% $Id: perf.tex,v 1.1.1.1 2005/07/25 20:21:39 cyilmaz Exp $
%\section{Performance Benchmarks}
% \section{Experimental Designs for Configuration Space Reduction}
% \label{perf-bench}

% Arvind, can you please merge this material into Section 2.3?!

In this section we present a new DCQA process whose aim is to improve
developer's visibility into the effect of software modifications on
system performance. We use BGML to specify the process and we use
Skoll to execute it.

When software systems change developers often run regression
tests to detect unintended functional side effects. In addition
to functionality, developers of performance\-intensive systems 
must also be wary unintended effects on performance. Thus, 
they will periodically run performance benchmarking tests
to detect such problems. 

As mentioned earlier these problem are compounded in highly
configurable systems. Although, these systems have a very large number
of valid configurations, our experience is that time and resource
constraints (and high change frequencies) severely limit the number of
configurations that can be examined.  For ACE+TAO, for example, only a
single default configuration is routinely benchmarked.  The result is
that developers get a very limited view of how their changes affect
the overall system and that problems not readily seen in the single
default configuration can escape detection until the system is
fielded.

To deal with this problem we propose a new DCQA process. This process
is carried out in two-phases. First, we execute a large-scale
formally-designed experiment across the Skoll grid. Here, we run
benchmarks on a wide-ranging, but sparsely distributed, set of
configurations. These configurations are chosen using a class of
experimental designs called screening designs~\cite{wu}. Screening designs
are highly economical and can reveal individual options 
that significantly affect performance (colloquially, first-order 
effects). The tradeoff is that they cannot reliably detect interaction
effects.  

{\bf Comments:}
\begin{itemize}
\item We may need to restate the last sentence in the paragraph
  above. The tradeoff is that in order to reduce the number of runs
  the screening experiments alias first-order effects with some
  higher-order effects. And the promise is that if the effect of these
  higher-order interactions are not significant one can reliably
  predict the first-order effects. 
\end{itemize}

Assuming relatively few first order effects exist, the process
continues executing on in-house resources. Now we exhaustively examine
all combinations of the first-order options, while randomly selecting
the settings for the remaining options.  Our hope is that by focusing
on only the first-order options, we can greatly reduce the
configuration space, while at the same time capturing a much wider
slice of the system's performance.

{\bf Comments:}
\begin{itemize}
\item We don't really choose the settings of the rest of the
  remaining options randomly. In fact, we choose their default settings.
\end{itemize}

%For example in our experiments, on ACE+TAO, we started with
%100 options, then ran tests for <XX> hours, collected results, and focused
%on 8 options.  Exhaustively tested (or reasonably small subset) all
%possible combinations of these 8 options in <YY> hours. Found that the
%performance range was reasonably well covered in much shorter time. Helped
%ACE+TAO developers find that their changes had unacceptable performance
%overhead for <MENTION SPECIFIC CONFIGURATIONS HERE> configurations.


%%%
% Arvind: To tie the BGML tool and screening experiments conducted by Skoll
% I have used the BGML use case scenario here to sort of motivate how the
% screening experiments can be generated from the modeling tool.
% Can folks please let me know if this makes sense?
% Thanks,
% Arvind
%%%

%%%%
% Arvind, I think we must move this somewhere where we show how BGML
% is combined with skoll in the context of screening first order
% configuration parameters - andy
%%%
%%%%
% Arvind: I am describing use case of BGML as the generation of the 
% screening experiments.
%%%




