\section{Discussion of Feasibility Study} 
\label{motex}

\subsection {Threats to Validity}
\label{validity-threats}

All empirical studies suffer from threats to their internal and
external validity.  For the experiments described in
Section~\ref{experiment}, we were primarily concerned with threats to
external validity since they limit our ability to generalize the
results of our experiment to industrial practice.  One potential
threat is that several steps in our process require human decision
making and input.  For example, developers must provide reasonable
benchmarking applications and must also decide which effects they
consider important.

Another possible threat to external validity concerns the
representativeness of the ACE+\-TAO+\-CIAO subject applications, which
are an integrated suite of software (albeit a very large suite with
over 2M+ lines of code).  A related issue is that we have focused on
the subset of the entire configuration space of ACE+\-TAO+\-CIAO that
only has binary options and has no inter-option constraints.  While
these issues pose no theoretical problems (screening designs can be
created for much more complex situations), there is clearly a need to
apply reliable effects screening to larger and richer configuration
spaces in future work to understand how well the process scales.

Another potential threat is that for the time period we studied, the
ACE+\-TAO+\-CIAO subject frameworks were in a fairly stable phase.  In
particular, changes were made mostly to fix bugs and reduce memory
footprint, but the software's core functionality was relatively
stable.  For time periods where the core software functionality is in
greater flux, {\em e.g.}, in response to new requirements from
sponsors or efforts to port the software to new and different
platforms, it may be harder to distinguish significant performance
degradation from normal variation.

\subsection{Hypotheses}
\label{mdd-benefits}

Despite the limitations described in Section~\ref{validity-threats},
we believe the study presented in Section~\ref{experiment} supports
our basic hypotheses presented in Section~\ref{design}.  We reached
this conclusion by noting that our study suggests that the (1)
MDE-based Skoll system allows QA engineers to quickly construct
complex DCQA processes, (2) reliable effects screening processes
provides developers with fast, cheap and reliable estimates of
performance across a system's entire configuration space; and (3)
developers of ACE+TAO+CIAO believe the technique provides them with
important development information.

\noindent{\bf Benefits from applying our MDE tools.} Our MDE tools
helped improve the productivity of QA engineers by allowing them to
create QA task models and to compose benchmarking experiments {\em
visually} rather than wrestling with low-level formats and source
code.  These tools thus resolve tedious and error-prone accidental
complexities associated with writing correct code by auto-generating
them from higher level models. For example, Table~\ref{ccmperfgen}
summarizes the BGML code generation metrics for a particular
configuration.

\begin{table}[h]
\center{
\caption {Generated Code Summary for BGML}
\begin{tabular}{|c|c|c|c|}\hline
{\bf Files} & {\bf Number} & {\bf Lines of Code} & {\bf Generated (\%)}
\tabularnewline \hline
    IDL & 3 & 81 & 100 \tabularnewline \hline
Source (.cpp) & 2 & 310 & 100 \tabularnewline \hline
 Header (.h) & 1 & 108 & 100 \tabularnewline \hline
 Script (.pl) & 1 & 115 & 100 \tabularnewline \hline
                                                                                            
\end{tabular}
\label{ccmperfgen}
}
\end{table}
This table shows how BGML automatically generates 8 of 10 required
files that account for 88\% of the code required for the experiment.
Since these files must capture specific information for each
configuration, these tools imply large improvements in productivity
for performing benchmarking QA tasks.  Similarly, OCML enabled us to
generate both syntactically and semantically correct middleware
configurations, thereby eliminating accidental complexity in
generating middleware configurations.

\noindent 
{\bf Reliable effects screening.}  Our experiments showed that the
reliable effects screening process was fast, cheap, and effective. We
came to this conclusion by noting that:
\begin{enumerate}
\item Screening designs can correctly identify important options
(Sections~\ref{step1} -- \ref{validate}).

\item These options can be used to produce reliable estimates of
performance quickly across the entire configuration space at a
fraction of the cost of exhaustive testing (Section~\ref{step2}).

\item The alternative approach of random or {\em ad hoc} sampling can
give highly unreliable results (Section~\ref{random}).

\item The reliable effects screening process detected performance
degradation on a large and evolving software system
(Section~\ref{inpractice}).

\item The screening suite estimates were significantly more precise
than the {\em ad hoc} process currently used by the developers of
ACE+\-TAO+\-CIAO (Section~\ref{inpractice}).
\end{enumerate}

\noindent {\bf User acceptance.}  Informally, we found that
ACE+\-TAO+\-CIAO developers have been quite happy with the results of
our experiments described in Section~\ref{experiment}.  As we move
towards fully integrating reliable effects screening into their
development processes they continue to find new ways in which this
information can help them improve their development processes,
including:

\noindent {\bf\em Using option importance to prioritize work.}  Our
quantitative results showed that options $L$, $M$, and $N$ did not
have a strong effect on latency and throughput.  These findings
surprised some ACE+TAO+CIAO developers, who had spent considerable
time optimizing code affected by these options.  Further investigation
showed that the options can have a somewhat larger effect, but only in
very specific circumstances.  The developers now see reliable effects
screening as a means to better understand how widespread the effects
of different pending changes may be.

\noindent {\bf\em Using changes in option importance to detect bugs.}
Prior to the release of an ACE+\-TAO+\-CIAO beta, developers noted a
significant ($\sim$25\%) drop in performance.
\iffalse The
degradation in performance was observed during a two week time windows
(between last week of August and first two weeks of September 2005).
\fi 
Since reliable effects screening had not yet been fully integrated
into their development processes, the ACE+\-TAO+\-CIAO developers fell
back on traditional {\em ad hoc} QA and debugging techniques.  When
they failed to identify the problem's cause, they froze their CVS
repository and used an exhaustive QA approach to painstakingly narrow
down the change that had degraded performance.

Ultimately they found the change that caused the degradation was a
feature addition that enabled TAO+\-CIAO to support the IPv6 protocol.
Specifically, they observed that a modification to TAO's connection
handler had degraded performance.  Interestingly, this code is
controlled by option $B$.  In retrospect, a reliable effects screening
would have shown a dramatic change in the importance of option $B$,
thus enabling the ACE+\-TAO+\-CIAO developers to localize the problem
quickly, {\em i.e.}, because one setting of option $B$ triggered the
buggy code, option $B$'s effect would have been 2.5 times greater
after the change than before it.  Based on this experience, the
developers now see added value in frequently recalibrating the
important options to alert them to changes in option importance.

\section{Usage Guidelines}
\label{guidelines}

This section focuses on providing some guidelines on how to use the
Skoll-based reliable effects screening process.  We examine how to
select an appropriate resolution and size for the screening designs,
how to identify the important options, and how to validate the basic
assumptions in identifying the important options.  We also summarize
on our experience to date applying this process on the
ACE+\-TAO+\-CIAO software frameworks.

\noindent {\bf Step 1: Choose the resolution.} Leverage {\em a priori}
knowledge of the software being tested, if it is available, to decide
the resolution of the screening experiment, {\em i.e.}, which
high-order effects are considered important vs. negligible.  If no or
limited {\em a priori} information is available, use screening
experiments in an iterative manner ({\it e.g.,} going from lower
resolutions to higher ones) to obtain this information.
Section~\ref{design} illustrated how we did this for the
ACE+\-TAO+\-CIAO software frameworks, where we selected 14 specific
options to explore based prior knowledge of a recent change.

\noindent {\bf Step 2: Choose the size.} Depending on available
resources and how fast the underlying software changes, determine the
maximum number of observations allowed in the screening experiment.
Note that the resolution of the experiment chosen in step (1) may
dictate the minimum size of the experiment.  If this size is not
feasible, consider lowering the resolution of the experiment by
carefully choosing the aliasing structure so that no potentially
important higher-order effects are aliased with lower-order ones.
Sections~\ref{step1} and~\ref{extensions} illustrate how we did this
for the ACE+\-TAO+\-CIAO software frameworks, where we created three
resolution IV designs with run sizes of 32, 64, and 128, and where we
created one resolution VI design with a run size of 2,048.

\noindent {\bf Step 3: Identify the important options.} After the
screening design is computed, conducted, and analyzed, use the
half-normal probability plots described in Section~\ref{step1} to
identify important options.  If no effects are important, these plots
will show a set of points on a rough line near $y=0$.  Any substantial
deviations from this line indicate important options.  Depending on
the benchmarking test and the desired precision of the performance
estimates, decide how large effects must be to warrant attention.
Section~\ref{step1} shows how we did this for the ACE+\-TAO+\-CIAO
software frameworks, where we identified 2 important and 3 arguably
important options.

\noindent {\bf Step 4: Validate the basic assumptions.} If needed,
validate the assumptions imposed by the choice of the resolution. Use
D-optimal designs described in Section~\ref{validate} to augment the
screening experiment to (1) dealias the important options and (2)
identify remaining higher-order effects.  Section~\ref{validate}
illustrated how we did this for the ACE+\-TAO+\-CIAO software
frameworks and showed that our basic assumptions helped and that our
initial analysis was therefore reliable.

\noindent {\bf Step 5: Estimate performance after changing software.}
Focusing on important options allows developers to reduce the
effective configuration space significantly by evaluating all
combinations of the important options, while randomizing the rest.
Section~\ref{step2} illustrated how we did this for the
ACE+\-TAO+\-CIAO software frameworks and showed that our approach gave
reliable estimates of performance across the entire configuration
space using only 40 seconds (for top-2 suite) or 2 minutes (for top-5
suite) of CPU time.

\noindent {\bf Step 6: Frequently recalibrate important options.}  The
importance of different options may change as software in a system
changes.  We therefore recommend frequent recalibration of the
important effects.  Although our feasibility study in
Section~\ref{experiment} does not show the need for recalibration, our
experience applying reliable effects screening to ACE+\-TAO+\-CIAO
over time indicates that recalibration is essential.
   
% LocalWords:  priori dealias
