% $Id: conclusions.tex,v 1.15 2006/07/16 22:03:43 atif Exp $
\section{Concluding Remarks}
\label{conclusions}

This paper presents a new {\em distributed continuous quality
assurance (DCQA)} process called ``reliable effects screening'' that
uses in-house and in-the-field resources to efficiently and reliably
detect performance degradation in performance-intensive systems that
have large configuration spaces.  The novelty of our approach stems
from its application of statistical quality control techniques to
efficiently identify a subset of system configurations that accurately
represents the performance of the entire configuration space.
Benchmarking this subset after a change provides a reliable estimate
of the distribution of performance across all configurations.
Unexpected changes in this distribution signal unintended side effects
that must be examined further.  

To evaluate the reliable effects screening process, we integrated it
with the Skoll DCQA environment and applied it to three large software
projects (ACE, TAO, and CIAO) using a grid of computing resources
managed by the Skoll DCQA environment.  Skoll's reliable effects
screening process is also supported by model-driven engineeering (MDE)
tools that control and manage its execution across a grid of in-house
and in-the-field computing resources.  The results of our experiments
in this environment indicated that:
\begin{itemize}
\item The reliable effects screening process helped developers detect
and understand performance bottlenecks across large configuration
spaces.

\item The ACE+\-TAO+\-CIAO developers used the information provided by
this process as a scalable defect detection aid, {\em i.e.}, when the
important options change unexpectedly (at recalibration time),
developers re-examined the frameworks to rapidly identify possible
problems with software updates.

\item ACE+\-TAO+\-CIAO developers also used information provided by
the reliable effects screening process to understand and arbitrate
disputes about subtle changes in framework and application performance
characteristics.
\end{itemize} 

In conclusion, we believe that this line of research is novel and
fruitful, though much R\&D remains to be done.  We are therefore
continuing to develop enhanced MDE-based Skoll capabilities and use
them to create and validate more sophisticated DCQA processes that
overcome the limitations and threats to external validity described
above.  In particular, we are exploring the connection between
design-of-experiments theory and the QA of other software systems with
large software configuration spaces.  We are also incorporating the
tools in the Skoll environment and the reliable effect screening
process into open-source software repositories, such as ESCHER
(\url{www.escherinstitute.org}).  Finally, we are conducting a much
larger case study using Skoll and reliable effect screening to
orchestrate the ACE+\-TAO+\-CIAO daily build and regression test
process with 200+ machines contributed by users and developers
worldwide, as shown on our online DCQA scoreboard at
\url{www.dre.vanderbilt.edu/scoreboard}.
   
\section{Acknowledgments}

This material is based on work supported by the US National Science
Foundation under NSF grants ITR CCR-0312859, CCF-0447864, CCR-0205265,
and CCR-0098158, as well as funding from BBN, Cisco, DARPA, Lockheed
Martin Advanced Technology Lab and Advanced Technology Center, ONR,
Qualcomm, Raytheon, and Siemens.  We thank the anonymous reviewers for
their comments that helped improve this paper.

% LocalWords:  aporter DCQA Skoll ITR CCR BBN


\iffalse
Skoll's MDE tools allowed visual composition of the experiments. This
allows QA engineers to focus on the execution and analysis of QA tasks
rather than low level details of synthesizing valid middleware
configuration and benchmarking code.
\fi
