% $Id: introduction.tex,v 1.22 2006/07/24 02:36:27 aporter Exp $

\section{Introduction}
%%%\vspace*{-0.2in}
\label{intro}
The quality of service (QoS) of many performance-intensive systems,
such as scientific computing systems and distributed real-time and
embedded (DRE) systems, depend heavily on various environmental
factors.  Example dependencies include the specific hardware and
operating system on which systems run, installed versions of
middleware and system library implementations, available language
processing tools, specific software features that are
enabled/\-disabled for a given customer, and dynamic workload
characteristics.  Many of these dependencies are not known until
deployment and some change frequently during a system's lifetime.

To accommodate these dependencies, users often need to tune
infrastructure and software applications by (re)adjusting many ({\em
i.e.}, dozens to hundreds) of compile- and run-time configuration
options that record and control variable software parameters.  These
options are exposed at multiple system layers, including compiler
flags and operating system, middleware, and application feature sets
and run-time optimization settings.  For example, there are $\sim$50
configuration options for SQL Server 7.0, $\sim$200 initialization
parameters for Oracle 9, and $\sim$90 core configuration options for
Apache HTTP Server Version 1.3.

Although designing performance-intensive systems to include such
configurations options promotes code reuse, enhnaces portability, and
helps end users improve their QoS, it also yields an enormous family
of ``instantiated'' systems, each of which might behave differently
and thus may need quality assurance (QA).  The size of these system
families creates serious and often under-appreciated challenges for
software developers, who must ensure that their decisions, additions,
and modifications work across this large (and often dynamically
changing) space.  For example, consider that:

\begin{itemize}
% Adam, Atif, it would be very helpful if we could cite some papers
% (ideally not our own) that substantiate these claims!
\item Option settings that maximize performance for a particular
environment may be ill-suited for different ones.  Failures can and do
manifest in some configurations, but not in others.  Similarly,
individual code changes can have different runtime effects in
different configurations.

\item Individual developers, especially those in smaller companies or
in open-source projects, may not have access to the full range of
hardware platforms, operating systems, middleware, class library
versions, etc. over which the system must run.  In these situations,
individual QA efforts will necessarily be incomplete.

\item Limited budgets, aggressive schedules, and rapidly changing code
bases mean that QA efforts frequently focus on a relatively small
number of system configurations, often chosen in an {\em ad hoc}
fashion.  Developers then unreliably extrapolate from this data to the
entire configuration space, which allows quality problems to escape
detection until systems are fielded.
\end{itemize} 

In summary, resources constraints and the large number of possible
system configurations make exhaustive evalution infeasible for
performance-intensive systems.  Developers therefore need (1) ways to
identify a small core set of configurations whose QA results can be
reliably generalized across all configurations and (2) support for
executing QA activities across a sufficiently rich and diverse set of
computing platforms.  To address these needs, our 
% Done: Adam, Atif, can you please cite several of our most relevant papers here?
research~\cite{ICSE05,ca,Schmidt:04b,cohenTSE06} has focused on system
support and algorithms for {\em distributed continuous quality
assurance}
% Adam, Atif, should we try to avoid using the DCQA acronym, i.e.,
% spell it out everywhere to make things earlier on the readers/reviewers?
(DCQA) processes.  DCQA helps improve software quality iteratively,
opportunistically, and efficiently by executing QA tasks continuously
across a grid of computing resources provided by end-users and
developer communities.

In prior work~\cite{Schmidt:04b}, we created a prototype DCQA support
environment called {\em Skoll} that helps developers create, execute,
and analyze their own DCQA processes, as described in
Section~\ref{skolloverview}.  To facilitate DCQA process development
and validation, we also developed {\em model-driven engineering} tools
for use with Skoll.  We then used Skoll to design and execute an
initial DCQA process, called ``main effects screening''~\cite{ICSE05},
whose goal was to estimate performance efficiently across all system
configurations (hereafter called the ``configuration space'').

Main effects screening borrows ideas from statistical quality
improvement techniques that have been applied widely in engineering
and manufacturing, such as Exploratory Data Analysis~\cite{tuk77},
Robust Parameter Design~\cite{wu}, and Statistical Quality
Control~\cite{sqc96}.  A central activity of these techniques is to
identify aspects of a system or process that contribute substantially
to outcome variation. We use similar ideas to identify important
configuration options whose settings define the distribution of
performance across all configurations by causing the majority of
performance variation.  Evaluating all combinations of these important
options (and randomizing the other options), thus provides an
inexpensive, but reliable estimate of performance across the entire
configuration space.

Although our initial work on main effects screening presented
in~\cite{ICSE05} showed promise, it also had several limitations.  For
example, the definition and execution of the process had many manual
steps.  To improve this, we have extended and better integrated
several model-driven 
% Doug, get this citation!
engineering (MDE)~\cite{Schmidt:06} tools, including the {\em Options
Configuration Modeling Language (OCML)}~\cite{OCML_ACMSE:04}, which
models configuration options and inter-option constraints, and the
{\em Benchmark Generation Modeling Language}
(BGML)~\cite{Schmidt:04k}, which models the QA tasks that observe and
measure QoS behavior under different configurations and workloads.
These MDE tools precisely capture common and variable parts of DCQA
processes and the software systems to which they are applied. They
also help reduce development and QA effort by generating configuration
files and many other supporting code 
% Doug, get this citation too!
artifacts~\cite{Kitty:06} needed to manage and control process
execution across heterogeneous computing resources.

Another limitation with our initial main effects screening process was
its dependence on strong and untested assumptions regarding the
absence of interactions among certain groups of options.  If these
assumptions do not hold in practice, our results could be wildly
incorrect.  Moreover, we had no way to assess the validity of the
assumptions without resorting to exhaustive testing, whose avoidance
motivated our DCQA process in the first place.

To remedy these problems, this paper describes further enhancements to
our earlier work that significantly broadens its applicability with
little additional operating costs.  Our new DCQA process, called
``reliable effects screening'' is implemented using Skoll and its MDE
tools, and relies on design-of-experiments (DoE) techniques called
``screening designs''~\cite{wu} and ``D-optimal
designs''~\cite{d-optimal}.  Our reliable effects screening process
first identifies a small subset of the most important
per\-for\-mance-rel\-ated configuration options by creating
formally-designed experiments and executing them across the Skoll
grid.  Whenever software changes occur thereafter, reliable effects
screening then uses a far smaller amount of computing resources to
estimate system performance across the entire configuration space by
exhaustively exploring all combinations of the important options,
while randomizing the rest.  This subsequent analysis can even be run
completely in-house, assuming appropriate computing platforms are
available, since the reduced configuration space is {\em much} smaller
than the original, and thus more tractable using only in-house
resources.
 
In addition to describing our new reliable effect screening DCQA
process, this paper also evaluates this process empirically on ACE,
TAO, and CIAO (\url{dre.vanderbilt.edu}), which are three widely-used,
production-quality, performance-intensive software frameworks.  This
evaluation indicates that (1) our reliable effects screening process
can correctly and reliably identify the subset of options that are
most important to system performance, (2) monitoring only these
selected options helps to detect performance degradation quickly with
an acceptable level of effort, and (3) alternative strategies with
equivalent effort yield less reliable results.  These results support
our contention that reliable effects screening can cheaply and quickly
alert performance-intensive system developers to changes that degrade
QoS, as well as provide them with much greater insight into their
software's performance characteristics.

The remainder of this paper is organized as follows:
Section~\ref{skolloverview} summarizes the Skoll DCQA environment;
Section~\ref{mes} describes how we extended Skoll to implement the new
reliable effect screening DCQA process to conduct performance-oriented
regression testing efficiently; Section~\ref{experiment} presents the
design and results of a feasibility study using ACE, TAO, and CIAO;
Section~\ref{guidelines} presents guidelines on how to use reliable
effect screening; Section~\ref{related} compares our research on
reliable effect screening with related work; and
Section~\ref{conclusions} evaluates threats to the validity of our
experiments and outlines future directions of our DCQA process and
tool research.
 
% LocalWords:  aporter inten sive mance middleware SQL DCQA Skoll OCML BGML QoS
% LocalWords:  ated
