\section {Related Work}
\label{related}
%%%%
% Arvind: Folks, can we need to add related work for DCQA techniques and also
% for DOE theory?
%%%%

This section compares our work on  with other related research
efforts, including applying DOE to software engineering, large-scale
testbed environments, feedback-based optimization techniques, and
generative techniques for synthesizing benchmarks.

\sqzpar\paragraph* {Applying DOE to software engineering} The use of
design-of-experiment (DoE) theory within software engineering has
largely focused on {\em interaction testing}, where the goal is to
compute and (sometimes) generate minimal test suites that cover all
combinations of specified program inputs, typically by computing
orthogonal arrays or covering arrays.  Some examples of this work
include Mandl \cite{mandl}, Brownlie {\em et al.} \cite{oats}, Dalal
{\em et al.}~\cite{dalal}, Burr {\em et al.}, Dunietz {\em et al.},
and Kuhn {\em et al.} Yilmaz {\em at al.}~\cite{ca} used covering
arrays as a configuration space sampling technique to support the
characterization of failure-inducing option settings.

\sqzpar\paragraph* {Large-scale benchmarking testbeds}
EMULab~\cite{White02} is a test\-bed at the University of Utah that
provides an environment for experimental evaluation of networked
systems.  EMULab provides tools that researchers can use to configure
the topology of their experiments, {\em e.g.}, by modeling the
underlying OS, hardware, and communication links.  This topology is
then mapped~\cite{EMULab:03} to $\sim$250 physical nodes that can be
accessed via the Internet.  The EMULab tools can generate script files
that use the Network Simulator (NS) (\url{www.isi.edu/nsnam/ns/})
syntax and semantics to run the experiment.  The reliable effect screeing project provides
a superset of EMULab that is not limited by resources of a single
testbed, but instead can leverage the large amounts of end-user
computer resources in the Skoll grid.  Moreover, the BGML model
interpreters can generate NS scripts to integrate our benchmarks with
experiments in EMULab.

\sqzpar\paragraph* {Feedback-driven optimization techniques}
Feedback-dri\-ven techniques involve using feedback control
loops~\cite{Lu:02} to adapt QoS measures. Our approach in reliable effect screeing uses
screening experiments to determine first order configuration
parameters impacting performance.  These results are then fed back
into performance models to provide information on the consequences of
mixing and matching configuration options at model building time.

The continuous compilation strategy~\cite{Childers:03} combines
aspects of offline (use program analysis to improve compiler-generated
code) and online analysis (where feedback control is used to
dynamically adapt QoS measures).  This strategy constantly monitors
and improves application code using code optimization techniques.
These optimizations are applied in four phases including (1) {\em
static analysis}, in which information from training runs is used to
estimate and predict optimization plans, (2) {\em dynamic
optimization}, in which monitors apply code transformations at
run-time to adapt program behavior, (3) {\em offline adaptation}, in
which optimization plans are actually improved using actual execution,
and (4) {\em recompilation}, where the optimization plans are
regenerated.  reliable effect screeing's model-based strategy can enhance conventional
hybrid analysis by tabulating platform-specific and
platform-independent information separately using the Skoll framework,
which does not incur the overhead of system monitoring since behavior
does not change at run-time.  New platform-specific information
obtained can be fed back into the models to optimize QoS measures.

\sqzpar\paragraph* {Generative techniques for synthesizing benchmarks}
There have been a several initiatives that use generative techniques
similar to reliable effect screeing for generating test-cases and benchmarking for
performance evaluation.  The MODEST~\cite{Rutherford:03} tool provides
a generative approach for producing (1) test cases, {\em i.e.},
test-code that is used to test the system and (2) test-harness, {\em
i.e.}, the scaffolding code required for test setup and tear down. In
this system, test cases are generated in parallel with the actual
system to provide users with the system and the test-code to reduce
maintenance costs.

The BGML MDD tool in reliable effect screeing focuses on empirical performance evaluation
rather than just test cases generation since generating functional
test cases may be amenable only to systems that can capture all inputs
to the system.  The MODEST approach captures this information via a
domain specification supplied as XML input.  In MODEST, all artifacts
conform to the same architecture, with variability only in the domain
specification given by the user.  This assumption rarely holds,
however, for performance-intensive systems, which often run on
heterogeneous combinations of OS, compiler, and hardware platforms.
In contrast, our MDD-based hybrid DCQA approach to benchmarking via
the reliable effect screeing process 
allows QA engineers to fully characterize the inputs to the system and
observe QoS behavior and variations across a wide range of platforms.
