% $Id: abstract.tex,v 1.14 2006/07/16 22:03:43 atif Exp $ 

Developers of highly configurable performance-intensive software
systems often use in-house performance-o\-ri\-ented ``regression
testing'' to ensure that their modifications do not adversely affect
their software's performance across its large configuration space.
Unfortunately, time and resource constraints can limit in-house
testing to a relatively small number of possible configurations,
followed by unreliable extrapolation from these results to the entire
configuration space.  As a result, many performance bottlenecks escape
detection until systems are fielded.

In our earlier work, we improved the situation outlined above by
developing an initial quality assurance process called ``main effects
screening''.  This process (1) executes formally designed experiments
to identify an appropriate subset of configurations on which to base
their performance-oriented regression testing, (2) executes benchmarks
on this subset whenever the software changes, and (3) provides tool
support for executing these actions on in-the-field and in-house
computing resources.  Our initial process had several limitations,
however, since it was manually configured (which was tedious and
error-prone) and relied on strong and untested assumptions for its
accuracy (which made its use unacceptably risky in practice).

This paper presents a new quality assurance process called ``reliable
effects screening'' that provides three significant improvements to
our earlier work.  First, it allows developers to economically verify
key assumptions during process execution.  Second, it integrated
several model-driven engineering tools to make process configuration
and execution much easier and less error prone.  Third, we evaluated
this process via several feasibility studies of three large,
widely-used per\-for\-mance-intensive software frameworks.  Our
results indicate that reliable effects screening can detect
performance degradation in large-scale systems more reliably and with
significantly less resources than conventional techniques.

% LocalWords:  QoS DCQA
