% $Id: skoll_summary.tex,v 1.26 2006/07/24 02:36:28 aporter Exp $
\section {An Overview of the Skoll DCQA Environment} 
\label{skolloverview}

To improve the quality of per\-for\-mance-inten\-sive systems across
large configuration spaces, our work focuses on {\em distributed
continuous quality assurance} (DCQA) processes~\cite{Schmidt:04b} that
evaluate various software qualities, such as portability, performance
characteristics, and functional correctness ``a\-round-the-world and
around-the-clock.''\footnote{Naturally, DCQA processes can also be
executed effectively in more constrained and smaller-scale
environments, such as company-wide intranets.}  To support this
methodology, we developed {\em Skoll}, which is a {\em model-driven
engineering} (MDE)-based DCQA environment
(\url{www.cs.umd.edu/projects/skoll}).  Skoll divides QA processes
into multiple {\it tasks}, each of which is implemented as a generic
process parametrized by one of several alternative configurations
expressed via MDE tools.  Example tasks might include running
regression tests in one of many system configurations, evaluating
system response time under one of several different input workloads,
or measuring code execution coverage using one of several
instrumentation schemes.
\begin{figure}[tbhp]
\centering \epsfig{file=figures/SkollTasksAndSubTasks.eps,scale=0.95}
\caption{The Skoll Distributed Continuous Quality Assurance Architecture} 
\label{fig:dcqa}
\end{figure} 
As shown in Figure~\ref{fig:dcqa}, these tasks are then intelligently
and continuously distributed to -- and executed by -- clients across a
grid of computing resources contributed by end-user and developer
communities. The results of individual tasks are returned to servers
at central collection sites, where they are merged and analyzed to
steer subsequent iterations and ultimately to complete the overall QA
process.

This section summarizes Skoll's key components and services, which
include MDE tools for modeling system configurations and their
constraints, a {\em domain-specific modeling language} (DSML) to
describe these models, algorithms for scheduling and remotely
executing tasks via planning technology that analyzes task results and
adapts the DCQA process in real time, a DSML to package the subtasks,
and techniques to interpret and visualize the results.

\noindent{{\bf QA task space.}}  Performance-intensive systems, such
as the ACE+\-TAO+\-CIAO QoS-enabled middleware, provide a range ({\em
i.e.}, $\sim$500) of configuration options that can be used to tune
its behavior.\footnote{All the ACE+TAO+CIAO's configuration options
are described at
\url{www.cs.wustl.edu/~schmidt/ACE_wrappers/TAO/docs/Options.html}.}
To be effective, DCQA processes must keep track of these options, in
addition to other environmental information, such as OS platform,
build tools used, and desired version numbers.  This information is
used to parameterize generic QA tasks and aids in planning the global
QA process, {\em e.g.}, by adapting the process dynamically and
helping interpret the results.

%
\iffalse
The cornerstone of Skoll is therefore its formal model of a DCQA
process' QA task space.  This model captures different configuration
options, environmental characteristics, their allowable settings, 
and any constraints among these. 
\fi
%

In Skoll, tasks are generic processes parameterized by QA task
options.  These options capture information that is (1) varied under
QA process control or (2) needed by the software to build and execute
properly.  These options are generally application-specific, including
workload parameters, operating system, library implementations,
compiler flags, or run-time optimization controls.  Each option must
take its value from a discrete number of settings.  For example, in
% Done: Adam, Atif, can you please add a citation for this other work?
other work, our QA task model include a configuration option called
{\tt OperatingSystem} so Skoll can select appropriate binaries and
build code for specific tasks~\cite{cohenTSE06}.

\iffalse                 % Commenting it out from here
% Folks, I do not think we use this anywhere else and this probably
% does not add to the paper since our goal is to give a quick overview
% - Andy
Defining a subtask involves mapping each option to one of its
allowable settings.  We call this mapping a {\em configuration} and
represent it as a set $\{$ $(V_1, C_1)$, $(V_2, C_2)$, $\ldots$,
$(V_N, C_N)$ $\}$, where each $V_i$ is an option and $C_i$ is its
value, drawn from the allowable settings of $V_i$.  In practice, not
all configurations are semantically useful, \eg, feature X may not be
supported on operating system Y.  We therefore allow {\it inter-option
constraints} that limit the setting of one option based on the setting
of another.

Skoll represents constraints as ($P_i \rightarrow P_j$), meaning ``if
predicate $P_i$ evaluates to $TRUE$, then predicate $P_j$ must
evaluate to $TRUE$.''  A predicate $P_k$ can be of the form $A$, $\neg
A$, $A \& B$, $A | B$, or simply ($V_i = C_i$), where $A$, $B$ are
predicates, $V_i$ is an option and $C_i$ is one of its allowable
values.  A {\em valid configuration} is a configuration that violates
no inter-option constraints.
\fi                      % Commenting it out to here

\noindent{{\bf QA task modeling.}}  The QA task model underlies the
DCQA process.  Our experience~\cite{Schmidt:04b,Schmidt:04k,ca} with
the initial Skoll prototype taught us that building these models
manually was tedious and error-prone.  We therefore developed and
integrated into Skoll the {\em Options Configuration Modeling Language
(OCML)}~\cite{OCML_ACMSE:04}. OCML is an MDE tool that provides a DSML
for modeling software configurations.  For example, OCML defines a
{\em numeric option} type for middleware options that can have numeric
values, {\em e.g.}, cache, buffer, or thread pool sizes.  OCML is
built atop the {\em Generic Modeling Environment (GME)}~\cite{GME:01},
which provides a meta-programmable framework for creating DSMLs and
generative tools via {\em metamodels} and {\em model interpreters}.
For the feasibility study in Section~\ref{experiment}, we used the
OCML MDE tool to define the configuration model visually and generate
the low-level formats used by other Skoll components.

\iffalse                 % Commenting it out from here
% probably don't need these details. - andy
We applied OCML to model and capture the various configuration
parameters provided by ACE+\-TAO+\-CIAO.  Figure~\ref{fig:ocml} shows
the TAO ORB configuration options modeled with OCML.
\begin{figure}[htpb]
\centerline{\epsfxsize=3in,\epsffile{figures/rule2.eps}}
\caption{OCML Modeling Paradigm Applied to TAO ORB}
\label{fig:ocml}
\end{figure}
Inter-option dependencies are captured as model constraints. For
instance, Figure~\ref{fig:ocml} shows that when the {\tt
ORBConnectionPurgingStrategy} is set to value {\tt NULL}, then neither
{\tt ORB\-Con\-nectionCacheMax} nor {\tt
ORBConnectionCachePurgePercentage} can be declared or given a
user-specified value.  OCML validates the modeled middleware configurations
and generates configuration files that can be used to configure the
middleware.  
For the feasibility study in Section~\ref{experiment}, we
used the OCML MDE tool to visually define the configuration model and
generate the low-level formats used by other Skoll components.
\fi                      % Commenting it out to here

\noindent{{\bf Exploring the QA task space.}} Since the QA task spaces
of many systems can be enormous, Skoll contains an {\em Intelligent
Steering Agent} (ISA) \cite{Schmidt:04b} that uses AI planning
techniques~\cite{Memon01c} to distribute QA tasks on available Skoll
clients.  When clients become available they send a message to the
Skoll server.  Skoll's ISA then decides which task to assign it by
considering many factors, including (1) {\em the QA task model}, which
characterizes the subtasks that can be assigned legally, (2) {\em the
results of previous tasks}, which capture what tasks have already been
done and whether the results were successful, (3) {\em global process
goals}, such as testing popular configurations more than rarely used
ones or testing recently changed features more heavily than unchanged
features, and (4) {\em client characteristics and preferences}, {\em
e.g.}, the selected configuration must be compatible with the OS
running on the client machine or users can specify preferences that
configurations must run with user-level -- rather than superuser-level
-- protection modes.

After a valid configuration is chosen, the ISA packages the
corresponding QA task into a {\em job configuration}, which consists
of the code artifacts, configuration parameters, build instructions,
and QA-specific code ({\em e.g.}, developer-supplied
regression/\-performance tests) associated with a software pro\-ject.
Each job configuration is then sent to a Skoll client, which executes
the job configuration and returns the results to the ISA.  By default,
the ISA simply stores these results.

In some experiments, however, we want to learn from incoming results.
For example, when some configurations prove faulty, it makes no sense
to retest them. Instead, we should refocus resources on other
unexplored parts of the QA task space.  When such dynamic behavior is
desired, DCQA process designers develop customized {\it adaptation
strategies} that Skoll uses to monitor the global process state,
analyze it, and use the information to modify future task assignments
in ways that improve process performance.  One example of an
adaptation strategy is the {\it nearest neighbor search strategy},
which allows a process to target failing QA task subspaces by
preferentially testing the ``neighbors'' of a failing configuration,
({\em i.e.}, other similar configurations that differ in one
configuration option value) to see if they also
fail~\cite{Schmidt:04b}.

\noindent{{\bf Packaging QA tasks.}} With the initial Skoll prototype,
developers who wanted to evaluate QoS issues had to provide
hand-written benchmark programs.  For example, ACE+TAO+CIAO developers
creating such benchmarks to measure latency and throughput for a
particular workload had to write (1) the header files and source code
that implement the measurements, (2) the configuration and script
files that tune the underlying ORB and automate running tests and
output generation, and (3) project build files ({\em e.g.}, makefiles)
required to generate executable binaries from source code.  Our
initial feasibility study~\cite{Schmidt:04b} revealed that this
process was tedious and error-prone.

To address these problems, we developed the {\em Benchmark Generation
Modeling Language (BGML)}~\cite{Schmidt:04n}, which is an MDE tool
that automates key QoS evaluation concerns of QoS-enabled middleware
and applications, such as (1) modeling how distributed system
components interact with each other and (2) representing metrics that
can be applied to specific configuration options and platforms.
Middleware/application developers can use BGML to graphically model
interaction scenarios of interest.  BGML automates the task of writing
repetitive source code to perform benchmark experiments and generates
syntactically and semantically valid source and benchmarking code.

\iffalse                 % Commenting it out from here
To model interactions, BGML provides (1) {\em test elements}, such as
operations, return-types, latency and throughput, that can be used to
represent generic operations or a sequence of operation steps and
associate functional QoS properties with them and (2) {\em workload
elements}, such as tasks and task-set that could be used to model and
simulate background load present during the experimentation process.
BGML maps these workload elements to individual platform-specific code
in the interpretation process.
\fi                      % Commenting it out to here

\begin{figure}[htpb]
\centerline{\epsfxsize=5.1in\epsffile{figures/BGML-Operation.eps}}
\caption{Associating QoS with an Operation in BGML} 
\label{bgml-operation}
\end{figure}
Figure~\ref{bgml-operation} depicts how QA engineers can visually
configure an experiment that measures end-to-end latency.  As shown in
the figure, the latency metric was associated with an operation ({\tt
WorkOrderResponse()}) using BGML.  BGML's {\tt TaskSet} element was
also used to create background tasks that invoked the {\tt
WorkOrderResponse()} operation continuously for a fixed number of
iterations.

\noindent{{\bf Analysis of results.}} Since DCQA processes can be
complex, Skoll users often need help to visualize, interpret, and
leverage process results.  Skoll therefore supports a variety of
pluggable analysis tools, such as Classification Tree Analysis
(CTA)~\cite{CART}.  In previous work~\cite{Schmidt:04b,ca}, we used
CTA to diagnose options and settings that were the likely causes of
specific test failures.  For the work presented in this paper, we
developed statistical tools to analyze data generated by the
formally-designed experiments described next.
