% $Id: evaluation.tex 19213 2009-08-17 10:06:43Z gokhale $
\sqzlarge
\section{Validating QUICKER-generated QoS Configurations}
\label{sec:eval}

This section validates the effectiveness of the QUICKER transformation
algorithms in meeting the QoS requirements of DRE systems.  We
demonstrate these validations in the context of a DRE system as a
proof-of-concept.  Extensive user studies are necessary to provide more
validation of the approach's effectiveness.

First we present a representative DRE system case study we used for the
evaluation. We then describe how we have applied Bogor model-checking
tool for verification of the generated configurations. Finally, through
empirical evaluation, we validate the generated QoS options. 

%\input{scenario}

\subsection{Validating BasicSP QoS Configuration Dependencies via Model-checking}
\label{sec:mcheck-basicsp}

The QoS extensions to the BIR format described in
Section~\ref{sec:modelcheck} are used in maintaining and resolving
dependencies between application components.  For example, consider a
real-time configuration of BasicSP scenario in which each of the
\emph{GPS}, \emph{AirFrame}, and \emph{NavDisplay} components are
configured to have \texttt{priority bands} for separate service
invocation priorities and the \emph{Timer} component is configured to
support multiple priority levels during generation of pulse events.
Given such a configuration, we have that \texttt{priority  band} values
at \emph{GPS} (client) component must match \texttt{ThreadPoolLanes} at
\emph{Timer} (server) component \emph{i.e.}, a direct configuration
dependency exists between these two components.

Further, since the pulse events are subsequently reported to
\emph{AirFrame} and \emph{NavDisplay} components there is a similar
indirect dependency between \texttt{band} values at these components and
\texttt{lanes} at  \emph{Timer} component. The dependency structure of
BasicSP scenario is maintained in QoS extensions to track such
dependencies between QoS options.  

Figure~\ref{fig:dependency} represents the dependency structure
generated using QoS extensions with the given configurations for our
BasicSP scenario. Occurrences of change in configurations of either of
the dependent components are followed by detection of potential
mismatches such that all dependencies are exposed and resolved during
application QoS design iterations. 

\begin{figure}[htb]
  \centering
  \includegraphics[width=\columnwidth,keepaspectratio]{dependency}
  \bfcaption{Dependency structure of BasicSP. $L_{c}$ denotes threadpool lane and
   $B_{c}$ denotes priority bands at component $c$. {\sc sd} and {\sc cp} indicate
   the {\sc server\_declared} and {\sc client\_propagated} priority models, respectively.}
  \label{fig:dependency}
\end{figure}

A similar approach can be carried out if the DRE system has multiple
different modes of operation, which in turn define multiple different
regions of operation.  Each such operating region may then have a set of
configuration options.  The Bogor-based model checking approach can now
verify the dependencies among options for connected components for
multiple modes of operation.  The verified sets of configurations can
then be used by the middleware to seamlessly adapt from one set to
another depending on the mode.

\subsection{Empirically Evaluating BasicSP QoS Configurations}
\label{sec:validate-basicsp}

In this section we empirically validate the effectiveness of the
generated QoS configurations for the BasicSP case study.

\noindent\textbf{Experiment Configuration.} We have used ISISLab 
(\url{www.dre.vanderbilt.edu/ISISlab}) for evaluating observed QoS
properties of DRE systems based on middleware QoS configurations
generated using our configuration process.  Each of the physical nodes
used in our experiments was a 2.8 GHz Intel Xeon dual processor, 1 GB
physical memory, 1 GHz network interface, and 40GB hard disks. Version
0.6 of our RT-CCM middleware CIAO was used running on Redhat Fedora Core
release 4 with real-time preemption patches. The processes that hosted
BasicSP components were run in the POSIX scheduling class SCHED\_FIFO,
enabling first-in-first-out scheduling semantics based on the priority
of the process.  

\begin{table}[htb]
\centering
\bfcaption{Generated QoS configuration for BasicSP}
\sqzlarge
\label{table:output}
\footnotesize
\begin{tabulary}{\columnwidth}{|C|C|C|C|C|}
  \hline
  \textbf{QoS configuration} & \textbf{Timer} & \textbf{GPS} & \textbf{Airframe} \\
  \hline \hline
  PriorityModel & {\sc sd} & {\sc cp} & {\sc cp} \\
  \hline
  \underline{ThreadPool} & & & \\
  stacksize & 1024 & 1024 & 1024 \\
  max\_buff\_reqs. & -- & 20 & 20 \\
  allow\_borro. & {\sc false} & {\sc false} & {\sc false} \\
  allow\_req\_buff & {\sc false} & {\sc true} & {\sc true} \\  
  \hline
  \underline{Lane} & & & \\
  static\_thrds & 4 & 8 & 8 \\
  dyna\_thrds & 0 & 0 & 0 \\
  \hline
\end{tabulary}
\vspace{-0.2in}
%\end{singlespace}
\end{table}

As the first step, we modeled BasicSP QoS requirements using the
requirements DSML described in Section~\ref{sec:transform}.
\texttt{bu\-rs\-ty\_cl\-ie\-nt\_re\-qu\-es\-ts} was set to \textsc{false}
for all components and
\texttt{fi\-xed\_pri\-ori\-ty\_ser\-vi\-ce\_ex\-ec\-uti\-on} attribute
was set to \textsc{false} for every component except 
\emph{Timer}.  Secondly, we applied our model transformation algorithm to the
requirements model above for generating detailed application configurations.
Table~\ref{table:output} captures some of the important QoS configurations
generated in our process. These configurations are represented as an 
application model.  In the final step, we apply model interpreters for
synthesizing descriptors required to configure the functional and QoS
properties of the application during deployment.


\begin{figure*}[bht]
\centering
\subfloat[End-to-end latency measurements]
  {
    \includegraphics[width=3.2in,keepaspectratio]{latency_work}%
    \label{fig:latency-work}
  }
\subfloat[Statistical dispersion of latency]
  {
    \includegraphics[width=3.2in,keepaspectratio]{sd_work}%
    \label{fig:sd-work}
  }
\bfcaption{Evaluating BasicSP QoS configurations against increasing workload at a constant 20Hz
         invocation rate.}
\label{fig:workload}
\end{figure*}

In evaluating effectiveness of our configuration process, we collected 
end-to-end latency measurements between
\emph{Timer} and \emph{NavDisplay} components. Earlier in 
Section~\ref{sec:modelcheck} we discussed how correctness of QoS
options can be verified using our process, the first experiment discussed 
below empirically evaluates the effectiveness of these options in meeting 20Hz 
operational display refresh rate of BasicSP from low to high workload conditions. 
Further, operational conditions of DRE system might change (unfavorably) during its 
execution. In order to evaluate the tolerance of our generated configurations
under such conditions, in the second experiment we measure the metrics discussed 
above when invocation rate is steadily increased. Each of these experiments 
were performed for a constant time period and after executing 10,000 warmup 
iterations. 

\noindent\textbf{Experiment 1: Increasing System Workload.}
Figure~\ref{fig:workload} plots the latency 
measurements under increasing system workload. The workload is 
characterized~\cite{Schmidt:02r} as a function performed with every 
client invocation.  The signature of the function is given
as:  $void\;\; work (int\; units);\;$ where $units$ argument specifies the
amount of processor intensive work performed per call.
The experiment was run for workload values of 10 through 80. End-to-end latency was observed to be
at an average value of $\sim$1925 as can be seen in Figure~\ref{fig:latency-work}.
Further, successive event-driven computations in the scenario exhibit an almost constant time
complexity, indicated by relatively small dispersion in latencies as plotted in
Figure~\ref{fig:sd-work}. 

\noindent\textbf{Experiment 2: Increasing System Invocation Rate.}
Performance of the generated configurations for BasicSP is given in 
Figure~\ref{fig:invc-rate}.  Throughout this experiment the rate of 
invocation was increased from a normal operational value of 20Hz to
a maximum of 40000Hz.  Latency results are shown in 
Figure~\ref{fig:latency-rate} which plots maximum, mean and minimum delay 
measurements for each invocation rate data point.  Even with increasing 
rate the mean latency did not change significantly and was observed 
to be consistently just above 1900 microsec. for the entire range of invocation
rates. Note that this is a desirable characteristic since even with an
unfavorable change in operational conditions (\emph{i.e.}, change in
invocation rate) the latency was observed to be constant. 

\begin{figure*}[htb]
\centering
\subfloat[End-to-end latency measurements]
  {
    \includegraphics[width=3.2in,keepaspectratio]{latency_rate}%
    \label{fig:latency-rate}
  }
\subfloat[Statistical dispersion of latency]
  {
    \includegraphics[width=3.2in,keepaspectratio]{sd_rate}%
    \label{fig:sd-rate}
  }
\vspace{-0.1in}
\bfcaption{Evaluating BasicSP QoS configurations against increasing
  invocation rate: All the plots use logarithmic X axis and linear Y
  axis.} 
\label{fig:invc-rate}
\end{figure*}

Jitter in latencies for each invocation rate is plotted in
Figure~\ref{fig:sd-rate} which shows that the deviation is bound between
a high value of 42.44 (at 40Hz) and low value of 26.68 (at 2500Hz).  At
frequencies 2500Hz and higher the jitter values became quite stable
showing  a maximum variation of only 2.11. Overall, our results indicate that
even under increased rate of invocation, the configurations perform effectively
in achieving BasicSP latency requirements.

