% $Id: evaluation.tex 16638 2009-01-29 19:00:10Z gokhale $
\sqzlarge
\section{Evaluating the QUICKER QoS configuration process}
\label{sec:eval}
\sqzlarge
This section evaluates our configuration process to verify the correctness
of its transformation algorithms and validate its effectiveness in meeting the
QoS requirements of DRE systems. First we present a representative DRE
system case study we used for the evaluation. Next we discuss our 
structural correspondence technique for proving that the input and output models 
of transformations used in our process are correctly mapped. We then describe how we have
applied Bogor model-checking tool for verification of the generated
configurations. Finally, through empirical evaluation, we validate the generated 
QoS options. 

\input{scenario}
\sqzlarge
\subsection{Verifying the correctness of QUICKER QoS configuration process}
\label{sec:verify}
\sqzlarge
Verifying our model-based configuration process entails verification of
correctness properties across the following two dimensions: (1) Correctness of
QoS mapping algorithms \emph{i.e.}, QoS options generated are equivalent to the 
QoS requirements from which these options are mapped. In our case, this translates
to verification of the QoS mapping/transformations used. (2) Correctness of
the generated QoS options themselves \emph{i.e.}, whether individual values of 
these options are appropriate locally (\emph{e.g.}, for a component) as well as 
globally (\emph{e.g.}, for all dependent components). This section discusses 
verification of our process across the above two dimensions.  
\input{verify}

\input{mcheck}
\begin{figure}[htb]
\centering
\subfloat[End-to-end latency measurements]
  {
    \includegraphics[width=3.2in,keepaspectratio]{latency_work}%
    \label{fig:latency-work}
  }
\subfloat[Statistical dispersion of latency]
  {
    \includegraphics[width=3.2in,keepaspectratio]{sd_work}%
    \label{fig:sd-work}
  }
\vspace{-0.1in}
\caption{Evaluating BasicSP QoS configurations against increasing workload at a constant 20Hz
         invocation rate.}
% \vspace{-0.2in}
\label{fig:workload}
\end{figure}

\sqzlarge
\subsection{Empirically evaluating BasicSP QoS configurations}
\label{sec:validate}
\sqzlarge
In this section we empirically validate the effectiveness of the generated QoS
configurations for the BasicSP case study.

\textbf{Experiment Configuration.} We have used ISISLab 
(\url{www.dre.vanderbilt.edu/ISISlab})
for evaluating observed QoS properties of DRE systems based on middleware QoS
configurations generated using our configuration process.  Each of the physical nodes used in our
experiments was a 2.8 GHz Intel Xeon dual processor, 1 GB physical memory, 1 GHz
network interface, and 40GB hard disks. Version 0.6 of our RT-CCM middleware
CIAO was used running on Redhat Fedora Core release 4 with real-time
preemption patches. The processes that hosted BasicSP
components were run in the POSIX scheduling class SCHED\_FIFO, enabling
first-in-first-out scheduling semantics based on the priority of the process. 

As the first step, we modeled BasicSP QoS requirements using the requirements DSML
described in Section~\ref{sec:transform}.
\texttt{bu\-rs\-ty\_cl\-ie\-nt\_re\-qu\-es\-ts} was set to {\sc false} for all
components and \texttt{fi\-xed\_pri\-ori\-ty\_ser\-vi\-ce\_ex\-ec\-uti\-on}
attribute was set to {\sc false} for every component except
\emph{Timer}.  Secondly, we applied our model transformation algorithm to the
requirements model above for generating detailed application configurations.
Table~\ref{table:output} captures some of the important QoS configurations
generated in our process. These configurations are represented as an 
application model. In the final
step, we apply model interpreters for synthesizing descriptors required to
configure the functional and QoS properties of the application during
deployment.
\begin{table}[htb]
\centering
\caption{Generated QoS configuration for BasicSP}
\sqzlarge
\label{table:output}
\footnotesize
\begin{tabulary}{\columnwidth}{|C|C|C|C|C|}
  \hline
  \textbf{QoS configuration} & \textbf{Timer} & \textbf{GPS} & \textbf{Airframe} \\
  \hline \hline
  PriorityModel & {\sc sd} & {\sc cp} & {\sc cp} \\
  \hline
  \underline{ThreadPool} & & & \\
  stacksize & 1024 & 1024 & 1024 \\
  max\_buff\_reqs. & -- & 20 & 20 \\
  allow\_borro. & {\sc false} & {\sc false} & {\sc false} \\
  allow\_req\_buff & {\sc false} & {\sc true} & {\sc true} \\  
  \hline
  \underline{Lane} & & & \\
  static\_thrds & 4 & 8 & 8 \\
  dyna\_thrds & 0 & 0 & 0 \\
  \hline
\end{tabulary}
\vspace{-0.2in}
%\end{singlespace}
\end{table}

In evaluating effectiveness of our configuration process, we collected 
end-to-end latency measurements between
\emph{Timer} and \emph{NavDisplay} components. Earlier in 
Section~\ref{modelcheck} we discussed how correctness of QoS
options can be verified using our process, the first experiment discussed 
below empirically evaluates the effectiveness of these options in meeting 20Hz 
operational display refresh rate of BasicSP from low to high workload conditions. 
Further, operational conditions of DRE system might change (unfavorably) during its 
execution. In order to evaluate the tolerance of our generated configurations
under such conditions, in the second experiment we measure the metrics discussed 
above when invocation rate is steadily increased. Each of these experiments 
were performed for a constant time period and after executing 10,000 warmup 
iterations. 

\noindent\textbf{Experiment 1: Increasing System Workload.}
Figure~\ref{fig:workload} plots the latency 
measurements under increasing system workload. The workload is 
characterized~\cite{Schmidt:02r} as a function performed with every 
client invocation.  The signature of the function is given
as:  $void\;\; work (int\; units);\;$ where $units$ argument specifies the
amount of processor intensive work performed per call.
The experiment was run for workload values of 10 through 80. End-to-end latency was observed to be
at an average value of $\sim$1925 as can be seen in Figure~\ref{fig:latency-work}.
Further, successive event-driven computations in the scenario exhibit an almost constant time
complexity, indicated by relatively small dispersion in latencies as plotted in
Figure~\ref{fig:sd-work}. 

\noindent\textbf{Experiment 2: Increasing System Invocation Rate.}
Performance of the generated configurations for BasicSP is given in 
Figure~\ref{fig:invc-rate}.  Throughout this experiment the rate of 
invocation was increased from a normal operational value of 20Hz to
a maximum of 40000Hz.  Latency results are shown in 
Figure~\ref{fig:latency-rate} which plots maximum, mean and minimum delay 
measurements for each invocation rate data point.  Even with increasing 
rate the mean latency did not change significantly and was observed 
to be consistently just above 1900 microsec. for the entire range of invocation
rates. Note that this is a desirable characteristic since even with an
unfavorable change in operational conditions (\emph{i.e.}, change in
invocation rate) the latency was observed to be constant. Jitter in latencies for each invocation rate is plotted in
Figure~\ref{fig:sd-rate} which shows that the deviation is bound between a high value
of 42.44 (at 40Hz) and low value of 26.68 (at 2500Hz).  At frequencies
2500Hz and higher the jitter values became quite stable showing 
a maximum variation of only 2.11. Overall, our results indicate that
even under increased rate of invocation, the configurations perform effectively
in achieving BasicSP latency requirements.
\begin{figure*}[htb]
\centering
\subfloat[End-to-end latency measurements]
  {
    \includegraphics[width=3.2in,keepaspectratio]{latency_rate}%
    \label{fig:latency-rate}
  }
\subfloat[Statistical dispersion of latency]
  {
    \includegraphics[width=3.2in,keepaspectratio]{sd_rate}%
    \label{fig:sd-rate}
  }
\vspace{-0.1in}
\caption{Evaluating BasicSP QoS configurations against increasing invocation rate:
         All the plots use logarithmic X axis and linear Y axis.}
\label{fig:invc-rate}
\end{figure*}
