% $Id: results.tex 16888 2009-02-21 03:45:29Z gokhale $
\section{Evaluating the Merits of the Transformation Algorithm}
\label{sec:results}
This section evaluates our approach to optimizing the original
deployment and configurations for component-based DRE systems.  We claim
that the quality of the resulting software architecture is improved if
it is able to demonstrate an improved performance.   We describe our
results in the context of our case study explained in
Section~\ref{sec:casestudy}.  We show how the end-to-end latency results
after applying our algorithm achieves considerable improvement over the
existing state-of-the-art.   Moreover, we also demonstrate a beneficial
side effect of our solution by discussing the algorithm can be combined
with additional backend optimization frameworks like the Physical
Assembly Mapper (PAM)~\cite{Schmidt:08e}.
 
\subsection{Experimental Setup \& Empirical Results}
\label{sec:results-main}
We have used ISISLab (\url{www.dre.vanderbilt.edu/ISISlab})
for evaluating our approach.  Each of the physical nodes used in our
experiments was a 2.8 GHz Intel Xeon dual processor, 1 GB physical
memory, 1 GHz network interface, and 40GB hard disks.  Version 0.6 of
our Real-time LwCCM middleware called CIAO was used running on Redhat
Fedora Core release 4 with real-time preemption patches.  The processes
that hosted BasicSP components were run in the POSIX scheduling class
SCHED\_FIFO, enabling first-in-first-out scheduling semantics based on
the priority of the process.  
 
To showcase our results, we first modeled the the BasicSP scenario and
generated the deployment and configuration metadata for each of its
components.  Note that the metadata is generated using the model
interpreters that encapsulate appropriate bin packing and schedulability
analysis techniques.   We collected the end-to-end latency metrics for
the BasicSP scenario using the initial deployment and configuration
metadata. 

We then applied the transformation algorithm~\ref{alg:optimize} to our
BasicSP model which resulted in more fine-grained optimizations to the
existing deployment and configuration metadata.  The BasicSP scenario
was then executed again with the updated QoS policies, and the results
were collected.  For both these experiments, the results were obtained
by repeating invocations for 100,000 iterations after 10,000 warmup
iterations and averaging them. 

Figures~\ref{fig:avg_latency} and ~\ref{fig:sd_latency} show the results
of applying our approach to BasicSP scenario comparing them to those
derived from the original deployment and configurations.  The figure
plots the average end-to-end latency and its standard deviation for the
invocations from \emph{Timer} to \emph{Na\-vD\-is\-pl\-ay} components in
BasicSP with and without our approach.  

As shown in Figure~\ref{fig:avg_latency}, the average latency improved
by $\sim$70\% when our technique was used for optimizing BasicSP QoS
configurations.  The standard deviation on the other hand, improved by
$\sim$59\% as plotted in Figure~\ref{fig:sd_latency}. 


\begin{figure}[htb]
  \centering  
    \includegraphics[width=0.7\textwidth,keepaspectratio]{average_latency}    
    \caption{Average end-to-end Latency}
\label{fig:avg_latency}
\end{figure} 

Without our approach, the initial BasicSP QoS configuration contained
separate policies for each of its four components.  Out of the four
components, only the \emph{Timer} component has 
\textsc{ser\-ver\_dec\-lar\-ed} priority model, while the rest of the
components have \textsc{cl\-ie\-nt\_prop\-aga\-ted} priority model.
Thus, as indicated on Lines~\ref{algo1:main1} and ~\ref{algo1:main2},
when Algorithm~\ref{alg:optimize} is applied to BasicSP, the QoS policy
set is reduced to a size of two, one for each kind of priority model.
This reduction in the size of the QoS policy set leads to the $\sim$70\%
improvement in end-to-end latency between \emph{Timer} and
\emph{NavDisplay} components.  

\begin{figure}[htb]
  \centering  
    \includegraphics[width=0.7\textwidth,keepaspectratio]{sd_latency}    
    \caption{Standard Deviation in Latency}
\label{fig:sd_latency}
\end{figure}

The third graph in each figure indicates the additional improvements in
end-to-end latencies accrued as a result of leveraging backend
optimization frameworks, such as PAM~\cite{Schmidt:08e}.  PAM is a
deployment-time technique that \emph{fuses} a set of components
collocated in a container to reduce memory footprint and latency between
service invocations.  Our approach simply indicates what components
should be part of the same container, however, individual components
continue to require their own stubs and skeletons, and other glue code,
which continues to be a source of memory footprint overhead.  Approaches
like PAM can then be used to eliminate this remaining overhead.

As shown in Figures~\ref{fig:avg_latency} and ~\ref{fig:sd_latency},
when applied in conjunction with PAM, our approach leads to a combined
improvement of $\sim$83\% in the end-to-end latency and $\sim$65\% in
the observed standard deviation in latency for BasicSP scenario.

\subsection{Discussion}
\label{sec:discussion}

Our transformation algorithm described in Sections~\ref{sec:algo} relies
on QoS configuration analyses in a platform-specific manner.  We
specifically showed how it has been realized in the context of a LwCCM
middleware implementation.  Naturally to extend it to other middleware
platforms requires a careful study of the other platform's configuration
space.  

The results indicated an improvement of ~$\sim$70\% in invocation
latency between an execution path consisting of four components (the 
execution path here refers to the invocations from \emph{Timer}, to
\emph{GPS}, to \emph{AirFrame}, and finally to \emph{NavDisplay}
components in BasicSP).  Recall that the BasicSP is an assembly of
components that is used in the context of larger DRE system
architectures.  With increasing scale of the DRE system, it becomes
necessary to leverage every opportunity for optimizations.  

We expect the improvements accrued using our approach to be even
higher.  This is because the reduction in end-to-end latency is
dependent upon how effectively the QoS policy sets $SQ_1$ and $SQ_2$ in
Algorithm~\ref{alg:optimize} are minimized.  Large-scale DRE systems
would have a number of QoS policies specified across their component
assemblies, and in general, would be expected to have more opportunities
to combine and reuse these policies leading to further latency
improvements.   
 

\iffalse
Existing deployment- and run-time techniques in standards-based
middleware have focused on improving the performance of the
infrastructure itself to optimize application-level QoS.  For example,
NetQoPE~\cite{Schmidt:08a} deals with improving the deployment framework
to improve network QoS provisioning by optimizing QoS policies.
PAM~\cite{Schmidt:08e} describes deployment-time techniques that allow
for \emph{fusing} of a set of components to reduce memory footprint and
latency between service invocations.

One of the key differences between these techniques and our work is that
we raise the level of optimizations to DRE system model-level.  Thus,
once the transformation algorithm has been applied, the modified
DRE system model (\emph{i.e.}, its QoS configuration specification) can
be further used by the developers in verification.  For example, these
models can be used to reason about the correctness of various
application properties using model-checking.

Our approach can be used in a complementary fashion with these
techniques to further improve the application QoS.  Since PAM is
essentially a model-driven tool, the modified DRE system QoS
configuration model resulting from applying our model transformation
algorithm can directly be used to investigate fusion opportunities for
the application.  As shown in Figures~\ref{fig:avg_latency} and
~\ref{fig:sd_latency}, when applied in conjunction with PAM, our
approach leads to a combined improvement of $\sim$83\% in the end-to-end
latency and $\sim$65\% in the observed standard deviation in latency for
BasicSP scenario.
\fi
