%% $Id: empirical.tex,v 1.19 2004/05/16 18:25:07 gokhale Exp $

\section{Performance Evaluation of Component Middleware-based
Real-time Event Services} 
\label{benchmarking}

With increasing use of QoS-enabled component middleware to develop and
deploy DRE systems, it is important to ascertain the real-time
performance of QoS-enabled component middleware, in particular that of
real-time publisher/\-subscriber services integrated within
QoS-enabled component middleware. This section provides results of
empirically evaluating performance of a real-time event channel in
CIAO, which is our QoS-enabled component middleware implementation. 
Our results focus on the container-level integration of event
channels described in Section~\ref{container-level}.

\subsection{Experiment Domain}
\label{exp_domain}
The domain we used for our experiments to empirically benchmark
performance of RT event channel integration within QoS-enabled
component middleware comprises a hypothetical aerial battlefield theatre
depicted in Figure~\ref{fig:UAV_scene}. 

\begin{figure}[htb]
\center{\includegraphics[width=3.3in]{figs/UAV_scene.png}}
\bfcaption{Hypothetical Aerial Warfare Scenario}
\label{fig:UAV_scene}
\end{figure}

In this military application scenario, a number of Unmanned
Aerial Vehicles (UAVs) simultaneously transmit surveillance video from
different regions in a battlefield. A UAV is a remote-controlled
aircraft that is launched in order to obtain a view of an engagement and
performs such functions as spotting enemy movements and locating
targets in a battlefield. 

A Distributor node receives video streams from UAVs and
forwards it to different receivers on the {\em Command and Operations
Control} (CAOC). The CAOC entity monitors these
video streams and is responsible for making tactical decisions about
vehicle deployment and weapon system guidance.  Some of the CAOC receivers
perform automatic target recognition to guide Unmanned Combat
Aerial Vehicles (UCAV), while others provide commands to 
launch weapons from ground stations.  

This application possesses features that make it a complex and
challenging problem, including large-scale, stringent real-time
requirements, resource constraints, and the distributed communication
environment (which includes low-bandwidth, high-loss wireless links)
that requires the use of publisher/\-subscriber paradigm.    

\iffalse
\subsection{Experimental Testbed}
\label{testbed}
We use Emulab~\cite{EMULab:03} as our testbed infrastructure for
executing the experimental scenario described in
Section~\ref{exp_domain}. The Emulab provides a novel approach to 
performing networking and distributed systems performance evaluation,
where realistic network behavior, network control and performance
measurements can be achieved using network emulation. The underlying
hardware used by Emulab is a cluster of computers that can be 
remotely configured by users to provide a virtual network topology and configure
various parameters of that network, including link bandwidth capacities,
node characteristics for use in routing, traffic shaping or traffic
generation, and link error rates. Using these configuration features,
users are able to emulate a variety of wireline and wireless
networks. An important feature of Emulab is its ability to execute
standard middleware, such as CCM implementations, on the user-defined
virtual network topology. For our empirical benchmarking, therefore,
we have chosen to use the Emulab testbed.
\fi

\subsection{Experiment Description}
\label{exp_desc}

This section describes our experiment by detailing the role of various
entities.
Figure~\ref{fig:BBN_UAV_v3} shows the topology of our 
aerial warfare theatre prototype. We modeled this topology using our
EQAL modeling tool~\cite{Schmidt:04e,FESML_ACMSE:04}. EQAL is a
modeling tool that resolves accidental complexities of configuring and
deploying large-scale publisher/\-subscriber component
middleware-based DRE systems. We applied our container-based real-time
event service~\cite{Schmidt:01k} to the prototype system. 

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/BBN_UAV_v3.png}}
\bfcaption{BBN UAV Scenario}
\label{fig:BBN_UAV_v3}
\end{figure}

In the prototype system shown in Figure~\ref{fig:BBN_UAV_v3}, we have
modeled and implemented seven different types of CIAO CCM components: 
Satellite component ({\tt Satellite}), CAOC component ({\tt CAOC}),
UAV Ground Station component ({\tt UAV\_GS}), UAV component ({\tt
UAV}), Engagement System component ({\tt Engagement\_Sys}), Battle
System ({\tt Battle\_Sys}), UCAV Ground Station component ({\tt
UCAV\_GS}) and UCAV component ({\tt UCAV}).  

The interactions among these components is described below:
\sqztiny
\begin{itemize}\crunchlist
\item {\bf Satellite:} Satellite is a component that is deployed for
      a surveillance task. Whenever it detects abnormal situations
      in the battlefield, it will emit an event of type {\tt
      TargetAbnormal}. The CAOC component is the only
      subscriber for such events.
\item {\bf CAOC:} The CAOC component acts as the command and control center
      for the battle. Whenever it receives a {\tt TargetAbnormal}
      event, it will emit an event of type {\tt PrepareCapture} 
      to instruct the UAV Ground Station (UAV\_GS) to prepare to
      capture an image in the field. The UAV\_GS is the only subscriber for
      this event type. Also, whenever there is any military need to engage
      the battle systems, the CAOC will emit a {\tt Prepare\_Engage} event 
      type to the Engagement System (Engagement\_Sys).
\item {\bf UAV\_GS:} When UAV\_GS component receives a {\tt
      PrepareCapture} event, it publishes a {\tt
      CaptureImage} event. Only those UAVs that are in the
      surrounding target field will receive the {\tt
      CaptureImage} event.  
\item {\bf UAV} When a UAV component receives a {\em CaptureImage}
      event, it will begin to execute the surveillance task by
      capturing images of the target field and transmitting
      the image back to the CAOC through the A/V Distributor. The image
      transmission is implemented via the CORBA A/V stream service. 
\item {\bf Engagement\_Sys:} Whenever the Engagement\_Sys component
      receives a {\tt Prepare\_Engage} type of event, it will
      publish an {\tt Engage} event to the UCAV Ground Station
      (UCAV\_GS) and Battle System (Battle\_Sys). The UCAV\_GS and
      Battle\_Sys are the only components that are the subscribers for
      {\tt Engage} events.  
\item {\bf Battle\_Sys:} Whenever the Battle\_Sys component receives an
      {\tt Engage} event, it will initiate engagement by preparing for 
      weapon launching.  
\item {\bf UCAV\_GS:} Whenever the UCAV\_GS component receives an
      {\tt Engage} event, it will dispatch a {\tt StartLaunch} event to
      the appropriate UCAV components.  
\item {\bf UCAV:} Whenever a UCAV component receives a {\tt
      StartLaunch} event, it will direct itself to the target location.
\end{itemize}

\sqztiny\noindent

There are a few interesting observations to note in this scenario: 
\begin{itemize}\crunchlist
\item most transmission links are wireless links imposing stringent
      bandwidth and latency constraints,
\item there is a need for prioritization between different types of
      event transmissions, and
\item reliability of delivery of event must be guaranteed. 
\end{itemize}
% Gan, George - since we did our experiments in our lab obvisoulsly we
% could not simulate the wireless links. We should says something to
% that effect here.

Building flexible application software and object-oriented middleware
that meets these requirements is challenging because the
need for determinism and predictability often results in
tightly-coupled designs. For instance, conventional mission-critical 
applications built with object-oriented middleware consist of closely
integrated responsibilities;  
each component, even with the benefit of a real-time event
service, will still have to
write application code to handle multiple aspects, such as
real-time event dispatching, scheduling and periodic event
processing. Tight coupling often yields inflexibility and thus can
substantially increase the effort and cost of integrating new and
improved system features. By using the container-based real-time event
service, however, it is still possible to build flexible and
loosely-coupled components, yet simultaneously meet QoS requirements. 

As shown in the Figure~\ref{fig:BBN_UAV_v3}, there are three event
propagation chains in the system. The first one begins when the Satellite
component emits a {\tt TargetAbnormal} event and ends when the UCAV components
receive the {\tt CaptureImage} events. The second one starts with the
CAOC emitting the {\tt PrepareEngage} event and terminates with the
Battle System 
receiving the {\tt Engage} event. The third one begins when the CAOC emits
the {\tt PrepareEngage} event and ends when the UCAVs receive the {\tt
StartLaunch} events. 

The most important metric for any event service is event latency, i.e., 
the time elapsed from when a supplier sends an event until the last consumer 
interested in the event receives it.
Our goal is to demonstrate and benchmark the publisher/\-subscriber
features in the experiment described above when used in the context of
a component middleware and a container-based event channel. The rest
of the section 
presents our experiment results with different real-time
event service settings, illustrating their effect on the provisioning
of event communication QoS in the DRE system. 

%%%%%%%%%%%%%%%%%%% results %%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experimental Measurement Approach}
\label{measurement_approach}

Measuring the latency in the event service is hard since events are
delivered  via a uni-directional flow of communication from suppliers
to consumers. 
As shown below, event delivery time and jitter is comparable
to the network propagation delay, because a distributed
clock precision is bounded by the jitter~\cite{Kopetz:97}, measuring
the latency of event propagation, where event supplier and event
consumer are deployed in different hosts, is impossible.
Fortunately, the latency for the centralized configuration can
be measured directly using a consumer located in the same
host as the supplier and measuring the roundtrip delay as
shown in Figure~\ref{fig:measure_approach}.

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/measure_approach.png}}
\bfcaption{Experimental Setup to Measure Event Service Latency}
\label{fig:measure_approach}
\end{figure}

% guys, can you please give a couple of sentences on the types of
% machines and network used - Andy
The performance tests 
were conducted on a testbed where each machine was a dual Intel Xeon
2.4G CPU, with 1G RAM running Windows XP Professional, and all these
machines are connected through 100M LAN. 
For all our measures we use the high-resolution timer 
(under 2 nanosecond resolution) available on Pentium processors.
This timer is implemented via a special register that counts 
the number of clock ticks since the CPU was reset.

\subsection{Metrics for Collocated Components}
\label{coloc_results}

Before we tested our prototype system and measured results in
the experiment situations where all CCM components are distributed
into different hosts, we first show our results where all the components 
are collocated in the same host and same process, {\em i.e.}, within
the same component server. 
By doing so we are able to mask the effects of network latencies, data
rate loss and distribution, thereby, permitting us to gauge the actual
performance of the real-time event dispatching within a QoS-enabled
component middleware. 
Figures~\ref{fig:colloc_UAV},~\ref{fig:colloc_UCAV}
and~\ref{fig:colloc_Battle} show the collocated event propagation
delays for the three event propagation chains described earlier,
respectively.  

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/UAV.png}}
\bfcaption{Collocated UAV Results}
\label{fig:colloc_UAV}
\end{figure}

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/UCAV.png}}
\bfcaption{Collocated UCAV Results}
\label{fig:colloc_UCAV}
\end{figure}

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/Battle.png}}
\bfcaption{Collocated Battle Results}
\label{fig:colloc_Battle}
\end{figure}

Our collocation experiment results indicate that the event dispatching
overhead is minimal thereby confirming that performance
optimizations of object-oriented real-time event dispatching are
preserved within a container-based solution.

In many real-world situations, multiple components are deployed within the
same process but still communicate with each other through 
CCM ports, {\em i.e.}, facets, receptacles, event sources, and event sinks.
To improve the performance and predictability of collocated component
communication, the process-collocation optimization was applied to CIAO's 
implementation. Process-collocation improves the performance and 
predictability transparencies for objects that reside in the same address space
as the servant implementation, while maintaining locality
transparency. To implement 
process-collocation, the ORB must identify the location of the component's 
reference without explicit application programmer intervention and without 
violating the policies specified by POAs and the component containers.
Once the ORB determines that a reference is collocated in the same process, 
all operation invocations can be forwarded to a special collocation
stub. The goal of process-collocation optimization is to ensure the
performance of accessing in-process collocated components is
comparable to accessing standard C++ components,  
while still providing predictability transparency in the framework.

\subsection{Metrics for Distributed Components}
\label{distrib_results}

In this experiment, we performed latency and jitter measurements on
the prototyped  
UAV application where components are deployed across multiple hosts. 
This experiment provides a realistic picture of the system 
performance in a typical computing environment. The network topology
is shown in Figure\~ref{fig:network-topology}. 

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/topology.png}}
\bfcaption{The Network Topology of Experiment for Distributed Case}
\label{fig:network-topology}
\end{figure}

Our measurements demonstrate that CIAO's container-based real-time event 
service provides the performance predictability needed by a representative 
DRE application. Delays for the same three event propagation chains
are given below. 
% Gan, can you expand this paragraph a little by explaining the system 
% requirements in more detail?

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/Distributed_UAV.png}}
\bfcaption{Distributed Case UAV Results}
\label{fig:dist_UAV}
\end{figure}

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/Distributed_UCAV.png}}
\bfcaption{Distributed Case UCAV Results}
\label{fig:dist_UCAV}
\end{figure}

\begin{figure}[htb]
\center{\includegraphics[width=3in]{figs/Distributed_Battle.png}}
\bfcaption{Distributed Case Battle Results}
\label{fig:dist_Battle}
\end{figure}

% Gan, can you expand this paragraph by explaining what the results tell us?
% Andy, I am not sure how to explain that our delay is dominated by
% inherent network delays. Any enhancement of this section would be greatly
% welcome!!
% OK, I did something - Andy
As can be seen from the above figures, the performance of the
federation of event channels integrated within different containers
continues to be predictable with minimal jitter, however, with an
additional cost of network latencies. Our testbed comprised networks
(LANs) that were lightly loaded. We are currently performing
additional experiments on the Emulab~\cite{EMULab:03} testbed, where we can
emulate different network types and link characteristics.

