% $Id: eval.tex,v 1.34 2005/11/23 19:54:35 gokhale Exp $

\section{Empirically Evaluating the ARMS MLRM}
\label{sec:eval}

This section describes the design and results of experiments we
conducted to empirically evaluate the ARMS MLRM architecture described
in Section~\ref{sec:mlrm}.  We focus on the MLRM's capabilities for
(1) dynamically managing computing and network resources to meet
changing mission requirements and (2) configuring various QoS
mechanisms within the middleware to satisfy QoS and operational
requirements.

\subsection{Hardware/Software Testbed and Evaluation Criteria}
\label{benchmarks}

The experiments used up to six Sun SPARC Solaris computers, with two
of the six being Sunfire V1280 servers with 12 multiprocessors and
other four being desktop uniprocessors.  All nodes ran the Solaris 8
operating system, which supports kernel-level multi-threading and
symmetric multiprocessing.  All nodes were connected over a 100 Mbps
LAN and used version 0.4.1 of CIAO and DAnCE as the QoS-enabled
component middleware infrastructure.  The benchmarks ran in the POSIX
real-time thread scheduling class~\cite{Khanna:92} to enhance the
consistency of our results by ensuring the threads created during the
experiment were not preempted arbitrarily during their execution.

Figure~\ref{fig:scenario} shows the TSCE scenario that guided the
experiments.  In this scenario many sensors (ED) colloborate with
planning processes (PLAN) and effectors (EFF) to detect different
operating conditions and make adaptive and effective decisions to
counteract harmful and adverse conditions.
\begin{figure}[htbp]
  \centering
  \includegraphics[width=\columnwidth]{motiv-scenario}
  \bfcaption{Operational Strings Deployed in ARMS MLRM Experiments}
  \label{fig:scenario}
\end{figure}
The sensors, planning processes, and effectors were implemented as
application components using CIAO; groups of application components
are connected in component assemblies to form operational strings.
Each operational string in Figure~\ref{fig:scenario} corresponds to a
different set of TSCE capabilities, such as anti-cruise missile
ship-self defense, mine avoidance, or crew entertainment.  The MLRM
therefore deploys different sets of operational strings depending on
the different contexts and mission modes.  For example, a particular
combination of sensors, planning processes, and effectors implemented
as application components and connected together can work in a
collaborative fashion to detect, plan, and implement a corrective
action to correspond to an sensed event, such as sensing of
unidentified airborne object and deciding what types of
counter-measures to enact.  These operational strings are managed by
the ARMS MLRM as described in Section~\ref{sec:mlrm-components}.

Our three main goals in conducting experiments on the TSCE scenario
shown in Figure~\ref{fig:scenario} were to:
\begin{enumerate}\cramplist
\item Evaluate the capability of the ARMS MLRM to deploy the
operational strings in different configurations and in different
orders so that the applications can be mapped or hosted into many
different combinations of physical processors, thereby ensuring that
MLRM can dynamically manage the available resources by redeploying
operational strings to different hardware nodes as nodes become
unavailable due to resource limitations or hardware failures.  The
goal was to demonstrate that at least 2 different dynamic allocations
can be achieved based on prevailing conditions.

\item Quantify the capability of MLRM to recover from data center
failures and redeploy primary operational strings to available data
centers in a timely manner.  The goal was to recover in less than 1/2
the time of 4 minutes it takes to recover in a manually managed TSCE.

\item Demonstrate the capability of MLRM to respond to an increasing
number of threats by ensuring that higher priority and more important
applications always operate, so that the utilization of resources are
under control and always available for providing key TSCE
capabilities.
\end{enumerate}
The remainder of this section describes the experiments we conducted
and evaluates how well the ARMS MLRM meets the stated goals.

\subsection{MLRM Capability to Deploy Infinite Set of Configurations}
\label{mlrm-deploy-infinite-configurations}

We first present empirical results that demonstrate ARMS MLRM's
ability to generate different deployment configurations of the same
set of operational strings, depending on different input ordering of
strings, different resource availabilities of the resources, and
different resource requirements of the operational strings.  We used
the TSCE scenario described in Figure~\ref{fig:scenario} to identify
five operational strings that map to five different TSCE
functionalities, including ship self defense, call for fire, and
undersea warfare and others.  Each operational string involved a
subset of the applications shown and contain a {\em critical path},
{\em i.e.}, start-to-finish processing flow over which an end-to-end
deadline must be met.

The goal of this experiment was to demonstrate that we could create a
variety of different configurations of the same set of five
operational strings by mapping those operational strings onto up to
six different hardware nodes in response to (1) changes in the
resource availabilities of the nodes, (2) changes in the events in the
TSCE scenario that causes new strings to be deployed beyond the
strings that are already deployed, (3) changes in the configurations
of the strings in response to the mode changes in the TSCE scenario,
and (4) changes in the size of the operational strings.  The five
operations strings comprise of 8, 9, 16, 17, and 20 application
components for a total of 70 application components in the deployed
system.  Application components are written using the CIAO RT-CCM
middleware and deployed and configured using DAnCE.

The results in Figure~\ref{fig:gt-1-exp} demonstrate that the ARMS
MLRM was able to generate six different configurations for those five
operational strings deployed on the six different hardware nodes.  The
five dynamic deployments differ from each other by the different order
in which various strings are deployed to satisfy different mission
needs.
\begin{figure}[htbp]
  \centering
  \includegraphics[width=\columnwidth]{gt1-exp}
  \bfcaption{Different Configurations of Operational Strings in Six Nodes}
  \label{fig:gt-1-exp}
\end{figure}
It can be observed, however, that in all cases dynamic deployment --
based on runtime use of bin-packing
algorithms~\cite{Kenyon:96,Baker:85} -- leads to a fairly balanced
workload across the processors.  These results indicate that
\begin{itemize}\cramplist
\item The ARMS MLRM allocation algorithms have the capability to
generate different sets of deployment configurations for the
operational strings over the same set of hardware nodes.  For static
allocation the highest processor utilization was $\sim$65 percent and
the lowest was $\sim$38 percent.  In contrast, all dynamic allocations
resulted in highest processor loading of less than 60 percent and no
processor is loaded less than 40 percent.  Keeping maximum
utilization is as low as possible is a DRM goal that ensures adequate
safety margins and system schedulabililty.

\item With the help of the standards-based CIAO and DAnCE RT-CCM
middleware that deploys and migrates operational strings, the ARMS
MLRM can quickly generate different sets of allocations for
operational strings depending upon resource availabilities and
hardware failures.  As a result, critical TSCE mission functionalities
need not be compromised in the face of hazardous events that can
trigger resource fluctuations and failures.
\end{itemize}

\subsection{MLRM Capability to Recover from Failures}
\label{mlrm-handle-failures}

Section~\ref{sec:mlrm-components} described how the ARMS MLRM manages
different sets of resources within a single pool and how the resources
within a pool can be utilized efficiently to deploy operational
strings by mapping them onto different nodes within a resource pool.
To support active replication, the MLRM deploys replica operational
strings in different resource pools so that mission mode functionality
can be available in a dependable manner even in the face of harmful
events.  To minimize the utilization of resources, however, not all
operational strings are duplicated and only certain mission-critical
operational strings are replicated and deployed under different
resource pools.

Figure~\ref{fig:gt-2} shows how operational strings consisting of a
combination of sensors and effectors in our TSCE are deployed together
with their replicas across two different resource pools.
\begin{figure}[htbp]
  \centering
  \includegraphics[width=\columnwidth]{GT2-scenario}
  \bfcaption{Operational Strings and Their Replicas Deployed in Different Resource Pools}
  \label{fig:gt-2}
\end{figure}
The two resource pools are {\em Pool-1.A} and {\em Pool-1.B} and each
of the resource pools have three nodes each.  The primary and the
secondary replicas for a particular operational string are distributed
across the two resource pools to ensure active replication.  For
example, the operational string deployed in the nodes {\em Mako} and
{\em Champion} of the resource pool {\em Pool-1.A} have their replicas
deployed in the nodes {\em Chaparal} and {\em Javelin} of the resource
pool {\em Pool-1.B}.  Below, we present empirical results that measure
how fast the ARMS MLRM can react to pool failures to reconstitute the
primary operational strings from the failed pools onto the pools that
are safe, thereby ensuring that critical mission functionality is
available throughout the lifecycle of the TSCE.

We used the TSCE scenario described in Figure~\ref{fig:gt-2} to
identify the steps the ARMS MLRM must take to reconstitute the primary
replicas from the {\em Pool-1.B} resource pool onto the {\em Pool-1.A}
resource pool in the case of the total failure of the {\em Pool-1.B}
resource pool.  We focused on the time the ARMS MLRM needed to (1)
detect the pool failure and decide what needs to be redeployed, (2)
redeploy the strings in the available pools and their resources, (3)
allow the OS to spawn a processing environment for the operational
strings, (4) give appropriate trigger signals to the strings so that
they can be part of the global mission functionalities by
colloborating with other strings, and (5) promote the new primaries
and secondaries in the new deployment configurations.

The results in Figure~\ref{fig:gt-2-exp} show that ARMS MLRM takes
$\sim$15 seconds to complete parts 1 and 2, which are pool failure
detection and redeployment, respectively.
\begin{figure}[htbp]
  \centering
  \includegraphics[width=\columnwidth]{gt2-experiment}
  \bfcaption{Redeployment of Effectors from Failed Pool}
  \label{fig:gt-2-exp}
\end{figure}
The vast majority of this time is taken up in pool failure detection.
Conservative timeout values were used for failure detection to
minimize false alarms.  With a more aggressive use of timeout, either
with capabilities native to a compute node or with custom hardware,
can reduce this number to well below a second.  The time taken by part
2 (redeployment decisions) are well below a second.  A relatively
large gap is noted for part 3 (time to spawn new processes to host
application components), which is handled by DAnCE and is outside the
direct control of ARMS MLRM.  Finally, we note another 15 second lag
for parts 4 and 5 (initialization and triggering) of the redeployment
process.  The time to recover is therefore well below 1/2 the 4
minutes it took to recover a manually managed system.

\subsection{MLRM Capability to Operate under Limited Resource Availabilities}
\label{mlrm-handle-overloads}

Section~\ref{sec:mlrm-components} described how the ARMS MLRM manages
different sets of resources within a single node in a pool and
provides the capability to maintain the operation of critical mission
capabilities by gracefully responding to overload conditions, such as
handling more tracks and threats by offloading, reallocating, or
reprioritizing other {\em not-so-important} operational strings
operating in a node.  Below, we present empirical results that show
(1) how the ARMS MLRM can handle increased loads in a node to handle
additional threats and tracks and (2) how the MLRM can do this
gracefully by offloading less important operational strings from a
node.  The goal is to ensure that critical mission capabilities meet
their QoS requirements in the face of increasing workloads.

Figure~\ref{fig:gt-3} shows a scenario in the TSCE where three nodes
are used to deploy two operational strings that provide {\em general
tracking} and {\em air threat tracking} capabilities.
\begin{figure}[htbp]
  \centering
  \includegraphics[width=\columnwidth]{figs/new-gt3-scenario}
  \bfcaption{Different Threat Tracking Components in the ARMS MLRM}
  \label{fig:gt-3}
\end{figure}
We measured the CPU utilization in the {\em Mako} node, where parts of
both the operational strings are deployed and launched.  We also
measured the {\em Mako's} CPU utilization as air threats started to
increase, prompting the {\em ED-1.1} sensor to send reactive events
and messages to {\em PLAN-3.1} planner on {\em Mako}.

The planner makes decisions and uses the effector {\em EFF-1.1} in the
{\em Checkmate} node to take evasive actions.  In this scenario, {\em
Mako}'s CPU utilization went up sporadically as it processed a series
of air threats.  In the face of the increasing air threats, however,
the critical operational functionalities provided by {\em Mako} (such
as the planning process provided by the {\em PLAN-3.1} planner) need
to operate correctly and efficiently.  To evaluate the benefits of
DRM, we therefore compared the results adding and removing ARMS MLRM
capabilities in the TSCE to highlight its capabilities.

Figure~\ref{fig:mlrm-disabled} shows the results of an experiment
where the ARMS MLRM capabilities were disabled.
\begin{figure}[htbp]
\sidebyside {
  \centerline{\includegraphics[width=3.3in]{gt3-mlrm-disabled}}
  \bfcaption{Handling Threats with MLRM Disabled} \label{fig:mlrm-disabled}} {
  \centerline{\includegraphics[width=3.3in]{gt3-mlrm-enabled}}
  \bfcaption{Handling Threats with MLRM Enabled}
  \label{fig:mlrm-enabled}}
\end{figure}
Here we observe that as processing load for Plan-1.1 increases beyond
a certain point the system starts to become unstable with actual
amount of resource consumed fluctuating significantly.  In contrast,
Figure~\ref{fig:mlrm-enabled} shows the results of enabling the ARMS
MLRM DRM capabilities, where even as load for Plan-1.1 increases the
MLRM makes the necessary dynamic adjustments to deliver steady
resource availability for this application component.

In summary, the experimental evaluations in this section validated
that the ARMS MLRM met its goals, {\em i.e.}:
% Jai, can you please make sure that these 4 goals correspond to the
% goals that are mentioned near the top of this section.  I think
% we're inconsistent since there were 3 goals there and 4 goals here!
\begin{itemize}
\item Multiple dynamic allocations were achieved - five were
demonstrated.

\item Recovery from a data center failure is achieved in well under
1/2 the time it takes for manual recovery

\item Under increased load MLRM makes the necessary resource
allocation adjustments to permit steady state increase in resource
availability to critical applications without any instability.

%%
%% JAI : The first three are the real goals. This one is a general statement.
%% Kind of a summary that Gautam has added.
%% We can move this at the top of this paragraph and say in summary,
%% we achieved the following goals.
%%
\item MLRM permits a range of automated, dynamic management of
resources that greatly extend operational flexibility of the system,
without requiring human intervention.
\end{itemize}

% Gautam, can you please tie the summary of the results here back with
% the goals we enumerated earlier?  Please try to make the discussion
% as ``quantitative'' as possible.
% Gautam : Attempt made in the list above.
