% $Id: eval.tex 3601 2006-07-24 19:35:41Z jai $

%\sqzlarge
\section{Empirically Evaluating the ARMS MLRM}
\label{sec:eval}
%\sqzlarge

This section describes the design and results of experiments we
conducted to empirically evaluate the ARMS MLRM architecture described
in Section~\ref{sec:mlrm}.  We focus on the MLRM's capabilities for
(1) dynamically managing computing and network resources to meet
changing mission requirements and (2) configuring various QoS
mechanisms within the middleware to satisfy QoS and operational
requirements.

%\sqzlarge
\subsection{Hardware/Software Testbed and Evaluation Criteria}
\label{benchmarks}
%\sqzlarge

The experiments used up to six Sun SPARC Solaris computers, with two
of the six being Sunfire V1280 servers with 12 processors and
other four being desktop uniprocessors.  All nodes ran the Solaris 8
operating system, which supports kernel-level multi-threading and
symmetric multiprocessing.  All nodes were connected via a 100 Mbps
LAN and used version 0.4.1 of CIAO and DAnCE as the QoS-enabled
component middleware infrastructure.  The benchmarks ran in the POSIX
real-time thread scheduling class~\cite{Khanna:92} to enhance the
consistency of our results by ensuring the threads created during the
experiment were not preempted arbitrarily during their execution.

%%
%% JAI: Adding a reviewer comment.
%% In Figure 6, there is a component CFGOP, in 4 of the 5 operational
%% strings. What is this? It seems that it is not introduced in the paper.
%%
%% Tom suggests the following:
%% The second paragraph in section 3.1 should have the first sentence modified to 
%% include CFGOP, something like:
%% Figure 6 shows the TSCE scenario that guided the experiments. In this 
%% scenario many sensors (ED) collaborate with planning processes (PLAN),  
%% configuration processes (CFGOP), and effectors (EFF) to detect different 
%% operating conditions and make adaptive and effective decisions to 
%% counteract harmful and adverse conditions.
%% JAI: addressed this here.
%%
%%
Figure~\ref{fig:scenario} shows the scenario that guided the
experiments.  In this scenario many sensors (ED) collaborate with
planning processes (PLAN), configuration processes (CFGOP), and
effectors (EFF) to detect different operating conditions and make
adaptive and effective decisions to counteract harmful and adverse
conditions.  The sensors, planning processes, and effectors were
implemented as application components using CIAO; groups of
application components are connected in component assemblies to form
operational strings.  Each operational string in
Figure~\ref{fig:scenario} corresponds to a different set of shipboard
computing capabilities.
% , such as anti cruise missile ship self defense,
% mine avoidance, or crew entertainment.
The MLRM therefore deploys different sets of operational strings
depending on the different contexts and mission modes.  For example, a
particular combination of sensors, planning processes, and effectors
implemented as application components and connected together can work
in a collaborative fashion to detect, plan, and implement a corrective
action to correspond to a sensed event, such as sensing of
unidentified airborne object and deciding what types of
counter-measures to enact.  These operational strings are managed by
the ARMS MLRM as described in Section~\ref{sec:mlrm-components}.

\begin{figure}[htbp]
  \centering
%  \includegraphics[width=\columnwidth, height=3in]{motiv-scenario}
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{motiv-scenario}
  \bfcaption{Operational Strings Deployed in ARMS MLRM Experiments}
  \label{fig:scenario}
\end{figure}

Our goals in conducting the experiments shown in
Figure~\ref{fig:scenario} were to:
\begin{enumerate}\cramplist
\item Evaluate the capability of the ARMS MLRM to deploy the
operational strings in different configurations and in different
orders so that the applications can be mapped or hosted into many
different combinations of physical processors, thereby ensuring that
MLRM can dynamically manage the available resources by redeploying
operational strings to different hardware nodes as nodes become
unavailable due to resource limitations or hardware failures.  The
goal was to demonstrate that at least 2 different dynamic allocations
can be achieved based on prevailing conditions.

\item Quantify the capability of MLRM to recover from data center
failures and redeploy primary operational strings to available data
centers in a timely manner.  The goal was to recover in less than half
the time of 4 minutes it takes to recover in a manually managed
system.

\item Demonstrate the capability of MLRM to respond to an increasing
number of threats by ensuring that higher priority and more important
applications always operate, so that the utilization of resources are
under control and always available for providing key shipboard
computing capabilities.
\end{enumerate}
The remainder of this section describes the experiments we conducted
and evaluates how well the ARMS MLRM meets the stated goals.

%\sqzlarge
\subsection{MLRM Capability to Deploy Infinite Set of Configurations}
\label{mlrm-deploy-infinite-configurations}
%\sqzlarge

We first present empirical results that demonstrate ARMS MLRM's
ability to generate different deployment configurations of the same
set of operational strings, depending on different input ordering of
strings, different resource availabilities of the resources, and
different resource requirements of the operational strings.  We used
the scenario described in Figure~\ref{fig:scenario} to identify five
operational strings that map to five different functionalities.
% , including ship self defense, call for fire, and undersea warfare
% and others.
Each operational string involved the applications shown and contain a
{\em critical path}, {\em i.e.}, start-to-finish processing flow over
which an end-to-end deadline must be met.

The goal of this experiment was to demonstrate that we could create a
variety of different configurations of the same set of five
operational strings by mapping those operational strings onto up to
six different hardware nodes in response to (1) changes in the
resource availabilities of the nodes, (2) changes in the events in the
scenario that causes new strings to be deployed beyond the strings
that are already deployed, (3) changes in the configurations of the
strings in response to the mode changes in the scenario, and (4)
changes in the size of the operational strings.  The five operations
strings comprise of 8, 9, 16, 17, and 20 application components for a
total of 70 application components in the deployed system (not all 70
applications are shown in the Figure~\ref{fig:gt-1-exp}.)  Application
components are written using the CIAO RT-CCM middleware and deployed
and configured using DAnCE.

The results in Figure~\ref{fig:gt-1-exp} demonstrate that the ARMS
MLRM was able to generate six different configurations for those five
operational strings deployed on the six different hardware nodes.  The
five dynamic deployments differ from each other by the different order
in which various strings are deployed to satisfy different mission
needs.
\begin{figure}[htbp]
  \centering
%  \includegraphics[width=\columnwidth,height=4in]{gt1-exp}
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{gt1-exp}
  \bfcaption{Different Configurations of Operational Strings in Six Nodes}
  \label{fig:gt-1-exp}
\end{figure}
It can be observed, however, that in all cases dynamic deployment --
based on runtime use of bin-packing
algorithms~\cite{Kenyon:96} -- leads to a fairly balanced
workload across the processors.  These results indicate that
\begin{itemize}\cramplist
\item The ARMS MLRM allocation algorithms have the capability to
generate different sets of deployment configurations for the
operational strings over the same set of hardware nodes.  For static
allocation the highest processor utilization was $\sim$65 percent and
the lowest was $\sim$38 percent.  In contrast, all dynamic allocations
resulted in highest processor loading of less than 60 percent and no
processor is loaded less than 40 percent.  Keeping maximum
utilization is as low as possible is a DRM goal that ensures adequate
safety margins and system schedulabililty.

\item With the help of the standards-based CIAO and DAnCE RT-CCM
middleware that deploys and migrates operational strings, the ARMS
MLRM can quickly generate different sets of allocations for
operational strings depending upon resource availabilities and
hardware failures.  As a result, mission-critical functionalities need
not be compromised in the face of hazardous events that can trigger
resource fluctuations and failures.
\end{itemize}

%\sqzlarge
\subsection{MLRM Capability to Recover from Failures}
\label{mlrm-handle-failures}
%\sqzlarge

Section~\ref{sec:mlrm-components} described how the ARMS MLRM manages
different sets of resources within a single pool and how the resources
within a pool can be utilized efficiently to deploy operational
strings by mapping them onto different nodes within a resource pool.
To support active replication, the MLRM deploys replica operational
strings in different resource pools so that mission mode functionality
can be available in a dependable manner even in the face of harmful
events.  To minimize the utilization of resources, however, all
operational strings may not be duplicated and only certain
mission-critical operational strings would be replicated and deployed
onto different resource pools. In tests results described here all
strings were duplicated since the overall test size is modest and we
had sufficient resources at hand.

Figure~\ref{fig:gt-2} shows how operational strings consisting of a
combination of sensors and effectors in our scenario are deployed
together with their replicas across two different resource pools.
\begin{figure}[htbp]
  \centering
%  \includegraphics[width=\columnwidth,height=3.5in]{GT2-scenario}
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{GT2-scenario}
  \bfcaption{Operational Strings and Replicas Deployed in Different Resource Pools}
  \label{fig:gt-2}
\end{figure}
The two resource pools are {\em Pool-1.A} and {\em Pool-1.B} and each
resource pool has three nodes.  The primary and the secondary replicas
for a particular operational string are distributed across the two
resource pools to ensure reliable replication.  For example, the
operational string deployed in the nodes {\em Mako} and {\em Champion}
of the resource pool {\em Pool-1.A} have their replicas deployed in
the nodes {\em Chaparal} and {\em Javelin} of the resource pool {\em
Pool-1.B}.  Below, we present empirical results that measure how fast
the ARMS MLRM reacts to pool failures to reconstitute the primary
operational strings from the failed pools onto pools that are safe,
thereby ensuring critical mission functionality is available
throughout the system lifecycle.

As shown in Figure~\ref{fig:gt-2}, the failure scenario begins with
the failure of host {\em Checkmate} and results in the loss of
secondary replicas for the primary applications in Pool 1.B.  The ARMS
MLRM is configured to {\em not} restart the failed secondaries for
experimental purposes.  We next failed Pool 1.B, which resulted in a
loss of the primary applications in Pool 1.B.  There is no active
replication since the secondaries failed first, so the MLRM must
restart the applications in the surviving resource pool to regain lost
capability.
%%
%% JAI : adding a reviewer comment.
%% Experiments (Figure 9) in Section 3.3 are interesting.
%% It will be better if the paper can give a little more
%% introduction on how the DRM detects resource failures
%% and reallocates applications. Is "half the 4 minutes" 
%% a little longer for system recovery?
%%
%% JAI: Adding the following lines as Tom suggested.
%%
%%
As described in Section~\ref{sec:mlrm-structure} and
Section~\ref{sec:mlrm-dynamics}, the ARMS MLRM defines Condition
Monitors, Determinators, and Response Coordinators to detect and
recover from pool failures.  We deployed Pool 1.B's failure detectors
in Pool 1.A to monitor heartbeats from Pool 1.B's Resource Status
Service\footnote{This failure detection scheme was sufficient to
evaluate the MLRM framework's failure and subsequent reactive resource
management capabilities.}.  When a failure was induced by
disconnecting Pool 1.B's network from the router, the failure
detectors detected the failure after missing three heartbeats, which
triggered the Pool Failure Response Coordinator to start failure
recovery actions.

As this scenario unfolded, we focused on the time the ARMS MLRM needed
to (1) detect the pool failure and decide what needs to be redeployed,
(2) redeploy the strings in the available pools and their resources,
(3) allow the OS to spawn a processing environment for the operational
strings, (4) give appropriate trigger signals to the strings so that
they can be part of the global mission functionalities by
collaborating with other strings, and (5) promote the new primaries
and secondaries in the new deployment configurations.  The results in
Figure~\ref{fig:gt-2-exp} show that ARMS MLRM takes $\sim$15 seconds
to complete parts 1 and 2, which are pool failure detection and
redeployment, respectively.
\begin{figure}[htbp]
  \centering
%  \includegraphics[width=\columnwidth]{gt2-experiment}
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{gt2-experiment}
  \bfcaption{Redeployment of Effectors from Failed Pool}
  \label{fig:gt-2-exp}
\end{figure}
The vast majority of this time is taken up in pool failure detection.
Conservative timeout values were used for failure detection to
minimize false alarms.  A more aggressive use of timeouts (either with
capabilities native to a compute node or with custom hardware) can
reduce this number to well below a second.  The time taken by part 2
(redeployment decisions) is well below a second.  A relatively large
gap is noted for part 3 (time to spawn new processes to host
application components), which is due to overhead in DAnCE that will
be fixed in a future release.  Finally, we note another 15 second lag
for parts 4 and 5 (initialization and triggering) of the redeployment
process.  The time to recover is therefore well below half the 4
minutes needed to recover a manually managed system.

%\sqzlarge
\subsection{MLRM Capability to Operate under Limited Resource Availabilities}
\label{mlrm-handle-overloads}
%\sqzlarge

Section~\ref{sec:mlrm-components} described how the ARMS MLRM manages
different sets of resources within a single node in a pool and
provides the capability to maintain the operation of critical mission
capabilities by gracefully responding to overload conditions, such as
handling more tracks and threats by offloading, reallocating, or
reprioritizing other {\em not-so-important} operational strings
operating in a node.  Below, we present empirical results that show
(1) how the ARMS MLRM can handle increased loads in a node to handle
additional threats and tracks and (2) how the MLRM can do this
gracefully by offloading less important operational strings from a
node.  The goal is to ensure that critical mission capabilities meet
their QoS requirements in the face of increasing workloads.

Figure~\ref{fig:gt-3} shows a scenario where three nodes are used to
deploy two operational strings that provide {\em general tracking} and
{\em air threat tracking} capabilities.
\begin{figure}[htbp]
  \centering
%  \includegraphics[width=\columnwidth]{figs/new-gt3-scenario}
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{figs/new-gt3-scenario}
  \bfcaption{Different Threat Tracking Components in the ARMS MLRM}
  \label{fig:gt-3}
\end{figure}
We measured the CPU utilization in the {\em Mako} node, where parts of
both the operational strings are deployed and launched.  We also
measured {\em Mako's} CPU utilization as air threats began to
increase, which we accomplished by configuring the {\em ED-1.1} sensor
to generate more messages -- and thereby more reactive events --
within the {\em PLAN-3.1} planner on {\em Mako}.

The planner makes decisions and uses the effector {\em EFF-1.1} in the
{\em Checkmate} node to take evasive actions.  In this scenario, {\em
Mako}'s CPU utilization went up sporadically as it processed a series
of air threats.  In the face of the increasing air threats, however,
the critical operational functionalities provided by {\em Mako} (such
as the planning process provided by the {\em PLAN-3.1} planner) need
to operate correctly and efficiently.  To evaluate the benefits of
DRM, we therefore compared the results adding and removing ARMS MLRM
capabilities to highlight its capabilities.

Figure~\ref{fig:mlrm-disabled} shows the results of an experiment that
disabled the ARMS MLRM capabilities.
\begin{figure}[htbp]
\sidebyside {
  \centerline{\includegraphics[width=3.3in]{gt3-mlrm-disabled}}
  \bfcaption{Handling Threats with MLRM Disabled} \label{fig:mlrm-disabled}} {
  \centerline{\includegraphics[width=3.3in]{gt3-mlrm-enabled}}
  \bfcaption{Handling Threats with MLRM Enabled}
  \label{fig:mlrm-enabled}}
\end{figure}
As the processing load for Plan-1.1 increased beyond a certain point
the system started to become unstable, with significant fluctuations
in the amount of resources consumed.  In contrast,
Figure~\ref{fig:mlrm-enabled} shows the results of enabling the ARMS
MLRM DRM capabilities.  as load for Plan-1.1 increased the MLRM made
the necessary dynamic adjustments to deliver steady resource
availability for this component.

In summary, the experimental evaluations in this section validated
that the ARMS MLRM met its goals, {\em i.e.}:
\begin{itemize}
\item Multiple dynamic allocations were achieved - five were
demonstrated.

\item Recovery from a data center failure is achieved in well under
half the time it takes for manual recovery

\item Under increased load MLRM makes the necessary resource
allocation adjustments to permit steady state increase in resource
availability to critical applications without any instability.
\end{itemize}
