% $Id: mlrm.tex 3601 2006-07-24 19:35:41Z jai $

%\sqzlarge
\section{The ARMS Multi-layered Resource Management Middleware}
\label{sec:mlrm}

This section describes the design and functionality of the component
middleware used to implement the ARMS MLRM architecture.

\subsection{ARMS MLRM Design Goals}
\label{sec:mlrm-goals}

Providing effective DRM capabilities for enterprise DRE systems
depends on several factors that span the domain- and solution-space.
For example, solutions to domain-specific issues, such as adapting to
mission mode changes, capability upgrades or resource failures, are
impacted by the choice of technologies and platforms used in the
solution space.  Addressing the complex problem of DRM as a single
unit, however, can become intractable due to the {\em tangling of
concerns} across the domain- and solution-space.  Hence, there is a
need to address DRM problems at different levels of abstraction, yet
maintain a coordination between these levels.  These design goals
motivate the ARMS MLRM framework described in this section, which we
applied to help resolve key DRM challenges in the ARMS program.

The goals of the ARMS MLRM design are to provide DRM solutions when
missions change, resources fail or become available, failures occur
due to damage, or new capabilities are added to the system.
\iffalse
\begin{figure}[htpb]
  \centering
  \includegraphics[width=0.75\columnwidth]{MLRM-goals}
  \bfcaption{Types of DRM Problems for Enterprise DRE Systems}
  \label{fig:event-classes}
\end{figure}
Figure~\ref{fig:event-classes} illustrates the types of DRM problems
and operational context that
\fi
We addressed the following types of DRM problems to meet the needs of
enterprise DRE systems:

$\bullet$ {\bf Mission mode changes}, where the goal is to enable a
much broader set of resource reallocations beyond mode changes and
behaviors typically provided by domain applications.
\iffalse
Due to the changes in environmental conditions, the availability of
hardware nodes in enterprise DRE systems often vary.  Moreover, the
fraction of capacity available for each application component on a
given node varies depending on the number, type, and workload of
application components hosted in each node.
\fi
Meeting this goal requires the ARMS MLRM to determine {\em at runtime}
which components should actually run in response to mission mode
changes.  Moreover, the MLRM must tune application performance
parameters dynamically using increasingly finer-grained precision, as
opposed to a coarse-grained, discrete set of configurations.

$\bullet$ {\bf Load changes}, where the goal is to tolerate {\em
out-of-spec} variations gracefully.
\iffalse
For example, a statically allocated enterprise DRE system responding
to radar tracks may be provisioned to handle a maximum of no more than
400 tracks.  A more effective design, however, should respond
gracefully to overload situations, such as handling more than 400
tracks by reallocating and/or reprioritizing other components in
accordance with mission importance parameters.
\fi
The ARMS MLRM therefore ensures that available resources are allocated
to the {\em currently most important} mission capabilities, allowing
the system to scale to even {\em out-of-spec} loads gracefully.

$\bullet$ {\bf Resource changes}, where the goal is to restore {\em
all} mission capabilities, even those that were not designed to
tolerate specific failures.  Through the automatic recovery from such
failures, the ARMS MLRM maximizes resource utilization and
performance, while simultaneously maximizing the value of mission
capabilities.  In addition, we wanted to expand mission capabilities
as new resources become available due to requirement or environment
changes.

$\bullet$ {\bf Capability upgrades}, where the goal is to allow
capabilities to be dynamically introduced into the system that were
not planned initially.  We wanted to capture the general behavior of
the newly introduced artifacts so the ARMS MLRM can determine the
resource requirements of these artifacts dynamically and make
appropriate allocations.  In particular, the ARMS MLRM should be able
to make dynamic resource allocation decisions for the newly introduced
artifacts, even when such capabilities were not part of the original
system.

$\bullet$ {\bf Platform upgrades}, where the goal is to support the
{\em heterogeneous} environment in which long-lived enterprise DRE
systems operate.  This environment involves diverse middleware,
operating systems, CPUs, and networks.
\iffalse
For example, our MLRM is designed to support
different versions of (1) distributed computing middleware, such as
TAO/CIAO in C++, JacORB/OpenCCM in Java, and OrbRiver in Ada, (2)
general-purpose POSIX-based operating systems, such as Linux and
Solaris, (3) real-time operating systems, such as VxWorks and LynxOS,
(4) hardware chipsets, such as x86, PowerPC, and SPARC processors, (5)
a wide range of high-speed wired interconnects, such as Gigabit
Ethernet and VME backplanes, and (6) different transport protocols,
such as TCP/IP and SCTP.
\fi

The ARMS MLRM services are implemented as a set of common and
domain-specific middleware services described in
Section~\ref{sec:mlrm-components} that communicate using the standard
middleware described in Section~\ref{ciao-dance} that coordinates and
encapsulates a wide range of operating systems, networks, programming
languages, and hardware.

%\sqzlarge
\subsection{The Component-based ARMS MLRM Design}
\label{sec:mlrm-components}
%\sqzlarge

The ARMS MLRM architecture integrates resource management and control
algorithms enabled by standard component middleware infrastructure
described in Section~\ref{ciao-dance} to achieve the enterprise DRE
system challenges described in Section~\ref{sec:motiv-chal} and the
ARMS MLRM design goals described in Section~\ref{sec:mlrm-goals}.
These design goals, combined with the complexity of the enterprise DRE
system domain, led us to model the DRM solution space as a layered
architecture comprising components at different levels of abstraction.
Figure~\ref{fig:mlrm-components} depicts the layers in the ARMS MLRM,
which has hundreds of different types and instances of infrastructure
components written in $\sim$300,000 lines of C++ code and residing in
$\sim$750 files developed by different teams at different locations.
The remainder of this section describes the structure and dynamics of
the ARMS MLRM.

\begin{figure}[htp]
  \centering
%  \includegraphics[width=5in, height=2.7in]{MLRM-components}
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{MLRM-components}
  \bfcaption{Components in the ARMS MLRM}
  \label{fig:mlrm-components}
\end{figure}

%\sqzlarge
\subsubsection{ARMS MLRM Structure}
\label{sec:mlrm-structure}
%\sqzlarge

%%
%% JAI: Adding a reviewer comment.
%% At the beginning of Section 2.3.1, a brief introduction of the 3 layers
%% can help readers to understand.
%% Tom suggested the following fix.
%%
%% JAI : Adding a reviewer comment.
%% Not all components shown in Fig 4 are covered in Section 2.3.1.
%% Tom suggests the following:
%% True not all components are detailed.  In the description on one layer we say 
%% "Key components in this layer include..."; I'd suggest that we add single 
%% sentence to the opening to say that only key components are described.  
%% The point (aside from the key components) is the layering, not the 
%% individual components.   So I added the "key" in front of 
%% "components" above, and a reference into section 2.3.2.
%%

We first describe the key goals of each layer of the ARMS MLRM
structure (as depicted in Figure~\ref{fig:mlrm-components}), including
the DRM Services layer, the Resource Pool layer, and the Physical
Resource layer, and explain how key components within these layers
help address the goals described in Section~\ref{sec:mlrm-goals}.  The
dynamics of these key MLRM components are described further in
Section~\ref{sec:mlrm-dynamics}.

% Patrick, can you please take a careful pass through the following
% two paragraphs and make sure they accurately explain how things work
% at this layer?!
$\bullet$ The {\bf DRM Services layer} is responsible for satisfying
shipboard computing missions, such as allocating system computing and
networking resources to tracks items of interest and planning
necessary actions.  The goal of this layer is to maximize the mission
coverage and reliability.
% in response to the resource management problems shown in Figure~\ref{fig:event-classes}.
The DRM Services layer receives explicit resource management requests
from applications, along with command and policy inputs.  It
decomposes mission requests into a coordinated set of software
allocation requests on resource pools within the {\em Resource Pool
layer}.  The DRM Services layer also monitors and coordinates the
execution of {\em operational strings}, which are sequences of
components that capture the partial order and workflow of a set of
executing software capabilities.

Key components in this layer include the (1) {\em Infrastructure
Allocator}, which determines the resource pool(s) where a package's
operational string(s) are deployed, (2) {\em Operational String
Manager}, which coordinates the proper deployment of operational
strings across resource pools.  The DRM services layer can manipulate
groups of operational strings by (1) deploying, stopping, and
migrating them depending on their mode of operation, (2) deploying
them in different configurations to support different modes, and (3)
deploying them in different orders depending on mode transitions
within the system.  The ARMS MLRM manages operational strings based on
their priority, {\em i.e.}, it (re)allocates resources for the
operational strings satisfying higher priority goals.  The MLRM may
therefore stop or migrate lower priority operation strings if
resources become scare so that mission critical and more important
operational strings remain operational.

$\bullet$ The {\bf Resource Pool layer} is an abstraction for a set of
computer nodes managed by a {\em Pool Manager} component.  The {\em
Pool Manager}, in turn, interacts with a {\em Resource Allocator}
component to run algorithms that deploy application components to
various nodes within the resource pool.  The goal of this layer is to
handle the two complementary capabilities:

\begin{enumerate}\cramplist
\item {\bf Proactively allocate resources so that QoS requirements for
all critical operational strings are satisfied within a single pool}.
The DRM services layer must ensure that resource allocations satisfy
QoS needs of operational strings.  The ARMS MLRM employs
importance-ordered, uni-dimensional bin-packing~\cite{Liu:00} of
worst-case application CPU resources to allocate operational strings
to computing nodes.  It also employs a network provisioning
algorithm~\cite{Das:2005} that allocates network bandwidth based on
operation string interactions.  The experiments in
Section~\ref{mlrm-deploy-infinite-configurations} demonstrate how
unique infrastructure resource allocations were generated in response
to different sequences of mission deployment requests.
% (an example of the {\em mission mode change} problem in Figure~\ref{fig:event-classes}).
When appropriate metadata is provided, ARMS MLRM also performs
end-to-end response time schedulabililty analysis~\cite{JunSun:97} to
verify that allocations can satisfy real-time deadlines of operational
strings.

\item {\bf Reactively re-allocate resources or operational modes to
tune deployed operational strings to current mission priorities.}  The
ARMS MLRM defines {\em Condition Monitors}, {\em Determinators}, and
{\em Response Coordinators} components~\cite{Schantz:98} to
detect, verify, and recover from classes of execution and performance
problems in deployed operational strings.  These components deploy
controllers for managing operational string overload.
% (an example of a {\em load change} problem from Figure~\ref{fig:event-classes}).
The experiments in Section~\ref{mlrm-handle-failures} and
Section~\ref{mlrm-handle-overloads} demonstrate how these services
help to (1) restore maximum capability after a resource pool failure
% (an example of a {\em resource change} problem from Figure~\ref{fig:event-classes})
and (2) manage operational string overload to allow critical
applications to continue operating.
\end{enumerate}

\iffalse
% Guys, let's omit this in the interest of space.  If necessary, we
% could put it in the Future Work section, but I still think we're
% taking up too much space, so let's omit it for now!
In Phase II of the ARMS program we are developing a framework, the
Resource Allocation and Control Engine (RACE), for deployment time
composition of allocation and control algorithms into DRM Services
tailored to individual resource pool requirements.
\fi

$\bullet$ The {\bf Physical Resources layer} deals with the specific
instances of resources in the system.  The goal of this layer is to
configure physical resources in accordance with dynamically-generated
allocations to support the QoS needs of a mission.  The ARMS MLRM
configures OS process priorities and scheduling classes of deployed
components across a variety of operating systems ({\em e.g.}, Linux,
Solaris, and VxWorks) in a manner that preserves execution precedence
specified in the dynamically generated allocation.

%\sqzlarge
\subsubsection{ARMS MLRM Dynamics}
\label{sec:mlrm-dynamics}
%\sqzlarge

%%
%% JAI : adding a reviewer comment.
%% In Section 2.3.2, it is not clear which are the two examples
%% or two workflows. It seems that there is only one workflow.
%%
%% Tom suggests the following:
%% This is a valid comment.   First, there are two #7 events; second,  
%% I would suggest that the reactive workflow have a dotted/shaded box 
%% around it to clearly delineate it from the proactive workflow ... 
%% or just take the main box and split it in two 
%% (allow some components and the deployed system to straddle both boxes).

The ARMS MLRM acts in response to incidences of resource management
problems.
% of the types shown in Figure~\ref{fig:event-classes}.
To illustrate the dynamic behavior of the ARMS MLRM, we next describe
the two examples shown in Figure~\ref{fig:mlrm-wflow}: a {\em
proactive workflow} in response to a mission mode change request and a
{\em reactive workflow} in response to a load change in an operational
string~\cite{Cross:01}.
\begin{figure}[htbp]
%  \centering \includegraphics[width=\columnwidth,height=3.5in]{mlrm-workflow}
  \centering \includegraphics[width=0.75\columnwidth,keepaspectratio]{mlrm-workflow}
  \bfcaption{MLRM Workflow}
  \label{fig:mlrm-wflow}
\end{figure}
The experimental results for both these workflows are described in
Section~\ref{sec:eval}. The sequence of activities happening in a
typical shipboard computing environment as the system goes through
proactive and reactive workflows are shown in
Figure~\ref{fig:mlrm-wflow} and described below:
\begin{enumerate}\cramplist

\item System mission management services selectively deploy and suspend
mission software packages in response to tactical needs.  To plan the
deployment at design time, a mission engineer specifies the sets of
operational strings to deploy based on the current mission and also
captures the system profile of all operational strings.

\item Deploying a new mission package begins with a deployment request
to the DRM Service layer's {\em Infrastructure Allocator}, which can
split operational strings across resource to balance resource load or
to satisfy various constraints ({\em e.g.}, place primary and replicas
in different geographic locations).

\item In accordance with each operational string's start-up order
constraints, the DRM services layer's {\em Operational String Manager}
directs the {\em Pool Manager} components in each resource pool to
deploy one or more operational strings.

\item If the deployment policy is dynamic, each {\em Pool Manager}
uses a {\em Resource Allocator} component in the {\em Resource Pool
layer} to assign applications to hosts, generate OS priorities and
scheduling classes, and coordinate with the {\em Bandwidth Broker}
component in the {\em Resource Pool layer} to reserve network
resources.  The {\em Resource Allocator} obtains the existing resource
allocations from the pool's {\em Resource Status
Service}~\cite{Zinky:2002} and analyzes schedulabililty of the
newly generated allocation to determine whether critical path
deadlines can be satisfied.

\item The {\em Pool Manager} directs the {\em Node Provisioner}
components on each node within the {\em Resource Pool layer} to
implement those allocation directives.

\item The allocation directives done by the {\em Node Provisioner}
include (1) launching new applications and configuring their QoS
parameters ({\em e.g.}, OS priority, scheduling class, and Diffserv
codepoint tagging) and (2) reconfiguring QoS parameters of previously
deployed applications that are shared by the newly deployed
operational string.  After all pool allocations are complete, the {\em
Operational String Manager} initiates the final start-up of the new
operational string(s) by invoking their application-level {\tt
start()} methods in accordance with their ordering constraints.

\item Deployed operational strings may declare various performance
requirements in their metadata, {\em e.g.}, a minimum average CPU
utilization for each application or an end-to-end deadline.  The ARMS
MLRM deploys {\em Condition Monitors} in the {\em Resource Pool layer}
to detect whether such performance requirements are being satisfied

\item When a {\em Condition Monitor} detects a performance requirement
violation, it notifies a {\em Determinator} in the {\em Resource Pool
layer}, which considers the violation in the broader context of
mission importance, resource availability, and policy directives to
determine whether the problem must be addressed and on what timeline.

\item When the {\em Determinator} decides that a problem must be
addressed it notifies a {\em Response Coordinator} in the {\em
Resource Pool layer} to initiate recovery actions. We developed a
coordinated set of {\em Condition Monitors}, {\em Determinators}, and
{\em Response Coordinators} components to support the string overload
problem described in the experiments discussed in
Section~\ref{mlrm-handle-overloads}.  In this case, the {\em Condition
Monitor} detected that after an increase in external threats, which
exceeded the design threshold, a critical application in the threat
response operational string was unable handle higher load because of
competing equally important threat tracking strings.

\item The {\em Response Coordinator} addresses starvation by lowering
the QoS parameters ({\em i.e.}, OS priority and scheduling class) of
the competing applications using resource allocation services provided
by the {\em Pool Manager}, {\em Resource Allocator}, and {\em Node
Provisioner} components in the local resource pool.  By reconfiguring
the system using these components, the ARMS MLRM helped ensure that
the maximum number of mission-critical operational strings were scaled
to meet out-of-spec load changes within available resource
constraints.  In this case, the ARMS MLRM temporarily elevates the
importance of the threat response string in accordance with changing
mission priorities, at the cost of reducing the performance of the
less important threat tracking string.
\end{enumerate}

%\sqzlarge
\subsection{The ARMS MLRM QoS-enabled Middleware Infrastructure}
\label{ciao-dance}
%\sqzlarge

To simplify development and enhance reusability, the ARMS MLRM
components described in Section~\ref{sec:mlrm-components} are based on
a QoS-enabled component middleware infrastructure consisting of the
{\em Component-Integrated ACE ORB} (CIAO)~\cite{Schmidt:03m} and the
{\em Deployment And Configuration Engine} (DAnCE)~\cite{Schmidt:05y}.
CIAO and DAnCE implement the {\em Real-time CORBA Component Model}
(RT-CCM).  RT-CCM combines standard Lightweight
CCM~\cite{CCM-light:03} mechanisms (such as codified specifications
for specifying, implementing, packaging, assembling, and deploying
components) and standard Real-time CORBA~\cite{RTCorba:02} mechanisms
(such as thread pools, priority preservation policies, and explicit
binding capabilities) to simplify and automate the (re)deployment and
(re)configuration of MLRM and application components in enterprise DRE
systems.  CIAO and DAnCE use patterns~\cite{Markus:02} to achieve
highly portable and standards-compliant component middleware and
reflective middleware techniques~\cite{Schmidt:01m} to support key QoS
aspects for enterprise DRE systems.

\iffalse
\begin{figure}[htb]
\begin{minipage}[b]{0.5\linewidth} % A minipage that covers half the page
\centering
\includegraphics[width=2.5in]{CCM-Layers}
  \bfcaption{CIAO Architecture} 
  \label{fig:ciao}
\end{minipage} %
\hspace{0.5cm} % To get a little bit of space between the figures
\begin{minipage}[b]{0.5\linewidth}
\centering
  \includegraphics[width=2.5in]{DAnCE}
  \bfcaption{DAnCE Architecture}
  \label{fig:dance}
\end{minipage}
\end{figure}
\fi


% This is all screwing things up but the above also does not work
\begin{figure}[htb]
\sidebyside {
%  \centerline{\includegraphics[width=2.3in]{CCM-Layers}}
  \includegraphics[width=2.3in]{CCM-Layers}
  \bfcaption{CIAO Architecture} \label{fig:ciao}}{
%  \centerline{\includegraphics[width=2.3in]{DAnCE}}
  \includegraphics[width=2.3in]{DAnCE}
  \bfcaption{DAnCE Architecture}
  \label{fig:dance}}
\end{figure}

As shown in Figure~\ref{fig:ciao}, {\em components} in CIAO are
implementation entities that collaborate with each other via {\em
ports}, including (1) {\em facets}, which define an interface that
accepts point-to-point method invocations from other components, (2)
{\em receptacles}, which indicate a dependency on point-to-point
method interface provided by another component, and (3) {\em event
sources/\-sinks}, which can be used to exchange typed events with one
or more components.  {\em Containers} in CCM provide a run-time
environment for one or more components that manages various
pre-defined hooks and strategies (such as persistence, event
notification, transaction, and security) used by the component(s).
Each container is responsible for (1) initializing instances of the
component types it manages and (2) connecting them to other components
and common middleware services.  There are two categories of
components in CIAO: (1) {\em monolithic components}, which are
executable binaries, and (2) {\em assembly-based components}, which
are a set of interconnected components that can either be monolithic
or assembly-based (note the intentional recursion).

MLRM and application component assemblies developed using CIAO are
deployed and configured via DAnCE, which implements the standard OMG
{\em Deployment and Configuration} (D\&C)
specification~\cite{DandC:03} shown in Figure~\ref{fig:dance}.  DAnCE
manages the mapping of MLRM and application components onto nodes in
our target environment.  The information about the component
assemblies and the target environment in which the components will be
deployed are captured in the form of standard XML assembly descriptors
and deployment plans.  DAnCE's runtime framework parses these
descriptors and plans to extract connection and deployment information
and then automatically deploys the assemblies onto the CIAO component
middleware platform and establishes the connections between component
ports.

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "JSS-2006"
%%% End: 
