% $Id: mlrm.tex,v 1.39 2005/11/23 19:54:35 gokhale Exp $

\section{The ARMS Multi-layered Resource Management Middleware}
\label{sec:mlrm}

Section~\ref{sec:intro} motivated the need for enterprise DRE systems
to adapt dynamically to changes in mission goals, variations in
resource availability, and capability upgrades.  This section
describes the component-based middleware used to implement the ARMS
MLRM architecture, focusing on the design goals that guided our
approach.

\subsection{ARMS MLRM Design Goals}
\label{sec:mlrm-goals}

Providing effective DRM capabilities for enterprise DRE systems
depends on several factors that span the domain- and solution-space.
For example, solutions to domain-specific issues, such as adapting to
mission mode changes, capability upgrades or resource failures, are
impacted by the choice of technologies and platforms used in the
solution space.  Addressing the complex problem of DRM as a single
unit, however, can become intractable due to the {\em tangling of
concerns} across the domain- and solution-space.  Hence, there is a
need to address DRM problems at different levels of abstraction, yet
maintain a coordination between these levels.  These design goals
motivate the ARMS MLRM framework described in this section, which we
use to resolve the DRM challenges of TSCEs.

The goals of the ARMS MLRM design are to provide DRM solutions when
missions change, resources fail or become available, failures occur
due to battle-damage, or new capabilities are added to the system.
\begin{figure}[htpb]
  \centering
  \includegraphics[width=5in]{MLRM-goals}
  \caption{Types of TSCE DRM Problems}
  \label{fig:event-classes}
\end{figure}
Figure~\ref{fig:event-classes} illustrates the types of DRM problems
and operational context that we addressed to meet the needs of TSCEs:

% Jai, YIKES - you had merged the discussion of operational strings
% from eval.tex here without explaining what you meant by this term,
% so I totally rewrote things to avoid mentioning this term until
% later when we talk about the MLRM architecture in detail.
$\bullet$ {\bf Mission mode changes}, where the goal is to enable a
much broader set of resource re-allocations beyond mode changes and
behaviors typically provided by domain applications.  Due to the
changes in environmental conditions, the availability of hardware
nodes in a TSCE vary.  Moreover, the fraction of capacity available
for each application component on a given node varies depending on the
number, type, and workload of application components hosted in each
node.  Meeting these goals requires the ARMS MLRM to determine {\em at
runtime} which components should actually run in response to mission
mode changes.  Moreover, the MLRM must tune application performance
parameters dynamically using increasingly finer-grained precision, as
opposed to a coarse-grained, discrete set of configurations.

$\bullet$ {\bf Load changes}, where the goal is to tolerate {\em
out-of-spec} variations gracefully.  For example, a statically
allocated TSCE responding to radar tracks may be provisioned to handle
a maximum of no more than 400 tracks.  A more effective TSCE design,
however, should respond gracefully to overload situations, such as
handling more than 400 tracks by reallocating and/or reprioritizing
other components in accordance with mission importance parameters.
The ARMS MLRM therefore ensures that available TSCE resources are
allocated to the {\em currently most important} mission capabilities,
allowing the TSCE to scale to even {\em out-of-spec} loads gracefully.

$\bullet$ {\bf Resource changes}, where the goal is to restore {\em
all} mission capabilities, even those that were not designed to
tolerate specific failures.  Through the automatic recovery from such
failures, the ARMS MLRM maximizes TSCE resource utilization and
performance, while simultaneously maximizing the value of mission
capabilities.  In addition, we wanted to expand mission capabilities
as new resources become available due to requirement or environment
changes.

$\bullet$ {\bf Capability upgrades}, where the goal is to allow
capabilities to be dynamically introduced into the system that were
not planned initially.  We wanted to capture the general behavior of
the newly introduced artifacts so the ARMS MLRM can determine the
resource requirements of these artifacts dynamically and make
appropriate allocations.  In particular, the ARMS MLRM should be able
to make dynamic resource allocation decisions for the newly introduced
artifacts, even when such capabilities were not part of the original
system.

$\bullet$ {\bf Platform upgrades}, where the goal is to support the
{\em heterogeneous} environment in which long-lived TSCEs operate.
For example, our TSCE is designed to support different versions of (1)
distributed computing middleware, such as TAO/CIAO in C++,
JacORB/OpenCCM in Java, and OrbRiver in Ada, (2) general-purpose
POSIX-based operating systems, such as Linux and Solaris, (3)
real-time operating systems, such as VxWorks and LynxOS, (4) hardware
chipsets, such as x86, PowerPC, and SPARC processors, (5) a wide range
of high-speed wired interconnects, such as Gigabit Ethernet and VME
backplanes, and (6) different transport protocols, such as TCP/IP and
SCTP.

The ARMS MLRM services are implemented as a set of common and
domain-specific middleware services (described in
Section~\ref{sec:mlrm-components}) that communicate using
standards-based middleware (described in Section~\ref{ciao-dance})
that uses a wide range of standard/\-proprietary operating systems,
networks, programming languages, and hardware.

\subsection{The ARMS MLRM QoS-enabled Middleware Infrastructure}
\label{ciao-dance}

To simplify development and enhance reusability, the ARMS MLRM is
based on a QoS-enabled component middleware infrastructure comprised of
the {\em Component-Integrated ACE ORB} (CIAO)~\cite{Schmidt:03m} and
the {\em Deployment And Configuration Engine}
(DAnCE)~\cite{Schmidt:05y}.  CIAO and DAnCE implement the {\em
Real-time CORBA Component Model} (RT-CCM)~\cite{Wang:03b}, which
combines standard Lightweight CCM~\cite{CCM-light:03} mechanisms (such
as codified specifications for specifying, implementing, packaging,
assembling, and deploying components) and standard Real-time
CORBA~\cite{RTCorba:02} mechanisms (such as thread pools, priority
preservation policies, and explicit binding capabilities) to simplify
and automate the (re)deployment and (re)configuration of application
components in enterprise DRE systems.  CIAO and DAnCE use
patterns~\cite{Markus:02,Schmidt:00a} to achieve highly portable and
standards-compliant component middleware and reflective middleware
techniques~\cite{Schmidt:04w,Schmidt:01m} to support key QoS aspects
for enterprise DRE systems.

\begin{figure}[htb]
   \includegraphics[width=6in,keepaspectratio]{figs/ciao-dance.eps}
   \bfcaption{CIAO and DAnCE Architecture}
   \label{fig:ciao}
 \end{figure}
As shown in Figure~\ref{fig:ciao}, {\em components} in CIAO are
implementation entities that collaborate with each other via {\em
ports}, including (1) {\em facets}, which define an interface that
accepts point-to-point method invocations from other components, (2)
{\em receptacles}, which indicate a dependency on point-to-point
method interface provided by another component, and (3) {\em event
sources/\-sinks}, which can be used to exchange typed events with one
or more components.  {\em Containers} in CCM provide a run-time
environment for one or more components that manages various
pre-defined hooks and strategies (such as persistence, event
notification, transaction, and security) used by the component(s).
Each container is responsible for (1) initializing instances of the
component types it manages and (2) connecting them to other components
and common middleware services.  There are two categories of
components in CIAO: (1) {\em monolithic components}, which are
executable binaries, and (2) {\em assembly-based components}, which
are a set of interconnected components that can either be monolithic
or assembly-based (note the intentional recursion).

Component assemblies in CIAO are deployed and configured via DAnCE,
which is an implementation of standard OMG {\em Deployment and
Configuration} (D\&C) specification~\cite{DandC:03} shown in
Figure~\ref{fig:ciao}.  DAnCE manages the mapping of application
components onto nodes in a target environment.  The information about
the component assemblies and the target environment in which the
application components will be deployed are captured in the form of
standard XML assembly descriptors and deployment plans.  DAnCE's
runtime framework parses these descriptors and plans to extract
connection and deployment information and then automatically deploys
the assemblies onto the CIAO component middleware platform and
establishes the connections between component ports.

\subsection{The Component-based ARMS MLRM Design}
\label{sec:mlrm-components}

The ARMS MLRM architecture integrates resource management and control
algorithms based on the standards-based CIAO and DAnCE RT-CCM
infrastructure described in Section~\ref{ciao-dance} to achieve the
enterprise DRE system challenges described in
Section~\ref{sec:motiv-chal} and the ARMS MLRM design goals described
in Section~\ref{sec:mlrm-goals}.  These design goals, combined with
the complexity of the TSCE domain, led us to model the DRM solution
space as a layered architecture comprising components at different
levels of abstraction.  Figure~\ref{fig:mlrm-components} depicts the
layers in the ARMS MLRM, which has hundreds of different types and
instances of infrastructure components written in $\sim$300,000 lines
of C++ code and residing in $\sim$750 files developed by different
teams at different locations.
\begin{figure}[htp]
  \centering
  \includegraphics[width=5in]{MLRM-components}
  \caption{Components in the  ARMS MLRM}
  \label{fig:mlrm-components}
\end{figure}
The remainder of this section describes the structure and dynamics of
the ARMS MLRM.

\subsubsection{ARMS MLRM Structure}

We first describe the key goals of each layer in
Figure~\ref{fig:mlrm-components} and explain how the RT-CCM-based
components within these layers help address the goals described in
Section~\ref{sec:mlrm-goals}.

% Patrick, can you please take a careful pass through the following
% two paragraphs and make sure they accurately explain how things work
% at this layer?!
$\bullet$ The {\bf DRM Services layer} is responsible for satisfying
TSCE missions, such as allocating system computing and networking
resources to respond to radar tracks and planning necessary actions or
events.  The goal of this layer is to maximize the mission coverage
and reliability in response to the resource management problems shown
in Figure~\ref{fig:event-classes}.  The DRM Services layer receives
explicit resource management requests from applications, along with
command and policy inputs.  It decomposes mission requests on the TSCE
into a coordinated set of software allocation requests on resource
pools within the {\em Resource Pool layer}.  The DRM Services layer
also monitors and coordinates the execution of {\em operational
strings}, which are sequences of components that capture the partial
order and workflow of a set of executing software capabilities.

Key components in this layer include the (1) {\em Infrastructure
Allocator}, which determines the resource pool(s) where a package's
operational string(s) are deployed, (2) {\em Operational String
Manager}, which coordinates the proper deployment of operational
strings across resource pools.  The DRM services layer can manipulate
groups of operational strings by (1) deploying, stopping, and
migrating them depending on their mode of operation, (2) deploying
them in different configurations to support different modes, and (3)
deploying them in different orders depending on mode transitions
within the TSCE.  The ARMS MLRM manages operational strings based on
their priority, {\em i.e.}, it (re)allocates resources for the
operational strings satisfying higher priority goals.  The MLRM may
therefore stop or migrate lower priority operation strings if
resources become scare so that higher priority operational strings
remain operational.

$\bullet$ The {\bf Resource Pool layer} is an abstraction for a set of
computer nodes managed by a {\em Pool Manager} component.  The {\em
Pool Manager}, in turn, interacts with a {\em Resource Allocator}
component to run algorithms that deploy application components to
various nodes within the resource pool.  The goal of this layer is to
handle the two complementary capabilities:

\begin{enumerate}\cramplist
\item {\bf Proactively allocate resources so that QoS requirements for
all critical operational strings are satisfied within a single pool}.
The DRM services layer must ensure that resource allocations satisfy
QoS needs of operational strings in a TSCE.  The ARMS MLRM employs
importance-ordered, uni-dimensional bin-packing~\cite{Liu:00} of
worst-case application CPU resources to allocate operational strings
to computing nodes.  It also employs a network provisioning
algorithm~\cite{Das:2005} that allocates network bandwidth based on
operation string interactions.
% Jai, do we indeed now refer to this stuff as Gate Metric 1 etc in
% the eval.tex file?
%%
%% JAI : no, we do not do that.
%% So i am removing the term Gate metric here and instead refer to the
%% appropriate subsections in the experimental results section.
%%
The experiments in Section~\ref{mlrm-deploy-infinite-configurations} demonstrate how
unique infrastructure resource allocations were generated in response
to different sequences of mission deployment requests (an example of
the {\em mission mode change} problem in
Figure~\ref{fig:event-classes}).  When appropriate metadata is
provided, ARMS MLRM also performs end-to-end response time
schedulabililty analysis~\cite{JunSun:97} to verify that allocations
can satisfy real-time deadlines of operational strings.

\item {\bf Reactively re-allocate resources or operational modes to
tune deployed operational strings to current mission priorities.}  The
ARMS MLRM defines {\em Condition Monitors}, {\em Determinators}, and
{\em Response Coordinators} components~\cite{Schantz:98, Paulo:03} to
detect, verify, and recover from classes of execution and performance
problems in deployed operational strings.  These components deploy
controllers for managing operational string overload (an example of a
{\em load change} problem from Figure~\ref{fig:event-classes}).  
%%
%% JAI : commenting out Gate metric experiments 2 and 3.
%% 
%%
The experiments in Section~\ref{mlrm-handle-failures} and
Section~\ref{mlrm-handle-overloads} demonstrate
how these services help to (1) restore maximum capability after a resource
pool failure (an example of a {\em resource change} problem from
Figure~\ref{fig:event-classes}), and (2) manage operational string
overload to allow critical applications to continue operating.
\end{enumerate}

\iffalse
% Guys, let's omit this in the interest of space.  If necessary, we
% could put it in the Future Work section, but I still think we're
% taking up too much space, so let's omit it for now!
In Phase II of the ARMS program we are developing a framework, the
Resource Allocation and Control Engine (RACE), for deployment time
composition of allocation and control algorithms into DRM Services
tailored to individual resource pool requirements.
\fi

$\bullet$ The {\bf Physical Resources layer} deals with the specific
instances of resources in the TSCE.  The goal of this layer is to
configure physical resources in accordance with dynamically-generated
allocations to support the QoS needs of a mission.  The ARMS MLRM
configures OS process priorities and scheduling classes of deployed
components across a variety of operating systems ({\em e.g.}, Linux,
Solaris, and VxWorks) in a manner that preserves execution precedence
specified in the dynamically generated allocation.

\subsubsection{ARMS MLRM Dynamics}
The ARMS MLRM acts in response to incidences of resource management
problems of the types shown in Figure~\ref{fig:event-classes}.  To
illustrate the dynamic behavior of the ARMS MLRM, we next describe the
two examples shown in Figure~\ref{fig:mlrm-wflow}: a {\em proactive
workflow} in response to a mission mode change request and a {\em
reactive workflow} in response to a load change in an operational
string~\cite{Cross:01}.
\begin{figure}[htbp]
  \centering \includegraphics[width=\columnwidth]{mlrm-workflow}
  \bfcaption{MLRM Workflow}
  \label{fig:mlrm-wflow}
\end{figure}
The experimental results for both these workflows are described in
Section~\ref{sec:eval}. The sequence of activities happening in a
typical TSCE environment as the system goes through proactive and
reactive
% Jai, do we indeed show these as numbered steps in the figure?  More
% importantly, do each of the steps in the figure correspond 1-to-1
% with the numbered steps described below?  If not, things need to be
% changed to correspond 1-to-1.
%%
%% JAI : yes, they correspond to 1-1.
%%
workflows are shown in Figure~\ref{fig:mlrm-wflow} and described
below:
\begin{enumerate}

\item TSCE mission management services selectively deploy and suspend
mission software packages in response to tactical needs.  A TSCE
mission engineer designs the sets of operational strings to be
deployed based on the current mission and also captures the system
profile of all operational strings.

\item Deploying a new mission package begins with a deployment request
to the DRM Service layer's {\em Infrastructure Allocator}, which can
split operational strings across resource to balance resource load or
to satisfy various constraints ({\em e.g.}, place primary and replicas
in different geographic locations).

\item In accordance with each operational string's start-up order
constraints, the DRM services layer's {\em Operational String Manager}
directs the {\em Pool Manager} components in each resource pool to
deploy one or more substrings.

\item If the deployment policy is dynamic, each {\em Pool Manager}
uses a {\em Resource Allocator} component in the {\em Resource Pool
layer} to assign applications to hosts, generate OS priorities and
scheduling classes, and coordinate with the {\em Bandwidth Broker}
component in the {\em Resource Pool layer} to reserve network
resources.  The {\em Resource Allocator} obtains the existing resource
allocations from the pool's {\em Resource Status
Service}~\cite{Zinky:2002} and analyzes schedulabililty of the
newly generated allocation to determine whether critical path
deadlines can be satisfied.

\item The {\em Pool Manager} directs the {\em Node Provisioner}
components on each node within the {\em Resource Pool layer} to
implement those allocation directives.

% Jai, do we actually show this as a separate step in this figure?
%%
%% JAI : yes. But the figure only shows a line going from the
%% the NodeProvisioner to the deployed system, but what it does
%% is explained here. The figure says "allocate and provision components",
%%
\item The allocation directives done by the {\em Node Provisioner}
include (1) launching new applications and configuring their QoS
parameters ({\em e.g.}, OS priority, scheduling class, and Diffserv
codepoint tagging) and (2) reconfiguring QoS parameters of previously
deployed applications that are shared by the newly deployed
operational string.  After all pool allocations are complete, the {\em
Operational String Manager} initiates the final start-up of the new
operational string(s) by invoking their application-level {\tt
start()} methods in accordance with their ordering constraints.

\item Deployed operational strings may declare various performance
requirements in their metadata, {\em e.g.}, a minimum average CPU
utilization for each application or an end-to-end deadline.  The ARMS
MLRM deploys {\em Condition Monitors} in the {\em Resource Pool layer}
to detect whether such performance requirements are being satisfied

\item When a {\em Condition Monitor} detects a performance requirement
violation, it notifies a {\em Determinator} in the {\em Resource Pool
layer}, which considers the violation in the broader context of
mission importance, resource availability, and policy directives to
determine whether the problem must be addressed and on what timeline.

\item When the {\em Determinator} decides that a problem must be
addressed it notifies a {\em Response Coordinator} in the {\em
Resource Pool layer} to initiate recovery actions. We developed a
coordinated set of {\em Condition Monitors}, {\em Determinators}, and
{\em Response Coordinators} components to support the string overload
problem described in the experiments discussed in
Section~\ref{mlrm-handle-overloads}.  In this case, the {\em Condition
Monitor} detected that (after an increase in external threats, which
exceeded the design threshold) a critical application, in a high
importance operational string, was being starved by a less important
competing operational string.

\item The {\em Response Coordinator} addresses starvation by lowering
the QoS parameters ({\em i.e.}, OS priority and scheduling class) of
the competing applications using resource allocation services provided
by the {\em Pool Manager}, {\em Resource Allocator}, and {\em Node
Provisioner} components in the local resource pool.  By reconfiguring
the TSCE using these components, the ARMS MLRM helped ensure that the
maximum number of mission-critical operational strings were scaled to
meet out-of-spec load changes within available resource constraints.
\end{enumerate}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "JSS-2006"
%%% End: 
