% $Id: challenges.tex 19213 2009-08-17 10:06:43Z gokhale $
\section{Sources of Complexity in Configuring Middleware for DRE Systems}
\label{sec:qos-challenges}

This section describes the challenges in configuring middleware, which
is needed to enforce the QoS policies of DRE systems.  To better
explain these challenges, we describe them in the context of a
representative application hosted on a specific middleware -- in our
case real-time CORBA~\cite{RTCORBA:05} and the CIAO CORBA component
middleware that builds upon it. 


\subsection{Overview of Real-time CORBA and Component Middleware}
\label{sec:rtcorba}


\begin{figure}[htb]
  \vspace{-.2in}
  \includegraphics[width=\columnwidth,keepaspectratio]{rtcorba}
  \vspace{-.2in}
  \bfcaption{Real-time CORBA Middleware}
  \label{fig:rtcorba}
\end{figure}

Figure~\ref{fig:rtcorba} illustrates the Real-time CORBA (RTCORBA)
middleware architecture.  RTCORBA extends traditional CORBA artifacts,
such as (a) the object request broker (ORB), which mediates the request
handling between clients and servers, (b) the portable object adapter
(POA), which manages the lifecycle of CORBA objects, (c) stubs and
skeletons, which are generated by an interface definition language (IDL)
compiler that hide the distribution aspects from the communicating
entities, with real-time policies and interfaces.

% \begin{figure}[htb]
%   \includegraphics[width=\columnwidth,keepaspectratio]{rtcorba_features}
%   \bfcaption{Real-time CORBA Middleware Features}
%   \label{fig:rtcorba_features}
% \end{figure}

RTCORBA defines standard interfaces and QoS policies that allow 
applications to configure and control (1) \emph{processor resources} via
thread pools, priority mechanisms, intra-process mutexes, and a global
scheduling service, (2) \emph{communication resources} via protocol
properties and explicit bindings, and (3) \emph{memory resources} via
buffering requests in queues and bounding the size of thread pools.

Some of the important RTCORBA features we use in our examples
described in this paper are detailed below.  Applications
leverage these features by programmatically specifying policies on the
POA using standard CORBA operations, such as \texttt{create\_POA} or
\texttt{validate\_connection}. 

\begin{BulletList}
  \item \emph{Thread pools --} RTCORBA supports concurrency and
  prioritized handling of requests by virtue of thread pools that can
  be created by enforcing the thread pool policy on the POA that
  manages the server object.  Thread pools may also comprise a special
  feature called \emph{lanes.} Lanes enable multiple priorities to be
  assigned to a set of threads so that requests can be executed at
  different priorities.  RTCORBA allows the notion of lane borrowing
  wherein if lanes of higher priority are exhausted due to multiple
  client requests, one or more lower priority lanes can have their
  priorities raised temporarily to handle additional requests.  The
  size of the thread pools can be fixed statically with additional
  provisions to increase or decrease the size by dynamic creation and
  removal of threads.

  \item \emph{Priority bands --} RTCORBA supports a mechanism called
  priority bands to alleviate priority inversions on network
  connections due to a lower priority request blocking a higher
  priority request.  Priority bands enable multiple simultaneous
  connections between communicating peers where each band designates a
  certain priority of the request.

  \item \emph{Priority handling --} RTCORBA defines two priority
  models for handling requests from a client to a server.  For
  requests that have fixed priorities that are determined offline
  through other tools, the \texttt{SERVER\_DECLARED} policy is
  enforced on the POA that handles such client requests.  If the
  request handling priority is determined online by a client, this
  priority is propagated to the server which is made possible by
  enforcing the \texttt{CLIENT\_PROPAGATED} policy on the POA that
  handles these client requests.

\end{BulletList}

The CIAO component middleware is an implementation of the lightweight
CORBA Component Model (LwCCM)~\cite{LWCCM} and leverages RTCORBA.  DRE
system developers can realize large-scale DRE systems by assembling and
deploying CIAO-based LwCCM components.  Components within LwCCM can
have four different kinds of ports.  The cardinality of each port can
be one or more.   The provided port called a \emph{facet} represents
an interface published by the component.  The required port called a
\emph{receptacle} provides a mechanism to the component to leverage
services offered by other components.  The other two ports are the
\emph{event source} and \emph{event sink} used to publish and
subscribe to events, respectively.  A component assembly is a reusable
block of connected components, which can be viewed as a logical,
composite component.

The applications within these
DRE systems can use publish/subscribe communication semantics (by
using the component event source and sink ports) or request/response
communication semantics (by using the facet and receptacle ports).  The
QoS properties of these applications are realized by appropriately
configuring the CIAO middleware and its underlying RTCORBA mechanisms.
Instead of using programmatic APIs to use these mechanisms, they use
declarative mechanisms, such as XML metadata.  Thus, application
developers are required to specify the middleware-specific QoS
mechanisms in XML.


%%% include scenario here %%%
\input{scenario}

\subsection{Middleware Configuration Challenges}
\label{sec:issues}

We now discuss why mapping the domain-specific QoS policies into
middleware QoS configurations is challenging for DRE system
developers.

\noindent\textbf{Challenge 1. Inherent complexity in translating QoS
policies to QoS configuration options.}  Translating QoS policies into
QoS configuration options is hard because it must transform semantics from
the application domain to the semantics of the underlying component
middleware.  QoS-enabled component middleware like CIAO leverage RTCORBA
mechanisms to configure (1) \emph{processor resources}, such as portable
priorities, end-to-end priority propagation, thread pools, distributable
threads and schedulers, (2) \emph{communication resources}, such as
protocol properties and explicit binding of connections, and (3)
\emph{memory resources}, such as buffering of requests.  

To translate the domain-specified QoS policies into middleware QoS
configurations, DRE system developers need a thorough understanding of
the underlying middleware platforms.  For example, in the BasicSP
scenario assume that the air frame component has multiple other
sensors like the GPS publishing events at different priorities.
Developers must be able to determine the right request handling model
to handle priorities, and determine the right concurrency model
including whether lanes must be used, how many of them should be
created, how many should be created statically versus dynamically,
among many other decisions.  DRE developers are seldom experts in
low level details of the middleware.

\noindent\textbf{Challenge 2. Satisfying pre/post conditions and
  invariants in QoS configuration.} Even if a DRE developer were to
translate the QoS policies into a subset of QoS configuration options,
it is also necessary for them to understand the pre-conditions,
invariants, and post-conditions of the different QoS configuration
options since these conditions affect middleware behavior.  For
example, to create a thread pool with lanes in a server component, the
following (non-exhaustive) pre-/post-conditions and invariants must be
satisfied: 

\begin{BulletList}
\item \emph{Pre-conditions.} A real-time POA created within a real-time
ORB must be available, and the range of priorities (for the different
lanes) and the type of priority mapping scheme chosen must be compatible,
\emph{i.e.}, within the limits. 

\item \emph{Post-conditions.} A thread pool-with-lanes corresponding
to the different priorities, along with the requested number of static
(pre-defined) threads must be made available for use.

\item \emph{Invariants.} The real-time ORB should be able to match
incoming request priorities to the corresponding lanes, and must always
handle incoming requests for higher priority lanes before incoming
requests for lower priority lanes.
\end{BulletList}

This problem is exacerbated by the plethora of options and choices of
valid values for each option, as well as by the fact that choosing one
value for a particular option may have side effects on other options.
These side effects are sometimes manifested as overt failures, such as
failure to perform a mapping of CORBA priority to the underlying OS
priority because of insufficient priorities in the OS to support the
choice of priority mapping scheme, \emph{e.g.}, \emph{direct} mapping.
They may also be manifested, however, as hard-to-reproduce and/or
hard-to-debug runtime failures that only emerge during field testing,
or after deployment, which are much harder to detect and fix.

\noindent\textbf{Challenge 3. Resolving dependencies between QoS
configuration options.} Even with a thorough understanding of
middleware QoS configuration options, manually mapping policies to
configurations of the middleware cannot scale as the number of
entities to configure increases.   This problem stems from
dependencies between the different QoS configuration options of each
component, such as the dependency between the CORBA priority of a
component, the chosen priority mapping scheme (to map CORBA priority
to native OS priority), and the priority-banded connections policy
(which selects the appropriate connection to route requests based on
the request invocation priority). 

If the components are connected, the side effect of the connection
between components may also induce an inter-component option
dependency.  Since these dependencies can grow quadratically, it is
infeasible for developers to manage these dependencies manually.  In
DRE systems with many components, the effects of changing a QoS
configuration option on a component may affect many other directly
connected components, their connected neighbors and so on.  These
dependencies can rapidly degenerate into a very large number of QoS
configurations.  Keeping track of dependencies between options and
propagating the changes in one option to all options affected by that
change is critical during the QoS configuration phase.   

For example, the BasicSP scenario is a reusable component assembly
that can be part of a larger workflow of components.  Moreover,
although the BasicSP workflow is reusable, it is conceivable that in
different use cases, the components such as the timer may operate with
different frequencies leading to different middleware configurations.
As the number of components increases, the number of inter-component
dependencies increases proportionally.  

Verifying the correctness of the values of the different QoS
configuration options in isolation and together with connected
components is critical to the successful deployment and ultimately the
operation of DRE systems.  Once again, it is hard to verify the
correctness of these values without automated tool support.   
Depending on the frequency of changes made to the domain-specified QoS
policies during the development process, relying simply on empirically
validating a change in QoS configuration options becomes time
consuming at this scale, which slows down the design process
considerably and permits subtle and pernicious errors to occur. What
is needed is an automated tool support that can assure an
application's evolution throughout its entire lifecycle.

\iffalse
%%%% commenting this out
\noindent\textbf{Challenge 4. Ensuring validity of QoS configuration
options with changes in QoS policies.}  QoS configuration options
effect the non-functional behavior of a system, and thus are affected
by changes in the system environment.  For a DRE system to operate
effectively in hostile environments, component middleware and their
associated QoS configuration options may need to adapt to their current
conditions.  While it is useful to change QoS configuration options at
runtime to effect changes in behavior (such as re-prioritizing or
increasing/\-decreasing resource usage), such dynamic reconfigurations
may incur another set of challenges.  

In particular, not only must we handle static QoS configuration problems
(such as checking validity of values and keeping track of dependencies),
there is typically little leeway to accommodate misconfiguration at
runtime.  It is non-trivial to change a running system because the
system might crash during reconfiguration due to misconfiguration of QoS
options.  Moreover, the reconfiguration process itself must be
predictable for the reconfiguration to have the desired effect on system
behavior.  In a DRE system, for instance, a reconfiguration done too
late may be worse than not performing a reconfiguration at all.

An exhaustive evaluation of possible choices of QoS configuration options
and validation of the reconfigured state is too time consuming to perform
at runtime and can delay the reconfiguration process itself, rendering it
useless.  Once again, tools are needed to help validate and automate this
reconfiguration process.  
\fi


\smallskip
The remainder of this paper shows how QUICKER helps to address these
challenges.
\iffalse
Section~\ref{sec:algo} describes our approach based on model-to-model 
translation that addresses Challenge 1.  Sections~\ref{sec:verify}
describes how QUICKER checks the validity of the values for the
different options and combinations of options according to the semantics
of the underlying middleware thereby addressing Challenge 2.
Section~\ref{sec:modelcheck} describes how QUICKER helps applications
evolve by automatically (re-)calculating the dependencies between
options, and can thus be used repeatedly by DRE developers during the
entire lifecycle thereby addressing Challenges 3 and 4.  Note that the model
checking is not applied at runtime.  We expect that system developers
will formally define the regions of operation and the desird QoS
policies as system conditions vary.  Accordingly QUICKER can synthesize
and verify the configurations for all the operating regions.
\fi

