% $Id: challenges.tex 16638 2009-01-29 19:00:10Z gokhale $
\section{Evaluating QoS Configuration Techniques for DRE Systems}
\label{sec:qos-challenges}

Although QoS-enabled component middleware like CIAO~\cite{Schmidt:06r} and MDE tools
like PICML simplify many aspects of assembly, packaging, resource
management, and deployment in DRE systems, the following key
challenges remain with respect to the configuration of QoS for
components that comprise the DRE systems (although our discussion
focuses on the CIAO QoS-enabled component middleware, these challenges
manifest themselves in any highly configurable component middleware
including EJB and Microsoft .NET framework):%. In this section, we
%first describe a representative DRE system case study, followed by
%a detailed discussion on the middleware configuration challenges for DRE
%systems similar to this case study.

\noindent\textbf{Challenge 1. Inherent complexity in translating QoS
policies to QoS configuration options.}  Translating QoS policies into
QoS configuration options is hard because it must transform semantics from
the application domain to the semantics of the underlying component
middleware.  QoS-enabled component middleware like CIAO provides mechanisms
to configure (1) {\em processor resources}, such as portable priorities,
end-to-end priority propagation, thread pools, distributable threads and
schedulers, (2) {\em communication resources}, such as protocol properties
and explicit binding of connections, and (3) {\em memory resources}, such
as buffering of requests.  To translate the QoS policies into QoS
mechanisms by configuring the QoS options, application developers need a
thorough understanding of the underlying middleware platforms.

\noindent\textbf{Challenge 2. Ensuring validity of QoS configuration
options.} Assuming that a domain expert can translate the QoS policies
into a subset of QoS configuration options, it is also necessary to
understand the pre-conditions, invariants, and post-conditions of the
different QoS configuration options since they affect middleware
behavior.  For example, to create a thread pool with lanes in 
a server component, the following (non-exhaustive) are the
pre-/post-conditions and invariants:
\begin{BulletList}
\item \emph{Pre-conditions.} A Real-time Portable Object Adapter
created within a Real-time ORB is available, and the range of
priorities (for the different lanes) and the type of priority mapping
scheme chosen are compatible, \emph{i.e.}, within the limits.

\item \emph{Post-conditions.} A thread pool with lanes corresponding
to the different priorities, along with the requested number of static
(pre-defined) threads is available for use.

\item \emph{Invariants.} The Real-time ORB will match incoming request
priorities to the corresponding lanes, and will always handle incoming
requests for higher priority lanes before incoming requests for lower
priority lanes.
\end{BulletList}

This problem is exacerbated by the plethora of options and choices of valid
values for each option, as well as by the fact that choosing one value for
a particular option may have side effects on other options.  These side
effects are sometimes manifested as overt failures, such as failure to
perform a mapping of CORBA priority to OS priority because of insufficient
priorities in the OS to support the choice of priority mapping scheme,
\emph{e.g.}, \emph{direct} mapping.  They may also be manifested, however,
as hard-to-reproduce and/or debug runtime failures that only emerge during
field testing, or after deployment, which are much harder to detect and
fix.

Validating the values of the different QoS configuration
options in isolation and together with connected components is
critical to the successful deployment and ultimately the operation of
DRE systems.  Once again, it is hard to validate these values without
automated tool support.  Sections~\ref{sec:algo} and ~\ref{sec:verify} therefore 
describe how the QUICKER MDE tool checks the validity of the values for the
different options and combinations of options according to the
semantics of the underlying middleware, {i.e.}, CIAO's Real-time CCM
support.

\noindent\textbf{Challenge 3. Resolving dependencies between QoS
configuration options.} Even with a thorough understanding of
middleware QoS configuration options, manual configuration of QoS policies
does not scale as the number of entities to configure increases.  This lack
of scalability stems from dependencies between the different QoS
configuration options of each component, such as the dependency between the
CORBA priority of a component, the chosen priority mapping scheme (to map
CORBA priority to native OS priority), and the priority-banded connections
policy (which selects the appropriate connection to route requests based on
the request invocation priority).  As the number of components increases,
the number of intra-component dependencies increases proportionally.  If
the components are connected, the side effect of the connection between
components may also induce an inter-component option dependency.  Since
these dependencies can grow quadratically, it is infeasible for
developers to manage these dependencies manually.

In DRE systems with many components, the effects of changing a QoS
configuration option on a component may affect many other directly
connected components, their connected neighbors and so on.  These
dependencies can rapidly degenerate into a very large number of QoS
configurations.  Depending on the frequency of changes, empirically
validating a change in QoS configuration options becomes
time consuming at this scale, which slows down the design process
considerably and permits subtle and pernicious errors to occur.

Keeping track of dependencies between options and propagating the
changes in one option to all options affected by that change is
critical during the QoS configuration phase.  What is needed is an
automated tool support that can assure an application's evolution
throughout its entire lifecycle.  Section~\ref{modelcheck} therefore
describes how the QUICKER MDE tool helps applications evolve by
automatically (re-)calculating the dependencies between options, and
can thus be used repeatedly by DRE developers during the entire
lifecycle.

\noindent\textbf{Challenge 4. Ensuring validity of QoS configuration
options with changes in QoS policies.}  QoS configuration options
effect the non-functional behavior of a system, and thus are affected
by changes in the system environment.  For a DRE system to operate
effectively in hostile environments, such as space missions, component
middleware and their associated QoS configuration options may need to
adapt to their current conditions.  Middleware that can only be
configured statically (\emph{i.e.}, at design- or
installation-time)---but does not allow dynamic reconfiguration---may
be of limited use in these scenarios.

While it is useful to change QoS configuration options at runtime to effect
changes in behavior (such as re-prioritizing or increasing/\-decreasing
resource usage), such dynamic reconfigurations may incur another set of
challenges.  In particular, not only must we handle static QoS
configuration problems (such as checking validity of values and keeping
track of dependencies), there is typically little leeway to accommodate
misconfiguration at runtime.  It is non-trivial to change a running system
because the system might crash during reconfiguration due to
misconfiguration of QoS options.  Moreover, the reconfiguration process
itself must be predictable for the reconfiguration to have the desired
effect on system behavior.  In a DRE system, for instance, a
reconfiguration done too late may be worse than not performing a
reconfiguration at all.

An exhaustive evaluation of possible choices of QoS configuration options
and validation of the reconfigured state is too time consuming to perform
at runtime and can delay the reconfiguration process itself, rendering it
useless.  Once again, tools are needed to help validate and automate this
reconfiguration process.  %Section~\ref{modelcheck} therefore describes how
%the QUICKER MDE tool helps evaluate possible choices of QoS configuration
%at design-time using model-checking, and shows how results of this
%design-time evaluation of possible choices can be used to select runtime
%QoS configurations by a dynamic QoS adaptation framework.

\smallskip
Due to the challenges described above, significant manual effort is
typically expended on QoS configurations for DRE systems. Even after 
much effort, it is common to identify
issues like frame overruns during integration testing, which results
in increasing the cost of development of DRE systems.  In some cases,
the problems are discovered after the system has been deployed.  It is
therefore critical that QoS configuration of component middleware is
performed via adequate tool support.  The remainder of this paper
shows how our QUICKER toolchain helps address these challenges in the
context of the CIAO QoS-enabled component middleware.
