% $Id: eval.tex,v 1.9 2006/01/04 19:35:52 dengg Exp $

%               Comments from the 3rd reviewer
%
% Many of the "evaluation" sections are rather light. The general
% message seems to be that more automation is a priori better, leading
% to evaluations such as (pp. 7-8): "...without DAnCE tool support it
% would be prohibitively hard to (re)install all 193 components on 26
% nodes in our ITS case study manually, while taking into
% consideration the heterogeneous software/hardware platforms, and
% variable component configuration, and process collocation
% strategies. In contrast, by using DAnCE in conjunction with PICML,
% the whole ITS component deployment process is automated and
% simplified for deployers." This strikes me as an overly biased
% towards the tool solution. However, the tool imposes costs of its
% own, such as taking time to learn, and costs some effort to use on
% its own. Meanwhile, if accomplishing this task manually is so
% prohibitively expensive, is it really the state of the practice for
% developers to undertake it in this way? Is there some other tool
% support that they would use? Or would they somehow define the
% problem differently so as to avoid this prohibitive bottleneck?

% Section 4: This is where I thought if we could aggregate all the
% evaluations from section 3. -andy
%%
%% Andy, I am not sure whether we should change our architecture or
%% not, since I always like using the "challenge ->
%% solution -> application to case study" format to organize paper. -Gan

\section{Evaluation}
\label{sec:eval}

Our motivation for integrating MDD tools with QoS-enabled component
middleware in the ITS case study was to (1) construct
software-intensive DRE systems using higher-level visual models that
facilitated more effective analysis, verification, and better
productivity than handcrafting software using third-generation
languages and (2) provide more reusable and composable abstractions
than using conventional object-oriented real-time middleware
platforms for DRE systems.  This section evaluates how well we
achieved these goals.

\subsection{Verifiability}
\label{sec:verify}

When developing large-scale DRE systems -- especially mission- or
safety-critical systems -- it is essential that system be verified
and validated against certain pre-defined domain rules before the
product is deployed.  As described in
Section~\ref{sec:its-config-concerns}, the WMGL DSML generated and
ensured the syntactic correctness of warehouse initialization code
that precisely define the physical layout of the warehouse,
characteristics of warehouse storages and facilities.  Moreover, the
embedded model checker of WMGL performed semantic analysis, such as
reachability analysis, of the warehouse configuration to ensure that
each warehouse storage cell could be reached by least one
transportation unit.  In addition, WMGL encapsulated domain
knowledge from warehouse domain experts in the form of additional
advanced warehouse model checkers.  Prior to this MDD tool support,
warehouse initialization and configuration scripts were handcrafted
manually, which was tedious and error-prone.

Prior work~\cite{Sharp:98,Sharp:03,Schmidt:04a} has shown that it is
hard for DRE developers to keep track of many complex dependencies
when configuring and deploying large-scale systems, even when using
component middleware.  Without MDD tool support, therefore, the effort
required to deploy large-scale DRE systems like ITS involves
hand-crafting deployment descriptor metadata in an {\em ad hoc}
manner.  Addressing this challenge effectively requires techniques
that can analyze, validate, and verify functional and QoS system
properties.

In our ITS case study, simple CORBA objects and more sophisticated CCM
components must coexist, which introduces complexities in interface
definition of components and interaction definitions between
components.  For example, while CORBA object interfaces can support
multiple inheritance, CCM components can only have only a single
component parent, so equivalent units of composition ({\em i.e.},
interfaces in CORBA objects and components in CCM) can have subtle
semantic differences.  Without automated support from MDD tools like
PICML, developers would have no systematic way to specify
interconnections and configurations.  Manually configuring the systems
via {\em ad hoc} techniques can overwhelm developers since many
configuration aspects are tangled with each other which spread
throughout different subsystems and layers of the ITS and component
middleware itself.  When using PICML, however, the problems associated
with multiple inheritance and semantics of tangled configuration can
be detected and resolved at design time.

\subsection{Reusability and Composability}
\label{sec:reuse}

Reusability and composability are important requirements,
particularly for developing large-scale DRE systems like ITS.
Diversified operating policies of ITS components can be customized
based on a particular warehouse deployment scenario.  In QoS-enabled
object-oriented middleware, such as Real-time CORBA and Real-time
Java, these configurations are made imperatively via invocations on
programmatic configuration interfaces. The drawbacks with
object-oriented approaches to reusability, however, are (1) impeded
reusability due to tight coupling of application logic with specific
QoS properties, such as event latency thresholds and
priorities~\cite{Schmidt:04e}, and (2) reduced composability due to
hard coded application interfaces, such as those integrated with
particular types of middleware services.

In contrast, component middleware, such as CIAO and DAnCE, enhances
reusability and composability by using metaprogramming techniques
(such as XML descriptor files) to specify component configuration
and deployment concerns declaratively.  This approach enables QoS
requirements to be specified later ({\em i.e}., just before runtime
deployment) in a system's lifecycle, rather than earlier ({\em
i.e.}, during component development).  When deploying an ITS product
instance, DAnCE parses the given XML configuration files and make
appropriate invocations on the corresponding service configuration
interfaces, which is useful for DRE systems like ITS that require
changeable customized QoS configurations for variable target OS,
network, and hardware platforms with different capabilities and
properties.

\begin{floatingfigure}[r]{0.55\columnwidth}
%\begin{wrapfigure}{r}{0.55\columnwidth}
  \centering
  \vspace{-0.8in}
  \includegraphics[width=0.53\columnwidth]{weight}
  \vspace{-0.5in}
  \caption{Weight of WorkflowManager Component in ITS}
  \label{fig:weight}
%\end{wrapfigure}
\end{floatingfigure}

\iffalse
\begin{figure}[!htb]
  \centering
%  \vspace{-0.5in}
% Gan, we need the figure for this one - andy
%  \includegraphics[width=0.7\columnwidth]{}
%  \vspace{-0.5in}
  \caption{Weight of WorkflowManager Component in ITS}
  \label{fig:weight}
\end{figure}
\fi

Figure~\ref{fig:weight} shows the weight distribution of a typical
ITS component WorkflowManager based on its functionality and
measured through {\em lines of code} (LOC) and the percentage of
code relative to the entire component lifecycle.  Other types of ITS
components have similar weight distributions.  As illustrated in
Figure~\ref{fig:weight}, the amount of code related to the
WorkflowManager component can be classified into five categories:
{\em Component Business Logic}, {\em Component Configuration}, {\em
Component Deployment}, {\em Object Stub \& Skeleton Glue code}, and
{\em XML descriptors}. Each category implements different aspect of
components during their lifecycles.

With object-oriented middleware, the {\em Object Stub and Skeleton
Code} is generated by the IDL compiler that shields application
developers from low-level network programming details.  This code
accounts for 58.9\% of the entire implementation.  The rest of the
code (more than 40\% of the implementation) must be hand written by
the application developers.  With the component-based approach,
conversely, most the code in the {\em Component Configuration} and
{\em Component Deployment} categories can be factored into reusable
CIAO component middleware infrastructure and then configured by MDD
tools like PICML and D\&C frameworks like DAnCE. For our ITS product
instance shown in Figure~\ref{fig:its-example}, where there are 8
components types and 193 component instances, more than $\sim$13,500
lines of code that were previously handcrafted are now refactored
into reusable component frameworks and configured and deployed using
PICML and DAnCE.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "isorc06-its"
%%% End:
