%$Id: evaluation.tex 16757 2009-02-09 05:03:03Z gokhale $

\sqzsmall
\sqzsmall
\section{Evaluating the Efforts Saved using MTS}
\label{sec:eval}
\sqzsmall
Because MTS was developed to enhance reusability, this section describes its
merits in terms of the reduction in effort to write the transformation
rules.  
%Second, since MTS provides higher order transformations, we also
%discuss the overhead incurred by the higher order transformations.
%
%
%\begin{table*}[htb]
%\centering
%\bfcaption{The size of the metamodel for the two case studies.}
%\label{tab:meta-size}
%\scriptsize
%\begin{tabulary}{\columnwidth}{|C|C|C|C|C|}
%  \hline
%  \textbf{Metamodel} & \textbf{\# of } & \textbf{\# of } & \textbf{\# of } \\  
%  & \textbf{modeling elements} & \textbf{attributes} & \textbf{connections} \\  
%  \hline \hline
%  Insurance &  &  &  \\
%  Enterprise &  &  &  \\
%  \quad\textit{SRC/TRGT} & 8 & 14 & 0 \\
%  \hline
%  QoS &  &  & \\
%  Configuration &  &  & \\
%  \quad\textit{SRC} & 3 & 2 & 2 \\
%  \quad\textit{TRGT} & 8 & 14 & 4 \\
%  \hline  
%\end{tabulary}
%\end{table*}
%\begin{table*}[htb]
%\centering
%\bfcaption{Distribution of variabilities for the two case studies.}
%\label{tab:perf-table}
%\scriptsize
%\begin{tabulary}{\columnwidth}{|C|C|C|C|C|C|}
%  \hline
%  \textbf{Data Point} & \multicolumn {2} {|c|} {\underline{\textbf{Insurance Enterprise}} } & \multicolumn {2} {|c|} {\underline{\textbf{QoS Configuration}} } \\      
%   & \textbf{Qualitative} & \textbf{Structural} & \textbf{Qualitative} & \textbf{Structural} \\  
%  \hline \hline
%  1 & 2 & 0 & 2 & 0 \\
%  \hline
%  2 & 2 & 2 & 4 & 0 \\
%  \hline
%  3 & 3 & 4 & 5 & 0 \\
%  \hline
%  4 & 3 & 6 & 5 & 2 \\
%  \hline
%  5 & 5 & 6 & 6 & 3 \\
%  \hline
%  6 & 6 & 7 & 8 & 3 \\
%  \hline
%  7 & 6 & 9 & n.a. & n.a.  \\
%  \hline  
%\end{tabulary}
%\end{table*}
%
The prototype implementation of MTS is part of the 
CoSMIC\footnote{CoSMIC is a MDE toolsuite used in the design and
  deployment of applications for QoS-enabled middleware. It is available
  from http://www.dre.vanderbilt.edu/cosmic/.} tool suite. All of our
experiments are based on CoSMIC version 0.5.7, with GME version 6.11.9
and GReAT version 1.6.0.    
% All the overhead 
% measurement experiments were run on a Windows XP SP2 workstation with 
% 2.66 GHz Intel Xeon dual processor and 2 GB physical memory.

% \sqzsmall
% \sqzsmall
% \subsection{Reduction in Development Effort using MTS}
% \label{sec:effort}
% \sqzsmall

% Amogh, this may be repetitive since we covered it earlier so I will
% look into trimming it.
Recall from Section~\ref{sec:soln} that to create a target model from
a source model using GReAT, developers need to execute the GR-engine that
executes all the translation rules of that model transformation. More
specifically, developers must first specify all the rules that transform
the elements of the source model to the target model.  Thereafter, the
GR-engine execution involves the following steps:  (1) executing the
\emph{master interpreter} that generates the necessary intermediate
files containing all the rules in the current transformation, (2)
compile these intermediate files, if not done already, and  (3) run the
generated executable. Steps 1 and 2 must be executed each time the model
transformation rules are modified -- which is the case with variants of
a family.

\iffalse
\begin{wraptable}[8]{R}{0.5\textwidth}
%\begin{table}[htb]
\vspace{-0.4in}
\centering
\bfcaption{Time taken in executing the two model transformations in GReAT.}
\label{tab:gr-tc}
%\vspace{-0.1in}
\scriptsize
\begin{tabulary}{\columnwidth}{|C|C|C|C|C|}
  \hline
  \textbf{Case} & \textbf{Master} & \multicolumn {2} {|c|} {$\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;$\underline{\textbf{GR-engine}}} \\  
  \textbf{Study} & \textbf{Interpreter} & \textbf{Compile \& execute} & \textbf{Execute} \\  
  \hline \hline
  Insurance &  &  & \\
  Enterprise & 16 sec. & 64 sec. & 8 sec.\\  
  \hline
  QoS &  & & \\
  Configuration & 16 sec. & 104 sec. & 12 sec.\\  
  \hline  
\end{tabulary}
%\end{table}
\end{wraptable}
\fi

\iffalse
Obviously, without the use of MTS, Steps 1 and 2 have to be re-executed
each time a new type of family instance (\emph{e.g.}, addition of a new
communication endpoint in insurance enterprise case study) has to be
supported by the model transformations.  Additionally, even for a single
family instance, modifying a particular mapping  (\emph{e.g.}, changing
the values of \texttt{Lane} attributes, for a particular
\texttt{mu\-lt\-i\_se\-rv\-ic\-e\_le\-ve\-ls} value in QoS configuration
case study), requires the re-execution of the first two steps above. 

Using MTS, on the other hand, the first two steps have to be executed
only once when the model transformation is being executed for the first
time. Since all the  instance-specific customizations/changes are done
in the corresponding VMM model, the developers only need to execute Step
3 after each change to produce output of the  transformation
(\emph{i.e.}, a new family instance). %For our two case studies, as shown
%in Table~\ref{tab:gr-tc}, using MTS leads to savings of upto 90\% in the
%time taken for a single transformation run, over non-templatized
%approach. 
\fi

Without MTS, model transformations for each variant of an application
family must expend effort in all of the above steps.  Our goal is to
evaluate MTS in terms of effort saved.  We focus on two
specific cases discussed below.

\noindent\textbf{Case 1: Newly added variant is subsumed by existing
  constraint specifications:} 
The existing constraint specification for the application family may be 
sufficient to capture all the variabilities of a new family
variant. Thus, the developers can create a new variant simply by
re-executing the same model transformation with the VMM model of the
variant as one of the inputs to the transformation.  Note that the first
two steps have to be performed only once when the model transformation
is being executed for the first time. Because all the instance-specific
customizations/changes are done in the corresponding VMM model,
developers only need to execute Step 3 after each change to produce
output of the  transformation (\emph{i.e.}, a new family
instance).  

In contrast, the traditional approach of one model transformation per
single (subset of) family instance(s) will require maintenance of
$I*R_n$ rules, where  $I$ is the number of family instances, and $R_n$
is the average number of rules per instance.  With MTS, assuming that
the average number of rules do not change, the total number of rules to
be maintained reduces by a fraction of $\frac{I - 1}{I}$.

\noindent\textbf{Case 2: New variant requiring additional constraint
  specifications:} 
If the variabilities of a new family variant are not completely captured
using existing constraint specification for the application family, MTS
requires enhancements to the constraint specification itself.  Such a
change necessitates executing the first two steps above once to produce
a new VMM which can be used to model variabilities in the new
variant.  Note that despite this change, the VMM models corresponding to
the existing variants will still be valid provided the changes in
constraint specification (because of a new family variant) are
orthogonal to the existing variabilities. 

\iffalse
\sqzsmall
\sqzsmall
\subsection{Performance Overhead of using MTS}
\label{sec:perf}
\sqzsmall
The rationale behind these experiments is to quantify the overhead
placed by the higher order transformations in Algorithms~\ref{algo:vmm}
and~\ref{algo:specialize} when the number of structural and quantitative
variabilities are increased.  The performance overhead was calculated in
terms of the time taken by each of these algorithms when used in the
context of each of the two case studies.  In all we identified (a
maximum of) fifteen variabilities for insurance enterprise case study,
and eleven variabilities for QoS configuration case study.  The  
performance overhead was measured by increasing the variabilities
in each case study from a minimum value of two to the maximum values
above.  The size of both the metamodels is given in
Table~\ref{tab:meta-size}.  Table~\ref{tab:perf-table} shows the
distribution of variabilities across the quantitative and structural
dimensions for these cases.

\begin{table}[htb]
\vspace{-0.4in}
\centering
\bfcaption{Details of the representative case studies.}
\label{tab:detailss}
\subfloat[The size of the metamodels.]
{
    \label{tab:meta-size}
\scriptsize
\begin{tabulary}{\columnwidth}{|C|C|C|C|C|}
  \hline
  \textbf{Metamodel} & \textbf{\# of } & \textbf{\# of } & \textbf{\# of } \\  
  & \textbf{modeling elmts.} & \textbf{attribs.} & \textbf{conns.} \\  
  \hline \hline
  Insurance &  &  &  \\
  Enterprise &  &  &  \\
  \quad\textit{SRC/TRGT} & 8 & 14 & 0 \\
  \hline
  QoS &  &  & \\
  Configuration &  &  & \\
  \quad\textit{SRC} & 3 & 2 & 2 \\
  \quad\textit{TRGT} & 8 & 14 & 4 \\
  \hline  
\end{tabulary}
}
\subfloat[Distribution of variabilities.]
{
    \label{tab:perf-table}
\scriptsize
\begin{tabulary}{\columnwidth}{|C|C|C|C|C|C|}
  \hline
  \textbf{Data Point} & \multicolumn {2} {|c|} {\underline{\textbf{Insurance Enterprise}} } & \multicolumn {2} {|c|} {\underline{\textbf{QoS Configuration}} } \\      
   & \textbf{Quantitative} & \textbf{Structural} & \textbf{Quantitative} & \textbf{Structural} \\  
  \hline \hline
  1 & 2 & 0 & 2 & 0 \\
  \hline
  2 & 2 & 2 & 4 & 0 \\
  \hline
  3 & 3 & 4 & 5 & 0 \\
  \hline
  4 & 3 & 6 & 5 & 2 \\
  \hline
  5 & 5 & 6 & 6 & 3 \\
  \hline
  6 & 6 & 7 & 8 & 3 \\
  \hline
  7 & 6 & 9 & n.a. & n.a.  \\
  \hline  
\end{tabulary}
}
%\vspace{-0.2in}
\end{table}

Figure~\ref{fig:overhead} shows the overhead involved in using MTS to
generate VMM (Step 1), and specialize the transformation (Step 4). In
general, the algorithms take slightly more time for QoS configuration
than the insurance enterprise, for the same number of variabilities,
which is attributed to the larger size of the combined size of the
source and target metamodels of the former. %  Algorithm~\ref{algo:vmm}
%was observed to be generally more sensitive to the variations in
%structural variabilities, when the quantitative variabilities were kept
%constant (\emph{e.g.}, data points 1-2, 3-4, and 6-7 in
%Table~\ref{tab:perf-table}).  
% Amogh, give some intuitive reasoning why?
%
% Andy, I have commented this statmt. out as I don't see a direct relationship
% between the rules of the algorithm with this observation. On its own, the
% statement seems to create confusion as opposed to giving a useful observation
% and so I think its best to avoid it.

\begin{figure}[htbp]
\vspace{-0.2in}
\centering
\subfloat[Insurance enterprise case study]
  {
    \includegraphics[width=2.2in,keepaspectratio]{insu-enterprise}%
    \label{fig:insu-ent}
  }
\subfloat[QoS configuration case study]
  {
    \includegraphics[width=2.2in,keepaspectratio]{qos-config}%
    \label{fig:qos-config}
  }
  \vspace{-0.1in}
\bfcaption{Overhead in using MTS for the development of templatized transformations. The Y axis denotes the time taken
by Algorithms~\ref{algo:vmm} and ~\ref{algo:specialize}.}
\label{fig:overhead}
\vspace{-0.2in}
\end{figure}

\iffalse
% Amogh, let us still keep this commented
%
% Andy, OK.
For a variation of \{Q=4, S=9\} in insurance enterprise case study where
Q and S denote the total variation in quantitative and structural
variabilities, respectively, the time complexity of
Algorithm~\ref{algo:vmm} increased by 350\% from an initial value of 6
seconds, while that of Algorithm~\ref{algo:specialize} increased by
380\% from an initial value of 5 seconds.  For QoS configuration case
study, with a total variation of \{Q=6, S=3\}, the increase was
$\sim$136\% and $\sim$300\%, for Algorithms~\ref{algo:vmm}
and~\ref{algo:specialize}, respectively.  
\fi

Thus, if the new family variant is already subsumed by the notation as discussed
in case 1 in Section~\ref{sec:effort}, the developers incur an additional 
overhead in using MTS only for the first time when each of these algorithms have to be 
applied (\emph{i.e.}, once for generating VMM, and once for creating temporary
objects in the model transformation).  Thus, the cost of using MTS is
amortized over the total number of transformation runs, during the
development cycle of that application family. For the remaining cases,
if the variabilities of the new variant are not captured by the existing 
specification, the two steps listed in Section~\ref{sec:effort} have to be
executed once after modifications in the specification have been made according
to the variabilities of the new variant. 
% Amogh, tie this back to the case 1 and 2 in the earlier results i.e.,
% tell the reader that if the variant is already subsumed by the
% notation then we don't pay any overhead except the first time. For the
% other case we do these steps but still the benefit is amortized. Can
% we also say a bit in terms of comparison with regular transformation
% overhead? 
%
% Andy, done.

\fi
