%% $Id: eval.tex,v 1.4 2005/12/09 12:39:19 gokhale Exp $
\section{Model-driven Performance Analysis}
\label{sec:eval_perf}

Section~\ref{sec:mdd} described the modeling languages and the
generative tools we have developed to create performance models of the
composed systems and to synthesize the metadata artifacts required to
run the analytical model solvers, simulators and benchmarking tools.
In this section we illustrate how the SRN model of the reactor pattern
described in Section~\ref{sec:reactor_perf} could be applied to a case
study and used for its performance analysis.

\subsection{Case Study: Event-Driven Software Architecture in Mobile Handhelds}
\label{sec:handheld}

Figure~\ref{fig:handheld-arch} illustrates a typical software
architecture for event demultiplexing and dispatching in mobile
handhelds.  Handhelds, such as PDAs, have been at the forefront of
ubiquitous computing and are getting increasingly complex due to 
the need to support multiple applications, such as email, 
web browsing, calendar management, multimedia support, and games.  
Since these handhelds are used in many critical domains such as 
patient healthcare monitoring and emergency response systems they
must provide exceptional performance.  Further,
since the software residing in these handhelds is primarily
responsible for providing the desired functionality, it is necessary
to analyze the software applications for their performance.  
In particular, in this paper we 
describe a methodology for the performance evaluation of 
such event-driven software applications, which are commonly
used in ubiquitous computing. 

\begin{figure}[!h]
\center{\includegraphics[width=3.2in]{eps/handheld-demux.eps}}
\bfcaption{Event Demultiplexing in Handhelds}
\label{fig:handheld-arch}
\end{figure}

The application mix available in the handhelds is aptly suited for
event-driven systems, which requires demultiplexing and dispatching
events to the correct event handlers in the handheld.  For example,
a user of the handheld could have set appointments in his/\-her
calendar, which might raise an event while the user may be in the
midst of web browsing.  Similarly it is conceivable that email
notification arrives while the user is in the midst of other
applications like web browsing or listening to audio.  In the context
of handhelds that are tailored to provide service to emergency
response personnel, one could conceive of scenarios where sensor
data from a phenomenon of interest is received by the handheld with
other notifications, such as SMS notifications from the command and
control, received by the handheld.  

\subsection{Performance Evaluation of Event-handling Software
  Architecture in Handhelds}
\label{sec:handheld_eval}

This section illustrates how we used the model-generated artifacts for
performance evaluation of an application mix consisting of two
services, namely, calendar service and email service in handhelds.
The user of the handheld device expects that he/\-she be notified of
the incoming emails  and the calendar events in a reasonable amount of
time.  Further, the probability of denying the service requests should
be minimal.  From the point of view of the service provider designing
the system, the rate at which the service requests are processed or
the throughput is of interest, since this will determine the quality
of the product and ultimately the revenue generated for the provider.

In order to implement the mix of these two services, the 
reactor pattern with the characteristics described in 
Section~\ref{sec:reactor_modeling} can be used to demultiplex the
events.  The SRN model of the reactor pattern can thus be used for the
performance analysis of the services.  In order to use the
SRN model, we designate the email notification requests as events
of type \#1, and calendar update events as events of type \#2.

The following listing illustrates a snippet of the model generated
artifacts that drive the SPNP analytical model solver.  In this case
it is a C program used by the SPNP solver.  Similar artifacts can be
generated by our modeling tools that drive simulations and empirical
benchmarks. 

\begin{verbatim}
void net () {
/* Places */

place("B1");
place("B2");
place("S1");
place("S2");
place("StSnp"); init("StSnp",1);
place("SnpInProg");

/* Timed transitions */ 
trans("A1"); rateval("A1",lam1);
trans("A2"); rateval("A2",lam2);
trans("Sr1"); rateval("Sr1",mu1); 
trans("Sr2"); rateval("Sr2",mu2);

/* Immediate transitions */
imm("Sn1"); probval("Sn1",1.0); priority("Sn1",100);
guard("Sn1",sn1);
imm("Sn2"); probval("Sn2",1.0); priority("Sn2",100); 
guard("Sn2",sn2);
imm("TStSnp"); probval("TStSnp",1.0); priority("TStSnp",60);
guard("TStSnp",gd);
imm("TESnp"); probval("TESnp",1.000);
guard("TESnp",gd1); priority("TESnp",120);

\end{verbatim}

In the design stage, it is important to assess the impact of
configuration options and parameters on the performance metrics. 
One of the configuration options is the buffer space available 
to hold the incoming events of each type.  This choice will have 
a direct impact on all the performance metrics.  Most importantly, 
from the user's perspective, the buffer space will influence the 
probability of denying the service requests.

We analyze the impact of the buffer capacities on the 
performance measures.  The values of the remaining parameters (except
for the buffer capacities) are reported in Table~\ref{table:param}.
We consider two values of buffer capacities $N_1$ and $N_2$. In the
first case, the buffer capacity is set to $1$ for both types of
events, whereas in the second case the buffer capacity of both types
of events is set to $5$.  The performance metrics for both these cases
are summarized in Table~\ref{table:perf}.  Because the values of the
parameters of the service requests from organization \#1 ($\lambda_1$,
$\mu_1$ and $N_1$) are the same as the values of the parameters for
the service requests from organization \#2 ($\lambda_2$, $\mu_2$, and
$N_2$), the throughputs, queue lengths, and the loss probabilities are
the same for both types of service requests for each one of the buffer
capacities as indicated in Table~\ref{table:perf}.  It can be observed
that the loss probabilities are significantly higher when the buffer
capacity is $1$ compared to the case when the buffer space is
$5$.  Also, due to the higher loss probability, the throughput is
slightly lower when the buffer capacity is $1$ than when the buffer
capacity is $5$.

For both buffer capacities, the total number of service requests 
from organization \#2, denoted $E_2$, is slightly higher than the 
total number of requests from organization \#1, denoted $E_1$. Because
the requests from organization \#1 are provided prioritized service
over the requests from organization \#2, on an average it takes longer
to service a request from organization \#2 than it takes to service a
request from organization \#1.  This results in a higher total number
of requests from organization \#2 than the total number of requests
from organization \#1.  This may result in a higher response time for
service requests from organization \#2 as compared with the service
requests from organization \#1.  The reward rates to obtain the
response times can be obtained using the tagged customer
approach~\cite{melamed84} and is a topic of our future research. 

The performance metrics obtained from the SRN model were validated
by the metrics obtained from simulation using CSIM~\cite{csim}.  The
metrics obtained using simulation are also summarized in 
Table~\ref{table:perf}.  

\begin{table}[htb]
\begin{center}
\caption{Parameter values}
\label{table:param}
\begin{tabular}{|c|c|c|} \hline 
Parameter & Event \#1 & Event \#2 \\ \hline 
Arrival rate & $\lambda_1$ = $0.400$/sec.& $\lambda_2$ = $0.400$/sec. \\ \hline 
Service rate & $\mu_1$ = $2.000$/sec. & $\mu_2$ = $2.000$/sec. \\ \hline 
Failure rate & $\gamma_1$ = $0.02$/sec. & $\gamma_2$ = $0.02$/sec. \\ \hline 
\end{tabular}
\end{center}
\end{table}

\begin{table}[htb]
\begin{center}
\caption{Impact of buffer capacity on performance measures}
\label{table:perf}
\begin{tabular}{|c|c|c|c|c|} \hline 
Measure & \multicolumn{4}{|c|}{Buffer space} \\ 
                    &\multicolumn{2}{|c|}{$N_1=N_2=1$} &\multicolumn{2}{|c|}{$N_1=N_2=5$} \\ 
                    & SRN & Simulation & SRN & Simulation \\ \hline 
$T_1$ & $0.37$/sec. & $0.36$/sec. & $0.39$/sec. & $0.39$/sec.  \\ \hline 
$T_2$ & $0.37$/sec. & $0.38$/sec. & $0.39$/sec. & $0.40$/sec. \\ \hline 
$Q_1$ & $0.064$     & $0.060$ & $0.12$ & $0.12$ \\ \hline 
$Q_2$ & $0.064$     & $0.065$ & $0.12$ & $0.13$ \\ \hline 
$E_1$ & $0.26$      & $0.25$  & $0.33$ & $0.34$  \\ \hline 
$E_2$ & $0.26$      & $0.26$  & $0.33$ & $0.34$ \\ \hline 
$L_1$ & $0.064$     & $0.068$ & $0.00024$ & $0.00024$\\ \hline 
$L_2$ & $0.064$     & $0.0059$ & $0.00024$ & $0.00024$ \\ \hline
\end{tabular}
\end{center}
\end{table}

It is rarely the case that the values of the arrival and service rates
can be estimated with accuracy in the design phase. The SRN model can
also be used to analyze the sensitivity of the performance measures 
to the parameter values. The results of sensitivity analysis are not
included in this paper due to space limitations.  
