% $Id: benchmarking.tex,v 1.4 2006/02/12 09:16:39 akogekar Exp $
\subsection{Benchmarking}
\label{sec:benchmarking}
To enable the performance analysis of composed system, the modeling
language provides a method to model benchmarking characteristics
for the system. This can be done in the ``Benchmarking
Aspect''. Figure~\ref{fig:benchmark_meta} shows the meta-model for the
Benchmarking paradigm. 
\begin{figure}[htbp]
	\centering
		\includegraphics[width=12cm]{benchmark_meta}
	\caption{The Benchmarking Meta-Model}
	\label{fig:benchmark_meta}
\end{figure}

The meta-model enables the system developer to model the workload
(number of threads, data, and the number of data exchanges), as well
as which metric to measure (latency or throughput). It is also
possible to set the handle time for each event handler. A sample model
which can be constructed using this paradigm is shown in
Figure~\ref{fig:benchmark-model}. 
\begin{figure}[htbp]
	\centering
		\includegraphics[width=8cm]{benchmark_model}
	\caption{Benchmarking Aspect}
	\label{fig:benchmark-model}
\end{figure}

In this case the developer has modeled two patterns, the Reactor and
the Acceptor-Connector, and the benchmarking characteristics to
analyze the performance of the Reactor pattern. Therefore the latency
and throughput metrics are attached to the Reactor pattern. In
addition, the developer has modeled the number of client threads and
the service time for each event handler in the Reactor Pattern.  

\subsubsection{The Interpreter for the Benchmarking Aspect}
The benchmarking aspect enables a user to model the benchmarking
characteristics of the system. Using the benchmarking interpreter, the
developer can generate benchmarking parameters for an existing
benchmarking library. These parameters can be the number of data
exchanges, the number of client threads, the data to be sent, the
number of event handlers and the service time (in case of
reactor). They are generated in XML format and can be used by an
existing library. 

\begin{verbatim}
- <benchmark_inputs>
    <connections>10</connections> 
    <data>ABCDEF</data> 
    <data_exchanges>200</data_exchanges> 
  - <reactor_inputs>
      <reactor_type>wfmo</reactor_type> 
      <handlers>2</handlers> 
      <service_time>Uniform</service_time> 
    </reactor_inputs>
  </benchmark_inputs>
\end{verbatim}

Our future work in this regard lies in auto-generating benchmarking code, instead of just generating benchmarking parameters for existing benchmarking libraries. Another approach which we are pursuing is the auto-generation of simulation models of the patterns using POSAML. This would be helpful in the design-time analysis of middleware systems.