% $Id: results.tex,v 1.3 2006/04/11 03:57:24 gokhale Exp $

\section{Experimental Results}
\label{sec:results}

The premise of our experiments is that there is a large scale compute platform
running mission critical software.  This platform is divided into clusters of
computers (or data centers) that are geographically separated as shown
in Figure~\ref{fig:setup}.  The failure
scenarios we envision involve catastrophic loss of entire clusters at once.
Our goal is to survive the failure by keeping the mission-critical
functionality available and having extremely low down-time.  We will present a
brief overview of our mission-critical software.  Then we will describe the
experimental configuration and metrics used for evaluation. Finally, we will
present our results.

%  (note to Joe, Andy: the software I'm talking about here is MLRM.  The fact
% that MLRM itself has similar goals for keeping Application Strings running is
% something that I'm not sure we want to get into since making the distinction
% would be horribly confusing)

The mission critical software comprises several applications, each with
their own distinct characteristics.  Certainly one of the most interesting
aspects of these experiments is that several different fault-tolerance
techniques were needed to achieve the overall level of fault-tolerance that our
mission-critical software required.  We used a combination of Active
Replication, Warm-Passive Replication, and Database Replication
(MySQL-Clustering).  The first two techniques were implemented at the
middleware layer.  The last technique, Database Replication, was used
``as is'', but had to be integrated with the actively replicated
middleware elements. 

% Matt, I put this as a common figure for both experiments - Andy
\begin{figure}[!htb]
  \centering
  \vspace{-1in}
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{gt3experiment}%
  \vspace{-1in}
  \caption{Experiment Testbed and Setup}
  \label{fig:setup}
\end{figure}

\subsection { Scenario A}
For our first scenario, we considered a single cluster failure.
% (include picture of topology:
%
%
% https://repo.isis.vanderbilt.edu/twiki/pub/ARMS/AnalysisWithGraphs/gt3experimentconfig.png ) 
The scenario called for a whole cluster to ``disappear'' (from the perspective
of the mission-critical software's user) instantaneously.  The nature of the
failure could be a major power failure or destruction of hosting facility.  We
simulated this failure by creating a network partition in such a way that
packets sent to hosts in the failed cluster would be dropped at a router.  We
believe that this is an accurate simulation of the failure, and has the
advantage that we are able to determine the exact time the failure occurred (as
opposed to trying to obtain a timestamp for physically pulling a plug).

\vspace{-.1in}
\subsubsection {Metrics}
We used two metrics to evaluate the results.  The first metric was a boolean:
{\em does the mission-critical functionality survive (i.e. is it still
  usable)}?  The second metric related to the speed of recovery, with
the goal being {\em to recover from the failure in under a second}.  The
recovery times for the actively replicated and passively replicated
elements were derived from instrumentation inserted at the middleware
layer.  The recovery time for the Database replication was derived
from an external program that made constant (as fast as possible)
queries on the Databases.  When the Database failover occurred, there
is a slight increase in the time it takes to do the query, since the
Database essentially blocks until it determines that one of the
replicated instances is gone (and never coming back).
% Paul's comments
% ``slight'' seems too small. Perhaps noticeable. Usual is 8-15ms we'd see
% 300ms on failure (but also during normal runs if load got high so,...)


\vspace{-.1in}
\subsubsection {Results}
As illustrated in Figure~\ref{fig:result_a}, we were able to verify
that the mission-critical functionality survived and was usable after
the failure.  While both the active replication and database
replication parameters could potentially be tuned to produce even
better results, the results from our experiments show that we are
clearly {\em in the ballpark} with respect to our goal of less than 1
second recovery.  

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{3a.eps}%
  \caption{Experimental Results for Scenario A}
  \label{fig:result_a}
\end{figure}

\subsection {Scenario B}
Our second scenario introduced cascading failures.  The point of this scenario
was to show that the fault-tolerant solution was robust.  For this scenario we
used three clusters.  We induced failure on one cluster, and then before the
failover was complete, induced failure on another cluster.

% this is based on Paul's clarification - andy
Due to the fact that we could not inject accurately in every case we
analyzed the runs after the fact and took only those runs where the
failure had been injected at the right time. 

\vspace{-.1in}
\subsubsection {Metrics}
The metrics used were the same metrics used in Scenario A.  The only difference
is that the "recovery time" is based on the time of the {\it first} induced
failure.  

\vspace{-.1in}
\subsubsection {Results}
As with Scenario A's results, we were able to verify that the mission-critical
functionality survived.  The recovery times are slightly higher than the single
failure case.  This is because we used knowledge of our fault-tolerance
mechanisms to induce the second (cascading) fault at the worst possible time
with respect to the recovery from the first failure.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.75\columnwidth,keepaspectratio]{3b.eps}%
  \caption{Experimental Results for Scenario B}
  \label{fig:result_b}
\end{figure}

\smallskip\noindent
Our results indicate that our FT solution enables fast and bounded
recovery both in a single failure and cascaded failure scenarios.

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "middleware-2006"
%%% End: 
