Empirical Evaluation of CORBA Component Model Implementations
            for Distributed Real-time and Embedded Systems

               Arvind S. Krishna and Douglas C. Schmidt
                {arvindk, schmidt}@dre.vanderbilt.edu

              Institute for Software Integrated Systems,
             Vanderbilt University, Nashville, Tennessee

Commercial off-the-shelf (COTS) middleware is now widely used to
develop distributed real-time and embedded (DRE) systems.  DRE systems
are themselves increasingly combined to form ``systems of systems''
that have diverse quality of service (QoS) requirements.  Earlier
generations of COTS middleware, such as Object Request Brokers (ORBs)
based on the CORBA 2.x standard, do not facilitate the separation of
QoS policies from application functionality, which makes it hard to
configure and validate complex DRE applications.  The new generation
of component middleware, such as the CORBA Component Model (CCM) based
on the CORBA 3.x standard, addresses the limitations of earlier
generation middleware by establishing standards for implementing,
packaging, assembling, and deploying component implementations.

There has been little systematic empirical study of the performance
characteristics of CCM middleware implementations in the context of
DRE systems.  As CCM platforms mature and become suitable for DRE
systems it is therefore desirable to devise a standard set of metrics
to compare and contrast different CCM implementations in terms of
their:

. Suitability, e.g., how suitable is the CCM implementation for DRE
  applications in a particular domain, such as avionics mission
  computing, total ship computing environments, or telecom systems.

. Quality of service, e.g., how well do CCM implementations provide
  predictable performance and consume minimal time/space resources
  to support applications in the DRE domain

. Conformance, e.g., to what extent does a CCM implementation conform
  to OMG standards by meeting the portability and interoperability
  requirements defined by the CCM specification.

This presentation will focus on the following topics pertaining to the
use of CCM for DRE systems:

1. The challenges involved in benchmarking different CCM
   implementations

2. Criteria for comparing different CCM implementations using key
   black-box and white-box metrics and

3. The design of CCMPerf, which is an open-source benchmarking toolkit
   that can be used to evaluate aspects of CCM implementations to
   determine their suitability for the DRE domain.

The presentation will also present results of benchmarking experiments
on various CCM implementations, including CIAO, Mico-CCM, Qedo, and
Star CCM.  For each of the ORBs the following categories of metrics
will be presented:

. Distribution middleware tests that quantify the overhead of
  CCM-based applications relative to applications based on earlier
  versions of the CORBA 2.x standard that do not support component
  run-time, configuration, and deployment capabilities.

. Common middleware services tests that quantify the suitability of
  using different implementations of CORBA services, such as Real-time
  Event and Notification Services.

. Domain-specific middleware tests that quantify the suitability of
  CCM implementations to meet the QoS requirements of a particular DRE
  application domain, such as static linking and deployment of
  components in avionics mission computing and total ship computing
  environments.