Configuring TAO's Components

Overview

As described in the options documentation, various components in TAO can be customized by specifying options for those components. This document illustrates how to combine these options to affect the behavior and performance of a TAO ORB, particularly its concurrency model.

TAO configures itself using the ACE Service Configurator framework. Options are therefore specified using the svc.conf file (if you want to use a different file name, use the -ORBSvcConf option). Configurations selected via a svc.conf file are global per-process, e.g., all ORBs in a process will share the same concurrency model configured via a svc.conf file. You can also setup default configurations for your programs. Please see the Programming Considerations for more detailed discussion on this.

Roadmap

Details for the following configurations are provided.

Configurating key components:

Server Concurrency Strategy.

Number of ORBs.

POA.

Collocation Table.

Forwarding Profile

orbsvcs Library

Configuration examples

Single-threaded, reactive model.

Multiple threads, single ORB, thread-per-connection model.

Multiple threads, single ORB, TAO thread-pool model.

Multiple threads, single ORB, Real-time CORBA thread-pool model.

Multiple threads, multiple ORBs, reactive model.

Multiple threads, multiple ORBs, thread-per-connection model.

Multiple threads, ORB-per-thread, thread-pool reactive model.

Each configuration has the following information:

Typical Use A brief description of the scenario and its typical use.

Number of Threads The number of threads used by ORB-related activities.

Thread Creator Identifies the creator of the threads discussed above.

Thread task Describes what task is undertaken for each thread.

Options Specifies the options for each service in order to utilize this configuration.

Programming Considerations
Configuration for Homogenous Systems

Compile-time options

Runtime-time

Configuration Suggestions

Typical Use	A brief description of the scenario and its typical use.
Number of Threads	The number of threads used by ORB-related activities.
Thread Creator	Identifies the creator of the threads discussed above.
Thread task	Describes what task is undertaken for each thread.
Options	Specifies the options for each service in order to utilize this configuration.

Configuring Key ORB Components

Number of ORBs --

e.g.

Server concurrency strategy -- The default server strategy factory provided by TAO supports two types of concurrency strategies that can be specified by adding the -ORBConcurrency flag in the Server_Strategy_Factory entry of the svc.conf file. This specifies the concurrency strategy an ORB uses. This strategy is orthogonal to the number of ORBs or threads that are configured in a server process.

reactive: The ORB handles requests reactively, i.e., the ORB runs in one thread and service multiple requests/connections simultaneously using the ACE_Reactor, which uses select or a similar event demultiplexing mechanism supported by the platform.

thread-per-connection: The ORB handles new connections by spawning a new thread whose job is to service requests coming from the connection.

Thread Pools

reactive: In this approach, each thread in the thread pool has an ORB that accepts and processes requests reactively. This is also called as the "ORB-per-thread" model
leader/followers: In this model, the user must create several threads, all of which invoke ORB::run, the ORB will select one of the threads to wait for incoming requests. This thread is called the leader thread and will process the first request that arrives to the ORB, but before doing so the ORB will selects another thread in the pool to become the leader. In other words the threads in the pool take turns to process the events.

Collocation Table -- An ORB can have several listening endpoints. If there are several ORBs in a process and a global collocation table is used, then all objects in the same process are considered collocated. If not, only objects reside in the same ORB are considered collocated. You can control the usage of global collocation table by passing the -ORBCollocation flag as an argument of ORB_init (most often thru the command line flags.)
orbsvcs Library -- By default, TAO's Makefiles build all the services that TAO supports. To reduce build time you can exclude unused services, by defining the TAO_ORBSVCS makefile variable. We recommend using one of these two approaches:

In your $(ACE_ROOT)/include/makeinclude/platform_macros.GNU file,
On the make command line, e.g., make TAO_ORBSVCS=RTEvent, or
Set (and export) a TAO_ORBSVCS environment variable.

Please see the ORBSVCS Makefile for the default setting of TAO_ORBSVCS.

Please note that the Naming Service will always be built, even if Naming is not specified in TAO_ORBSVCS. That's because many examples, tests, and presumably applications use it.

Configuration Examples

The following are common ORB configurations used by TAO applications.

Single-threaded, reactive model.

Typical Use This is the default configuration of TAO, where one thread handles requests from multiple clients via a one Reactor. It is appropriate when the requests (1) take a fixed, relatively uniform amount of time and (2) are largely compute bound.

Number of Threads 1

Thread Creator OS or whoever creates the main ORB thread in a process.

Thread task The single thread processes all connection requests and CORBA messages.

Synchronization considerations Application servants needn't be concerned with synchronizing their interactions since there's only one thread active with this model.

Options The default settings should work just fine (by default, the -ORBReactorType is tp). However, you can apply the following options to improve performance:
TAO_Advanced_Resource_Factory: -ORBReactorType select_st, -ORBInputCDRAllocator null
TAO_Server_Strategy_Factory: -ORBconcurrency reactive (default),
TAO_Client_Strategy_Factory: -ORBConnectorLock null

Typical Use	This is the default configuration of TAO, where one thread handles requests from multiple clients via a one Reactor. It is appropriate when the requests (1) take a fixed, relatively uniform amount of time and (2) are largely compute bound.
Number of Threads	1
Thread Creator	OS or whoever creates the main ORB thread in a process.
Thread task	The single thread processes all connection requests and CORBA messages.
Synchronization considerations	Application servants needn't be concerned with synchronizing their interactions since there's only one thread active with this model.
Options	The default settings should work just fine (by default, the `-ORBReactorType` is `tp`). However, you can apply the following options to improve performance: `TAO_Advanced_Resource_Factory`: `-ORBReactorType select_st`, `-ORBInputCDRAllocator null` `TAO_Server_Strategy_Factory`: `-ORBconcurrency reactive` (default), `TAO_Client_Strategy_Factory`: `-ORBConnectorLock null`

Check out the Grid for an example of a default configuration and this for an example of settings that would provide better performance.

Multiple threads, single ORB, thread-per-connection model.

Typical Use This configuration spawns a new thread to serve requests from a new connection. This approach works well when there are multiple connections active simultaneously and each request-per-connection may take a fair amount of time to execute.

Number of Threads 1 thread for the ORB, plus 1 thread for each connection.

Thread Creator Programmer must set up the main thread which the ORB lives. The ORB is responsible to create new threads upon new connections.

Thread task The main thread handles new connections and spawns new threads for them. Other threads handle requests for established connections.

Synchronization considerations To avoid race conditions, application servants may need to synchronize their methods if multiple clients can access them concurrently.

Options TAO_Server_Strategy_Factory: -ORBConcurrency thread-per-connection

Typical Use	This configuration spawns a new thread to serve requests from a new connection. This approach works well when there are multiple connections active simultaneously and each request-per-connection may take a fair amount of time to execute.
Number of Threads	1 thread for the ORB, plus 1 thread for each connection.
Thread Creator	Programmer must set up the main thread which the ORB lives. The ORB is responsible to create new threads upon new connections.
Thread task	The main thread handles new connections and spawns new threads for them. Other threads handle requests for established connections.
Synchronization considerations	To avoid race conditions, application servants may need to synchronize their methods if multiple clients can access them concurrently.
Options	`TAO_Server_Strategy_Factory`: `-ORBConcurrency thread-per-connection`

IDL_Cubit is a good example on using multiple threads, thread-per-connection configuration.

Multiple threads, single ORB, TAO thread-pool model.

Typical Use This model implements a highly optimized, TAO-specific thread pool that minimizes context switching and thread creation costs. In this model, the programmer is responsible of spawning a group of threads, creating an ORB instance, and then instructing all the threads to run the ORB event loop. When a request comes in, one of these waiting threads in the pool will handle the request.

Number of Threads Thread for the ORB, plus the number of threads used by the thread pool.

Thread Creator Pre-spawned by the main thread.

Thread task Blocking on the reactor to wait for its turn to handle a request.

Synchronization considerations To avoid race conditions, application servants may need to synchronize their methods if multiple clients or multiple thread from the same client can access them concurrently.

Options The default ORB settings support this concurrency configuration, though you'll need to spawn the extra threads in the pool explicitly.

Typical Use	This model implements a highly optimized, TAO-specific thread pool that minimizes context switching and thread creation costs. In this model, the programmer is responsible of spawning a group of threads, creating an ORB instance, and then instructing all the threads to run the ORB event loop. When a request comes in, one of these waiting threads in the pool will handle the request.
Number of Threads	Thread for the ORB, plus the number of threads used by the thread pool.
Thread Creator	Pre-spawned by the main thread.
Thread task	Blocking on the reactor to wait for its turn to handle a request.
Synchronization considerations	To avoid race conditions, application servants may need to synchronize their methods if multiple clients or multiple thread from the same client can access them concurrently.
Options	The default ORB settings support this concurrency configuration, though you'll need to spawn the extra threads in the pool explicitly.

MT_Server is a good example on using multiple threads, single ORB, TAO thread-pool configuration.

Multiple threads, single ORB, Real-time CORBA thread-pool model.

Typical Use This model implements a highly optimized Real-time CORBA thread pool that minimizes context switching, and thread creation costs. In this model, the application is responsible for starting up the ORB and then calling the Real-time CORBA thread pool factory methods that internally spawn a pool of threads. When a request comes in, one of these waiting threads in the pool will handle the request in accordance with the various CORBA policies.

Number of Threads Thread for the ORB, plus the number of threads used by the thread pool.

Thread Creator Static threads are pre-spawned by the ORB, which also spawns any dynamic threads on-demand.

Thread task Blocking on the reactor to wait for its turn to handle a request.

Synchronization considerations To avoid race conditions, application servants may need to synchronize their methods if multiple clients or multiple thread from the same client can access them concurrently.

Options The ORB must be configured to work with Real-time CORBA, at which point the default settings work fine.

Typical Use	This model implements a highly optimized Real-time CORBA thread pool that minimizes context switching, and thread creation costs. In this model, the application is responsible for starting up the ORB and then calling the Real-time CORBA thread pool factory methods that internally spawn a pool of threads. When a request comes in, one of these waiting threads in the pool will handle the request in accordance with the various CORBA policies.
Number of Threads	Thread for the ORB, plus the number of threads used by the thread pool.
Thread Creator	Static threads are pre-spawned by the ORB, which also spawns any dynamic threads on-demand.
Thread task	Blocking on the reactor to wait for its turn to handle a request.
Synchronization considerations	To avoid race conditions, application servants may need to synchronize their methods if multiple clients or multiple thread from the same client can access them concurrently.
Options	The ORB must be configured to work with Real-time CORBA, at which point the default settings work fine.

Thread_Pool is a good example on using multiple threads, single ORB, Real-time CORBA thread-pool configuration.

Multiple threads, multiple ORBs, reactive model.

Typical Use In this configuration, there are multiple ORBs in a process with multiple threads. Each thread handles requests reactively. This model is good for hard real-time applications that require different thread priorities for the various ORBs.

Number of Threads One thread for each ORB.

Thread Creator The main process (thread).

Thread task Service the requests from associating ORB.

Synchronization considerations Application servants needn't be concerned with synchronizing their interactions if there's no cross ORB/thread access.

Options The default settings just works fine. However, one could use
TAO_Advanced_Resource_Factory: -ORBReactorType "which", for a thread-safe platform-specific reactor.

Multiple threads, multiple ORBs, thread-per-connection model.

Typical Use	In this configuration, there are multiple ORBs in a process with multiple threads. Each thread handles requests reactively. This model is good for hard real-time applications that require different thread priorities for the various ORBs.
Number of Threads	One thread for each ORB.
Thread Creator	The main process (thread).
Thread task	Service the requests from associating ORB.
Synchronization considerations	Application servants needn't be concerned with synchronizing their interactions if there's no cross ORB/thread access.
Options	The default settings just works fine. However, one could use `TAO_Advanced_Resource_Factory`: `-ORBReactorType "which"`, for a thread-safe platform-specific reactor.

Typical Use This approach provides a range of thread priorities plus connections that don't interfere with each others.

Number of Threads One thread for each ORB, plus one thread for each connection.

Thread Creator Main threads creates threads running ORBs. They, in turns, create connection handling threads.

Thread task There are threads running ORB's event loops which handle connection requests and handler threads which service requests from established connections.

Synchronization considerations To avoid race conditions, application servants may need to synchronize their methods if multiple clients can access them concurrently.

Options TAO_Server_Strategy_Factory: -ORBConcurrency thread-per-connection

Typical Use	This approach provides a range of thread priorities plus connections that don't interfere with each others.
Number of Threads	One thread for each ORB, plus one thread for each connection.
Thread Creator	Main threads creates threads running ORBs. They, in turns, create connection handling threads.
Thread task	There are threads running ORB's event loops which handle connection requests and handler threads which service requests from established connections.
Synchronization considerations	To avoid race conditions, application servants may need to synchronize their methods if multiple clients can access them concurrently.
Options	`TAO_Server_Strategy_Factory`: `-ORBConcurrency thread-per-connection`

MT_Cubit is a good example on using multiple threads, multiple ORBs, and thread-per-connection configuration.

Multiple threads, multiple ORBs, thread-pool model.

Typical Use This model incorporates the advantage of using thread-pool while allowing hard real-time system to handle requests in different priority.

Number of Threads One thread for each ORB, plus the total number of threads in all thread pools

Thread Creator Pre-spawned by the main thread.

Thread task Handle incoming request for the ORB event loop it is waiting on.

Synchronization considerations Application servants needn't be concerned with synchronizing their interactions if there's no cross ORB/thread access.

Options The default settings work well for this

Typical Use	This model incorporates the advantage of using thread-pool while allowing hard real-time system to handle requests in different priority.
Number of Threads	One thread for each ORB, plus the total number of threads in all thread pools
Thread Creator	Pre-spawned by the main thread.
Thread task	Handle incoming request for the ORB event loop it is waiting on.
Synchronization considerations	Application servants needn't be concerned with synchronizing their interactions if there's no cross ORB/thread access.
Options	The default settings work well for this

Programming Considerations

There are several ways to pass option flags into TAO's components.

The plain vanilla approach is do nothing. All TAO components use their default settings.
The most common use case is to use a file called svc.conf. On most platforms, TAO programs automatically search and read in the file. The disadvantage of this approach is you always need a svc.conf file if you want to do use non-default configuration.
You can use -ORBSvcConf filename to use a config file that is not called svc.conf. Specifying -ORBSvcConf exclude the reading of default svc.conf file.
If you don't want the application users to worry about setting up or knowing about svc.conf files, you can call TAO::ORB::default_svc_conf_entries() before calling the first ORB_init() in your program to set up the default svc.conf entries. In this case, if a TAO application cannot find a svc.conf file, it will configure TAO's components using the default settings. You can still use a svc.conf file or use -ORBSvcConf option to tune the program.
TAO programs evaluate the configuration settings in the following order:
1. File specified in -ORBSvcConf command-line option, if one exist. Otherwise, the svc.conf in the start-up directory will be evaluated, if one exist.
2. Default entries set by TAO::ORB::default_svc_conf_entries(), if ones exist.
3. Default configuration as specified in this document.
Notice that the first encountered component settings are always the ones take effect. For example, if you set the entries for Resource_Factory and Server_Strategy_Factory using TAO::ORB::default_svc_conf_entries() in a program and you also have a file called svc.conf which has an entry for Resource_Factory. This program will use the entry for Resource_Factory in the svc.conf file, the entry for Server_Strategy_Factory set in the program, and the in-stock Client_Strategy_Factory that TAO defines.
Some platforms do not support reading of svc.conf files or perhaps you would rather not to use this feature. In these cases, you can define TAO_PLATFORM_SVC_CONF_FILE_NOTSUP in your ACE config.h file and recompile the TAO libraries. When this flag is set, TAO programs will not try to search for the default svc.conf file. You can still use -ORBSvcConf to initialize the components (assuming the platform supports it).

On these platform, you can alter the default settings for TAO components by defining the following macros in your config.h file:
- TAO_DEFAULT_RESOURCE_FACTORY_ARGS
- TAO_ADVANCED_RESOURCE_FACTORY_ARGS
- TAO_DEFAULT_SERVER_STRATEGY_FACTORY_ARGS
- TAO_DEFAULT_CLIENT_STRATEGY_FACTORY_ARGS
The ACE Makefiles fakesvcconf flag can be used to define TAO_PLATFORM_SVC_CONF_FILE_NOTSUP. To define that macro, just add fakesvcconf=1 to your make invocation.
See orbconf.h for an example.

Configuration for Homogenous Systems

Compile-time options
Many real-time applications run on homogenous environments, TAO (and ACE) can take advantage of this fact by simplifying the server side demarshaling; to enable this feature you have to edit the $ACE_ROOT/ace/OS.h file and enable the macro ACE_DISABLE_SWAP_ON_READ.
In this systems it is also common that server and the client startup and shutdown simultaneously, in those circumstances there is no need to check the timestamps in the POA, another macro (POA_NO_TIMESTAMP) can be used for this purpose.
Users running in embedded systems may also need to modify the default options for TAO, the macros TAO_DEFAULT_RESOURCE_FACTORY_ARGS, TAO_ADVANCED_RESOURCE_FACTORY_ARGS, TAO_DEFAULT_CLIENT_STRATEGY_FACTORY_ARGS and TAO_DEFAULT_SERVER_STRATEGY_FACTORY_ARGS can be used for those purposes. If the footprint size is an issue users may consider writing custom strategy factories that only create the right strategies, this eliminates the parsing code for the different options.
Run-time options
Unix systems that support local IPC (formerly known as Unix domain sockets) can take advantage of TAO's UIOP pluggable transport protocol to improve performance considerably. To use TAO's UIOP pluggable protocol, simply specify a UIOP endpoint on the command line using the -ORBEndpoint option described in the options documentation. Additional information about TAO's UIOP pluggable protocol can be found in the release notes.

Some embedded systems run without the benefit of a DNS server, in that case they can use the -ORBDottedDecimalAddresses option; the ORB will avoid the use of hostnames in the profiles it generates, thus clients don't need to do any name resolution. Use the compile-time define TAO_USES_DOTTED_DECIMAL_ADDRESSES in $TAO_ROOT/tao/orbconf.h to make this the default behavior.

Configuration Suggestions

Choosing the right configuration is hard and, of course, depends on your application. In the following section we will attempt to describe some motivations for features in TAO, hopefully that can guide you through the choice of your configuration options.

ORB-per-thread -- The main motivation behind this options is to minimize priority inversion, since threads share no ORB resources no locking is required and thus, priority is preserved in most cases (assuming proper support from the OS). If you are not too concerned about priority inversion try to use a single ORB, using ORB-per-thread has some tradeoffs (like calling ORB_init on each thread, activation of a servant is more complicated, etc.) Some of the problems, can be minimized, but they require even more careful analysis.
As the reader will note this is a delicate configuration option, the rule of thumb should be not to use ORB-per-thread unless it is really required.
Collocation tables -- Why would an application not want to use the global collocation table? Because a collocated method invocation is run in the client's thread-of-control. If objects are to serve requests only at a well known priority the application can be configured with the ORB-per-thread option, and the object is activated only in the thread (ORB) corresponding to the desired priority. But using a global table would subvert the priority assignment (because calls would run at the priority of the client).

Single-threaded vs. Multi-threaded Connection Handlers -- The Client_Connection_Handler is the component in TAO that writes the requests to the underlying transport socket; this is also the component that reads the response back from the server.

While waiting for this response new requests to the local ORB can arrive, this is the so-called nested upcall support. TAO supports two mechanisms for handling nested upcalls, the default uses the leader-follower model to allow multiple threads to wait on a single reactor for several concurrent requests; sometimes this configuration can be an overkill, if only one thread is using a reactor at the same time a lighter weight implementation can be used.

This configuration is controlled by the -ORBClientConnectionHandler option, good opportunities to use this option are:

Single threaded servers
Servers running in ORB-per-thread mode (pseudo single threaded.)
Pure clients that will never receive a request

Allocator for input CDR streams -- Normally the application has no access to this buffer, and it is only used on the demarshaling of arguments (or results). It is almost always better to use the "-ORBInputCDRAllocator null" option since it will allocate memory from a thread specific allocator and it will not need locks to manage that memory.

In some cases the user may gain access to the CDR stream buffer: TAO makes no copies when demarshaling octet sequences, instead the octet sequence simply points to the CDR buffer, since the octet sequence does not own this buffer a copy must be made if the user wants to keep the buffer after the upcall.

The user can, however, increase the reference count on the CDR stream buffer, thus allowing her to extend the lifetime of this buffer. Still passing this buffer to another thread and attempting to release it in that thread will result in some memory leak or corruption. Users willing to use this feature of TAO can still do so, if they use a global allocator for their input CDR stream, but that will introduce extra locking on the critical path.

As the reader can see this is an option that has limited applicability and requires careful consideration of the tradeoffs involved.

Back to the TAO component options and tuning documentation.