An earlier version of this article appeared in the C++ Report magazine, January 1999.
During the past decade, my colleagues and I have worked with hundreds of telecommunication, aerospace, and medical companies and written millions of lines of code while developing widely reusable middleware components and frameworks [Schmidt:01,Schmidt:97] for networked applications. We've also had the opportunity to document several dozen patterns [Schmidt:00] and architectures [POSA:96] that guide the design of these components and frameworks. In addition, we've taught hundreds of tutorials and courses on these topics for thousands of developers and students. In spite of formidable non-technical and technical challenges, we've identified a solid body of knowledge, experience, and software artifacts that can significantly enhance the systematic reuse of networked application software.
In this article, I outline common reasons why reuse has failed in the past. I then discuss proven steps that organizations, projects, and individuals can take to avoid these traps and pitfalls. The key themes pervading this article are:
Like many other promising techniques in the history of software, however, systematic reuse of software has not universally delivered significant improvements in quality and productivity. There have certainly been successes, e.g., sophisticated frameworks of reusable components are now available in OO languages running on many OS platforms. In general, however, these frameworks have focused on a relatively small number of domains, such as graphical user-interfaces or C++ container libraries like STL. Moreover, component reuse is often limited in practice to third-party libraries and tools, rather than being an integral part of an organization's software development processes.
In theory, organizations recognize the value of systematic reuse and reward internal reuse efforts. In practice, many factors conspire to make systematic software reuse hard, particularly in companies with a large installed base of legacy software and developers. In my experience, non-technical impediments to successful reuse commonly include the following:
I've also observed that developers often put too much faith in language features, such as inheritance, polymorphism, templates, and exception handling, as the primary means to foster reuse. Unfortunately, languages alone don't adequately capture the commonality and variability of abstractions and components required to build and systematically apply reusable software in complex application domains.
Over the past decade, I've worked with many companies, including Boeing, Cisco, Ericsson, Iridium, Kodak, Lucent, Motorola, SAIC, Siemens, and Sprint, building reusable networked applications using OO design techniques and programming languages [Schmidt:00,Schmidt:01]. These projects have applied a range of reusable middleware tools including CORBA, the ACE framework [Schmidt:01], which is a C++ framework that implements many patterns for concurrent networked applications, and TAO [Schmidt:97], which is a high-performance, real-time implementation of CORBA that leverages the framework components in ACE.
The following discussion summarizes the lessons my colleagues and I have learned making systematic reuse work with ACE and TAO, which have been deployed in a wide range of commercial applications in many domains, including aerospace, data- and tele-communications, medical, process automation, simulation, and financial services. ACE and TAO are freely available, open-source software packages that consist of frameworks, components, examples, and regression tests containing over one million lines of C++ code. More than 100 person-years of effort have been invested to develop, validate, optimize, and document these reusable assets.
Although there is no magic methodology or process that's guaranteed to foster systematic reuse, I have personally seen the recommendations below applied successfully numerous times over the past decade on many projects at many companies around the world.
The defense industry in the 1980's is a good example of the ``reinvent rather than reuse'' phenomenon. During the defense buildup in the Reagan administration, defense-related R&D funding was abundant. Not surprisingly, little systematic reuse was achieved, even though the Ada programming language was designed explicitly to support software reuse. Starting in the early 1990's, however, the substantial decrease in DoD funding had a powerful impact on the defense industry. Contemporary aerospace and defense contractors now aggressively seek to integrate "commercial-off-the-shelf" (COTS) software and hardware in order to remain competitive in their changing marketplace.
Ideally, an organization's software process should reward developers who invest the time and effort to build, document, and reuse robust and efficient components. For instance, a reward system could be built into project budgets, with incentives based on the number of software components reused by individuals or groups. I still find companies, however, whose processes measure programmer productivity solely in terms of the number of lines of source code written from scratch, which penalizes developers who attempt to reuse existing software.
In my experience, ``open-source'' development processes are an effective process for creating attractive reuse magnets. Open-source processes have yielded many widely used software tools and frameworks, such as Linux, Apache, GNU, ACE, and TAO. The open-source model allows users and developers to participate together in evolving software assets. One of the key strengths of this model is that it scales well to large user communities, where application developers and end-users can assist with much of the quality assurance, documentation, and support [Schmidt+Porter:01].
As a general rule, software development does not scale up as the number of developers increases. The main problems stem from the increased human communication and coordination costs associated with large projects. A team of 10 good developers can therefore typically produce much better quality software systems with much less effort and expense than a team of 1,000 developers.
In contrast, however, software debugging does scale up as the number of developers helping to debug the software increases. The main reason for this, of course, is that all other things being equal, having more people test the code will identify its ``error-legs'' much more quickly than having just a few people testing code. A team of 1000 testers will therefore typically find many more bugs than a team of 10 testers.
Moreover, open-source development efforts tend to have short feedback loops between the point when a bug is discovered and the bug is fixed. This increases the incentive for the user community to help with the debugging process since they are ``rewarded'' by rapid feedback and fixes once bugs are identified. In addition, because the source code is available for inspection, developers in the user community can often help fix any bugs they find, which further amortizes the overall debugging effort and improves software quality rapidly.
In general, the level of experience required to succeed with systematic reuse depends largely on whether programmers are trying to develop reusable components or to use them. I've found that developing reusable frameworks and components for complex domains, such as telecom or avionics, requires highly experienced and skilled architects and developers. These individuals must be trained and empowered to create, document, and support horizontal middleware platforms that reduce the effort required to develop vertical applications.
In general, the more complex the domain, the greater the skills and leadership required to develop effective reusable middleware that can encapsulate complex communication protocols and mechanisms for concurrency, locking, persistence, fault tolerance, connection management, event demuxing, and service configuration. When middleware architects and developers are successful, they create component abstractions that hide these error-prone and tedious mechanisms and protocols. Application developers therefore needn't be as experienced with complex systems-level technologies since they can program to these higher-level component abstractions.
I've also observed, however, that horizontal middleware platform efforts generally fail when application developers are (1) too inexperienced, (2) the domain is sufficiently challenging, and (3) the middleware team lacks sufficient training, resources, time, or empowerment to create a stable platform. It's therefore important that developers at all levels improve their technical skills and learn how to apply good software principles, patterns, and practices.
In my experience, many companies have a hard time getting even relatively small systematic reuse efforts to work right the first time. It's therefore essential to resist the urge to institutionalize ambitious corporate-level reuse initiatives before the people and the processes have reached the proper maturity. I've found that a much more effective process in practice is to iteratively grow reuse efforts out of smaller-scale successes that allow developers and managers to build the necessary skills and confidence.
For systematic reuse to succeed therefore, management must have the vision and resolve to support the incremental evolution of reusable software assets. Fred Brook's observation that ``Plan to throw the first one away, you will anyway'' [Brooks:75] applies as much today as it did 25 years ago. Increasingly, an 80% solution that can be deployed and evolved incrementally is preferable to waiting for a 100% solution that never ships. This phenomenon is referred to as the ``piecemeal growth'' model of software deployment, sometimes referred to by the more colorful name ``Worse is better'' [Gabriel:98]. According to this model, ``the best is the enemy of the good'' since striving for perfection makes it hard to hit tight market windows and gain initial market share.
In contrast, reuse efforts that try to work top-down, e.g., from high-level domain analysis, are highly likely to fail. The culprit is often the lack of close feedback loops between developers of reusable middleware and developers of applications. For instance, I've observed that it's usually counter-productive to create reuse teams that build middleware frameworks and components in complete isolation from application teams.
Since reuse efforts rarely have sufficient resources to please all possible customers, it's important to be goal-directed, rather than exhaustive, in determining which assets to develop, enhance, and maintain. Without intimate feedback from application developers, therefore, the software artifacts produced by component teams rarely address core business problems and won't be reused effectively.
It's also important that the relationship between application developers and reuse groups be mutually synergistic. For instance, reuse teams should be responsive to fix problems that inevitably arise in their middleware. Likewise, application developers should actively help to improve reusable artifacts, rather than waiting passively for the reuse team to find and fix all the problems. While it's possible to depend upon developer altruism and good will to achieve these goals, it's usually more effective to institutionalize a reward system with incentives to encourage effective relationships between developers and users.
For large-scale, long-lived networked applications it's essential to design and use architectures that transcend any specific technology or middleware standard. For instance, programming directly to a proprietary middleware API is risky since these APIs can rapidly become obsolete. I've therefore found it's more fruitful to develop networked applications based on a common architecture that can be instantiated on a range of middleware and OS platforms.
An effective pattern I've seen applied repeatedly to organize a common software architecture is to use (1) frameworks in the horizontal middleware platform layers and (2) components in the vertical application layers. Components are self-contained instances of abstract data types (ADTs) that can be plugged together to form complete applications. Common examples of components include COM+ controls and CORBA Object Services.
The relationship between frameworks and components is highly synergistic, with neither subordinate to the other. Frameworks can be used to develop components, whereby component interfaces provide Facades for internal class structures inside a framework. Moreover, components can be used as ``pluggable strategies'' within a framework.
Compared with frameworks, components are less tightly coupled and can support binary-level reuse more readily. For example, application developers can reuse components without having to subclass from existing base classes. In my experience, application developers are generally more comfortable and successful programming with components than they are customizing frameworks. Conversely, frameworks are useful for middleware teams because they help to simplify the development of horizontal platform software. Naturally, components can also be used to develop infrastructure and middleware.
The urge to apply one-dimensional solutions to complex problems isn't limited to technologists, however. For instance, there is a school of thought that claims only the non-technical impediments to reuse are worth addressing since systematic reuse fails solely for economic and organizational reasons, not technological ones. According to this perspective, investing in education or training to improve the technical skills of developers is pointless because it has no impact on success.
Unfortunately, one-dimensional non-technical solutions are no better than one-dimensional technological solutions. Managerial and organizational support is certainly desirable and essential for large-scale adoption of systematic reuse across an enterprise. In my experience, however, this support is not sufficient, nor even always necessary, to succeed with systematic reuse, particularly within smaller parts of organizations.
Moreover, focusing solely on organizational and economic impediments at the expense of technology and skills-building, can yield a corporate culture of ``learned helplessness.'' Developers suffering from this malady often postpone improving their design and reuse skills until the entire organization is ``cured.'' This approach is as futile as waiting for all the customer requirements to solidify before engaging in architecture and design phases.
Failing to invest in technology and education can greatly hamper a company's ability to compete effectively, particularly when time-to-market is crucial to success. It can also cause companies to become dangerously out of touch with contemporary software practice, where an increasing number of companies are in fact succeeding with systematic reuse and COTS technology adoption. Not surprisingly, many of the large companies that suffered the most during the economic downturn in 2001 were also companies that most strongly resisted adopting systematic reuse and COTS.
I therefore believe that we must not wait passively for organizational and economic problems to be resolved completely before building the technical skills and experience level of developers. Instead, we must initiate and support skills-building education now and sustain them over time. These skills are ultimately required to succeed with systematic reuse, in particular, and high-quality software development, in general.
For systematic reuse to succeed, it's crucial that developers who build reusable assets are empowered to own, maintain, and enhance these assets. Ultimately, reusable components and frameworks are only as good as the people who build and use them. In my experience building complex networked applications, there's simply no substitute for hiring high-quality, experienced software developers. The key to success is to hire developers whose skills and knowledge of patterns and design principles have withstood the test of time.
In practice, of course, it's hard to find top-notch software developers. Ironically, many companies---particularly large ones---still treat their developers as interchangeable, ``unskilled labor,'' who can be replaced easily. I'm increasingly noticing, however, that companies who respect and reward their top-notch software developers consistently outperform those who don't.
Systematic reuse is largely a by-product of good designs and experienced developers. Education is crucial to help improve developers' design skills. Fortunately, developing good reusable software requires many of the same set of skills, such as knowledge of architectures, patterns, frameworks, and components, necessary to develop good software in general. The time and effort spent on education will pay off therefore, whether or not developers actually write reusable software artifacts.
Ultimately, organizations that attempt systematic reuse without providing an incubation environment will lose their faithful. Many of these faithful will be the most experienced developers or those most capable of coming up to speed quickly. In markets driven by ``Internet cycle times,'' the loss of valuable developers can devastate an organization's long-term competitive viability.
Keeping the faith requires keeping abreast of external R&D developments and global technology trends. In my travels throughout the software industry, I am continually amazed by the rate at which reuse and COTS middleware is being adopted in many businesses and application domains. I suspect that the pundits who dismiss reuse as a myth simply haven't spent enough time ``in the trenches'' lately to recognize the speed at which the software industry is moving away from programming applications from scratch to integrating applications out of reusable frameworks and components.
I want to stress, however, that these technological solutions alone are not silver bullets [Brooks:87]. I firmly believe the promise of systematic reuse for networked applications will not be fully realized until we address both technical and non-technical impediments effectively. I also strongly believe, however, that there's no virtue in waiting for organizational and managerial maladies to be resolved completely before improving the education and experience of software developers. Fortunately, most software professionals are eager to hone their technical skills, so future impediments to successful reuse will be largely self-imposed.
[Brooks:87] Frederick P. Brooks, ``No Silver Bullet: Essence and Accidents of Software Engineering,'' IEEE Computer, Volume 20, Number 4, April 1987, 10-19.
[Foote:97] Brian Foote and Joseph Yoder, ``The Selfish Class,'' in Pattern Languages of Program Design 3, eds. Robert C. Martin, Dirk Riehle, and Frank Buschmann, Addison Wesley, 1997.
[Gabriel:98] Richard P. Gabriel, ``Patterns of Software, Tales from the Software Community,'' Oxford University Press, 1998.
[GoF:95] Gamma et al., Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley, Reading, MA, 1995.
[POSA:96] Buschmann et al., Pattern-Oriented Software Architectures, Wiley & Sons, 1996.
[Schmidt:97] Douglas C. Schmidt, David Levine, and Sumedh Mungee, ``The Design and Performance of Real-Time Object Request Brokers,'' Computer Communications, Volume 21, No. 4, April, 1998.
[Schmidt+Fayad:97] Douglas C. Schmidt and Mohamed Fayad, ``Object-Oriented Application Frameworks,'' Communications of the ACM, Special Issue on Object-Oriented Application Frameworks, Vol. 40, No. 10, October 1997.
[Schmidt:00] Douglas C. Schmidt, Michael Stal, Hans Rohnert, and Frank Buschmann, Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Objects, Wiley and Sons, 2000.
[Schmidt:01] Douglas C. Schmidt and Steve Huston, C++ Network Programming: Mastering Complexity with ACE and Patterns, Addison-Wesley, 2001.
[Schmidt+Porter:01] Douglas C. Schmidt and Adam Porter, Leveraging Open-Source Processes to Improve the Quality and Performance of Open-Source Software, 1st Workshop on Open Source Software Engineering, ICSE 23, Toronto, Canada, May 15, 2001.
Back to Douglas C. Schmidt's home page.
Last modified 21:03:47 CST 16 December 2012