Abstract: This paper uses an established reuse metric
and return on investment (ROI) model to present a new metric for
estimating the financial benefit of software development within
product lines. This metric can help organizations develop
a business case to support the early development of shared software
for use across families of related applications and products.
The discussion and metrics apply (1) early in the software life-cycle
for estimating product line benefits, and (2) after product release
for calculating the benefits that resulted from product line development.
Product line software development has attracted significant attention due to the rich opportunities it brings to development organizations (Blaney et al., 1996). By designing and building software for repeated use across many products, an organization receives direct benefits in terms of lower development costs, faster cycle times, lower maintenance costs, and higher quality software. The organization also receives indirect benefits that stem from commonality of the internal software architecture (for maintenance) and commonality of function (for users).
In fact, product line development has much in common with domain analysis and domain engineering. The primary difference between the two lies in how we define the scope of each. Software developers (technical persons) usually define the boundaries of a domain based on issues related to the development of the software solution. Managers usually define the boundaries of product lines based on customer-desired attributes and the potential market for products (Poulin 1997b). Both concepts represent approximately the same level of abstraction and have the same purpose in terms of improving the software development process. For this reason, the metrics in this paper apply equally well to both product lines and Domain-Specific Software Architectures (DSSAs) (Tracz, 1991).
We seek to estimate the realistic benefits of product line software development by means of simple, easy-to-understand metrics. In the next section we will start by reviewing the standard reuse metrics and economics model (Poulin 1997a). In Section III we introduce "The Economics of Product Lines." The paper concludes with examples showing how to use the metrics.
The discussion in this paper assumes that the users of the metrics have adopted and adhere to a formal framework that defines what to count as reuse. This paper uses the framework for counting a Reused Source Instruction (RSI). Without such a framework, no metric program could possibly deliver consistent and reliable results (Poulin 1997a).
Finally, the examples in the paper use lines of code (LOC) for units. Although LOCs have well-known deficiencies, they also have a number of advantages. These advantages include serving as an excellent overall indicator of total effort on a project as well as providing us a convenient unit to use for explaining the concepts in this paper. Since the concepts do not depend on units of measurement, we can use these metrics with any unit (for example, function points) simply by making the appropriate substitutions.
The vast majority of reuse economic models incorporate some form of Cost-Benefit (C-B) analysis. This approach has an intuitive appeal because it uses the commonsense axiom that "If it costs less than the benefits, we should do it." However, none of the C-B models in existence today outline the many C-B factors that influence software development with reuse, domain analysis, and product lines. Even if we had such a list, we would find gathering and quantifying every individual C-B factor a daunting, if not impossible, task.
To solve the problem caused by the inherent incompleteness of Cost-Benefit methods, we use two abstractions called the Relative Cost of Reuse (RCR) and the Relative Cost of Writing for Reuse (RCWR). RCR represents the ratio of the portion of the effort that it takes to reuse software without modification to the cost normally incurred to develop it for one-time use. For example, if you can reuse software for only 20% of the cost of new development, we say that RCR = 0.2. On the other hand, RCWR represents the ratio of the portion of the effort that it takes to develop reusable software to the cost of writing it for one-time use. For example, if it costs an additional 50% effort to complete a domain analysis, create a more generic design, provide additional documentation, etc., we say that the RCWR = 1.5.
Every organization will have its own values for RCR and RCWR. Even within organizations the values for RCR and RCWR will fluctuate according to environmental factors such as the experience of the reuser, organizational processes, the complexity of the software, etc. However, a plethora of quantitative evidence shows that we can assign reasonable default values of RCR = 0.2 and RCWR = 1.5 (Poulin 1997a). The examples that follow will use these suggested default values. Organizations that lack their own detailed quantitative data can also use these values in order to create a realistic business case for product line development in their environment.
Our standard reuse economic model uses RCR and RCWR to calculate the total Reuse Cost Avoidance (RCA) of reuse and the Additional Development Costs (ADC) incurred by writing reusable software. In essence, the total financial benefit of reuse to a project, Project ROI, consists of the sum of the benefits experienced by each of the n organizations participating in the project minus the costs of building the software for each organization to reuse.
To use this model, we first determine the RCA. Because a project benefits from reusing software during the development and the maintenance phases of the life-cycle, we need to estimate both of these factors. During the development phase, we use the value of RCR to estimate the Development Cost Avoidance (DCA). We say that if it took "RCR amount of effort" to locate, evaluate, and integrate a component into our application rather than write it new, we saved an amount of effort proportional to (1-RCR) and the size of the component, in RSI. Specifically:
The Service Cost Avoidance (SCA) accrue in two ways. First, reusable software tends to have far fewer defects than average; evidence shows up to a 10x quality improvement in reusable software (Matsumoto 1995). This means the reuser eliminates a significant maintenance cost. Second, when a reuser experiences a defect, the reuser saves service costs by not having to support the software it reuses. Normally, these service costs come from having to fix errors located after the product has gone into production. If we know the historical rate of these errors and the cost to fix them, we can estimate the SCA for a reused component by:
Of course, the total reuse benefit results from the sum of these two avoided costs:
Next, we need to adjust this benefit by subtracting the additional costs that the project incurred by developing shared software. To do this we use the abstraction RCWR. In short, we incur an additional cost proportional to (1-RCWR) and the amount of reusable software that we write. This additional cost equals:
Now we will put this all together in an example.
Example: Your company has just delivered an application to one of your major customers. In all, three development teams worked to develop the three primary subsystems needed for the application. These teams reused 10k LOC, 15k LOC, and 5k LOC each for a total of 30k RSI. Most of this RSI came from a domain-specific repository of routines that your company has developed. However, during the project your three teams identified and developed 1500 LOC of routines that all three teams needed for their subsystems.
Because you do not have actual RCR and RCWR data for your teams, you decide to use the default values of RCR = 0.2 and RCWR = 1.5. You call your company cost engineers and find out that on projects of this type your developers have a historical error rate of 1.5 errors per thousand LOC and it costs $10,000 to fix these errors after product release. Your cost engineer tells you to use the industry standard new code development cost of $100/LOC for your calculations.
You calculate the DCA and SCA to find the total RCA:
So,
The total benefit of $2.95 million comes from saving 80% of the estimated new development cost and from not having to fix errors in the code you would have written. We now calculate the investment made in producing the 1500 LOC of shared components, and then subtract this investment cost from the RCA to find the Project ROI.
The resulting benefit to the project equals:
This example shows how to use the abstractions for the relative costs of reuse in the reuse metric models. The metric estimates the total financial benefit of reuse to your project as almost $3 million. This metric model does not assume any special development process or architecture; it simply reports reuse benefits based on the amount of software that an organization did not have to write and the amount of software that it developed for multiple uses.
We would like to apply what we know about reuse economics
to software development with product lines. In the absence of
an organized reuse program that might encourage up-front planning
and architecture development, reuse happens primarily as a matter
of convenience. Developers might locate utility routines in a
reuse library or salvage software from previous projects. In
this way your project may achieve some benefits. However, this
kind of unorganized approach to reuse will never approach the
potential possible through domain analysis, domain engineering,
and the planned sharing of software across related applications
within a product line.
Extensive experiences with software reuse show us that the failure to plan for reuse will limit reuse levels to at most 20% of an application. This comes from the amount of an application that typically consists of the low-level, general purpose utilities that developers find convenient to reuse. However, the study of domains and product lines reveal that we often have up to about 85% common code across applications within the same domain or product line. How do we use this information in an economic model to motivate or simply reflect on the benefits experienced from the sharing of software across related applications?
Intuitively, we find it easy to believe that reuse can lead to substantial cost reductions. As early as 1988 Gaffney et al., set out to study the relationship between the relative costs of reuse and determine the effects of amortizing the costs and benefits across many reusers. To explain this relationship, they created the Payoff Threshold (n0). The Payoff Threshold tells us how many times we have to reuse a component before we recover the investment we made in order to develop the component. In our terminology:
Using our default values of RCR = 0.2 and RCWR = 1.5, we find that n0 = 1.88. This means that if we have a domain or product line that we expect to contain more than one related application, we can expect to have a positive return on our investment in shared software. This simple business case confirms our intuition, but it only foreshadows the much larger benefits possible when working with product lines that contain perhaps dozens of related products.
Remember that we plan to use our metrics in two situations:
In the latter case we will of course use actual project data to accurately calculate how much CSW each product actually used. We find this the much easier situation because we have the information we need, and we end up with more reliable values. In the former case, however, we must somehow determine a priori the critical values needed in our metrics.
We quickly see that the number of products, n, that we intend to build plays a key role in our product line metrics. Obviously, the number of products in the product line determines the upper bound on the number of times we can expect to use the CSW that we develop. However, not every product will use all of the CSW. More specifically, then, we need to know how much of each product will come from the use of CSW.
Determining "the market" for the CSW has traditionally posed a significant obstacle. In conventional, component-based reuse, we sometimes developed "reusable" components simply because they seemed as if they might find multiple uses. However, we cannot build a product line business case based on the optimistic wishes of "build it and they will come." Malan and Wentzel addressed this issue by assigning a probability function to each component in their reuse library; this probability reflected the chances of a particular component in the library eventually becoming a part of a future product (Malan and Wentzel 1993). However, how do we assign a probability function to each component in the library? To complicate the matter, the degree of uncertainty about the evolution of a product line tends to increase over time. Thus, "the market" for CSW five years into the product life-cycle will have a different probability profile than for products we build today. Malan and Wentzel expound on, without finding a solution, the difficulty surrounding the assignment of these probabilities.
A more accurate approach would determine the market from the results of a formal domain or product line analysis (Arango 1993). This information should become available during the design of the product line software architecture. For each application or product, the analysis reveals the amount of CSW that each individual product will use.
Finally, we can obtain a rough estimate of our market
based on our prior experience in the product line or with similar
product lines. In any case, we face the hideous difficulties
and unreliability inherent in software size estimation. Independent
of the method, we use the information to determine the benefits,
in terms of Reuse Cost Avoidance, to each product that we intend
to build.
A. Calculating the costs of product line development
In our standard reuse metric equation, we reduced the calculated benefits of reuse by the additional costs we incurred by writing software for reuse by others. We justify not including the full cost of developing this software in the model because we realize that we would have developed the software anyway. We consider this a planned expense and our equation only takes into account the difference in development costs over writing the software for one-time use. We also do not consider the costs of supporting the software we reuse because we expect the original developers (or the reuse librarian) to provide that support if necessary. Our product line ROI metric differs from our standard reuse metrics in that it assumes full life-cycle costs for the CSW.
For software development with product lines, DSSAs, or any planned reuse program, we highly recommend a management organization that dedicates a team to the development and support of the CSW (Poulin 1997a, p. 94). We call this the Reuse Development Team. The product line manager should organize this team very early in the development cycle and well before organizing the development teams for any products. This team has the critical responsibility of conducting a thorough review of the product requirements, conducting a domain analysis of all affected domains, designing the software architecture for the product line or domains (DSSAs), designing and implementing the CSW for the software architecture, and then supporting the CSW for the lifetime of the product line.
We explicitly factor the costs of this effort into our Product Line ROI metrics. Like RCA, we need to account for both the development and maintenance phases of the life-cycle. The full cost of developing the CSW, CSWdev_costs, will depend on RCWR, the amount of the CSW, and the historical cost of developing code for one-time use. The equation resembles that for ADC, with CSW substituted for Code written for reuse by others. The costs of maintenance, CSWservice_costs, will exactly equal the SCA experienced by each product that reuses the CSW.
The total CSWcosts equals the sum of these two values:
Note that an alternate method of calculating CSWcosts
would simply take the total budget of the Reuse Development
Team and assign that value to CSWcosts. If
we wish to calculate the Product Line ROI on an annual
basis we would use the annual budget for the development team.
For a product line life-cycle that might span several years, we
would discount the annual values using a Net Present Value (NPV)
analysis as explained in the conclusion of this paper. However,
because we normally use the budget of the Reuse Development Team
to calculate our actual value for RCWR, we view this method as
equivalent to using RCWR, as presented above.
B. Product Line ROI
Having explained the investment costs, CSWcosts, we couple that with the benefits experienced by each product in our product line. For a product line of n products, the Product Line ROI equals the sum of the costs avoided by each product, RCAi, minus the investment we made in the common software:
We now give two examples of how to use the Product
Line ROI metric. In the first example, we show how to build
a business case for using product line development on an upcoming
suite of related products. In the second example, we show how
to estimate the benefits of product line software development
in a case study involving an avionics simulator and trainer product
line.
A. Developing a business case for future product line development
Estimating the sizes and costs of future development poses challenges in many respects. When determining the amount and nature of the CSW, as well as the potential market for the CSW, we must take into account information such as the total expected size of the product line and the number of products we expect to build. We presented methods to address this in the introduction to The Economics of Product Lines.
Because the economics of product lines depend heavily on the number of uses of each component, we really would like the best market information available for our estimates. To do this we depend on a domain analysis in order to determine the global opportunities for reuse and expected number of potential uses for individual components. However, a full domain analysis may require significant resources in terms of experienced persons and may take months to complete (Tracz 1993).
Perhaps the simplest approach calculates the Product Line ROI for the estimated reuse level across all estimated software development. If the organization has recent experience on a similar project, we can use that experience to estimate the Product Line ROI for the upcoming product line. This will result in an approximate value that, especially early in a project, may serve as our best and only estimate. The following example uses this approach to develop a business case for product line development.
Example: You seek to justify an investment in product line development for an upcoming Management Information System (MIS) that will consist of up to 20 related business applications. The total size of each application will vary from perhaps 50k LOC to 300k LOC, with the total estimated software development estimated at 1.2 million LOC. Your development team just completed work on a similar program for a large customer, where your team achieved consistent reuse levels of 23%, primarily from reuse of 27k LOC of CSW you developed specifically for that project. You intend to use this experience as the basis for estimate (BOM) in your business case.
Rather than take 23% of the total 1.2 million LOC development estimate, you decide a more conservative reuse level of 20%. This means that you expect 20% of the 3.4 million LOC to come from CSW, for a total of 240k LOC RSI. Again, because of the similarity between this and your previous project, you expect to obtain this reuse primarily from about 30k LOC of CSW that you develop specifically for this project. You take these numbers to the lead software architect, who validates your assumptions.
Before you start building products, you need to study, design, and build the CSW. During product development, you will need to support the CSW. To find these costs:
The total costs equal:
To offset this investment of almost $5 million, we calculate the expected benefits based on the estimated reuse levels on a project involving 1.2 million LOC. To do this, we use the equation for RCA:
So the total benefit comes to:
To complete the example:
This example shows almost an $18 million ROI on an
approximately $5 million investment. This result can provide
a formidable data point for the justification of planned development
of products using the product line approach.
C. Calculating the Product Line ROI following product development
In the next example, we will estimate the benefits on a completed project. Unlike during the planning phase when we must depend on estimates or probabilities, we have the advantage of knowing the actual reuse levels on our products. As before, we balance the costs of producing and supporting the product line CSW against the costs avoided by each product in the product line.
Example: Lockheed Martin
Federal Systems developed the Core product line for avionics
simulator and trainers. Bardo et al., describe how the team started
product line development with an intense review of the expected
new simulator implementations (Bardo et al. 1996). This review
led to a generic software requirements specification (SRS), software
architecture, and set of reconfigurable Ada and C components.
These components comprised approximately 50,000 LOC of CSW for
functions such as real-time controls, sensor simulation, navigation,
and operator interface. As of early 1996, the team had developed
six products and achieved very impressive reuse levels of 11-96%,
primarily through the use of this CSW. Table 1 details these
results, showing a total Reused Source Instructions (RSI) of
215,815 LOC.
To begin our Product Line ROI analysis, we first calculate the CSWcosts for the 50k LOC of CSW developed by the Core team. We will continue to use our default values for RCR, RCWR, costs, and error rates in these examples.
The total costs equal:
We next calculate the expected benefits based on the actual reuse levels on Core of 215,815 LOC. Again, we use the equation for RCA:
The total benefit comes to:
To complete the example:
The benefit of $12.26 million represents a factor of 1.49 from an investment in CSW of $8.25 million, which we can expect to grow as the team develops additional Core products. In a completed project at Lockheed Martin Federal Systems, the Automated Tracking and Monitoring System (ATAMS) product line, the development team delivered command and control systems to the U.S. Air Force for approximately one-fifth of the originally estimated cost (Randall et al., 1996).
Product line software development and domain-specific software architectures can lead to tremendous gains and benefits during software development. The systematic reuse methods they entail extend the conventional reuse of software components to include shared requirements, design models, architectures, and skills of the people who implement the CSW and subsequent products. This paper addresses the problems of:
To achieve these goals, this paper uses an established reuse metric and economics model. Based on a proven method, the metrics in this paper use straightforward assumptions to explain the economics of software product line development.
These metrics can provide significant insights with only minimal
cost and impact to the business. To maintain this simplicity,
these metrics do not include factors for the time value of money
and therefore directly apply to projects of limited duration.
For longer projects (e.g., those spanning 3-5 years or more)
we need to apply a standard Net Present Value (NPV) calculation
to the metrics for each year (time period) of the analysis (Favaro
1996). However, we place a very high value on making our metrics
understandable and accessible to everyone involved in software
development. To add factors for NPV, risk, expected value, the
cost of delayed investments (Sullivan 1996), etc., you should
consult with a qualified cost engineer.
Acknowledgments
A special thank-you goes to Dr. F.G. (Pat) Patterson for his support
and contributions both to this paper and the special issue of
this journal.
References
Arango, Guillermo (1993), "Domain Analysis Methods," in Software Reusability. Wilhem Shaefer, Ruben Prieto-Diaz, and Masao Matsumoto, eds. Ellis Horwood, Chichester, U.K..
Bardo, Tim, Dave Elliot, Tony Krysak, Mike Morgan, Rebecca Shuey, and Will Tracz (1996), "CORE: A Product Line Success Story," Crosstalk: The Journal of Defense Software Engineering, Vol. 9, No. 3, pp. 24-28.
Blaney, G., Evans, R. P., Kemp, K. M., Patterson, F. G., Jr., & Reddy, J. (1996). Conclusions and Recommendations. In W. M. Lawbaugh (Ed.), Proceedings of the Workshop: A NASA Focus on Software Reuse, 23-27 September 1996, George Mason University, Fairfax, Virginia (pp. 1-18). Washington: National Aeronautics and Space Administration.
Favaro, John (1996), "A Comparison of Approaches to Reuse Investment Analysis," Fourth International Conference on Software Reuse (ICSR'4), Orlando, FL, 23-26 April, pp. 136-145.
Gaffney, John E., Jr., and Thomas Durek (1988), "Software Reuse- Key to Enhanced Productivity; Some Quantitative Models," Software Productivity Consortium, SPC-TR-88-015.
Malan, Ruth and Kevin Wentzel (1993), "Economics of Software Reuse Revisited," Hewlett-Packard Technical Report HPL-93-31.
Matsumoto, Masao (1995), "Research to Shed Light on Software Shadows," IEEE Software, Vol. 12, No. 5, p. 16.
Poulin, Jeffrey S. (1997a). Measuring Software Reuse: Principles, Practices, and Economic Models. ISBN 0-201-63413-9, Addison-Wesley, Reading, MA.
Poulin, Jeffrey S. (1997b), "Software Architectures, Product Lines, and DSSAs: Choosing the Appropriate Level of Abstraction," Proceedings of the 8th International Workshop on Software Reuse (WISR'8), Columbus, OH, 23-26 March.
Randall, Richard L., David J. Bristow, Jesse G. Foster, and Maj. Dennis D. Kaip (1996), "Product-Line Reuse Delivers a System for One-Fifth the Cost in One-Half the Time," Crosstalk: The Journal of Defense Software Engineering, Vol. 9, No. 8, pp. 25-26.
Sullivan, Kevin (1996), "Software Design: The Options Approach," Proceedings of the Second International Software Architecture Workshop (ISAW-2), San Francisco, CA, 14-15 October, pp. 15-18.
Tracz, Will, Ed. (1991). Software Reuse: Emerging Technology. IEEE Computer Society Press, Los Alamitos, CA.
Tracz, Will, Lou Coglianese, and Patrick Young (1993),
"A Domain-Specific Software Architecture Engineering Process
Outline," ACM SIGSOFT Software Engineering Notes,
Vol. 18, No. 2, pp. 40-49.
Biography
Jeffrey S. Poulin (Jeffrey.Poulin@lmco.com) MD 0210,
Lockheed Martin Federal Systems, Owego, New York, 13827. http://www.owego.com/~poulinj.
Dr. Poulin works as a Senior Programmer and software architect with Lockheed Martin Federal Systems (formally Loral Federal Systems and IBM Federal Systems Company) in Owego, NY. As a member of the Advanced Technology Group, he works software reuse and architecture issues on a variety of software development efforts across Lockheed Martin.
From 1991-1993 Dr. Poulin worked in the IBM Reuse Technology Support Center (RTSC) where he led the development and acceptance of the IBM software reuse metrics and return on investment (ROI) model. He also organized, edited, contributed to, and published a complete library of resources on the IBM Reuse Method. Active in numerous professional activities and conference committees, Dr. Poulin has over 40 publications, to include a book on reuse metrics and economics published by Addison-Wesley. A Hertz Foundation Fellow, Dr. Poulin earned his Bachelors degree at the United States Military Academy at West Point and his Masters and Ph.D. degrees at Rensselaer Polytechnic Institute in Troy, New York.