THE IBM REUSE METRICS: QUANTIFYING SOFTWARE REUSE 5 March 1993 Jeffrey S. Poulin International Business Machines Corporation Reuse Technology Support Center PO Box 950 Department A80D, Building 996 Poughkeepsie, New York 12602 PoulinJ at TDCSYS2 email: poulinj@tdcsys2.vnet.ibm.com ii The IBM Reuse Metrics: Quantifying Software Reuse ABSTRACT ________ Reuse metrics must accurately reflect the amount of effort saved. We must _____________________________________________________________________________ have realistic measures to ensure the credibility of the value we place on _____________________________________________________________________________ reuse and to separate reuse benefits from those of other technologies also _____________________________________________________________________________ competing for limited investment dollars. This paper defines the IBM reuse _____________________________________________________________________________ metrics. Used in conjunction with a planned reuse program, the potential of _____________________________________________________________________________ reuse serves as a powerful motivator. We derive three reuse metrics from _____________________________________________________________________________ readily available software data elements and use these metrics in a return on _____________________________________________________________________________ investment model that establishes a sound business justification for reuse. ___________________________________________________________________________ OVERVIEW ________ Management traditionally uses software metrics to quantify the software development process. With an emerging technology, however, metrics must extend beyond their traditional role. Applying reuse metrics must also encourage the practice of reuse. Most organizations do not practice formal reuse or remain reluctant to invest in a formal reuse program. Reuse metrics must assist in the technology insertion process by providing favorable process improvement statistics and by placing emphasis on activity conducive to reuse. Our experience shows great motivation results from establishing a positive financial return using a sound ROI model. When reviewing reuse experience reports, the reader must consider results in their reported contexts. The interpretation of any measurement depends on the collection method, environment, and numerous other factors. Most reports lack sufficient supporting information. The reader therefore does not have the details required to repeat the experiment or case study, understand the results, or cogently agree with or disagree with the conclusions [17]. This paper explains the method IBM uses to quantify and report reuse. ORGANIZATIONAL GOALS Before we define metrics, we must establish organizational goals [5]. In doing so, we find a very controversial question: "What should we measure as reuse?" For example, we may describe reuse as the building of applications from building blocks of reusable parts, porting, using operating systems, using high-level languages, or using spreadsheets, databases, and similar products. Although these tools and techniques may greatly increase the pro- ductivity of the average programmer, their use does not always constitute "reuse." Ultimately, we must define what to measure and report as reuse by the goals of the organization. Because we input the IBM metrics to an ROI model we must also ensure the metrics reflect effort saved, both by quanti- fying the level of reuse in an organization and by determining the investment value of reuse. Specifically, the IBM reuse metrics have the following goals: 1. Reuse metrics must reflect effort saved. 2. Reuse metrics must encourage reuse. Along with the above two goals, we use the following three general defi- nitions of reuse throughout the paper. These common definitions describe 1 software reuse; each provides a necessary but not always sufficient condition for reuse. They serve useful in helping to resolve the numerous issues asso- ciated with applying metrics in an organization. 1. Reuse means using an existing component in a new context. 2. Reuse means raising of the level of abstraction in programming. 3. Reuse means not writing what you would have written. The real difficulty in reuse metrics lies in judging the third definition. Often, the question hinges on whether the programmer uses or reuses software. ____ ______ Furthermore, as a technique becomes ingrained in a culture, it transitions from "novel" to "expected." What we consider reuse today we may simply con- sider "used" three years from now. The IBM reuse metrics provide input to business cases justifying investments in reuse. Because of this, we must distinguish between the usual software development process and how the process changes with reuse. Therefore, resolving most of the issues in reuse metrics depends on focusing on effort saved, or the value of reuse. WHAT TO MEASURE AS REUSE We must differentiate between code recovery, which results in new software to maintain, and planned reuse, in which we assemble or tailor products from building blocks of reusable software. We must also define reuse based on "who" uses the component. To improve the practice of reuse we must understand that we expect good design and management within development organizations and that we find com- munication less common between organizations. Although many groups work closely together, communication, essential to the simple exchange of informa- tion and critical to sharing of software, becomes more difficult as the number of people involved grows and natural organizational boundaries emerge. Therefore, measurements must encourage reuse across these organizational ______ boundaries. We determine that for a component to count as reuse, an organization that did ______ not develop and does not maintain the component must use the component. Because software development organizations vary we define a "typical" organ- ization as either a programming team, department, or functional group of about eight people or more. Also, although organizational size may indicate how well communication within and between organizations takes place, we must also consider functional boundaries between organizations. For example, a small programming team may qualify as an organization if it works independ- ently. For consistency, we report the type and size of the reporting organization when reporting reuse metrics. This provides an informal check on the flexi- bility allowed in selecting the most appropriate boundary for the organiza- tion. Selection of an inappropriate small boundary would distort the value of the metrics upward and an inappropriate large boundary would result in low 2 The IBM Reuse Metrics: Quantifying Software Reuse reuse values. Changing the organizational boundary between reports would eliminate any possibility for comparisons and evaluation of the reuse program. REUSING VERSUS USING COMPONENTS Many organizations report the reuse percent of a product or a new release of a product. The organizations intend to indicate the portion of their product (normally expressed in lines of code, or LOC) saved by reusing software from other products or product releases. In either case, the effort attributed to reuse comes from completely unmodified reusable components. We make reusable components easy to identify in new products by using straightforward cri- teria; if use of a component saved having to develop a similar component, we record it as reuse. However, although an organization may "use" a component numerous times, an organization can "reuse" a component only once. We make this distinction to increase the accuracy of our return on investment analysis of projects. Since we expect organizations to develop subprograms for often-needed services and to use components previously developed for a product or previously developed by themselves, we do not credit them with savings resulting from these activities. We consider this good software development, a practice expected from every programmer. In one actual example, a project reported 11 kloc of reuse on a relatively small application. Closer inspection revealed 5120 lines of the 11 kloc came from one 10-line reusable macro and that all 5120 lines came from the same module. A code review revealed that the original code: Do i := 1 to 512 MACRO(i); consisted of 2 instructions (the DO..WHILE and the call to MACRO) and 10 reused instructions. However, the optimizer automatically unrolled the loop to yield: MACRO(1); MACRO(2); .... MACRO(511); MACRO(512); The report therefore contained 512 source instructions and 5120 reused instructions. This does not accurately reflect the productivity or reuse on the project. For this reason, IBM considers that an organization may "use" a part numerous times, but an organization may "reuse" a part only once. REUSE METRICS _____________ 3 OBSERVABLE DATA We calculate reuse metrics from the following observable data elements which IBM [11] and other companies [7] have used for many years. Managing the software process depends on complete and accurate data; sensible reuse metrics require certain data about the process. We can usually measure observable data directly from the product. For example, we can directly measure the different classes of source instructions. Observable data may also come from historical files that we collected for a variety of reasons related to managing the software development process. Historical data that IBM normally collects include costs for software development, defect rates, and maintenance costs. Shipped Source Instructions (SSI) Shipped Source Instructions (SSI) indicate the number of non-comment instructions in the source files of a product. SSI do not include Reused ___ Source Instructions (RSI). A call to a reusable part counts as one SSI. When reporting reuse measures for development organizations, SSI include all the source instructions the organization maintains. The lines of code actu- ally written by someone for a product contribute to the SSI. Changed Source Instructions (CSI) Changed Source Instructions (CSI) indicate the number of non-comment source instructions that programmers wrote new, added, or modified in a product _________________________ release. CSI does not include Reused Source Instructions (RSI) or unchanged ___ base instructions from prior releases of the product. CSI includes all source instructions from partially modified components. A call to a reusable part counts as one CSI. Reused Source Instructions (RSI) Reused Source Instructions (RSI) indicate the source instructions an organ- ization ships but does not develop or maintain. RSI come from completely unmodified components. Programmers normally locate these components in a reuse library. Base instructions from prior releases of a product and source instructions from partially modified parts do not count as RSI. Source Instructions from a reused part count once per organization inde- pendent of how many times one calls or expands the part. Source Instructions Reused by Others (SIRBO) Source Instructions Reused by Others (SIRBO) indicate the amount of an organ- ization's source instructions that other organizations reuse. This indicates how much an organization contributes to reuse. A successful reuse program depends not only on organizations reusing software but also them helping other organizations reuse software. SIRBO measures both the parts contrib- uted for use by others and the success of those parts. Organizations writing 4 The IBM Reuse Metrics: Quantifying Software Reuse successful reusable parts will have a very high SIRBO because SIRBO increases every time another organization reuses their software. This encourages organizations to generate high quality, well-documented, and widely- applicable reusable components. To measure SIRBO, for each part an organization contributes to a library, take the sum of: cabove Example: If an organization contributes the following to a reuse library: a ________ 10kloc module in use by 5 other departments, a 25kloc macro in use by 6 other departments, and an unused 75kloc macro, the organization's SIRBO equals: < (5%depts. times 10%kloc) plus (6%depts. times 25%kloc)> cabove Like RSI, the number of times the same organization invokes or calls the part does not affect SIRBO. The same rules apply for counting RSI; use of a reus- able part saved having to develop the part one time, not one time for every call to the part. SIRBO changes with time. As more organizations reuse the components, the SIRBO of the donating organizations increases. Software Development Cost (Cost per LOC) To determine the financial benefit of reuse, we must know the cost of devel- oping software without reuse. We generally can obtain this new software development cost from historical data kept by the financial planners and man- agement of the organization. If necessary, calculate the new software cost by adding all the expenses of the organization, including overhead, and dividing by the total output (in LOC) of the organization. Software Development Error Rate (TVUA rate) No amount of testing, inspection, or verification can guarantee that we have released a product without errors. Although emphasis on quality and strict adherence to development processes leads to better products, errors inevi- tably reveal themselves in the marketplace. Every development organization has a historical average number of errors, or TVUAs ("Total Valid Unique _____ Program Analysis Reports") uncovered in their products. Note that we normally design and test software components built for reuse to stricter standards than those for normal program product components. We justify the additional cost of this effort by the savings gained when other organizations do not have to develop and maintain a similar component. The additional testing results in increasing the quality of the component, so we can expect fewer errors from reusable code. 5 Software Error Repair Cost (Cost per TVUA) To quantify the benefit of the increased quality of reusable components, we need the historical average cost of maintaining components with traditional development methods. As with software development cost, we generally obtain this figure from financial planners and management in the organization. If necessary, calculate the software error repair cost by taking the sum of all costs, including overhead, of repairing latent errors in software maintained by the organization and dividing by the number of errors repaired. Although software maintenance includes enhancements to products, the cost of increasing function does not factor into the software error repair cost. DERIVED METRICS The observable data elements combine to form three derived reuse metrics. The first two metrics indicate the level of reuse activity in an organization as a percentage of products and by financial benefit. The third metric includes recognition for writing reusable code [15]. 1. Reuse Percent; the primary indicator of the amount of reuse in a product or practiced in an organization. Derive Reuse Percent from SSI, CSI, and RSI. 2. Reuse Cost Avoidance; an indicator of reduced total product costs as a result of reuse in the product. Derive Reuse Cost Avoidance from from SSI, CSI, RSI, TVUA rates, software development cost (cost per LOC), and maintenance costs (Cost per TVUA). 3. Reuse Value Added; an indicator of leverage provided by practicing reuse and contributing to the reuse practiced by others. Derive Reuse Value Added from SSI, RSI, and SIRBO. Reuse Percent The Reuse Percent indicates the portion of a product, product release, or organizational effort that we attribute to reuse. The importance of the Reuse Percent comes from the ease of calculation and ease of understanding. Unfortunately, it has little value without a supporting framework. Many com- panies report their reuse experiences in terms of "reuse percent" but few describe how they calculate the values. They commonly include unplanned reuse in the value, making it difficult to assess actual savings or produc- tivity gains. Since we clearly define RSI, we believe our Reuse Percent metric reasonably reflects the development effort saved. REUSE PERCENT OF A PRODUCT: To calculate the Reuse Percent of a product (or first release of a product): Reuse Percent equals > times 100 percent Product Example: If a product consists of 65kloc SSI and an additional 35kloc ________________ from a reuse library, then find the Reuse Percent of the product by: 6 The IBM Reuse Metrics: Quantifying Software Reuse Reuse Percent equals <35%kloc over <35%kloc plus 65%kloc>> times 100 percent equals 35 percent REUSE PERCENT OF A PRODUCT RELEASE: For a new release of a product, RSI comes from reusable components that did not appear in previous releases (or the product base) of the product. We count calls to component used in pre- vious releases as one new or changed source instruction (CSI). Calculate the Reuse Percent of a product release using: Reuse Percent equals > times 100 percent Product Release Example: If a release of a product consists of 7k CSI plus 3k ________________________ "new" RSI from a reuse library, then calculate the Reuse Percent for this product release as: Reuse Percent equals <3%kloc over <3%kloc plus 7%kloc>> times 100 percent equals 30 percent REUSE PERCENT FOR AN ORGANIZATION: All software developed and maintained by an organization counts as the organization's SSI. Any software used by the organization but maintained elsewhere counts as their RSI. We calculate the Reuse Percent of an organization using: Reuse Percent equals > times 100 percent Organizational Example: If a programming team develops and maintains 80k SSI _______________________ and the team additionally uses 20k RSI from a reuse library, then the Reuse Percent for the team is: Reuse Percent equals <20%kloc over <20%kloc plus 80%kloc>> times 100 percent equals 20 percent Reuse Cost Avoidance (RCA) The Reuse Cost Avoidance estimates the financial benefit of reuse. When used with a clear definition of RSI, this metric shows the tremendous return on investment potential of reuse. Because we use RCA when performing return on investment (ROI) analysis of reuse programs, RCA helps with the insertion of reuse technology. Reusing software requires less resources than new development, but does not come free. The developer still must search for, retrieve, and assess the suitability of reusable components before finally choosing the appropriate part for integration into the product. Although reuse requires effort to understand and integrate reusable parts, studies show that the cost of this effort consists of only about 20% of the cost of new development [18], [8]. Based on this relative cost of reuse, we estimate the financial benefit __________________________ attributable to reuse during the development phase of a project: 7 cabove However, we typically spend only about 40% of the software life cycle in development [10]; significant maintenance benefit also results from reusing quality software. We may estimate this benefit as the cost avoidance of not fixing errors (TVUAs) in newly developed code [9]. We can represent this savings as: cabove The total Reuse Cost Avoidance becomes: cabove cabove Example: If an organization has a historical new code development cost of ________ $200 per line, a TVUA rate of 1.5/kloc, and a cost to fix a TVUA of $43k, then we calculate the RCA for integrating 20k RSI into a product by: labove labove labove labove Reuse Value Added (RVA) The previous two metrics measure how much organizations reuse software. We must also motivate contributions of reusable software. The Reuse Value Added provides a a metric that reflects positively on organizations that both reuse software and help other organizations by developing reusable code. We use a ratio, or productivity index for the RVA; organizations with no involvement in reuse have an RVA=1 . Contributions to reuse and reusing software positively correlate with the ratio. An RVA=2 indicates the organization becomes approximately twice as effective as the same organization without reuse. In this case, the organization doubled its productivity (or effectiveness) either directly (by reusing software) or indirectly (by maintaining software that other organizations use). Observe that we do not take into account the additional costs of writing code for reuse nor the relative costs of reusing code. We seek only an indicator of 8 The IBM Reuse Metrics: Quantifying Software Reuse the multiplicative effect of planned reuse in an organization. To represent the total effectiveness of a development group, use: equals <(SSI plus RSI) plus SIRBO> over Example: A programming team maintains 80kloc and uses 20 kloc from a reuse ________ library. In addition, five other departments reuse a 10kloc module the pro- gramming team contributed to the organizational reuse library. Calculate the RVA of the programming team as: labove ' ' labove over 80%kloc equals 1.9> In this example, the RVA of 1.9 indicates the programming team became approx- imately 1.9 times more effective than the same team without reuse. Some organizations organize to obtain the most benefit possible from reuse. For example, the Mid-Hudson Valley Programming Laboratory and the IBM Federal Systems Company in Rockville, Maryland [12] dedicate programming teams to develop and maintain shared software or site-wide reuse libraries. Corporate parts centers, such as the Boeblingen software center, also develop and main- tain software for IBM-wide use. Experience shows that although these types of teams may have modest values for the Reuse Percent metric, they have extremely high values for the RVA metric. This high RVA indicates the tremen- dous programming leverage they provide to their organizations. RELATED WORK ____________ Since an essential step in assessing the success and effectiveness of a reuse program includes quantifying results, several groups have reported on reuse metrics. However, the uniqueness of these metrics lies in the attention given to the definition of RSI and in attempting to present reuse as "real effort saved." Although [1] differentiates between reuse within an organiza- tion and reuse from sources external to the organization, no other paper addresses how to measure the classes of reuse nor do they provide a concen- trated definition of RSI. Related work includes metrics to assess the reusability of components and to determine the suitability of a component for use in a new application [4], [6]. However, our work focuses on measuring the amount of reuse and the value of reuse to an organization. Although we find that many organizations use "reuse percent," the majority of published methods focus on financial analysis. Other organizations also find that the financial benefit of reuse convinces project managers and planners to invest in formal reuse programs [13]. In 1988 Gaffney and Durek [9] published a comprehensive model addressing business case analysis of reuse. They premise their model on the need to amortize the cost of the reuse program, including the additional cost to build reusable components, across all projects using the component. When doing cost benefit analysis for software reuse, one needs to consider the 9 long-term benefits and associated costs which apply to every project using the component. They argue that a better economic estimate includes the number of times the organization reuses the component. Other groups present cost-benefit models of reuse [14] and [2]. The NATO model consists of listing the major benefits and costs of reuse, then applying time-value of money formulas to adjust for future values. The Barns and Bollinger model determines the cost/benefit of reuse by subtracting investment costs of producing reusable software from the reduced consumer development costs minus adaptation costs. FUTURE WORK ___________ Future work includes further validation of these measures and ROI model. This includes comparing the predicted versus actual costs avoided and com- paring increased productivity rates with the values calculated in the metrics and the ROI model. Although the model uses industry experience for default values in the equations, we use actual values when we have them. Future work will include methods to quantify reuse in areas other than software (e.g., Design, Test Case, Information Development). CONCLUSION __________ Managing the software process depends on sound business decisions based on accurate measurements. This paper introduces the following three metrics for software reuse: Reuse Percent, Reuse Cost Avoidance, and Reuse Value Added. The metrics rely on easy to collect data, provide reasonable representations of reuse activity, and encourage reuse. Most importantly, the metrics provide reliable input to the corporate reuse ROI model, where we carefully and realistically define the benefits attributable to reuse. With emerging technologies such as software reuse, the value of metrics and ROI analysis goes beyond the traditional benefits of assuring the quality of reusable components, demonstrating the success of a program, and improving the ability to plan and predict for future projects. They also serve to encourage reuse by providing feedback on the results of a reuse program and by highlighting the benefits of a organizational reuse effort. REFERENCES __________ [1] Banker, Rajiv D., Robert J. Kauffman, Charles Wright, and Dani Zweig, "Automating Output Size and Reusability Metrics in an Object-Based Com- puter Aided Software Engineering (CASE) Environment," Unpublished Tech- _________________ nical Report, 25 August 1991. _____________ [2] Barns, B.H. and T.B. Bollinger, "Making Reuse Cost Effective," IEEE ____ Software, Vol 8., No.1, Jan, 1991, pp. 13-24. _________ [3] Bourland, D. David and Paul Dennithorne Johnston, ed., To Be or Not: An ________________ E-Prime Anthology, International Society for General Semantics, San ___________________________________________________________________ Francisco, CA, 1991. 10 The IBM Reuse Metrics: Quantifying Software Reuse [4] Basili, Victor R., H. Dieter Rombach, John Bailey, and Alex Delis, "Ada Reusability Analysis and Measurement,," Empirical Foundations of Infor- ________________________________ mation and Software Science V, Atlanta, GA, 19-21 Ocotber 1988, pp. __________________________________ 355-368. [5] Basili, V.R., and R.W. Selby, "Paradigms for Experimentation and Empir- ical Studies in Software Engineering," Reliability Engineering and _____________________________ System Safety, Vol. 32, 1991, pp. 171-191. ______________ [6] Caldiera, Gianluigi and Victor R. Basili, "Identifying and Qualifying Reusable Software Components,," IEEE Software, Vol. 24, No. 2, February ______________ 1991, pp. 61-70. [7] Daskalantonakis, M.K., "A Practical View of Software Measurement and Implementation Experiences within Motorola," IEEE Transactions on Soft- ___________________________ ware Engineering, Vol. 18, No. 11, November 1992, pp. 998-1010 _________________ [8] Favaro, John, "What price reusability? A case study," Ada Letters, Vol. ____________ 11, No. 3, Spring 1991, pp. 115-24. [9] Gaffney, John E., Jr. and Thomas Durek, "Software Reuse- Key to Enhanced Productivity; Some Quantitative Models," Software Productivity Consor- _____________________________ tium, SPC-TR-88-015, April 1988. ____________________ [10] "Software Engineering Strategies," Strategic Analysis Report, Gartner ____________________________ Group, Inc., April 30, 1991. [11] "Corporate Programming Measurements (CPM)," V 4.0, IBM Internal Docu- __________________ ment, 1 November 1991. _____ [12] Margano, Johan, and Lynn Lindsey, "Software Reuse in the Air Traffic Control Advanced Automation System," paper for the Joint Symposia and __________________ Workshops: Improving the Software Process and Competitive Position, 29 ____________________________________________________________________ April-3 May 1991, Alexandria, VA. [13] Margano, Johan, and Thomas E. Rhoads, "Software Reuse Economics: Cost Benefit Analysis on a Large Scale Ada Project," Proceedings of the ____________________ International Conference on Software Engineering, 11-15 May 1992, _____________________________________________________ Melbourne, Australia, 1992, pp.338-348. [14] "Standard for Management of a Reusable Software Component Library," NATO ____ Communications and Information Systems Agency, 18 August 1991. ______________________________________________ [15] Poulin, Jeffrey S. and W.E. Hayes, "IBM Reuse Methodology: Measurement Standards," IBM Corporation Document Number Z325-0682, 16 July 1992. __________________________________________ [16] Poulin, Jeffrey S. and Joseph M. Caruso "A Reuse Measurement and Return on Investment Model," Proceedings of the Second International Workshop __________________________________________________ on Software Reusability, Lucca, Italy, 24-26 March 1993. ________________________ [17] Rombach, H. Dieter, "Design Measurement: Some Lessons Learned," IEEE ____ Software, March 1990, pp.17-25. _________ 11 [18] Tracz, Will, "Software Reuse Myths," ACM SIGSOFT Software Engineering ___________________________________ Notes, Vol. 13, No. 1, Jan 1988 p. 17-21. ______ BIOGRAPHY _________ JEFFREY S. POULIN joined IBM's Reuse Technology Support Center, Poughkeepsie, New York, in 1991 as an advisory programmer. His primary responsibilities include developing and applying corporate standards for reusable component classification, certification, and measurements. Dr. Poulin has worked in the area of software reuse since 1985 and helped lead the development and acceptance of the IBM software reuse metrics. He participates in the IBM Corporate Reuse Council, the Association for Computing Machinery, and Vice- Chairs the Mid-Hudson Valley Chapter of the IEEE Computer Society. Dr. Poulin earned his Bachelors degree at the United States Military Academy at West Point, New York, and his Masters and Ph.D. degrees at Rensselaer Polytechnic Institute in Troy, New York. 12 The IBM Reuse Metrics: Quantifying Software Reuse