Despite the benefits of having a fixed set of measures that could be applied to all situations, the approach of goal-based measurement has been much more successful in practice. Goal-based measurement relies upon the idea that measures must be tied to the organizational context, in particular to organizational goals, and that there is no set of measures that can be standardized and universally applied.
The academically most influential work developing industrial measurement programs was performed by Basili and his colleagues at the University of Maryland by developing the Goal-Question-Metric (GQM) approach (Basili and Rombach 1988, Basili et al. 1994). This approach, which is grounded in the authors’ experience with software measurement at the NASA Software Engineering Laboratory, is based upon the idea that measures must be developed based upon an organization’s explicitly stated measurement goals. The authors’ experiences with collecting measurement data (Basili 1984), showed that collecting data without a clear view of what it will be used for is likely to be counterproductive and lead both to missing data and the collection of data that will never be used, but stored in “data graveyards”.
Using the GQM, measures are defined top-down by first stating formal measurement goals, then asking questions that need to be answered in order to meet the goal, and finally, define measures that answer the questions. This exercise helps build a GQM tree that makes implicit the mental models people have of the object under study. This knowledge is important when interpreting the results of the measures. Figure below shows the goal tree, and a goal template used by the GQM approach. While not originally part of the GQM approach, the goal template was developed to help formulate measurement goals. As shown in the figure, each measure can be tied to several questions, and several questions to each goal.
Figure. The GQM Approach (Lassenius 2006)
While intuitively nice and simple, the GQM has not been without its critics. More specifically, it has been deemed to be more an approach for brainstorming than a real method, e.g., (Card1993), due in particular to its lack of a guiding process and the gap between the definition of goals and the selection of measures. Partly addressing this problem, subsequent work on GQM has developed extensions to the approach, such as the abstraction sheet (Briand et al. 1996), as well as documented the GQM process in greater detail (Solingen and Berghout 1999). The GQM approach has also become popular in industry – many of the existing experience reports, like (Grady 1992, Daskalantonakis 1992, Fuggetta et al. 1998) have based their measurement programs on the GQM, albeit the earlier ones quite informally.
In addition to the work expanding upon the GQM, the most well-known approaches to building industrial measurement programs, including the ami approach (Pulford et al. 1996), and the GQ(I)M approach (Park et al. 1996) are goal-driven. Common to these approaches is the identification of business goals that are further broken down into GQM measurement goals, thus adding another layer of abstraction to the process. Goal-based measurement is also widely adopted in fields outside software measurement. Many performance measurement approaches rely on the definition of goals and deriving related measures. In particular the popular Balanced Scorecard (BSC) approach for strategic measurement (Kaplan and Norton 1992) belongs to this group. The BSC requires companies to define measurable objectives in at least four perspectives: the customer, internal business process, learning and growth, and financial perspectives. This is claimed to help management get a “balanced” view of the organization replacing purely financial management with other complementing perspectives. The BSC and GQM have also been proposed to be natural to combine (Lassenius et al. 1999, Goethert and Fisher 2003).
The main problem with approaches like the GQM and its relatives lies in the idea of context-based measurement. Blindly subscribing to the idea of always developing new measurements for each new undertaking might, while perhaps given the right expertise and infinite resources, produce the best possible set of measures for each situation. Its main problem, in addition to cost, lies in lost ability to compare and benchmark, also making the development of predictive models harder since the data collected might not be comparable e.g. between projects. However, in some cases, e.g., when dealing with one-off development projects, this might not be a problem from a practical point of view. It is also possible to reuse (parts of) a GQM tree for similar situations, thus achieving some of the benefits of fixed sets of measures discussed above.
References:Basili, V. A methodology for collecting valid software engineering data. IEEE Transactions on Software Engineering, 10(6):728-738, 1984. Basili, V. and Rombach, H. The tame project: Towards improvement-oriented software environments. IEEE Transactions on Software Engineering, 11(6):758-773, 1988. Basili, V., Caldiera, G. and Rombach, H. The goal question metric approach. In Encyclopedia of Software Engineering. Wiley, 1994. Briand, L., Differding, C. M. and Rombach, H. Practical guidelines for measurement-based process improvement. Software Process – Improvement and Practice, 2:253-280, 1996. Card, D. N. What makes for effective measurement. IEEE Software, 10(6):94-95, 1993. Daskalantonakis, M. K. A practical view of software measurement and implementation experiences within motorola. IEEE Transactions on Software Engineering, 18(11):998-1010, 1992. Fuggetta, A. , Lavazza, L., Morasca, S., Cinti, S., Oldano, G. and Orazi, E. Applying GQM in an industrial software factory. ACM Transactions on Software Engineering and Methodology, 7(4):411-448, 1998. Goethert, W. and Fisher, M. Deriving enterprise-based measures using the balanced scorecard and goal-driven measurement techniques. Technical Report CMU/SEI-2003-TN-024, CMU/SEI, 2003. Grady, R. B. Practical Software Metrics for Project Management and Process Improvement. Prentice-Hall, Englewood Cliffs, NJ, 1992. Kaplan, R. and Norton, D. The balanced scorecard – measures that drive performance. Harvard Business Review, 70(1) :71-79, 1992. Lassenius, C. “Software Development Control Panels: Concepts, a Toolset and Experiences”, Doctoral dissertation, Helsinki University of Technology, 2006. Park, R., Goethert, W. and Florac, W. A. Goal-driven software measurement – a guidebook. Technical report, Software Engineering Institute, 1996. Pulford, K., Kuntzmann-Combelles, A. and Shirlaw, S. A Quantitative Approach to Software Management: The ami Handbook. Addison-Wesley,Wokingham, England, 1996. Solingen, V. R. and Berghout, E. The GoałQuestion/Metric Method. McGraw-Hill International, Maidenhead, UK. 1999.