Perhaps the most fundamental agile measurement is the cumulative measure for delivered customer value in terms of Running Tested Features. Ron Jeffries has described the measurement and its implications in a web article (Jeffries 2004).
Dubinsky et al. (2005) describe a case study of measurement approach of agile development in the software development unit in Israeli Air force. The organization is a military organization with a culture of fixed regulations and rather strict project approval and rules. This case study describes a successful measurement approach in a context that is challenging for agile methods. They report that the measurement system was developed based on a set of identified risks in the development process that the measurements were supposed to help control and mitigate.
Janus et al. (2012) present a framework for combining traditional QA measurement practices with agile development process. They had applied traditional code quality measurements and a suite of static analysis tools in agile context. They used Goal-Question-Metric method for designing and improving the measurement program.
Korhonen (2012) describes a case study of the effects of agile transformation on defect management practices in three large, distributed development organizations. Korhonen reports that the most important measurement in the three cases was number of open faults. The two differences in agile process in comparison to the waterfall measurements were measuring the open faults more frequently, i.e., in each sprint, and setting the target levels for open faults much lower. She concludes that in agile development context fault levels must be tracked frequently (at least in every sprint) and on product level. In addition, in one of the cases immature agile practices caused cumulative quality debt in terms of open faults that enforced to include a long fixing and stabilization period at the end of the release.
Ktata and Ghislain (2010) describe a case study of creating a measurement program for SCRUM teams in a single company. Goal-Question-Metric method was used for designing the measurement program. They concluded that in comparison to traditional teams, agile teams have different needs when trying to establish a measurement program. Agile people are more interested in value delivery, technical debt, and multiple aspects related to team dynamics. They also report that agile teams, too, are reluctant and resistant to change, will cooperate to collect the data as soon as there are proper tools that support the data collection.
Marschall (2007) reports case experiences of transforming a six month big-bang release cycle into continuous release process and increasing product quality. They found that the running-tested-features measurement and small, frequent releases into production were the key practices contributing their success.
Talby et. al (2006) report a case study of a large scale XP process. They describe their tailored XP practices from the viewpoint of testing and QA and describe and reason the key measurements in their process.
Kulas (2012) has summarized the literature on agile measurements in her Master’s thesis.
Table 2 Proposed agile measurements in empirical research
|Measured concept||Proposed measure||Description||Rationale||Source|
|Product size (amount of completed work)||Test points||Test point is one test step in an automatic acceptance testing scenario or one line of unit tests.||The advantage of test points is that the amount of acceptance tests for a given feature is usually proportional to the feature’s size and complexity. This cannot be said on the count of lines of code, or lines of specifications. Encourages writing tests, and anything partially implemented is not counted.||Dubinsky et al. 2005Talby et al. 2005
|Tests||Number of unit tests||Gives first insight to the product size.||Janus et al. 2012|
|Running tested features (RTF)||Number of user stories that have received customer sign-off and are ready to be released (or released)||Measure true progress in terms of release quality features. To encourage developers to finish the features instead of just implementing without testing and validation.||Marschall 2007|
|Test coverage||Percentage of source code covered by tests (statement and branch)||Measures how well the tests cover the new code.||Janus et al. 2012Marschall 2007|
|Test growth ratio||The ratio of number of tests (or covered code) vs. added amount of source code||Gives better view to test coverage than absolute test coverage in a context where large amount of legacy code does not have tests.||Janus et al. 2012|
|The development pace||CI Pulse||Number of check-in operations occur per day.||“Steady” pulse; good. “Spiky” pulse means that most check-ins are grouped at the end of iterations; people don’t integrate enough; bad.||Dubinsky et al. 2005|
|Version branch life-time||Number of days developer’s private branch has been un-merged with the trunk||If the developers work long times on a branch without merging the trunk into their code, it will lead to merging difficulties in the story release phase.||Marschall 2007|
|Fault cycle time (longevity)||The number of days between finding a fault and verification of the fix. For a project-level measure, the number of faults with the same defect cycle time was counted.||Short fixing time has several benefits: fast fixing requires less effort from developers, working on a clean and stable code base is more efficient, avoiding the overhead of prioritizing and planning defect fixes, avoiding unpleasant customer negotiations over which faults to fix.If there is large number of defects with a long cycle time, it could be a sign of possible problems with either correcting the faults or verifying the corrections.||Korhonen 2012Talby et al. 2005
|Progress against plans||Burn-down||Graph of the remaining work in days versus the remaining human resources in days.||Shows are we going to meet the goals of the release or iteration during the release. And if not, help to scope down early. Mitigating the risk of missing release goals due to the lack of a clear view of progress during the release.||Dubinsky et al. 2005|
|Quality||Faults||Number of found faults per iteration||Measures the achieved quality level. Estimating the needed fixing and rework effort in the forthcoming iteration(s).||Dubinsky et al. 2005|
|Open faults||Number of open faults at given points in time. An open fault is a problem which has been found in the software but not yet corrected, or the correction has not yet been verified.||Open faults graph can be used to visualize release readiness. If a large number of critical faults are not yet corrected, or if the number of open faults is increasing over time, the software is not ready for deliveries. The expected pattern is stabilizing and then decreasing to a minimal number of open faults before the release.||Korhonen 2012|
|Broken builds||Number of broken builds over time. Build break occurs if automated test fails or static code analysis reports a sever violation in the new code.||Each broken build indicates a quality problem. Some broken builds are to be expected if the CI practices work effectively.||Janus et al. 2012Marschall 2007|
|Violations from static code analysis tools||Number of low criticality violations from static code analysis tools.||Measures the change in the code quality.||Janus et al. 2012|
|Effort||Project level testing and fault repair effort||Effort in hours for maintaining and running regression tests and fixing faults||Project level statistics are stable enough to be useful for estimating the required (regression) testing and fault fixing effort in forthcoming iterations.||Talby et al. 2005|
Characteristics of measurements in agile context:
- The goal-based approach for creating the measurement program is common in agile context. Standard measurement-sets or approaches are not typically included in agile development process as such. Instead, the development organizations are designing and tailoring their measurements program based on their needs and identified problems or risks.
- In agile context the measurements focus on the customer value (e.g., in terms of running-tested-features) and the progress in not tracked in terms of the phase products or other artifacts.
- In most cases, the development hours are not tracked (at least not for the development management purposes). Instead, for steering the development work it is relevant how many hours (or days) of work is left and how much resources available for that work.
- The effort estimations and tracking is performed for the purpose of estimating the capability of completing the iteration (and release) according to the plan. If not, the measurements give early indication of insufficient velocity in order to scope down the iteration contents.
- Code quality and technical debt are important for developers and common quality measurements in agile contexts.
- Higher-level quality measurements are often simple, focusing on the remaining problems instead of trying to quantify the absolute level of quality.
- In agile context the trends in the measures over time tend to be more important than absolute target values. The aim is to be able to detect early if progress slows down or quality starts to deteriorate.
- In agile context the principle is to measure early and often and keep the target levels strict, in order to avoid accumulated quality problems (debt).
P.S: Drop us a message if you want to take a look at the referred sources.