Delivering value from: benchmarking

by: Grant Rule

Benchmarks establish baseline measures of performance from which the results of improvement initiatives can be measured. They can be derived for Application Development & Enhancement projects, or for Maintenance & Support activities, enabling management and staff understand recent performance with respect to other organisations and competitors.

However, benchmarking is often seen as a costly and unproductive exercise which fails to achieve any meaningful outcomes.

Grant Rule looks at the issues which combine to give benchmarking IT a bad name, and how to address some of the problems in order to get the most value out of a benchmarking exercise.

Benchmarking has its place in the toolset for performance improvement – after all, if you don’t know where you’re starting from, how can you measure improvement achieved? It can also be useful to evaluate where your project performance sits in relation to industry standards. However, there are a number of reasons why benchmark reports often fail to deliver meaningful data which can be used to inform management planning and strategy.

Unreliable source data. Accurate and reliable measurement of the outputs from the application development & enhancement (AD&E), and/or support & maintenance (S&M) processes and projects is crucial. The quantity or size measurement of the outputs is a key component to calculations of:

• unit cost (size/total_expenditure);
• productivity (size/effort);
  
• velocity (size_delivered/time_elapsed) or cycle_time (elapsed_time/unit_size);
• product quality (#defects/unit_size)… also called ‘defect density’;
• capacity (size_delivered/calendar_month) for a group or department; etc.

This is why ‘size measurement’ is emphasised in the old CMM for Software, and the current CMMI for Development. However, SMS continues to find that organisations really use very poor measures of ‘the functionality delivered’ to customers and end-users… and even worse measures of the qualitative requirements imposed on the software. It amazes me that people are prepared to base critical business decisions, and to bet their careers and companies, on the poor, ambiguous data that is usually available.
SMS were recently called in by a client concerned by the strange benchmark results they had obtained. We conducted an FPA Audit of a set of IFPUG r.4.2 FP size measures and had to report that ‘not one FP count could be verified or validated’! Of course, the whole benchmark exercise which was based on this source data proved a waste of time and money. Unfortunately, this is not a unique situation.

Delayed reporting. There can be an inordinate delay between the Senior Executive’s wish for a benchmark and the publication of the report. SMS has observed that it will frequently take 6-9 months for a benchmark report to be delivered to the organisation which commissioned it. It seems pretty obvious to me that, in order to derive maximum value from a benchmark exercise, the client data used for comparisons needs to be as current as possible.

Backward-looking information. There is always a tendency for a benchmarking exercise to be backward-looking, analysing the work typical in the past. This does not necessarily reflect the nature of current activities and the work expected in future, or – critically - the process performance required to deliver current and future requirements.

Provides no analysis of process. A benchmarking exercise typically only analyses half the entire picture. It is a measure of past performance, but does not explain why the performance is as observed. While measuring performance is absolutely necessary, you also need to understand the process(es) that lead to that performance.

A benchmark, based on the old/existing processes, won’t help predict the performance of a radically new process… which is likely to be just what is needed!

Let’s look at each of these points in turn, and what can be done to mitigate the negative effects on the value of a benchmarking exercise.

Unreliable source data

The most precise, accurate and widely-applicable functional size measurement (FSM) method available is the award-winning, ISO standard COSMIC (ISO 19761) which recently passed its 9th birthday and now is in its 3rd release (ref: < www.cosmicon.com > ). It also happens to be easier and cheaper to collect. From the perspective of both management and the software practitioner this is a massive improvement over the older FSM methods, such as IFPUG and MkII Function Point Analysis (FPA). Although SMS supports all 3 Functional Size Methods (it’s the client’s choice) and these older methods are still widely used, we often find their implementation to be very poor.

We do find it somewhat incredible that there are still organisations who rely on simple counts of the number of lines of source code - as if ‘code’ is what the customer wishes to purchase! This is a measure of input, not a measure of output, and is unlikely to provide worthwhile benchmarking data.

Based on my 35 years experience in IT, I entirely agree with Compass’ own Ms. Geraldine Fox, when she says, “An independent third party is essential to executing an effective benchmarking initiative, in terms of defining the metrics and the comparative groups, normalizing the reference data, and ensuring a meaningful and relevant apples-to-apples comparison.”

Delayed reporting

The length of time taken to prepare a benchmarking report is largely determined by the length of time it takes to collect the data. This is why SMS devised the Fast Data Collection Study (FDCS), which exploits the specialist expertise of SMS consultants to enable us to complete data collection for a benchmark in 3-6 weeks rather than the typical 6-9 months.

Backward-looking information

As noted above, a benchmark is backward looking – even with fast data collection, the data may not be helpful for forward planning, especially if the benchmark comparison identifies an urgent need for change. It may tell you there’s a need to do something, but not what to do. Our own experience in working with the major benchmark providers (Compass, Gartner, SPR, CSC, IBM, Quantimetrics, etc) and indeed in providing benchmarks ourselves, is that a benchmark should contribute to an overall strategy based on an understanding of the sources of waste, and how the process needs to change.

SMS supports our clients in obtaining a rounded perspective of their service operations, advocating value stream mapping and process appraisals, combined with performance measurement and benchmarking. We also recommend applying the principles of lean product development and service delivery to identify the improvement opportunities – and of course we can mentor and assist management and creative, customer-facing staff during the implementation of their change to more mature, higher performance value streams. That’s our business.

Provides no analysis of process

Benchmarking an organisation’s process performance with respect to that achieved by others can identify the gross mismatches, and establish an understanding of the urgent need for change - the ‘burning platform’. However, it often proves very difficult for those involved to translate the ‘performance gaps’ observed into a meaningful Management Action Plan.

This provides the rationale for all those organisations that commission model-based appraisals, such as the CMMI-based SCAMPI, and those using ITIL, ISO9001, or CoBIT, etc. However, this is not necessarily the best approach. Applying a model “out of the box” will not deliver results unless the implementation is informed by an analysis of the succession of steps in the conception-to-delivery value stream. Process performance; technology used; the people involved; and the company culture must also be taken into account.

Sometimes small-scale incremental improvements (‘kaizen’) are sufficient, but often a radical re-organisation & re-alignment of the process steps (‘kaikaku’) is necessary.

It is important to apply systems thinking to the entire system and avoid local optimisation that is detrimental to delivering value to the customer.

Conclusion

For benchmarking to be valuable, it must contribute to an overall strategy for change and improvement. Deming had it right in ‘Out of the Crisis’ (MIT 1986):

• be of constant purpose – from change management control to the leadership of change;
• build in quality;
• minimise lifetime costs across the whole;
• continuously seek improvement;
• train on-the-job;
• promote cross-functional team-work;
• substitute leadership for numerical goals and slogans;
• engage everyone in the improvement of services. Everyone has a part to play, and service improvement is achieved bottom-up, rather than through exhortation and target-setting from the top down.

...........................................................................

Copyright © Software Measurement Services Ltd.