Performance Assessment

by: Mick Burn-Murdoch

To compare project performance, identify improvement opportunities and make good estimates for future projects, you need to assess the performance of existing projects.

Our Performance Assessment service will assess the performance of a project, immediately upon completion.

Several SMS clients are using Performance Assessment to track the effectiveness and efficiency of their processes with respect to past projects and as a means of providing the fast feedback cycles needed to encourage process improvement.

The basis for this service is facilitated data collection and analysis using product size, effort, duration and defect information. The analysis is performed using simple statistical techniques such as regression analysis.

A Complement to Benchmarking

Benchmarks are conducted periodically, with a typical frequency of between 12 and 24 months. Preparing for, conducting and analysing the results takes time. In our experience, there may be 6 months or more between the decision to benchmark and the delivery of the report. Projects may have been completed for well over a year before results are available and feedback given.

Performance Assessment is complementary to any existing benchmarking techniques which are in use. It allows people to assess the performance of their processes on completion of each project. Benchmarks are still needed so an organisation can check itself with respect to competitors at home and abroad. Whereas a benchmark may contribute to selection of an improvement strategy, Performance Assessment contributes to tactical `in-flight', continuous process improvement activities.

Using the Technique

The typical output from the technique is a set of models.

A model is created for each specific group of applications. The results give a standard profile against which subsequent projects can be measured. Examination of the data to obtain the best possible correlation will show which group of applications demonstrate the most effective and efficient processes. Each group of applications needs to be uniquely identified by an appropriate set of characteristics.

For each group of applications a `standard' performance will be calculated. The standard consists of a formula that can be used to calculate the expected performance of a project. The formula for `y' (the effort or duration) gives the figures for calculating a project's ideal budget. If a project's actual results fall below the line, its processes are more efficient than those to be expected based on previous projects. If the project's actuals are above the line, its processes are less efficient. In either case, investigation will allow for lessons to be learned and process improvements made.

When considering the relative performance of a project remember that the degree of correlation between data and formula means it is unlikely that projects will match the formula precisely. Only projects outside a band of, say, ± 1 standard deviation or ± 10% would be considered as `exceptions'.

The derived formula for `y' will give a line on an X-Y graph. This does not have to be straight; in most cases it will not be. In some cases it will give what is, at first sight, a complex mathematical formula. If you do not feel happy calculating Natural Logarithms in your head it is perfectly possible to use a spreadsheet to resolve the formula.

Actuarial Systems COBOL

The formula for R-squared shows the degree of correlation between data from previous projects and the formula for the standard. The possible range is from 0 to 1, with 1 being a perfect fit. It is recommended that the data in the model be examined to achieve a correlation of at least 0.75.

Care needs to be taken when using the models, so that the implications are properly understood. Data collection should be as accurate as possible. If the data collection is not sufficiently accurate it will be very difficult to achieve correlations as reliable as 0.75.

As new projects are completed, the models can be updated to reflect the latest results. Again, care needs to be taken that only valid data is added to models otherwise correlations will be unnecessarily distorted.

Benefits

Organisations using Performance Assessment benefit from the fast feedback of performance information. This highlights processes that are tending to become inefficient and prompts further analysis of the root causes, facilitating control of change (e.g. introduction of new technology or techniques) and `tuning' of familiar practices.

Additionally, the data collected for a Performance Assessment can be used for estimating, to predict the cost and duration of future projects. One SMS client has achieved an estimation accuracy of ± 9% for all projects estimated over a 12 month period.