De-risking the programme portfolio with reference class forecasting

Programmes that run over budget, behind schedule and don’t deliver the intended promises have many underlying factors that cause poor performance. Studies by various scholars have categorised poor performance to a category of causes such as psychological, technical and political.

These variety of underlying causes have sparked numerous studies to create solutions to these problems. Academics, including Bent Flyvbjerg (Professor of Major Programme Management at Oxford University's Saïd Business School), have found that basing forecasts on actual outcomes of implemented programmes gives an estimate devoid of the psychological issues - biases and strategic misrepresentation - that exist during the planning stages.

One technique that is being used in infrastructure programmes is Reference Class Forecasting (RCF). RCF, at its simplest, is a technique where one looks at a reference class of similar programmes to the one about to be embarked upon and, using this reference class, works out the statistical probability of where the forecast should reside depending on the risk appetite.

RCF provides an outside view of the likely contingency need of a specific project. The systematic approach using programme data creates a significantly more accurate estimate when planning programmes and as a result more realistic business cases.

The five-step approach to applying RCF

Applying the RCF in an organisation would comprise of a five-step approach, comprising the following:

1. Select a reference class.
2. Assess the distribution of outcomes.
3. Make an intuitive prediction of your project’s position in the distribution.
4. Assess the reliability of your prediction.
5. Correct the intuitive estimates – using the optimism uplift indicator.

However, before organisations can begin to use RCF, they need to decide which approach is needed to create the reference class(es), i.e. an organised completed programme database.

A few points that need to be thought through, but it’s by no means an exhaustive list:

• Organisations must first decide whether they want to embark on creating reference classes based on their own programme data, or to benchmark across the industry.
• Creating robust reference classes for the various types of programmes requires data.
• Data that can be compared easily to provide statistical outcomes.
• Most large organisations have a diverse set of programme data which is stored in more than one system. Do we need to understand the data flows across various systems?

Once the technical accessibility has been established, the next stage is to analyse the type of programmes that the organisation is running and the corresponding programme data that is available.

The key to creating a reference class is not only collecting cost data but to standardise, determine the granularity that is required and create a process that enables a constant flow of data into the reference class. Only then can the methodology of applying RCF be practised. Also, not forgetting the change management that must underpin the successful adoption of RCF.

My takeaway - if your programmes are not running at optimal levels, then consider a systematic approach to help reduce the causes of poor performance.

Pindy Bhullar has over 20 years’ experience working in the financial services industry with over 16 years at UBS, one of the largest wealth management banks in the world.

She is currently a Director working on Commercial Deals for UBS. She has experience of managing global technology implementations, transformational change, strategy and process based programmes. Pindy is passionate about programme management and has recently completed the MSc in Major Programme Management at Saïd Business School, University of Oxford and received a distinction in her thesis.

Merv Wyeth
APM Benefits Management SIG Co Chair