What is the difference between a trial and a pilot?

Introduction

Pilot projects and trials are a good way to reduce risk on projects that don’t fit into a life cycle well understood by the implementing organisation.

The definition of pilot and trial is often not clear and the purpose of this web briefing is to propose a standard meaning and offer some guidance regarding best practice for both activities.

Definition: Pilot

The pilot project is an initial small-scale implementation that is used to prove the viability of a project idea. This could involve either the exploration of a novel new approach or idea or the application of a standard approach recommended by outside parties but which is new to the organisation. An example of this would be the standard implementation approach for a new off-the-shelf package.

The pilot project enables an organisation to manage the risk of a new idea and identify any deficiencies before substantial resources are committed.

Definition: Trial

The trial forms part of a committed project and is intended to manage the risk of implementation roll out. It tests the implementation approach and its purpose is to manage the risk of implementation.

Basis for success

As with all project activities success is founded on clarity of objectives and outcomes, clear communication with the team and thorough expectation setting with stakeholders. 

Carefully written terms of reference and governance structures are key to success.

The use of pilot and trial stages can - if managed properly - significantly improve the chances of success. Many of the causes of major programme failure as identified by Richard Bacon in ’Conundrum – Why every government get things wrong – and what we can do about it’ can be addressed to some extent at least by the use of pilot and trial activities:

• Ill-defined roles and responsibilities – clear terms of reference and objectives created for and confirmed during pilot.
• Corporate amnesia with lessons not being learnt or applied – both pilot and trial activities have a strong focus of lessons learned.
• Skills / experience gaps – opportunity to identify and close these gaps before significant resources have been expended.
• Woeful user / stakeholder engagement – both pilot and trial activities require positive stakeholder engagement and also offer the opportunity to test wider stakeholder management plans.
• Poor financial management – small scale initial activities offer the opportunity for greater control over finances
• Sloppy procurement – again starting with a small scale pilot offers the opportunity to keep buying under control.
• Inadequate / inappropriate risk management – use of pilot and trial stages in a large project is a key risk mitigation tactic.
• Misunderstanding the cultural challenges. A well run pilot scheme can tease out many potential cultural issues and enable them to be addressed in the project plan.

The pilot

With a pilot study there is generally a high degree of uncertainty and there needs to be recognition on the part of both the project team and the governing stakeholders that the purpose of the pilot stage is primarily to prove viability, not deliver an agreed outcome.

There is a clear need for a control structure that enables prompt cancellation but also allows for the potential for radical changes in scope and direction if required. The intention is to create a controllable budget for people and supplies that will enable the project team to confirm that the underlying idea is sound.

The pilot study will confirm viability and scalability and enable proposed processes and procedures to be tested. It will confirm the appropriateness and safety of any tools proposed and also confirms that any working practices are safe and comply with organisational/statutory standards. It also enables the benefits to be tested and a more reliable investment appraisal to be created for the main project.

Typically deliverables would include:

• products required for project,
• robust lessons learned, risks and issues logs,
• benefits assessment,
• viability report and recommendation,
• route map for implementation,
• revised investment appraisal and project plan.

Note that most of these deliverables are essentially commentary on the viability of the proposed approach and not the actual outcomes produced by the approach.

The trial

A trial is a small-scale implementation planned in before the main rollout of a project.

It enables the project team to test logistics, communications, stakeholder management plans and the effectiveness of any rollout tools. It can also enable deficiencies in project team briefing, training and skills to be ironed out before the main rollout begins.

There is a clear need for a control structure that enables prompt cancellation but also allows for the potential for changes in approach, scope and direction if required. The intention is to create a controllable budget for people and supplies that will enable the project team to confirm that the underlying plan for rollout is sound.

The trial enables a more accurate budget and plan to be produced for the main roll out and as with the pilot stage offers an opportunity for the benefits to be revisited in the light of practical rollout experience.

A trial does have some down sides, care needs to be taken to ensure that scalability is proven during the process and prolonged running with only the trial group can put business as usual activities under a great deal of strain.

Typically, deliverables would include:

• products required for project: systems, processes, procedures, communications plans, training plans etc,
• interfaces between trial and non-trial business areas,
• benefits assessment,
• lessons learned,
• risks and issues,
• revised full roll-out route-map and plan,
• revised budget and investment appraisal,
• evaluation of equipment and tools.

Again, as with the pilot activity, many of the deliverables are commentary on the success of the trial not just the outputs required for a successful trial.

Governance

There can be a tendency on the part Senior Responsible Owners (SRO) and other senior governing stakeholders to see pilot and trial activities as a quick and cheap way to get their objectives met. 

This needs to be countered by careful briefing, the agreement of clear objectives and the introduction of a robust governance structure. 

A gate-based structure may be appropriate but whether or not formal gates are adopted a regular formal forum at which necessary changes to scope or approach can the addressed and either approved or rejected is critically important. The active involvement of the SRO in these sessions is vital as is a willingness to cancel or make radical changes if necessary.

If the SRO is more accustomed to holding project managers to account for delivery than being actively involved in the project process there may be a need for the project manager to spend more time with them outside of the formal sessions. To get the most out of pilot and trial activities the SRO really needs to be very close to detail of what is going on.

It may be that adopting a formal behavioural model such as SCARF - Status, Certainty, Autonomy, Relatedness, and Fairness - will help in drawing the SRO and project team closer together. Given the critical importance of the SROs active involvement in the process, time spent at the beginning of the process addressing any engagement issues will be time well spent.

As with all project activities, lessons learned and risks should be captured on an on-going basis and fed into the decision making process. The governance approach needs to be lesson and risk centric and offer the opportunity to respond flexibly and quickly to new discoveries. 

For pilot studies, governance needs to reflect the uncertainty of scope and regularly address the question of continuation and direction. At the end of the study, a clear way forward (which can include not going ahead) needs to be formally identified. 

For trials, governance needs to focus on the effectiveness of the plan being trialled, the risks associated with scalability and the complexities of the interface between the ‘trial user group’ and the rest of the business running with the existing business model.

Conclusion

The use of the pilot study can significantly reduce the risks associated with adopting novel or ‘new to organisation’ approaches to projects. It also enables organisations to get a firmer idea of the costs and benefits of a project before committing.

Trial implementations offer the opportunity to reduce the risk of rollout failure by enabling all aspects of the roll-out to be tested on a small and well defined user community before committing to the full implementation. As such they should form part of the risk-mitigation strategy for any sizeable rollout activity.

Attributions

Martin Acari, David Bewick, Robert Blakemore, John Bryden, Jim Dale, Richard E. Renshaw, Darren Jaundrill, David John Fairhall, Andy Nichols, Colin Parker, Adrian Pyne, Andrew Rickerd, Alastair Smart, Clare Young.

Web briefings

Web briefings are developed from the APM Community, an online community of project professionals. They result from discussions and questions asked within the community, the content is developed from users’ responses and edited by us. 

Web briefings are constantly evolving within the online community and are intended as a guide to issues within the profession.

APM Body of Knowledge 7^th edition

The APM Body of Knowledge 7^th edition is a foundational resource providing the concepts, functions and activities that make up professional project management. It reflects the developing profession, recognising project-based working at all levels, and across all sectors for influencers, decision makers, project professionals and their teams.