Analytics versus experience: What’s more critical for the portfolio director?
Portfolio Directors' Dinner Series February 2023

Insights from the practitioners

Barely a day goes by without a new article or thought-piece about the power of analytics and data. But equally, people are increasingly recognising the power of experience, or ‘The Master Builder’, to deliver big, complex projects time and again. Common sense would imply that the two concepts are polar opposites, so this seemed like an ideal topic to discuss with our Portfolio Directors for the third APM Portfolio Directors’ Dinner. The subject was, therefore, Analytics versus Experience: What’s More Critical for the Portfolio Director?

We explored the topic with two key questions:

The clear goal was to explore what enables strong portfolio management – clever use of modern analytical techniques, or deep wisdom and experience. And for good measure we finished off by asking what heuristics or “rules of thumb” our portfolio directors had learned from experience that might be useful to our audience.

The evening began with a discussion of whether an analytical mindset was more critical to the portfolio or the programme manager – basically, whether the portfolio view required more quantitative capability than the pure delivery view.

The initial reaction was that the portfolio manager’s role was the more analytical of the two because, by definition, the portfolio manager needed to scan a wider landscape of change and business as usual than the programme manager. 

However, the conversation then expanded into two interesting areas.

First, we talked about the subjective and very people-focused nature of the portfolio manager’s role. People generally agreed that, at the portfolio level, most of the work depended on managing the relationships between the people who defined the strategy at the macro level, both with each other, and with the transformational elements of the business. Strategy development is often about balancing subjective opinions of “what good looks like”, and rarely about presenting a simple algorithm or analytical breakdown to tell the board what their strategy should be. As such, although presentation of the data in a clear and analytical fashion to help decision making was critical, understanding and working within that stakeholder environment was equally crucial. So analytics was important, but needed to be tied to excellent stakeholder engagement and “emotional intelligence”.

Second, the split between the “strategic” view of the portfolio manager and the more “tactical” view of the programme manager regarding data drove an interesting discussion. Participants generally accepted that the portfolio manager needs to be in control of their data (collection, analysis and presentation), because of the size of the “change estate” the portfolio manager is responsible for. However, some people pointed out that the programme manager might not have as large a change estate, but would most likely need far more detail about the change they did preside over, to manage the complexities and nuances of delivering change in a complex, modern business environment. We concluded that the programme manager might not require the same breadth of data as a portfolio manager, but they would probably need much more depth of data in the space they were operating in.

This in turn drove a fascinating discussion about whether, with advances in modern analytics, both the portfolio and programme manager could “have their cake and eat it”. Could modern data capture, automation and analytics tools give them both the breadth and depth of data analytics across large change estates that would enable them to make better and faster decisions in relation to their daily activities?

The main requirements driving the portfolio manager’s analytical mindset are:

Analytics are therefore critical to allowing the good portfolio manager to understand what is going on across the large portfolio and support strong decision making with good data – a key element of building trust with the senior stakeholder community. But a) this is useless without excellent stakeholder skills that enable the portfolio manager to navigate the strong personalities and subjective nature of strategic alignment, and b) the nature of the analytics approach favours rapid analysis of a wide breadth of data rather than detailed analysis of specific depth of data – although, with improvements in data analytics techniques, might the advanced portfolio manager be able to both have their cake and eat it in a data sense?

Everyone accepted that strong data and analytics skills were critical for the modern portfolio manager, but there was some debate about the place in the hierarchy of analytics versus experience.

Historical data and the portfolio

Many people around the table had had positive experiences of using historical data tools, such as reference class forecasting or benchmarking, to help make decisions. Indeed, these were the most common data analytics methods used by the participants, in contrast to more “cutting edge” techniques, such as predictive analytics algorithms. This was, however, tempered by a number of stories about the limitations of analytics. “We use historical data and artificial intelligence (AI) to give us higher confidence and predictability when t-shirt sizing” was a typical example, showing that the analytics tool was used as a very clear aid to human decision making, rather than as a replacement for it.

At the other end of the spectrum, there were a number of stories about where analytics had gone wrong and led to fundamentally unsound decisions and, worse, a loss of confidence in the data and data analytical capability of the Portfolio Management Office. “We were looking to solve a problem that wasn’t really an issue. The data wasn’t good enough and it didn’t work.”

Another key area of concern with using historical data to predict future situations and trends was that, almost by definition, the portfolio space is strategic, subjective and very, very changeable – changed both by internal and external factors. Historical data tells you an awful lot about activities in a specific environment and time – both of those things will have changed, so your interpretation of historical data must be sensitive to this fact.

Data and the whole organisation

Some participants made a particularly interesting observation about how the whole organisation engaged with data and data analytics. In essence, this was that discussing data as an aspect of portfolio or programme management misses the point – if an organisation uses data as an asset, understands the importance of data quality, routinely captures, cleans, and then uses that data to make well-informed and rapid decisions, that organisation will be in a strong position to use data analytics to inform its programme and portfolio management. Where that’s not the case – they will struggle.

This would imply that, rather than simply trying to improve data in isolation around the portfolio board, if a portfolio manager wishes to use data as part of their arsenal, they should look to assess and improve their organisation’s cultural approach to data as an asset rather than as a blocker. To quote one of the participants: “Data should flow throughout the organisation as a single source of truth,” – therefore everyone should have an analytical mindset, with the varying levels of management focusing on different aspects of what the data is saying.

Data, trust and the portfolio manager

A final point that flowed through the discussion was about data and trust. This went deeper than whether we should trust the raw data collected by the portfolio manager (i.e. a data quality issue) – it was more about whether the currently available analytics tools were powerful enough to give insights and decisions that could be acted upon. A point the group kept coming back to was that in the portfolio office, “trust is massive; you have to be seen as an honest broker”. Given that the trust placed in the portfolio manager is so critical – and that a loss of trust can be deadly to the ability of the portfolio manager to influence – can we take the data and insights of the currently available analytics tools at face value? The overwhelming opinion here was “no”, for two primary reasons:

1. It is the responsibility of the portfolio manager to layer insight and context over the data. The analytics tool can help direct the manager’s attention and inform that context, but ultimately it needs to be the portfolio manager teasing out the insight.
   
   
2. The very subjective nature of the portfolio manager’s role means that analytics tools can only take the insight so far – they can inform, but not decide.

At the end of the debate, however, there was a general sense that making the discussion a battle between analytics versus experience was far too much of an oversimplification of reality in the portfolio office. In short, it’s not a binary issue. We need both data and experience – with the particular balance being influenced by the specific cultural environment (i.e. the data culture maturity, the level of trust, the scale of the portfolio etc.). And once again the debate came back to trust. If, as a portfolio manager, you are fully trusted, you will be able to gather context with the data tools and pull out insights with experience that will allow you to make observations and recommendations about the way ahead, not accusations about what has gone before. “The skill is to tease out the right conversations.” Ultimately, it is the role of the portfolio manager to triangulate the data and the stakeholders in order to drive fast, accurate decision making.

To finish off the dinner, we asked our portfolio directors for their favourite heurisms or “rules of thumb” that they’d used to drive successful transformation across their organisations:

• Stop over-validating after you’ve decided. It’s about having just enough information to make the decision. We’ve got so much data we feel we should use it, so we can end up navel-gazing.
• All change is about people. Data is nothing without a human element.
• If it sounds too good to be true, it probably is. Be careful of the watermelon status: green on the outside, red on the inside!
• Hire portfolio managers for attitude. The role is very different in different organisations and so attitude, ability to cope with ambiguity and the ability to swing between the tactical and the strategic is key.
• Be comfortable with someone being annoyed with you all the time – “We’re the deliverers of bad news”. You can’t please everyone; someone will always be disappointed or annoyed.
• Be confident in your own convictions – and you will need to be resilient and stand by what you say, as there will always be challenge if you’re doing the job right!
• Be someone with project delivery discipline but not project delivery language, and
• Caveat it every single time!

We hope you’ve found these insights on data, experience and the portfolio manager of interest.

As the APM Portfolio Specific Interest Group (SIG), we work hard to both share and promote good practice in the Portfolio space, and we couldn’t do this without the help and support of a range of experienced portfolio professionals who give freely of their time to help us with conferences, webinars and events like this.

We’d particularly like to thank the people listed below, who gave up their evening to make the portfolio directors’ dinner such a successful event:

• Oliver Fisher, HM Revenue & Customs
• Simon Harwood, Fidelity International
• Karina Singh, IPA
• Stella Zuegel, Kubrick Group
• Mike Chavez, Environment Agency
• Scott Sellars, NATS
• Richard Moor, Royal Mail Group
• Petula Allison, Yorkshire Building Society
• Adam Skinner, Valcon
• Andrew Goodman, Department of Work & Pensions
• Kessia Willoughby, Estee Lauder
• David Gifford (rich picture artist), Inscript Design
• Steve Clark, Valcon
• Rich Charman, Serco
• James Stokes, Elexon
• Michelle Downs, Vanquis
• Nicky Walsh, NFU Mutual
• Paul Morgan, Mars
• Dave Corbin, Gleeds

And, of course, APM, which supports and funds so much of our activity as a SIG. We hold these dinners annually. If you are a portfolio director or executive and would like to be involved, please contact us. We’d also be happy to hear from you if you are interested in speaking at one of our conferences or a webinar, or if you would just like to be more involved with our SIG. Please get in touch.

APM Portfolio Management Specific Interest Group

The Portfolio Management SIG ensures organisations invest in the ‘right’ projects/programmes to support their strategic objectives.