Why so many data science projects fail to deliver

WTW’s Tim Rourke on why data science teams have traditionally underperformed in the insurance sector.

The insurance industry has historically been – and continues to be – highly data-led. Insurance professionals understand the value of data and the need for continual investment. As computing capability has expanded, the ability of data science to turn traditional insurance problems from descriptive, backwards-looking views to highly accurate, predictive insights has also advanced.

Today, insurers continue to be data-rich entities, gaining deeper insights captured more quickly than previously possible. For example, there has been fast-growing interest in using machine learning to improve claims operations via informed call routing decisions, or the ability to spot emerging problems early on and trigger the engagement of human intervention for remediation.

In the turbulent markets that the UK personal lines industry currently faces, data science can, when combined with experienced decision makers, deliver a compelling advantage to ride the “perfect storm” more effectively.

Yet, although insurers are increasingly using data to generate value, firms have so far done this with varying degrees of success. At the executive and senior leadership level, there is concern that significant investment in data science teams - and the technology infrastructure required to deploy these methods – is not delivering the practical, pragmatic business change or value they would like or expect.

The grace and favour once afforded to executives around data science as an “R&D” activity has passed, and the expectation of clear value from the investment is now being demanded. Close observation of the market has revealed five of the biggest drivers of underperforming data science teams:

Trading off accuracy and value creation

Insurers face potentially conflicting challenges between how data scientists have been trained to work and the actual needs of the business. Where model accuracy and predictiveness might be the ultimate focus for data scientists, many insurance leaders are keen to see swift and actionable insights that can result in material change and measurable value. They are also – within limits – more than prepared to compromise on predictiveness.

The trade-off between model predictiveness and value continues to be a well-socialised issue. How leadership balances both requirements is not an easy problem to solve and the time required to allow this challenge to find its natural equilibrium is not always palatable – or indeed practical or desirable.

A lack of technical challenge

This is a situation that occurs with leadership who have not previously used advanced analytics techniques in their earlier careers. For example, those who may have cut their teeth on GLMs and do not understand these new methods as deeply. Therefore, their ability to challenge model performance or outputs effectively is reduced. This can manifest as an inability to identify and therefore steer the team away from pitfalls and, as a result, the data science function failing to deliver sufficient commercial value. It can also present as a reluctance or slowness to apply these methods, due to fear or lack of understanding, that may impact future commerciality.

Naïvety

There is a certain level of naïvety in the approaches taken by data science teams, which stems from a lack of understanding of the very specific, niche problems faced by insurers. Model instability, for example, is where data science techniques are able to create an inherent variability (more so than with historical methods), which when deployed in an insurance context can lead to unintended and detrimental outcomes. What data scientists choose to model is sometimes misguided, so it is imperative that insurance specialists and data scientists work together, sharing common goals to achieve the best outcomes for their business.

Managing massive model real estate

For organisations that have great data, the opportunity to model is enticing, and with well-built models, the value is unquestionable. However, models need maintenance and attention, as neglect risks leading to poor insight and decision making. So, with a large model real estate, it is easy for data scientists to spend a disproportionate amount of time on being glorified handle-turners, rather than spending the time in generating material insights from models to create genuine business change and value.

Insufficient governance and control

Data science teams can lose sight of appropriate governance. It is critical to bring together data scientists and subject-matter experts to design systems that offer greater visibility of what models are doing, with more transparent governance that is sufficiently understood by the wider business and external stakeholders. The excuse of data science methods being opaque and uninterpretable is no longer an option, with the best having good control over the impact of their models.

The UK insurance market is seeing an explosion in the use of data science, with both winners and losers. Bad data science often involves clever people doing clever things with data, but too often failing to filter through the organisation to drive real change or generate commercial value. This results in poor return on investment, but more importantly acts as a weight around the ankles of data science teams, resulting in reduced productivity and attrition.

Insurers who are pulling ahead of the pack are the ones thinking about how they can create the structure and culture to empower data science teams to deliver value. They also have a strategy and clear vision around team structure, what to model, deployment and maintenance, as well as having the technical expertise to ensure the implementation is robust and real business value is unlocked from data science, targeted at solving meaningful problems. Those who are successful in navigating these challenges are seeing significant tangible returns.