AI Blind Trust Could Trigger Corporate Failures

The AI boom could lead to major corporate failures as bosses trust answers they don’t understand. Anaplan co-founder, Michael Gould says businesses are repeating the mistakes made with complex spreadsheets on a far bigger and more dangerous scale. The rush to adopt artificial intelligence will lead to major corporate failures as business leaders increasingly make decisions based on forecasts, analysis and recommendations they do not understand, according to Anaplan co-founder Michael Gould.

The warning follows concerns raised by the Bank of England about growing risks surrounding the AI investment boom and whether the huge sums being invested in the technology will ultimately deliver the returns businesses and investors expect.

Gould, who has spent more than 40 years building financial planning and modelling systems and is now the founder of financial modelling software company Kaleidoscope.com, believes the greatest danger facing businesses is not that AI will produce obviously incorrect answers, but that executives will become increasingly confident in sophisticated outputs they are unable to question or explain.

Michael Gould said: “I think we will see major businesses fail because senior executives trusted answers produced by systems they did not properly understand. AI can produce an incredibly convincing forecast, report or recommendation in seconds. The danger comes when the person making the decision cannot explain the assumptions behind it, where the data came from or what happens when those assumptions turn out to be wrong.

“We are creating the conditions for businesses to make bad decisions faster, at greater scale and with more confidence than ever before.”

Gould is keen to stress that this is not an argument for standing still on AI. He believes ignoring the technology is not a realistic option for any business, and that those who adopt it well stand to gain a real and lasting competitive advantage, while those who fail to adopt it risk being left behind.

Gould said: “This is not a case for standing still. Businesses that adopt AI well will gain a genuine competitive advantage, and those that ignore it altogether risk being left behind. The potential benefits are real and substantial, but only for organisations that pair the technology with proper understanding and judgement. Used carelessly, the same technology becomes a serious liability rather than an asset.”

Gould believes companies are rushing to invest in artificial intelligence while ignoring longstanding weaknesses in the way they plan, forecast and make decisions.

Businesses now have access to more financial information, customer data, operational dashboards and analysis than at any point in history, yet many still rely on fragmented systems, conflicting forecasts and financial models understood by only a handful of employees.

Gould said: “Most businesses do not have a data problem, they have an understanding problem.

“Finance has one forecast, sales has another and operations is working from different assumptions. Management receives dozens of reports and dashboards and is expected to turn them into a coherent view of what the business should do next. Adding AI to that environment does not fix the problem. It simply gives companies the ability to produce more answers from the same weak foundations.”

Gould says the risks mirror problems businesses have faced with spreadsheets for decades, but AI could dramatically increase their scale.

Spreadsheets transformed financial planning by allowing businesses to build increasingly sophisticated models. However, as models became more complex, organisations often became dependent on a small number of employees who understood the formulas and assumptions behind them.

Errors became harder to identify, assumptions were buried inside thousands of cells and senior executives were asked to approve significant decisions based on models they could not properly interrogate.

Gould said: “We have already watched this happen with spreadsheets. Businesses built models that became so complicated that only one or two people understood how they worked. Senior management trusted the output because it looked precise.

“AI is the same problem with the accelerator pressed down. The technology is more powerful, the outputs are more convincing and the person making the decision can be even further removed from the assumptions producing the answer.”

According to Gould, one of the biggest mistakes businesses are making is treating artificial intelligence as a way to predict the future more accurately. He argues that no technology can reliably predict customer behaviour, economic shocks, competitor activity, changing costs or political decisions. Instead, businesses should use financial modelling and AI to understand how different events could affect their organisation and prepare for multiple possible outcomes.

Gould said: “Business leaders need to stop asking AI to tell them what is going to happen. The future is not a single number waiting to be calculated.

“The useful questions are: What happens if sales fall by 20 per cent? What if our biggest customer leaves? What if costs rise? How long does our cash last? Which assumptions could destroy the plan? AI should help executives explore uncertainty, not give them false confidence that uncertainty has disappeared.”

Gould believes the pressure on executives to demonstrate that their companies are adopting artificial intelligence is encouraging businesses to invest before understanding exactly what problems they are trying to solve.

He said: “Nobody wants to be the CEO who admits they do not have an AI strategy. So businesses are buying tools, hiring teams and adding AI features without first asking whether they understand their own organisation well enough to use the technology properly.

“The winners from AI will not necessarily be the companies spending the most money or moving the fastest. They will be the businesses that understand what drives their performance, which assumptions matter and how changes in one part of the organisation affect everything else.”

Gould believes the debate about whether the enormous sums flowing into artificial intelligence will ultimately produce sufficient returns is distracting attention from the risks created by the technology inside individual businesses.

He added: “The Bank of England is asking whether investors and businesses have become too confident about the financial returns AI will deliver. Business leaders should ask themselves a similar question about the decisions they are making.

“If an AI system recommends investing millions of pounds, closing a division, changing prices or cutting jobs, can the executives approving that decision explain why? If the answer is no, we have a much bigger problem than an AI investment bubble.”