What people miss about AI and ROI

NANDA – an MIT unit building agentic AI infrastructure – shocked markets recently with eye-popping claims that 95% of businesses were failing to realize returns from Gen AI. The study captures a hard reality: AI adoption is difficult work.

But there was a flaw in it as well. What was spun as a failure is in reality a different f-word: a feature of this technological transformation. AI may pay off or not, but only assessing financial returns and using just a six-month time period, as MIT did, is hardly enough to judge.

For banks, in our view, ROI from AI investments will take years. They’re still getting their heads around the systems and controls needed to move AI into production, what productivity gains are actually worth and how to tie compensation to AI goals. As that happens, the vast majority of gains come through in other ways that MIT ignores, like costs avoided or time saved (see: “3 ways to measure AI returns,” The Brief, June 12).

Take the most recent earnings round: Bank CEOs are using lower payrolls to prove AI efficacy (see: “One trick banker,” The Brief, July 24). They’re pointing to time they’ve gotten back, like CIBC’s report of 600,000 hours saved by its Gen AI platform. Or they’re referencing adoption milestones – like Bank of America eclipsing three billion client interactions with AI assistant Erica – to keep investors patient as they wait, presumably not like Godot, for ROI to turn up.

What the MIT report gets right is the need for accountability on AI investment – especially when the metrics banks are using aren’t as iron-clad as dollars and cents. Headcount reductions aren’t always seamless. CommBank found that out the hard way last month when it backed off plans to replace 45 customer support workers with AI after its tool didn’t handle complaints as expected. Adoption rates aren’t always reported consistently; some banks look at daily users, others monthly. And the formula for hours saved tends to vary by firm and by team.

Bottom line: Before the infrastructure to scale AI is finished, judging ROI by financial uplift is myopic. But but but … if banks want investors to hang around until those revenue gains actually materialize in 2026, 2027 or beyond – some smart folks we know say you’re really looking at 2030 – they need to be clearer that the metrics they can report will enable financial returns in the future.

Use cases featuring agentic AI may still be few and far between (see: “Why agentic is so hard,” The Brief, June 26). Research into it certainly isn’t, new data from Evident shows.

Of the more than 300 papers banks have published in 2025, more than 6% focus on how to give AI more autonomy – the fifth-highest share. Banks aren’t just giving lip service to the much-hyped topic; they’re putting real resources into letting their most bookish employees try to crack it.

JPMorganChase published a paper that defines agentic AI, describes the technical and governance needs to make it work and explores how several applications of the tech were created. Intesa Sanpaolo’s highly-cited paper details a better way to understand why agentic systems fail. Capital One’s contribution shows how the bank built a dataset of realistic conversations and step-by-step tasks that agents might be asked to handle – something it can turn to when it needs to understand why new tools act the way they do or how to improve them.

Why it matters: Our data shows banks are moving away from theoretical topics – like scientific discovery or health research, two popular areas in 2024 – in favor of AI research that relates clearly to business applications. This work on agentic AI shows banks see it as being in that camp.

"Humans will not be operating the banking system in 30 years. Banks will be AI companies in 15 to 20 years."

- Brett King, futurist, at Bankers’ Meet 2025 in Dhaka, August 14

We sat down yesterday with Joe Bonanno, global head of data, analytics and innovation at Citi Wealth, to give us the backstory on how his team rolled out a new AI tool to the bank’s wealth management team in just six months.

Use Case: Advisor Insights
Line of Business: Wealth management
Vendor: Google
Bank: Citi

Why it’s interesting: The bank’s AI tool helps advisors improve client interactions by creating what Bonanno described as an “optimal engagement model.” Instead of relying on call cycles (pre-scheduled check-ins), advisors get lists of clients to contact on a given day based on information that might make them more likely to respond – personal information (like birthdays or anniversaries), market movements that relate to their portfolios or news in the region they live.

How it works: The tool creates a pre-populated, personalized email for the clients determined to be most in need of a check-in from the bank. The advisor then reviews it and is free to add any personal touches. It’s “the fact that you’re constantly dripping on your clients with interesting, relevant things,” he said. “If you're not talking to me, the odds are I'm probably not going to do more business with you.”

How they did it: The bank took the tool from an idea to production in six months. Citi organizes itself in agile squads, something Bonanno insisted upon when he joined the wealth unit in 2024. One squad did the “sausage making” of preparing the data, one handled the campaign distribution and one built the UX and the interface. But the secret sauce to the quick rollout was the bank’s unified data store: “Think of every trade, transaction, position, balance, email, click, phone, log, web log in an organized way, in a central database repository,” Bonanno said. “The fact that I have that means I can fly.” It doesn’t hurt to have a close relationship with Google to harness best practices either, he quipped: “We probably go to Google’s offices twice a week.”

By the numbers: The bank judges success by looking at how much more likely a client is to make a transaction after one of the notes the system prompts advisors to send. “We want to see a combination of steps,” Bonanno said. “Did they reply or call back within a [five-day] window? Did they transact?”

What comes next: The bank is “working on an LLM version” of the tool that can power “the generation of the ideas, the measurement and the feedback loop,” Bonanno said. The future of Advisor Insights is marrying it up with AskWealth, the ChatGPT-like tool the bank also rolled out last week. The two tools together will allow advisors to ask questions about clients and the bank’s research at the same time so interactions can get even more personalized.

In this section, we translate AI vernacular (and propose our own).

Last month, Google reported that a single Gemini query consumes about five drops of water. Mistral this summer said creating a page of text uses about as much water as you’d need to grow a small radish.

We present “Radishmetrics” – our in-jest term for the emerging vernacular of how AI labs are talking about how much energy their tools use. The metrics may seem crude-ités, but sustainability data about AI is becoming more detailed, and banks are paying attention.

Lloyds’ head of AI and advanced analytics Rohit Dhawan last week said the bank now assesses “energy use and data center efficiency” for its major AI projects. Earlier this year, Paul Dongha, head of responsible AI at NatWest, noted that banks might need to think about how and when to use AI in order to keep emissions under control in the future.

Banks are lagging other parts of the financial industry on Gen AI adoption, a new report on the post-trade industry from Citi says. The survey of 537 securities industry leaders showed bankers were less likely than asset management or institutional investing peers to have AI projects running in areas like KYC or sales and product distribution.

Trouble in paRAGdise? A new paper from DeepMind says vector embeddings – the way RAG systems usually identify which documents to search in order to return the right answer to a question – have clear limitations. The researchers found that systems will ignore some of your data because of how they determine what’s relevant to a particular question. The upshot? Combining vector embeddings with other methods of searching can address weak spots, the researchers said.

MUFG continues to strike partnerships with homegrown AI companies. Japan’s biggest bank is taking a 5% stake in LayerX, betting that it can deploy 60 use cases in contracts, invoices and sales processes through the partnership and save 200,000 work hours per year. It follows news from May that MUFG was partnering with Sakana AI.

Santander hired Arantxa Sarasola as its new head of data and AI for Spain, reporting to the bank’s chief data and AI officer Ricardo Martín Manjón and Ignacio Juliá, CEO of Santander Spain. Like Juliá, Sarasola joins from ING. It’s part of a flurry of recent moves made by the Spanish bank to take on rival BBVA – the current AI leader in Spain in the Evident AI Index.

Ryan Duffy is joining BMO as its U.S. chief data and analytics officer and head of data commercialization. Duffy spent a decade at EY, leading financial services data management and data governance.

David Solganik joined Raymond James as its head of AI strategy. Solganik was previously head of AI and data strategic initiatives for RBC’s U.S. wealth management unit and served as the head of decision sciences at Morgan Stanley before that.

Danske Bank is hiring a new head of Gen AI architecture and transformation to “lead the technical evolution of enterprise-wide Gen AI platforms.” It follows the bank’s appointment of Kasper Tjørntved Davidsen as chief AI officer in June.

State Street is looking for a new chief data and AI officer, a move to centralize the bank’s strategy under “a single executive level leader responsible for unlocking the full value of data and AI across the enterprise.”

Chandra Kapireddy, firmwide head of Gen AI, ML and analytics at Truist, is stepping away from the bank, he wrote on LinkedIn. He’d been at the bank since July 2024.

New research from BlackRock could pave the way for AI agents to become your next portfolio manager. For now though, the agents’ performance suggests investment manager jobs are safe.

The firm built a multi-agent system that mimics a team of equity researchers: A “fundamental agent” parses 10-Ks, a “sentiment agent” reads news and a “valuation agent” crunches historical prices. Then they “debate” the merits of their findings with one another until they find a consensus for a recommendation.

The firm randomly chose 15 tech stocks to hold over four months that served as a baseline portfolio. It then tested to see if the agents could outperform them employing two different investment strategies on those stocks: one where the agents were risk-averse and one where they were neutral.

Results were mixed: When the agent teams played it more cautiously, they underperformed the baseline portfolio. When they were risk-neutral, they outperformed it. More testing is needed before fund managers have to start worrying about updating their resumes.

Tues 9 - Weds 10 Sept

Artificial Intelligence in Financial Services Conference, London

Mon 22 - Weds 24 Sept

Responsible AI Summit, London

Mon 29 - Thurs 2 Oct

Sibos, Munich