Model dating game

By declaring Anthropic a “supply chain risk,” and limiting how its models can be used in government systems, Washington just reminded banks why the AI model bachelor life is the right choice.

The Claude-maker has spent the last year courting – and winning the business of – lenders (see: “Bye, bye OpenAI” The Brief, Jan. 29). Should the Trump Administration’s label stick (and Anthropic sued the government this week to remove it), banks may have to unwind Anthropic use in some corners of their business. The bigger risk here for banks is overreliance. Anthropic’s situation feels unique, but we’re so early in the AI race that architecting around one provider, no matter how good their model, could leave them scrambling to unwind their systems if that vendor runs into trouble, political or otherwise.

Banks are racing to be “model agnostic” and building systems that can swap models in and out to combat that. Roughly half of the firms we track now work with more than one tech partner on their public use cases, new data from the Evident Use Case Tracker shows, more than double last year. Those that aren’t don’t just risk the higher costs of vendor lock-in; they risk needing to rebuild whole systems if a vendor becomes unusable.

The flexibility matters because no model is best – or most cost-effective – for everything a bank does. “Right now, GS AI is not just based on a specific model,” Goldman Sachs CIO Marco Argenti said last year of the bank’s firmwide AI productivity tool. “It actually gives you the choice of the latest and greatest model.” That means the bank can use the right model for the right job.

Actually doing that at scale though is an engineering challenge. “The hard problem was never ‘can we access a foundation model;’ it's everything that sits around the model,” Biswa Sengupta, chief AI technologist for LLM Suite at JPMorganChase, told us last month (see: “Show me the money,” The Brief, Feb. 27). The hard part is developing “secure orchestration across heterogeneous LLMs” – in other words making sure that tools can run several models side-by-side.

Banks are working on that, RBC CEO Dave McKay told investors this week: “We’ve built models, we’ve learned how to train models. Our data’s in great shape,” he said. “Retraining those LLMs on the AI side has gone faster…and largely on the timelines and the cost structure we thought.”

Which models get retrained and plugged in – and which labs make them – may not be the same in a year. But banks need to make sure that even if a model-maker loses, they don’t get brought down with it.

The pace of LLM development is faster than ever and harder to parse. In this series, we guide you through how to decipher the ever-flowing announcements one piece at a time. This week: benchmarks. See also our first entry on costs.

OpenAI announced four new products this past week, including GPT-5.4, the new version of its flagship model. To figure out how much of an upgrade it is, you may well need a Ph.D.

There’s no shortage of information: The company, by our count, shared more than 150 scores to illustrate how the model performs on different benchmarks. All that data should make the answer clearer. It often does the opposite.

Benchmarks today are hyper-specific and spiky: A model might show a 25% improvement on a niche test for agentic computer use. That doesn’t mean it’s 25% better at the kind of work a bank actually cares about, like extracting covenants from credit agreements or drafting internal memos.

Even when benchmarks cover tasks banks care about, the results are hard to trust. OpenAI said its new ChatGPT for Excel beat Claude on investment banking tasks. It claimed the gold using a benchmark the firm itself designed – and hasn’t released the inner workings of. On top of that, providers curate their scorecards: Each release highlights a handpicked mix of benchmarks, often without clear head-to-head comparisons. OpenAI’s post, for example, largely omitted direct match-ups with the latest models from Anthropic and Google.

Most industries don’t have this problem. Internet providers track dozens of network metrics, but customers look at download speed to judge quality. Car companies publish pages of specs, yet horsepower or fuel economy tell buyers the skinny. AI doesn’t have that yardstick yet. Artificial Analysis and LiveBench have standardized leaderboards, but they still only capture some of what models actually do.

Banks are taking this into their own hands. More than one-quarter of the papers published at December’s NeurIPS conference – the year’s biggest AI academic gathering – focused on how to evaluate model performance (see: “Paper trail,” The Brief, Dec. 4). BMO, meanwhile, has a whole Applied AI Evaluations team that it’s currently hiring into. One day there may be a common language for vendors. For now, whether a model is actually better is something banks need to prove for themselves.

The size of Bank of America’s “AI catalyst” team, a group the bank put together to jumpstart AI usage at the bank, the bank’s co-president Dean Athanasia said at the RBC Capital Markets investor event this week. “They represent all the different areas of the bank, and they work to make sure – and individually make sure – that we’re driving AI into every single area,” he said. Tech chief Hari Gopalkrishnan and strategy head Jeff Busconi jointly lead the group.

Why it matters: Banks are fighting on two fronts to get the most out of AI – deployment and adoption. Deployment comes when banks wire frontier tech into legacy systems. One Bank of America executive reportedly summed that process up to an Nvidia employee like this: “You sold us a Formula 1 race car…now you have to help us as local car mechanics drive the race car.” Even after that though, banks need to get employees to use it: “We have very different usage rates now with the AI tools,” said Commerzbank CEO Bettina Orlopp at the same RBC event this week. “We are currently doing everything to democratize the AI usage so that everybody is really using it and really taking the benefits out of it.” That takes all hands – including teams like Bank of America’s catalyst group – on deck.

Kevin Cole joined JPMorganChase as head of Chase Digital Assistant. Cole was previously responsible for Fargo, the assistant at Wells Fargo and Erica, Bank of America’s offering.

UBS elevated Andrin Solèr to be AI driven process automation and governance lead. He's been with the bank since 2020.

Katie Adams is now head of AI and data culture at Lloyds. Adams’ new remit includes “helping people understand where AI fits in their work,” she wrote on LinkedIn.

Caroline Trang was promoted to be head of AI tooling and delivery at NAB. Trang has been with the Australian bank since 2015.

RBC hired Sabarinathan Vairamani to be director of Gen AI and machine learning. He spent 13 years at Cognizant.

Neobank Revolut applied for a U.S. bank charter and appointed Cetin Duransoy CEO in the U.S. Earlier in his career, Duransoy was global head of installment and loyalty products at Visa. He also spent more than a decade in various roles at Capital One. The neobank this week also secured a U.K. banking license.

Citi is building an agentic AI marketplace, job listings show. It will allow “business teams to safely discover, deploy, and govern AI agents across the enterprise.”

The buy versus build dilemma is as old as time, and the right recipe for banks isn’t always one or the other. In this week’s “Corner” we dig into why HSBC chose the buy route with legal AI startup Harvey as it enables its lawyers with tech.

Use Case: AI-powered legal assistant 
Vendor: Harvey 
Bank: HSBC

Why it's interesting: HSBC partnered with Harvey to deploy AI across its global legal function days after Anthropic released its own legal AI skills in January (see: “Skills pay the bills, The Brief, Feb. 12). The legal assistant can answer questions from the bank’s lawyers and perform tasks across areas like compliance, litigation, contracts and due diligence. Legal work isn’t a differentiating point for a bank’s business, but it’s critical work. That makes it a prime place to rely on a specialist vendor rather than spend resources developing a tool in-house.

How it works: Harvey draws on over 250 legal sources and will be piloted for legal research, due diligence, contract review, and regulatory compliance. It runs a multi-model, agentic system, which aligns with banks’ model-agnostic approach. “This isn’t just about deploying new technology,” said Bob Hoyt, the bank’s chief legal officer. “It’s about reimagining how an in-house legal function can operate by combining the speed and efficiency of AI with the expertise and judgement of our legal professionals.”

By the numbers: Harvey reports that users save between 15 and 25 hours per month. Scaled to HSBC's estimated 1,100 to 1,300 legal staff, that implies roughly 16,500 hours saved monthly, or 10% of the bank’s lawyers’ total time.

Bigger picture: Legal work is increasingly part of banks’ AI roadmap. “We are just introducing in the legal department and legal AI to do everything around contract management,” said Commerzbank CEO Bettina Orlopp this week. Not every bank is looking to vendors though: BPCE’s chatbot MAiA can handle some legal tasks, the bank said. And Société Générale built an AI assistant for legal review.

“We believe that the role of an engineer will radically change in the future. Instead of an engineer coding on their keyboard, they're gonna be overseeing 10 agentic bots that's doing 80% of the code development, and they'll take it the last mile.”

–Brendan Coughlin, president of Citizens Financial, at the RBC Capital Markets investor conference, March 10

DBS generated SGD 1 billion ($785 million) in economic value from its analytics and AI initiatives, the bank reported this week. But to measure how much that’s transforming the business, CEO Tan Su Shan is using a different metric: The number of “operating model transformations” it completes. These “involve re-engineering processes for human and AI collaboration, leveraging tools to enhance workflows, reskilling our people for new and augmented roles, and implementing simplified, adaptive organisational structures,” she wrote. So far the bank has completed nine, three ahead of schedule. The Singaporean bank also reported more-traditional metrics, too: It cut code deployment time by 25%, and its corporate banking assistant “contributed to a 23% increase in customer satisfaction.”

JPMorganChase limited lending to private credit funds and BlackRock limited redemptions in recent days as fears that falling SaaS valuations would bleed into the opaque private credit ecosystem. Bank leaders argue AI prowess may be the solution to those jitters: “How much money do you wanna lend to high-risk, high-reward AI companies, recognizing that some of them may go under?” said BMO Capital Markets CEO Alan Tannenbaum at the RBC investor event this week. “We have amazing resources in a large bank. You’ve got Kristin Milchanowski, who’s our head of AI for the bank…We sit down with her and say, ‘here are the companies that we’re talking to, we’d love your perspective on whose technology we should be betting on’ because that’s embedded in some of these decisions.”

Mastercard (#2 in the inaugural Evident AI Index for Payments) began rolling out its new “Virtual C-Suite,” a family of agents that advise on finances, marketing, and security. The payments giant is kicking off the program with an agentic CFO, available to give “big picture insights” to small businesses. The payments network website shows the firm will also roll out virtual versions of a chief information officer, a CMO and a COO.