Why cheap AI ain't cheap

Banks were already losing sleep over how much cash they were wiring to the big AI labs. In comments heard and much discussed in the last week in the tech and business worlds, Palantir CEO Alex Karp added another concern: Companies choosing to “waste [their] time” running up token bills with AI labs’ models are handing over their best ideas to them as well.

“What he's referring to there is that these enterprises are at risk of transferring their knowledge, their know-how, their trade secrets, their customer data to these model providers who might eventually decide to compete with them,” said former White House AI Czar David Sacks, discussing Karp's comments. “Enterprises are waking up to this threat and they're not happy about it.”

The obvious escape hatch for businesses is to send less work to the frontier labs and use cheaper, open source models instead.

Here’s the catch: Cheap AI is cheap for a reason. As part of their offering, the big labs send forward deployed engineers who do a lot of the work to make it possible for businesses – not least highly regulated ones like banking – to deploy this technology. That includes cleaning up messy data and ensuring they’re secure in the eyes of customers and regulators. Open source models push more of that work back onto the bank, which adds up to costs beyond tokens.

Hedge fund Bridgewater last week showed what banks have to do before the model bill actually comes down. Working with Thinking Machines, the AI firm founded by former OpenAI CTO Mira Murati, the firm detailed how it fine-tuned Qwen, a Chinese open source model from Alibaba. The firm wanted AI that could pick important details out of documents and make judgment calls about whether news or central bank reports or research briefs mattered to investors. Off-the-shelf models could do that reasonably well, but not well enough that Bridgewater employees found it useful for everyday work. Bridgewater’s staff went back through their documents and, with Thinking Machines’ guidance, marked up documents so the AI could think more like an employee. After that process, the customized open source model beat every frontier lab’s model and ran at nearly one-fourteenth the cost.

That kind of fine tuning takes time (and money). Even when a firm gets them working though, they – and the savings they deliver – can be fragile. Smaller, open source models need tighter prompts and aren’t as polished when people dump more information into them. Coinbase CEO Brian Armstrong said his firm had cut its AI bill in half in part by switching to Chinese models like GLM 5.2 and Kimi 2.7, but the key was in educating his workforce on how to use them with more finesse. That meant teaching employees to cut off access to certain tools so they didn’t inadvertently run up bills and to prompt them in more exact ways.

Bottom line: Big AI labs are pricey for a reason. Banks have a way of escaping the bills, but cheap AI only gets cheap once you pay to make it good.

Banks are racing to give their systems more autonomy, but Morgan Stanley’s lesson from one of its messiest, judgment-heavy jobs is that AI can be just as helpful when the leash stays on. In this week’s “Corner,” we look at how an agentic trade accounting tool the bank uses freed up tens of thousands of hours.

Use Case: FIXR
Vendor: n/a
Bank: Morgan Stanley

Why it’s interesting: Morgan Stanley has roughly 100 controllers tasked with closing the book on the bank’s trades each day. Their job includes hunting down and fixing any discrepancies that may exist in the different systems that record a trade. One trade, for example, might have a slightly different settlement date in a finance system or a different exchange rate in a trade-capture system. It adds up to hundreds of thousands of mismatches (called “breaks”) that then need to be reconciled before a trade gets passed back to the desk the next morning. The bank’s agentic system now helps trace those breaks back to their source and suggests the likely fix.

How it works: When a break gets flagged, FIXR uses three agents to determine the best fix. One gauges the feedback controllers have given in the past about the type of break that shows up, one looks for patterns in how the humans have dealt with certain types of mismatches in the past and a third turns those patterns into recommended actions. The controllers make the final call every time.

How they did it: The bank trained the system by having it observe controllers’ processes and judgment calls, which it then turned into fixed rules. That means the model powering the tool doesn’t have to “think” as hard about how to solve problems that come up. Because humans have to respond to every suggestion an agent makes, the model keeps learning over time. “If you have an opportunity to make things very prescribed and repeatable, that’s cheaper in terms of token consumption,” said Todd Johnson, managing director and head of process engineering, automation and AI application at the bank. “Have the LLM do the stuff where you don’t need that kind of deterministic workflow.”

By the numbers: FIXR cut the time it took to reconcile breaks from six hours to two or three. It adds up to roughly 1,500 hours saved per week, Johnson said.

Bigger picture: Controllers in New York make a median of $106 per hour, according to the Occupational Information Network. Game that out and FIXR’s efficiency gains translate to roughly $160,000 per week and north of $8 million annually, our back-of-the-envelope math suggests. At the same time, the tool reduces risk associated with fully-manual reviews of breaks. Goldman Sachs has also turned to agents to handle back office trade accounting – with supervision – as a part of its Anthropic tie-up (see: “Agents move beyond coding,” The Brief, Feb. 12)

Bankers descended on Seoul this week to rub shoulders with the academic set at ICML, one of the world’s premier academic AI conferences. Some 6,000 papers sketch out what the future of almost every industry might look like.

For financial institutions, it showed something else – just how far away the future actually is. Three big themes in the papers throw some icy water on the agentic future banks have been promised.

FLIGHTY AGENTS: For all the talk of AI agents gaining Ph.D.-level skills, inside the workplace they still act more like an intern on a Friday. A team from Princeton tested more than a dozen frontier models and found that for all the gains in what models can do, there’s been almost no improvement in how reliable they are, at least on certain tasks. The same agent given the same task twice produces different answers, they found. And small rephasings of a prompt still derail them. In banking, that means they can’t really be let off the leash: Intesa Sanpaolo research found that 68% of AI agents can’t take more than 10 steps without a human stepping in.

• THE FIX: Research from Capital One showed one potential solution. The bank designed a system that scores how an agent is doing as it completes each step of a task so that failures can get flagged up to humans earlier, before any real damage gets done.

FORECAST FOG: Time-series foundation models, which learn from long series of numbers rather than words and could be used for forecasting in markets, liquidity or fraud, have been pitched as the next ChatGPT for banking. The research in Seoul suggests holding the purchase order. Tested across a broad range of industries, the leading models’ accuracy swings so widely from one domain to the next that the forecasts may well be a coin flip.

• THE IMPLICATION: There won’t be a one-size-fits-all forecasting model available for banks off the shelf anytime soon. If a lender wants this kind of forecasting model for its messy data, it may as well add it to the ‘build’ backlog now.

KEEPING TABS: The way banks pay for LLMs is ripe for fraud, says another paper. When you submit a request to a specific LLM, like Anthropic’s Claude Mythos, it returns an output and charges you based on how much “thinking” the LLM did. But that work is hidden. Labs don’t reveal it to prevent competitors from copying it, and customers can’t confirm which model was used. Your request to Mythos could be silently passed to a cheaper or downgraded model without you knowing.

• THE PATCH: The paper proposes a way for agents to log every step and create verifiable receipts of their actions so third-party auditors can check that customers get what they pay for. It would be a step in the right direction, but that much babysitting means building real trust in these systems is going to take time yet.

That’s the boost to Nubank’s AI transaction NPS score – its measure of customer satisfaction with AI interactions – after the bank rebuilt the customer service AI agents handling card delivery questions. The trick was “simulation-maxxing,” as Aman Gupta, the firm’s principal machine learning scientist, wryly put it. The firm set up simulated support chats and had AI judges check, at scale, how the agents performed. It then knew where to put extra training in, like making sure the agents always asked for the right information. It took the same approach for customer service agents handling other tasks: debt management support satisfaction jumped 40 percentage points, credit-limit support gained five, card management was up 38 and product explanation improved by 12 percentage points. In four of the five areas, the newly-trained AI agents were less likely to need to kick questions up to humans than previous bots.

Zoom out: Banks talk plenty about AI saving money, but less about whether customers actually like the tech. Fewer than 5% of public AI use cases with outcomes across the 50 banks we track cite happier customers as the main return. That may be changing as backlash against AI grows and businesses are starting to want to show the tech benefits to customers. Citi, for example, said in May that AI helped lift the digital NPS in its cards business by five points.

JPMorganChase will use SambaNova’s AI chips to power some of the AI models the bank runs inside its own data centers, SambaNova announced this week. “It sends a message to the banking industry that it’s time not to completely depend on cloud services,” said SambaNova CEO Rodrigo Liang. “These banks want heterogeneous [infrastructure].” The announcement came as the AI chipmaker closed a $1 billion series F funding round, which values the firm at $11 billion. It’s the latest signal in banking that lenders want to own more of the AI stack and become self-sufficient: Last month, PNC CEO Bill Demchak said the bank “will have our own GPU compute” so it isn’t as reliant on external providers (see: “Tokenflation,” The Brief, June 11).

OpenAI’s most powerful models are open for business: The Trump administration lifted restrictions on GPT-5.6 models, which include Sol, the flagship version of the model as well as cheaper versions, Terra and Luna. The White House had previously blocked OpenAI from releasing the models to the public in a similar way as it did for Anthropic’s Fable 5 model (see: “Claude gets paroled,” The Brief, July 2). The White House seems to be moving faster to clear new releases: Anthropic’s model was on the shelf for 19 days before its ultimate approval, while OpenAI’s was cleared in a week.

Scotiabank joined Canadian firms Sun Life, TELUS and Lightworks to launch the AI Consortium, a group that will build shared software focused on keeping AI systems in check. The group’s first project is an Agentic Control Plane, a tool that shows what AI agents are doing around a business in real time, Scotiabank CIO Tim Clark wrote. The tool is currently processing two trillion tokens per month from the member organizations. Next, the group is building an AI Operations Center, a system which will give better insights into “performance, resilience and cost management,” and an AI Token Exchange, essentially a shared way for members to buy and use AI capacity without each needing to build the infrastructure from scratch.

Elon Musk’s SpaceXAI released Grok 4.5, a new model built to “handle difficult, long-running tasks,” including finance work. It’s the first release since the firm announced its intention to buy Anysphere, the company behind the AI coding tool Cursor, which it developed this model alongside. It’s priced cheaper than Anthropic’s Opus 4.8 and bests it on three commonly-used software engineering benchmarks. “Our internal assessment is that Grok 4.5 is roughly comparable to Opus 4.7, but much faster,” Musk wrote on X. The firm has been pushing Wall Street firms, namely Morgan Stanley and Apollo Global Management to use Grok, but there’s a lot of catching up to do: None of the 50 banks we track have a public use case that uses Musk’s firm’s models.

Emirates NBD (#1 in the Evident AI Index for Banks - MEA) partnered with Techstars to develop AI startups by providing “early-stage startups with mentorship, funding, and direct access to Emirates NBD’s banking infrastructure to scale innovative financial technologies.” These kinds of accelerator and incubator programs have had mixed results in other parts of the world: BNP Paribas runs B! Up Accelerate, a fintech and insurtech accelerator, which lets startups pilot tech within the bank’s ecosystem. Barclays, meanwhile, wound down RISE, its incubator for fintechs, last year.

“The growing use of Chinese AI models by U.S. companies raises serious concerns…[those] AI models are designed to advance Beijing’s narratives, censor dissent, and reflect CCP ideology and values.”

–U.S. State Department spokesperson, in an interview, July 8

Forget private equity or an MBA, the best investment banking exit opportunity may now be a frontier lab. OpenAI is hiring an investment banking subject matter expert to “define what excellent AI-assisted banking work looks like and turn that standard into better models and products,” a new job description shows. The role, which pays north of $200,000 in base salary and offers equity, asks for two years of investment banking experience, which includes live deal execution.

Goldman Sachs hired Evan Kotsovinos as a partner and head of asset and wealth management engineering. He was previously VP of privacy, safety and security at Google. Before that, he worked at American Express and Morgan Stanley in infrastructure roles.

JPMorganChase hired Quazi Haydar as a principal engineer and executive director. He was previously a software engineer and executive director at Goldman Sachs.

Jane Seah left Goldman Sachs to join Tomoro, the Edinburgh-based AI firm acquired by the OpenAI Deployment Company, the AI services firm the ChatGPT-maker launched earlier this year to help companies adopt the tech.