Data-driven insights and news
on how banks are adopting AI

The Year of Gen AI (scratch that) Agentic AI

12 December 2024

TODAY’S BRIEF

A never-too-early Merry Christmas and best wishes for a happy AI-fueled 2025 from our team here at Evident.

We have something special on offer today – our first annual Evident awards for the best (and not) in AI and in banking in the past year. See who was brilliant and what deserved a few lumps of coal. Plus we grade ourselves on our predictions for this year and look ahead to 2025.

But first the most interesting ideas and trends from the gathering of AI minds at the Conference on Neural Information Processing Systems (NeurIPS). Along with ICLR and ICML (who names these things!?), Vancouver is the place to be this week if you’re an AI researcher . . . or an AI recruiter. One clear trend: generative AI is going to share the stage with agentic AI, as our Use Case Corner also finds.

The Brief is 2,345 words, an 8 minute read. If it was forwarded to you, please subscribe. You can read this Brief online here. Find out more about our membership offers here. We want to hear from you: [email protected].

— Alexandra Mousavizadeh & Annabel Ayles

TOP OF THE NEWS

LONG LIVE THE LLM

Image generated by Adobe Firefly

What’s preoccupying AI’s 15,000 best and brightest gathered in Vancouver this week? While we spend a lot of time thinking about the practicalities of AI adoption in finance, the world's top researchers are more focused than ever on foundation models.

In the 4,500 papers accepted at NeurIPS, you see a clear and growing focus on large language models. The other notable spike is in the interest in diffusion, the tech behind image and video generative AI.

LLM Boom

The number of accepted AI research papers at the NeurIPS conference by topic in 2024 compared to 2023

Source: https://neurips2024.vizhub.ai/

Main takeaways

We went through some of the papers and listened in on the conversations so far, and here are our main takeaways.

Agentic is hot. Meta and Microsoft showed off new ways to build agentic AI systems. Microsoft’s AutoGen team highlighted multi-agent workflows, using a team of AI agents with defined roles and toolsets. For example if one agent was tasked with finding things to do on holiday, another might handle scheduling, and a third booking. For a practical example of a multi-agent system, see our interview with Caesar Sengupta. For more on banks and agentic AI, scroll down to the Use Case Corner section.

Sticker shock. As the cost of Gen AI goes up for everyone, Andy Liu, AI engineering team lead at Bloomberg, offered some tips to reduce them: Use powerful models to experiment, then scale back to cheaper alternatives by getting a smart model to teach a cheap one (knowledge distillation). Develop a unified platform for LLMs, he added, an approach already used by banks such as Goldman Sachs.

A new kind of flexibility… While traditional machine learning requires training a new model for each new task, LLMs already understand text and can tackle novel challenges simply by adjusting the prompt—like asking them to translate, summarize, or rewrite copy. But LLMs are mostly limited to text and image inputs. Researchers from Capital One, Shopify, and IBM showed off foundation models for new domains like tables (e.g. spreadsheets), recommendations, and forecasting.

Banks are turning out. Conferences like NeurIPS are important places to scout for talent and exchange ideas. Four of the 50 Evident Index banks (Capital One, RBC, NatWest and CommBank) now sponsor the conference, up from two last year. Researchers from six have had papers accepted this year. They share the field’s current obsession with LLMs. Capital One submitted a paper on how to use LLMs to improve data quality, RBC looked at how to make them more efficient. Morgan Stanley’s paper is on how to improve financial forecasts using the latest AI techniques.

All too human. “Do LLMs dream of bull markets?” asked CommBank. Their paper found that LLMs given personas can accurately reflect human decision making when making investments, including risk appetite and impulsivity. We’re not sure the model’s capable of dreaming, but it’s able to simulate human behavior, say the authors.

NOTABLY QUOTABLE

"“Data is as critical as algorithms. To this day, no matter how much progress we make in deep learning, in AI, in GenAI, so much of it relies on data.”"

- Fei-Fei Li, Co-Director of the Stanford Institute for Human-Centered AI, speaking December 11 at NeurIPS 2024

USE CASE CORNER

FROM GEN AI TO AGENTIC

The conversations at NeurIPS will filter into the real world quickly. Get ready for a whole new world of use cases in 2025. Just half a year ago, generative AI was the rage and felt relatively new (See “74 Ways to Use AI”). As banks build out Gen AI applications (the so-called Wave 2), now there’s a new new thing coming over the horizon.

And that is … Agentic AI. Australia’s CommBank is a pioneer, the first and so far only Index bank out with a live agentic AI use case. The bank is deploying agents to solve some of the 15,000 payment disputes raised by its customers every day.

But banks will be cautious to give the machines too much control over their customers’ finances. Vendors like Bud Financial are piloting AI agents in lower risk use cases, such as maximizing interest earnings on deposits or avoiding overdraft fees. For more on this topic, check out the “The Agent Revolution” discussion at last month’s Evident Symposium in New York.

Generative AI, say hello to the customer. And customers can say goodbye to the basic chatbot.

Throughout 2024, banks have been rolling out generative AI tools that give both staff and external customers greater access to progressively more advanced capabilities, performance, and personalization. Here are a few of our favorites:

State Street’s generative AI assistant on its Alpha platform will let customers generate custom insights based on State Street’s proprietary data and research about assets that the custodian bank holds for clients.
Lunar, a Nordic neobank, launched a GPT-4o–powered voice customer assistant that’s able to have live human-like phone conversations with customers. This tool can do what traditional voice chatbots struggled with, such as pauses, or changes in the questions posed by customers.
BBVA’s customer insights tool, which combines NLP and pre-trained transformer models to automatically address qualitative feedback that customers provide through the bank’s app via push notifications. This way, BBVA can get more customer feedback that is not captured through traditional methods, such as Net Promoter Score (NPS).

Holiday Honors

In the spirit of the season, we’re making lists, checking them twice, and figuring out who in the world of AI and banking has been naughty and nice. Some are easy picks for best-ofs, breakthroughs, and a few duds. Some forced us to make hard choices. By no means exhaustive – and with a bit of tongue in cheek – we’re delighted to present our inaugural Evident Holiday Honors List for 2024.

BREAKOUT STARS

Our first prize goes to the people who somehow broke out in 2024 by claiming a voice or doing something notable in the field of AI and banking.

Robin Vince: The BNY CEO turned heads by leaning America’s oldest and leading custodian bank into AI. He touted efficiency gains while with frankness saying the payoff won’t come before 2026 soonest, made BNY the first global bank to integrate an AI supercomputer into its tech infrastructure and struck a unique partnership with Nvidia to take advantage of the bank’s huge trove of data. Coming from the leader of a by necessity more risk-averse house, Vince’s vote of confidence resonated.

David Solomon: Don’t count the Goldman Sachs chief executive among the irrationally exuberant on AI. But in 2024, Solomon made himself and his bank central to the conversation. For the first time, he talked about it in his shareholder letter and earnings call. In deploying its first Gen AI tool, Goldman spent months centralizing their in-house AI tech before rolling out the platform. The firm is a leading voice on AI and its future.

Sergio Ermotti: The UBS chief spoke with actions. By deploying Microsoft AI CoPilot to 50,000 employees, he made the bank one of the largest users in financial services to date. UBS’s M&A "Co-pilot" puts in customer hands a generative AI tool that analyzes company databases, generates buy-side ideas, and identifies potential buyers when it’s time to sell. Ermotti says AI is more about “evolution not revolution,” but his moves in 2024 were anything but incremental.

BIGGEST MISS OF THE YEAR

ROI NEEDS MORE FOCUS

Banks didn’t talk enough about the ROI on the billions spent on AI – and that’s a problem. You have to know where the gains are in order to be able to capitalize on them. We are expecting more reporting on ROI in 2025.

BEST BOOKS

Our Evident Book Club read a lot. These three stood out in 2024.

I Am AI: A Novelette by Ai Jiang

“A short, emotionally impactful story that was a thought provoking view on the trade-off between human creativity and greater efficiency” – Andrew, Index Team

The Eye of the Master: A Social History of Artificial Intelligence, by Matteo Pasquinelli

“A refreshing challenge to how we think about the ‘intelligence’ bit of AI” – Sam, Intelligence Team.

Code Dependent: Living in the Shadow of AI, by Madhumita Murgia

“Diverse set of use cases that highlight the lived experiences of those impacted by AI from around the world” – Mellisha, Data Team

Technology and the Rise of Great Powers: How Diffusion Shapes Economic Competition, by Jeffrey Ding

“Timely exploration on how emerging technologies such as AI could influence the US-China power balance” – Colin, Intelligence Team

BEST PODCASTS

Me, Myself, and AI, by Sam Ransbotham and Shervin Khodabandeh

Great guests and practical case studies of implementing AI in business

No Priors, by Elad Gil and Sarah Guo

Nice balance between technical depth and investment-focussed discussion

Hard Fork, by Kevin Roose and Casey Newton

Lighter, fun, but punchy editorial podcast from NYT journalists

Aarthi and Sriram Show, by Aarthi Ramamurthy and Sriram Krishnan

Lots of insight, lots of humour, with top innovators for guests

BREAKOUT REGION

RULE, BRITANNIA!

UK banks ran away with this one. Based on Evident’s 2024 Key Findings Report, the average UK bank increased its Index score +4.5 points – nearly twice the Index average (+2.4 points). While they still lag behind the North American leaders, their particularly strong showing in Leadership and Transparency reflect a ramp up in activity amongst these players.

CODA

OUR 2024 PREDICTIONS REPORT CARD

A year ago today, we put out our 2024 predictions. We preach transparency for banks, and try to live by the same credo. We also think that what we got wrong can be as revealing as what we got right.

Generative AI Accelerates from Promise to Reality

How did we do? 🤷 Partial Credit

Context: From Morgan Stanley to JPMorganChase, banks are showcasing generative AI tools. Out of over 160 use cases identified for consideration in our Use Case Corner, 47% qualify as examples of Gen AI. But upon closer inspection, only 24% of these use case announcements provide concrete information on outcomes (i.e. business impact). So lingering questions about ROI will persist into 2025.

AI Leaders Expand Presence on Operating Committees

How did we do? ❌ MISS

Context: While 41 out of 50 Evident AI Index banks now have at least one Executive Committee member focused on AI, this number has remained virtually unchanged year-over-year. In fact, instead of a rapid expansion of conventional C-suite roles (see “The C-Suite’s Hottest New Job: The Chief AI Officer”), we’ve observed banks are reluctant to give AI strategy to newly minted roles. That may not be a bad thing. You can avoid redundancy and siloing. But we still got it wrong.

AI Talent Escapes Right Sizing & Cost Reduction

How did we do? ✅ HIT

Context: Ding ding ding. Year-on-year, we observed the AI workforce found across the Evident AI Index banks surged by over 17%. Moreover, every talent capability area (e.g. AI Development, Data Engineering, Model Risk, AI Implementation) is growing at 5-25x faster than overall headcount (+1.4%).

Banks Shift in Mindset from What Can AI Do? >> What ROI Does it Create

How did we do? ✅ HIT

Context: As banks move from experimentation to implementation, they’re slowly rolling out evidence of initial ROI. Fifteen out of 50 banks now make substantive AI disclosures in Investor Relations materials linking AI to strategic priorities, emerging use cases, and measurable business impact (up from only 6 banks in 2023). However, only two banks (DBS and JPMC) have attempted to provide a comprehensive estimate of overall business impact from AI investments. 2025 will test which of these banks can move from sporadic to systematic reporting.

Banks Provide the Blueprint for Emerging AI Regulations

How did we do? ❌ MISS

Context: By far, one of the most anti-climactic gatherings this year was the Network of AI Safety Institute (AISIs) in San Francisco two weeks following the U.S. election. While much of our view of what AI under a Trump Administration looks like remains speculative, we’re fairly confident that synching up with the EU AI Act won’t be part of the Trump policy agenda.

Open Question: Will the Gap Between Leaders vs. Laggards Shorten or Widen

How did we do? ✅ HIT

Context: Ok – yes, Evident failed to achieve consensus on this non-prediction. But surprisingly, these two things can be true at the same time. The established Index leaders (JPMC, Capital One, and RBC) faced increased competition from their immediate peers, with 10 banks now scoring over 40 out of 100 points in the Evident AI Index (diminishing gap among top performers). Meanwhile, these same 10 banks improved their standings 2x faster than the overall Index (widening gap between leaders and laggards).

ICYMI

We published our 2025 Predictions following the Evident AI Symposium. We’ll use our first issue of The Brief next month to have a deeper look at the year to come. If you have a thought or prediction on what 2025 will bring, please share it with us at [email protected].