The 2024 Evident AI Index Results

Evident's Co-founders share what has changed in the latest ranking, which key findings were most surprising, and how the current state of play is expected to shift in 2025 and beyond.

Key Takeaways:
1. While the top of the ranking remained relatively consistent year on year, the top-10 banks are continuing to enhance their overall performance at a faster rate than the overall Index, underscoring continuing bifurcation in AI maturity.
2. While there's no longer a single bank in the Index that doesn't address AI in their investor relations materials, barely half of the 50 banks are reporting on specific AI use cases.
3. Consequently, next year represents a consequential moment for banks to demonstrate where and how AI is impacting a given bank's bottom line. In a single word: Outcomes.

The unhyped use cases: where are financial services companies generating value from LLMs?

Amidst the explosion of hype around AI use cases, the panelists highlighted where they are currently seeing under-reported applications of AI, where financial institutions are finding early value, and specifically, where those applications will be implemented first across the wider business.

Key Takeaways:
1. Not enough attention is placed on having the enabling platform and culture to deploy AI use cases at scale, as many "press worthy" applications are those that don't escape from a siloed part of the organization
2. Prior deployments of AI/ML applications over the past decade have often been overlooked, but have already brought tremendous value in capital-intensive processes such as credit underwriting and fraud detection
3. Looking forward, value will shift to addressing key pain points that persist in the customer experience. As self-improving AI platforms inherit more of the cognitive burden from humans, end users of banking services will invariably benefit from more personalized applications and services.

In conversation with Manny Roman, CEO & Managing Director, PIMCO

In this keynote session, Manny Roman (PIMCO) served up nuanced, macro insight on what we might expect from the second Trump administration. On AI specifically, Roman shared what improvements the technology is bringing to his firm, as well as productivity gains he ultimately expects to impact the wider labor market.

Key Takeaways:
1. The effect of AI on the productivity of business at PIMCO has been transformative. The company can now accomplish more with less people. Full stop.
2. Building chips in the US is not a trivial task, which puts the incoming administration's pro-energy, pro-business policy at odds with potential tariffs on key technology inputs. Even with considerable investment, it could take up to a decade or more to replicate Taiwan's capability in semiconductor fabrication, warned Roman.
3. One of the first places AI is likely to produce real "alpha" is code generation.

The insight machine: how is AI driving insights and knowledge creation?

Given limited public reporting on ROI, panelists were challenged to describe use cases in active production - and which, if any, are driving revenue and/or impacting the bottom line.

Key Takeaways:
1. David Wu (Morgan Stanley) stressed the importance of deploying labor-saving applications to bank employees that are most adjacent to clients, namely Wealth Management advisors that benefit from the ability to surface insights from 100,000+ source documents.
2. Chintan Mehta (Wells Fargo) unpacked the benefits not just from new use cases, but the gains from "ensemble modeling" that helps surface unexpected benefits of AI applications outside the business need for which they were originally conceived.
3. Gabriel Stengel (Rogo) suggested productivity of entry-level analysts could potentially double over a 5-year horizon, based on efficiency gains observed in adjacent industries.
4. Sid Khosla (EY) outlined the phases of evolution of generative AI from historical information synthesis to pre-emptive conversation prompts—laying out how we should expect multi-agent systems to unfold in an enterprise setting.

The politics of artificial intelligence: how are geopolitical dynamics affecting AI and business?

In this keynote session, Ian Bremmer (Eurasia Group) provided a lively overview of challenges to global governance stemming from AI, as well as a preview of what we can expect from a post-election geopolitical landscape.

Key Takeaways:
1. Both US tech and foreign policy will become increasingly hawkish under a Trump administration, which includes key areas in the semiconductor value chain.
2. International agreements on AI governance remain unlikely under the second Trump administration, as the President will prioritize one-on-one "deals" with individual countries as opposed to multilateral frameworks.
3. The future of AI governance is frequently (and mistakenly) conflated with the nuclear arms race. The absence of a direct agreement between the dominant players (US and China) could potentially be offset via public-private partnerships modeled after the Financial Stability Oversight Council (FSOC).

The power of experimentation: how are banks innovating and creating value through experimentation?

Panelists offered their experience on what's required to master the art of the possible with AI through iterative assessments of performance, continuing evaluation of value creation, and crucially, ongoing feedback from end users.

Key Takeaways:
1. CommBank’s culture of testing is laser-focused on understanding what the impact of AI tools are for their customers.
2. For each use case being tested, CIBC has two key assessment tools: one to measure the opportunity for the business and another to calculate the value it can generate.
3. Tweaking and testing models is no longer the focus of developers. Instead, the main focus has shifted to evaluating the data available for training, fine tuning, and implementation, says Alex Ratner (Snorkel AI).

In conversation with Anthony Scaramucci, Founder & Managing Partner, SkyBridge

The Mooch is loose! In a free-ranging discussion of Trump Act II, Anthony Scaramucci (SkyBridge) provided his personal insight into some of the implications of the incoming President's team of advisors on policy specific to AI and crypto.

Key Takeaways:
1. Any proposed tariffs on AI infrastructure need to be surgical in order to prevent foreign powers taking market share away from American businesses in areas where they do not demonstrate a competitive advantage.
2. Understanding Elon Musk’s evolving approach to AI across his business interests will be critical to understanding Trump’s approach to development and regulation of AI moving forward.
3. Legal and regulatory delays to spot bitcoin ETFs saved the industry by exposing fraud and delaying the current valuation cycle by two years. There's a lesson there for managing the AI bubble...

The enabling factors: how can architecture, data and cloud enable AI?

From the perspective of designing data architecture and implementing cloud infrastructure, what goes into enabling AI? How do you put it together? And what challenges crop up as banks accelerate from experimentation to implementation?

Key Takeaways:
1. In areas such as code generation, the most senior developers often produce the best results from new toolsets. This suggests a need to revisit where and how new application licenses are rolled out, and how expert users document best practices to achieve and scale desired results.
2. Trust in your underlying data helps offset the "black box" explainability issues with LLM applications. Regardless of what happens next in the regulatory environment, overinvestment in data security and governance now will pay serious dividends tomorrow.
3. Alpha derived from the next wave of AI technologies lies in the quality of unstructured data, which in turn, requires a proactive approach to controls around that data.

The breakthrough potential: how are financial services companies embracing emerging technologies?

The AI industry is entering a new hardware paradigm, as firms move from training models to extracting inferences. Meanwhile, the leading edge is making early bets on quantum. As hardware customisation becomes more impactful to the speed and scale of each successive wave of AI deployments, future-proofing your software stack becomes critical.

Key Takeaways:
1. AI and Quantum have a symbiotic relationship. Beyond Quantum providing the processing power to fuel AI growth, AI systems are also being deployed to overcome roadblocks to making Quantum viable (e.g. error correction). Consequently, venture capitalists like James Wise (Balderton Capital) are betting AI winners will demonstrate an ability to optimize for diversity in the hardware layer of their tech stack.
2. Quantum is becoming more stable, and thereby more promising. The industry has moved from hypothesis testing and science to engineering and business. Some banks like JPMC are already at the forefront of this research not only in banking, but also in the broader ecosystem.
3. We are moving to a period where specialised hardware will be targeted at the use-case level, leveraging architectures that combined both accelerator and quantum hardware.

The innovation advantage: how can banks benefit from the most promising AI research?

This session covered a host of AI research topics, ranging from what's currently front of mind, to how applied research feeds into business applications, to the most exciting new topic of inquiry on the horizons.

Key Takeaways:
1. Lessons learnt in AI research have fed into three buckets of use cases that Goldman Sachs focusses on developer productivity, knowledge worker support and supporting client-facing staff
2. Progress in reinforcement learning - where systems can take decisions in complex environments to get some future reward - is an exciting trajectory in AI research for Giuseppe Nuti (UBS) given its alignment with how decisions are made in finance.
3. The opportunity of mass personalisation from AI is an area that Deborah Yang is most excited by based on existing research.

The agent revolution: how could human-machine partnerships be optimised, now and in the future?

Panelists disentangled a critical theme that kept emerging throughout the day's agenda: AI agents. This session covered the differences between AI agents vs. LLMs, in terms of their respective benefits, development frameworks, and substitution between human- vs. machine-driven actions.

Key Takeaways:
1. Developing an internal definition of an AI agent is key: JPMorganChase have designed their own framework, while Vahe (Cognaize) suggests an AI agent requires a sensory input, reasoning engine, the tools for it to action, and a mission for the agent to achieve.
2. Identify the intended outcome early in the development process, as this provides not only the intended guardrails, but also a guide to how much should a human still be in the loop throughout the workflow. As Sumitra from JPMorganChase put it, these agents may need babysitting in the short term, but once the trust is built up they will need to be let go.
3. For at least two banks (JPMorganChase and TD Bank), AI agents should be coming to production within the next year...

The cutting-edge applications: how can companies integrate AI into a new technology stack?

For the final session of the day, panelists from leading enterprise vendors (including Microsoft, Salesforce, and SambaNova) provided their future-forward guidance on adoption of AI agents - and specifically, what that means for frontier applications within financial services.

Key Takeaways:
1. Looking forward, the rate at which the number of discrete AI agents will rival humans in an enterprise is not five years out - it's imminent.
2. You need three things when experimenting with agentic AI: flexibility of the enabling platform, low latency when you combine agents, and checkpoints to debug increasingly complex systems.
3. Different customers will have different requirements that favor standardized, off-the-shelf solutions versus highly customized, proprietary solutions.