The analysis evaluated sampled earnings call transcripts across U.S., European, and Latin American companies. Each transcript was independently scored across six quality dimensions on a 1–10 scale. Across all six categories, Aiera’s transcripts achieved an average score of 8.04, compared to 5.74 for the benchmark, a 40% overall quality improvement.

Aiera outperformed the benchmark in every category, including:

• Speaker Identification: +3.87 points (79% improvement)
• Formatting Consistency: +3.74 points (83% improvement)
• Readability: +1.95 points (31% improvement)
• Verbatim Accuracy: +1.58 points (27% improvement)

All transcripts undergo Aiera’s four-stage process to reach “Human Perfect” status.

1. Real-Time Transcription – AI Generated live transcript text from audio with no delay
2. Speaker Final – Editor reviews the AI-generated transcript against audio for accuracy, speaker attribution, proper nouns, and financial figures.
3. First Draft Final (FD) – Formatting polish, metadata verification, compliance review, and structural consistency check.
4. Human Perfect (HD) – Final sign-off and publication readiness.

AI-First. Human Perfected.

Aiera’s transcription workflow begins with AI generation and is elevated through structured human review. On average, Aiera’s AI engine produced a first-draft transcript in just 6 minutes following the completion of the event, regardless of call length.

Across the sample, the average earnings call lasted approximately 50 minutes, meaning transcripts were generated roughly 8x faster than call duration. The AI-generated first draft’s verbatim accuracy scores competitively against the benchmark’s final reviewed version. After human review, the final Human Perfect version quality exceeds the benchmark, driven by improved speaker verification and verbatim corrections.

Why Quality Matters

The analysis identified recurring deficiencies in the benchmark transcripts, including:

• Unresolved “Unknown Analyst” labels
• Incorrect company names and technical terminology
• Meaning reversals due to transcription errors
• Omitted financial disclosures marked as “[Audio Gap]”

Aiera’s transcripts resolved unknown speakers, corrected material errors, recovered indiscernible content, and included structured navigation features such as timestamps, speaker directories, and keyword metadata in every file.

“These results reinforce the benefits of AI in combination with human oversight,” said Ken Sena, CEO and Co-Founder of Aiera. “When AI is combined with structured human review, it offers speed and delivery that institutions can trust.”

James Nguyen, CEO of INFLXD, added: “This benchmarking study shows what happens when purpose-built AI meets rigorous human editorial standards. Together with Aiera, we’re defining a new quality benchmark for financial transcription.”

Setting the Standard for AI-Native Research

Aiera’s Human Reviewed Transcripts are produced through its exclusive partnership with INFLXD. They reflect Aiera’s consortium-led model, built to align AI innovation with institutional standards for transparency, attribution, and auditability.

As AI reshapes financial research, independent validation matters. This benchmarking analysis confirms that Aiera’s governed, AI-native infrastructure delivers transcripts that are both faster and more accurate, built to meet institutional standards.

Download the Benchmark Report here: https://aiera.com/transcript-benchmark-report/

About Aiera
Aiera is a consortium-backed, AI-powered financial research and event intelligence platform built for institutional investors. Governed by leading Wall Street research providers, Aiera enables compliant, fully auditable access to high-value research content, spanning discovery, analysis, summarization, and distribution, with complete IP governance and attribution. The platform empowers research providers to adopt AI while retaining full control of their content, and gives buy-side firms centralized, entitlement-aware access to AI-ready research through vendor-agnostic APIs, MCP, and embeddable components across existing and emerging distribution channels.

Media Contact:
Carol Hickins
PR for Aiera
[email protected]

The post Aiera’s Human Reviewed Transcripts Set New Bar as Institutional Standard appeared first on Aiera.

Transcript:

Now with these new features, Claude can go much deeper into your direct workflows, and that is across hedge fund, private equity, and investment banking. And we would love to show you what this actually looks like in action. Starting with hedge funds.

Let me show you how Claude helps credit analysts move from market signal to investment decision in minutes, not hours.

We’re following Yuri, a credit analyst at a multi-strategy fund, Steady Capital. It’s 10:00 AM, and he notices the Walmart 2030 bonds moving tighter. Portfolio review is at 2:00 PM. Yuri wants to know—is this a trading opportunity or a fundamental shift?

He starts simple: “Show me Walmart’s bond curve and compare it to Costco and Target.” Claude connects to LSEG. Within seconds, an interactive dashboard is ready. Yuri sees the full Walmart curve with live data. The 2030 is tight, but there is a steep jump up to 2037 and 2038. The market wants compensation for a longer duration.

Claude automatically pulls comps for Costco and Target, and Walmart trades tighter than both of them. In under a minute, Yuri sees that Walmart is priced as best in class.

Now, do the fundamentals support it? Yuri asks Claude: “Dig deeper on Walmart’s credit quality. Analyze the last three earnings calls, check Third Bridge experts, and give me a write-up.”

Claude posts Aiera earnings transcripts for Q4, Q1, and Q2 while searching Third Bridge for former Walmart executives. Everything happens in parallel. Within minutes, Claude has a complete credit analysis ready.

It found patterns across three quarters:

• consistent sales growth
• surging e-commerce
• advertising up nearly 50%, and new high-margin businesses scaling

Every claim is verifiable, with links that go directly to Aiera’s platform and the exact transcript Claude analyzed.

Claude also found former executives who actually built these systems. One confirms the e-commerce profitability story. Another explains their competitive advantage on tariffs, supplier relationships, and domestic sourcing give them structural protection.

Claude synthesizes everything: this is a trend, not a blip. Three quarters of strong execution, experts confirming it’s real, with every data point linked to source.

It’s 1:45 PM. Yuri walks into the portfolio review with this dashboard and comprehensive deep-dive. The story is clear. The market is pricing Walmart as best in class and fundamentals back it up.

Claude’s view: short bonds are priced to perfection, and longer bonds offer better value if this transformation continues.

30 Minutes. 3 platforms. 1 story with full transparency. That’s the difference between data and conviction.

The post Watch Claude and Aiera Bring Real-Time Intelligence to Investment Workflows appeared first on Aiera.

Enables financial professionals to access Aiera directly within Claude, enhancing research, analysis, and decision-making where workflows live across the GenAI ecosystem

New York, NY — October 27, 2025 — Aiera, the world’s leading investor events and intelligence platform, today announced its collaboration with Anthropic to bring Aiera’s trusted investor events data and real-time market intelligence into Anthropic’s Claude for Financial Services. The integration enables financial professionals to access Aiera’s human-verified data directly within Claude, enhancing research, analysis, and decision-making across GenAI workflows.

Built using Anthropic’s MCP (Model Context Protocol) open standard, customers can now seamlessly and reliably unlock Aiera’s deep library of investor intelligence within Anthropic’s Claude, or a model of their choosing.

“Our integration with Claude for Financial Services marks a major leap forward in how professionals access and act on our real-time intelligence,” said Bryan Healey, CTO and Co-Founder at Aiera. “Today’s announcement is a testament to our commitment to meet customers where their workflows live, offering flexibility and choice across the AI ecosystem while ensuring they can rely on Aiera’s trusted data wherever insights are needed.”

Claude for Financial Services is designed to help financial institutions safely and effectively apply generative AI across research, portfolio management, and client advisory workflows. By integrating Aiera’s verified event data, Claude for Financial Services enables users to generate insights, summaries, and answers grounded in the most comprehensive investor event coverage available.

“MCP gives us a universal standard for providing dynamic context to intelligent systems,” said Bryan Healey, CTO and Co-Founder at Aiera, “while still maintaining security, attribution, and traceability.”

With coverage for over 45,000+ events annually, Aiera is the world’s leading investor events platform delivering visibility into virtually every corporate, investor, and macroeconomic event impacting global markets. Approximately 25% of this content is uniquely available on Aiera, making it the most comprehensive and differentiated dataset in the industry.

Aiera’s coverage includes:

• 31,000+ corporate earnings, investor days, shareholder & special situation meetings
• 12,000+ conference presentations
• 3,000+ macro and regulatory briefings

This represents more than 13,000+ global companies across most major indices, including S&P 500, DJIA, NASDAQ COMP, Russell 3000, FTSE 100, DAX, CAC 40, and S&P/ASX, as well as over 100+ marcomacro and regulatory entities.

About Aiera:

Aiera is the leader in generative AI event solutions for financial services professionals. Now backed by a first-of-its-kind consortium of leading Wall Street banks and research providers, Aiera integrates broker research, expert network content, financial news, transcripts, filings, and other public data sources into its technology and platform solutions. Trusted by many of the world’s top asset managers and financial institutions, Aiera delivers its capabilities through a secure dashboard, customizable components, and enterprise-grade APIs – empowering more informed, real-time decision-making across the investment workflow. Learn more at aiera.com.

Media Contact:
Carol Hickins
PR for Aiera
[email protected]

The post Aiera and Anthropic Announce Integration of Aiera’s Trusted Market Intelligence into Claude for Financial Services appeared first on Aiera.

New York, NY — October 23, 2025 — Aiera, the world’s leading investor events and intelligence platform, today announced “Human-Reviewed” Transcriptions in addition to a number of speed, coverage, and performance improvements to its overall events experience. Pushing its continually improving real-time transcription accuracy to 99% within one to three hours of event completion, Aiera’s new offering combines advanced AI automation with expert human reviewers to ensure unmatched precision, superior diarization, and accurate speaker attribution.

“Aiera provides the fastest and most complete transcription offering anywhere, from original sourcing to transcription and review. The result is reliability and speed at scale, giving clients quality transcripts they can trust,” said Ken Sena, CEO and Co-Founder of Aiera.

With coverage for over 45,000+ events annually, Aiera is the world’s leading investor events platform delivering visibility into virtually every corporate, investor, and macroeconomic event impacting global markets. Approximately 25% of this content is uniquely available on Aiera, making it the most comprehensive and differentiated dataset in the industry.

Aiera’s coverage includes:

• 31,000+ corporate earnings, investor days, shareholder & special situation meetings
• 12,000+ conference presentations
• 3,000+ macro and regulatory briefings

This represents more than 13,000+ global companies across most major indices, including S&P 500, DJIA, NASDAQ COMP, Russell 3000, FTSE 100, DAX, CAC 40, and S&P/ASX, as well as over 100+ macro and regulatory entities.

“With today’s announcement, we continue to deliver on our promise to bring to market actionable intelligence, bridging the gap between human insight and AI scale,” said Gavin Skinner, Chairman and COO of Aiera. “This milestone demonstrates how advanced automation and human expertise can work in harmony to set a new standard for trustworthy, next-generation market intelligence.”

Aiera is the only platform with both real-time and human-edited transcripts available for licensing to downstream clients and/or API for quantitative analytics. This would include data providers, research platforms, and investment workflows, enabling firms to integrate verified event data directly into their models, dashboards, and decision systems.

Today’s announcement follows recent news of the accelerated availability of Third Bridge content on the Aiera platform.

About Aiera:
Aiera is the leader in generative AI event solutions for financial services professionals. Now backed by a first-of-its-kind consortium of leading Wall Street banks and research providers, Aiera integrates broker research, expert network content, financial news, transcripts, filings, and other public data sources into its technology and platform solutions. Trusted by many of the world’s top asset managers and financial institutions, Aiera delivers its capabilities through a secure dashboard, customizable components, and enterprise-grade APIs – empowering more informed, real-time decision-making across the investment workflow. Learn more at aiera.com.

Media Contact:
Carol Hickins
PR for Aiera
[email protected]

The post Aiera Sets New Industry Standard in Transcription Coverage, Speed, & Accuracy for Investor Events appeared first on Aiera.

LONDON, UK – September 17, 2025 – Third Bridge, the leading expert network, and Aiera, a pioneer in generative AI solutions for financial professionals, today announced the accelerated availability of Third Bridge content on the Aiera platform.

Following the formation of a groundbreaking consortium featuring ten of Wall Street’s largest investment banks and Third Bridge as the sole provider of expert network content in June 2025, this expedited integration marks an important milestone in Aiera’s development roadmap.

As the first new content provider on the Aiera platform, Third Bridge’s high-quality content library can now be seamlessly accessed, empowering investment professionals and industry analysts with even more informed, real-time decision-making capabilities across their critical investment workflows.

“Third Bridge is committed to providing decision-makers with the high-quality insights they need, exactly when and where they need them,” said Mike Grubert, Managing Director at Third Bridge. “By accelerating this integration with Aiera, we can embed our expert insights directly into the AI-driven research workflows our clients already use, enhancing their ability to make informed decisions. We’re excited about this as the first of many future developments for this partnership.”

The collaboration highlights a shared vision for a long-term partnership that will benefit both companies and their clients.

“This is just the beginning of our work together,” said Chad Doerge, President and Chief Revenue Officer at Aiera. “We’re thrilled to deliver Third Bridge’s unique content to the platform, accelerating our deployment timeline to provide clients with a new level of data engagement that goes far beyond a typical distribution channel. We look forward to building on this foundation with more joint developments in the future.”

This development is the first of many for the Aiera consortium, which aims to enhance language model accuracy, fortify intellectual property protections, and deliver a unified, cutting-edge solution for clients integrating generative AI into their workflows.

“Expert insights are one of the core content sets that financial professionals rely on every day. By partnering directly with content owners, we ensure that products are tested and validated for accuracy, by those who authored the original content, before going to market,” said Gavin Skinner, Chairman and COO at Aiera. “Over the coming months, Aiera will be expanding its core platform and technology offerings, as we continue to deliver on our promise to bring responsible generative AI solutions to the financial services industry.“

About Third Bridge:

Third Bridge is a leading global investment research firm established in 2007, with a team of over 1,500 employees worldwide, dedicated to fueling decisions with expert insights. We accelerate and enhance decision-making for investors and business leaders by unearthing unique expert insights across multiple sectors, geographies, and topics. For nearly 20 years, we’ve helped clients access knowledge on demand from experts, in-person and through our Library covering over 65,000 companies.

For media enquiries, please contact us at [email protected].

About Aiera:

Aiera is the leader in generative AI event solutions for financial services professionals. Now backed by a first-of-its-kind consortium of leading Wall Street banks and research providers, Aiera integrates broker research, expert network content, financial news, transcripts, filings, and other public data sources into its technology and platform solutions. Trusted by many of the world’s top asset managers and financial institutions, Aiera delivers its capabilities through a secure dashboard, customizable components, and enterprise-grade APIs – empowering more informed, real-time decision-making across the investment workflow. Learn more at aiera.com.

The post Accelerated access to Third Bridge expert insights on the Aiera platform appeared first on Aiera.

This marks an exciting milestone for Aiera as we officially announce our consortium with ten of Wall Street’s largest investment banks and research providers, including Third Bridge, and Microsoft as our strategic technology partner. This powerful collaboration sets the stage for Aiera to redefine how AI will support financial research – driven by innovation, built for compliance, and designed to serve the industry as a whole.

The financial services sector is undergoing a generational transformation. The rise of large language models and generative AI is reshaping how research and data are created, distributed, and consumed. This disruption brings tremendous promise, but also critical questions around content ownership, IP protection, entitlements, and model accuracy.

At Aiera, we are meeting this moment with purpose. Our mission is to set a new standard for responsible, intelligent AI in financial services. We are building a comprehensive platform powered by best-in-class generative AI, trained on proprietary content – with permission – from the world’s leading research providers, including expert network and financial news media, along with our own proprietary event transcripts, event documents and corporate filings. This provides Aiera with a truly differentiated foundation for innovation.

We are actively collaborating with our new partners to develop solutions that ensure:

• Content creators retain control of their intellectual property.
• Generative AI tools reflect original sentiment, advice, and recommendations.
• Every response is accurate, attributable, and entitlements-aware.
• Metrics and transparency are delivered back to content owners.

Specifically, the new Aiera experience will leverage specialized financial language models, secure chat interfaces (with in-line citations), and configurable agentic workflows. It will enable intelligent content discovery, summarization, generation, and review, all while being delivered via dashboards, mobile apps, and APIs. Aiera is proud to be tasked with setting new industry standards in model accuracy, IP protection, content sanitization, and source traceability – built on a uniquely deep and permissioned content base, with what will be the most rigorous validation process.

These capabilities will serve the full financial ecosystem – sell-side, buy-side, corporates, aggregators, and FinTech platforms. While the rollout will occur over the next 6–9 months, many capabilities are already available on the existing Aiera platform, and we invite you to join us on this exciting journey.

Sincerely,

• Gavin Skinner, Chairman & COO
• Ken Sena, Co-Founder & CEO
• Bryan Healey, Co-Founder & CTO

The post Letter from Aiera on Our Series B appeared first on Aiera.

New York, NY | Jun 9, 2025 – Aiera, the leader in generative AI event solutions for financial professionals, today announced the successful closing of $25 million in Series B funding. Led by a consortium that includes ten of Wall Street’s largest investment banks and one of the industry’s preeminent global expert network firms, Third Bridge, Aiera aims to transform AI-driven financial research solutions in a collaborative and compliant manner.

As the financial industry undergoes a fundamental shift in how research is created, distributed, and consumed, Aiera is committed to accelerating AI innovation by enhancing language model accuracy, strengthening intellectual property protections, and providing a unified solution for all customers integrating generative AI into their workflows.

“By bringing together the world’s top financial research institutions, we are ensuring AI innovation is developed responsibly within the highly regulated research domain,” said Gavin Skinner, Chairman and COO of Aiera. “Our focus is on clear standards for intellectual property protection, entitled content access, accurate chat-prompt responses, and citation integrity. Aiera’s technology solutions will be made available to all content creators, content consumers, existing content aggregators, and new and emerging platforms within the financial services industry.”

As part of this transformative initiative, Microsoft has joined Aiera as its strategic technology partner, providing cutting-edge Azure infrastructure to power Aiera’s next generation of GenAI solutions.

A Buy-Side Advisory Council has also been established, composed of senior leaders from a dozen of the world’s largest long-only and hedge-fund firms. This council will play a key role in shaping Aiera’s technology offering by providing strategic insights, development recommendations, and product testing feedback.

“This marks a defining moment in our industry,” said Ken Sena, CEO and Co-Founder of Aiera. “Together with our partners, we are building a secure and scalable AI ecosystem – one that enhances the value of research, safeguards intellectual property, and empowers more informed, real-time decision-making. We’re honored by this collective commitment and excited to lead this next chapter of industry innovation.”

About Aiera:
Aiera is the leader in generative AI event solutions for financial services professionals. Now backed by a first-of-its-kind consortium of leading Wall Street banks and research providers, Aiera integrates broker research, expert network content, financial news, transcripts, filings, and other public data sources into its technology and platform solutions. Trusted by many of the world’s top asset managers and financial institutions, Aiera delivers its capabilities through a secure dashboard, customizable components, and enterprise-grade APIs – empowering more informed, real-time decision-making across the investment workflow. Learn more at aiera.com.

Media Contact:
Carol Hickins
PR for Aiera
[email protected]

The post Aiera Closes Series B Funding Round with Ten of Wall Street’s Largest Research Firms Joining Board appeared first on Aiera.

New York, NY | February 26, 2025 – Aiera, a leading provider of AI-powered event streaming and analytics for institutional finance, is excited to announce expanded availability through key authoring solution partners – BlueMatrix, Eidosmedia, and Mediasterling. Alongside this expansion, Aiera is introducing the ability for publishers to embed interactive, verifiable excerpts from corporate event transcripts directly within research reports, articles, and presentations – which Aiera is calling “Transcrippets.”

Enhancing the Research Experience with Embedded Event Insights

‍“Sometimes, what is said in real-time carries nuances that don’t fully translate when read,” said Gavin Skinner, Chairman and Chief Operating Officer at Aiera. “By integrating not just the transcript, but the actual audio into the authoring experience, we’re giving analysts a more dynamic way to enhance their research and engage their clients.”

Transcrippets offers an interactive, multimedia experience within published research. Each Transcrippet contains key contextual information – including speaker details, company name, ticker, event type, date, and point-in-time stock price – alongside an embedded play button for instant audio playback. This seamless integration ensures that analysts and investors can access market-moving insights without leaving the research report.
‍

Example of Transcrippet from Mastercard (MA) Q4 2024 Earnings Call

To explore Aiera’s Transcrippets and broader offerings, click here. For media inquiries or to learn more about publishing with Aiera, journalists can reach out to [email protected].

About Aiera:
Aiera delivers a comprehensive AI-powered streaming and insights platform covering over 13,000 equities and 45,000+ annual events, including corporate earnings calls, government and central bank announcements, and investor conferences. Since its founding in 2017, Aiera has redefined access to market-critical information through advanced natural language processing, automatic speech recognition, and proprietary AI models. Trusted by top global asset managers, Aiera’s platform provides accurate, actionable insights tailored for institutional finance, available through an intuitive interface, robust APIs, and customizable components.
‍
About BlueMatrix:
BlueMatrix is the industry standard and the world’s largest publisher of investment research. For over two decades BlueMatrix connects the world’s leading asset and wealth managers with nearly 1,000 research firms in more than 50 countries and serves internal teams across multi-national corporations from its offices located in Durham (HQ), New York, London, Edinburgh, and Timisoara. BlueMatrix facilitates the equitable exchange of critical investment insights by improving the efficiency, collaboration, and security across the complete information lifecycle. The ecosystem is designed to meet users’ bespoke needs, from compliance tracking to interactive publishing, by removing friction from the publication, dissemination, consumption and application of investment research content. For more information, visit BlueMatrix’s website at bluematrix.com.
‍
About Eidosmedia:‍
Eidosmedia is a leading global provider of enterprise content management solutions, specializing in capital markets. Eidosmedia solutions are used by four of the top ten global investment banks, leading rating agencies and independent research providers, as well as portfolio managers and analysts at major asset management and wealth management companies globally, in addition to the principal business publications in Europe and the USA. Trusted by over 8,000 investment research professionals worldwide, Eidosmedia empowers capital market participants to digitize the information supply chain, enhancing efficiency through its enterprise-grade intelligent automation platform. Learn more at eidosmedia.com/financial-services.
‍
About Mediasterling:
Mediasterling is a software provider specializing in research authoring, workflows, and client portals. The company has over 25 years’ experience implementing design-driven technology solutions for financial and professional services firms to create high quality, branded content – from investment research, to pitches and presentations, to report templates, and more. Learn more at mediasterling.com.
‍
Media Contact:
Carol Hickins
PR for Aiera
[email protected]

The post Aiera Expands Availability to BlueMatrix, Eidosmedia, and Mediasterling with Innovative New Feature appeared first on Aiera.

[Written by Jacqueline Garrahan, Director of Machine Learning at Aiera | February 4, 2025]

The recent release of DeepSeek’s R1 model has sparked considerable discussion in both Silicon Valley and Wall Street. Perceived shifts in AI training economics have resulted in major market movements, most notably Nvidia’s record single-day market capitalization loss, with shares falling nearly 17% following DeepSeek’s announcement.

DeepSeek, a Hangzhou-based AI research company, began their journey as an AI-focused hedge fund backed by Chinese hedge fund High-Flyer, before pivoting to fundamental AI research in 2023. In the past month, DeepSeek has made two significant model releases. First, their DeepSeek-V3 model, released on December 26, 2024. Second, their January 20, 2025 release of DeepSeek-R1, which garnered significant market attention and spurred wide AI stock selloffs. In their V3 technical notes, DeepSeek reported training consumption of 2.788 million GPU hours on Nvidia H800 chips. At an estimated cost of $2 per GPU hour, this totals approximately $5.576 million, a significant fraction of Altman’s self-reported $100 million required for the training of GPT-4 in 2023.

But, the machine learning landscape has changed dramatically since 2023 and recent market reactions appear to be based on misunderstandings about AI economics rooted in that bygone era. Anthropic CEO Dario Amodei has publicly stated that training Claude 3.5 Sonnet – currently the top-performing model in Aiera’s benchmarks – cost “a few $10M’s”. So, rather than being “AI’s Sputnik moment”, as Marc Andreessen characterized the R1 release in an X post on January 26, 2025, DeepSeek’s releases are a natural continuation of research-based improvements in model architecture. The results align with the industry’s modern training paradigms, where:

1. Initial pretraining remains resource-intensive
2. Secondary reinforcement learning stages for reasoning capabilities are relatively inexpensive
3. Mixture of experts architectures are enabling more efficient scaling for model intelligence

These developments imply that the relationship between computational resources and model performance is evolving in ways that may not be fully reflected in current market valuations, though hardware remains crucial for AI development.

‍Aiera’s Benchmark Results

‍
According to our latest benchmarks, the AI landscape continues to be led by Anthropic’s Claude 3.5 Sonnet (released October 2024), with a cumulative score of 78.04% across our core metrics. Notably, DeepSeek’s R1 scores only second to Claude 3-5 Sonnet on our most reasoning intensive task, the Financial Q&A benchmark, which requires both comprehension and analysis against dense financial text.

According to technical notes, DeepSeek’s R1 was trained using reinforcement learning with chain-of-thought reasoning from DeepSeek-V3 checkpoints. Interestingly, while reasoning is improved, we see regressions on all other measured benchmarks of which all are less complex than the Financial Q&A task.

At Aiera, this regression underscores our conviction that task-specific benchmarking is integral to product-fit.

Like Dario Amodei highlighted in his recent blog post, DeepSeek’s December release of DeepSeek-V3 marked a more significant moment than its successor, demonstrating innovations in both architecture and training efficiency. DeepSeek-V3 introduced a Mixture-of-Experts (MoE) architecture with 671B total parameters, but only 37B activated per token, optimizing efficiency while maintaining high performance. Multi-Head Latent Attention (MLA) reduced memory overhead, Auxiliary-Loss-Free Load Balancing was used for better MoE stability, and Multi-Token Prediction (MTP) employed to enhance inference speed. Training was highly optimized through FP8 mixed precision, and the application of the DualPipe parallelism algorithm minimized idle GPU time, and introduced near-zero communication overhead for cross-node MoE training, which reduced computational costs significantly. Post-training improvements included distillation from DeepSeek-R1 for better reasoning, supervised fine-tuning, and reinforcement learning to align the model with human preferences.

These innovations are remarkable in that they demonstrate humankind’s ability to drive the frontier of the intelligence-to-compute capabilities, but these innovations hardly exist in a vacuum. DeepSeek’s research built upon previous research conducted by Google, Nvidia, Microsoft, Meta, and Toshiba among others.

‍The Bigger Picture: An Evolving Landscape

‍

The DeepSeek story illuminates several key trends shaping the future of AI development. First, the democratization of AI capabilities is accelerating, driven by architectural innovations rather than raw computational power. This shift challenges the conventional wisdom that bigger is always better, suggesting instead that targeted improvements in specific capabilities may be the next frontier of AI advancement.

However, the story isn’t just about technical capabilities. DeepSeek’s success, built on their prescient accumulation of 10,000 Nvidia GPUs before export controls, highlights the continuing importance of hardware access in AI development.

This presents a complex picture for investors: While training costs are declining, the strategic value of chip access remains high.

The market’s dramatic reaction to DeepSeek’s releases reflects an industry in transition.

The AI landscape is moving from an era where progress was measured primarily by model size and training costs to one where architectural innovation and targeted optimization drive advancement. This evolution suggests that future market leaders may emerge not from who can spend the most on training, but from who can innovate most effectively in model architecture and training methodology.

Want to learn more? Book a demo of the Aiera platform today.

The post DeepSeek’s R1 Release: Separating Signal from Noise appeared first on Aiera.

[Written by Ken Sena, Co-Founder and CEO of Aiera | December 13, 2024]

At Aiera, our ML team benchmarks all new language model releases. So, we thought it would be a good time to revisit our scoring across the major providers, and provide understanding as to performance and price. Specifically, we use these benchmarks to gain insight into:

1. Performance and perceived scale limitations

2. Pricing efficiencies based on newer model architectures being released

Notably, Google’s newest release, Gemini Flash 2.0 Experimental, stands out to us as being among the most economically efficient for the industry, while ranking in the top 5 for us as far as absolute performance. Why this is and how we get to this conclusion requires an understanding of our benchmarking approach, which we will review.

Aiera LLM Benchmarking

The Aiera ML team has developed proprietary tests to generate benchmarks on a variety of financial domain tasks, including sentiment, summarization, Q&A, and speaker identification. Our benchmarking informs which models we deploy for which tasks, offering key flexibility and responsiveness to our platform and to our users, who rely on the quality of these model-derived outputs.

As our users tend to be institutional investors, they know well that the quality of a language model output, measured in sourcing accuracy and citations, requires transparency.

We have developed a standardized view across the major model providers, offering contextually relevant, accurate, and well-cited information tailored to our specific use cases. Further, while benchmark performance does not always translate directly to real-world utility, by holding the tests constant across providers, it does offer comparison.

Comparison of Model Providers

Aiera has benchmarked 28 unique models to date, with measured performance reflecting a quantitative assessment on the quality of outputs across many critical features of the Aiera platform, including sentiment, summarization, Q&A, and speaker identification.

As part of this process, we derive an Aiera score, which is set on a scale of 0 to 100 and based on the four measured benchmarks.

While each model is scored individually, we can compare providers in terms of maximum individual scores achieved, where OpenAI, at 11 models evaluated, had the most opportunities, followed by Anthopic and Google at 6 each.

Performance: Are We Seeing Scaling Limitations?

A question that often comes up is whether we are seeing scale limitations based on current benchmarking. This is a difficult question to answer because most models are closed-source where we don’t know the exact parameter size. However, if we take one model series, Anthropic’s Claude, we find performance improvements to be mixed. Specifically, with more recent releases, we see continued improvement around Speaker Identification and Q&A, but diminished, or even some declining performance, where Sentiment and Summarization are concerned.

Price: Are We Seeing Efficiency Gains?

Again, large providers such as OpenAI, Anthropic, and Google operate as closed-source entities, offering limited transparency into their pricing models. This makes it challenging to assess the true cost-efficiency or alignment of their resource allocation with the underlying architecture and operational expenses. However, by evaluating the cost of these models relative to our benchmarked performance, we feel we can glean some insight into comparative efficiency.

Model intelligence is a function of the complexity of the composing neural network. Larger models perform better than their smaller counterparts with similar architectures due to their increased capacity to capture and generalize complex patterns in data. However, this performance gain often comes at the cost of higher computational requirements, longer training times, and greater energy consumption, underscoring the need for efficient architectures that optimize both size and performance.

Competing architectures, including transformer-based models, RNN variants, and emerging approaches like mixture-of-experts (MoE), aim to balance scalability, interpretability, and computational efficiency, with MoE models standing out for their ability to activate only a subset of parameters dynamically, offering enhanced performance while more effectively managing resource consumption.

In other words, by throwing cost-efficiency into the mix, model selection becomes much more interesting.

Google Gemini 2.0 Flash Experiment – Thesis Supporting

To calculate cost, we multiply the quantity of input and output tokens by their cost (in millions). We then standardize the cost distribution between 0 to 100, where the Gemini Flash models are far less expensive because they leverage a hybrid architecture combining the strengths of transformer-based architectures (used in models like GPT) with advanced techniques for fine-tuned efficiency and performance. Moreover, whereas historically these lighter-weight faster models tended to be less performant compared to the larger model sizes, we were impressed to find that yesterday’s Gemini 2.0 Flash Exp release ranked in our top 5 for the year for overall performance while likely at a fraction of the cost.

We look forward to sharing more model updates to come as we focus on delivering the best possible performance for Aiera and our users.Enjoy the holidays!

The post Aiera Year End Language Model Benchmarking Review 2024 appeared first on Aiera.