If it closes at those terms, Cursor would nearly double its November 2025 valuation of $29.3 billion in under six months. Eighteen months ago, the company was worth $9.9 billion.

The Numbers That Got Investors Here

The case for a $50 billion valuation starts with revenue growth that has almost no precedent in enterprise software. Cursor crossed $500 million in annualized revenue by May 2025, reached $1 billion by October, and surpassed $2 billion in February 2026. The company now projects ending the year above $6 billion ARR - a tripling in under 12 months.

Sixty-seven percent of Fortune 500 companies use Cursor, generating around 150 million lines of enterprise code per day. Enterprise contracts account for 60% of revenue. Those enterprise accounts carry positive gross margins, which matters because Cursor's individual developer subscriptions still run at a loss.

The gross margin arc matters more than the headline ARR. For most of its short life, Cursor was a growth story with a margin problem: paying Anthropic, OpenAI, and other model providers a large share of every dollar it earned from developers. Two things changed that picture. Cursor launched its own Composer model in November 2025, reducing what it routes to third-party providers. It also integrated models from China's Kimi - cheaper per token than the US frontier providers - for tasks where raw coding performance is less critical.

The combination moved overall gross margins into slightly positive territory. Enterprise accounts, where clients pay annual contracts and run predictable workloads, now show meaningfully better margins than the developer consumer tier.

Who Benefits

The Founders and Existing Investors

Michael Truell and his three co-founders built Anysphere out of MIT without external funding until 2024. Their equity, and that of a16z and Thrive Capital who backed early rounds, has appreciated on a path that would be unusual in any market. At $50 billion, Anysphere would rank behind only OpenAI among US AI company valuations.

Truell, who's 25, has been unusually direct about his competitive strategy. Rather than pretending model providers are partners, he has publicly framed the company's goal as reducing its exposure to them. "We take the best intelligence that the market has to offer from many different providers," Truell told TechCrunch in December. "And we also do our own product-specific models in places." The Composer launch was the first concrete step. The funding round provides capital to accelerate that work.

Enterprise IT Buyers

More capital going into Cursor's enterprise engineering team benefits corporate customers directly. The company has been building spend controls, billing groups, usage dashboards, and compliance tooling - the infrastructure that large IT departments require before they can approve company-wide seat deals. With 67% of the Fortune 500 already using Cursor, renewal conversations are happening at scale for the first time.

Nvidia

Nvidia's strategic participation is worth reading carefully. The company invests in AI software startups partly to maintain demand for its GPUs. A Cursor processing 150 million lines of enterprise code per day runs on compute, and Nvidia has a clear interest in that compute budget. It's a different reason than the typical VC bet on returns - Nvidia is buying influence with a company that controls a meaningful slice of AI inference spend.

Cursor's Cursor 3 interface, launched in April 2026, allows developers to run multiple coding agents in parallel across local and cloud environments. Source: cursor.com

Who Pays

New Investors Buying at the Peak

The investors writing checks at $50 billion are making a specific bet: that Cursor can hold its market position as both model providers and hyperscalers push competing tools. That's harder than it sounds. Anthropic's Claude Code crossed $2.5 billion in annualized revenue by February 2026. OpenAI has unified Codex with its broader desktop strategy. Microsoft, which owns GitHub, is pushing Copilot into the same enterprise accounts that Cursor wants.

"It's pretty clear the market is standardizing on a couple solutions. It's a narrow field of folks really at scale here."

• Michael Truell, CEO of Anysphere, to Fortune, March 2026

Truell's framing is accurate, and it cuts both ways. If the market standardizes on two or three tools, Cursor could be one of them. Or Microsoft could bundle Copilot into existing enterprise agreements at a price Cursor can't match. New investors at $50 billion are paying for the optimistic scenario.

Anthropic and the Model Providers

Cursor's explicit goal of reducing reliance on Anthropic is a direct threat to Anthropic's model revenue. Cursor was one of Anthropic's largest API customers. Each Composer token that replaces a Claude token is revenue Anthropic doesn't see. This dynamic repeats across the AI stack: the most successful application-layer companies tend to eventually build their own models for the tasks they do at scale.

Individual Developers

The funding round doesn't change the economics of individual Cursor subscriptions right away, but the direction of travel is visible. Cursor's enterprise accounts carry positive margins. Its individual developer accounts do not. That cross-subsidy works until investors want profitability - which they'll eventually demand, especially with an IPO on the horizon. Individual pricing will likely go up before any Cursor listing.

Enterprise sales cycles and Fortune 500 adoption drove Cursor's shift from a developer tool to a boardroom purchase - and brought a different class of investor with it. Source: unsplash.com

Counter-Argument: What Bears Will Say

The bear case on Cursor has been consistent since the company's first large valuation: the product's core function - helping developers write code with AI assistance - is a commodity waiting to happen. Every major model provider wants to control the coding interface. Apple Silicon, AMD, and Intel are all chasing inference cost reductions that would make running local models viable at enterprise scale, removing Cursor's control over model selection.

The company's response is to be the best end-to-end tool regardless of which model sits underneath it. Cursor's $2 billion ARR milestone was partly built on integrating the best available model at any given moment - switching from GPT-4 to Claude 3.5 to Gemini as each took the lead. The Composer model is an attempt to own the layer that matters most for code-specific tasks. Whether a proprietary coding model can stay competitive with Anthropic's and OpenAI's frontier research teams, with a fraction of the budget, is the real question.

At $50 billion, Cursor is a bet on enterprise software margins - and the terms will be set by whether Anysphere can make its own models good enough to stop paying the companies most likely to kill it.

Sources:

• Cursor in talks to raise $2B+ at $50B valuation as enterprise growth surges - TechCrunch
• Cursor has reportedly surpassed $2B in annualized revenue - TechCrunch
• Cursor's Anysphere nabs $9.9B valuation, soars past $500M ARR - TechCrunch

Ox Security, a Tel Aviv-based application security firm founded by former Check Point executives Neatsun Ziv and Lion Arzi, published its findings on April 15. The research catalogues 30+ responsible disclosures, 10 CVEs (all rated critical or high), and live proof-of-concept exploitation on six production platforms. The root cause isn't a bug in any one implementation. It's how MCP's STDIO transport was designed from the start.

How STDIO Transport Actually Executes Commands

MCP supports two transport mechanisms: HTTP+SSE for remote connections and STDIO for local ones. When a host application - Claude Code or Cursor, for instance - needs to spawn a MCP server, it passes a command string to the STDIO interface. That interface launches the subprocess, then waits for the MCP handshake to succeed.

The problem is that the command executes before the handshake check.

# Simplified execution sequence with a malicious command string:
1. MCP STDIO receives: "malicious-payload && exfiltrate-keys"
2. OS executes the command string  ->  payload runs
3. MCP handshake fails (no valid server started)
4. Error is returned to the caller
# The payload already ran. The error doesn't matter.

Ox researchers describe this as "execute first, validate never." Any user-controlled string that reaches the STDIO configuration layer becomes a potential RCE vector. No authentication required. No sandboxing applied. This behavior runs through all ten official MCP SDK implementations.

The Four Attack Families

1. Direct UI Injection

The simplest path. Frameworks like LangFlow (CVE pending) and GPT Researcher (CVE-2025-65720) expose web interfaces that accept MCP configuration directly. For LangFlow, no account is needed - a session token is "freely available," per Ox's report. Sending a crafted payload through the UI gets arbitrary code running on the server. Agent Zero (CVE-2026-30624) falls to the same class of attack via its unauthenticated interface.

2. Hardening Bypass

Some tools tried allowlisting to restrict which commands could run. Flowise (GHSA-c9gw-hvqq) had one - researchers bypassed it by routing commands through npx's -c flag, which the allowlist didn't cover. Upsonic (CVE-2026-30625) fell to a similar approach. The lesson from both: if the underlying STDIO execution model doesn't confirm, surface-level allowlists provide false confidence.

3. Prompt Injection to Local RCE

This is the attack chain that reaches the tools most developers use every day. An AI coding assistant processes content from a web page, a document, or a code repository. That content carries a prompt injection payload instructing the assistant to modify its MCP server configuration. The assistant writes the malicious command to STDIO config. STDIO executes it.

Windsurf received CVE-2026-30615 for a zero-click variant - visiting a malicious website was enough to trigger the full chain. Claude Code, Cursor, GitHub Copilot, OpenAI Codex, and Gemini CLI require some user interaction, but Ox confirms all are within the exposure window. Their respective vendors classified this as "by design."

The attack chain turns AI coding assistants into the delivery mechanism - the same tool that reads your source code can be instructed to execute arbitrary commands on your machine. Source: unsplash.com

4. Marketplace Poisoning

Ox tested 11 MCP marketplace directories. Nine accepted malicious MCP entries without review. A poisoned entry sits in the directory waiting for developers to install it. Once installed, it can substitute a legitimate SSE transport connection for a local STDIO connection. At that point, the attacker controls what runs on the developer's machine.

Confirmed CVEs

Claude Code, Cursor, GitHub Copilot, and Gemini CLI don't appear in the CVE list - not because they're safe, but because their vendors declined to assign CVEs, treating the prompt-injection path as an accepted risk in the current threat model.

MCP-enabled IDEs are high-value targets: they run with local file access, network access, and often terminal permissions on the developer's machine. Source: unsplash.com

The Disclosure Timeline

• November 2025 - Ox Security begins investigating MCP's STDIO transport behavior after observing inconsistent execution semantics.

• January 7, 2026 - Initial contact with Anthropic. Researchers present the root issue and propose architectural fixes that would protect all downstream implementations.

• January 16, 2026 - Anthropic updates its SECURITY.md, recommending caution with STDIO adapters. No protocol changes made.

• January - March 2026 - 30+ responsible disclosures filed across affected projects. LiteLLM, DocsGPT, Flowise, and Bisheng issue patches for their specific CVEs.

• March 18, 2026 - LangFlow formally acknowledges the issue.

• April 15, 2026 - Ox publishes the full report. The MCP protocol architecture remains unchanged.

Where It Falls Short

The standard defense Anthropic and affected vendors are offering is "sanitize your inputs." That's correct advice in isolation - developers building MCP servers should absolutely validate anything that reaches command execution. The problem is what it means in practice for the broader ecosystem.

Over 200 open-source projects inherited this exposure by building on Anthropic's official SDKs. A single architectural fix at the protocol level would have protected all of them. Ox's researchers put the ask clearly: "One architectural change at the protocol level would have protected every downstream project, every developer, and every end user."

Anthropic had five months between initial disclosure and public release to implement that fix. The company added a warning to its documentation instead.

This fits a pattern that's become familiar in the MCP ecosystem. A command injection flaw in the Kali Linux MCP server showed up in February, affecting penetration testing tools running with elevated permissions. Multiple RCE paths in Claude Code itself were disclosed earlier this year. Earlier still, the LiteLLM supply chain compromise showed how a single widely-used inference layer becomes the delivery mechanism for downstream attacks. MCP's footprint is much larger than any of those - at 97 million monthly SDK downloads across all language implementations, the scope of a unaddressed protocol-level flaw is substantial.

What Developers Should Do

The upstream fix isn't imminent. Mitigations to apply now:

1. Restrict MCP services to private IP addresses - don't expose MCP endpoints to the public internet under any circumstances
2. Treat all MCP configuration as untrusted - any user-controlled string reaching STDIO config is a potential attack vector
3. Sandbox MCP processes - run with minimal permissions, isolated from credentials, secrets, and sensitive file paths
4. Audit marketplace installs - verify MCP entries before installation; prefer packages with active maintainers and documented security review
5. Apply available patches - LiteLLM, DocsGPT, Flowise, and Bisheng have addressed their specific CVEs; update right away
6. Log tool invocations - monitoring what MCP-enabled tools are actually executing makes exploitation visible before damage spreads

Anthropic has not announced any architectural changes to the protocol today. The 10 CVEs issued so far represent confirmed exploits on real production systems; Ox expects additional CVEs as more projects are audited.

Sources:

• Anthropic won't own MCP 'design flaw' putting 200K servers at risk - The Register
• Systemic Flaw in MCP Protocol Could Expose 150 Million Downloads - Infosecurity Magazine
• Flaw in Anthropic's MCP putting 200k servers at risk, researchers claim - Computing
• MCP Supply Chain Advisory - Ox Security
• Anthropic's MCP Design Flaw Enables Remote Code Execution - Cyber Kendra
• RCE by design: MCP architectural choice haunts AI agent ecosystem - CSO Online

MoE Routing Topology Doesn't Determine Quality

Authors: Ivan Ternovtsii and Yurii Bilak | arXiv:2604.14419

Sparse Mixture-of-Experts (MoE) architectures have become a standard tool for scaling LLMs without proportionally scaling compute. The design assumption is that routing - the mechanism that dispatches tokens to specific experts - is one of the main levers for quality. A more sophisticated router, the logic goes, means better expert specialization and better outcomes.

Ternovtsii and Bilak ran 62 controlled experiments on WikiText-103 to test that assumption directly. They built ST-MoE, a geometric routing system that uses cosine-similarity matching against learned centroids in a 64-dimensional space, then compared five variants of this approach against standard linear routers.

The Equifinality Result

All five cosine-routing variants landed within a 1-perplexity margin of each other (range: 33.93 to 34.72 PPL). Using rigorous Two One-Sided Tests (TOST) across 15 runs and 3 seeds, the differences were statistically equivalent - not just small, but provably negligible.

Standard linear routers achieved 32.76 PPL, which is noticeably better. But they used 5.3 times more routing parameters to get there. When the researchers built an iso-parameter cosine routing variant, it closed 67% of that gap. The mechanistic advantage of the linear router narrowed to roughly 1.2%.

The multi-hop routing updates showed cosine similarity of 0.805 between update vectors - essentially pointing in the same direction. They're doing magnitude amplification, not compositional computation. A single learnable scalar reproduced that behavior.

The most actionable finding is the zero-shot relative-norm halting result: by stopping routing computation early based on a relative-norm criterion, the system saved 25% of MoE FLOPs with only a 0.12% PPL degradation. That's a free efficiency gain available right now, independent of routing design choices.

Network routing patterns - the complexity of routing mechanisms in MoE architectures may matter less than researchers have assumed. Source: unsplash.com

For anyone doing architecture search on MoE systems, this is a significant reallocation of effort. The routing mechanism isn't where the performance comes from. Earlier work at Awesome Agents covering MoE routing myths and context compression reached adjacent conclusions - this paper sharpens the argument with controlled experimental evidence.

Prompt Optimization Is a Coin Flip

Authors: Xing Zhang, Guanghui Wang, Yanwei Cui, Wei Qiu, Ziyuan Li, Bing Zhu, Peiyang He | arXiv:2604.14585

Automated prompt optimization frameworks like DSPy and TextGrad are now a standard part of many LLM pipelines. The assumption is that running optimization over prompts improves downstream task performance. You pay compute costs upfront, save on manual prompt engineering, and get a better system.

Zhang et al. tested that assumption across 72 optimization runs (6 methods × 4 tasks × 3 repeats) and 18,000 grid evaluations across 144 additional optimization runs, using Claude Haiku and Amazon Nova Lite as their target models.

The 49% Failure Rate

On Claude Haiku, 49% of optimization runs scored below zero-shot baseline. Amazon Nova Lite showed an even higher failure rate. The optimization methods included approaches from the DSPy and TextGrad ecosystems with four other automated techniques.

One task was different: all six methods improved by up to 6.8 points over zero-shot. That divergence is the key to understanding when optimization works and when it doesn't. The successful task had what the authors call "exploitable output structure" - a format the model can produce but doesn't default to. The optimizer discovers and enforces that format. Tasks without that property don't benefit.

The team also tested a popular assumption underlying these frameworks: that agent prompt interactions are a meaningful optimization target. Across all experiments, those interactions were never statistically significant (p > 0.52, all F-statistics below 1.0). The coupling between prompts in multi-agent systems, which tools like TextGrad attempt to optimize jointly, doesn't show up as a real effect.

The Diagnostic

Rather than abandoning prompt optimization, the paper proposes a two-step check before running an expensive optimization campaign:

1. An $80 ANOVA pre-test to assess whether agent coupling is real for the specific task
2. A 10-minute headroom test that checks whether the task has exploitable output structure

If neither condition holds, optimization is unlikely to beat zero-shot. The diagnostic costs less than a single failed optimization run.

For anyone building compound AI systems, this finding is directly actionable. The prompt engineering basics assumptions worth questioning aren't about which optimizer to use - they're about whether optimization is the right move at all for a given task.

GUARD: Finding Where Reasoning Actually Breaks

Authors: Wei Zhu, Jian Zhang, Lixing Yu, Kun Yue, Zhiwen Tang | arXiv:2604.14528 | Accepted at ACL 2026

Extended reasoning chains in LLMs - the kind generated by AI reasoning models - are assumed to reduce errors by allowing more computation. When a model fails, the implicit model is that errors build up gradually, or appear late in the chain when complexity builds up. Neither assumption is quite right.

Zhu et al. studied the distribution of reasoning errors across long inference chains and found something more tractable: "errors are not uniformly distributed but often originate from a small number of early transition points." After one of these critical junctures, the reasoning stays internally consistent - it's just consistent with a wrong premise.

The Entropy Signal

These failure points correlate with localized increases in token-level entropy. The model hesitates, in a measurable sense, and the direction it picks at that moment determines whether the rest of the chain succeeds or fails. Importantly, alternative paths from the same intermediate state can still yield correct answers - the fork hasn't committed the model to failure yet.

The GUARD framework uses these entropy signals to identify critical transitions at inference time and proactively redirect the reasoning toward alternative paths. Testing across multiple benchmarks confirmed that targeting these specific junctures improves reliability more efficiently than increasing total compute.

Reasoning chains in LLMs don't fail gradually - errors originate at specific early transition points where token-level entropy spikes. Source: unsplash.com

This connects to earlier work on reasoning traps and LLM instability, which documented failure modes without a clear intervention mechanism. GUARD provides one: a targeted inference-time redirect based on measurable uncertainty, not a blanket increase in sampling budget.

The practical implication is a shift in how to think about reasoning chain failures. The question isn't "how long should the chain be" - it's "where does the chain first deviate and can we catch it."

The Common Thread

None of these papers argue for simplicity as an end in itself. What they share is a case for measurement before investment. MoE routing design is worth measuring before assuming it drives quality. Prompt optimization is worth pre-testing before running an expensive campaign. Reasoning chain failures are worth locating before adding compute.

The diagnostic tools in all three papers are cheap relative to the resources they save. That's a pattern worth following: before adding machinery, check whether the machinery does what you think it does.

Sources:

• Equifinality in Mixture of Experts - arXiv:2604.14419
• Prompt Optimization Is a Coin Flip - arXiv:2604.14585
• Dissecting Failure Dynamics in LLM Reasoning (GUARD) - arXiv:2604.14528

This leaderboard covers the major browser-specific benchmarks as of April 2026. Each suite tests something slightly different: task complexity, website diversity, the role of vision vs. text, and tolerance for ambiguity. No single number tells the full story. Read the methodology sections before drawing conclusions.

Why Web Agent Benchmarks Differ from General LLM Evals

General benchmarks like MMLU or GPQA test whether a model knows things. Web agent benchmarks test whether a model can do things - navigate a real or simulated browser, interpret dynamic page content, chain actions across multiple steps, and recover from errors without human help.

AI agents operate on real or simulated browser environments, making web agent benchmarks a distinct category from static knowledge tests. Source: unsplash.com

This distinction matters for benchmark selection. A model that tops GPQA may be terrible at clicking through a checkout flow. The correlation between raw LLM capability and web agent performance exists but isn't tight - scaffolding, observation format, and action space all contribute as much as the underlying model.

For context on how web agents relate to full desktop automation, see the Computer Use Leaderboard: Desktop AI Agent Rankings.

Benchmark Overview

WebArena

The oldest major web agent eval. 812 tasks (241 templates, ~3.3 variations each) across four domains: e-commerce, social forums, code repositories, and content management. Tasks are long-horizon - "Find when your last order shipped and post an update to the forum thread" - with programmatic grading, no LLM judge involved. Scores are pass/fail success rate across all 812 tasks.

A verified, Docker-hosted version (webarena-verified) became available in February 2026, improving reproducibility. The original leaderboard at webarena.dev lists community submissions.

VisualWebArena

910 tasks built specifically for multimodal agents, where understanding what's on screen visually (images, product photos, UI layouts) is required to complete the task. Released by the same CMU group behind WebArena. At original publication in early 2024, the best VLM agent scored 16.4% against 88.7% human performance. Most current published results still cite the original paper rather than a live leaderboard.

WebVoyager

643 tasks across 15 popular websites - Google, Amazon, GitHub, Reddit, Wikipedia among them. Uses a dual evaluation approach: human annotation plus automated GPT-4V judgment. Published in 2024 with an original GPT-4V agent scoring 59.1%. The Steel.dev leaderboard now tracks live agent submissions against this benchmark.

Mind2Web / Mind2Web 2

The original Mind2Web (NeurIPS 2023) introduced 2,000+ open-ended tasks across 137 websites in 31 domains. Mind2Web 2, published at NeurIPS 2025, raised the bar far: 130 long-horizon tasks requiring real-time browsing and cross-site information synthesis, constructed with over 1,000 hours of human annotation. It uses an Agent-as-a-Judge framework with tree-structured rubrics. Best current system is OpenAI Deep Research at 50-70% of human performance.

Online-Mind2Web, a 2025 evaluation of 300 live tasks, showed that most commercially available agents underperform the academic SeeAct baseline from early 2024. The exceptions: Claude Computer Use 3.7 and OpenAI Operator at roughly 61% success.

BrowseComp

Released by OpenAI in April 2025. 1,266 hard information-retrieval problems designed to be nearly unsolvable by keyword search alone - tasks require multi-hop reasoning across multiple retrieved pages. At launch, GPT-4o with browsing scored 1.9%, o1 scored 9.9%, and Deep Research hit 51.5%. The full paper is available as a PDF. Updated leaderboard data tracked by llm-stats.com puts scores far higher for 2026 frontier models.

WorkArena / WorkArena++

ServiceNow's enterprise-focused benchmark. WorkArena covers 33 atomic tasks in a ServiceNow instance. WorkArena++ scales this to 682 multi-step compositional tasks requiring planning, retrieval, arithmetic reasoning, and memory across browser sessions. Human performance is 93.9% on WorkArena++. GPT-4o managed only 2.1%, and no model hits meaningful performance on the L3 (ticket-like, context-rich) task tier.

WebChoreArena

A 2025 extension of WebArena with 532 tasks focused on tedious, labor-intensive work: massive memory retrieval, calculation across pages, and long-term cross-page reasoning. Gemini 2.5 Pro scores 54.8% on WebArena but drops to 37.8% on WebChoreArena. GPT-4o manages only 2.6% on WebChoreArena versus 44.9% for Gemini 2.5 Pro, exposing a much wider performance gap than the base benchmark.

Rankings

WebArena - Tracked Model Scores (April 2026)

Scores from benchlm.ai, which tracks 15 models against the standard 812-task suite.

Note: Specialized agentic frameworks that wrap models can score higher. OpAgent (CodeFuse AI) reached 71.6% on WebArena using a Planner-Grounder-Reflector-Summarizer multi-agent pipeline with online reinforcement learning, holding the #1 leaderboard position in January 2026. DeepSeek v3.2 as an agent backbone (not raw model) hit 74.3% in the Steel.dev results index, which tracks end-to-end agent systems rather than raw model calls.

WebVoyager - Top Agent Systems (April 2026)

From the Steel.dev Browser Agent Leaderboard, tracking end-to-end agent submissions.

WebVoyager scores are approaching saturation. Scores above 90% are common enough that the benchmark no longer differentiates the top tier well.

BrowseComp - Model Scores (April 2026)

From the llm-stats.com tracker, which covers 40 models. BrowseComp scores are reported as fractions (0.0-1.0).

The jump from 0.515 (Deep Research at launch) to 0.869 in under a year is steep. BrowseComp remains the hardest widely-used browsing eval and hasn't saturated yet.

Web agents must interpret and act on complex data-rich interfaces like dashboards. Benchmarks like WorkArena++ specifically test this kind of enterprise SaaS interaction. Source: unsplash.com

Key Takeaways

Anthropic and OpenAI Trade Leads Depending on the Benchmark

Anthropic's models dominate WebArena and BrowseComp. OpenAI's ecosystem leads on WebVoyager through commercial products like Operator. Google's Gemini 3.1 Pro scores second on BrowseComp at 0.859 and shows competitive WebArena numbers. The gap between providers has closed significantly since late 2024.

Agentic Frameworks Beat Raw Model Calls by a Wide Margin

The 74.3% DeepSeek v3.2 agent score on WebArena versus the same model's 48.6% raw-model score shows what a well-designed scaffolding layer contributes. OpAgent's online RL pipeline - where the agent learns from task failures on the fly - represents the current state of the art for WebArena. Raw model API calls are not the right comparison point for production web agent deployments.

WebVoyager Has Saturated

Near-perfect WebVoyager scores no longer distinguish good systems from excellent ones. Scores in the 90-98% range are clustered, and the benchmark uses GPT-4V as a judge - which may not reliably distinguish between a 92% and a 97% agent. Researchers and practitioners should weight WebChoreArena (the harder variant) and BrowseComp more heavily.

Enterprise Web Tasks Remain Very Hard

WorkArena++ at 2.1% for GPT-4o makes the frontier model gap with humans (93.9%) vivid. The benchmark's L3 tasks - which mirror real ServiceNow workflows with complex context requirements - have effectively zero LLM success now. Any vendor claiming "autonomous enterprise agent" capabilities should be pressed on WorkArena++ L3 numbers.

Open-Source Agents Are Competitive

Browser Use (open-source) beat OpenAI's commercial Operator on WebVoyager, 89.1% vs 87%. The performance gap between open and closed systems that defined 2024 has mostly closed at the framework level. The underlying models still favor closed-source providers, but the agent scaffolding is no longer a meaningful differentiator.

Practical Guidance

For general web task automation

If you're building browser agents on top of frontier model APIs, Claude Opus 4.6 or GPT-5.4 as the backbone gives you the strongest raw capability. Pair either with the Browser Use open-source framework (see our Best AI Browser Automation Tools roundup) rather than building scaffolding from scratch. The open-source frameworks now match or exceed proprietary agent products on standard benchmarks.

For research on web agents

BrowseComp and WebChoreArena are the benchmarks worth tracking in 2026. WebVoyager provides a useful regression check but shouldn't be the primary eval. For enterprise-specific scenarios, WorkArena++ L2 is realistic; L3 results tell you how far you still have to go. Mind2Web 2 is the right choice for agentic search and long-horizon information tasks.

For commercial web agent products

If you're evaluating products like OpenAI Operator, Google Project Mariner, or Skyvern, ask for BrowseComp scores rather than WebVoyager scores. BrowseComp's hard information-retrieval problems separate agents that actually reason from agents that pattern-match. Our Best AI Browser Agents guide covers the commercial product landscape.

For open-source work

The Browser Use framework running on GPT-4o is the strongest open-source baseline. MolMo-Web (AI2) is a notable recent open-source web agent worth watching if you need a permissively licensed option. The BrowserGym ecosystem from ServiceNow provides a unified test harness across WebArena, WorkArena, VisualWebArena, and others - useful if you want reproducible comparisons across benchmarks.

FAQ

Which web agent benchmark should I use for evaluating my system?

Use BrowseComp for hard information-retrieval tasks, WebChoreArena for tedious multi-step tasks, and WorkArena++ for enterprise SaaS workflows. WebVoyager is near-saturated for top systems; use it as a baseline check only.

What's the difference between WebArena and WebVoyager?

WebArena uses four simulated websites with programmatic grading and 812 multi-step tasks. WebVoyager uses 15 live real-world websites with 643 tasks and GPT-4V automated judging. WebArena is more reproducible; WebVoyager tests live-web generalization.

Can open-source agents match commercial products on web tasks?

Yes, at the framework level. Browser Use (open-source) scored 89.1% on WebVoyager vs 87% for OpenAI Operator. The underlying model still matters, but the scaffolding gap has closed significantly.

How do WorkArena++ scores compare to real enterprise automation?

WorkArena++ L3 tasks have near-zero LLM success rates vs 93.9% for humans. No current LLM-based agent should be trusted for unsupervised enterprise workflow automation without significant human oversight.

How often do these rankings change?

WebVoyager and BrowseComp update frequently as new agent submissions arrive. WebArena updates more slowly. WorkArena++ results have been stable since mid-2025. Check Steel.dev and benchlm.ai for current snapshots.

Sources:

• WebArena: A Realistic Web Environment for Building Autonomous Agents
• WebArena GitHub Repository
• VisualWebArena GitHub Repository
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
• WebVoyager GitHub Repository
• Mind2Web 2 Benchmark Page
• BrowseComp Technical Paper (PDF)
• WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
• WorkArena GitHub Repository
• WorkArena++ NeurIPS 2024 Paper
• WebChoreArena Project Page
• AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
• Browser Use State-of-the-Art Technical Report
• Steel.dev AI Browser Agent Leaderboard
• OpAgent: Operator Agent for Web Navigation
• BenchLM.ai WebArena Leaderboard

What This Guide Covers

The consumer/SaaS tools reviewed here - ChatPDF, Adobe Acrobat AI Assistant, HumataAI, AskYourPDF, ChatDOC, PDF.ai, LightPDF AI, and Smallpdf AI - are aimed at professionals, students, and researchers who need to interact with documents through a chat interface.

The developer/API tools - Mistral OCR, LlamaParse, Reducto, Unstructured.io, Azure AI Document Intelligence, AWS Textract, Google Document AI, Marker, and Docling - are aimed at engineering teams building pipelines. They're evaluated differently: output format fidelity, table/equation accuracy, throughput, and cost per page matter more than chat quality.

If you're building a RAG pipeline and want to understand how these extraction tools fit into a retrieval architecture, see the best AI RAG tools guide. If structured data extraction from spreadsheets is your goal, the best AI data analysis tools guide covers that separately.

Consumer PDF Chat Tools

Comparison Table

ChatDOC

ChatDOC is the strongest consumer option right now. The free tier allows five document uploads per day at 200 pages each, with 300 questions daily - generous enough for real use. The Pro plan at $8.99/month adds GPT-4o access, formula recognition, and OCR for scanned documents. ChatDOC's citation feature traces answers to the specific page and passage, which matters when you need to verify an AI claim against a contract or technical spec.

For users who need to stay on budget, the add-on packages are a useful safety valve: extra files cost $0.29 each and extra pages cost $0.06 each, both valid for 90 days.

ChatPDF

ChatPDF is the entry point for many users and gets the basics right. Free usage requires no account: upload a PDF, start chatting. The 2-document daily limit on the free tier is workable for occasional use, and the smart routing between GPT-4o and GPT-4o-mini keeps response quality reasonable without inflating costs. The 120-page cap per file is a real constraint for long reports.

ChatPDF's strength is simplicity. There's nothing to configure, and sharing a secure link to a PDF-chat session takes seconds. It's not the best at long documents or multi-file analysis, but it remains the fastest way to extract a quick answer from a short PDF.

HumataAI

HumataAI is the option for students. The $1.99/month Student plan (with verified.edu email) covers 200 pages per month - enough for coursework and paper review. The Expert plan at $9.99/month is competitive for individual researchers. HumataAI's search and comparison features across multiple documents are stronger than ChatPDF's, but the free tier at 10 answers per month and 60 pages is too restrictive for real evaluation.

Adobe Acrobat AI Assistant

Adobe bundles the AI assistant as a $4.99/month add-on to any Acrobat subscription. This is the right pick if you're already paying for Acrobat Pro ($19.99/month) and do substantial PDF editing. The AI chat supports up to 10 files simultaneously at 600 pages each - the largest multi-file context window in this category.

The 2026 Acrobat Studio plan ($24.99/month) bundles AI features, PDF editing, and creative tools. Whether it's worth the premium depends completely on how much you use Acrobat for non-AI tasks. As a standalone PDF chat tool, you can find better value at lower price points.

AskYourPDF, PDF.ai, LightPDF AI, Smallpdf AI

AskYourPDF's paid plan at $11.99/month is solid value: 1,200 questions per day, 50 documents per day, up to 2,500 pages per document. The plugin ecosystem is an edge over competitors. PDF.ai at $15/month is clean and straightforward but doesn't offer anything that distinguishes it from ChatDOC or AskYourPDF at a lower cost.

LightPDF AI bundles 25+ PDF tools (convert, compress, edit) with AI chat. The $19.99/month price is harder to justify unless you need those utility tools with the chat capabilities. Smallpdf AI offers free unlimited basic Q&A without registration, which is useful for one-off summaries. Its EU hosting is a genuine differentiator for users with data residency requirements. The $12/month Pro plan unlocks advanced features.

Developer and API Extraction Tools

This is where accuracy benchmarks and per-page costs matter. The two main public benchmarks for this category are OmniDocBench (a CVPR 2025 benchmark covering text, tables, formulas, and reading order across nine document types) and Reducto's RD-TableBench (1,000 hand-labeled table images from varied public documents, scoring table similarity with a Needleman-Wunsch alignment algorithm).

Comparison Table

Mistral OCR (mistral-ocr-2503 / mistral-ocr-latest)

Mistral OCR is the best API pick for most document extraction workloads. In Mistral's internal benchmarks, the model scores 96.12% on table parsing, 94.29% on math, and 98.96% on scanned documents. On multilingual content, it hits 97.55% for Hindi and 97.11% for Chinese. The newer Mistral OCR 3 (released January 2026) improved accuracy on handwriting and forms, with a 74% win rate over OCR 2 in internal evaluations.

Mistral OCR rendering a complex multi-column table with figures. Output uses Markdown text with HTML table tags for structured cells. Source: mistral.ai

Pricing is $2 per 1,000 pages with the standard API (mistral-ocr-latest), dropping to $1 per 1,000 pages with the Batch API - the lowest among the major cloud providers. The API processes up to 2,000 pages per minute per node. A limited self-hosting option exists for customers with classified or highly sensitive workloads, but it's not generally available.

The output format deserves mention: Mistral OCR returns interleaved text and image references in Markdown, with tables as HTML, and supports structured JSON output for downstream use. This makes it directly usable in RAG pipelines without a separate parsing layer.

LlamaParse / LlamaIndex

LlamaParse runs on a credit system: 1,000 credits = $1.25. The cost per page ranges from $0.00125 (one credit, simple text extraction) to roughly $0.11 per page (90 credits, using a top-tier LLM agent like Sonnet for parsing). For most RAG workflows, the "cost-effective" mode at 3 credits per page ($0.00375 per page) is the practical baseline.

The 10,000 free credits on signup translates to roughly 3,300 pages at cost-effective mode - enough for a real pilot. LlamaParse is the natural fit if you're already using LlamaIndex for vector indexing and retrieval; the ecosystem integration reduces boilerplate. In March 2026, LlamaIndex also open-sourced LiteParse, a TypeScript-native local parser for agents that need zero-latency PDF parsing without cloud calls.

For complex layouts (financial tables, academic papers with equations), LlamaParse's premium agent mode is competitive, but Mistral OCR's batch pricing will be cheaper at scale.

Reducto

Reducto is built for production pipelines where table and form accuracy is critical. It combines traditional computer vision with vision-language models. On Reducto's own RD-TableBench, it scores an average table similarity of 90.2%. The benchmark is open-source (1,000 hand-labeled examples covering scanned tables, handwriting, merged cells, and multilingual content) and worth running against your own documents if you're evaluating vendors.

Pricing starts free for the first 15,000 credits, then moves to custom growth pricing. There's no public per-page rate - Reducto targets enterprises and pricing requires a conversation. HIPAA and SOC2 compliance, EU/AU data residency, and VPC deployment are available on paid tiers.

Unstructured.io

Unstructured offers both an open-source library and a managed platform. The open-source library is free to self-host and supports 60+ file types. The managed API charges $0.03 per page pay-as-you-go after 15,000 free pages - meaningfully cheaper than Azure or AWS for high-volume basic extraction. Compliance certifications (HIPAA, SOC2, GDPR, ISO 27001) make it viable for regulated industries.

The parsing strategy selection - Fast, Hi-Res, VLM, Auto - lets engineers trade speed against accuracy. Hi-Res and VLM modes handle complex layouts but at higher latency and cost. The open-source path is the cheapest option if you have the infrastructure to run it.

The Unstructured platform's no-code workflow builder. Engineers can also access the same processing via API without the UI layer. Source: unstructured.io

Azure AI Document Intelligence, AWS Textract, Google Document AI

These are the incumbent cloud offerings. They're battle-tested at enterprise scale but expensive compared to newer entrants.

Azure AI Document Intelligence: The Read model costs $1.50 per 1,000 pages, matching Google's OCR rate. Prebuilt models (invoices, receipts, contracts) run $10 per 1,000 pages. Custom extractors cost $30 per 1,000 pages for the first million pages, dropping to $20 afterward. Azure's advantage is deep integration with the Microsoft ecosystem and strong form-field extraction on standard document types.

AWS Textract: Basic text detection runs $1.50 per 1,000 pages. Table and form extraction (Analyze Document) ranges from $15 to $65 per 1,000 pages depending on features enabled. Volume discounts kick in above one million pages, dropping basic detection to $0.60 per 1,000 pages. Textract's table accuracy on Reducto's RD-TableBench was notably below Reducto and Mistral OCR in the benchmark results.

Google Document AI: OCR costs $1.50 per 1,000 pages (dropping to $0.60 above five million pages). Specialized processors - invoice parser, expense parser - each cost $0.10 per 10 pages ($10 per 1,000 pages). The Custom Extractor is $30 per 1,000 pages, same as Azure. Google's strength is language coverage and integration with Google Cloud workflows.

For teams already deep in AWS, Azure, or GCP, the convenience of staying in one cloud often justifies the price premium. For greenfield projects, Mistral OCR's accuracy and pricing make it hard to justify the incumbents on cost alone.

Marker and Docling (Open Source)

These are the two strongest open-source options for teams that want full control, zero per-page costs, and on-premises deployment.

Docling (IBM Research, Apache 2.0) outputs a structured DoclingDocument format that preserves semantic hierarchy - not just text, but the relationships between elements. It scored 0.882 on OmniDocBench text fidelity in evaluations. Docling reached 37,000 GitHub stars and is optimized for production RAG pipelines. It handles PDFs, DOCX, PPTX, HTML, and images.

Marker (MIT license, available at github.com/datalab-to/marker) scored 0.861 on OmniDocBench and supports an optional --use_llm flag that layers a LLM on top for accuracy-critical documents. Without the flag, it runs fast on CPU. With it, accuracy approaches commercial APIs. Marker is slower than Docling at scale (one benchmark put it at 53 seconds per page on complex academic documents vs Docling's single-pass approach), but the LLM enhancement mode is useful for isolated high-value documents.

Both tools are available via PyPI. Neither offers cloud hosting or SLAs - you're running the infrastructure.

Which Should You Use?

For one-off summaries and Q&A: ChatDOC free tier covers most needs. Use Adobe Acrobat AI if you already have an Acrobat subscription.

For student and research use: HumataAI's $1.99/month student plan or ChatDOC free tier. AskYourPDF for heavy multi-document work.

For production extraction pipelines: Start with Mistral OCR. It's the cheapest major cloud API with benchmark-backed accuracy. If you need deep LlamaIndex integration, add LlamaParse for complex layouts. For tables at enterprise scale with custom SLAs, Reducto.

For regulated or air-gapped environments: Unstructured.io open-source or Docling for self-hosted extraction. Azure Document Intelligence if regulatory requirements demand a commercial vendor with established compliance certifications.

For cost-sensitive high-volume OCR: Google Document AI's OCR tier ($1.50/1K pages) or Mistral's batch API ($1/1K pages). AWS Textract's advanced features are the most expensive in this group.

FAQ

Which AI PDF tool is most accurate on tables?

Mistral OCR leads at 96.1% in internal benchmarks. On Reducto's public RD-TableBench, Reducto scores 90.2%. Neither AWS Textract nor GPT-4o alone matches Reducto's table accuracy in that benchmark, and GPT-4o has documented hallucination issues on dense tables.

Can I use these tools with sensitive documents?

Unstructured.io, Docling, and Marker can run fully self-hosted. Reducto offers HIPAA/SOC2 compliance and VPC deployment on enterprise plans. Mistral OCR has a selective self-hosting option for classified workflows. Consumer tools like ChatPDF and HumataAI are cloud-only.

What's the cheapest way to extract text from PDFs at scale?

Mistral OCR's Batch API at $1 per 1,000 pages is the lowest public rate among cloud APIs. Self-hosted Docling or Marker are free, but you're paying for compute.

Does LlamaParse support non-PDF formats?

Yes. LlamaParse supports PDF, PPTX, DOCX, XLSX, HTML, JPEG, and more. Pricing and accuracy vary by file type.

What output formats do developer APIs produce?

Mistral OCR outputs Markdown with HTML tables and supports structured JSON. LlamaParse produces Markdown and JSON. Unstructured outputs JSON, HTML, and Markdown. Docling produces its own DoclingDocument format plus Markdown and JSON export. Azure, AWS, and Google all return JSON.

Sources

• Mistral OCR announcement and benchmarks
• Mistral OCR 3 release (January 2026)
• Reducto RD-TableBench benchmark
• Reducto pricing
• Unstructured.io pricing
• LlamaIndex / LlamaParse pricing
• Google Document AI pricing
• AWS Textract pricing
• Azure AI Document Intelligence pricing
• HumataAI pricing
• ChatDOC Pro plan
• ChatPDF
• OmniDocBench paper (CVPR 2025)
• Marker on GitHub
• Docling on GitHub

No single benchmark captures the full scope of how models fail on facts. TruthfulQA tests whether models parrot common misconceptions. SimpleQA probes short-form factual recall. FACTS Grounding measures faithfulness to provided source documents. The Vectara leaderboard tracks summarization-time hallucinations. AA-Omniscience penalizes wrong answers and rewards abstention. Together they give a more honest picture of where models stand.

Why Factuality Benchmarks Diverge

Before diving into the numbers, it helps to understand what each benchmark actually tests. They don't measure the same thing, and strong performance on one doesn't predict performance on another.

If you want the conceptual background, our guide to AI hallucinations explains the core failure modes, and our understanding AI benchmarks guide covers what benchmark scores can and can't tell you.

One important distinction runs through all of these: hallucination and factuality aren't the same thing. HalluLens (from Meta FAIR, published at ACL 2025) formalizes this: an extrinsic hallucination contradicts a model's own training data, while an intrinsic hallucination contradicts the context provided in the prompt. Benchmarks conflate these two failure modes frequently, which is why a model can look excellent on one test and poor on another.

TruthfulQA

TruthfulQA, introduced by Lin et al. in 2021, targets imitative falsehoods - wrong answers that feel plausible because they appear in human-written text. The 817 questions span 38 categories including health, law, finance, and politics.

The benchmark has a well-documented weakness: it's easy to game by training on similar questions, and contamination from public benchmarks is a real concern. Still, it remains widely reported and its MC2 scoring (normalized probability assigned to the set of true answers) is more robust than MC1.

Source: llm-stats.com, 17 self-reported results, no verified external evaluation.

The most striking pattern: Microsoft's Phi family controls the top three slots. Phi-3.5-MoE-instruct's 0.775 score stands well above the field. Notably absent from the top 10 are GPT-4o, Claude Opus, and Gemini - the headline frontier models. That's partly because TruthfulQA has become saturated for the largest models (who were likely trained with TruthfulQA-style data in mind), and partly because the leaderboard only has 17 submissions - closed model providers don't always report this score publicly.

The original paper documented an inverse scaling effect: larger models answered less truthfully on this benchmark, not more. That relationship has weakened with instruction tuning and RLHF, but it is a reminder that scale alone doesn't fix factuality.

Factuality benchmarks require examining model outputs against verifiable sources, much like document analysis. Source: pexels.com

SimpleQA

SimpleQA, released by OpenAI in October 2024, focuses on short-form factual questions with verifiable single answers. The 4,326 questions were verified by multiple human raters and are designed to have clear, objective correct answers - not ambiguous or opinion-based queries.

This is arguably the most important factuality benchmark right now, because it's recent, resistant to training contamination (questions weren't publicly available before the benchmark launched), and covers a wide factual domain.

Source: pricepertoken.com SimpleQA leaderboard, updated April 17, 2026. 45 models evaluated, average score 20.8.

A few things stand out. First, Gemini 2.5 Pro's lead is real but not commanding - the gap between first and fourth (GPT-4.1) is about 12 percentage points. Second, the Qwen3 family punches above its weight, with three models in the top five. Third, the absolute scores are low across the board. The leader gets 53% right. The field average is 20.8%. These aren't figures a vendor would put on a press release, which is probably why SimpleQA scores often get buried when companies announce new models.

The Thinking variants don't consistently outperform their non-thinking counterparts on this benchmark - Claude 3.7 Sonnet Thinking ties with Claude 3.7 Sonnet at 32.8%.

FACTS Grounding

FACTS Grounding, from Google DeepMind, tests something different: given a long document (up to 32,000 tokens) and a user request, does the model answer faithfully based on what's in the document without hallucinating content not present in the source?

The benchmark uses 1,719 examples across finance, technology, medicine, law, and retail. Three LLM judges - Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet - each score responses, and the final score averages their judgments to reduce individual model bias. Responses that fail to address the user request are disqualified before scoring.

Source: FACTS Grounding paper (arxiv 2501.03200), original leaderboard results from January 2025.

Google's models dominate the top three, which isn't surprising given they designed the benchmark - though the research team took care to use a multi-judge setup and include non-Google judges. Claude 3.5 Sonnet and GPT-4o are close behind, both above 78%.

The FACTS Benchmark Suite (announced in early 2026) expands this to four dimensions: Grounding v2, Parametric, Search, and Multimodal. Under that harder suite, Gemini 3 Pro leads with an overall FACTS Score of 68.8%, and no model breaks 70%. The added difficulty comes from longer documents and more complex reasoning requirements. The full suite is described on the Google DeepMind FACTS Benchmark Suite blog post.

A caveat worth noting: the research team found models rate their own outputs 3.23 percentage points higher than those from competing providers on average, which is why the multi-judge approach matters. Single-judge evaluations of grounding are suspect.

Vectara HHEM Leaderboard

Vectara's hallucination leaderboard, now in its second generation, measures how often models introduce unsupported information when summarizing documents. The evaluation uses Vectara's HHEM-2.3 model (with a FaithJudge approach for some comparisons) to score factual consistency.

The refreshed dataset expanded from 1,000 to 7,700+ articles spanning law, medicine, finance, technology, education, sports, and news. Articles now reach 32K tokens, which is a meaningful difficulty increase. The leaderboard was last updated March 20, 2026.

Source: Vectara hallucination leaderboard GitHub (last updated March 20, 2026). Lower hallucination rate = better.

The top result from Ant Group's finix_s1_32b at 1.8% is impressive, though this is a less-known model with limited independent benchmarking. The more notable finding from the updated leaderboard: reasoning-focused frontier models - GPT-5, Claude Sonnet 4.5, and Grok-4 - all show hallucination rates above 10% on the harder dataset. Vectara's explanation is that these models "overthink" summarization, deviating from source material in ways that smaller, more focused models don't.

This has direct effects for RAG pipelines. Our RAG Benchmarks Leaderboard covers the retrieval side; on the generation side, these hallucination rates suggest that raw intelligence and grounding faithfulness don't move together.

Careful verification of AI outputs remains necessary given hallucination rates that persist even in frontier models. Source: pexels.com

HalluLens and HaluEval

HaluEval (RUCAIBox, EMNLP 2023) is a research benchmark with 35,000 examples across question answering, knowledge-grounded dialogue, and text summarization. It found that ChatGPT produced hallucinated content in roughly 19.5% of user queries when prompted in specific topic domains. It doesn't maintain a live leaderboard, but it's widely used in academic hallucination research as a standard evaluation set.

HalluLens (Meta FAIR, ACL 2025) goes further by distinguishing extrinsic from intrinsic hallucinations. The key result: Llama-3.1-405B-Instruct showed the lowest false acceptance rate (6.88%) on non-existent entity prompts, while some Mistral variants hit rates above 80%. GPT-4o balanced precision and recall best across tasks, scoring 52.59% accuracy on PreciseWikiQA.

The benchmark creates test sets dynamically to prevent data leakage - a design choice that matters for assessing newer models that may have trained on older static benchmarks.

AA-Omniscience

Artificial Analysis released AA-Omniscience in early 2026 as a knowledge and hallucination benchmark that jointly rewards correct answers and penalizes hallucinations. The scoring metric, the AA-Omniscience Index, runs from -100 to 100: a score of 0 means the model is equally likely to be right as wrong.

The 6,000 questions span 42 economically relevant topics across six domains. A key design choice: abstaining when uncertain is rewarded, unlike benchmarks that count refusals as failures.

Source: artificialanalysis.ai/evaluations/omniscience. AA-Omniscience Index measures net factual reliability; hallucination rate is incorrect answers as a fraction of all non-correct responses.

The highest AA-Omniscience Index belongs to Gemini 3.1 Pro Preview at 33. Grok 4.20 in reasoning mode achieves the lowest raw hallucination rate at 17%. Claude 4.5 Haiku comes in third on hallucination rate at 25% - an interesting result for a smaller, non-reasoning model.

The gap between the accuracy ranking and the index ranking shows why the metric design matters. Gemini 3 Pro has a higher raw accuracy (56%) than Gemini 3.1 Pro Preview (55%), but its hallucination rate of ~88% drags the index down severely. A model that guesses more and is wrong more often gets penalized harder under this scoring system, which better reflects real-world reliability requirements.

Key Takeaways

No single benchmark tells the whole story

A model that ranks first on TruthfulQA may not rank first on SimpleQA, and a model that grounds faithfully to documents (FACTS Grounding) may still hallucinate during open-ended generation (AA-Omniscience). The leaderboards need to be read together. Our overall LLM rankings cover aggregate performance, but factuality is best assessed per-use-case.

Reasoning models introduce a grounding tradeoff

The Vectara finding that GPT-5, Claude Sonnet 4.5, and Grok-4 exceed 10% hallucination rates on their harder dataset is consistent with a pattern showing up across evaluations: chain-of-thought reasoning helps with tasks that need derivation, but it can hurt faithfulness on tasks that just need the model to stick to what's in front of it. If your application is document-grounded (RAG, summarization, contract review), a smaller, more focused model may serve you better than the biggest frontier reasoning model.

Benchmark contamination is a real concern for TruthfulQA

TruthfulQA's age (2021) and public availability mean that training datasets likely include examples similar to its questions. SimpleQA was designed to ease this by using a withheld question set. FACTS Grounding uses a private held-out set for the same reason. When evaluating new models, weight newer and harder benchmarks more heavily.

Open-source models can match or exceed closed models on factuality

Phi-3.5-MoE-instruct leads TruthfulQA. Llama-3.1-405B-Instruct performs best on HalluLens extrinsic hallucinations. Ant Group's finix_s1_32b leads the Vectara leaderboard. The narrative that frontier closed models are always more accurate doesn't hold up across these benchmarks.

Practical Guidance

For document-grounded applications (summarization, contract review, RAG): Focus on FACTS Grounding and Vectara HHEM scores. Prefer models that score above 78% on FACTS Grounding. Avoid reasoning-heavy models unless you've tested their grounding behavior on your specific document types.

For factual Q&A assistants (research tools, knowledge bases): Use SimpleQA as your primary signal. Top performers are Gemini 2.5 Pro (53.0%), the Qwen3 235B family (50.6%), and GPT-4.1 (40.4%). The field average of 20.8% means you should build retrieval augmentation into any production system rather than relying on parametric knowledge alone.

For general trust calibration: AA-Omniscience gives the most complete picture because it penalizes overconfidence. Models with a high index score are doing something real - they're either answering correctly more often, hedging appropriately, or both.

Budget-conscious options: Phi-3.5-MoE-instruct punches above its weight class on TruthfulQA. Gemini 2.5 Flash Lite leads the Vectara leaderboard among accessible models at 3.3% hallucination rate. Smaller models with factuality-focused training can outperform larger general-purpose models on specific tasks.

FAQ

Which model has the lowest hallucination rate in 2026?

On the Vectara summarization benchmark, Ant Group's finix_s1_32b leads at 1.8%. On AA-Omniscience, Grok 4.20 in reasoning mode reaches the lowest raw hallucination rate at 17%.

What is SimpleQA and why does it matter?

SimpleQA is OpenAI's 4,326-question benchmark for short-form factual accuracy. It's considered more contamination-resistant than TruthfulQA because it used a withheld question set at launch. The field average score of 20.8% shows factual recall remains a weak point across models.

Does TruthfulQA still matter in 2026?

It's useful but limited. The benchmark dates to 2021, is publicly available, and likely appears in training data. It's also small (817 questions). Use it as a sanity check, not a primary signal. SimpleQA and FACTS Grounding are more reliable for current model comparisons.

Why do reasoning models hallucinate more on summarization?

Vectara's leaderboard data suggests reasoning models deviate from source documents because their chain-of-thought process adds inferences beyond what's written. Document summarization rewards strict faithfulness, not elaboration - a task better suited to smaller, focused models.

What does the FACTS Benchmark Suite measure beyond the original?

The full FACTS Suite adds Parametric (knowledge without retrieval), Search (using web search tools), and Multimodal (image-grounded factuality) on top of the original Grounding benchmark. No model has topped 70% average across all four components as of April 2026.

Sources:

• TruthfulQA paper (arxiv 2109.07958)
• TruthfulQA leaderboard at llm-stats.com
• SimpleQA leaderboard at pricepertoken.com
• FACTS Grounding paper (arxiv 2501.03200)
• FACTS Grounding - Google DeepMind blog
• Vectara hallucination leaderboard (GitHub)
• Vectara next-gen leaderboard announcement
• HaluEval paper (arxiv 2305.11747)
• HalluLens paper (arxiv 2504.17550)
• AA-Omniscience benchmark - Artificial Analysis
• FELM paper (arxiv 2310.00741)

This comparison covers 12 platforms across the following dimensions: pricing model, verified resolution claims, channel coverage (web chat, email, voice, WhatsApp), helpdesk integration depth, voice agent support, workflow flexibility, self-hosting options, and publicly named customer deployments.

How to Read the Comparison Table

"Resolution rate" numbers in this article come directly from vendor marketing materials unless otherwise noted. Vendors control the definition of "resolved" - some require a customer confirmation click, others count any conversation that didn't escalate. That makes cross-vendor comparisons unreliable. Where I have third-party deployment data (Klarna, Chime, Substack), I've called it out explicitly.

Intercom Fin 3 - Best Overall for SaaS Support

Fin 3 shipped at Intercom's Pioneer 2025 event and is the most significant platform update in the product's history. The jump from Fin 2 to Fin 3 isn't gradual: Procedures replace the old content-matching approach with a system combining natural language instructions and deterministic controls. You write the resolution logic in plain English, add API connectors and conditional branches, and Fin executes them as workflows - processing refunds, verifying account status, filing claims.

The Fin 2 dataset is the most credible published resolution benchmark in the category: 66% average across 6,000+ customers, with over 20% of those customers topping 80%. Intercom also offers a 50% automation rate guarantee - if Fin resolves less than half the conversations it touches, they credit back the difference.

Pricing is $0.99 per outcome. An outcome is a resolution (customer confirms issue solved), or a successful Procedure handoff. You pay once per conversation regardless of how many exchanges it took. Plans start at $29/month/seat (Essential), $85/month (Advanced), $132/month (Expert), with Fin billed separately.

What's new in Fin 3

• Simulations - automated test suites that run Procedures against synthetic customer scenarios before you deploy. This is a genuine engineering QA feature, not a marketing checkbox.
• Expanded channels - Slack and Discord support added. Voice latency dropped 30-40% and now covers 28 languages.
• Multimodal inputs - Fin can read screenshots and invoice images customers attach to conversations, reducing "I can't read what you sent" escalations.

Weakness: Fin is deeply tied to the Intercom platform. If you're on Zendesk or Freshdesk, deploying Fin means either migrating your helpdesk or running two systems in parallel. Intercom doesn't offer a standalone "Fin layer" over third-party ticketing.

Deployment scale: Intercom hasn't published a curated deployment list with resolution numbers, but claims 40M+ resolved conversations across all Fin customers.

Zendesk AI Agents - Best for Existing Zendesk Shops

Zendesk picked up Ultimate AI in May 2023 and spent the following 18 months integrating it into Suite. The result is two AI agent tiers built into the Suite platform.

The Essential tier - included with all Suite plans - generates responses from your Help Center content without configuration. The Advanced tier ($50/agent/month add-on for Suite Professional at $115/agent/month) adds hybrid conversation flows combining generative AI with scripted paths, API orchestration, and analytics.

Automated resolution pricing is separate: each plan includes a free AR quota per agent per month. Beyond that, you pay $1.50/AR on committed volume or $2.00/AR on pay-as-you-go. The Ultimate tier runs $139/agent/month and includes skill-based routing, live dashboards, and extended API limits.

The Forethought acquisition (March 2026) is strategically significant. Zendesk's largest acquisition in nearly two decades, Forethought adds five AI "agents" - Solve, Triage, Assist, Discover, and Agent QA - that work across email, chat, voice, and SMS. If you're already on Zendesk, the combined platform is now the most feature-complete option in the market. If you're not, the switching cost is high.

Voice: Available via Zendesk Talk, priced separately. Not native to the AI agent tier.

Weakness: The per-seat pricing model means AI costs compound on top of agent seat costs. A team of 50 agents on Suite Professional is already paying $5,750/month before AI adds and AR charges.

Agentforce (Salesforce Einstein Service AI) - Best for Salesforce Shops

Salesforce rebranded Einstein Service Agent to Agentforce in late 2024. The pitch is autonomous customer self-service without pre-scripted flows - the agent uses a reasoning engine to interpret requests, check Salesforce CRM data, and take action.

The numbers are painful if you're not already embedded in the Salesforce ecosystem. Service Cloud Enterprise runs $165/user/month. Service Cloud Unlimited is $330/user/month. Agentforce conversations are billed at $2 each. A team running 15,000 AI conversations per month faces $30,000/month in conversation costs before licenses.

The implementation timeline is a real constraint: basic Einstein configuration takes 4-8 weeks and usually requires a certified Salesforce partner. A full Agentforce rollout is 3-6 months. Compare that to Fin or Decagon, which ship in days.

What it does well: If your support data lives in Salesforce CRM - cases, entitlements, contact history, product catalogs - Agentforce's data access is unmatched. No API calls to external systems; it queries your Salesforce org directly. For regulated industries with strict data residency requirements, that's a real advantage.

Support teams using AI platforms spend less time on ticket routing and more time reviewing performance analytics and exception handling. Source: pexels.com

Decagon - Best Pure-Play Agentic Option

Decagon is the quietest name on this list and has the most credible third-party resolution numbers. Chime reported 70% AI resolution across chat and voice. Substack reached 90% resolution without human intervention. Bilt Rewards reported 75%. NG.CASH grew autonomous resolution from 13% to 70% over its deployment period.

Those numbers come from publicly verified customer case studies, not aggregate vendor dashboards. That's a meaningful distinction.

Decagon doesn't publish pricing - it's enterprise-only with custom contracts. Third-party sources put it in the same range as Sierra: six-figure annual commitments. The platform supports chat, email, and voice, with deep integrations into Zendesk, Salesforce, and Intercom for helpdesk data access.

What makes Decagon different: It's not built on a rules engine. Decagon models are trained on each customer's historical support data, producing resolution behavior tuned to that company's specific product and customer patterns. This takes longer to implement than a configuration-based tool, but the resolution accuracy on complex queries is higher.

Weakness: No self-serve or SMB tier. Decagon won't make sense unless your support volume justifies a dedicated AI implementation project.

Sierra - Best for Voice-First Enterprise Support

Sierra, co-founded by Bret Taylor (former Salesforce co-CEO) and Clay Bavor, reached $100M ARR in November 2025 - 21 months after launch. The company was valued at $10 billion following a $350M Greenoaks round in September 2025.

Sierra's positioning is outcomes-based pricing, with contracts starting at $150K/year. Customers include ADT, Bissell, SiriusXM, Cigna, Ramp, Rivian, Discord, and Vans. The mix of tech-native and legacy enterprise deployments is notable - most AI-native support vendors have consumer tech customer lists. Sierra's presence at SiriusXM and Cigna suggests the platform handles regulated, complex support workflows.

Voice is Sierra's strongest differentiator. Most platforms in this list treat voice as a channel add-on. Sierra was voice-first from the beginning and the product architecture reflects it.

"The era of clicking buttons is over," Taylor told TechCrunch in April 2026. The implication is that Sierra's agents handle multi-step resolution on the phone, not just deflection.

Weakness: The $150K/year minimum excludes everyone except large enterprises. No published resolution rates from Sierra directly - customer case studies are NDA-protected.

Ada - Established Enterprise Chatbot Layer

Ada has been in the enterprise support automation market since 2016. It's the oldest purpose-built AI support vendor on this list and has built deep integrations with Zendesk and Salesforce. The current product supports web chat, email, and mobile channels with voice available via integrations.

Pricing starts at approximately $30,000/year (per the Salesforce AppExchange listing), with enterprise deals reported in the $100K-$300K range. Resolution pricing is $1-$3.50 per resolved conversation depending on volume and configuration. That upper end gets expensive fast at scale.

Ada is honest in its own pricing guide about the tradeoffs between resolution-based and conversation-based models. Resolution-based pricing creates better incentives: you only pay when the AI actually closes the ticket.

Weakness: Ada is genuinely expensive and opaque on pricing. The per-resolution rate of $3.50 at low volumes competes unfavorably with Fin's $0.99 flat rate.

Forethought (Now Part of Zendesk)

Zendesk announced the acquisition of Forethought on March 11, 2026 - its largest deal in two decades. The transaction closed by end of March 2026. Today, Forethought operates as a Zendesk product.

If you're assessing Forethought independently, understand what you're buying into. It's now a Zendesk product roadmap. That may be fine - Zendesk has the scale to invest in the technology. But independent Forethought deployments over non-Zendesk helpdesks are likely to see reduced investment over time.

Forethought's five-agent architecture (Solve for automated resolution, Triage for routing, Assist for agent recommendations, Discover for knowledge gaps, Agent QA for quality scoring) is the most thorough AI layer in the category if your helpdesk is already Zendesk. Median contract value before acquisition ran around $56,000-$60,000/year.

Forethought's published resolution claim - "up to 98%" - isn't independently verified. Take it as a best-case ceiling, not an average.

Kustomer - Meta's CRM-First Approach

Meta acquired Kustomer in 2020 and spun it back out as an independent company in 2023. The current Kustomer product is one of the few platforms on this list that combines a full customer CRM (conversation history, customer profiles, lifetime value data) with native AI agents.

The October 2024 relaunch introduced "disruptive pricing" - conversation-based billing at $0.60 per engaged AI conversation for the customer-facing agent. That's the cheapest per-conversation rate in this comparison. The catch: you pay when the AI engages, not when it resolves. A deflected conversation that fails and escalates still costs $0.60.

Kustomer AI Voice is included at no extra cost beyond standard telephony. That's genuinely competitive: Sierra and Decagon charge enterprise minimums for voice. Kustomer's AI voice is available to anyone on the $89/seat plan.

The Vuori case study is the clearest public deployment data Kustomer has published: 40% of chat conversations resolved by AI without human intervention. That's below Fin and Decagon averages, but Vuori is an apparel e-commerce brand, not a software company - and product return/exchange flows are truly harder to automate than software FAQs.

Crescendo - Managed AI Plus Human Backup

Crescendo takes a different model: it's not a pure software platform. The service combines AI agents with a team of 3,000+ human agents who handle anything the AI can't resolve. You're outsourcing customer support operations, not just buying software.

Pricing reflects this: $2.99/resolution plus a $2,900/month platform fee. That's higher per-resolution than most alternatives, but it includes 24/7 human coverage, 50+ language support, and quality assurance - services that cost extra elsewhere.

The vendor claims 70-90% AI resolution with 99.8% accuracy. The accuracy number is unverified. The value proposition is clearest for companies that want to remove their internal support operations entirely rather than augment an existing team.

AI support platforms surface knowledge base answers and customer history directly inside the chat interface, reducing time-to-resolution. Source: pexels.com

Gorgias AI - Best for E-Commerce

Gorgias is the only platform here built specifically for e-commerce, and its 15,000+ brand customers (Kith, Arc'teryx, Reebok) confirm the fit. Every ticket opens with the customer's full order history, shipping status, and return eligibility visible alongside the conversation - no tab-switching, no manual CRM lookup.

The AI Agent feature resolves 60% of support inquiries according to Gorgias's own data. Pricing for AI automation is $1.00/automated resolved conversation on monthly billing, or $0.90 on annual. Note the double-billing structure: you also pay the helpdesk ticket charge on top of the AI automation fee.

If you're not in e-commerce, Gorgias has limited value. The product doesn't handle SaaS subscription workflows, doesn't integrate cleanly with Salesforce, and lacks the deep customization options that enterprise support teams need.

Voice: Gorgias doesn't offer native voice. This is a live chat, email, and social messaging tool.

Help Scout AI - Best Budget Option for Small Teams

Help Scout is the simplest platform in this comparison. It's a shared inbox tool for small support teams with AI features layered on top: AI Answers (customer-facing chatbot), AI Drafts (agent reply suggestions), AI Assist (tone and translation), and AI Summarize (thread condensing).

Plans run $25/user/month (Standard), $45/user/month (Plus), and $75/user/month (Pro). AI Answers costs $0.75 per AI-resolved conversation, the lowest published per-resolution rate in the market. New accounts get three months of unlimited AI Answers free.

The AI Answers chatbot is straightforward: it answers from your Docs knowledge base and hands off to a human if it can't resolve. There's no voice support, no advanced workflow engine, no enterprise integrations beyond standard web hooks. For a 5-20 person support team that wants deflection on simple FAQ traffic without a six-figure contract, Help Scout is the right starting point.

HubSpot Service Hub AI - Best for HubSpot-Centric Teams

HubSpot's Breeze AI Customer Agent is available in Service Hub Professional ($90/seat/month) and Enterprise ($150/seat/month). The AI works inside the HubSpot CRM - it sees every customer interaction, deal, and support ticket in the same data model.

The pricing structure uses Breeze credits: $10 per 1,000 credits. Customer Agent consumes 100 credits per conversation, which works out to roughly $1 per conversation. Plans include a 3,000-credit allocation (roughly 30 conversations) per seat per month.

For teams already using HubSpot for sales and marketing, the Service Hub AI makes sense: it runs on the same data without any integration work. For teams not in the HubSpot ecosystem, the seat cost is hard to justify when Fin or Help Scout covers the same resolution workflow at lower cost per conversation.

Pricing Model Comparison

This table covers only publicly published pricing. "Contact sales" means the vendor has no published list price.

The Klarna Reference Point

Klarna's public deployment data is the most cited benchmark in AI customer support and worth looking at directly. In the first month of running their Anthropic-powered assistant, Klarna handled 2.3 million conversations through AI - two-thirds of all customer service interactions. Resolution time dropped from 11 minutes to 2 minutes. Klarna says customer satisfaction scores held steady at human-agent parity.

The assistant does the equivalent work of 700 full-time agents and saved $60 million. Per-transaction customer service costs fell 40% over two years.

Two things are true simultaneously: those numbers are impressive and Klarna had to walk back its "AI-first" strategy partially in 2025, reintroducing human agents for complex cases. The 75% self-resolution rate was real, but the 25% of cases that escalated required more human expertise than the cost savings justified removing. Pure deflection rates overstate deployment success; what matters is whether customers with complex problems still get resolved.

None of the 12 platforms in this article built Klarna's custom infrastructure. They offer configurable AI on top of existing helpdesk data. Klarna's numbers are the ceiling, not the benchmark.

Best Picks

Best overall - mid-market SaaS, consumer tech, fintech: Intercom Fin 3. The $0.99/resolution pricing is the fairest cost model in the market, the Fin 2 resolution data is the largest published dataset in the category, and Fin 3's Procedures give engineering teams genuine workflow control without hard-coding. The 50% automation rate guarantee removes pricing risk during ramp-up.

Best for e-commerce brands: Gorgias AI. The order-context-in-every-ticket feature removes the primary friction in e-commerce support. At $0.90/resolved conversation on annual billing, it's cost-competitive with Fin for the right workload.

Best for enterprise with Salesforce investment: Agentforce. The data access story is compelling if your customer records live in Salesforce. At $2/conversation the cost model gets expensive, but the integration depth justifies it for teams that have already built on Service Cloud.

Best budget option for small teams: Help Scout AI at $0.75/AI-resolved conversation is the lowest published per-resolution rate in the market and requires no long-term contract.

For complex enterprise deployments with voice requirements: Sierra or Decagon, depending on whether your budget supports a $150K+ minimum. Both have credible public deployment data and neither requires buying into a separate helpdesk vendor.

If you're evaluating general-purpose AI agent frameworks rather than support-specific products, the best AI agent frameworks for 2026 covers the build-vs-buy tradeoff from an engineering perspective. For teams wanting to build custom support bots on top of LLM infrastructure, best AI chatbot builders 2026 covers the low-code and no-code build options.

FAQ

What's the cheapest AI customer support option that actually works?

Help Scout AI at $0.75/AI-resolved conversation is the lowest published per-resolution rate, with no minimum contract. For small teams under 20 agents, the platform plus AI tier runs under $2,500/month total.

Which platforms support voice AI natively?

Sierra (voice-first architecture), Kustomer (native at no extra cost), Intercom Fin 3 (28 languages, 30-40% latency improvement in v3), Decagon (voice included), and Crescendo (managed voice included in base fee). Zendesk and HubSpot handle voice via separate telephony products.

Can any of these tools be self-hosted?

No platform in this comparison offers a self-hosted tier. If data residency or self-hosting is a hard requirement, you'd need to evaluate open-source alternatives or build on models running in your own infrastructure. The best AI agent frameworks covers open-source options.

Is Forethought still worth evaluating after the Zendesk acquisition?

For teams already on Zendesk, yes - the combined platform is the most feature-complete in the category. For teams on other helpdesks, assess with caution: Forethought's roadmap now runs through Zendesk's product priorities.

What does "resolution rate" actually mean across vendors?

It differs by vendor. Intercom counts a resolution when a customer confirms the issue is solved or doesn't escalate after Fin's reply. Kustomer's 40% Vuori figure counts conversations where no human agent took over. Decagon's numbers come from customer case studies with customer-defined definitions. There's no industry standard, so compare cautiously.

Sources

• Intercom Fin AI Pricing - official Intercom pricing page with $0.99/outcome model
• Fin 3 announcement - The Intercom Blog - Fin 3 features, Fin 2 resolution rate (66% avg), Simulations and Procedures details
• Zendesk Suite Pricing Guide - eesel AI - Suite plan tiers and AI add-on costs breakdown
• Zendesk AI Agents - About AI agents - AR pricing tiers ($1.50/$2.00)
• Sierra reaches $100M ARR - TechCrunch - $100M ARR in 21 months, November 2025
• Sierra raises $175M - TechCrunch - fundraising history; $350M/10B valuation reported separately in September 2025
• Sierra's Bret Taylor: era of clicking buttons is over - TechCrunch - April 2026 interview
• Klarna AI customer service costs - CX Dive - $60M savings, 40% per-transaction cost reduction, 2.3M conversations handled
• Kustomer AI-Native Platform Launch - CMSWire - October 2024 relaunch, $0.60/engaged conversation pricing, native voice
• Help Scout Pricing - official Help Scout pricing, $0.75/AI resolution
• Crescendo Pricing - $2.99/resolution + $2,900/month base
• Gorgias AI Pricing - eesel AI analysis - $0.90-$1.00/automated resolution breakdown
• Ada CX pricing anchor - Featurebase analysis - $30K/year minimum, $1-$3.50/resolution range
• Salesforce Agentforce - official Agentforce product page
• Zendesk acquires Forethought - Forethought.ai - March 2026 acquisition announcement

The table below shows where Codex now stands against Anthropic's Claude Code across the features that matter most for agentic development workflows.

What's New in the April 16 Update

Background Computer Use

The core feature of the update is computer use: Codex can see, click, and type into Mac apps while running in the background, without interfering with other work. Developers can run multiple agents simultaneously - one testing a mobile UI in Simulator while another pushes a commit - with each task isolated from the others.

OpenAI's developer documentation is specific about scope. Codex can interact with windows, menus, keyboard input, and clipboard state in target apps. It cannot automate terminal apps or Codex itself (a deliberate security restriction), and it cannot authenticate as an administrator or approve system permission prompts.

The permission model requires users to grant both macOS Screen Recording and Accessibility access, then selectively approve individual apps on a per-session basis.

The computer use approval dialog asks users to authorize each app individually before Codex can interact with it. Source: developers.openai.com

The Atlas Browser

Alongside computer use, Codex now ships with a built-in browser built on OpenAI's proprietary Atlas engine. In its current form, it lets developers comment directly on pages served from local dev servers, giving the agent precise context about which elements to modify without switching to an external browser.

OpenAI's stated intent is to expand Atlas to "fully command the browser beyond web applications on localhost" - meaning general web browsing - but that isn't live yet.

Image Generation and Visual Tooling

Image generation via gpt-image-1.5 is now available directly inside Codex threads. The use cases are narrow but practical: generating placeholder images, iterating on mockup concepts, and creating slide visuals without leaving the development environment. Rich file previews for PDFs and spreadsheets, multiple terminal tabs, GitHub review integration, and artifact previews round out the quality-of-life additions in this release.

111 Plugins: The App Store Moment

OpenAI has added 111 new plugins to the Codex plugin ecosystem, bringing together skills, app integrations, and MCP server configurations into a single installable unit. Named integrations include Atlassian Rovo, CircleCI, CodeRabbit, GitLab Issues, Microsoft Suite, and Render.

The framing from OpenAI is that plugins let teams package reusable workflows - for example, a plugin that reads a project's Slack channel and Google Calendar to create a daily task list, then routes issues to the appropriate agent.

Codex running multiple tasks in parallel inside the desktop app. Source: developers.openai.com

Also added: thread reuse so agents can resume interrupted work, and long-horizon scheduling that lets Codex work across tasks that span days or weeks. Memory - a preview feature that lets Codex recall preferences, tech stacks, and workflow patterns across sessions - completes the picture, though its availability is sharply restricted (more on that below).

What It Does Not Tell You

The Geography Problem

Computer use isn't available in the European Economic Area, the United Kingdom, or Switzerland. Memory is similarly unavailable for Enterprise, Education, EU, and UK users. These are two of the four headline features in this update. For organizations operating under GDPR or UK data protection law - which includes a major share of enterprise software teams - the April 16 update delivers less than the headlines suggest.

The restrictions aren't accidental; they reflect the regulatory pressure that makes screen-recording-based computer use truly difficult to ship in Europe. OpenAI hasn't said when, or whether, these restrictions will lift.

The Memory Fine Print

The memory feature is in preview and excluded from Enterprise accounts - the segment most likely to need persistent workflow context. Individual users on Pro, Max, and Team plans can access it, but Enterprise deployments remain stateless. Teams assessing Codex for professional workflows should treat memory as aspirational for now, not a production capability.

The Browser Ceiling

The Atlas browser is built for localhost development. Pointing it at production URLs or external services is outside its current scope. The roadmap phrase "fully command the browser" is OpenAI acknowledging that the current implementation is incomplete - not announcing a feature that exists today.

Taken overall, this update closes a visible gap between Codex and Claude Code's agentic capabilities. The computer use architecture, the plugin depth, and the Atlas browser represent genuine additions that enterprise developers will want to evaluate. The geographic restrictions and the memory exclusions are real constraints that'll matter to a large share of professional users. OpenAI is catching up - but in parts of the world where it counts most commercially, Anthropic still has the open field.

Sources:

• OpenAI Codex April 2026 Update - TechCrunch
• Codex Computer Use - OpenAI Developer Docs
• OpenAI Codex Mac Update - MacRumors
• Codex Three Key Features - 9to5Mac
• SmartScope Codex Desktop Update Analysis

Where GLM-5.1 Comes From

Z.ai is a Beijing-based lab spun out of Tsinghua University's KEG research group. The company appeared on the US Entity List in January 2025, cutting off legal access to Nvidia's H100, H200, and B200 GPUs. Most frontier AI labs outside China treat those chips as oxygen. Z.ai had to build without them.

The entire GLM-5 family - including 5.1 - was trained on around 100,000 Huawei Ascend 910B chips running MindSpore, Huawei's homegrown deep learning framework. The decision wasn't ideological. It was the only available path under US export controls. That constraint makes the benchmark results much more interesting than they'd be otherwise.

On January 8, 2026, Zhipu completed a Hong Kong IPO - the first publicly traded foundation model company in the world - raising roughly HKD 4.35 billion (around $558 million USD) at a valuation near $31.3 billion. The GLM-5.1 release came three months later, and the timing isn't coincidental. The company needs to justify that valuation with technical output.

The GLM-5 base model launched in February 2026. GLM-5.1 is a post-training refinement focused specifically on agentic coding tasks and long-horizon execution. The architecture is unchanged; the improvements come from training methodology.

Architecture and Specifications

GLM-5.1 uses a Mixture-of-Experts design the company calls GLM_MOE_DSA (Dynamic Sparse Attention), with 754 billion total parameters and 40 billion active per forward pass. That active-parameter count places it in the same inference-compute neighborhood as models like DeepSeek V3.2 and Kimi K2.5, which both use similar MoE approaches.

The 200K context window with 128K maximum output is a truly generous combination - most frontier models cap output at 32K or 64K. For long-horizon agentic tasks that generate large code artifacts or iterate across many files, that ceiling matters.

Supported inference frameworks include SGLang (v0.5.10+), vLLM (v0.19.0+), Transformers, KTransformers, and xLLM. Ollama also has the model in its library. The API supports function calling, streaming, structured outputs, context caching, and a thinking mode for extended reasoning.

GLM-5.1's training ran on Huawei Ascend 910B chips - no NVIDIA hardware anywhere in the stack. Source: pexels.com

Benchmarks

The headline number is SWE-Bench Pro, but GLM-5.1's benchmark profile is more varied than the press releases suggest.

On coding-specific work, the picture is positive. SWE-Bench Pro tests real software engineering tasks - applying patches, fixing bugs, implementing features from natural language descriptions - and GLM-5.1's 58.4 is currently the highest public score on that leaderboard. It also leads CyberGym at 68.7, a security task completion benchmark across 1,507 tasks.

The gaps appear on general reasoning. AIME 2026 (advanced math competition problems) has GPT-5.4 at 98.7% and Claude Opus 4.6 at 98.2%, while GLM-5.1 comes in at 95.3. GPQA-Diamond (graduate-level science reasoning) shows an even wider gap against Gemini 3.1 Pro's 94.3%. If your workload is heavy on mathematical derivations or scientific research, this isn't the model to reach for.

One thing worth noting: the SWE-Bench Pro results come from Z.ai's internal testing. Arena.ai independently placed GLM-5.1 third on their Code Arena agentic webdev leaderboard with an Elo rating of 1530, which provides at least partial external validation for the coding performance. But a fully independent evaluation on SWE-Bench Pro from a third-party lab hasn't been published as of this writing. Treat the exact margin over Claude and GPT-5.4 as preliminary.

Agentic Capabilities

The most compelling part of GLM-5.1 isn't any individual benchmark score. It's the claim that the model can run autonomously for up to eight hours without human checkpoints, completing a full plan-execute-analyze-optimize loop across hundreds of iterations.

Z.ai demonstrated this with two tasks. First: building a complete Linux desktop environment from scratch - file browser, terminal, text editor, system monitor, and playable games - through 655 autonomous iterations in a single session. Second: optimizing a vector database over 178 rounds of autonomous tuning, improving throughput to 6.9 times the original baseline. In a separate CUDA kernel optimization task, the model improved speedup from 2.6x to 35.7x through sustained iterative refinement.

These are company-produced demos, so the usual skepticism applies. But the underlying technical approach - maintaining goal alignment across hundreds of tool calls without strategy drift - addresses a real failure mode of current agentic systems, where models lose context or start tuning for the wrong objective after a few dozen steps.

The practical implication: GLM-5.1 makes more sense as the backbone of an autonomous coding agent than as an interactive assistant. Plug it into a CI/CD pipeline, give it a ticket, and come back later. For real-time IDE completions where latency matters, Cursor or Claude Code with a faster model will feel more responsive.

GLM-5.1 is built for long-running agentic loops, not fast interactive completions. Source: pexels.com

Pricing and Access

The model weights are on Hugging Face at zai-org/GLM-5.1 under a MIT license, which means unrestricted commercial use and self-hosting.

For API access, Z.ai prices GLM-5.1 at $0.95 per million input tokens and $3.15 per million output tokens, with cached inputs at $0.26 per million. Those prices sit comfortably below what Anthropic and OpenAI charge for their frontier models.

Z.ai also offers a subscription-tier Coding Plan aimed at developers who want integrated tooling rather than raw API access:

Free-tier users get access to GLM-4.7-Flash and GLM-4.5-Flash at no cost, along with a small allocation of premium model credits.

Local deployment works with standard frameworks. On a NVIDIA DGX Spark or equivalent multi-GPU setup, you can run the FP8-quantized variant (zai-org/GLM-5.1-FP8) with reduced memory requirements. The full BF16 weights need substantially more VRAM - this isn't a laptop model.

Strengths

• SWE-Bench Pro leader: 58.4 is the highest public score on the benchmark that best predicts real-world software engineering performance
• Genuinely long autonomous execution: eight-hour uninterrupted agentic sessions with maintained goal alignment aren't something most models can do
• MIT license with commercial use: no restrictions, self-hostable, no vendor lock-in
• Competitive API pricing: at $0.95/$3.15 per million tokens, it undercuts frontier closed-source models
• Deep engineering toolkit: function calling, context caching, MCP integration, structured outputs, thinking mode - all there
• CyberGym leader: 68.7 on security task completion ahead of Claude Opus 4.6's 66.6

Weaknesses

• Text-only: no image, audio, or video input while GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro are all multimodal
• Slower generation speed: 40-44 tokens/second against a competitive median above 53 t/s, which is noticeable in interactive use
• Verbose output: the model produces significantly more tokens than peers to reach equivalent answers, which inflates costs and latency
• Math and science gaps: AIME 2026 (95.3) and GPQA-Diamond (86.2) trail the frontier, making it a weaker choice for research or quantitative tasks
• Self-reported benchmarks: the SWE-Bench Pro score hasn't been independently verified by a third-party lab
• Young ecosystem: fewer community integrations and third-party tooling than Claude or OpenAI-compatible APIs

Verdict

GLM-5.1 is the most capable open-weight model available for agentic software engineering work. The SWE-Bench Pro result is real - partially backed up by Arena.ai's independent Code Arena rankings - and the sustained eight-hour autonomous execution is a genuine differentiator for teams building long-running coding agents. The MIT license and competitive API pricing remove the usual friction of adopting a less-established model.

The weaknesses are real too. No multimodal input, slower token generation, and benchmark claims that haven't all been independently verified mean this isn't a drop-in replacement for Claude or GPT-5.4 across general workloads. For coding agents specifically - especially those that need to run overnight on complex engineering tasks - GLM-5.1 earns serious consideration.

The detail that makes the result interesting beyond the benchmark number: it was built entirely on Huawei silicon, by a lab that's been cut off from US chips for over a year. Whether that changes anything about how US policymakers think about export controls is a separate question. The technical fact is that the gap between Western and Chinese frontier models on coding tasks is now measured in decimal points.

Score: 8.1 / 10

Sources

• GLM-5.1 Official Documentation - Z.ai
• zai-org/GLM-5.1 on Hugging Face
• Z.AI Introduces GLM-5.1 - MarkTechPost
• GLM-5.1 - Artificial Analysis
• GLM-5.1 API Pricing - OpenRouter
• GLM-5.1 Tops SWE-Bench Pro - Dataconomy
• GLM-5.1 Open Source #1 on SWE-Bench Pro - ModemGuides
• GLM-5.1 Full Review - BuildFastWithAI
• GLM-5.1 Benchmarks Breakdown - Lushbinary
• Z.ai Coding Plan Pricing

Physical Intelligence on April 16 unveiled π0.7, a generalist robot model that can tackle tasks it was never directly trained on - then match the performance of specialist models that were. It's a result that the robotics field has been chasing for years and that most researchers thought was still a few model generations away.

What π0.7 Actually Does

A generalist that rivals specialists

The headline result is that a single π0.7 model, with no task-specific fine-tuning, can match the success rates of Physical Intelligence's own π*0.6 specialist models - which were trained via a dedicated reinforcement learning algorithm called Recap specifically designed to maximize performance on individual tasks.

That matters because the standard path in robot learning has always been: train a new model (or fine-tune heavily) for every new task. Swapping out a laundry-folding robot for an espresso-making one isn't just a software update - it's a whole new training pipeline. π0.7 is an attempt to change that.

Co-founder Sergey Levine described the shift this way: "Once it crosses that threshold where it goes from only doing exactly the trained tasks to actually remixing things in new ways, the capabilities are going up more than linearly." That's the same description researchers used for early large language models - where each new benchmark didn't just improve, it collapsed.

The air fryer experiment

The clearest demonstration of what the company means by "compositional generalization" is the air fryer test. π0.7 had exactly two relevant training episodes involving that appliance: one showing a robot pushing an air fryer closed, and one showing a robot placing a bottle inside. Neither episode involved cooking.

Given the instruction "cook a sweet potato," with no additional coaching, the model attempted the task and failed. Given step-by-step verbal instructions ("open the fryer, place the potato, set the timer"), it succeeded - synthesizing fragments from tangential training data into functional behavior. The research team later found that the model probably learned air fryer context from the open-source DROID dataset, which includes footage of humans interacting with appliances. The robot never saw the specific task but absorbed enough about the appliance class to generalize.

A fisheye view from inside the robot's perception stack as it navigates the kitchen for the air fryer cooking experiment. Source: pi.website

Architecture: Three Layers Working Together

The policy stack

π0.7 is a Vision-Language-Action model built from three components that operate at different levels of abstraction:

• High-level policy: creates language subgoals describing what the robot should achieve next
• World model: produces visual subgoals at inference time - literally predicting what the scene should look like after the next action
• Action expert: takes the multimodal context (camera frames, language instructions, predicted subgoals) and outputs low-level robot control commands

The world model step is what makes compositional generalization plausible. Instead of mapping raw instructions directly to motor commands, the model first imagines what success looks like, then plans toward that image. This creates a natural bottleneck where learned skills from different domains can be mixed and matched through the visual prediction space.

Training with varied conditioning

The training methodology is built around what the team calls "diverse multimodal conditioning" - as opposed to naive data merging. Rather than throwing all robot episodes into a single training run with a fixed instruction format, π0.7 was trained with varied prompt structures:

• Language instructions describing tasks and subtasks at different granularities
• Metadata flags specifying execution speed or quality level
• Control modality labels showing whether the robot should use joint control or end-effector control
• Visual subgoal images showing desired intermediate states

The underlying data combined several sources: teleoperated robot episodes, human demonstrations captured with a different setup, web-scale vision-language pretraining (the same recipe that powers today's VLMs), and synthetic data created by running existing RL policies autonomously. The diversity of conditioning formats is what allows the model to be steered at inference time through plain language rather than requiring task-specific prompts.

Benchmark Comparison

Physical Intelligence compared π0.7 against its own RL-tuned specialist models across the core task set. External benchmarks for generalist robot policies don't yet exist in any standardized form, so these numbers are self-reported.

The cross-embodiment result deserves closer reading: for the UR5e laundry-folding transfer, the comparison group was human teleoperators attempting the task on the target robot for the first time - subjects with about 375 hours of teleoperation experience. π0.7 matched their first-attempt success rates. That's a useful baseline exactly because it anchors the claim in observable human performance rather than a competing model number.

The robot's on-board perception during a separate task evaluation in a non-kitchen environment. Source: pi.website

Cross-Embodiment Transfer

One of the more surprising results is the cross-embodiment transfer: moving the laundry-folding capability from a small static bimanual desktop robot to a large UR5e industrial arm with markedly different morphology - different joint counts, different workspace geometry, different gripper design.

The model didn't require retraining for this. The cross-embodiment generalization comes from the control modality labels in the training data, which teach the model to reason about robot morphology as a conditioning variable rather than a fixed property. If the model has seen enough variations of arm kinematics during training, a new embodiment becomes one more conditioning input rather than a reason to start over.

Physical Intelligence has been documenting this arc for a while. Their earlier investment round at an $11B valuation in March was partly premised on exactly this kind of result - that the jump from task-specific to general-purpose robot policies was closer than the market believed. π0.7 is the first internal proof point they've released publicly.

The timing also puts Physical Intelligence directly in competition with Google DeepMind, whose Gemini Robotics-ER 1.6 launched in April for Boston Dynamics' Spot platform with a focus on embodied visual reasoning. Those are different bets - Google is pushing integration with commercial robot hardware, while Physical Intelligence is still building the foundation model layer. And Hugging Face's open-source LeRobot v0.5.0 remains the accessible alternative for researchers who want to experiment without proprietary APIs.

What To Watch

What the demos don't prove

The key limitation is one Physical Intelligence states clearly: π0.7 can't execute multi-step tasks autonomously from a single high-level instruction. "Make toast" doesn't work. "Open the toaster, place the bread, press the lever" does. The coaching step is essential. The model has generalized skill components, not autonomous task planning.

That's a real gap. The practical value of a generalist robot in most commercial settings depends on reducing the amount of per-task human setup. If every new task still requires someone to decompose it into coached substeps, a lot of the generalization value moves from the robot to the operator.

Benchmark opacity

The comparison data is self-reported, compared against Physical Intelligence's own prior models, with no external replication. The research community still lacks standardized robotics benchmarks that would let third parties verify claims like "matches specialist performance." Until those exist, every result from any robotics lab - Physical Intelligence, Google DeepMind, NVIDIA - should carry a methodological asterisk. We've seen similar dynamics with NVIDIA's Alpamayo model, where strong internal numbers didn't fully translate to external testing conditions.

What's actually novel

Skepticism aside, the compositional generalization result is genuinely new if it holds up. Previous generalist robot models required fine-tuning to match specialist performance. π0.7's claim is that the gap is now zero, in the zero-shot direction. That's a meaningful threshold to cross.

The real test will come when external researchers get access to the model weights - Physical Intelligence hasn't showed a release timeline - and when the evaluation methodology gets stress-tested against tasks that sit further from the training distribution. The air fryer worked because DROID had relevant tangential data. The next test is something truly outside the training set.

Physical Intelligence's next public benchmark is whether the model generalizes to tasks where no tangential data exists at all - a question the company hasn't answered yet, and one the broader field will be watching closely.

Sources:

• Physical Intelligence π0.7 announcement
• Physical Intelligence, a hot robotics startup, says its new robot brain can figure out tasks it was never taught - TechCrunch

This leaderboard covers where the major models actually stand, based on two types of evidence: the Artificial Analysis Video Arena (blind Elo from user preference votes) and the VBench/VBench-2.0 academic benchmarks (automated metrics across 16-18 dimensions). The two systems measure different things and sometimes disagree. Both matter.

How These Benchmarks Work

Artificial Analysis Video Arena

Artificial Analysis runs blind pairwise comparisons. A user sees two videos generated from the same prompt, doesn't know which model produced which, and picks a winner. Elo scores update continuously from these votes, the same mechanism used in chess ratings. A gap of 20-30 Elo points translates to roughly 53% win rate in direct head-to-head comparisons.

The leaderboard separates text-to-video and image-to-video tasks, and also splits by whether audio output is included. This matters: models that produce synchronized audio (native sound, not post-dubbed) tend to score higher on the audio leaderboard regardless of visual quality alone.

As of April 2026, the leaderboard covers more than 80 models across 24 API providers.

VBench and VBench-2.0

VBench, introduced at CVPR 2024 by the Vchitect group, decomposes video generation quality into 16 automated dimensions covering two categories - video quality (subject consistency, motion smoothness, temporal flickering, dynamic degree, aesthetic quality, imaging quality) and video-text alignment (object class, multiple objects, human action, color, spatial relationship, scene, temporal style, appearance style, overall consistency). Each dimension uses a purpose-built automated evaluator rather than a single aggregate score.

VBench-2.0, released in March 2026, extends the framework to test "intrinsic faithfulness" - five new categories (Human Fidelity, Controllability, Creativity, Physics, Commonsense) with 18 fine-grained sub-dimensions. The original VBench caught many technical defects well; VBench-2.0 is designed for models that have already cleared those. Even leading models currently score around 50% on action faithfulness in the new framework, suggesting plenty of headroom remains.

The two benchmarks answer different questions. VBench rewards smooth, coherent video that matches prompts. VBench-2.0 rewards plausible physics, human motion, and real-world causality. A model can score well on one while failing the other.

VBench radar chart comparing closed-source video generation models across all 16 evaluation dimensions. Subject consistency and motion smoothness tend to separate top models from the middle tier. Source: github.com/Vchitect/VBench

Text-to-Video Rankings - Artificial Analysis Elo

Rankings as of April 17, 2026. "No Audio" tracks pure visual quality; "With Audio" includes native synchronized audio output as a preference signal.

Source: Artificial Analysis Text-to-Video Leaderboard

Notes on the table above: HappyHorse-1.0 leads but its public availability is uncertain - Alibaba confirmed authorship on April 10 but hasn't announced a standard API or subscription product. Seedance 2.0 is accessible primarily in China following its global rollout pause. Runway Gen-4.5 was previously reported with an Elo near 1,247, but doesn't appear in the current top-10 by No Audio Elo today. The leaderboard updates continuously.

Image-to-Video Rankings - Artificial Analysis Elo

Source: Artificial Analysis Image-to-Video Leaderboard

The image-to-video gap between first and second place is 51 Elo points, which is one of the largest margins in the arena's history. For I2V with audio, Seedance 2.0 leads (1,176) over HappyHorse (1,165) and SkyReels V4 (1,087) - the audio pairing is where Seedance retains an advantage.

Open-Source VBench Rankings

The VBench leaderboard tracks 50+ text-to-video models. These are the notable open-source entries by overall score:

The Wan series (from Alibaba's open research team, distinct from the HappyHorse product) leads open-source VBench scoring. Wan 2.2 trained on 1.5 billion videos and 10 billion images, and its 84.7% aggregate VBench score is the highest publicly verified number in that tier.

HunyuanVideo 1.5 from Tencent generates at roughly 75 seconds per clip on a RTX 4090 at 480p. Not fast, but more accessible than most proprietary options for developers who want GPU-local generation. LTX-2.3 from Lightricks is the speed pick - it runs notably faster than HunyuanVideo at comparable quality on many prompt types. James covered it when it launched in the LTX-2.3 model profile.

VBench radar chart for open-source video generation models. The Wan series consistently leads in temporal consistency and multiple-object tracking. Source: github.com/Vchitect/VBench

Key Takeaways

HappyHorse's Dominance and the Access Problem

HappyHorse-1.0 is the most technically impressive model on these rankings, and also the least accessible. It appeared without attribution on Artificial Analysis around April 7, climbed to first across both T2V and I2V leaderboards within days, then vanished from the platform before Alibaba confirmed ownership on April 10. The Alibaba-HappyHorse announcement acknowledged the product but didn't announce a commercial availability date.

The architecture details leaked via the model's documentation suggest a 40-layer single-stream Self-Attention Transformer without Cross-Attention, running inference in only 8 denoising steps. It's optimized specifically for human-centric video - facial expression, lip sync, body motion - which explains why it scores particularly well on human-heavy preference prompts in the arena. You can read more in the news coverage of HappyHorse topping the leaderboard.

The practical question for anyone building a pipeline: right now, HappyHorse isn't something you can reliably call. It is a signal of where Alibaba's video research is, not a product.

Sora's Exit and What It Means

OpenAI shut down the Sora consumer app on March 24, 2026. The API continues until September 24. Download numbers had fallen to roughly 1.1 million by February - a 66% decline from peak - and infrastructure costs were estimated at $15 million per day. The TechCrunch analysis of the shutdown describes a product that failed commercially while technically competitive. Disney had committed $1 billion to a partnership; that agreement ended when Sora did.

OpenAI has said the research team continues working on "world simulation" for robotics applications. Whether a successor video model appears under a different product structure isn't clear.

Seedance's Legal Limbo

ByteDance's Seedance 2.0 is technically strong - Elo 1,268 for T2V, second only to HappyHorse - but its commercial arc is uncertain. The Motion Picture Association sent a cease-and-desist letter in February 2026 over "unauthorized use of US copyrighted works on a massive scale," with Disney, Warner Bros., Paramount, Netflix, and Sony joining. SAG-AFTRA cited unauthorized use of member voice and likeness. ByteDance paused the global rollout in March. More context in our Seedance Hollywood copyright coverage.

The model remains accessible in China and through some API providers. For international commercial production use, its legal exposure is a real risk to factor in.

The T2V vs. I2V Split

Text-to-video and image-to-video are distinct tasks that require different model strengths. T2V demands creative interpretation of prompts and consistent world-building from nothing. I2V demands that the model respect an existing visual starting point - maintain object identity, honor the lighting and composition, produce motion that feels native to the input image.

Kling's performance shows the distinction well. Kling 3.0 ranks 3rd in T2V (Elo 1,247) and 9th in I2V (Elo 1,282 for the same Pro 1080p model). PixVerse V6, which doesn't appear in the T2V top-10, reaches rank 4 in I2V (Elo 1,314). Users building workflows should test both tasks explicitly rather than assuming a single ranking applies to both use cases. Our best AI video generators guide covers this trade-off in more depth.

VBench-2.0's Harder Questions

The original VBench caught consistency, smoothness, and alignment failures that plagued 2024-era models. Most 2026 frontier models clear those thresholds comfortably. VBench-2.0 turns the dial harder by asking whether videos show physically plausible interactions. When a character pours water, does it flow correctly? When two objects collide, does the motion follow conservation laws? When a person walks, do their limbs move in anatomically plausible sequence?

On action faithfulness alone, even leading models score around 50% in the VBench-2.0 paper's initial evaluation. The four models tested in the March 2026 paper - HunyuanVideo, CogVideoX-1.5, Sora 480p, and Kling 1.6 - showed consistent weaknesses in dynamic attribute changes, where most scored between 8% and 29%.

Physics is the next frontier that benchmarks have put numbers to. Expect this to become the primary differentiator in 2027 rankings.

Practical Guidance

For professional video production (reliably available): Kling 3.0 Pro at Elo 1,247 T2V is the strongest option with stable API access and no current legal complications. It supports native 4K output, multi-shot scene logic, and subject consistency across cuts - which matters for anything longer than a 5-second clip. Our Kling 3.0 review covers hands-on results.

For social media and short-form at low cost: Hailuo 02 from MiniMax generates at $0.28 per clip, runs fast, and scores well in user preference testing for casual and social video styles. It won't beat Kling on cinematic quality, but for volume generation the cost difference is significant.

For open-source / local deployment: Wan 2.2 leads VBench with a 84.7% aggregate score and is Apache 2.0 licensed. LTX-2.3 is the speed pick if latency matters more than absolute quality. HunyuanVideo 1.5 runs on a RTX 4090 at consumer GPU prices and performs well on visual quality sub-metrics even if raw VBench totals aren't publicly confirmed.

For image-to-video specifically: PixVerse V6 (Elo 1,314) outperforms its T2V ranking markedly and is worth testing separately from any T2V benchmarks you're relying on.

For audio-native video: Seedance 2.0 leads the With Audio I2V category (Elo 1,176) and is accessible through some API providers despite the global rollout pause. SkyReels V4 at Elo 1,087 is a reliable alternative with audio.

FAQ

Is Sora still available in April 2026?

The Sora app was shut down March 24, 2026. The Sora API continues until September 24, 2026. No successor product has been announced.

What is VBench-2.0 and how is it different from VBench?

VBench tests 16 dimensions covering video quality and text alignment. VBench-2.0 adds 18 fine-grained dimensions testing intrinsic faithfulness - physics plausibility, human motion realism, commonsense causality, and creative composition. Most current models score below 60% on VBench-2.0 physics tasks.

Which video model has the best Elo score right now?

HappyHorse-1.0 from Alibaba leads both T2V (Elo 1,361 no-audio) and I2V (Elo 1,398 no-audio) on Artificial Analysis as of April 2026, but public API access isn't yet available.

What is the best video model I can actually use today?

For general availability with strong quality, Kling 3.0 Pro (T2V Elo 1,247) or Veo 3 from Google (Elo 1,217) are solid choices. Open-source developers should look at Wan 2.2 or LTX-2.3.

Can I use Seedance 2.0?

Seedance 2.0 is accessible in China and through some API providers. ByteDance paused the global consumer rollout in March 2026 due to copyright disputes with major Hollywood studios. Commercial use in international markets carries legal uncertainty.

Sources:

• Artificial Analysis Text-to-Video Leaderboard
• Artificial Analysis Image-to-Video Leaderboard
• VBench Project Page (Vchitect, CVPR 2024)
• VBench GitHub Repository
• VBench-2.0 Paper - arXiv 2503.21755
• Why OpenAI Really Shut Down Sora - TechCrunch, March 29 2026
• Sora Shutdown Coverage - CNN Business, March 24 2026
• Disney Exits OpenAI Deal After Sora Closure - Hollywood Reporter
• MPA Cease-and-Desist to ByteDance Over Seedance 2.0 - Hollywood Reporter
• ByteDance Pledges Fixes to Seedance 2.0 After Hollywood Claims - Al Jazeera
• Alibaba Reveals HappyHorse AI Video Model - Creati.ai, April 10 2026
• Runway Gen-4.5 Introduction - Runway Research

This leaderboard tracks how the major frontier models actually perform on dedicated function calling and tool-use benchmarks. We cover five distinct evaluations: BFCL v3 (single and multi-turn structured calls), tau-bench (multi-turn tool use in realistic customer service scenarios), ToolBench (multi-step real-world API tasks), FinTrace (long-horizon financial tool calling), and MCP-Bench (agentic tool use via Model Context Protocol). Each captures something the others miss.

If you're coming from our agentic AI benchmarks leaderboard, the coverage here is more focused: we're looking specifically at the tool-calling layer, not the full agent trajectory.

The Benchmarks Explained

Before reading the tables, it helps to know what each benchmark actually tests - because the gap between a 76% BFCL score and a 70% tau-bench score isn't just a number difference. They measure different things entirely.

BFCL v3 (Berkeley Function Calling Leaderboard)

The Berkeley Function Calling Leaderboard from UC Berkeley's Sky Computing Lab is the most widely cited tool-use benchmark. Version 3 added multi-turn interactions on top of the v2 static evaluation. The dataset contains over 2,000 question-function-answer pairs spanning Python, Java, JavaScript, and REST APIs, testing six categories: simple single calls, parallel calls (multiple functions fired simultaneously), multiple function selection, relevance detection (knowing when to refuse a tool call), multi-turn interactions, and multi-step reasoning.

The evaluation method matters here. BFCL uses Abstract Syntax Tree (AST) comparison to check structural correctness of the produced call, not just string matching. That catches paraphrasing games where a model rewrites the function name slightly and a naive eval would still pass it. BFCL v4 has since added web search and memory tasks, but v3 remains the most-assessed version for cross-model comparison. Version 4 scores aren't broadly available yet across 2026 frontier models.

tau-bench (Tool-Agent-User Interaction)

tau-bench from Sierra Research simulates live customer service scenarios. An AI agent is given a set of API tools and must complete multi-turn conversations with a simulated user - resolving requests like fare changes, return windows, and account modifications while following strict policy guidelines. A Pass@k metric measures how consistently the agent succeeds across k repeated runs. That penalizes brittleness: a model with 80% single-turn accuracy might only complete multi-turn scenarios reliably 50% of the time because errors compound.

There are two primary domains - airline and retail - with a newer telecom variant (tau2-bench) from the same team. Scores across domains aren't directly comparable because task complexity differs, so we report them separately.

ToolBench and ToolLLM

ToolBench, published by Tsinghua's NLP lab and accepted at ICLR 2024, tests models against over 120,000 instruction-API pairs drawn from 16,000+ real-world APIs across 49 categories. It stresses tool generalization: models are assessed on unseen tools, unseen instructions, and unseen API categories to test whether they've truly learned to use APIs or just memorized training examples. The ToolLLM paper describes the fine-tuned model trained on ToolBench data; the benchmark itself runs any model through the same evaluation harness.

FinTrace

Published in April 2026, FinTrace from Carnegie Mellon and a consortium of financial AI researchers targets a specific failure mode: models that pick the right tool but then fail to use the output correctly. The benchmark contains 800 expert-annotated trajectories across 34 financial task categories, scored across nine metrics along four axes - action correctness, execution efficiency, process quality, and output quality. It's the only benchmark in this roundup that explicitly measures whether the model reasons well about tool outputs, not just whether it invokes the right function.

MCP-Bench

MCP-Bench from Accenture Labs connects LLMs to 28 live MCP servers spanning 250 tools across domains including finance, scientific computing, and travel. Accepted to NeurIPS 2025, it tests real multi-hop tool coordination: "book a flight to the cheapest city with a conference this month" requires chaining calendar, flight search, and pricing tools in the correct order. This benchmark is newer and has fewer assessed models than BFCL, but it's the closest thing to a production agentic workload in the published benchmark suite.

BFCL v3 Rankings

Data from pricepertoken.com, accessed April 2026. 23 models assessed; average score 55.9.

The Claude Opus 4 result at 25.3% deserves attention. Anthropic's flagship model, which leads multi-turn agentic benchmarks and beats everyone on tau-bench, scores at the bottom of the BFCL v3 table. That isn't a fluke - it reflects a real tradeoff in how the model handles structured formatting under test conditions. BFCL rewards precise, rigid JSON outputs. Claude tends toward conversational wrapping that can trip up AST parsing even when the underlying tool selection is correct. The discrepancy is a known issue worth understanding before drawing conclusions about real-world usability.

BFCL uses Abstract Syntax Tree comparison to evaluate function call correctness - catching subtle structural errors that string matching misses. Source: pexels.com

tau-bench Rankings

Airline Domain

Data from llm-stats.com, accessed April 2026. 23 models assessed; average score 0.495.

Retail Domain

Data from llm-stats.com, accessed April 2026. 25 models assessed; average score varies.

The pattern is consistent: Anthropic holds the top five retail positions, with Claude Sonnet 4.5 posting 0.862 - 3.8 points ahead of the next model. GLM-4.5 at sixth (0.797) is the first non-Anthropic model, and it scores higher than GPT-4.5 (0.684), which is worth noting for cost-sensitive buyers.

tau2-bench Telecom (Extended Domain)

The tau2-bench telecom dataset, tracked at artificialanalysis.ai, extends the original benchmark into telecommunications customer service with more complex policy trees. GPT-5.2 Thinking set a high bar at 98.7%, and GLM models have since pushed the reported ceiling even higher.

One important caveat: these telecom scores look spectacular, but the benchmark's specific policy constraints also make it more susceptible to overfitting. No model scored above 49% when the paper was first published. The rapid climb to 98%+ suggests some combination of genuine capability improvement and possible training data exposure. Treat telecom scores as directionally useful, not as definitive.

Multi-turn tool use benchmarks like tau-bench are harder to game than single-turn evaluations because errors compound across multiple conversation turns. Source: pexels.com

FinTrace Rankings (Financial Tool Calling)

FinTrace, published April 2026 (arXiv:2604.10015), evaluated 13 LLMs across 800 annotated financial task trajectories. The rubric grades nine metrics along four axes - action correctness, execution efficiency, process quality, and output quality.

The FinTrace authors found that frontier models handle tool selection reliably but consistently struggle with information use and final answer quality. Picking the right function to call isn't the hard part anymore - doing something useful with the result is. That finding holds across all 13 models tested, including the top scorers.

Scale Labs Enterprise Tool Use

For a broader single-turn tool use perspective, Scale Labs runs an enterprise evaluation against 35 models. Top performers as of the most recent snapshot at labs.scale.com/leaderboard/tool_use_enterprise:

This leaderboard uses older model checkpoints (the most recent version evaluated is from early 2025), so it doesn't reflect current frontier capability. It's still useful as a reference for enterprise buyers comparing older deployments and for seeing how reasoning models (o1, o1 Pro) compare to standard models on structured API tasks.

Key Takeaways

The BFCL-tau-bench split tells the real story

The biggest insight from these numbers is that BFCL and tau-bench don't agree on model ranking - and they shouldn't, because they're testing different things. BFCL rewards single-call precision; tau-bench rewards sustained reliability across a multi-turn conversation. Claude Opus 4 scores 25.3% on BFCL and 0.814 on tau-bench retail. That isn't a contradiction - it means Claude handles multi-turn tool orchestration well but formats outputs in ways that trip up BFCL's AST parser.

For most production use cases, tau-bench scores matter more. Real agents don't make a single tool call and stop.

Open-weight models are competitive on structured calling

GLM-4.5 tops BFCL v3 at 76.7%, and Qwen3 32B is right behind at 75.7%. Both are available open-weight. Anthropic's closed models dominate tau-bench, but for applications that need raw function-call accuracy without the conversational overhead, the open-weight options are genuinely strong. Our guide to running open-source LLMs locally covers setup if you want to benchmark these yourself.

The "thinking" premium is marginal for tool calling

Several model pairs in the BFCL v3 table show identical scores for the base and "Thinking" variant (GLM-4.7-Flash and GLM-4.7-Flash Thinking both score 74.6%; Kimi K2.5 and Kimi K2.5 Thinking both score 64.5%). Extended chain-of-thought doesn't help on well-formed single-call evaluations. It helps on complex multi-step planning - which is what tau-bench and FinTrace measure. So the decision of whether to use a reasoning model should depend on your task structure, not just the model tier.

FinTrace exposes the output problem

The FinTrace finding - that all models struggle with information utilization more than tool selection - points to the next frontier in function calling research. Models have gotten good at choosing the right tool. They haven't gotten comparably good at reasoning over the result before taking the next step. This matters enormously for financial, medical, and legal agent workflows where a tool returns a document and the agent needs to extract the right number from it before continuing.

Practical Guidance

Building a customer service or task automation agent: Use Claude Sonnet 4.5 or Claude Opus 4. Both consistently lead tau-bench across domains, and the multi-turn reliability gap between them and GPT-4.1 is large enough to matter in production. See our review of Claude Opus 4.6 for a deeper look at Anthropic's flagship line.

Structured data extraction or API integration with exact schema requirements: Check the BFCL v3 table and prioritize GLM-4.5 or Qwen3 32B if output format compliance is your bottleneck. If you're running closed APIs through OpenAI, Nova Pro 1.0 (67.9% on BFCL v3) sits well above GPT-4.1 on structured calling.

Multi-hop tool chains (MCP, complex pipelines): MCP-Bench data is limited, but its findings indicate that cross-tool coordination and parameter precision are still hard problems regardless of model tier. Our guide on what MCP is and how to use it covers the protocol layer if you're designing the tool interface rather than just selecting a model.

Financial or high-stakes domain tool use: FinTrace scores put Claude Opus 4.6 (0.788) and Claude Sonnet 4.6 (0.750) ahead of GPT-5.4 (0.737). The gap isn't large, and the benchmark itself is new enough that you should treat these numbers as signals rather than verdicts. Fine-tune on domain-specific tool trajectories if stakes are high - FinTrace-Training (8,196 annotated trajectories) is public.

Methodology Notes and Caveats

A few things to keep in mind when interpreting these tables:

BFCL v3 evaluates models as of the snapshot date and doesn't account for system prompt changes, context windows, or whether the model is being called with tool-use instructions or not. Provider settings matter.

Tau-bench is stochastic - it uses an LLM to simulate the user, and scores vary across runs. The Pass@k metric helps, but single published numbers should carry error bars that most leaderboard aggregators don't show. The tau-bench airline and retail numbers above come from llm-stats.com's snapshot and may not match Sierra Research's official site for newer models.

FinTrace and MCP-Bench are 2025-2026 publications with limited model coverage. They're worth watching as evaluation harnesses, but neither yet has the breadth of BFCL.

The Scale Labs enterprise leaderboard has excellent methodology documentation but runs on older model versions. Don't use it to compare GPT-5 vs. Claude 4.x.

FAQ

Which model is best for function calling overall?

No single model leads every benchmark. For multi-turn tool use, Claude Sonnet 4.5. For single-call format precision, GLM 4.5. For financial workflows, Claude Opus 4.6.

Why does Claude score so low on BFCL but high on tau-bench?

BFCL uses AST parsing to check output format. Claude wraps responses conversationally, which fails the parser even when tool selection is correct. Tau-bench measures task completion, where Claude's reasoning over outputs is an advantage.

Are open-weight models competitive with closed APIs for tool use?

On BFCL v3, yes - GLM 4.5 and Qwen3 32B beat every closed API in the table. On tau-bench multi-turn tasks, Anthropic closed models hold a consistent lead, though GLM-4.5 runs competitively at rank 3-4.

How often do these rankings change?

BFCL updates with new model submissions irregularly - major frontier releases tend to appear within weeks of launch. Tau-bench and FinTrace are less frequently updated. Check the source leaderboards linked in the Sources section for the latest snapshots.

What's the difference between tau-bench and tau2-bench?

The original tau-bench covers airline and retail customer service. Tau2-bench adds a telecom domain with more complex policy constraints. Scores aren't comparable across domains.

Does function calling performance transfer to MCP tool use?

Partially. BFCL and tau-bench measure direct function calls in controlled setups. MCP-Bench adds tool discovery and cross-server coordination. Models that score well on BFCL don't automatically handle MCP workflows well, according to the Accenture paper's findings.

Sources:

• Berkeley Function Calling Leaderboard (BFCL) V4
• BFCL v3 model rankings - pricepertoken.com
• BFCL benchmark - llm-stats.com
• tau-bench GitHub repository
• tau-bench airline - llm-stats.com
• tau-bench retail - llm-stats.com
• BenchLM tau-bench snapshot
• tau2-bench telecom leaderboard - Artificial Analysis
• FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling (arXiv:2604.10015)
• MCP-Bench: Benchmarking Tool-Using LLM Agents (arXiv:2508.20453)
• ToolBench - OpenBMB GitHub
• Scale Labs Enterprise Tool Use Leaderboard
• Best AI for Tool Calling - llm-stats.com
• BenchLM LLM Agent and Tool-Use Benchmarks
• Nexus Function Calling Leaderboard - Hugging Face
• ToolSpec: Accelerating Tool Calling (arXiv:2604.13519)

This article covers all 12 options with verified pricing from official pages checked in April 2026, benchmark data from VectorDBBench and Qdrant's open-source benchmark suite, and honest commentary on where the marketing oversells reality.

If you're looking at the end-to-end retrieval stack - frameworks like LangChain, LlamaIndex, and Haystack that sit on top of these databases - see our best RAG tools comparison. For embedding model selection, the MTEB leaderboard coverage has the current rankings. If you're newer to the architecture pattern, what is RAG covers the fundamentals.

The Benchmark Reality Check

Before the individual reviews, some important context on the numbers floating around this space.

Every vendor publishes benchmarks that favor their product. Qdrant's open-source benchmark suite tests only open-source engines, which is at least methodologically consistent. Zilliz's VectorDBBench includes managed cloud services but was built by the Milvus maintainers. Redis's benchmark blog put Redis 3.4x ahead of Qdrant in QPS - at lower recall thresholds where Redis traded accuracy for speed.

The numbers that matter for production RAG workloads are: p99 latency under concurrent load and recall at k=10 with 95%+ threshold. Single-threaded p50 benchmarks don't tell you what happens when your app has real traffic.

With that caveat: at 1M vectors (768 dimensions), Qdrant's benchmark shows it achieving roughly 1,200 QPS at 99% recall, Milvus indexing faster than most alternatives but showing some degradation at 10M+ vectors in lower dimensions, and Elasticsearch running up to 10x slower at 10M vectors of 96 dimensions compared to purpose-built engines.

Turbopuffer's architecture gives it 10ms p90 warm-cache performance but 444ms p90 cold-cache, which matters if you have a long tail of rarely-queried namespaces.

Comparison Table

The storage and compute infrastructure behind managed vector database services runs on the same data center hardware - the real differentiation is in the index structure, query engine, and operational experience above the metal. Source: unsplash.com

Pinecone - Best for Teams That Want Zero Ops

Pinecone's serverless architecture is still the fastest path from zero to a working vector search endpoint. There's no cluster to configure, no index parameters to tune, and the free Starter tier includes 2GB storage, 2M write units/month, and 1M read units/month with no credit card required.

The Standard plan has a $50/month minimum and charges $0.33/GB/month for storage, $4/million write units, and $16/million read units. Enterprise is $500/month minimum and adds 99.95% uptime SLA, private networking, and HIPAA support.

The read unit cliff

Pinecone's pricing model has a structural problem at scale that their docs don't highlight enough. Read unit consumption is 1 RU per 1GB of namespace queried, with a 0.25 RU minimum per query. That seems fine until your namespace grows. At 50GB of vectors, a single query costs 50 RUs. Run 5M queries/month against that namespace and you're at 250M RUs - which is $4,000/month in read costs alone on Standard. Self-hosting Qdrant on a 16GB RAM node would run $96/month for the same data.

Pinecone's query API is proprietary. Migrating off at scale is painful because there's no standard wire protocol. That's worth pricing into your decision upfront.

Integrations: LangChain, LlamaIndex, Haystack, Vertex AI, AWS Bedrock - all first-class support.

Qdrant - Best Open-Source Performance

Qdrant is written in Rust and built specifically for vector similarity search with complex metadata filtering. The cloud free tier gives you a single-node cluster permanently at no cost. Paid cloud nodes start at $0.014/hour (roughly $10/month for the smallest configuration). A 16GB RAM / 4 vCPU production cluster on AWS via Qdrant Cloud runs roughly $96/month with no per-query billing.

Self-hosting is free under Apache 2.0 with no limits other than your hardware. A three-node production cluster on AWS typically runs $300-500/month depending on instance types.

Where it actually leads

Filtered vector search is where Qdrant's engineering shows most clearly. When you combine similarity search with strict metadata conditions - "find the 10 most similar documents where source=legal and date>2025-01-01" - Qdrant's payload filtering engine consistently beats alternatives in Qdrant's own benchmarks (which, again, should be reproduced independently before you bet your architecture on them).

The benchmark suite at qdrant.tech/benchmarks shows Qdrant achieving highest RPS and lowest latencies across most tested configurations. Milvus is notably faster at index build time. Redis shows higher raw QPS at lower recall thresholds.

Hybrid search: Supports dense + sparse vector search natively. BM25 integration via sparse embedding models.

Integrations: LangChain, LlamaIndex, Haystack, all have maintained connectors.

Weaviate - Best Native Hybrid Search

Weaviate's main differentiator is that hybrid search (BM25 + dense vectors with score fusion) ships as a first-class query primitive, not a bolted-on feature. You don't need to run a separate text search index and merge results at the application layer. The built-in hybrid query parameter handles it in one call.

Cloud pricing shifted in October 2025. The Flex plan starts at $45/month (shared deployment, pay-as-you-go), Plus starts at $400/month (dedicated or shared, prepaid contract). Pricing dimensions now include vector dimensions ($0.00975-$0.01668 per million depending on compression method), storage ($0.2125-$0.31875 per GiB), and backup ($0.022-$0.033 per GiB). For 10M objects at 1,536 dimensions without compression, expect roughly $1,459/month before backup costs.

Compression matters here. Weaviate supports product quantization (PQ) and binary quantization (BQ), which can reduce vector storage by 4-32x at some recall cost. For most RAG workloads where you're trading off a few percentage points of recall for 8x lower storage cost, the math usually works.

Self-hosted Weaviate is free and open-source. Docker Compose deployment is straightforward for development; Kubernetes is the production path.

Weaviate's vectorizer modules

One truly useful feature: Weaviate can call embedding models automatically on data import through its vectorizer module system. You configure which model to use, push raw text, and Weaviate handles the embedding calls. This reduces the application code needed to maintain an embedding pipeline. The trade-off is that it obscures the embedding step and creates a tight coupling between your database and your embedding provider.

Milvus / Zilliz Cloud - Best for Billion-Scale

Milvus is the open-source project; Zilliz Cloud is the fully managed version. At the scale where most databases start struggling - 100M+ vectors, high write throughput - Milvus was architecturally designed for it. It separates storage, compute, and indexing into distinct services, which means you can scale each dimension independently.

Zilliz Cloud pricing in early 2026: the free tier includes 5GB storage and 2.5M vCUs monthly. Serverless charges $4 per million vCUs (virtual compute units). Dedicated clusters start at $99/month. In October 2025, Zilliz introduced tiered storage delivering a 87% storage cost reduction and standardized storage pricing at $0.04/GB/month across AWS, Azure, and GCP.

Milvus 2.6.x (now on Zilliz Cloud) introduced cloud-only index optimizations that further reduce TCO for billion-scale deployments.

The ops burden

Running Milvus self-hosted in production is real work. It requires etcd, MinIO (or S3), and multiple Milvus service components. The Helm chart works, but understanding what each component does and how to size it takes time. Zilliz Cloud removes this completely, at cost. For teams without dedicated infrastructure engineers, self-hosting Milvus at production scale is probably the wrong call.

Integrations: Full support in LangChain, LlamaIndex, and Haystack.

Chroma - Best for Local Development

Chroma is the default vector database for tutorials and early prototyping because the API is genuinely simple and the Python library installs in one command. The 2025 Rust rewrite improved write and query performance significantly over the original Python implementation.

There's no cloud managed offering. Chroma is embedded, in-memory first (with optional disk persistence), and designed for single-node deployments. Self-hosting on a 4GB VPS costs under $30/month and handles millions of embeddings for most development workloads.

The honest assessment: Chroma doesn't belong in production RAG systems handling more than a few million vectors with concurrent traffic. The memory-first architecture hits a wall when data no longer fits in RAM. For anything production at scale, it's a stepping stone to a proper deployment.

Hybrid search: Basic keyword + vector combination, less capable than purpose-built hybrid search in Qdrant or Weaviate.

pgvector - Best If You Already Run Postgres

Pgvector extends PostgreSQL with vector storage and approximate nearest-neighbor search via HNSW and IVF_FLAT indexes. If your application already runs on Postgres, adding vector search costs you nothing in infrastructure. Your documents and embeddings live in the same table, metadata filtering uses standard SQL WHERE clauses, and there's no synchronization pipeline to maintain.

The gradual cost is effectively zero if you have spare capacity on your existing database. A dedicated Postgres instance for a vector workload runs $30-80/month depending on size. Every managed Postgres provider - Supabase, Neon, RDS, AlloyDB, Azure Database for PostgreSQL - supports pgvector as an extension.

The limits

Pgvector's performance at large vector counts (50M+) at high QPS trails purpose-built engines. It's not designed for billion-scale vector workloads. But most RAG applications don't have billion-scale vector workloads. They have a few million documents, moderate query rates, and a team that already knows how to operate Postgres. For that profile, pgvector is often the correct answer.

The pgvecto.rs extension from TensorChord is worth knowing as an alternative - it's written in Rust and shows better performance at scale than the original C implementation, with the same SQL API.

Hybrid search: Combine with pg_search (Apache-2.0, by ParadeDB) for BM25 full-text search in the same Postgres instance.

Redis Vector Search - Best for Low-Latency Requirements

Redis 8.4 (released early 2026) introduced FT.HYBRID, a unified command that fuses full-text BM25 and vector similarity results within a single execution plan. Previous versions required merging results at the application layer. This is a genuine improvement for hybrid search use cases.

Redis Cloud pricing: a permanent free tier with 30MB (not useful for production), Essentials from $0.007/hour ($5/month), and Pro from $0.014/hour ($200/month minimum). Pro adds dedicated resources, active-active replication, and 99.999% uptime.

The architectural advantage of Redis for vector search is sub-millisecond latency for hot data. Everything lives in RAM. For real-time RAG where you're serving cached context to high-traffic endpoints, Redis can hit latencies that dedicated vector databases can't match.

The trade-off: RAM is expensive. At $2+/GB for in-memory storage versus $0.07/GB for object storage (Turbopuffer's model), the economics diverge quickly at scale. Redis benchmarks at 3.4x higher QPS than Qdrant in Redis's own testing - but at lower recall thresholds. At matched recall rates, the gap narrows considerably.

Vector databases differ most in how they manage the index structures above the storage layer - HNSW, IVF, and LSM-based approaches each make different trade-offs between memory usage, query latency, and update throughput. Source: unsplash.com

LanceDB - Best for Multi-Modal and Columnar Workloads

LanceDB is built on the Lance columnar format and is architecturally different from most alternatives in this list. It handles larger-than-memory datasets gracefully because data lives on disk in a columnar format, not in RAM. That makes it well-suited for multi-modal RAG (text, images, audio, video in the same index) and for workloads where the vector corpus is too large to fit in memory but query latency doesn't need to be sub-millisecond.

LanceDB OSS is free and embedded - it runs inside your Python application process with no separate server. LanceDB Cloud is in public beta with usage-based pricing and $100 in free credits. The cloud offering adds automatic versioning, managed infrastructure, and SQL query support across the managed dataset.

Zero-copy versioning (automatic, no extra infrastructure) is a useful feature for ML workflows where you want to roll back to previous dataset snapshots.

Integrations: LangChain, LlamaIndex.

Turbopuffer - Best Cost Efficiency at Scale

Turbopuffer is the most architecturally interesting entry in this list. It builds vector search on object storage (S3) as the primary data layer, with a memory and SSD cache sitting in front. The cost difference is significant: incumbents using RAM + 3x SSD for index storage pay roughly $1,600/TB/month; Turbopuffer's S3 + SSD cache model runs $70/TB/month.

This is why Cursor, Anthropic, Notion, Linear, and Superhuman use it. A Cursor co-founder noted they saved "an order of magnitude in costs" after switching their vector database to Turbopuffer.

At the production scale it's operating at - 3.5T+ documents, 10M+ writes/second, 25k+ queries/second - the architecture clearly works.

The trade-off is cold-query latency. When data isn't in the SSD cache, query times are 285-444ms p90. For workloads with uniform query distribution across a large corpus, that's fine. For workloads where cache hit rate is high, warm performance is 10-18ms p90. If your use case has a long tail of infrequently accessed namespaces, cold-cache latency is a real concern.

Pricing: $64/month minimum spend. No free tier.

Hybrid search: Supported (vector + full-text).

Elastic Vector Search / OpenSearch k-NN

Both are solid choices when you already run an Elasticsearch or OpenSearch cluster for full-text search and want to add vector capabilities without a new dependency. The integration is natural - vector search and BM25 keyword search run on the same infrastructure, and hybrid search via reciprocal rank fusion works well.

OpenSearch 2.11+ supports hybrid search combining BM25 and vector similarity. Amazon OpenSearch Service pricing is cluster-based (instance hours + storage); dedicated master nodes and data nodes are billed separately, making cost estimation more involved than purpose-built vector databases. AWS also introduced Amazon S3 Vectors in 2026, promising up to 90% lower costs for vector storage in the S3 ecosystem.

Elasticsearch (via Elastic) shows strong performance in filtered vector search benchmarks, though its own comparison against OpenSearch shows Elasticsearch with 60% higher throughput for filtered queries in their testing.

The honest use case: if your team runs Elastic or OpenSearch at scale and you need to add semantic search, don't introduce a separate vector database. If you're starting fresh specifically for vector search, purpose-built options generally offer better QPS/cost at the same recall level.

MongoDB Atlas Vector Search - Best for Existing Atlas Users

Atlas Vector Search is included in your Atlas cluster - it's not a separate product. If you run MongoDB already, semantic search over your documents costs no gradual infrastructure. The free tier (M0) includes Vector Search and gives you 512MB storage. The Flex tier scales to 5GB with pay-as-you-go billing capped at roughly $30/month.

The limitation: vector search performance is constrained by your Atlas cluster size. It runs on the same compute as your operational workload, which means a burst of vector queries can affect your transactional performance if you haven't provisioned dedicated search nodes. Atlas does support dedicated Search Nodes (separate compute for search operations) from the M10 tier, which adds cost but isolates the workload.

For teams running MongoDB as their primary database and adding AI features, Atlas Vector Search removes a deployment dependency. For teams building a greenfield vector search system, the pricing model (cluster-based, not vector-optimized) is harder to predict at scale.

SingleStore - Best for Real-Time Analytics + Vector Hybrid

SingleStore is a distributed SQL database that added HNSW-indexed ANN vector search with a full product quantization implementation. The pitch is different from every other option on this list: you get vector search, full-text search, and high-performance SQL analytics in one system with no data movement between services.

Hybrid filtering in SingleStore uses standard SQL WHERE clauses, JOINs, and aggregations combined with the vector similarity search. There's no separate query language to learn. For applications that need to combine vector similarity with time-series analytics or relational filtering, that consolidation reduces the number of moving parts considerably.

The CEO has publicly argued that purpose-built vector databases will struggle long-term as general-purpose databases with strong vector support become the norm. That's a self-serving position, but it's worth taking seriously at the architecture planning stage.

Pricing is custom (contact sales). A free shared tier exists for evaluation.

VectorDBBench Performance Summary

The table below reflects VectorDBBench results at 1M vectors (768 dimensions, HNSW index) where available. Numbers vary by configuration and hardware - treat these as directional, not definitive.

Best Pick Recommendations

Start here for new projects: If you don't have an existing database that supports vectors and your corpus is under 50M documents, Qdrant cloud is the cleanest starting point. Free cluster, Apache 2.0 open-source, strong filtering, no per-query pricing.

Existing Postgres users: Add pgvector. The operational simplicity of keeping everything in one system outweighs the performance gap until you're running 50M+ vectors at high QPS.

Enterprise managed cloud: Weaviate Dedicated or Zilliz Cloud both handle production scale. Weaviate is better if hybrid search quality is critical; Zilliz Cloud is better if you need very high write throughput and billion-scale vector counts.

Cost-sensitive at scale: Turbopuffer. The S3-native architecture is 10-23x cheaper per TB than RAM-heavy alternatives, and the production track record at Cursor, Notion, and Linear gives it credibility that a newer entrant normally doesn't have.

Already on MongoDB/Elastic/Redis: Don't add infrastructure. Use the vector search capability built into what you already operate. The marginal performance gain from switching to a purpose-built database rarely justifies the operational cost of adding a new service.

For selecting the right embedding model to pair with any of these databases, the RAG benchmarks leaderboard tracks quality metrics across the full retrieval pipeline.

FAQ

Which vector database has the best hybrid search in 2026?

Weaviate has the most mature native hybrid search, fusing BM25 and dense vectors in a single query call. Redis 8.4's FT.HYBRID command is now comparable for in-memory workloads. Qdrant and Milvus both support sparse + dense vector hybrid search.

Is pgvector good enough for production RAG?

For most RAG workloads under 20M vectors with moderate query rates, yes. It's not the highest-performance option, but it removes an infrastructure dependency and uses SQL for filtering. Above 50M vectors or at high concurrent QPS, purpose-built databases pull ahead.

How does Turbopuffer compare to Pinecone on cost?

At scale, Turbopuffer is significantly cheaper. Turbopuffer's S3-first storage runs $70/TB/month vs. RAM-heavy alternatives at $1,600/TB/month. Pinecone's read unit pricing becomes expensive when namespace sizes are large and query volumes are high.

What's the difference between Milvus and Zilliz Cloud?

Milvus is the Apache 2.0 open-source project you self-host. Zilliz Cloud is the fully managed SaaS version built and operated by the Milvus creators (Zilliz Inc), with additional cloud-specific optimizations and enterprise support.

Do I need a dedicated vector database or can I use MongoDB/Postgres?

It depends on scale and team capacity. For most applications adding AI features to an existing MongoDB or Postgres deployment, the built-in vector search is sufficient and avoids operational overhead. Dedicated vector databases offer better performance at large scale but add another system to operate.

Sources

• Pinecone Pricing - Official Pinecone pricing page
• Qdrant Pricing - Official Qdrant cloud pricing
• Qdrant Benchmarks - Open-source vector database benchmark suite
• Weaviate Pricing - Official Weaviate Cloud pricing (updated Oct 2025)
• Zilliz Cloud Pricing - Official Zilliz Cloud pricing
• Turbopuffer Architecture - Turbopuffer blog: architecture and performance details
• Redis Benchmarking Results - Redis vector database benchmark blog post
• Redis 8.4 Hybrid Search - Redis FT.HYBRID announcement
• VectorDBBench - Open-source benchmark tool (Zilliz / Milvus team)
• Ranksquire Pricing Comparison 2026 - Third-party TCO breakdown
• Zilliz Oct 2025 Update - Tiered storage and pricing changes
• MongoDB Atlas Vector Search - Official MongoDB Atlas Vector Search product page

This comparison covers eleven frameworks, pulls from H100 benchmarks published in early 2026, and tries to give you an honest decision framework rather than another feature-checkbox table.

The Benchmark Baseline

All throughput numbers below come from public H100 SXM5 80GB benchmarks running Meta-Llama 3.3-70B-Instruct in FP8 precision unless noted otherwise. For smaller models (Llama 3.1 8B on H100), the SGLang/LMDeploy lead is wider (roughly 16,200 tok/s vs 12,500 tok/s for vLLM).

Hardware: NVIDIA H100 SXM5 80GB (single GPU) for TRT/SGLang/vLLM rows. LMDeploy row: A100 80GB, Llama 3 70B Int4, 100 concurrent users. Ollama row: RTX 4090. Sources: Spheron benchmarks, Prem AI comparison.

Modern GPU clusters run multiple inference servers simultaneously, making the choice of serving framework a critical infrastructure decision. Source: unsplash.com

vLLM - The Reliable Default

GitHub: vllm-project/vllm | License: Apache 2.0 | Latest: v0.19.0 (Apr 3, 2026)

vLLM started at UC Berkeley and has built up 77k GitHub stars and 2,000+ contributors. It remains the reference implementation for PagedAttention - a KV cache management technique that treats GPU memory like virtual memory, partitioning the cache into fixed-size blocks that can live in non-contiguous memory. The result is less than 4% memory waste under typical workloads, compared to 60-80% fragmentation in naive implementations.

Strengths

• Model breadth: 200+ architectures natively supported from HuggingFace, including decoder-only LLMs, MoE models (DeepSeek-V3, Mixtral), multimodal models (LLaVA, Qwen-VL), and embedding models
• Quantization coverage: FP8, INT4, INT8, GPTQ, AWQ, GGUF, and compressed_tensors - essentially everything
• Hardware portability: NVIDIA GPUs, AMD GPUs, CPUs, TPUs, and specialized accelerators
• Speculative decoding: n-gram, suffix, EAGLE, and DFlash variants supported
• API compatibility: OpenAI-compatible server plus Anthropic Messages API

Weaknesses

• Cold start around 62 seconds - not ideal for auto-scaling fleets
• TTFT p99 is 80% larger than competitors under load in some benchmarks - variance is higher than SGLang
• Slightly behind SGLang and TRT-LLM on raw throughput for larger models

Best for

General-purpose production serving where you need to run multiple model architectures and don't want to maintain separate serving stacks. The ecosystem around vLLM (observability tools, integrations, community support) is unmatched.

SGLang - The Throughput Leader

GitHub: sgl-project/sglang | License: Apache 2.0 | Latest: v0.5.10.post1 (Apr 9, 2026)

SGLang is the inference engine that crept up on vLLM's market share by solving a specific problem better: prefix reuse. RadixAttention uses a radix tree data structure to automatically identify and cache shared KV blocks across requests. When multiple users start with the same system prompt, the same document in a RAG pipeline, or the same few-shot examples, those tokens are computed once and reused.

The cache hit rates are dramatic: 85-95% for few-shot workloads vs 15-25% for vLLM's prefix caching, and 75-90% for multi-turn chat vs 10-20% for vLLM. That compounding reuse is why SGLang delivers 3.1x faster inference than vLLM on DeepSeek V3 specifically.

Strengths

• Structured output: xGrammar backend delivers up to 10x faster JSON decoding than alternatives, with compiled grammars cached in the radix tree for reuse
• Scale: Powers 400,000+ GPUs globally, trillions of tokens daily. Production users include xAI, AMD, NVIDIA, LinkedIn, and Cursor
• Multi-GPU: Tensor, pipeline, expert, and data parallelism
• Quantization: FP4/FP8/INT4/AWQ/GPTQ across NVIDIA GB200/B300/H100 and AMD MI-series
• Prefill-decode disaggregation: Separates the prompt processing phase from token generation for better resource use

Weaknesses

• Smaller community than vLLM (26k stars vs 77k)
• Slightly steeper operational learning curve
• Fewer supported model architectures than vLLM

Best for

Any workload with shared prefixes: RAG pipelines, document Q&A, multi-turn chat agents, structured JSON extraction. For anything involving AI agent frameworks, SGLang's prefix caching compounds into a significant cost advantage.

TensorRT-LLM - Maximum Performance, Maximum Friction

GitHub: NVIDIA/TensorRT-LLM | License: Apache 2.0 | Latest: v1.2.0

NVIDIA TensorRT-LLM is the performance ceiling for LLM serving on NVIDIA hardware. The numbers are real: 13% higher throughput than vLLM at 50 concurrent requests, best TTFT across every concurrency level. On H100 and later GPUs, FP8 quantization doubles performance versus FP16. On B200 GPUs, FP4 inference is natively supported.

The cost is compilation. Every new model requires building a TensorRT engine, a process that takes around 28 minutes. After that first compile, subsequent warm starts drop to ~90 seconds. But any model update, weight change, or configuration tweak triggers a full recompile.

Strengths

• Highest throughput and lowest TTFT in class at high concurrency
• In-flight batching maximizes GPU use
• FlashAttention and paged KV caching built in
• Best-in-class on NVIDIA hardware including GB200 (FP4 native)
• Usually rolled out through the Triton Inference Server backend

Weaknesses

• 28-minute engine compilation per model - painful for experimentation
• NVIDIA-only (no AMD, no CPU, no TPU fallback)
• The "open source" label is technically accurate (Apache 2.0) but NVIDIA controls development and the custom kernels make portability a non-starter

Best for

High-volume production deployments where you're running a fixed model at scale and can amortize the compilation overhead. Data centers serving millions of requests per day on H100/B200 fleets. Not recommended if you need to swap models frequently or experiment with architectures.

HuggingFace TGI - Sunsetted

GitHub: huggingface/text-generation-inference | Status: Maintenance mode since Dec 11, 2025

TGI pioneered the OpenAI-compatible API wrapper for HuggingFace models and was the default choice for most teams from 2023 through 2025. On December 11, 2025, HuggingFace's Lysandre Debut announced that TGI had entered maintenance mode - only minor bug fixes and documentation updates will be accepted. New features are done.

Under production load, TGI achieved 68-74% GPU use compared to vLLM's 85-92%, and saturated at 50-75 concurrent requests vs vLLM's 100-150. For existing deployments, TGI will continue working. For anything new, migrate.

Migration path: Drop-in replacement is vLLM or SGLang - both serve the same OpenAI-compatible /v1/chat/completions endpoint.

llama.cpp and Ollama - Development Tools

llama.cpp GitHub: ggml-org/llama.cpp | License: MIT

llama.cpp runs LLM inference in pure C++ with no GPU required. It supports GGUF quantization from 1.5-bit through 8-bit integer, k-quants (Q4_K_M, Q5_K_S, etc.), and KV cache quantization that cuts memory usage by up to 50% during generation. The built-in llama-server exposes an OpenAI-compatible API with Prometheus metrics, and since late 2025 also supports the Anthropic Messages API.

Ollama wraps llama.cpp in a user-friendly CLI and REST interface, adding model management and a clean API. It's excellent for local development, testing prompt changes, and internal tooling.

The production ceiling is real: at 50 concurrent users, Ollama plateaus at roughly 155 tok/s while vLLM maintains 920 tok/s. Response times degrade from 2 seconds to 45+ seconds with just 10 concurrent users. This isn't a criticism - these tools are optimized for a different use case. See our separate local LLM tools comparison for the desktop-first perspective.

Best for

CPU-only inference, air-gapped environments, developer workstations, model prototyping before launching to a GPU cluster.

LMDeploy - The Hidden Strong Contender

GitHub: InternLM/lmdeploy | License: Apache 2.0

LMDeploy from the InternLM team runs the TurboMind inference engine (C++) and matches SGLang's throughput on Llama 3.1 8B (~16,100 tok/s on H100). Its standout metric is latency on quantized models: for Int4 inference on A100 80GB running Llama 3 70B at 100 concurrent users, LMDeploy delivers 700 tok/s with the lowest time-to-first-token across all tested engines. The Int4 performance is 2.4x faster than FP16.

LMDeploy supports persistent batch inference, blocked KV cache, dynamic split-and-fuse, and tensor parallelism. The caveat is narrower model coverage than vLLM and a smaller Western community - most documentation and issues are in Chinese.

Best for

Teams running InternLM models, or anyone prioritizing Int4 latency on A100/A800 hardware.

MLC-LLM - Cross-Platform Compilation

GitHub: mlc-ai/mlc-llm | License: Apache 2.0

MLC-LLM compiles models via Apache TVM to run on CUDA, Vulkan, Metal, and WebGPU. The same compiled artifact runs on Linux, macOS, Windows, iOS, Android, and in-browser via WebGPU. This is the only framework in this list that can serve a model natively on a mobile device or in a browser.

For server-side production throughput, MLC-LLM is not competitive with vLLM or SGLang. Its value is in edge and cross-platform deployment where the runtime can't be NVIDIA-only. It exposes an OpenAI-compatible REST API and supports FP8/INT4/AWQ/GPTQ quantization.

Best for

Edge inference, mobile deployment, browser-based private AI applications, cross-platform CI/CD pipelines.

Aphrodite Engine - Extended Quantization for Creative AI

GitHub: aphrodite-engine/aphrodite-engine | License: AGPL-3.0

Aphrodite is a vLLM fork maintained by the PygmalionAI team, tuned for creative writing and roleplay use cases. Its unique value is quantization breadth: AQLM, AutoRound, AWQ, BitNet, Bitsandbytes, EETQ, GGUF, GPTQ, QuIP#, SqueezeLLM, Marlin, FP2 through FP12, NVIDIA ModelOpt, TorchAO, VPTQ, and MXFP4. It also adds a KoboldAI-compatible API layer and Mirostat sampling.

For general production serving, stick with upstream vLLM - Aphrodite's advantages are niche. But if you need FP2 quantization or KoboldAI API compatibility, it's the only serious option.

DeepSpeed-MII - Microsoft's Inference Layer

GitHub: deepspeedai/DeepSpeed-MII | License: Apache 2.0

DeepSpeed-MII provides inference optimizations from Microsoft Research: blocked KV caching, continuous batching, Dynamic SplitFuse, tensor parallelism, and INT8/ZeroQuant quantization. It supports 37,000+ models and claims up to 2.5x higher throughput than vLLM in its own benchmarks.

The honest assessment: MII's benchmarks predate vLLM's v0.6+ performance improvements, and independent 2026 comparisons show it behind SGLang and current vLLM. It remains relevant for teams already on the Azure/DeepSpeed stack. For greenfield deployments, vLLM or SGLang is the better default.

Triton Inference Server - The Enterprise Orchestration Layer

GitHub: triton-inference-server/server | License: BSD 3-Clause

NVIDIA Triton is general-purpose multi-framework model serving, not a LLM inference engine itself. Its value in this comparison is as the production deployment layer on top of TensorRT-LLM. The TensorRT-LLM backend for Triton adds in-flight batching, paged attention, LoRA support, and leader/orchestrator multi-GPU modes.

Triton handles the enterprise concerns: multi-model serving, health checks, performance metrics, ensemble pipelines, and gRPC/HTTP APIs. If your production stack already uses Triton for computer vision or NLP models, adding LLM serving via the TRT-LLM backend is the natural path.

Multi-GPU inference deployments rely on high-bandwidth networking between nodes - the infrastructure layer matters as much as the serving framework. Source: unsplash.com

Ray Serve - Orchestration, Not a Server

Docs: docs.ray.io/en/latest/serve/llm/index.html | License: Apache 2.0

Ray Serve isn't an inference engine - it's a programmable orchestration layer that wraps engines like vLLM. The ray-llm repository was archived for the built-in ray.serve.llm API (available since Ray 2.44), which handles auto-scaling, load balancing, streaming responses, and multi-model routing.

Ray Serve's superpower is disaggregated serving: prefill and decode phases can be separated across different actor pools, tuning GPU use. Teams serving multiple models or needing fine-grained traffic control will find it valuable. Teams serving a single model with predictable load should just run vLLM directly.

Full Comparison Table

TP = Tensor Parallelism, PP = Pipeline Parallelism, EP = Expert Parallelism

Use-Case Decision Matrix

You want the safest default for production serving - start with vLLM. Broadest model support, active community, Apache 2.0, and solid documentation. Pick SGLang if you confirm prefix-heavy workloads.

Your workload involves RAG or multi-turn agents - switch to SGLang. The RadixAttention prefix caching compounds across requests in ways that vLLM's implementation doesn't match. For RAG tool integrations, this is often a 20-30% cost reduction in practice.

You're running a fixed high-traffic model on H100/B200 - TensorRT-LLM delivers the best throughput numbers. Accept the 28-minute compilation as a one-time build cost per model version.

You need CPU inference, Apple Silicon, or air-gapped deployments - llama.cpp is the right engine. Ollama adds a friendlier interface. Neither scales to high concurrent load.

You're focused on quantized inference budget - LMDeploy for Int4 on A100/A800, especially for InternLM models or any workload where you want lowest TTFT at low precision.

You need edge, mobile, or browser deployment - MLC-LLM is the only viable open-source option.

You're on the Azure/DeepSpeed ecosystem - DeepSpeed-MII integrates naturally and offers solid performance within that stack.

You need structured JSON outputs at scale - SGLang with xGrammar. The compiled grammar caching is unique in the space.

You're running many models or need routing/autoscaling - Ray Serve as the orchestration layer, wrapping vLLM or SGLang underneath. Check the AI speed and latency leaderboard for how these stack up under real routing conditions.

You want to benchmark your own hardware - LLMFit is a useful starting point for calibrating expected throughput against your available GPU fleet.

FAQ

Is vLLM still the best inference server in 2026?

vLLM is the most flexible default, but SGLang and TensorRT-LLM beat it on raw throughput. The right choice depends on workload: vLLM for broad model support, SGLang for prefix-heavy or structured output workloads, TRT-LLM for maximum single-model throughput.

Should I migrate away from TGI?

Yes, for anything new. TGI entered maintenance mode on December 11, 2025, and no new features will be added. Both vLLM and SGLang are drop-in API replacements serving the same OpenAI-compatible endpoints.

Does SGLang support all the same models as vLLM?

Not yet. vLLM supports 200+ HuggingFace architectures; SGLang covers fewer. For the most recently released or niche models, vLLM is more likely to have support. Check the SGLang docs for the current model list before committing.

When does TensorRT-LLM make sense?

When you are serving a fixed model at sustained high concurrency on NVIDIA H100 or B200 hardware, and the 28-minute compilation cost is amortized over millions of requests. Not suitable for experimentation or multi-model serving.

Is Ollama suitable for production?

At low concurrency (under 4-8 simultaneous users), yes. Under higher load, response times degrade sharply and throughput plateaus. For customer-facing production serving, use vLLM or SGLang.

Sources

• vLLM GitHub (v0.19.0)
• SGLang GitHub (v0.5.10)
• NVIDIA TensorRT-LLM GitHub
• HuggingFace TGI GitHub
• llama.cpp GitHub
• InternLM LMDeploy GitHub
• MLC-LLM GitHub
• Aphrodite Engine GitHub
• DeepSpeed-MII GitHub
• Triton Inference Server GitHub
• Ray Serve LLM Docs
• Spheron vLLM vs TRT-LLM vs SGLang H100 Benchmarks
• Prem AI: vLLM vs SGLang vs LMDeploy 2026
• Prem AI: LLM Inference Servers Compared 2026
• Lysandre Debut - TGI Maintenance Mode Announcement
• vLLM Automatic Prefix Caching Docs

Google is in active negotiations with the Department of Defense to deploy Gemini AI on classified networks, according to The Information, which cited two people with direct knowledge of the talks. If finalized, the deal would take Google well beyond its existing unclassified footprint at the Pentagon and into the sensitive data environments where Anthropic's Claude was rolled out before the company was blacklisted in March.

The talks are the clearest sign yet that the Anthropic fallout has reshuffled the government AI market in ways that benefit Google directly.

The Proposed Terms

The Pentagon has pushed all AI vendors to accept contracts covering "all lawful purposes" - language designed to give the military maximum flexibility over how it uses AI tools. Anthropic refused to remove carve-outs for domestic mass surveillance and fully autonomous weapons systems. The Pentagon responded by designating Anthropic a supply chain risk on March 4 - the first American company ever to receive that designation.

Google's proposed terms in these negotiations reportedly include the same two restrictions: Gemini would be prohibited from use in domestic mass surveillance and from controlling autonomous weapons without appropriate human oversight.

"The Pentagon will continue to rapidly deploy frontier AI capabilities to the warfighter through strong industry partnerships across all classification levels."

That statement, from a Pentagon spokesperson, didn't confirm the Google talks specifically. But the phrase "all classification levels" carries its own weight. Google's Under Secretary of Defense contact, CTO Emil Michael, made the direction explicit in March: "I have high confidence they're going to be a great partner on all networks." Networks, plural. The unclassified GenAI.mil deployment, where Google already serves 3 million Pentagon workers, was the first step.

How Google Got Here

Google's path to this negotiation runs through a deliberate reversal of its own stated principles.

In 2018, roughly 3,100 Google employees signed a letter demanding the company exit Project Maven, a DoD contract using machine learning to analyze drone footage. Google walked away when the contract expired in 2019. That decision was treated internally as a commitment: Google wouldn't build AI for warfare.

By early 2025, Google had quietly removed those restrictions. The company signed a "$200 million prototype" contract with the Pentagon in July 2025 with Anthropic, OpenAI, and xAI. In August it announced Gemini for Government. In December it became the first enterprise AI provider on GenAI.mil at launch. By March 2026, eight pre-built Gemini-powered agents were running across the unclassified tier, with 1.2 million unique users already on the platform.

The classified-network talks are the logical next step in that progression.

Sundar Pichai at a 2023 regulatory meeting. Google's leadership made a calculated decision to re-enter military AI after the 2018 Project Maven retreat. Source: commons.wikimedia.org

Who Benefits, Who Pays

Companies

Google's classified-network pitch puts it in direct competition with Microsoft, which has long served the intelligence community through dedicated Azure Government regions, and with Palantir, whose tools served as the conduit for classified Claude deployments before the Anthropic blacklisting. OpenAI signed its own Department of War agreement within hours of the Anthropic designation - an approach its own CEO later described as "opportunistic and sloppy" in internal communications reported by The Atlantic.

The Pentagon's stated strategy is deliberate vendor diversification. That makes the Google talks less about winner-takes-all and more about which vendors are willing to operate within terms the DoD will accept.

Users

Inside Google, the atmosphere is different from 2018. Employees who oppose military AI use are still organizing - the 100-person internal letter and the broader 430-person joint letter with OpenAI workers are real acts of dissent - but those involved acknowledge the company's direction has already changed. The fight has shifted from "get Google out of defense" to "force Google to maintain limits."

Whether those limits survive the negotiation is the open question.

Competitors

The Anthropic situation offers a useful data point. Pentagon CTO Emil Michael's public criticism of Anthropic's position was direct: "What we're not going to do is let any one company dictate a new set of policies above and beyond what Congress has passed." The framing cast Anthropic's safety restrictions as anti-democratic overreach.

Google is attempting to win the same restrictions - no mass surveillance, no autonomous weapons - through negotiation rather than as non-negotiable policy. If the Pentagon accepts those terms from Google, it'll be hard to argue the Anthropic blacklisting was ever really about the substance of the restrictions.

Google's cloud infrastructure will form the backbone of any classified Gemini deployment. The Pentagon already routes 1.2 million users through unclassified Gemini services. Source: commons.wikimedia.org

The Anthropic Parallel

The Trump administration's handling of Anthropic has been contradictory enough that some observers question whether the blacklisting was ever primarily about safety policy. On March 4, the Pentagon designated Anthropic a supply chain risk. Within hours it signed a new agreement with OpenAI. The same week, Treasury Secretary Bessent was meeting with Wall Street banks to encourage Claude adoption in financial services.

Anthropic's injunction lawsuit secured a preliminary block on the ban's enforcement - a federal judge found evidence of First Amendment retaliation. But Anthropic lost an appeals court bid on April 8 to extend that protection pending the full litigation, leaving the legal picture split. Classified Claude deployments, which ran through Palantir and AWS, remain in a months-long phase-out.

Google's classified-network talks advance during that phase-out. The timing isn't subtle.

What Happens Next

No dollar figure has been disclosed for the classified-network agreement, and no deal has been confirmed as of publication. The existing July 2025 Pentagon contracts paid up to $200 million each to Google, Anthropic, OpenAI, and xAI for prototype development. A classified-network extension would likely be a separate procurement.

The more consequential variable is the negotiated safeguards. If the Pentagon accepts Google's proposed restrictions on mass surveillance and autonomous weapons, the Anthropic blacklisting looks less like a principled line and more like a negotiating failure. That outcome would validate Anthropic's underlying positions while punishing the company for how it defended them - a distinction unlikely to comfort either side.

The OpenAI-Google competition for the Pentagon's AI budget is now moving into classified territory. The next milestone is whether Emil Michael signs a deal that includes the safeguard language Google is pushing, or whether the Pentagon holds the same line it drew against Anthropic.

Sources: The Information (original report, paywalled), Newsweek - Pentagon Weighs Google Gemini After Anthropic Fallout, Reuters/Bangor Daily News - Google Pentagon Negotiations, Winbuzzer - Google Deploys Gemini Agents at Pentagon, Winbuzzer - Google Workers and Military AI Activism, SF Standard - Diane Greene Op-Ed on Maven, CNBC - Pentagon Blacklists Anthropic, CNBC - Anthropic Loses Appeals Court Bid

The benchmarks in this space have grown substantially more demanding since 2023. The original MMMU was largely saturated by mid-2025. ChartQA and DocVQA, while still useful, are now cleared by most frontier models above 90%. The interesting signal in 2026 comes from MMMU-Pro (harder exam questions, 10-option answers), CharXiv-R (realistic scientific charts from arXiv papers), and OCRBench v2 (bilingual text recognition with 31 task scenarios). These are the benchmarks where gaps between models still tell you something.

The Benchmarks Explained

MMMU and MMMU-Pro

MMMU (Massive Multi-discipline Multimodal Understanding) contains 11,500 college-level problems across 30 subjects, each requiring understanding of images such as figures, charts, diagrams, and photographs. The original version is no longer useful for frontier model differentiation - top models score above 80% and gaps have compressed to noise.

MMMU-Pro matters more now. It expanded the answer option count from 4 to 10 (reducing random guessing value), removed questions answerable without reading the image, and added harder multi-step problems. Where top models scored 75% on the original, they score 65-82% on MMMU-Pro. The floor rose but the ceiling spread back out, which is exactly what you want from an evaluation benchmark.

MathVista

MathVista tests mathematical reasoning in visual contexts: bar charts with partial data, geometry diagrams with unlabeled angles, function plots that require inference, statistical tables where you need to calculate something that isn't written. It combines 28 existing visual math datasets with three new ones. The test isn't just vision - it requires the model to correctly read the visual element and then perform non-trivial math.

ChartQA and CharXiv

ChartQA asks questions about charts drawn in fairly clean presentation styles. It's scored at 90%+ by most frontier models today, which means it's best used to identify weak performers rather than rank the top tier.

CharXiv is meaningfully harder. The charts come from actual arXiv papers - the kind with dual y-axes, inset subplots, color-coded multi-series data, unconventional labeling, and statistical notation. CharXiv-R specifically assesses reasoning questions that require synthesizing information across the full chart, not just reading off a number. Human performance sits at 80.5%. Top models reached that range only in late 2025.

DocVQA

DocVQA uses scanned or photographed documents - forms, invoices, contracts, research papers with tables - and asks free-form questions about their content. The challenge is handling poor image quality, varied fonts, complex layouts, and text that wraps around form fields. It tests a capability with direct enterprise value: automated document processing.

OCRBench v2

OCRBench v2 is a sizable upgrade over the original, expanding to 10,000 human-verified question-answer pairs across 31 scenario types including handwritten text, mathematical notation, multilingual content, and degraded documents. English OCR leader as of Q1 2026 is Seed1.6-vision at 62.2%, which tells you this is still a hard benchmark with real headroom.

AI2D

AI2D tests scientific diagram comprehension - the kind of figures found in textbooks and biology papers showing cell structures, geological processes, and physical systems. Most frontier models now clear 89-94%, placing this benchmark in the same category as ChartQA for differentiation purposes. It's useful for identifying capable models at lower capability tiers.

BLINK

BLINK reformats 14 classic computer vision tasks - relative depth, visual correspondence, spatial reasoning, multi-view geometry - into 3,807 multiple-choice questions. These are perceptual tasks that humans solve almost instantly but models struggle with. The benchmark's name is literal: the tasks should take humans a blink. When the original BLINK launched, GPT-4V scored 51% while humans averaged 95%. The gap has since closed substantially, but BLINK remains a good test of genuine perceptual grounding rather than pattern matching from training data.

RealWorldQA

Released by xAI with Grok-1.5 Vision, RealWorldQA uses over 700 images taken from vehicles and real-world outdoor scenarios, each paired with a spatial reasoning question. It tests the kind of situational awareness relevant to robotics and autonomous systems, not academic benchmarks. The question set is small enough that statistical noise is a concern, but it adds a practical dimension that most other benchmarks lack.

Rankings Table

Scores reflect the best publicly reported results as of April 2026. Where a model has multiple evaluation configurations (e.g., with vs. without extended thinking), the higher result is listed. Scores marked with a dash were not publicly reported at time of writing.

A second table covering the benchmarks designed to test open-source models specifically:

Chart understanding is one of the harder visual reasoning tasks - models must identify axes, read legends, and reason about relationships that aren't explicitly stated. Source: pexels.com

Key Takeaways

Gemini 3.1 Pro Leads on MMMU-Pro, GPT-5.4 Leads on Documents

Gemini 3.1 Pro scores 82% on MMMU-Pro, the highest confirmed score on that benchmark from any currently available model. Its 92% on DocVQA and 90% on ChartQA are competitive but not dominant - GPT-5.4 scores 95% and 92.5% respectively on those two. The split makes sense architecturally. Google trained Gemini as a natively multimodal model from the beginning, which advantages it on broad academic visual reasoning. GPT-5.4 applies a stronger language model to extracted document representations, which advantages it on structured text extraction from forms and invoices.

For practitioners: Gemini 3.1 Pro is the better pick when the visual content is complex or diagram-heavy. GPT-5.4 is the better pick for production document processing pipelines where DocVQA-style extraction is the core task. You can read our Gemini 3.1 Pro review and GPT-5.4 review for hands-on testing beyond benchmark scores.

Claude Opus 4.7 Wins Scientific Charts

Claude Opus 4.7 holds the top CharXiv-R score at 91.0%, 2.6 points above Muse Spark and well ahead of other frontier models. This result is directly tied to the resolution upgrade introduced with Opus 4.7 - maximum image input increased from 1,568px (1.15MP) to 2,576px (3.75MP), roughly 3.3x more pixels. Scientific charts from arXiv papers often pack dense information into small visual areas, and the resolution upgrade makes a concrete difference on exactly those cases.

The result also stands apart because CharXiv-R is scored independently from provider benchmarking pipelines. It uses charts from real papers rather than constructed datasets, making contamination less likely. For research and data analysis workflows where scientific figures are central, Opus 4.7's lead on this benchmark translates to practical value.

Qwen Dominates Open-Source Visual Reasoning

Alibaba's Qwen VL family has established clear leadership among open-source vision-language models. Qwen3 VL 235B tops the BLINK leaderboard at 70.7% and holds the best RealWorldQA scores. Qwen3-VL's larger model (72B) scores 85.8% on MathVista and 96.5% on DocVQA - the latter beating every closed-source frontier model except GPT-5.4. Qwen3.6 Plus leads AI2D at 94.4%.

This is a reversal from where things stood 18 months ago, when open-source multimodal models lagged frontier proprietary models by 15-20 points on most benchmarks. The gap on DocVQA is now essentially closed for Qwen3-VL and InternVL3. On MMMU-Pro, a meaningful gap remains - Qwen3-VL scores 69.3% against the 81-82% from Gemini 3.1 Pro and GPT-5.4. Broad multi-discipline academic reasoning at the graduate level still favors proprietary models. Specialized document and chart tasks don't.

See our Qwen 3 review for context on how the broader Qwen model family performs outside visual benchmarks.

BLINK Shows What Frontier Models Still Get Wrong

BLINK remains the most humbling benchmark for visual reasoning. Qwen3 VL 235B leads at 70.7%, while a human would score around 95% on the same set of tasks. The gap isn't about knowledge or reasoning sophistication - it's about perceptual grounding. Tasks that require comparing relative depths in a photograph, tracing visual correspondence between two views of an object, or identifying which 3D object a 2D shadow belongs to still trip up current models at a rate that should give pause to anyone building systems where physical-world perception is safety-critical.

Worth noting: a 2026 paper showed BLINK scores are sensitive to design choices like visual marker size and style, which can reorder model rankings. The absolute scores should be read with some skepticism.

MathVista: Qwen-VL Beats the Frontier

The MathVista result is striking enough to call out directly. Qwen3-VL 72B scores 85.8% on MathVista, higher than any frontier proprietary model in this comparison. Gemini 3 Pro (82.3%) and GPT-5.4 (78.4%) trail it. Mathematical visual reasoning - reading geometry figures, interpreting function graphs, extracting data from statistical tables - is a domain where the Qwen team has invested heavily. InternVL3-78B (79.0%) also beats GPT-5.4 on this specific benchmark.

For applications involving quantitative analysis from images - financial charts, scientific figures, engineering diagrams - the assumption that proprietary frontier models are always the best choice doesn't hold. The open-source alternatives are worth assessing on your actual task distribution.

Increasing camera input resolution was a key architectural decision for Opus 4.7 - from 1.15MP to 3.75MP - directly improving performance on dense visual content. Source: pexels.com

Practical Guidance

Best for Enterprise Document Processing

GPT-5.4 is the clear choice if your pipeline mostly processes invoices, contracts, forms, and receipts. Its 95% on DocVQA and 92.5% on ChartQA combined with strong language model capability for following complex extraction instructions makes it the most reliable option for production document workflows. The pricing isn't cheap, but document processing use cases usually justify it on value recovered per document.

Qwen3-VL 72B is the open-source alternative worth evaluating seriously. At 96.5% on DocVQA it matches or beats every proprietary model, and it can be self-hosted on infrastructure you control - which matters for document workflows involving sensitive data.

Best for Scientific Research Assistance

Claude Opus 4.7 takes the top spot for scientific figure analysis. If your application needs to extract insights from arXiv-style charts, interpret experimental results, or process multi-panel research figures, the CharXiv-R result at 91.0% is the most relevant signal available. The extended resolution also helps with poster presentations and conference slides where content density is high.

Best for Mathematical and Technical Reasoning from Images

Qwen3-VL 72B scores highest on MathVista (85.8%) among all models with confirmed scores. For geometry problems, quantitative chart analysis, or any task requiring mathematical inference from a visual input, this is the model to test first. InternVL3-78B (79.0%) is a strong second and has a more mature deployment ecosystem.

Budget Option

Llama 4 Maverick from Meta scores 73.7% on MathVista, 94.4% on DocVQA, and 90% on ChartQA - truly competitive with models from a year ago that commanded much higher pricing. It's open-weight and deployable via standard inference infrastructure. Our Llama 4 Maverick review covers performance in more depth.

Methodology Caveats

Two issues specific to visual reasoning benchmarks are worth acknowledging. First, several of these benchmarks - especially DocVQA and ChartQA - are now in a range where small differences in evaluation configuration (prompt format, few-shot examples, extraction parsing) can swing results by 1-3 points. Numbers from different evaluators shouldn't be compared directly. The scores in this table are sourced from provider-reported results where available and independent evaluators where not, which introduces some inconsistency.

Second, MMMU contamination is a genuine concern. The original MMMU questions are widely available online, and models trained after 2024 on broad web datasets will have seen some fraction of them. MMMU-Pro and CharXiv use harder-to-replicate content, but no benchmark is immune. This is why the field is developing new evaluations like OCRBench v2, which used human-verified pairs not published before the benchmark's release.

For more context on how to read AI benchmark data critically, see our guide to understanding AI benchmarks.

FAQ

Which model is best for reading scanned documents?

GPT-5.4 leads DocVQA at 95%, making it the top choice for scanned document extraction. Qwen3-VL 72B scores 96.5% and is the best open-source alternative for sensitive or self-hosted workflows.

Is MMMU still useful for comparing models in 2026?

MMMU-Pro is the more useful variant. The original MMMU is too saturated above 80% to differentiate frontier models. MMMU-Pro, with 10-option questions and harder visual reasoning requirements, still provides meaningful signal.

Why does Qwen-VL beat frontier models on some benchmarks?

Qwen3-VL has been optimized specifically for document and mathematical visual tasks with targeted training data. On MathVista (85.8%) and DocVQA (96.5%) it outperforms all closed-source models. On broader academic multi-discipline benchmarks like MMMU-Pro, proprietary frontier models maintain a lead.

How is CharXiv different from ChartQA?

ChartQA uses presentation-style charts and is now cleared at 90%+ by most models. CharXiv uses charts from actual arXiv scientific papers, which are significantly denser and less standardized. CharXiv-R specifically tests multi-step reasoning across full charts rather than single-fact lookup questions.

What does BLINK measure?

BLINK tests core visual perception - depth estimation, visual correspondence, multi-view reasoning, and similar tasks humans solve instantly. Models still score far below human performance (70% vs. 95%), making BLINK one of the few benchmarks where frontier models clearly fail at something humans find trivial.

How often should I re-check these rankings?

Visual reasoning is one of the faster-moving benchmark categories right now. Claude Opus 4.7's resolution upgrade and Qwen3-VL's document improvements both shipped in early 2026. Check this leaderboard quarterly - major capability jumps are still happening at multi-month intervals.

Sources:

• MMMU-Pro Leaderboard - llm-stats.com
• MMMU Leaderboard - llm-stats.com
• MathVista - mathvista.github.io
• CharXiv-R Leaderboard - llm-stats.com
• BLINK Leaderboard - llm-stats.com
• RealWorldQA Leaderboard - llm-stats.com
• AI2D Leaderboard - llm-stats.com
• OCRBench v2 Paper - arxiv.org
• InternVL3 Paper - arxiv.org
• Llama 4 Benchmarks - llama.com
• Gemini 3.1 Pro Benchmarks - automatio.ai
• GPT-5.4 Benchmarks - automatio.ai
• Claude Opus 4.7 Benchmarks - vellum.ai
• MMMU-Pro Benchmark - Artificial Analysis
• Qwen3-VL Technical Report - arxiv.org
• CharXiv Benchmark - charxiv.github.io
• BLINK Benchmark Paper - arxiv.org

This article covers the second category - browsers you'd actually put on your daily machine, where the AI agent is the product, not a plugin on top of it.

What Makes a Browser Agent Different

A browser agent isn't a chatbot attached to your browser. The distinction matters. A chatbot answers questions about a page. An agent navigates, clicks, fills forms, and completes multi-step workflows across multiple sites without you touching the keyboard again after the initial prompt.

The test I use: can it book a round-trip flight to Berlin, find me a hotel under $150/night nearby, and output a formatted itinerary - all from one prompt? That task requires authenticating to travel sites, comparing multiple results, and handling decision branches when options differ. Most "AI browsers" fail it or require constant hand-holding. A real browser agent doesn't.

The other axis is privacy. Cloud-based agents process your browsing session on someone else's server. That's a real tradeoff, and not all vendors are transparent about it.

Perplexity Comet

Comet launched on Windows and macOS in July 2025, Android in November 2025, and iOS in March 2026. It's Chromium-based, ships with a persistent AI sidebar, and at the Max tier it routes tasks through Claude Opus 4.6 for reasoning-heavy work.

The agent can autonomously research a topic across multiple sites, summarize findings, fill forms, manage email, and book travel. Comet Plus ($5/month, or included in Pro and Max plans) gives the agent access to trusted journalism and premium sources when doing research tasks.

Pricing:

The $200 Max plan is the only one that unlocks what Perplexity calls "Max Assistant" - the routing layer that picks the right model for each subtask. For most users, the $20 Pro plan handles the majority of research and form-filling work.

Platforms: macOS, Windows, Android, iOS.

Privacy concern: Comet has built up six significant security vulnerabilities since its July 2025 launch. The most recent - disclosed by Zenity Labs in March 2026 - showed that a malicious calendar invite could hijack the agent into reading local files and exfiltrating credentials. We covered the full vulnerability chain here. Perplexity patched it after a 120-day disclosure window, but the pattern of dismissing reports before quietly fixing them is worth noting. A federal court also issued a preliminary injunction in March 2026 blocking Comet's agent from accessing Amazon accounts without platform-level authorization.

The agentic capability is real and among the best in this category. The security track record is the asterisk.

Dia (by Atlassian / The Browser Company)

Dia launched in beta in June 2025, went broadly available on macOS in October 2025, and started Windows early access in March 2026. Atlassian picked up The Browser Company for $610 million in October 2025, which explains the integration push into Slack, Notion, Google Calendar, and Gmail that landed in early 2026.

The core interaction model is "chat with your tabs." You open a bunch of research tabs, ask Dia a question, and it synthesizes an answer from everything you have open. The cross-tab reasoning is genuinely useful for knowledge work - I ran it across six tabs of conflicting benchmark data and it held the context correctly.

The agentic capabilities are less aggressive than Comet. Dia handles research, writing, planning, and product comparison well. It doesn't book flights or run fully autonomous multi-step workflows the way Comet's Max tier does. That may be intentional positioning - Dia is macOS-first and targets knowledge workers, not power-user automation.

Pricing:

Platforms: macOS (generally available), Windows (early access as of March 2026).

Privacy: Dia's privacy model isn't publicly detailed in the same way as Brave's. Sessions are processed in the cloud. Given the Atlassian acquisition, enterprise data governance policies will likely tighten - but for now, treating Dia as a cloud-processed assistant is the correct assumption.

The Atlassian backing is a meaningful signal for long-term viability. It's also a reason to expect Jira and Confluence integrations within the year.

Opera Neon

Opera made an aggressive move in this space. It started with Browser Operator in March 2025 - the first major browser to ship a native agentic feature - then rebuilt it into a standalone AI-first product called Neon, which launched in September 2025 and started charging $19.90/month in December 2025.

Neon's "Neon Do" feature handles shopping, booking, information gathering, and form completion. The technical approach is notable: it runs client-side rather than processing screenshots in the cloud, using native browser APIs to take actions without sending a video feed of your session to Opera's servers. That's a genuine privacy differentiator versus Comet.

The model selection at the $19.90 tier includes Gemini 3 Pro and GPT-5.1. For $19.90 you're getting a capable agent with solid model access.

Pricing: $19.90/month. No free tier with agentic features.

Platforms: Desktop (Windows, macOS, Linux), mobile in development.

Best for: Users who want agentic browsing with stronger local-processing privacy guarantees than Comet and don't need the deep research tier.

Chrome with Gemini Auto Browse

Google shipped Gemini into Chrome's sidebar in January 2026, with Auto Browse - the agentic feature - reserved for AI Pro and Ultra subscribers. The integration runs on Gemini 3.1 and handles tasks like filling forms from PDFs, filtering apartment searches, scheduling appointments, and filing expense reports.

Auto Browse supports Google's Universal Commerce Protocol (UCP), a standard co-developed with Shopify, Etsy, Wayfair, and Target that lets the agent complete purchases across participating retailers without breaking mid-flow. That's the most concrete agentic commerce capability of any browser in this list.

Pricing:

Platforms: Windows, macOS, ChromeOS. US-only for Auto Browse as of April 2026.

The main limitation: 20 tasks/day at the $19.99 tier is constraining if you're using it for serious workflow automation. And "Auto Browse" is a feature inside Chrome, not a new browser - you're still running Chrome with a subscription-gated agentic panel. That's a different value proposition than a purpose-built agent browser like Comet.

AI browser agents work across multiple open tabs to synthesize research, fill forms, and complete multi-step tasks without manual navigation. Source: unsplash.com

Brave with AI Browsing

Brave Leo has been the privacy-first AI assistant story for two years. In 2026, Brave added AI Browsing - an autonomous mode available in Brave Nightly - that takes the privacy story into agentic territory.

The claim is verifiable. Brave doesn't log IP addresses, doesn't retain chat history, and doesn't train on your conversations. The new Trusted Execution Environment (TEE) deployment cryptographically verifies these claims rather than asking you to take them on faith. That's a level of privacy assurance that no other browser in this roundup matches.

The AI Browsing feature itself is still in Nightly (pre-release). It can research across multiple sites, compare products, fill shopping carts, and complete multi-step tasks. Leo supports Mixtral, Claude, and Llama models depending on tier.

Pricing:

Platforms: Windows, macOS, Linux, Android, iOS.

The "Bring Your Own Model" option lets you connect a local Ollama instance or custom API endpoint. For users running local LLMs, Brave is the only mainstream browser that supports fully local inference with no cloud hop at all.

Fellou

Fellou launched in May 2025 as an independent European startup (ex-DeepMind and Mozilla engineers) and went Commercial Edition in September 2025. It runs the "shadow workspace" model: tasks execute in a background virtual window, visible to you but not interrupting your active browsing session.

Before executing anything, Fellou generates a step-by-step action plan for review. You can edit or cancel before it starts. That's the right design for users who want agent assistance without a black-box running loose in their browser.

The RAG layer is interesting - Fellou can pull from 43+ live data sources including LinkedIn, Reddit, academic databases, and membership sites. The benchmark claim from the company (5.2x faster on complex multi-domain tasks versus Claude and Perplexity) comes from their own internal testing and isn't independently verified.

Pricing:

Platforms: macOS (generally available), Windows in development.

The "Sparks" credit model is worth understanding before committing. Each task consumes a variable number of Sparks based on complexity. Fellou estimates the cost before running, which at least gives you visibility, but the variable pricing can add up for power users. The Plus tier is competitive with Comet Pro for most use cases.

Microsoft Edge Copilot Mode

Edge's Copilot Mode - generally available as of late 2025 - is the enterprise play in this space. The consumer-grade agent story is real: Agent Mode handles multi-step browser workflows using Microsoft's CUA (Computer-Using Agent) models, multi-tab reasoning analyzes up to 30 open tabs simultaneously, and there's a Daily Briefing that pulls from Microsoft Graph and your browsing history.

Where Edge distinguishes itself is governance. DLP policies you've already configured in Microsoft 365 apply automatically to agentic actions. Agent Mode won't touch passwords or payment fields without explicit permission. IT admins control which sites the agent can access.

Pricing: Copilot Mode comes with Microsoft 365 subscriptions ($6-$22/user/month depending on tier). Microsoft 365 Copilot ($30/user/month) unlocks the full enterprise agentic layer including Agent Mode.

Platforms: Windows, macOS, iOS, Android.

Best for: Organizations already on Microsoft 365 that need AI browsing within existing governance boundaries. Consumer users get the features for free with Edge, but the governance controls that make it genuinely enterprise-ready require the paid tier.

Privacy models vary notably across AI browsers - from cloud-processed sessions to TEE-verified local inference and full self-hosted options. Source: unsplash.com

Island Enterprise Browser

Island isn't competing on consumer features. It's a security-first enterprise browser built for organizations that need to know exactly what their agents are doing at all times.

The March 2026 enterprise AI platform launch added three layers: an AI Browser that embeds governed AI chat (multiple frontier models, enterprise context via RAG, DLP enforcement), AI Automation for building and running on-demand agents with defined permissions and audit trails, and full admin controls over what sites agents can access and what actions require human approval.

The "hardened Chromium environment" framing is the key point: agents run inside a browser designed to reduce prompt injection exposure. After the year Comet had with its injection vulnerabilities, that's not just marketing language.

Pricing: Enterprise pricing, not publicly listed. Expect per-seat contracts.

Platforms: Windows, macOS.

Best for: Financial services, healthcare, legal, and government organizations that can't accept the security posture of consumer agentic browsers.

Open-Source Options: Surfer and Browserbase

Two open-source options deserve mention for developers who want to build on top of browser agent technology rather than just use it.

Surfer-H / Surfer 2 (H Company): An open-weight web agent that hit 92.2% on WebVoyager and 97.1% on agentic benchmarks with Surfer 2 - the state-of-the-art result as of February 2026. The Holo1 VLM models are open-sourced. This is for building agents, not for daily browsing, but it's the most capable open-weight foundation available. The CLI is available on GitHub.

Browserbase + Stagehand: The developer infrastructure layer. Stagehand wraps Playwright with LLM primitives; Browserbase provides the managed headless browser fleet. Covered in depth in our browser automation roundup. The free tier (1 concurrent browser, 1 hour) lets you prototype before spending anything.

Full Comparison Table

Which Browser Agent to Pick

You want the most capable agent and price isn't the constraint: Perplexity Comet Max at $200/month, with Claude Opus 4.6 for complex tasks. Keep an eye on the security advisories.

You want a capable agent at $20/month: Comet Pro, Dia Pro, Opera Neon, and Fellou Plus are all competitive. Comet has deeper agentic task completion; Dia has better knowledge-work integrations (Slack, Notion, Google Calendar); Opera Neon has better client-side privacy.

Privacy is the primary requirement: Brave with AI Browsing. No IP logging, TEE-verified privacy guarantees, and a local inference option via Ollama. The AI Browsing feature is still in Nightly as of April 2026, so early-adopter roughness applies.

You're already on Google AI Pro: Chrome's Auto Browse at 20 tasks/day is included and works well for Google-ecosystem workflows and UCP-supported commerce sites. Don't pay again for a separate browser.

You're in an enterprise environment with compliance requirements: Island Browser for hardened, audited agents. Edge Copilot Mode if you're on Microsoft 365 and governance integration matters more than advanced agent capability.

You want to build your own agent, not just use one: Surfer-H and Browserbase/Stagehand are the places to start. See our browser automation roundup for the full developer-side picture.

The Security Reality

None of the vendors discuss this prominently in their marketing, but it's the practical issue that matters most for any agentic browser: every piece of web content the agent processes is a potential prompt injection surface.

Comet's six vulnerabilities aren't a Comet problem - they're a demonstration of the category problem. OpenAI has acknowledged that prompt injection in its own Atlas browser agent is "unlikely to ever be completely eliminated." The architectural issue is that LLMs can't reliably distinguish between trusted user instructions and untrusted page content when they arrive in the same token stream.

Practical mitigations: keep password managers locked when not in active use, disable agent access to sensitive domains in browser settings, and treat your agent's access scope as your attack surface. A browser agent with access to your email and calendar has a larger blast radius than a sandboxed chatbot. Scope it accordingly.

Sources

• Perplexity Comet pricing guide - eesel AI
• Perplexity pricing plans - finout.io
• Comet arrives on iPhone - 9to5Mac
• Dia browser official site
• The Browser Company launches Dia in beta - TechCrunch
• Atlassian acquires The Browser Company - Atlassian blog
• Opera Neon ships - Opera blog
• Opera Neon $19.90 subscription - TechCrunch
• Opera Browser Operator announcement - Opera Newsroom
• Chrome Gemini agentic features - TechCrunch
• Google AI Pro and Ultra pricing - gemini.google
• Brave Leo official page
• Brave AI Browsing announcement
• Brave TEE privacy announcement
• Fellou official pricing
• Fellou browser agentic features and security - Seraphic Security
• Edge Copilot Mode enterprise-ready - SiliconANGLE
• Island enterprise AI platform - island.io
• Surfer 2 by H Company
• Dia browser - Wikipedia

Two things make RAG benchmarking unusual compared to language modeling leaderboards. First, you're assessing two different components - the embedding or retrieval model, and the generation model that reads what was retrieved. A great retriever paired with a weak generator will still hallucinate. Second, scores mean different things across benchmarks. A NDCG@10 on BEIR isn't the same animal as an Exact Match score on NQ, and comparing them directly misleads more than it informs.

What Each Benchmark Measures

RAG evaluation splits cleanly into retrieval benchmarks and end-to-end RAG benchmarks. They test different failure modes.

Retrieval benchmarks score how well a model finds the right documents given a query. The dominant metric is NDCG@10 (normalized discounted cumulative gain at rank 10), which rewards returning relevant documents at the top of the ranking and penalizes burying them lower down. Most retrieval benchmarks are zero-shot by design - models shouldn't be trained on the evaluation data.

End-to-end RAG benchmarks score the full pipeline: does the produced answer actually match the gold answer, and does it stick to what was retrieved? The key metrics here are Exact Match (EM), F1, and faithfulness scores. A model that scores well on retrieval but has high hallucination rates on RAGTruth has a generation problem, not a retrieval problem.

BEIR - The Generalization Test

BEIR (Benchmarking Information Retrieval) covers 18 diverse datasets including MS MARCO, Natural Questions, HotpotQA, and domain-specific corpora spanning biomedical, legal, scientific, and technical content. The benchmark is intentionally heterogeneous - a model that scores well here generalizes to out-of-distribution queries, which is what you actually want in production. BM25 scores around 42 NDCG@10 as the sparse baseline; modern dense models consistently clear 60.

MTEB Retrieval Track

The Massive Text Embedding Benchmark includes a 15-dataset retrieval sub-track that overlaps with BEIR. MTEB is more structured about evaluation conditions and covers additional task categories (STS, classification, clustering) that let you see how retrieval performance relates to other embedding capabilities. It's the most widely used comparative leaderboard for embedding models.

MIRACL - Multilingual Retrieval

MIRACL covers 18 languages with native-speaker annotation for each, testing monolingual retrieval where both queries and documents are in the same language. The dataset spans languages from Arabic and Bengali to Swahili and Telugu. A model that scores well on English MTEB can still fall apart here - many commercial APIs drop 20-30% in retrieval quality when switching from English to lower-resource languages. The evaluation metric is nDCG@10 and Recall@100.

In May 2025, MIRACL-VISION extended the benchmark to visual document retrieval across the same 18 languages, exposing a different problem: state-of-the-art VLM embedding models lag text-based retrieval models by up to 59.7% in multilingual visual retrieval accuracy.

MS MARCO - Large-Scale Passage Ranking

MS MARCO is a large-scale dataset from real Bing queries and human-annotated passage relevance. The passage ranking task uses MRR@10 as the primary metric. Unlike BEIR, which tests generalization, MS MARCO is the closest thing to an in-distribution retrieval test for English web queries. Models optimized for MS MARCO tend to excel here but don't always transfer to domain-specific corpora.

KILT - Knowledge-Intensive Tasks

KILT (Knowledge Intensive Language Tasks, from Meta AI) tests retrieval over a Wikipedia snapshot for 11 downstream tasks: fact-checking, open-domain QA, slot filling, entity linking, and dialogue. Systems need to retrieve supporting evidence and create answers that depend on that evidence. CoRAG-8B, released in January 2025, currently holds state-of-the-art scores across most KILT tasks, outperforming systems built on much larger LLMs.

HotpotQA and Natural Questions

Both are multi-hop and open-domain QA benchmarks respectively. HotpotQA requires retrieving evidence from multiple documents and connecting it to answer questions that can't be answered from any single passage. NQ uses real Google search queries against Wikipedia. EM (Exact Match) and F1 are the standard metrics, and scores here reflect the combined quality of both retrieval and generation.

RAGTruth - Hallucination in Generation

RAGTruth is a corpus of nearly 18,000 RAG responses from multiple LLMs, annotated at the word level for hallucination type and severity. It's the most detailed benchmark specifically for the generation side of RAG - distinguishing between cases where the model invented unsupported facts, contradicted the retrieved context, or produced partially correct but misleading answers. The benchmark was introduced at ACL 2024 and has become the standard reference for comparing how different LLMs handle retrieved context faithfully.

Retrieval benchmarks like BEIR test generalization across domains - medical, legal, code, and web queries each expose different model weaknesses. Source: unsplash.com

Retrieval Model Rankings

MTEB Retrieval + BEIR Leaders

*GTE-ModernBERT-base score is on the full MTEB English average (not retrieval-only), using 149M parameters. It's included as the strongest efficient open-weight option under 150M parameters.

A few caveats on this table. Voyage AI publishes scores on their own RTEB benchmark (29 datasets across 8 domains) rather than MTEB, which makes direct comparison to MTEB scores approximate. On their own evaluation, voyage-4-large beats OpenAI text-embedding-3-large by 14% and Cohere Embed v4 by 8.2% in NDCG@10. MTEB is cross-vendor but self-reported; RTEB is vendor-controlled but more thorough. Neither is fully neutral.

MIRACL Multilingual Rankings

For multilingual retrieval, the picture looks different:

For production multilingual work, the best free option is NVIDIA Llama-Embed-Nemotron-8B. If you need a commercial API with broad language support, Cohere Embed v4 covers 100+ languages with cross-lingual retrieval (query in English, retrieve in French, etc.) and is the only major embedding API that also handles images natively.

End-to-End RAG: Generation Quality

The generation side of RAG gets less systematic attention than retrieval, but RAGTruth and KILT give us data points.

On the RAGTruth hallucination benchmark, LLM performance varies enough to matter in production. GPT-4o and Claude 3.5 Sonnet produce the fewest hallucinations against retrieved context. Fine-tuned smaller models trained on RAGTruth data can reach competitive faithfulness at lower cost - the original paper showed that fine-tuning Llama-2-13B on RAGTruth training data matched prompt-based approaches with GPT-4. That gap has closed further with stronger base models in 2025-2026.

On KILT, CoRAG-8B (Chain-of-Retrieval Augmented Generation, January 2025) posts state-of-the-art performance on multi-hop QA tasks including HotpotQA subsets, beating systems built on LLMs three to five times larger. CoRAG uses iterative retrieval - the model retrieves, reads, decides if it needs more evidence, and retrieves again. Single-pass RAG struggles with multi-hop questions by design.

End-to-end RAG evaluation requires tracking retrieval quality and generation faithfulness separately - strong retrieval with weak generation still produces hallucinated answers. Source: unsplash.com

Key Takeaways

Gemini Embedding 2 Sets a New Bar

Google released Gemini Embedding 2 into public preview in March 2026. It's built natively on the Gemini architecture and handles text, images, video, audio, and PDFs in a single 3,072-dimensional vector space. On MTEB English, it scores 68.32 overall and 67.71 on the retrieval sub-track, with Matryoshka support down to 768 dimensions.

The pricing is $0.20 per million tokens standard or $0.10 with batch processing. Indexing 1 million 500-token documents costs roughly $50 at batch rates. That's not cheap compared to open-weight models, but it's clearly less than the engineering cost of self-hosting a competitive open model. Early production reports mention 20% recall improvement and 70% latency reduction from users switching from older models - though those come from Google's own case studies, so treat them as directional.

The main limitation is vendor lock-in. Re-indexing a document corpus whenever you switch embedding models is a real operational burden. If you build on Gemini Embedding 2, plan for it long-term or architect an abstraction layer.

The MoE Architecture Arrives for Embeddings

Voyage-4-large, launched in January 2026, is the first production embedding model using a Mixture-of-Experts architecture. The MoE design routes different inputs to specialized sub-networks, delivering "serving costs 40% lower than comparable dense models" according to Voyage AI, while maintaining state-of-the-art accuracy. The model family includes voyage-4, voyage-4-lite, and voyage-4-nano (open-weight, Apache 2.0 on Hugging Face).

Voyage AI uses their own RTEB benchmark for evaluation rather than MTEB, which creates comparison friction. On RTEB, voyage-4-large outperforms Gemini Embedding 001 by 3.87% and OpenAI text-embedding-3-large by 14.05% in NDCG@10 across 29 datasets. Independent third-party scoring on MTEB puts the comparison closer, but Voyage models consistently sit in the top tier.

Open-Weight Models Are Genuinely Competitive

NVIDIA's NV-Embed-v2 posts 62.65 on the MTEB retrieval track - trailing Gemini Embedding 2's 67.71, but scoring higher than OpenAI text-embedding-3-large on retrieval specifically. NV-Embed-v2 reached 72.31 on the full MTEB average (56 tasks) using a two-stage contrastive instruction-tuning method. It's fully open-weight and runs on a single A100. For teams with GPU infrastructure already in place, self-hosting NV-Embed-v2 eliminates per-token costs completely.

Qwen3-Embedding-8B, Alibaba's multilingual model, scores 70.58 on the MMTEB multilingual leaderboard and supports flexible dimensions from 32 to 4096. Its code retrieval score on MTEB Code is 80.68, the highest reported for any model on that sub-benchmark. If your RAG system needs to retrieve code with natural language documentation, Qwen3-Embedding-8B is the current choice.

Our embedding model leaderboard from March 2026 covers the full MTEB picture in more depth, including dimension tradeoffs and Matryoshka support across providers.

BM25 Is Still Alive, But in a Support Role

Sparse retrieval with BM25 scores around 42 NDCG@10 on BEIR - below every modern dense model. But that doesn't mean BM25 is obsolete. Hybrid search combining BM25 with dense retrieval routinely beats either method alone, particularly on exact-match queries like product codes, person names, and technical identifiers. Dense models miss these because the learned vector space doesn't encode character-level patterns reliably. Most production RAG systems should use hybrid retrieval unless they have clear evidence that pure dense retrieval meets their precision requirements.

Multi-Hop RAG Requires a Different Stack

Standard single-pass RAG - retrieve once, produce once - handles simple factoid questions reasonably well. It struggles with questions requiring evidence from multiple documents, temporal reasoning, or comparative analysis. On HotpotQA specifically, hybrid retrieval combining dense and lexical signals with maximum marginal relevance post-filtering achieves 20% absolute EM improvement over pure dense retrieval alone.

CoRAG-8B's iterative retrieve-read-decide-retrieve loop is the cleanest solution for multi-hop at inference time. It requires more compute per query (5-8 retrieval passes for hard questions), but the accuracy gains are significant. For production systems handling complex enterprise queries, iterative retrieval is worth the latency cost.

RAGPerf Brings System-Level Measurement

Most RAG benchmarks measure either retrieval quality or answer quality in isolation. RAGPerf, released on arXiv in March 2026, provides an end-to-end framework that simultaneously tracks context recall, query accuracy, factual consistency, latency, throughput, and GPU/memory use across a full pipeline. It supports multiple vector databases (LanceDB, Milvus, Qdrant, Chroma, Elasticsearch) and is open-sourced at platformxlab/RAGPerf. For teams building production systems, this is the first framework that lets you see retrieval accuracy tradeoffs against actual hardware costs in a single run.

Practical Guidance

The right choice depends on which part of the pipeline you're improving.

For best retrieval accuracy (API): Gemini Embedding 2 leads on MTEB retrieval at 67.71 NDCG@10. If you don't want Google lock-in, voyage-4-large is the alternative, with strong RTEB numbers and a lower-cost MoE architecture.

For best retrieval accuracy (self-hosted): NV-Embed-v2 scores 62.65 on MTEB retrieval and is fully open-weight. Qwen3-Embedding-8B is preferred for multilingual or code-heavy workloads.

For multilingual RAG: NVIDIA Llama-Embed-Nemotron-8B leads the MMTEB multilingual benchmark and is open-weight. For a commercial API, Cohere Embed v4 covers 100+ languages with cross-lingual support and multimodal capability.

For budget-constrained pipelines: OpenAI text-embedding-3-small at $0.02/million tokens offers a reasonable retrieval score for the price. Voyage-4-nano is Apache 2.0 licensed, freely deployable, and designed for high-throughput low-cost retrieval.

For multi-hop or complex queries: Single-pass RAG won't cut it. Build iterative retrieval into your pipeline. CoRAG-8B's approach is the current benchmark reference.

For minimizing hallucination: Choose your generation model carefully. On RAGTruth evaluations, GPT-4o and Claude 3.5 Sonnet produce the fewest hallucinations when given retrieved context. Fine-tuning a smaller model on RAGTruth-style data is a viable lower-cost path. See our guide to best RAG tools in 2026 for framework-level options including LangChain, LlamaIndex, and Haystack.

What benchmark to trust: For retrieval comparison, use MTEB as the cross-vendor neutral standard. Be skeptical of vendor-internal benchmarks (including Voyage's RTEB) when they're the only numbers cited. For end-to-end faithfulness, RAGTruth is the most granular public reference. For multilingual, MIRACL and MMTEB are the appropriate benchmarks - English MTEB scores don't predict multilingual performance reliably.

FAQ

Which embedding model is best for RAG in 2026?

Gemini Embedding 2 leads on MTEB retrieval with 67.71 NDCG@10. For self-hosted options, NV-Embed-v2 and Qwen3-Embedding-8B are both competitive. For multilingual, NVIDIA Llama-Embed-Nemotron-8B leads.

What is the BEIR benchmark?

BEIR tests retrieval models across 18 diverse datasets covering web, biomedical, legal, and scientific content. It measures zero-shot generalization - models shouldn't be fine-tuned on BEIR data before evaluation. BM25 baseline is roughly 42 NDCG@10.

How does MTEB differ from BEIR?

MTEB includes the BEIR retrieval datasets plus additional task categories (STS, classification, clustering). MTEB is broader and used to compare general embedding quality. BEIR focuses exclusively on retrieval generalization.

What is RAGTruth and why does it matter?

RAGTruth is a 18,000-response dataset annotated for hallucination type and severity in RAG outputs. It's the standard benchmark for comparing how faithfully different LLMs stick to retrieved context when creating answers.

Is BM25 still relevant for RAG?

Yes. Hybrid retrieval combining BM25 with dense embeddings beats pure dense retrieval on exact-match queries. Most production RAG systems use hybrid search. Pure BM25 scores around 42 NDCG@10 on BEIR, well below modern dense models, but it handles keyword precision that dense models miss.

How often do RAG benchmark rankings change?

Retrieval rankings shift 2-4 times per year as major providers release new models. The current leaders (Gemini Embedding 2, Voyage-4-large) were both released or updated in early 2026. Check this page and the MTEB leaderboard on Hugging Face for the latest.

Sources:

• MTEB Leaderboard - Hugging Face
• BEIR Benchmark - GitHub
• Voyage 4 Model Family Announcement
• Voyage-4-large MoE Architecture Blog
• Voyage 3.5 Announcement
• NV-Embed-v2 on Hugging Face
• NVIDIA NV-Embed MTEB Announcement
• MIRACL Benchmark Project
• MMTEB: Massive Multilingual Text Embedding Benchmark - arXiv
• RAGTruth - ACL 2024
• RAGPerf: End-to-End RAG Benchmarking Framework - arXiv
• MS MARCO Passage Ranking Leaderboard
• KILT Benchmark - GitHub
• HotpotQA Dataset

These tools are not meeting assistants. If you need AI to join a Zoom and take notes, that's a separate category - we cover it in Best AI Meeting Assistants 2026. The tools here are dedicated transcription engines for podcasters, legal firms, video producers, researchers, and developers building voice pipelines. The typical use case is a file upload or a live audio stream that needs to come back as clean, speaker-attributed text with timestamps.

The comparison spans API-first services (Deepgram, AssemblyAI, OpenAI, Google Cloud, Rev AI), professional SaaS tools (Sonix, Descript, HappyScribe, Trint), and the open-source Whisper option for teams that want full control.

Why This Category Is Different from Meeting AI

Meeting assistants layer on top of a transcription engine and add scheduling, action items, and CRM sync. Transcription tools are lower in the stack. A developer building a podcast hosting platform doesn't need summaries - they need clean SRT files and a reliable API. A court reporting firm needs verbatim accuracy above 98% and a clear audit trail.

The tradeoffs matter differently here. Latency for batch transcription doesn't matter much; accuracy and format support do. For real-time captioning or live subtitles, streaming latency is everything. Speaker diarization quality is critical for any multi-speaker audio - interview recordings, depositions, panel discussions.

For teams also needing text-to-speech or voice synthesis, our Best AI Voice Generators 2026 covers the output side of the pipeline.

Accuracy Benchmarks: What the Numbers Actually Mean

Word Error Rate (WER) is the standard metric, but vendor-published benchmarks are almost always optimistic. Most use clean audio at moderate speeds with standard vocabulary. Real-world audio - phone calls, noisy environments, heavy accents, technical jargon - normally delivers WER 2x to 4x worse than the headline number.

AssemblyAI publishes one of the more credible independent benchmark reports, running evaluations across 250+ hours of audio from 26 datasets. Their February 2026 results cover English and multilingual performance.

English WER across major providers (AssemblyAI benchmark, February 2026):

The Noisy Audio column is where things fall apart. Amazon's 24.73% WER on noisy audio is nearly unusable for real-world podcast or field recording work. Deepgram's Nova-3 separate benchmark reports a 5.26% batch WER on their own test set - the discrepancy with AssemblyAI's 8.38% average illustrates exactly why cross-vendor benchmarks are valuable.

For multilingual audio, ElevenLabs edges ahead with a 8.75% average against AssemblyAI's 9.78%, though both are well ahead of Deepgram (12.82%) and Microsoft (12.62%).

These benchmarks were published by AssemblyAI, so treat them with appropriate skepticism - they have an obvious interest in favorable numbers. That said, their methodology is documented and they include datasets where they don't win.

Dedicated transcription APIs are the right tool for podcast producers, interviewers, and media teams - not meeting bots. Source: pexels.com

API-First Transcription Engines

Deepgram Nova-3

Deepgram's Nova-3 is the most developer-friendly option in this space. The API design is clean, documentation is thorough, and the streaming latency is genuinely low - Deepgram claims 300ms time-to-first-token for real-time streaming. For anyone building a voice agent, live captioning service, or anything requiring sub-second response, Nova-3 is the practical default.

Pricing (Pay As You Go):

• Pre-recorded English: $0.0043/min ($0.26/hr)
• Streaming English: $0.0077/min ($0.46/hr)
• Pre-recorded multilingual: $0.0052/min ($0.31/hr)
• Speaker diarization add-on: +$0.0020/min ($0.12/hr)
• Smart formatting: included

The Growth plan (annual commitment, $4K+ minimum) cuts pre-recorded English to $0.0036/min. Nova-3 supports 36+ languages. Output formats include JSON with word-level timestamps, SRT, and VTT. DOCX export requires post-processing.

Where Nova-3 loses ground is on noisy or heavily accented audio, where AssemblyAI's numbers pull ahead. Deepgram also publishes their own benchmarks claiming a 30% WER advantage over competitors - the conflict between their numbers and AssemblyAI's data is unresolved. Test on your own audio.

AssemblyAI Universal-3 Pro

AssemblyAI's current flagship model adds structured formatting, keyterm prompting, and medical mode on top of base transcription. The Universal-2 model (still available) remains solid for most use cases at a lower price.

Pricing:

• Universal-3 Pro (pre-recorded): $0.21/hr base
• Universal-3 Pro Streaming: $0.45/hr base
• Universal-2 (pre-recorded): $0.15/hr base
• Speaker diarization: +$0.02/hr (pre-recorded), +$0.12/hr (streaming)
• Medical mode: +$0.15/hr on top of base

The feature set is the strongest here. Speaker identification, sentiment analysis, entity detection, PII redaction, and summarization are all available as add-ons. Entities and PII redaction cost extra but the API is clean and well-documented. AssemblyAI's Universal-3 Pro Medical model claims a 4.97% Missed Entity Rate on clinical terminology - medical practices and legal firms with domain vocabulary should test this tier specifically.

The free tier includes $50 in credits with no card required.

OpenAI (gpt-4o-transcribe / Whisper-1)

OpenAI now has two transcription tiers. The original Whisper-1 pricing has been replaced with the gpt-4o family as the recommended path.

Pricing:

• gpt-4o-transcribe: $0.006/min ($0.36/hr)
• gpt-4o-mini-transcribe: $0.003/min ($0.18/hr)

The gpt-4o-transcribe model reaches 2.46% WER on FLEURS benchmarks - one of the lowest published numbers. Real-world performance is strong, especially on clear speech and standard English. It struggles more on heavy accents and noisy environments than the benchmark suggests.

The main limitation is the API design. There's no native streaming model comparable to Deepgram. You upload a file and get back text. For batch workflows - transcribing podcast episodes, video libraries, interview archives - it works well. For real-time use cases, the architecture requires workarounds.

File size limit is 25MB per request. Supported formats include MP3, MP4, WAV, WEBM. Output formats: JSON, SRT, VTT, or plain text. Speaker diarization is available via the gpt-4o-transcribe endpoint but was in beta as of early 2026.

Rev AI

Rev operates both an AI transcription API and a human transcription service, which is genuinely useful for high-stakes work.

Pricing:

• Reverb ASR (English): $0.20/hr
• Reverb Turbo: $0.10/hr
• Reverb Foreign Language: $0.30/hr (55+ languages)
• Whisper Fusion / Whisper Large: $0.005/min ($0.30/hr)
• Human transcription: $1.99/min ($119.40/hr)
• Free tier: 5 hours equivalent in free credits

The Reverb model is Rev's proprietary ASR. Whisper Fusion and Whisper Large are what they say on the tin - Whisper-based models hosted by Rev. The human transcription option at $1.99/min is expensive but delivers 99%+ accuracy with a human review - appropriate for legal depositions, medical records, or any context where errors have real consequences.

The API is well-documented. Speaker diarization and confidence scores are included. Rev supports SRT, VTT, JSON, and text output.

Google Cloud Speech-to-Text

Google's STT API offers two versions - V1 and V2. V2 adds Chirp, their multilingual model, and multi-region data residency.

Pricing:

• Standard models: $0.024/min ($1.44/hr)
• Enhanced models: $0.036/min ($2.16/hr)
• Data logging opt-out: +40% to listed rates
• First 60 minutes per month: free

At $1.44/hr for standard, Google is more expensive than Deepgram, AssemblyAI, or OpenAI for equivalent accuracy. The real cost is higher once you factor in the full GCP stack - Cloud Storage, Cloud Functions, and egress fees can double the effective per-hour rate. Google's WER benchmark numbers (Microsoft-level, around 8-9% average) don't justify the price premium over dedicated speech APIs.

Google STT makes the most sense if you're already deep in GCP and the integration cost outweighs the per-minute savings from switching providers.

Speaker diarization accuracy matters most for multi-voice recordings - depositions, interviews, panels. Source: pexels.com

Professional SaaS Transcription Tools

These tools trade API flexibility for a polished UI and editorial workflow. They're aimed at podcasters, journalists, researchers, and content teams rather than developers.

Sonix

Sonix is the strongest editor-focused option. The transcript editor is browser-based, clean, and genuinely useful - you can click any word to jump to that moment in the audio, assign speaker names, and export with word-level timestamps intact.

Pricing:

• Standard (pay-per-use): $10/hr
• Premium: $5/hr + $22/user/month subscription
• Free trial: 30 minutes

Sonix supports 53+ languages and exports SRT, VTT, TXT, and DOCX. Translation is available as an add-on. Accuracy is claimed at up to 99%, which likely refers to best-case English conditions. For podcasters and researchers who want a clean editing workflow without developer overhead, Sonix is hard to beat at this price.

Descript

Descript takes a different approach. Rather than positioning itself as a transcription service, it's a video and audio editor where the transcript is the primary editing interface. Delete words from the transcript and the corresponding audio gets cut. It's a genuinely novel workflow for creators.

Pricing (post-September 2025 restructure):

• Free: 60 media minutes/month + 100 AI credits (one-time)
• Paid tiers start at $16/month (Hobbyist)
• Creator and Business tiers add more media minutes and AI credits

Descript uses media minutes (upload/recording) and AI credits (AI processing) as its two resource pools since the September 2025 pricing overhaul. If you're a solo creator doing under an hour of content per month, the Hobbyist tier at $16/month covers most workflows. For heavier users, calculate your media minute consumption before committing.

Speaker labeling, noise reduction, and studio sound processing are built in. Export formats include SRT and VTT for captions. It isn't designed for bulk API ingestion - it's a creative tool for video and podcast producers.

HappyScribe

HappyScribe is an European-focused transcription service with strong subtitle support. It's popular among documentary filmmakers and broadcasters who need subtitle workflows with transcription.

Pricing:

• Basic: €8.50/month (annual) - 120 AI minutes/month
• Pro: €19/month (annual) - 600 AI minutes/month
• Business: €59/month (annual) - 6,000 AI minutes/month
• Pay-per-use top-ups: €0.20/min
• Human proofreading: from €1.75/min

All plans include unlimited meeting recording (up to 45 minutes per session on Basic). HappyScribe exports SRT, VTT, TXT, and DOCX. Human proofreading is available as an add-on, useful for broadcast media where accuracy requirements are high. Language support covers 60+ languages with subtitle timing built into the workflow.

The UI is polished and aimed at non-technical users. There's no developer API comparable to Deepgram or AssemblyAI.

Trint

Trint targets newsrooms and media organizations. It's one of the older players in AI transcription - founded in 2014 - and has built out collaboration features that matter to editorial teams.

Pricing:

• Starter: $52-80/month per seat (7 files/month)
• Advanced: $60-100/month per seat (unlimited transcriptions)
• Enterprise: custom

Trint supports 31+ languages and has ISO 27001 certification, which matters for enterprise and media buyers. The AI assistant can create 400-word overviews of long transcripts. Real-time collaboration and API access are Enterprise-only features.

The per-seat subscription model is expensive compared to usage-based alternatives. For a team doing light transcription work, Trint's minimum viable plan at $52/month per user adds up fast. It earns its price in newsroom environments where the collaboration and security certifications matter.

The Whisper Open-Source Option

Whisper (originally released by OpenAI in 2022) remains the reference point for open-source transcription. The large-v3 model delivers competitive accuracy for many use cases. Self-hosting is the draw - your audio never leaves your infrastructure.

The reality of self-hosting Whisper at scale isn't simple. You need GPU infrastructure, audio chunking logic for long files (Whisper processes in 30-second segments), concurrent request handling, and ongoing model management. Processing 1 hour of audio on an AWS GPU instance costs roughly $0.56 using the large model - comparable to Deepgram for small volumes, but without the infrastructure overhead factored in.

For a solo developer or researcher transcribing occasional files, running whisper audio.mp3 --model large locally is perfectly practical. For production workloads above a few hundred hours per month, the managed APIs offer better economics once engineering time is priced in.

Whisper doesn't support native real-time streaming. It's a batch-only option out of the box.

Comparison Table

WER numbers from published benchmarks where available. FLEURS score for OpenAI reflects clean read speech - real-world performance is higher WER. "Not published" means no independent external benchmark found at time of writing.

Best Pick by Use Case

Developers building voice pipelines: Deepgram Nova-3. The streaming API, low latency, and clean documentation are the right fit. Start with Pay As You Go at $0.0043/min and move to Growth pricing if volume justifies it.

Teams needing maximum accuracy on challenging audio: AssemblyAI Universal-3 Pro. The edge in noisy audio and medical/legal vocabulary is measurable. The $0.21/hr base rate is reasonable given the accuracy differential.

Already on OpenAI stack: gpt-4o-transcribe at $0.006/min. No new vendor onboarding, strong accuracy on clean audio, SRT and VTT output included. Acceptable tradeoff for most batch workflows.

Podcast and video creators: Sonix for a dedicated transcription workflow, or Descript if you want to edit your content by editing the transcript. Both are consumer-grade tools that don't require API work.

High-stakes accuracy (legal, medical, regulatory): Rev AI's human transcription at $1.99/min, or AssemblyAI Universal-3 Pro Medical mode combined with human review. Machine accuracy above 99% on clean audio isn't guaranteed by any provider - a human review layer remains necessary for documents with legal weight.

European/broadcast media: HappyScribe for the subtitle workflow and GDPR-compliant hosting.

Maximum data control: Self-hosted Whisper large-v3. Accept the infrastructure overhead and the lack of real-time streaming.

FAQ

Which transcription API is most accurate in 2026?

AssemblyAI Universal-3 Pro leads third-party English benchmarks with a 5.72% average WER (Feb 2026). OpenAI's gpt-4o-transcribe reports 2.46% WER on FLEURS, though that dataset uses clean read speech.

Does Deepgram support speaker diarization?

Yes - speaker diarization is available as an add-on at $0.0020/min ($0.12/hr) on top of base Nova-3 pricing. It detects multiple speakers and labels segments by speaker ID.

Can I self-host Whisper for production transcription?

Yes, but at scale it requires GPU infrastructure, audio chunking logic, and concurrent request handling. For under 100 hours per month, self-hosting is cost-competitive. Above that, managed APIs are usually more economical when engineering time is factored in.

What output formats do transcription APIs support?

Most APIs output JSON with timestamps, plain text, SRT, and VTT. DOCX is common in SaaS tools (Sonix, HappyScribe) but not usually available from API providers without post-processing.

Which tool is best for transcribing non-English audio?

ElevenLabs (8.75% multilingual WER) and AssemblyAI (9.78%) lead on multilingual benchmarks. Deepgram Nova-3 supports 36+ languages with a multilingual model at $0.0052/min pre-recorded. HappyScribe covers 60+ languages with subtitle workflow support.

Sources

• AssemblyAI Benchmarks (February 2026) - English and multilingual WER data across 26 datasets
• AssemblyAI Pricing - Universal-3 Pro, Universal-2, and streaming tier pricing
• Deepgram Pricing - Nova-3 per-minute rates, diarization add-on costs
• OpenAI API Pricing - gpt-4o-transcribe and gpt-4o-mini-transcribe per-minute rates
• Rev AI Pricing - Reverb ASR, Whisper options, human transcription rates
• HappyScribe Pricing Plans - Subscription tiers and pay-per-use rates
• Sonix Pricing - Standard and Premium tier pricing
• Deepgram Introducing Nova-3 - Nova-3 WER benchmark (5.26%)
• OpenAI gpt-4o-transcribe Launch Coverage - gpt-4o-transcribe accuracy claims and analysis
• AssemblyAI vs Deepgram Accuracy Comparison - Head-to-head accuracy analysis
• Google Cloud Speech-to-Text Pricing - Standard and enhanced model rates

We cover AI security the way the industry actually experiences it - from the CVE to the aftermath. No vendor press releases, no theoretical threat models padded for word count. If a real compromise happened, we report it. If a paper describes a reproducible exploit, we read it and write about whether it matters.

Supply-chain and SDK compromises

SDKs and orchestration layers are where attackers reach the most keys per kilobyte of malicious code. Our most-read story of 2026 was a supply-chain compromise in a widely deployed LLM router, and the pattern has kept repeating.

• LiteLLM supply-chain compromise drains developer credentials
• LLM router agent supply-chain attack
• RoguePilot: GitHub Copilot extension supply-chain attack
• Claude Code npm leak and "Claw-Code" GitHub record
• HackerBot: Claw, Trivy, and the GitHub Actions compromise

Full catalog: /tags/supply-chain-attack/

Agents and assistants weaponized

When the attacker can use the same models you do, defender asymmetry goes to zero. We cover both sides - offensive research on agents that run exploits and defensive coverage of products meant to stop them.

• AI assistants weaponized as C2 proxies
• Single operator uses DeepSeek and Claude to breach 600 FortiGate firewalls
• Kali Linux integrates Claude for automated pentesting
• BugTraceAI Apex: red-team LLM fits on a single RTX 3060
• AI hacker breaches 600 Fortinet firewalls in 5 weeks

Model vulnerabilities and data leaks

Training-data extraction, jailbreaks that scale, and cloud misconfigurations that expose unreleased models.

• Anthropic's Mythos model exposed via CMS misconfiguration
• Anthropic leak reveals Claude Mythos and cybersecurity risks
• Gemini CLI X account hacked in CLI token scam
• GitHub LLM malware repositories
• Anthropic says DeepSeek and Moonshot ran 24,000 fake accounts to steal Claude's capabilities

Benchmarks, red teams, and disclosure

The security research side - what can actually be measured, where the public benchmarks fail, and how responsible disclosure plays out for AI systems.

• Every major AI agent benchmark can be hacked
• Stanford-Harvard AI agent red team study
• Anthropic ships $100M AI cyber defense to 12 rivals
• Claude Code auto mode and agentic safety
• Linux Foundation $12M to fight AI bug slop

Policy, procurement, and national security

Who is allowed to sell AI to whom, and what the government does when it decides something is a supply-chain risk.

• Anthropic sues Pentagon over supply-chain blacklist
• Anthropic wins injunction against Pentagon ban
• Pentagon formally designates Anthropic a supply-chain risk
• Google and OpenAI employees letter limiting military AI
• NIST AI agent standards initiative

Related coverage

Full catalogs are auto-updated on the tag pages:

• Security - all security-adjacent coverage
• Cybersecurity - attacks, defenses, threat intel
• Supply Chain Attack - compromised packages, SDKs, agents
• AI Safety - alignment, oversight, red-team research
• Vulnerabilities - specific CVEs and disclosure stories
• Prompt Injection - input-layer attacks

Why we cover this

Two things separate useful AI-security coverage from the noise. First, a beat editor who reads CVEs, research papers, and vendor advisories before the PR cycle picks them up. Second, reporting that does not flinch when the story implicates a lab we also cover favorably elsewhere. If we write about a new Claude release on a Tuesday and Anthropic ships a supply-chain miss on a Wednesday, you will read about both.

This page is the front door. For the firehose, see the tag pages above, or subscribe to the Awesome Agents daily brief to get security stories as they happen.