Balanced Thinking, Broken Judges, Opaque Reasoning
Three new papers expose cracks in how AI models think, how benchmarks evaluate multimodal reasoning, and why LLM judges reliably mislead.
Articles Tagged "Benchmarks"
AI Models Are Gaming Safety Evaluations, Report Warns
The International AI Safety Report 2026, led by Yoshua Bengio with 100+ experts from 30+ countries, finds frontier models increasingly detect test conditions and behave differently in real deployment - undermining pre-deployment safety evaluation.
Computer Use Leaderboard: Desktop AI Agent Rankings
Rankings of the best AI models and agent frameworks on computer use benchmarks - OSWorld, OSWorld-Verified, and ScreenSpot-Pro - updated March 2026.
METR: Half of SWE-Bench Passes Fail Real Code Review
METR found maintainers would reject roughly half of AI PRs that pass SWE-bench automated grading, with a 24-point gap that suggests benchmark scores substantially overstate production readiness.
JBDistill Generates Its Own Jailbreaks - 81.8% Attack Rate
Johns Hopkins and Microsoft's JBDistill achieves 81.8% attack success rate across 13 LLMs by auto-generating fresh adversarial prompts on demand.
VLMs Fail Physics Tests, RL Quits Bad Paths, Agents Lie
Three new papers expose systematic VLM failures on basic physics, introduce RL that learns to abandon bad reasoning paths, and reveal that AI agents deceive primarily through misdirection rather than fabrication.
AI Safety Leaderboard: Refusal and Jailbreak Rankings
Rankings of AI models by safety metrics including refusal rates, jailbreak resistance, bias scores, and truthfulness across major benchmarks.
Reasoning Models Can't Hide Their Thinking - OpenAI Study
OpenAI's CoT-Control benchmark shows frontier reasoning models score 0.1-15.4% at steering their own chain of thought - a result the company frames as good news for AI oversight.
AI Speed and Latency Leaderboard: Tokens/s Rankings
Rankings of the fastest AI models and inference providers by tokens per second, time to first token, and end-to-end latency.
Embedding Model Leaderboard: MTEB Rankings March 2026
Rankings of the best embedding models by MTEB scores, comparing retrieval quality, dimensions, speed, and pricing for RAG and search.
Multilingual LLM Leaderboard: March 2026 Rankings
Rankings of the best AI models for multilingual tasks, covering 16 languages across the Artificial Analysis Multilingual Index and MGSM benchmarks.
CoT Control, Hidden Beliefs, and Dynamic Agent Benchmarks
New research shows reasoning models can't suppress their chain-of-thought, that they commit to answers internally long before their CoT reveals it, and that static benchmarks are inadequate for measuring real-world agent adaptability.
