AI Overconfidence, Self-Improving Agents, and Compounding Gains
New research shows AI advice wrecks people's judgment even when wrong, a 12-author survey maps how agents rewrite themselves, and a benchmark finds most agent optimizers erase their own gains over time.
7 min read
Three papers landed today that, read together, describe the same problem from different angles: it's getting harder to know when an AI system is actually right, whether it's a person leaning on a chatbot or an agent leaning on its own upgrades. One is a five-experiment behavioral study on why people stop saying "I don't know" once AI is in the room. Another is a 12-author survey, co-written by Jürgen Schmidhuber, that tries to formalize what "self-improvement" even means for a deployed agent. The third is a hard number: most published agent-optimization methods don't survive contact with new tasks.
TL;DR
- AI advice suppresses "I don't know" - across five experiments with 3,132 people, having AI access nearly removed willingness to abstain, cutting accuracy to about a third of baseline even as confidence roughly doubled.
- Self-Improvements in Modern Agentic Systems - a new survey frames every self-improving agent as a foundation model plus a scaffold, then classifies improvement methods by what they update and what signal drives the update.
- Do Agent Optimizers Compound? - on Terminal-Bench 2.0, two of three tested optimization methods lost ground once new tasks arrived; only one that builds in regression control kept improving.
When AI advice makes people worse, not better
Chiara Marcoccia, Walter Quattrociocchi, and Valerio Capraro ran five experiments, four preregistered plus one replication, with 3,132 participants answering hard questions. Each question came with the option to decline: say "I don't know" instead of guessing. The researchers rigged the AI advice to be wrong on purpose, which let them isolate a single variable. Not whether AI helps when it's right, but what just having it around does to a person's willingness to admit uncertainty.
The answer isn't subtle. Access to AI advice nearly removed people's willingness to abstain. Participants answered more questions across the board. But their accuracy fell to roughly one-third of the non-AI baseline, and their confidence in those wrong answers nearly doubled. People weren't just following bad advice. They were following it while feeling more certain than the group with no AI access at all.
The paper's most interesting twist involves money. One condition penalized wrong answers and rewarded correct ones, a direct financial incentive to reintroduce caution. It worked, partially. Participants under the incentive suspended judgment more often and scored higher than the unincentivized AI group. But their accuracy never came close to the no-AI baseline. Paying people to be careful softened the effect. It didn't reverse it.
That's a hard result for anyone building AI-assisted decision tools, from medical triage to customer support to hiring pipelines. Our own guide to catching AI hallucinations covers the mechanics of why models state falsehoods with a straight face; this study adds the missing half of the picture, which is that a fluent wrong answer changes how the human reasons, not just what they type next. Incentive design alone won't fix an interface problem. If a system never offers an "I'm not sure" option as a first-class output, users will keep manufacturing false confidence to match it.
The study isolated AI's mere presence, not its accuracy, as the variable that erodes judgment suspension.
Source: unsplash.com
A framework for agents that rewrite themselves
The second paper is a survey, not a new method, but it's notable for who signed it. Zhe Ren, Yimeng Chen, Dandan Guo, and eight co-authors, including Jürgen Schmidhuber, whose Gödel machine work from 2003 is the intellectual ancestor of most self-improving-agent research, attempt something surveys rarely pull off: an actual formal framework rather than an annotated bibliography.
Their move is to define a modern agent as two parts. A foundation model, plus an operational scaffold of prompts, memory, tools, and control logic wrapped around it. Self-improvement, in their framing, is a "self-induced update operator" that commits changes to either the model's parameters or the scaffold itself. That distinction matters more than it sounds. Fine-tuning a model's weights and rewriting its own system prompt are both "self-improvement" in casual usage, but they have completely different failure modes, verification requirements, and rollback costs.
From there, the survey organizes prior work along two axes: what gets updated, and what signal triggers the update. Reward signals, human feedback, self-critique, environmental outcomes. It's a useful map for anyone trying to place a specific paper (a memory-editing method, a prompt-evolution technique, a weight-level RL fine-tune) into the broader landscape rather than treating each as a one-off. We covered a related self-evolution angle in our look at agent memory circuits and evolution, and the MCP-related risks of self-evolving agents making false edits in our sycophancy and self-evolving agents roundup. This survey gives both of those specific findings a home in a shared taxonomy.
Schmidhuber's 2003 Gödel machine work, self-referential systems that provably improve their own code, underlies much of the taxonomy this survey proposes.
Source: wikimedia.org
Where the survey is honest is in its closing section: evaluation remains unsolved. Measuring whether an agent has truly improved, versus overfit to its own recent history or gamed its own reward signal, is exactly the open problem the third paper below tries to answer empirically.
Most agent optimizers don't survive a second round
RELAI's technical report, authored by Wenxiao Wang, Priyatham Kattakinda, and Soheil Feizi, asks a question that should have been asked years ago: when a paper reports that an agent-optimization method improved performance, is that gain a stable property of the method, or a one-time artifact of tuning against a fixed benchmark?
Their test is a two-phase continual-learning setup built from hard Terminal-Bench 2.0 tasks. Optimize an agent once, measure it, then introduce new tasks and optimize again. Does the second round of gains stack on top of the first, or does it erode what the first round built? They compared three approaches under identical budgets: GEPA, Meta Harness, and RELAI's own Verifiable Continual Learning (RELAI-VCL).
Terminal-Bench 2.0 tasks span real software engineering and systems work, the kind of environment where an optimizer's shortcuts either generalize or quietly fail.
Source: unsplash.com
All three beat the unoptimized baseline in the standard, single-phase setting that most papers report. The divergence shows up only in phase two.
| Method | Lifelong average pass rate |
|---|---|
| RELAI-VCL | 76.4% |
| GEPA | 66.0% |
| Meta Harness | 64.6% |
| Unoptimized baseline | 58.7% |
GEPA's optimized agent transferred below the unoptimized baseline once new tasks arrived, meaning the first round of optimization actively hurt generalization. Meta Harness transferred positively but plateaued, unable to improve further once given a second optimization budget. Only RELAI-VCL both transferred well and kept climbing, reaching the highest pass rate at every stage measured.
"Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize."
That's a specific, falsifiable claim: shortcut solutions look like progress on a fixed benchmark and fall apart the moment the task distribution shifts, unless the optimizer is explicitly checking for regressions as it goes. For teams evaluating coding agents or building their own agent-harness tooling, the practical takeaway is blunt. A benchmark score from a single optimization pass tells you almost nothing about whether that gain survives the next round of real-world failures.
The common thread
All three papers point at the same gap: the difference between a system that looks better on a snapshot and one that's actually more reliable over time. People overestimate AI advice because a confident wrong answer looks identical to a confident right one in the moment. Agent optimizers overestimate their own progress because a benchmark score after one optimization pass looks identical whether or not it generalizes. And the self-improvement survey's core contribution, separating "what changed" from "what signal caused the change," is really an attempt to build the vocabulary needed to ask "did this actually get better, or did it just get better at the test."
None of these papers solve that gap. What they do is give practitioners better instruments for spotting it: abstention rates as a metric worth reporting alongside accuracy, regression control as a required component of any optimization loop, and a shared framework for classifying what kind of "improvement" a paper is actually claiming.
Sources:
