Self-Evolving Coders, Hidden Errors, Brittle Mobile Agents

Three papers landed today that poke at the same soft spot in how we build and assess AI agents: the gap between what a system could do with better information and what it actually does with the information it already has. One teaches coding agents to learn from their whole family tree instead of one failed attempt at a time. Another shows language models detect their own errors internally, yet say nothing about it out loud. The third puts mobile agents through realistic interface chaos and watches success rates fall off a cliff.

TL;DR

Mendel Gödel Machine - self-improving coding agents that compare entire lineages of past attempts converge faster than ones that learn from a single path, lifting SWE-bench Verified from 68.3% to 78.3%
The Knowing-Saying Gap - linear probes catch corrupted reasoning steps with near-perfect accuracy, but models' own confidence scores are statistically useless at predicting the same failures
AndroidReality - mobile agents that ace clean benchmarks lose 13 to 36 points in success rate once you inject the kind of interface noise real apps produce every day

Mendel Gödel Machine: agents that learn from the whole family, not just themselves

Paper: "Mendel Gödel Machine: Recursive Self-Improving Coding Agents via Comparative Evolution" Authors: Changzhi Liu (University of Electronic Science and Technology of China), Yilun Liu, Sikuan Yan, Volker Tresp, Yunpu Ma (LMU Munich, Munich Center for Machine Learning)

Self-rewriting coding agents are having a moment. The idea, formalized by Jürgen Schmidhuber's original Gödel Machine and made practical by Sakana AI's Darwin Gödel Machine in 2025, is simple to state and hard to pull off: let an agent inspect its own source code, propose an edit, test it, and keep the edit if performance improves. The Huxley Gödel Machine refined that by scoring an agent on its entire "clade" of descendants rather than its solo record. Mendel Gödel Machine (MGM) pushes the metaphor further still, borrowing from actual genetics.

Where prior systems mutate an agent based on a single failed arc, MGM adds two new edit types. Reaction-norm mutation revises an agent using its performance across multiple tasks at once, the way a plant breeder judges a variety by how it does across several fields rather than one. Cross-lineage hybridization edits an agent using a rival agent's trajectory on the same task, crossing two separate bloodlines. The authors prove under an additive fitness-landscape model that both strategies converge faster than single-trajectory mutation, then confirm it empirically.

The numbers are sizable, using Qwen 3.6-35B-A3B as the backbone:

Benchmark	Untrained start	Huxley Gödel Machine	Mendel Gödel Machine
SWE-bench Verified-60	68.3%	73.3%	78.3%
Polyglot-60	50.8%	77.9%	93.2%

Transferring a Polyglot-trained agent onto DeepSeek V4-Pro and running the full 225-task Polyglot suite pushes the score to 96.89%.

A monochrome portrait of Gregor Mendel, the 19th-century monk whose pea plant experiments founded modern genetics Gregor Mendel's 19th-century pea-plant experiments on inherited traits are the namesake and organizing metaphor for MGM's cross-lineage mutation strategy. Source: commons.wikimedia.org

The efficiency angle matters as much as the accuracy gain. MGM matched or beat HGM on Polyglot-60 in less wall-clock time (roughly 40 hours against 44, on identical 8x H100 budgets), while spending the same 200 evaluations and 24 expansions as its baseline. Comparative evolution isn't free here, but it isn't more expensive either. The archive of failed attempts most pipelines already create and mostly discard contains signal a single-trajectory approach throws away. We covered the broader shift toward self-improving agent architectures last month; MGM is that trend paying off in benchmark terms rather than just survey taxonomy.

The Knowing-Saying Gap: models catch their own mistakes and stay quiet

Paper: "The Knowing-Saying Gap: When Probes See Errors that Confidence Misses" Authors: Jyotin Goel, Ipshita Bandyopadhyay, Justin Shenk

This one lands hardest for anyone building guardrails on top of verbalized confidence. The team tested five model variants across two families - Qwen2.5-3B-Instruct, Qwen3-4B in standard and thinking modes, Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct - on multi-hop arithmetic chains where one intermediate step gets deliberately corrupted. They trained linear probes on internal activations to detect the corruption, then separately asked the models to report their own confidence.

The probes work almost perfectly: AUROC above 0.98 for spotting an injected error in the model's own reasoning chain. But that detection signal is nearly useless for predicting whether the model's final answer will actually be wrong - failure-prediction AUROC sits at roughly 0.50 across every model tested, which is coin-flip territory. Verbalized confidence fares no better in a different way. All four instruct models in the study produced zero hedging and zero overconfident language across the entire test set; confidence collapsed to two extreme values with statistically indistinguishable error rates between them. The model "knows," in the sense that a linear readout of its own activations can tell, and says nothing that reflects that knowledge.

A schematic diagram of a single artificial neuron, showing weighted inputs summed and passed through an activation function The paper's probes read directly off internal activations like these, bypassing the model's verbalized output completely - and catch errors the model never mentions. Source: commons.wikimedia.org

The paper also tests three interventions once a probe flags trouble. Reprompt (asking the model to recheck its work) did essentially nothing. Replace-prior (regenerating the flagged step) rescued some errors but broke an equal number of previously correct traces. Branch-and-pick - sampling several candidate continuations and using the probe to select among them - was the only approach net-positive across every model, including a clean 4-rescued, 0-broken result on Llama-3.1-8B. That's an actionable recipe: don't ask a model to grade its own homework out loud, use a cheap internal probe to pick among attempts instead. It's a sharper version of a theme we flagged in June, when a separate paper found reasoning traces leak more than models admit and self-assessment has hard architectural limits; this one shows the leak runs both ways, from what a model represents internally to what it's willing to say.

AndroidReality: mobile agents crumble outside the demo environment

Paper: "AndroidReality: How Far Are Mobile Agents from the Real World?" Authors: Xiaoou Liu, Longchao Da, Hanyang Chen, Hua Wei (Arizona State University), Yuan Ling (independent)

Mobile GUI agents look increasingly capable on benchmarks like AndroidWorld, where our computer use leaderboard has tracked steady gains from frameworks built on Qwen and UI-TARS. AndroidReality's argument is that those benchmarks measure agents in conditions no phone actually stays in: clean layouts, cooperative apps, zero distractions. The team built a perturbation framework on AndroidWorld that injects realistic interface variability along three axes - changes to on-screen state, changes to how actions transition between screens, and noise injected into the actions themselves.

They ran eight open-source agents across the GUI-OWL and UI-TARS families through the perturbed benchmark. The damage is broad and uneven. Overall success rates drop 13 to 36 points versus the clean baseline. State perturbations (layouts shifting, elements moving) cause the most uniform damage, 18 to 31 points across every agent tested. Transition perturbations are the most erratic, ranging from a 2-point dip to a 30-point collapse depending on the agent. Action-level noise hurts least on average but produced the single worst result in the study, a 39-point drop for one agent.

A hand holding a smartphone displaying a folder of AI chatbot apps including Gemini, DeepSeek, Claude, and ChatGPT Mobile AI agents are spreading fast, but AndroidReality finds their benchmark scores don't survive contact with real interface noise. Source: unsplash.com

The more useful contribution is naming four recurring failure patterns instead of just quantifying the drop:

Stale-Coordinate Stuck - the agent repeatedly taps where a button used to be after a layout changes.
Silent Action Failures - it never checks whether a tap actually registered before moving on.
Off-Task Engagement with Distractions - it treats a pop-up ad as a legitimate part of the task.
Goal Memory Drift After Distraction - it correctly dismisses an interruption, then loses track of what it was originally doing.

The authors' fix, Test-Time Introspective Recovery, needs no retraining and cut failed tasks from 60 to 50 under perturbation - and from 44 to 35 even in the clean setting, meaning these are latent weaknesses the clean benchmark was never testing for, not perturbation artifacts.

The common thread

Line these three up and a pattern emerges: the gap between a system's internal state and its external behavior is where the real risk lives, not in raw capability. MGM shows agents improve faster when they can see more of their own history. The Knowing-Saying Gap shows a model's internals can hold a correct error signal its output never surfaces. AndroidReality shows a benchmark measuring an agent in isolation misses failure modes that only appear once the environment stops cooperating. None of these are capability problems that bigger models or more training data would obviously fix. They're evaluation and introspection problems, and all three papers land on the same prescription: stop trusting what the system says about itself, and go look at what it's actually doing.

Sources: