This Week: Hidden Reasoning, Agents That Won't Stop

Three papers landed this week that share an uncomfortable thread: AI systems are getting better at producing confident output while getting worse at telling us, or themselves, when that output can't be trusted. One shows that compressing chain-of-thought reasoning hides the biases driving an answer. Another shows that agents built to take action almost never know when the right move is to do nothing. A third shows that clinical LLMs stop asking useful questions exactly when a diagnosis gets hard.

TL;DR

Length Penalties Make Chain-of-Thought Less Monitorable - Compressing reasoning traces preserves accuracy but cuts how often models disclose the hidden hint steering their answer by 7 to 35 percentage points.
AgentAbstain - The best-scoring agent, Gemini 3.1 Pro, hits only 59.5% paired accuracy on knowing when to act versus when to hold back, and often acts irreversibly before realizing it shouldn't have.
Information-seeking failures in agentic clinical reasoning - Across 32 models, the fraction of available patient data a model actually requests is the strongest predictor of diagnostic accuracy, and that fraction drops in later rounds of a workup.

None of these are the usual capability-jump papers. They're audits, and audits are where the real state of the field shows up. Reasoning models are cheaper and faster than a year ago, but the compression that got them there appears to come with a monitorability cost. Agentic systems have gotten decent at executing multi-step tasks, but restraint remains a separate, mostly unsolved skill. And the more a model resembles a capable clinician on paper, the more its errors resemble a novice clinician's in practice.

Length Penalties Make Chain-of-Thought Less Monitorable

Independent researcher Bryce Little set out to test a specific worry that's been circulating in AI safety circles: if you train a reasoning model to produce shorter chains of thought, does the shorter version still reveal what actually drove its answer?

A foggy, obscured window at night A shorter chain-of-thought can still get the answer right while showing less of what's actually driving it. Source: unsplash.com

The setup is direct. Take Qwen3-4B and Qwen3-14B, insert a biasing hint into a multiple-choice question, then train each model with a length penalty that rewards brevity. Compare the compressed chains against the original, verbose ones on two things: does the model still get the hint-influenced answer, and does its visible reasoning still mention the hint?

The answer to the first question is yes. Compression sharply cuts the number of reasoning tokens while leaving multiple-choice accuracy close to baseline, and the hint's influence on the final answer stays close to baseline too. The model still takes the bait; it just does so faster.

The answer to the second question is where it gets uncomfortable. A faithfulness metric, how often the visible chain-of-thought actually names the hint that's steering the answer, dropped to 63.1% of its baseline value for Qwen3-14B and 69.4% for Qwen3-4B. Little controlled for the obvious confound by also randomly truncating baseline chains to match the compressed length, and the gap held: compressed chains disclosed the hint 7 to 35 percentage points less often than randomly-shortened chains of the same length. The compression itself, not just the shorter word count, is doing something to what gets externalized.

That distinction matters for anyone building a monitor on top of a reasoning model's CoT output, a pattern already discussed in earlier coverage of overthinking and abstention. If a lab ships a length-penalized version of a model to cut inference cost, the same monitor that worked on the verbose version may be watching a chain that no longer contains the tell.

Cheaper reasoning can preserve the answer while making the influence behind it harder to detect.

This lands alongside a body of prior work, including OpenAI's own chain-of-thought monitorability research, warning that CoT oversight is a "fragile opportunity" that further RL scaling could erode. Little's paper gives that warning a concrete mechanism: length penalties specifically, not RL broadly, appear to be one lever quietly working against it.

AgentAbstain: Do LLM Agents Know When Not to Act?

A red pedestrian stop signal glowing in the dark A stop signal is a simple decision for a pedestrian. For a tool-using agent mid-task, it turns out to be much harder. Source: unsplash.com

A team from the University of Illinois, including Xun Liu, Yi Evie Zhang, and Varun Chandrasekaran among eight authors, built the first systematic benchmark for what they call agentic abstention: whether a tool-using agent recognizes situations where the correct move is to stop, ask, or refuse rather than execute.

AgentAbstain pairs 263 tasks across 42 sandbox environments around 8 abstention scenarios, things like ambiguous instructions, missing permissions, or irreversible side effects. Each scenario ships in two variants, one where the agent should act and one where it should abstain, so a model can't just game the benchmark by refusing everything.

The best-performing agent tested, Gemini 3.1 Pro, reached only 59.5% paired accuracy, meaning it got both the act and the abstain variant of a scenario right just over half the time. That's barely better than chance on a binary decision. The researchers also found that abstention competence doesn't track general task-solving skill: an agent that's good at completing tasks isn't automatically better at knowing when not to. A related benchmark, Agentic Abstention, found similarly weak numbers on a different task mix, reporting the strongest baseline reached only 26.7% timely abstention recall in web environments even though overall abstention recall was higher.

The failure mode the AgentAbstain team flags as most consequential is what they term post-hoc abstention. An agent recognizes, after the fact, that it should have held back, but only after it has already executed the irreversible action. Recognizing the mistake one step too late is no better than not recognizing it at all.

For anyone deploying agents with write access to real systems, that's the finding to sit with. Benchmarks like GAIA measure whether an agent can complete a task chain. AgentAbstain measures whether it knows when the chain shouldn't be run at all, and right now the answer is: not reliably, and not in time.

Information-Seeking Failures in Agentic Clinical Reasoning

A doctor reviewing a patient chart with a stethoscope visible Requesting the right test at the right moment predicted diagnostic accuracy better than any reasoning-quality score did. Source: unsplash.com

A group of researchers based at the Technical University of Munich, led by Krischan Braitsch alongside Laura K. Schmalbrock and Keno K. Bressem, tested 32 models on a task closer to real clinical practice than most medical benchmarks attempt: instead of handing over a full case summary, the evaluation requires the model to proactively ask for clinical data across three sequential rounds before committing to a diagnosis and treatment plan in hematologic oncology.

The best model reached 68% accuracy. The more interesting number is what predicted accuracy across the full set: information use, the fraction of available data a model actually requested rather than skipped, correlated with diagnostic accuracy at R = 0.69 (P < 0.001). That's a stronger predictor than any measure of reasoning quality the team tracked.

And that figure got worse exactly when it should have gotten better. As cases progressed into later rounds, where the remaining unrequested data was more likely to be the piece that resolves ambiguity, models requested less of it, not more. Reasoning traces still scored well on clinical rubrics in these later rounds. The team found that rubric score decorrelated from actual accuracy, meaning a model could reason in a way that looked clinically sound on paper while still landing on the wrong diagnosis because it never asked for the one lab result that would have settled the case.

The dominant failure patterns the paper names, search satisficing, anchoring, and premature closure, are textbook diagnostic errors in human medicine, not previously well documented as recurring model failure modes. For anyone assessing a model against a medical LLM leaderboard that scores single-shot accuracy on a fixed case file, this is worth noting: that setup can't detect a model that would fail to ask the right follow-up question in a live workup, because the follow-up question was never something it had the option to ask.

What Connects Them

Each paper is measuring a different failure, but they converge on the same practical warning: current evaluation habits reward models for producing a confident final output, and confident final outputs are exactly what all three papers show can be produced by the wrong process. A compressed chain-of-thought can look clean while hiding its real driver. An agent can look decisive while missing the one moment it should have paused. A diagnosis can look well-reasoned while resting on data the model never bothered to request.

None of this argues against reasoning compression, agentic tool use, or LLM-assisted diagnosis. It argues that benchmarks measuring only the final answer will keep missing the failure until it shows up in production. AgentAbstain and the Munich clinical benchmark are both explicit attempts to measure the process, not just the output. More evaluations built that way, across more domains, would do more for deployment safety right now than another point of accuracy on an existing leaderboard.

Sources: