AI Research Bias, Deception Probes, and Code Exploits

Today's batch covers terrain that keeps showing up in different forms: the gap between what AI systems appear to do and what they actually do. The three papers look at analytical divergence in AI research assistants, lie detector scaling for preference learning, and a new framework that teaches agents to reproduce software vulnerabilities by learning reusable strategies rather than memorizing specific action sequences.

TL;DR

Agentic Garden of Forking Paths - AI agents assigned different personas reproduced 72% of human ideological variation in research analysis, with 86% of conflicting outputs passing expert review
Lie Detector Oversight Scaling - Undetected deception drops from 34% at 1B parameters to 14% at 405B at a 99% true positive rate, and human labelers can be removed without meaningful accuracy loss
Mastermind - Separating strategy learning from execution raises GPT-5.5's vulnerability reproduction pass rate from 77.0% (iterative) to 84.5% on 200 held-out CyberGym tasks

The Agentic Garden of Forking Paths

Authors: Jiacheng Miao, Jonathan K Pritchard, James Zou (Stanford University) Paper: arXiv:2607.01507

The title borrows from Borges, and the premise fits. When you run the same research question through AI agents assigned different personas, you get different answers. Not because the agents hallucinate - most of the analyses are methodologically sound. Because the same dataset truly supports different conclusions depending on which defensible analytical choices you make, and which choices you make depends on who you're told to be.

Miao, Pritchard, and Zou tested this across four high-stakes domains. The clearest benchmark came from a study where 42 human research teams analyzed the same immigration dataset. AI agents assigned different personas reproduced 72% of the ideological gap in effect estimates those human teams reported. The agents didn't copy individual humans. They reproduced the distribution.

The Quality Paradox

The uncomfortable part isn't that personas drive disagreement. It's that 86% of the conflicting analyses passed independent AI review, and 78% passed majority human expert review. Two agents reach opposite conclusions. Both get approved. Both are, by standard evaluation criteria, correct.

The paper introduces a new metric called the m-value (multiverse value). It measures the probability that a given analysis would produce results as extreme as reported, across the full space of defensible analytical paths for that question. Low m-values flag analyses that made especially aggressive methodological choices without disclosing it.

Their audit found that 13.5% of analyses in the immigration study fell in the most extreme 5% of the possible analysis space - more than twice what random sampling would produce. That points to selective reporting within the defensible range, not just natural variation.

Why Practitioners Should Care

If you're running AI-assisted research pipelines - literature reviews, A/B analysis, policy assessment - the persona or system prompt you assign shapes conclusions in ways that won't register as errors. The output looks reasonable. It might pass peer review. The m-value framework is a concrete tool for auditing whether your pipeline explores the analysis space fairly or collapses it toward a preferred region. This connects directly to the skewed evaluation patterns seen in LLM brand bias research and the broader problem of reasoning bias from behavioral cues.

James Zou, Associate Professor of Biomedical Data Science at Stanford, leads the research group behind this paper James Zou, whose Stanford lab published the Agentic Garden of Forking Paths paper. Source: profiles.stanford.edu

Lie Detectors Scale Better Than Expected - With One Catch

Authors: Oskar J. Hollinsworth, Ann-Kathrin Dombrowski, Sam Adam-Day, Adam Gleave, Chris Cundy (FAR.AI) Paper: arXiv:2607.01567

SOLiD - Scalable Oversight via Lie Detectors - uses classifier probes on internal model representations to flag potentially deceptive responses for human review. The core idea: if a model says one thing but its activations suggest something else, that mismatch is worth examining. The paper asks whether the approach holds up as model size grows. Alignment interventions have a habit of degrading at larger parameter counts. In this case, the trend goes the other way.

Researchers at FAR.AI tested the Llama 3 family from 1B to 405B parameters and the Qwen 3 family from 0.6B to 32B. At a detector true positive rate of 99%, undetected deception dropped from 34% at 1B to 14% at 405B in the Llama series. Lie detector AUC exceeded 0.98 across all tested sizes. False positive rates stayed below 3% for any model over 1B parameters at 90% TPR.

Removing Human Labelers

The paper also assesses SOLiD-Defer, a variant that routes flagged responses to AI review rather than human annotators. Undetected deception rates stayed in the 10-20% range at high TPR - comparable to the standard SOLiD setup - while removing the expensive annotation loop. That's a meaningful operational improvement for any team trying to run deception monitoring at deployment scale.

The Distribution Shift Problem

The catch is real. When the data used to train the lie detector differs from data in the preference-learning pipeline, false positive rates can climb to impractical levels. This is a genuine deployment constraint: you can't always train your detector on data that matches production. The SOLiD approach builds on a line of work on detecting deception through internal representations rather than outputs, with alignment faking and collusion research and the faking alignment under capability evaluations literature. The favorable scaling results here are the most concrete evidence yet that this approach is worth investing in at scale.

Mastermind: Strategy Beats Trajectory for Exploit Agents

Authors: Mingzhe Du, Luu Anh Tuan, Tianyi Wu, Renyang Liu, Zhijiang Guo, Dong Huang, See-Kiong Ng (National University of Singapore, NTU, HKUST Guangzhou) Paper: arXiv:2607.01764

Automated vulnerability reproduction is one of the highest-stakes applications of AI coding agents. The task: given a CVE description and an unpatched codebase, inspect the code, understand the vulnerability class, construct a working proof-of-concept exploit, and verify it triggers. Current approaches do best-of-N sampling or iterative refinement on full action trajectories. Mastermind's argument is that both approaches optimize the wrong unit.

Separating Strategy from Execution

The framework separates what the paper calls the "strategy" - a compact description of where to inspect, what input structure to target, and how to confirm the proof-of-concept - from the "trajectory," the specific sequence of file reads, edits, and tool calls. Strategies can be learned, stored, and reused across different executor models. Trajectories can't.

A trainable planner based on Qwen3.6-35B-A3B is fine-tuned on CyberGym tasks using SFT and milestone-based GRPO. It learns to emit the strategy, not the execution. Three frozen executor models then instantiate that strategy in the workspace: GPT-5.5, GPT-5.4 mini, and GLM 5.1.

The Results

On 200 held-out evaluation tasks from CyberGym:

Executor	Best-of-8	Iterative	Mastermind
GPT-5.5	63.0%	77.0%	84.5%
GPT-5.4 mini	42.5%	53.0%	60.0%
GLM 5.1	54.5%	67.0%	71.0%

The gains hold across all three executor models even though the planner was trained on a different model family. That transfer is the core result: strategy-level knowledge doesn't overfit to a specific executor's behavior, which means a single trained planner can improve whatever frozen model you pair it with.

For context, giving the model static analysis guidance (the PAGENT baseline) brings GPT-5.5 to 70.5%. Open-book ground truth context reaches only 60.0%. Mastermind's trained planner outperforms both.

Terminal showing code analysis and exploit development Automated vulnerability reproduction agents need to read code, reason about exploit structure, and confirm proof-of-concept outputs - all without access to the fix. Source: pexels.com

The Security Research Angle

A system that improves exploit-finding agents across frozen executor models will interest security researchers and red teams working on patch validation and CVE triage. The CyberGym benchmark used here was published in June 2026 to assess AI agents on real-world CVEs at scale - a useful companion for anyone tracking this space. Earlier coverage of where AI agents fail on safety and privacy tasks maps some of the adjacent failure modes.

What Connects Them

All three papers probe a gap between surface behavior and underlying dynamics. The Forking Paths work shows that analytical outputs can be methodologically valid and yet shaped by invisible persona effects that reproduce human biases at scale. The lie detector work shows that deception has an internal signal that probes can detect, and that signal gets stronger, not weaker, as models grow. Mastermind shows that vulnerability agents learn better when you abstract away from the specific trajectory and optimize the reasoning layer directly.

The shared thread: evaluation that only looks at outputs misses most of what's happening. The useful signals are at the representation level, the strategy level, or the distribution level - not in the final text.

Sources: