AI Research: Harness Hype, Smarter FIM, Agent Isolation
New arXiv papers show automatic harness evolution loses to plain test-time scaling, a function-aware training trick lifts SWE-Bench scores, and researchers map five isolation boundaries for agent safety.
7 min read
Three papers posted to arXiv's cs.AI feed this week share a common thread: they each puncture something the agent community has been taking on faith. One tests whether letting an agent rewrite its own scaffolding is worth the compute. Another shows that a training tweak most labs haven't tried yet moves SWE-Bench numbers more than expected. The third argues that agent safety needs a vocabulary borrowed from operating systems, not from chatbot alignment.
TL;DR
- Rethinking the Evaluation of Harness Evolution for Agents - automatic harness search underperforms simple parallel sampling once compute budgets are matched, and barely generalizes to held-out tasks
- Function-Aware Fill-in-the-Middle Mid-Training - masking whole functions instead of random spans during mid-training lifts SWE-Bench-Verified by up to 3.2 points at no inference cost
- Isolation as a First-Class Principle for LLM-Agent System Safety - a five-boundary taxonomy (user-agent, agent-tool, agent-execution, agent-agent, system-environment) reframes agent security around where trust crosses a line, not what the model says
Harness evolution doesn't beat scaling when you check the math
Every few weeks a new paper claims that letting an LLM agent redesign its own scaffolding, the code that builds prompts, calls tools, and checks results, beats a fixed harness. A team led by Yike Wang, with co-authors including Hannaneh Hajishirzi and Yulia Tsvetkov, decided to check whether those claims survive a fair fight.
Their complaint is methodological. Most harness-evolution papers search and evaluate on the same benchmark, which invites overfitting, and they rarely compare against a matched-budget baseline. If a harness-evolution method burns ten times the inference budget of a static harness, of course it wins. The question is what happens when you hold the budget constant.
The researchers tested three published harness-evolution approaches, Meta-Harness, Agentic Harness Engineering (AHE), and AEVO, against two simple test-time scaling baselines: parallel sampling and sequential refinement. All ran on Terminal-Bench 2.1, an 89-task corrected version of the CLI benchmark that's become a standard proving ground for agentic coding tools.
The results aren't close. With unit-test feedback available, parallel sampling hit 86.0% pass@1 and sequential refinement reached 84.3%. Harness evolution, using AHE as the primary instantiation, landed at 75.8%. Without test feedback, plain sampling still beat it, 72.3% against 67.4%. Worse for the harness-evolution camp, the improvements barely transferred: gains on held-out tasks averaged just 0.6 points.
The paper's core recommendation isn't "stop evolving harnesses." It's "stop declaring victory without a matched-budget baseline and a held-out test set."
For teams building agent products, the practical takeaway is blunt. Before investing engineering time in a system that rewrites its own scaffolding, run the comparison this paper insists on: same compute, same feedback signal, a test set the search process never saw. Several times that comparison alone erases the advantage.
Matched-budget comparisons are the kind of unglamorous check that decides whether a claimed improvement is real or an artifact of extra compute.
Source: unsplash.com
A function-shaped hole in how coding models are trained
Coding models are usually mid-trained on fill-in-the-middle (FIM) tasks: hide a random span of code, ask the model to predict it from the surrounding context. Yubo Wang and colleagues, including Wenhu Chen, argue that random spans teach the wrong lesson for agent work.
Their observation: a coding agent's action-observation loop, where it calls a tool, gets a result, and continues reasoning from that result, looks structurally like a function call. Arguments get bound, a value comes back, and downstream code has to use that value correctly. Standard FIM, which masks arbitrary chunks of text, doesn't specifically train that skill.
The fix is function-aware FIM. The method parses a codebase's abstract syntax tree to build a program dependency graph, connecting callers to callees and sibling methods within a class. It then scores every function on two axes: complexity (lines of code, cyclomatic complexity, nesting depth) and inferability, a five-signal measure of how much a masked function's behavior can be recovered from caller specificity, callee count, type annotations, documentation, and class membership. Functions selected by this filter, and only those, get masked for FIM training. Some also get a chain-of-thought reason produced by Gemini 3 Flash and inserted into the masked span, so the model learns to reason before it completes.
The team mid-trained Qwen2.5-Coder-Instruct at 7B and 14B, plus Qwen3-8B, on a 2.6-billion-token corpus pulled from 968 GitHub repositories, then ran standard post-training pipelines (R2E-Gym, SWE-Smith, SWE-Lego) on top. Gains on SWE-Bench-Verified ran from +2.8 points on the 7B model to +3.2 on Qwen3-8B; SWE-Bench-Lite gains were larger, from +3.67 up to +5.4 points depending on size. Notably, the lift showed up on non-agentic coding and general tool-use benchmarks too, which suggests the function-call bias is doing something more general than memorizing SWE-Bench patterns.
The authors are upfront about the limits. The corpus is Python-only, cross-base evidence rests on a single non-Qwen2.5 configuration, and the method assumes modular code. Feed it monolithic scripts, generated code, or notebooks, and the pipeline finds fewer functions worth masking.
Function-aware FIM masks whole functions selected by a complexity-inferability filter instead of random spans, teaching models the same argument-in, value-out pattern agents rely on when calling tools.
Source: unsplash.com
For anyone mid-training their own coding model, this is a mid-training recipe change with no added inference cost. That combination, free at serving time, several points of SWE-Bench improvement, is rare enough to be worth testing against your own eval suite before the next release cycle.
Agent safety needs boundaries, not better prompts
A 13-author survey led by Huihao Jing takes a different angle on agent safety than most alignment papers: instead of asking whether a model's outputs are safe, it asks where trust crosses a boundary it shouldn't.
The paper organizes agent safety around five boundaries. The user-agent boundary keeps user-supplied content from being treated as privileged instructions, the failure mode behind direct prompt injection, multi-turn jailbreaks, and persistent compromise through memory injection. The agent-tool boundary governs whether a tool's returned output gets treated as trusted control rather than bounded evidence; the paper argues defenses need to move "from call-level safety to interface-level safety" with contextual integrity checks on tool metadata.
The agent-execution boundary separates planning from acting, the gap where unsafe code execution, malicious browser clicks, and vision-language-action failures live. The agent-agent boundary treats inter-agent messages as bounded contributions rather than unverified control signals, since one compromised message in a multi-agent system can spread like an infection across the whole population. The system-environment boundary is the classic indirect prompt injection surface: a poisoned webpage, email, or RAG index absorbed into context and treated as if it carried authority.
| Boundary | Core failure | Example attack |
|---|---|---|
| User-agent | User content treated as policy | Prompt injection, memory poisoning |
| Agent-tool | Tool output treated as command | Malicious tool metadata, wrong routing |
| Agent-execution | Plans executed without mediation | Unsafe code exec, browser hijack |
| Agent-agent | Inter-agent messages trusted blindly | Prompt infection across agent swarms |
| System-environment | External content granted authority | Indirect injection, RAG poisoning |
None of these threats are individually new. What the survey adds is the observation that serious failures usually cross more than one boundary in sequence: user input overrides control at the first boundary, steers tool selection at the second, and triggers unsafe execution at the third. The proposed fix is "isolation-by-construction," keeping trust boundaries explicit and checked at every hop instead of relying on a single upstream filter to catch everything.
That framing lines up with what we've seen play out in practice: agents breaking in ways that are hard to detect until they've already caused damage, often because a boundary nobody was watching got crossed quietly. A taxonomy alone won't stop an attack, but it gives security teams a checklist that "monitor the model's outputs" never provided: which of the five boundaries does this deployment actually enforce, and which does it just assume?
Multi-step deployments, like package installs across a boot sequence, are exactly the kind of agent-execution surface the isolation taxonomy is built to reason about.
Source: unsplash.com
The pattern underneath
Line these three papers up and a shared skepticism shows through. The harness-evolution paper distrusts a benchmark number until it's been checked against a matched baseline. The FIM paper distrusts the assumption that random-span masking is the right training signal for agentic code. The isolation survey distrusts the idea that model-level safety training is enough once an agent starts calling tools and talking to other agents.
None of this is pessimism about agents working. It's a demand for the same rigor that reward hacking research has already forced onto RL training: show the baseline, name the boundary, prove the number survives contact with a held-out test.
Sources:
