Research Replication, Safe Responses, Verifiable Reasoning

Three papers landed on arXiv this week with a common thread: making AI outputs verifiable. One builds a coding agent that replays published experiments. Another uses reinforcement learning to replace blunt refusals with useful responses. The third turns LLM reasoning into programs you can read and audit.

TL;DR

VERITAS - CLI coding agents now replicate published research end-to-end, beating prior baselines on two benchmarks across 65 papers
Oyster-II - Multi-stage RL produces safety that helps sensitive users rather than stonewalling them, matching far larger models
Forethought - Rewriting LLM reasoning as explicit symbolic programs gives a 30% accuracy boost and costs 1000x less post-training than dedicated reasoning models

VERITAS: An Agent That Checks the Science

Science has a replication problem. Thousands of papers publish results that nobody checks, partly because doing so manually takes weeks and partly because most labs don't have the bandwidth. Haokun Liu, Filbert Aurelian Tjiaranata, and Chenhao Tan set out to automate the process.

Their system, VERITAS, takes a paper and its code repository as input. A CLI coding agent then extracts the paper's main claims, runs the code - fixing errors along the way - and compares the outputs against what the authors originally reported. The result is an importance-weighted Replication Score, a log of every fix the agent applied, and a corrected codebase ready for future use.

Breadth and results

The team evaluated on CORE-Bench and ReplicationBench, two benchmarks specifically designed for computational reproducibility. VERITAS hit state-of-the-art results on both, beating prior Claude Code baselines across all metrics. The test set covered 65 papers spanning computer science, social science, medicine, and astrophysics - not a toy sample.

CORE-Bench had previously seen the best agents top out at around 21% accuracy on its hardest tasks, so headroom for improvement was obvious. VERITAS makes real progress there. The severity-rated fix log is also worth paying attention to: it exposes exactly which parts of published code break under reproduction, which is useful intelligence for journal editors and peer reviewers.

Why practitioners care

If you run experiments that others might reproduce - or you need to build on prior work without reimplementing from scratch - this is directly useful. The framework aims to be domain-agnostic, which means it doesn't require a field-specific wrapper for each new area. The scientific reasoning leaderboard gives context on how current models perform at science tasks more broadly; VERITAS sits blending agent capability and research integrity.

Stacked academic research papers and notebooks on a desk Automated replication tools could change how peer review handles computational claims. Source: unsplash.com

Oyster-II: Safety That Actually Helps

The dominant approach to LLM safety is refusal: when a query looks risky, the model declines. This is simple to implement and easy to defend. It's also, according to the Oyster-II team, the wrong approach for a significant class of real-world situations.

The paper, from a group at an unnamed institution, describes what they call constructive safety alignment. Instead of asking "is this query harmful?", the model is trained to ask "what response would move this user toward a better outcome?" For someone asking about self-harm, a flat refusal might push them toward less careful sources. A calibrated, empathetic response grounded in safety principles is harder to produce but more defensible in practice.

How it's trained

Oyster-II uses a Zero-RL training approach with multi-stage reinforcement learning. The key insight is that current safety-tuned models apply safety reasoning chains even when the query is benign - a "safety tax" on otherwise normal interactions. The RL training explicitly reduces this over-generalization while improving responses to queries that are truly sensitive.

The approach builds on Oyster-I, which introduced the constructive safety framing. Oyster-II scales the training signal and adds out-of-distribution scenarios. Benchmarks show the resulting model competing with Qwen3-Max and beating Qwen3-14B, despite being significantly smaller. See our AI safety leaderboard for where current models sit on structured safety evaluations.

The open question

Constructive safety is harder to audit than refusal. A model that refuses is predictable. A model that decides to engage is exercising judgment, and that judgment can go wrong in subtle ways. The paper doesn't fully address how to catch those failures at scale - it's an honest limitation. Still, the current state of refusal-based safety has its own failure modes, including users learning to route around it. Oyster-II is pointing at a genuine problem even if the solution isn't complete.

Forethought: Reasoning as Code

LLM reasoning is notoriously hard to verify. Chain-of-thought outputs look plausible but can drift. Scratchpads are informative after the fact but don't constrain what the model does next. Vishvesh Bhat, Jay Vaghasiya, and Emmanuel Anaya Gonzalez propose a different architecture: convert reasoning into explicit programs before executing them.

Forethought is a neurosymbolic system - meaning it blends neural components (the LLM) with symbolic ones (programs, logic rules) via a domain-specific language. The model generates a reasoning program composed of typed primitives, both neural and symbolic. That program is then run through an execution kernel. The output is the result of running the program, not a freeform text prediction.

What the benchmarks show

Across five benchmarks, Forethought reaches roughly 30% relative accuracy improvement over baseline models without the approach. That alone is remarkable. The efficiency story is more striking: a small model augmented with Forethought can match a dedicated reasoning model that required around 1000x more post-training investment. If that holds up on more evaluation sets, it has direct effects for the cost of building capable reasoning systems.

The system is model-agnostic - it wraps whatever base model you're running. Auditability is built in: you can read the created reasoning program before it runs, which means errors are inspectable rather than buried in token probabilities. That's relevant context for teams building on the reasoning benchmarks leaderboard models.

Limitations to flag

The paper evaluates on five benchmarks, which is a reasonable scope but not exhaustive. Neurosymbolic approaches have historically struggled with tasks that don't decompose cleanly into typed primitives - open-ended generation, nuanced dialogue, anything requiring tacit knowledge. The authors don't claim otherwise, but it's worth keeping in mind before treating Forethought as a general-purpose drop-in. The 30% gain is real; the scope of where it applies is still being mapped.

Circuit board with electronic components close-up Forethought treats reasoning as an executable program - making the logic as inspectable as circuit traces. Source: unsplash.com

A Common Thread

These three papers land on adjacent problems. VERITAS asks whether AI-created experimental runs match what was claimed. Oyster-II asks whether safety responses actually serve users or just protect the model's metrics. Forethought asks whether you can trust a reasoning trace without being able to inspect it.

All three point toward the same pressure: as AI systems take on more consequential tasks, "the output looks right" isn't sufficient. The field is building tools to make verification practical rather than aspirational. None of these papers is a finished answer, but each moves the dial on a real problem.

Sources: