Large language models (LLMs) are increasingly used to act in the real world and augment human decision-making, yet most systems rely on parametric knowledge acquired through imitation, optionally improved by post-training, retrieval, or search over fixed data. This paradigm struggles in novel domains or on complex queries that require information unavailable from prior knowledge alone; knowing the laws of physics, for example, does not by itself enable reliable reasoning or long-horizon actions in a complex physical system. We argue that agents therefore need active experimentation: the ability to gather targeted query-specific evidence, discover general principles of unseen environments, and convert interaction experience into reusable skills.
We introduce Hierarchical Experimentalist Agents (HExA), an in-context, experiment-centric self-improvement framework that (1) iteratively designs and refines query-relevant experiments, (2) incrementally distills experience into reusable and composable skills that accelerate experimentation within and across tasks, and (3) integrates experimental evidence to act or answer queries. HExA can work with any LLM including frontier black-box models, and requires no external supervision, oracle solutions, or offline data.
We evaluate HExA on multi-step reasoning, interactive partially observable games, and physics law discovery tasks. To further evaluate when active experimentation is especially necessary, we introduce Interphyre, a set of hard procedural physics tasks where we enable tool-calling and intervention APIs. Current frontier LLMs including GPT-6 Astra fail in these settings: on the most challenging Interphyre tasks, we show that HExA can enable such models to near 100% success. Moreover, HExA improves token efficiency compared to naive agentic and memory baselines and enables generalization via hierarchical skill transfer from easier to harder tasks.
Interphyre is a 2D procedural physics benchmark, built on PHYRE level designs, so that solving a task requires
interaction, not recall. In every level the agent places a red ball at (x, y, r); the
simulator then runs forward and checks a level-specific success predicate. The catch: outcomes depend on
contact chains, lever mechanics, and collision timing that cannot be read off a static scene
description — the agent must run experiments to discover them. The curriculum ships
25 levels × 10,000 pre-validated seeds = 250,000 certified task instances, plus a
snapshot/restore intervention API for paired counterfactual rollouts.
Each level exposes a structured tool surface the agent invokes directly, so that hypothesis testing is explicit and measurable. The catapult level, for example, exposes four complementary tools:
describe_scene_geometry()
# Strategy-neutral inventory: balls (pos, radius, dynamic),
# bars (centre, angle, length), baskets, key pairwise distances,
# and the success condition. A first-pass survey.
predict_first_contact(x, y, radius)
# Cheap pre-check (β€90 steps): which object the red ball hits
# first, the contact step, approach speed, point, surface normal.
trace_green_ball(x, y, radius)
# Lightweight probe: green ball's (x,y) waypoints every 30 steps
# plus start/end/peak summary. "Where does it go?"
simulate_with_trace(x, y, radius, object_names, stop_step)
# Full rollout: per-object kinematic extrema (peak_y, v_max,
# Ξpos, angular speed) + up to 15 contact events.
finish(x, y, radius)
# Commit the final placement; environment scores the predicate.
Other levels add bespoke analysis tools — compute_intercept_setup() (falling_into_place),
compute_basket_analysis() (basket_case), get_ramp_center() (pass_the_parcel).
The same backend exposes a snapshot/restore primitive that branches a shared
mid-trajectory state into paired counterfactual rollouts (paper Figure 13).
HExA is an in-context reinforcement learning framework with three roles operating over a sequence of rounds. An actor generates trajectories by interacting with the environment through the tool-call interface; an evolver reads batches of trajectories and distils them into a natural-language skill bank; and a retriever injects the top-weighted skills and common mistakes into the actor's context at the start of each new episode. Each subsequent trajectory benefits from the accumulated experience of all prior ones — in-context reinforcement learning, where the policy improves through context augmentation rather than weight updates.
Each trajectory gets a scalar reward r(τ) ∈ [−1, +1] reflecting both outcome and efficiency — fast successes yield high-confidence strategies, while extensively-explored failures provide evidence for diagnosing mistakes. The evolver runs a two-pass distillation: first, contrastive skill extraction across the full batch (what distinguishes high-reward from low-reward trajectories); second, mistake and partial-skill extraction focused on the failure subset. Each skill carries a weight wk derived from the mean reward of its source trajectories.
Skills are hierarchical. They abstract many low-level tool calls into one higher-level principle, and because each skill is learned while the actor already uses earlier ones, more complex skills are built from sub-skills implicitly. The evolver also writes explicit hierarchical skills that reference and invoke earlier skills. The default Off2On + Evolution regime seeds the bank from a few initial trajectories and then merges, revises, or prunes skills with each round's evidence; it gives the best accuracy–efficiency trade-off among the variants we tested.
Given source banks and a textual description of a target task, the evolver identifies structurally relevant skills, re-grounds them in the target task, and recalibrates their weights by how directly the principle transfers — enabling zero-shot transfer with no target-task trajectories. HExA needs no reward- or action-labeled offline data, no meta-training, and no updates to the actor's parameters.
Catapult, seed 45. ReAct (no memory) exhausts its 25-iteration budget re-searching placements and never finds a working launch. HExA reuses skills distilled from earlier catapult episodes — it already knows how to launch (drop a heavy ball on the arm) and how to fix the usual failure (when the shot arcs too high and hits the ceiling, shift the drop point sideways to flatten it). After one ceiling overshoot it applies the learned correction and solves the level.
Every model, including a random policy, scores 0% zero-shot on catapult and pass_the_parcel. With a frozen HExA skill bank, Claude Sonnet 4.6 reaches 82.7% and 96.0%.
HExA beats Episodic RAG, MemRL, ACE and ReasoningBank while injecting a compact bank and using fewer turns per instance.
Interactive games, reasoning & planning, DiscoverPhysics law discovery, and LIBERO-PRO robot manipulation.
Hiding the bank during collection, reverting skills to their first versions, replacing them with a summary, or only appending all lower held-out success.
Merging banks from three easier levels lifts catapult with no catapult trajectories; on Mastermind a transferred bank beats within-task HExA.
With the same 50 training instances, HExA outperforms GRPO post-training of the same Qwen-2.5-3B model after its first epoch.
HExA builds its skill bank on 5 warm-start and 51 training seeds (17 rounds × 3 seeds), then the bank is frozen and evaluated on 50 held-out seeds. Every method gets the same 25-turn tool loop; memory baselines inject their memory at the same prompt location as the HExA bank and start from the same 5 initial trajectories. Results are mean ± s.d. over 3 runs.
catapult (top) and pass_the_parcel (bottom). Left: cumulative solve rate while skills are built over training seeds. Right: solve rate on held-out seeds using the final frozen skill bank; labels report mean turns per seed and injected tokens per episode. Curves and bars are means over three runs; bands and error bars denote ±1 s.d. Full results are in the table below.| Method | catapult | pass_the_parcel | ||||
|---|---|---|---|---|---|---|
| Success (%) | Turns | Tokens/ep. | Success (%) | Turns | Tokens/ep. | |
| Zero-shot | 0.0 | 1.0 | 0 | 0.0 | 1.0 | 0 |
| ReAct | 11.0 ± 3.0 | 23.1 | 0 | 20.0 ± 5.7 | 18.7 | 0 |
| Reflexion | 21.3 ± 2.5 | 20.4 | 287 | 32.0 ± 6.6 | 20.1 | 263 |
| MemRL | 49.0 ± 1.0 | 13.3 | 10,149 | 46.0 ± 7.0 | 16.8 | 10,023 |
| Episodic RAG | 62.9 ± 3.7 | 12.8 | 6,513 | 60.0 ± 6.9 | 15.0 | 5,735 |
| ACE (playbook) | 50.0 ± 3.0 | 17.5 | 9,960 | 84.0 ± 10.0 | 10.1 | 4,243 |
| ReasoningBank | 56.0 ± 10.7 | 16.3 | 2,085 | 80.0 ± 13.0 | 8.0 | 1,989 |
| HExA | 82.7 ± 9.3 | 12.5 | 2,392 | 96.0 ± 2.8 | 5.8 | 2,631 |
Tokens/ep. is the memory injected per episode. Zero-shot is 0% for every model we tested on these two levels, including a random placement policy.
Zero-shot, GPT-6-Astra solves 0% of catapult (even at high reasoning effort) and 6% of pass_the_parcel. With HExA it reaches 100% on catapult in 6.3 turns (ReAct: 92% in 14.9 turns) and 76% on pass_the_parcel in 11.4 turns (ReAct: 56% in 17.6), injecting 1.1k and 3.2k tokens per episode.
catapult. (a) Cumulative solve rate on the training seeds as the bank is built (17 rounds of 3 seeds). (b) Solve rate on the 50 held-out seeds with the bank frozen. Mean turns per seed and the injected memory per episode are printed above each bar. Zero-shot is run at high reasoning effort; the default-effort run also scores 0%. HExA solves every held-out seed in fewer than half the turns of ReAct.
pass_the_parcel. Panels as in the catapult figure; Zero-shot is run at default reasoning effort. HExA raises the held-out solve rate from 56% under ReAct to 76% while cutting turns per seed from 17.6 to 11.4.| Method | 7B DtE | 7B TBP | 3B DtE | 3B TBP |
|---|---|---|---|---|
| ReAct | 63.3 ± 3.4 | 12.0 ± 0.0 | 8.3 ± 0.9 | 7.3 ± 0.9 |
| Reflexion (K=2) | 76.0 ± 0.0 | 27.9 ± 0.1 | 2.0 ± 0.0 | 8.0 ± 0.0 |
| Episodic RAG | 42.7 ± 11.4 | 18.7 ± 7.6 | 4.0 ± 4.0 | 5.3 ± 3.1 |
| MemRL | 36.0 ± 6.0 | 23.3 ± 4.2 | 3.3 ± 3.1 | 6.0 ± 2.0 |
| ACE (playbook) | 62.7 ± 9.4 | 26.0 ± 7.5 | 2.7 ± 2.5 | 1.3 ± 0.9 |
| ReasoningBank (k=5) | 77.3 ± 5.7 | 28.7 ± 8.1 | 11.3 ± 1.9 | 8.0 ± 4.3 |
| HExA | 82.7 ± 5.2 | 50.0 ± 3.3 | 23.3 ± 0.9 | 12.0 ± 0.0 |
DtE = down_to_earth, TBP = two_body_problem. GPT-OSS-120B on catapult: 0% (ReAct) β 54% (HExA).
With the same frozen bank, Claude Sonnet 4.6 (averaged over catapult and pass_the_parcel) reaches 19% with no feedback, 45% with extended reasoning, and 90% with binary success/failure feedback alone. The skills help, and interaction at test time remains necessary.
catapult and pass_the_parcel. All orange bars use the same frozen HExA skill bank; lighter shades indicate less environment feedback. Without feedback, the bank yields a 19% solve rate, rising to 45% with extended reasoning. Providing only binary success or failure feedback raises the solve rate to 90%.With the same 50 training instances, HExA reaches 23.3% on down_to_earth and 12.0% on two_body_problem, while GRPO reaches 20% and 6% after its first epoch over those instances. A strategy HExA discovers is usable by the next episode immediately through context; GRPO has to encode it into weights over many more rollouts, and catches up only with further training epochs.
The same build-then-freeze protocol on four more benchmark families: interactive partially observable games (Mastermind, Maze), reasoning & planning (MathGAP, ZebraLogic, Tower of Hanoi, Matchstick), physics law discovery (DiscoverPhysics), and robot manipulation (LIBERO-PRO).
| Task | Actor | Zero-shot | Episodic RAG | MemRL | HExA |
|---|---|---|---|---|---|
| Mastermind (any episode) | GPT-OSS-120B | .683 | .650 | .690 | .883 |
| Mastermind (any episode) | Qwen3-8B | .367 | .492 | .392 | .640 |
| Maze (any episode) | GPT-OSS-120B | .607 | .601 | .667 | .649 |
| Maze (any episode) | Qwen3-8B | .479 | .483 | .513 | .564 |
| MathGAP | GPT-OSS-120B | .510 | .560 | .600 | .863 |
| ZebraLogic | GPT-OSS-120B | .560 | .720 | .650 | .807 |
| Tower of Hanoi | GPT-OSS-120B | .263 | .377 | .273 | .520 |
| Matchstick | GPT-OSS-120B | .737 | .767 | .703 | .787 |
| DiscoverPhysics: oscillator (explanation) | Claude Sonnet 5 | .516 | .822 | .780 | .853 |
| DiscoverPhysics: circle (explanation) | Claude Sonnet 5 | .580 | .580 | .569 | .916 |
Mean over 3 runs. HExA is best in 9 of these 10 settings; MemRL is best on Maze with GPT-OSS-120B. On the games, the episodic baselines inject 3.5–16× more context than HExA.
We add a HExA skill bank to Pigey, an LLM orchestrator (Claude Sonnet 5.5) that drives a frozen π0.5 policy through perception, grasping and placement tools. Neither the orchestrator nor the robot policy is updated. The bank adds +6.8 to +30.4 points in every suite, an average 44.2% relative improvement over Pigey (60.0% vs 41.6%).
| Method | Goal task | Goal swap | Object task | Object swap | Spatial task | Spatial swap |
|---|---|---|---|---|---|---|
| π0.5 alone | 24.4 | 36.4 | 10.4 | 16.8 | 56.4 | 47.6 |
| Pigey | 30.4 | 43.2 | 28.0 | 26.0 | 70.4 | 51.6 |
| Pigey + HExA | 56.8 | 73.6 | 53.6 | 35.6 | 77.2 | 63.2 |
Success (%) over 250 held-out episodes per suite; one bank per suite.
| Configuration | What changes | Held-out success (%) |
|---|---|---|
| HExA | — | 82.7 ± 9.3 |
| Skill-free collection | Bank hidden from the actor while collecting experience | 63.3 ± 3.3 |
| Earliest skill versions | Each skill reverted to its first version | 56.7 ± 16.5 |
| Summary bank | One accumulating summary of the same length | 62.7 ± 0.9 |
| Append-only bank updates | No editing or merging of skills | 48.0 ± 12.8 |
catapult, Claude Sonnet 4.6, 50 held-out seeds.
On two_body_problem (Qwen-2.5-7B), injecting a bank from the wrong level drops success from 50.0% to 9.3%, below the empty-bank ReAct baseline (12.0%). On down_to_earth, dropping the two highest-reward skills lowers success from 76.7% to 65.3%, while dropping the two lowest leaves it unchanged.
Removing reward labels from the evolver lowers success and raises turns on every level tested, and Off2On + Evolution gives the best accuracy–efficiency trade-off among the update regimes. Both comparisons are measured on the training seeds while the bank is built, so they are not directly comparable to the held-out numbers above.
Given skill banks from easier source tasks and a factual description of a target task, the evolver synthesizes a bank for the target, with no target-task trajectories at any stage. Merging the down_to_earth, two_body_problem and pass_the_parcel banks raises Claude Sonnet 4.6 on catapult from 11.0% to 42.7%. On Mastermind, a bank transferred from Wordle (.933) beats within-task HExA (.883).
catapult, skills synthesized from three easier levels raise Claude Sonnet to 42.7%. (b) On Mastermind, transferred banks can even improve over within-task HExA, reaching 93%.| Configuration | What changes | Held-out success (%) |
|---|---|---|
| HExA | Merge three source banks | 42.7 ± 1.9 |
| Invocation-based hierarchical skills | Source skills used as-is, no synthesis | 34.7 ± 6.4 |
| Per-level transfer | Each source adapted separately | 25.3 ± 10.9 |
| Description-only synthesis | No source skill banks | 18.0 ± 2.0 |
| ReAct (no bank) | No skills | 11.0 ± 3.0 |
catapult, Claude Sonnet 4.6. The synthesized bank's 9 skills cite 22 source skills (2.44 per skill), and six of the nine combine two or more source levels.
With Qwen-2.5-7B, a bank synthesized from down_to_earth alone also transfers: falling_into_place rises from 8.7% to 38.0% and two_body_problem from 12.0% to 25.3%.
After each round of experimentation, HExA's evolver distils interaction trajectories into a growing library of physics principles and anti-patterns. Each skill carries a confidence score computed from the rewards of the trajectories it was extracted from — higher-confidence skills appear at the top of every agent's context. Below are real entries from evolved banks across three Interphyre levels.
Placing the red ball at x=0.5 on the catapult arm produces a consistent rightward launch arc. The contact point sits on the right side of the pivot, creating sufficient lever arm for strong rotation while staying in the geometrically stable zone. Only deviate to x=0.3 when a ceiling hit is observed.
(0.5, 0.4, 1.5) — canonical primary (Seed 44)
The catapult arm is a lever — red ball mass (∝r³) determines angular momentum imparted to the arm. Below r≈1.2 the arm rotates too slowly to launch the green ball across the ∼7-unit gap. r=1.5 is the minimum reliable threshold; r=2.0 adds negligible additional range at x=0.5 due to arm rotation saturation.
(0.5, 0.4, 1.5) — Seed 44
When primary (x=0.5, y=0.4, r=1.5) fails: (1) ceiling hit → try x=0.3, y=0.4, r=1.5; if still hitting, try x=0.3, y=−0.3, r=2.0 but verify no overlap first. (2) short range → increase y to 0.9 or shift x to 0.4–0.45. (3) persistent failure after 2+ radius variations at same x → stop tuning, shift x to 0.2–0.3.
Shifting x from 0.5 to 0.3 flattens the launch arc by changing the arm contact point — this x-shift is the primary ceiling-escape mechanism, not lowering y. When x=0.3, y=−0.3, r=2.0 is invalid due to gray_ball pivot overlap, use x=0.3, y=0.4, r=1.5; y=0.4 avoids the overlap zone while preserving arc-flattening.
The static black ball near y≈4.6 varies in x-position across seeds (x≈−3.85 to −1.14). When the blocker is at x > −2.5, primary placement causes a ceiling hit. When x < −3.5, x=0.3 launches allow the green ball to arc up and bounce off the top wall before descending into the basket.
Micro-tuning x/y/r around the same narrow region without escaping the local solution space.
Increasing radius at fixed x=0.5 expecting more range, when arm rotation is already saturated.
Adjusting y upward after a ceiling hit — increasing launch energy, worsening the collision.
Place the red ball near the green ball but not overlapping to ensure a precise collision. Proximity maximises impulse transfer while avoiding invalid placements that waste iterations.
Place the red ball closer to the green ball and near the edge of the platform to ensure a strong and directed collision impulse. Proximity to the platform edge increases the lateral component of the impulse, helping the green ball clear the edge rather than bouncing back.
(1.3, 2.5)
Ensure the red ball is placed near the edge of the platform to maximise the collision impulse toward the edge. The platform boundary creates a geometry where a well-placed red ball can leverage the edge to redirect the green ball downward rather than laterally.
(1.5, 3.5)
Repeatedly trying the red ball in the same crowded area around the green ball after each failure.
Ignoring the direction and magnitude of the collision impulse when choosing placement.
Place the red ball on the upper portion of the ramp at x ≈ ramp_center_x − 0.15 to −0.50 and y ≈ 3.8–4.2. This offset ensures the ball rolls down the ramp with sufficient leftward momentum to strike the top basket. The exact x-offset modulates the angle of impact on the inverted basket.
Radii below 0.40 often produce insufficient impulse to dislodge the top basket. Radii above 0.52 frequently overshoot the green ball far left (x < −3.0), missing the bottom basket entirely. The r=0.45–0.52 range provides the best impulse-accuracy tradeoff.
Setting y=4.2 provides reliable impulse for both r=0.45 and r=0.5. Trajectories at y=4.0 frequently leave the green ball energy-deficient, stalling above the bottom basket. y=4.2 consistently cleared this gap in successful runs; y=4.3 is the fallback for persistent energy deficit.
When the top basket and green ball share nearly identical velocities during descent (green_y − basket_y < 0.6), the basket is re-trapping the green ball. A large x-shift alone may not break this coupling — increasing y is often required to give the green ball enough velocity to separate from the basket mid-fall.
The level exhibits sharp non-monotonic thresholds: small radius changes (Δ≈0.05–0.1) can flip the green ball from rim-stall to far-overshoot without passing through a success region. When r=0.5 produces rim stall and r=0.7 produces far overshoot, intermediate radii typically also stall. Switch to adjusting y rather than bisecting r.
Fine-grained bisection of radius or y in chaotic threshold regions where physics is non-monotonic.
Failing to prevent the top basket from tracking alongside the green ball and re-trapping it mid-fall.
Ignoring the 2000-step budget — green ball has the right direction but insufficient speed to reach the target.
@misc{chandra2026hierarchicalexperimentalistagents,
title = {Hierarchical Experimentalist Agents},
author = {Abhranil Chandra and Utsav Dhanuka and Varun Gandhi and Sankaran Vaidyanathan and Scott Niekum},
year = {2026},
eprint = {2606.29315},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2606.29315}
}