Hierarchical Experimentalist Agents

LLM agents that learn like scientists — actively experiment, distill reusable hierarchical skills, and transfer them across tasks, entirely in-context

1University of Massachusetts Amherst
Preprint
Overview of the HExA framework: (a) ReAct, (b) HExA meta-episodes with a skill bank, (c) skill transfer to unseen harder tasks
Figure 1. Overview of the HExA framework. (a) Baseline ReAct: each episode is solved independently with no cross-episode learning. Each episode consists of alternating tool calls for experimentation, terminating in success (✓) or failure (×). (b) HExA: inner episodes are processed in successive rounds of meta-episodes. After some inner episode, an evolver distills the trajectory into a Skill Bank of reusable skills and recurring mistakes that condition subsequent episodes. Repeated interaction helps evolve and refine the task-specific Skill Bank. (c) Transfer to unseen tasks: each source task runs the HExA loop independently to build its Skill Bank. The resulting banks are synthesized by the evolver into a single generalizable Skill Bank for zero-shot transfer to harder target tasks, with no target trajectories used during bank construction.
0% β†’ 83% / 96%
Claude Sonnet 4.6 on the two hardest Interphyre levels (catapult / pass_the_parcel), zero-shot vs. HExA
100%
GPT-6-Astra on catapult with HExA (0% zero-shot, 92% ReAct), in 6.3 turns instead of 14.9
11% β†’ 42.7%
Zero-shot transfer to catapult from skills learned on three easier levels, with no catapult trajectories
15 / 16
Task–actor settings where HExA is best across games, reasoning & planning, DiscoverPhysics and LIBERO-PRO

Abstract

Large language models (LLMs) are increasingly used to act in the real world and augment human decision-making, yet most systems rely on parametric knowledge acquired through imitation, optionally improved by post-training, retrieval, or search over fixed data. This paradigm struggles in novel domains or on complex queries that require information unavailable from prior knowledge alone; knowing the laws of physics, for example, does not by itself enable reliable reasoning or long-horizon actions in a complex physical system. We argue that agents therefore need active experimentation: the ability to gather targeted query-specific evidence, discover general principles of unseen environments, and convert interaction experience into reusable skills.

We introduce Hierarchical Experimentalist Agents (HExA), an in-context, experiment-centric self-improvement framework that (1) iteratively designs and refines query-relevant experiments, (2) incrementally distills experience into reusable and composable skills that accelerate experimentation within and across tasks, and (3) integrates experimental evidence to act or answer queries. HExA can work with any LLM including frontier black-box models, and requires no external supervision, oracle solutions, or offline data.

We evaluate HExA on multi-step reasoning, interactive partially observable games, and physics law discovery tasks. To further evaluate when active experimentation is especially necessary, we introduce Interphyre, a set of hard procedural physics tasks where we enable tool-calling and intervention APIs. Current frontier LLMs including GPT-6 Astra fail in these settings: on the most challenging Interphyre tasks, we show that HExA can enable such models to near 100% success. Moreover, HExA improves token efficiency compared to naive agentic and memory baselines and enables generalization via hierarchical skill transfer from easier to harder tasks.

The Interphyre Environment

Interphyre is a 2D procedural physics benchmark, built on PHYRE level designs, so that solving a task requires interaction, not recall. In every level the agent places a red ball at (x, y, r); the simulator then runs forward and checks a level-specific success predicate. The catch: outcomes depend on contact chains, lever mechanics, and collision timing that cannot be read off a static scene description — the agent must run experiments to discover them. The curriculum ships 25 levels × 10,000 pre-validated seeds = 250,000 certified task instances, plus a snapshot/restore intervention API for paired counterfactual rollouts.

Level gallery

catapult level being solved
catapult Tier 2 Β· hard A green ball rests on a pivoting arm; it must reach the blue ball in a basket on a far ledge, past a ceiling blocker. Sonnet 4.6 zero-shot: 0%
pass_the_parcel level being solved
pass_the_parcel Tier 2 Β· hard A green ball is trapped under an inverted basket; it must reach the blue ball in a basket below the platform. Sonnet 4.6 zero-shot: 0%
down_to_earth level being solved
down_to_earth Tier 1 Gravity-driven drop past an obstruction. Sonnet 4.6 zero-shot: 68%
two_body_problem level being solved
two_body_problem Tier 1 Coordinate two interacting bodies via momentum transfer. Sonnet 4.6 zero-shot: 14%
falling_into_place level being solved
falling_into_place Tier 1 Timed interception: cross a platform edge to meet a falling jar. Sonnet 4.6 zero-shot: 33%
tipping_point level being solved
tipping_point Tier 1 Unstable equilibrium — nudge a balanced structure the right way. Sonnet 4.6 zero-shot: 0%
basket_case level being solved
basket_case Tier 1 Container avoidance: deflect the green ball away from a basket. Sonnet 4.6 zero-shot: 19%
cliffhanger level being solved
cliffhanger Tier 1 Ledge/edge dynamics under gravity. Sonnet 4.6 zero-shot: 0%

The tool-call & intervention API

Each level exposes a structured tool surface the agent invokes directly, so that hypothesis testing is explicit and measurable. The catapult level, for example, exposes four complementary tools:

describe_scene_geometry()
    # Strategy-neutral inventory: balls (pos, radius, dynamic),
    # bars (centre, angle, length), baskets, key pairwise distances,
    # and the success condition. A first-pass survey.

predict_first_contact(x, y, radius)
    # Cheap pre-check (≀90 steps): which object the red ball hits
    # first, the contact step, approach speed, point, surface normal.

trace_green_ball(x, y, radius)
    # Lightweight probe: green ball's (x,y) waypoints every 30 steps
    # plus start/end/peak summary. "Where does it go?"

simulate_with_trace(x, y, radius, object_names, stop_step)
    # Full rollout: per-object kinematic extrema (peak_y, v_max,
    # Ξ”pos, angular speed) + up to 15 contact events.

finish(x, y, radius)
    # Commit the final placement; environment scores the predicate.

Other levels add bespoke analysis tools — compute_intercept_setup() (falling_into_place), compute_basket_analysis() (basket_case), get_ramp_center() (pass_the_parcel). The same backend exposes a snapshot/restore primitive that branches a shared mid-trajectory state into paired counterfactual rollouts (paper Figure 13).

How HExA Works

HExA is an in-context reinforcement learning framework with three roles operating over a sequence of rounds. An actor generates trajectories by interacting with the environment through the tool-call interface; an evolver reads batches of trajectories and distils them into a natural-language skill bank; and a retriever injects the top-weighted skills and common mistakes into the actor's context at the start of each new episode. Each subsequent trajectory benefits from the accumulated experience of all prior ones — in-context reinforcement learning, where the policy improves through context augmentation rather than weight updates.

The HeXA actor, skill evolver, and evolving skill bank feedback loop
Figure 2. The HExA actor-evolver loop. The actor receives distilled skills in its system prompt and runs on physics puzzles, producing trajectories annotated with rewards (successes ✓, failures ×). The Skill Evolver analyzes these trajectories and curates the evolving Skill Bank: skills (max M) are merged, pruned, or promoted by reward signal, and common mistakes (max N) are updated accordingly. Each skill carries a generation index tracking its lineage across evolution rounds. The updated bank is injected back into the actor's system prompt for the next round, closing the loop.

Reward-guided, two-pass distillation

Each trajectory gets a scalar reward r(τ) ∈ [−1, +1] reflecting both outcome and efficiency — fast successes yield high-confidence strategies, while extensively-explored failures provide evidence for diagnosing mistakes. The evolver runs a two-pass distillation: first, contrastive skill extraction across the full batch (what distinguishes high-reward from low-reward trajectories); second, mistake and partial-skill extraction focused on the failure subset. Each skill carries a weight wk derived from the mean reward of its source trajectories.

Hierarchical, evolving skills

Skills are hierarchical. They abstract many low-level tool calls into one higher-level principle, and because each skill is learned while the actor already uses earlier ones, more complex skills are built from sub-skills implicitly. The evolver also writes explicit hierarchical skills that reference and invoke earlier skills. The default Off2On + Evolution regime seeds the bank from a few initial trajectories and then merges, revises, or prunes skills with each round's evidence; it gives the best accuracy–efficiency trade-off among the variants we tested.

Cross-task transfer

Given source banks and a textual description of a target task, the evolver identifies structurally relevant skills, re-grounds them in the target task, and recalibrates their weights by how directly the principle transfers — enabling zero-shot transfer with no target-task trajectories. HExA needs no reward- or action-labeled offline data, no meta-training, and no updates to the actor's parameters.

One Episode, Two Agents

Catapult, seed 45. ReAct (no memory) exhausts its 25-iteration budget re-searching placements and never finds a working launch. HExA reuses skills distilled from earlier catapult episodes — it already knows how to launch (drop a heavy ball on the arm) and how to fix the usual failure (when the shot arcs too high and hits the ceiling, shift the drop point sideways to flatten it). After one ceiling overshoot it applies the learned correction and solves the level.

ReAct — 25 iterations, FAILURE   placement (0.5, −1.0, 2.0)
ReAct failure: 25 iterations, no green–blue contact
HExA — 6 iterations, SUCCESS   placement (0.3, 0.9, 1.5)
HeXA success: 6 iterations, lever flings green ball over the blocker into the basket

Core Claims

Results on Interphyre

HExA builds its skill bank on 5 warm-start and 51 training seeds (17 rounds × 3 seeds), then the bank is frozen and evaluated on 50 held-out seeds. Every method gets the same 25-turn tool loop; memory baselines inject their memory at the same prompt location as the HExA bank and start from the same 5 initial trajectories. Results are mean ± s.d. over 3 runs.

Held-out success on the two hardest levels (Claude Sonnet 4.6)

Building and using the skill bank on catapult and pass_the_parcel: HExA 83% and 96% held-out, above all memory baselines
Figure 3. HExA uses experimentation and learned skills for higher success with greater efficiency. Claude Sonnet 4.6 on catapult (top) and pass_the_parcel (bottom). Left: cumulative solve rate while skills are built over training seeds. Right: solve rate on held-out seeds using the final frozen skill bank; labels report mean turns per seed and injected tokens per episode. Curves and bars are means over three runs; bands and error bars denote ±1 s.d. Full results are in the table below.
Methodcatapultpass_the_parcel
Success (%)TurnsTokens/ep.Success (%)TurnsTokens/ep.
Zero-shot0.01.000.01.00
ReAct11.0 ± 3.023.1020.0 ± 5.718.70
Reflexion21.3 ± 2.520.428732.0 ± 6.620.1263
MemRL49.0 ± 1.013.310,14946.0 ± 7.016.810,023
Episodic RAG62.9 ± 3.712.86,51360.0 ± 6.915.05,735
ACE (playbook)50.0 ± 3.017.59,96084.0 ± 10.010.14,243
ReasoningBank56.0 ± 10.716.32,08580.0 ± 13.08.01,989
HExA82.7 ± 9.312.52,39296.0 ± 2.85.82,631

Tokens/ep. is the memory injected per episode. Zero-shot is 0% for every model we tested on these two levels, including a random placement policy.

A stronger frontier actor: GPT-6-Astra

Zero-shot, GPT-6-Astra solves 0% of catapult (even at high reasoning effort) and 6% of pass_the_parcel. With HExA it reaches 100% on catapult in 6.3 turns (ReAct: 92% in 14.9 turns) and 76% on pass_the_parcel in 11.4 turns (ReAct: 56% in 17.6), injecting 1.1k and 3.2k tokens per episode.

GPT-6-Astra on catapult: training and held-out solve rates
Figure 4. GPT-6-Astra on catapult. (a) Cumulative solve rate on the training seeds as the bank is built (17 rounds of 3 seeds). (b) Solve rate on the 50 held-out seeds with the bank frozen. Mean turns per seed and the injected memory per episode are printed above each bar. Zero-shot is run at high reasoning effort; the default-effort run also scores 0%. HExA solves every held-out seed in fewer than half the turns of ReAct.
GPT-6-Astra on pass_the_parcel: training and held-out solve rates
Figure 5. GPT-6-Astra on pass_the_parcel. Panels as in the catapult figure; Zero-shot is run at default reasoning effort. HExA raises the held-out solve rate from 56% under ReAct to 76% while cutting turns per seed from 17.6 to 11.4.

Open-weight actors (Qwen-2.5, 3 runs)

Method7B DtE7B TBP3B DtE3B TBP
ReAct63.3 ± 3.412.0 ± 0.08.3 ± 0.97.3 ± 0.9
Reflexion (K=2)76.0 ± 0.027.9 ± 0.12.0 ± 0.08.0 ± 0.0
Episodic RAG42.7 ± 11.418.7 ± 7.64.0 ± 4.05.3 ± 3.1
MemRL36.0 ± 6.023.3 ± 4.23.3 ± 3.16.0 ± 2.0
ACE (playbook)62.7 ± 9.426.0 ± 7.52.7 ± 2.51.3 ± 0.9
ReasoningBank (k=5)77.3 ± 5.728.7 ± 8.111.3 ± 1.98.0 ± 4.3
HExA82.7 ± 5.250.0 ± 3.323.3 ± 0.912.0 ± 0.0

DtE = down_to_earth, TBP = two_body_problem. GPT-OSS-120B on catapult: 0% (ReAct) β†’ 54% (HExA).

Is experimentation still needed once the bank is built?

With the same frozen bank, Claude Sonnet 4.6 (averaged over catapult and pass_the_parcel) reaches 19% with no feedback, 45% with extended reasoning, and 90% with binary success/failure feedback alone. The skills help, and interaction at test time remains necessary.

Same frozen bank under different evaluation-time feedback
Figure 6. Effect of evaluation-time feedback. Held-out solve rate for Claude Sonnet 4.6, averaged over catapult and pass_the_parcel. All orange bars use the same frozen HExA skill bank; lighter shades indicate less environment feedback. Without feedback, the bank yields a 19% solve rate, rising to 45% with extended reasoning. Providing only binary success or failure feedback raises the solve rate to 90%.

In-context policy improvement vs. GRPO post-training (Qwen-2.5-3B)

With the same 50 training instances, HExA reaches 23.3% on down_to_earth and 12.0% on two_body_problem, while GRPO reaches 20% and 6% after its first epoch over those instances. A strategy HExA discovers is usable by the next episode immediately through context; GRPO has to encode it into weights over many more rollouts, and catches up only with further training epochs.

Beyond Interphyre

The same build-then-freeze protocol on four more benchmark families: interactive partially observable games (Mastermind, Maze), reasoning & planning (MathGAP, ZebraLogic, Tower of Hanoi, Matchstick), physics law discovery (DiscoverPhysics), and robot manipulation (LIBERO-PRO).

Games, reasoning & planning, and law discovery

TaskActorZero-shotEpisodic RAGMemRLHExA
Mastermind (any episode)GPT-OSS-120B.683.650.690.883
Mastermind (any episode)Qwen3-8B.367.492.392.640
Maze (any episode)GPT-OSS-120B.607.601.667.649
Maze (any episode)Qwen3-8B.479.483.513.564
MathGAPGPT-OSS-120B.510.560.600.863
ZebraLogicGPT-OSS-120B.560.720.650.807
Tower of HanoiGPT-OSS-120B.263.377.273.520
MatchstickGPT-OSS-120B.737.767.703.787
DiscoverPhysics: oscillator (explanation)Claude Sonnet 5.516.822.780.853
DiscoverPhysics: circle (explanation)Claude Sonnet 5.580.580.569.916

Mean over 3 runs. HExA is best in 9 of these 10 settings; MemRL is best on Maze with GPT-OSS-120B. On the games, the episodic baselines inject 3.5–16× more context than HExA.

HExA vs zero-shot, Episodic RAG and MemRL on games, reasoning and DiscoverPhysics
Figure 7. HExA improves frozen actors across all evaluated benchmark families. Averaged scores across task families of interactive games, reasoning and planning, and DiscoverPhysics; exact metrics differ, and full details are in the paper's appendix.

Robot manipulation on LIBERO-PRO

We add a HExA skill bank to Pigey, an LLM orchestrator (Claude Sonnet 5.5) that drives a frozen π0.5 policy through perception, grasping and placement tools. Neither the orchestrator nor the robot policy is updated. The bank adds +6.8 to +30.4 points in every suite, an average 44.2% relative improvement over Pigey (60.0% vs 41.6%).

MethodGoal taskGoal swapObject taskObject swapSpatial taskSpatial swap
π0.5 alone24.436.410.416.856.447.6
Pigey30.443.228.026.070.451.6
Pigey + HExA56.873.653.635.677.263.2

Success (%) over 250 held-out episodes per suite; one bank per suite.

LIBERO-PRO: HExA skill bank improves Pigey in all six suites
Figure 8. HExA improves a robot-manipulation agent on all six LIBERO-PRO perturbation suites. Success rate on the Pigey agent, whose LLM orchestrator drives a frozen π0.5 policy, over tasks 0–9 on held-out initial states (250 episodes per bar, one built bank per suite). π0.5 gives the task instruction directly to the policy with no orchestrator; Pigey adds the Claude Sonnet 5.5 orchestrator; HExA is the same orchestrator with the frozen HExA skill bank injected. The bank adds +6.8 to +30.4 points over the same orchestrator without it, with the smallest gain on the suite where the no-bank agent already succeeds most often (spatial task).

What Drives the Gain

Evolving skills beats summarizing or appending experience

ConfigurationWhat changesHeld-out success (%)
HExA—82.7 ± 9.3
Skill-free collectionBank hidden from the actor while collecting experience63.3 ± 3.3
Earliest skill versionsEach skill reverted to its first version56.7 ± 16.5
Summary bankOne accumulating summary of the same length62.7 ± 0.9
Append-only bank updatesNo editing or merging of skills48.0 ± 12.8

catapult, Claude Sonnet 4.6, 50 held-out seeds.

Bank content, not context length

On two_body_problem (Qwen-2.5-7B), injecting a bank from the wrong level drops success from 50.0% to 9.3%, below the empty-bank ReAct baseline (12.0%). On down_to_earth, dropping the two highest-reward skills lowers success from 76.7% to 65.3%, while dropping the two lowest leaves it unchanged.

Bank content and ranking ablations
Figure 9. How the skill bank content drives the gains. Top: removing skills or mistakes hurts, while unnecessary wrong-level skills hurt and go below the ReAct baseline. Bottom: removing high-reward skills hurts substantially more than removing low-reward ones; permuting rewards also degrades performance.

Reward labels and update regime

Removing reward labels from the evolver lowers success and raises turns on every level tested, and Off2On + Evolution gives the best accuracy–efficiency trade-off among the update regimes. Both comparisons are measured on the training seeds while the bank is built, so they are not directly comparable to the held-out numbers above.

Reward ablation: with vs without reward labels
Figure 10. Solve rate (%) per level; the number above each bar is the mean turns to solve. The no reward bar treats every success and failure identically, so the evolver prunes from its own prior rather than from environment feedback.
Update-regime variants: accuracy vs turns
Figure 11. HExA variants accuracy–efficiency comparison. Solve rate vs. mean turns to termination for each skill-bank initialization and update strategy with ablations on Sonnet 4.6. Colour marks the strategy and shape marks the task; the upper left is better. On both levels, Off2On + Evolution is the strategy with highest accuracy and efficiency. Arrows show its gain over Zero-shot.

Hierarchical Skills Transfer to Unseen Tasks

Given skill banks from easier source tasks and a factual description of a target task, the evolver synthesizes a bank for the target, with no target-task trajectories at any stage. Merging the down_to_earth, two_body_problem and pass_the_parcel banks raises Claude Sonnet 4.6 on catapult from 11.0% to 42.7%. On Mastermind, a bank transferred from Wordle (.933) beats within-task HExA (.883).

Transfer: catapult from three source levels 42.7% vs ReAct 11.0%; Mastermind transferred banks
Figure 12. HExA on transferred hierarchical skill banks improves zero-shot performance on target tasks. (a) On catapult, skills synthesized from three easier levels raise Claude Sonnet to 42.7%. (b) On Mastermind, transferred banks can even improve over within-task HExA, reaching 93%.

Synthesis is what makes transfer work

ConfigurationWhat changesHeld-out success (%)
HExAMerge three source banks42.7 ± 1.9
Invocation-based hierarchical skillsSource skills used as-is, no synthesis34.7 ± 6.4
Per-level transferEach source adapted separately25.3 ± 10.9
Description-only synthesisNo source skill banks18.0 ± 2.0
ReAct (no bank)No skills11.0 ± 3.0

catapult, Claude Sonnet 4.6. The synthesized bank's 9 skills cite 22 source skills (2.44 per skill), and six of the nine combine two or more source levels.

With Qwen-2.5-7B, a bank synthesized from down_to_earth alone also transfers: falling_into_place rises from 8.7% to 38.0% and two_body_problem from 12.0% to 25.3%.

Inside the Skill Bank

After each round of experimentation, HExA's evolver distils interaction trajectories into a growing library of physics principles and anti-patterns. Each skill carries a confidence score computed from the rewards of the trajectories it was extracted from — higher-confidence skills appear at the top of every agent's context. Below are real entries from evolved banks across three Interphyre levels.

Skills 5 principles · generation 10
x β‰ˆ 0.5 Is the Primary Launch Sweet Spot 0.91

Placing the red ball at x=0.5 on the catapult arm produces a consistent rightward launch arc. The contact point sits on the right side of the pivot, creating sufficient lever arm for strong rotation while staying in the geometrically stable zone. Only deviate to x=0.3 when a ceiling hit is observed.

When: Always as the first placement attempt. Deviate only on ceiling hit or overlap. Example: (0.5, 0.4, 1.5) — canonical primary (Seed 44)
Seeds: 44
Use Large Radius (β‰₯1.5) for Sufficient Launch Energy 0.90

The catapult arm is a lever — red ball mass (∝r³) determines angular momentum imparted to the arm. Below r≈1.2 the arm rotates too slowly to launch the green ball across the ∼7-unit gap. r=1.5 is the minimum reliable threshold; r=2.0 adds negligible additional range at x=0.5 due to arm rotation saturation.

When: Always use r=1.5 as baseline. Scale down only when overlap constraints require it. Example: (0.5, 0.4, 1.5) — Seed 44
Seeds: 44
Three-Tier Fallback Sequence for Failed Primary 0.93

When primary (x=0.5, y=0.4, r=1.5) fails: (1) ceiling hit → try x=0.3, y=0.4, r=1.5; if still hitting, try x=0.3, y=−0.3, r=2.0 but verify no overlap first. (2) short range → increase y to 0.9 or shift x to 0.4–0.45. (3) persistent failure after 2+ radius variations at same x → stop tuning, shift x to 0.2–0.3.

When: Immediately when ceiling hit or short range is observed. Do not micro-tune the failed placement. Example: Seed 44 primary hit ceiling → x=0.3, y=0.4, r=1.5 succeeded
Seeds: 44
x=0.3 Is the Ceiling-Escape Position; y=0.4 Is Gray-Ball-Safe 0.93

Shifting x from 0.5 to 0.3 flattens the launch arc by changing the arm contact point — this x-shift is the primary ceiling-escape mechanism, not lowering y. When x=0.3, y=−0.3, r=2.0 is invalid due to gray_ball pivot overlap, use x=0.3, y=0.4, r=1.5; y=0.4 avoids the overlap zone while preserving arc-flattening.

When: Primary (x=0.5) causes ceiling hit. Try x=0.3, y=0.4, r=1.5 before attempting y=−0.3. Example: Seed 44: x=0.3, y=0.4, r=1.5 → green ball bounced off top wall into basket
Seeds: 44
Ceiling Blocker Lethality Depends on Its x-Position 0.87

The static black ball near y≈4.6 varies in x-position across seeds (x≈−3.85 to −1.14). When the blocker is at x > −2.5, primary placement causes a ceiling hit. When x < −3.5, x=0.3 launches allow the green ball to arc up and bounce off the top wall before descending into the basket.

When: Check ceiling blocker x via describe_scene_geometry. If x > −2.5, ceiling escape is needed. Example: Seed 44: black_ball at x=−3.63 → green ball bounced off top_wall then landed in basket
Seeds: 44
Common Mistakes 3 anti-patterns · generation 10

Micro-tuning x/y/r around the same narrow region without escaping the local solution space.

Why it happensThe catapult arm is the most obvious mechanism — after 2–3 failures, agents try small perturbations instead of qualitatively different placements.
How to avoidAfter 2 failures within x±0.2, move to a completely different x zone (x=0.5 → x=0.3) rather than fine-tuning.

Increasing radius at fixed x=0.5 expecting more range, when arm rotation is already saturated.

Why it happensLinear energy intuition: more mass = more energy = more range. But lever rotation saturates before additional mass converts to green ball velocity at x=0.5.
How to avoidWhen r=1.5 fails at x=0.5, never try r=2.0. Change x-position to 0.3 instead.

Adjusting y upward after a ceiling hit — increasing launch energy, worsening the collision.

Why it happensHigher drop = more kinetic energy at impact. But more energy produces a higher-arcing launch, worsening the ceiling hit.
How to avoidAfter a ceiling hit, shift x left to 0.3 (keeping y=0.4) to flatten the arc. Never increase y above 1.5 in response to ceiling hits.
Skills 3 principles · generation 13–14
Optimal Initial Positioning Near the Green Ball 0.90

Place the red ball near the green ball but not overlapping to ensure a precise collision. Proximity maximises impulse transfer while avoiding invalid placements that waste iterations.

When: When placing the red ball for the first time.
Seeds: 51, 52, 53
Red Ball Positioning for Strong Directed Impulse 0.88

Place the red ball closer to the green ball and near the edge of the platform to ensure a strong and directed collision impulse. Proximity to the platform edge increases the lateral component of the impulse, helping the green ball clear the edge rather than bouncing back.

When: When the green ball is near the centre of the platform and needs a directed push toward the edge. Example: (1.3, 2.5)
Seeds: 52
Platform Edge Geometry: Maximise Lateral Impulse 0.56

Ensure the red ball is placed near the edge of the platform to maximise the collision impulse toward the edge. The platform boundary creates a geometry where a well-placed red ball can leverage the edge to redirect the green ball downward rather than laterally.

When: When the red ball needs to push the green ball off the platform edge. Example: (1.5, 3.5)
Seeds: 52, 53
Common Mistakes 2 anti-patterns · generation 13–14

Repeatedly trying the red ball in the same crowded area around the green ball after each failure.

Why it happensAgent lacks confidence in other regions and defaults to the same vicinity, treating failures as positioning errors rather than strategy errors.
How to avoidShift x or y by at least 0.5 units after two consecutive failures from the same region before any further fine-tuning.

Ignoring the direction and magnitude of the collision impulse when choosing placement.

Why it happensAgent focuses on avoiding overlap rather than engineering the right impulse vector — treating placement as a constraint problem rather than a physics problem.
How to avoidApproach from the side opposite the desired green ball exit direction to ensure the impulse is correctly directed.
Skills 5 principles · generation 12
Ramp Launch: Upper-Mid Zone Placement 0.90

Place the red ball on the upper portion of the ramp at x ≈ ramp_center_x − 0.15 to −0.50 and y ≈ 3.8–4.2. This offset ensures the ball rolls down the ramp with sufficient leftward momentum to strike the top basket. The exact x-offset modulates the angle of impact on the inverted basket.

When: Always use as the primary strategy; compute x-offset from get_ramp_center. Example: ramp_center_x=3.401 → try x=2.9; ramp_center_x=3.774 → try x=3.3
Seeds: 51, 53
Safe Radius Range: r = 0.45–0.52 0.88

Radii below 0.40 often produce insufficient impulse to dislodge the top basket. Radii above 0.52 frequently overshoot the green ball far left (x < −3.0), missing the bottom basket entirely. The r=0.45–0.52 range provides the best impulse-accuracy tradeoff.

When: Default to r=0.45 or r=0.5 on first attempt; do not exceed r=0.55 without overshoot analysis. Example: Seed 52: r=0.6 → rim stall, r=0.7 → overshoot to x=−4.73; r=0.5 is safer.
Seeds: 51, 52, 53
y = 4.2 Is the Preferred Default Launch Height 0.87

Setting y=4.2 provides reliable impulse for both r=0.45 and r=0.5. Trajectories at y=4.0 frequently leave the green ball energy-deficient, stalling above the bottom basket. y=4.2 consistently cleared this gap in successful runs; y=4.3 is the fallback for persistent energy deficit.

When: Use y=4.2 as the first-attempt height; verify y + r ≤ 4.95 to avoid ceiling clamp. Escalate to y=4.3 only if green ball still stalls. Example: Seed 51: x=2.9, y=4.2, r=0.45 → SUCCESS after y=4.0 failed
Seeds: 51, 53
Basket-Fall Coupling: Velocity-Match Signals Re-Trapping 0.87

When the top basket and green ball share nearly identical velocities during descent (green_y − basket_y < 0.6), the basket is re-trapping the green ball. A large x-shift alone may not break this coupling — increasing y is often required to give the green ball enough velocity to separate from the basket mid-fall.

When: After any simulation where basket_y is within 0.6 of green_ball_y in the final state, increase y by 0.15–0.20 rather than shifting x. Example: Seed 51: x=2.7, y=3.8 → re-trap; x=2.9, y=4.2 → separation → SUCCESS
Seeds: 51
Overshoot Detection: Physics Has Binary Threshold Behaviour 0.90

The level exhibits sharp non-monotonic thresholds: small radius changes (Δ≈0.05–0.1) can flip the green ball from rim-stall to far-overshoot without passing through a success region. When r=0.5 produces rim stall and r=0.7 produces far overshoot, intermediate radii typically also stall. Switch to adjusting y rather than bisecting r.

When: Two radius values produce opposite failure modes (stall vs. overshoot) — do not bisect. Hold r constant and increase y to add energy instead. Example: Seed 52: r=0.5/0.6/0.65 → rim stall; r=0.7 → far overshoot. Bisecting r was futile.
Seeds: 52
Common Mistakes 3 anti-patterns · generation 12

Fine-grained bisection of radius or y in chaotic threshold regions where physics is non-monotonic.

Why it happensAgent treats the parameter space as smooth and continuous, assuming bisection will converge. Threshold regions have discontinuous jumps.
How to avoidBisect at most once between opposite failure modes, then switch to a completely different parameter axis (e.g., from r to y).

Failing to prevent the top basket from tracking alongside the green ball and re-trapping it mid-fall.

Why it happensAgent adjusts x when basket re-trapping occurs, not recognising that more vertical energy (higher y) is needed to break velocity coupling.
How to avoidWhen final basket_y is within 0.6 of green_ball_y, increase y by 0.15–0.20 rather than shifting x alone.

Ignoring the 2000-step budget — green ball has the right direction but insufficient speed to reach the target.

Why it happensAgent interprets near-miss positions (green above blue) as close to success, when the green ball has actually stopped with steps exhausted.
How to avoidCheck final velocities — if green_ball velocity ≈ 0 at step 2000 with green still above blue, this is an energy-deficit failure requiring a y or r increase.

BibTeX

@misc{chandra2026hierarchicalexperimentalistagents,
  title         = {Hierarchical Experimentalist Agents},
  author        = {Abhranil Chandra and Utsav Dhanuka and Varun Gandhi and Sankaran Vaidyanathan and Scott Niekum},
  year          = {2026},
  eprint        = {2606.29315},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2606.29315}
}