Pinductor

The POMDP, and what stays hidden

A partially observable Markov decision process is a tuple of states, actions, observations, a transition model, an observation model, a reward and an initial-state distribution. The agent never sees the state s. It keeps a belief Q, a distribution over states, and updates it by Bayes' rule after every action and observation:

That update needs T, O and ρ₀. In the paper's setting none of them is given; they must be learned from trajectories that contain observations, actions, rewards and termination flags only. Rewards and termination are outcomes of the transition, not perceptual fields: the agent receives them each step but the observation program is never asked to reproduce them.

The panel in the stage's corner prints the current observation exactly as the code does: a 3×3 integer image (2 wall, 1 empty, 11 lava, 10 goal, 12 the agent's own cell), the direction, and the carried object.

Five gridworlds, one observation interface

The arXiv paper evaluates on five deterministic MiniGrid tasks that vary in what the model has to get right. Each button loads a real initial state from the released N = 10 demonstration files and replays that demonstration; the switcher also carries the three stochastic variants of Appendix D.1 and the two non-grid suites of the camera-ready version.

• Empty: an empty room, the sanity check; every method except random solves it.
• Corners: the goal sits in a random corner of a 10×10 room. With a 3×3 view the room is almost rotationally symmetric, so the difficulty is disambiguating where you are.
• Lava: a terminal hazard. A model that does not encode lava as terminating produces a planner that takes the shortcut and dies. Reward is sparse, so the model must combine the description and the trajectories to learn what lava means.
• Four Rooms: long-horizon navigation through narrow doorways toward a goal the agent cannot see; each doorway choice is a near-blind decision under the belief over goal location. (The released code uses a 15×15 layout.)
• Unlock: two rooms separated by a locked door. Find the key, pick it up, toggle the door, traverse, reach the goal. The model must encode the carrying channel and the door-state transition from observation traces alone.

Two suites are not grids. In Blocks World six numbered blocks are dealt afresh over three columns each episode and the gripper, which does not know its own column, sees only the stack beneath it and the block in its hand; the target tower is drawn faintly on the left column, and the replayed demonstration is one of the fifteen breadth-first-optimal solutions the language model was shown. In PopGym MineSweeper a click returns the number of mines adjacent to that cell; the replays are the fifty rollouts the paper used as demonstrations, of which twenty-three end on a mine and twenty-five clear the board; the default replay is a cleared board. Switch to the true-state view to see where the mines were. These two suites matter later, when the question becomes whether the language model learns from data or recalls MiniGrid.

What earlier methods quietly assumed

WorldCoder, GIF-MCTS and most LLM program-synthesis methods assume the world is fully observable. The closest predecessor, POMDP Coder (Curtis et al., CoRL 2025), does learn POMDP programs, but after every episode it is handed the ground-truth states it could not observe at decision time, and its refinement prompts compare predicted states with recorded states.

Toggle the two. POMDP Coder learns from the whole grid plus the true pose at every step. Pinductor learns from the cells the agent actually saw, strictly less information. In robots operating in cluttered spaces or agents in imperfect-information games, neither online nor post-hoc state access exists, so a method that needs it cannot be applied at all.

Both methods otherwise share the pipeline, the language model (Qwen 3.6 Plus), the belief-space planner and the evaluation seeds. The one difference the paper isolates is where the learning signal comes from: hidden states, or particle-filtered beliefs.

The language model proposes the world as code

The prompt contains a natural-language environment description, the ten state-free trajectories, and a code API describing the state, action and observation types. It asks for one Python program implementing four functions: initial_func, transition_func, observation_func, reward_func. Pinductor makes a single joint call per candidate; POMDP Coder makes four per-component calls.

Below are four candidate hypotheses for Lava. They are hand-written stand-ins shaped like the model outputs reproduced in the paper's Appendix I, and they are the programs every interactive panel on this page executes. The stage shows K = 10 states sampled from each candidate's initial-state distribution, the world it imagines before seeing anything.

Score by filtering, not by state labels

For each candidate m, Pinductor runs a particle filter under the candidate's own dynamics along a recorded trajectory. Particles start from ρ₀ᵐ, are pushed through Tᵐ under the recorded actions, and each predicts an observation with the candidate's observation program. Because that program is usually deterministic, its prediction is softened into a kernel likelihood through an observation distance:

The belief Q is resampled to integer particle counts after every observation. When the total post-observation weight drops below K, fresh particles are drawn from ρ₀ᵐ, rolled through the whole action history and kept with probability proportional to their cumulative kernel likelihood (rejuvenation). The score is the filtered expected log-likelihood, up to constants the negative expected observation energy under the belief the candidate itself induces.

Execution becomes debugging feedback

The next prompt does not carry only a scalar score. It summarises where the candidate failed: execution errors, trajectory segments with large observation distance, reward and termination mismatches, and the particle filter's health. The block below is assembled live from the filter lab's current state, in the format of the refinement prompt reproduced in Appendix I.

The prompt also carries a query-by-committee signal. The true transition is unavailable, so the transition programs proposed so far act as a committee: for a belief particle s and an action a, each votes for a next state, and disagreement is the normalised vote entropy

computed on belief particles visited during filtering, for realised and counterfactual actions. Contexts with high entropy are shown to the model together with the observation and the action, so it sees where the current family of programs is unsure. The signal is diagnostic only; it never enters the selection objective.

Refinement as a tree, not a chain

Refinement-by-execution repeats proposal, scoring and feedback over J rounds. Each round picks one existing candidate as the parent, asks the language model for M = 5 revised programs, scores them and adds them to a persistent pool. The parent is chosen with UCB1, so high-scoring candidates are refined more often but less-explored branches keep an exploration bonus:

The smoothed denominator keeps the bonus finite for unexpanded nodes, so a fresh node cannot dominate on the bonus alone. Programs that crash during filtering receive −∞, stay in the pool for their diagnostics, and can never be selected. After all rounds the final program is sampled from the near-best set, every candidate within one empirical standard deviation of the best score, with a softmax at temperature T = 0.1:

Plan in belief space with the program you chose

The selected program is used directly by a belief-space planner, the determinized A* of POMDP Coder (PO-DA*). Nodes are beliefs; an edge applies an action and branches on the predicted observation. Each branch costs

with a budget of 5,000 belief expansions. The entropy term is the one hyperparameter where the two LLM methods differ: 1.0 for Pinductor, 0 for POMDP Coder, because tuning it helped Pinductor and did nothing for the baseline. During the episode the belief is tracked with the same distance kernel used for scoring, so the object the planner consumes is the object the score evaluated.

Close the loop

After each evaluation episode the new trajectory is appended to the replay buffer and a fresh REx round runs online (five attempts), so the program keeps improving from the agent's own experience. The protocol behind every number in the next section: ten offline demonstrations, three evaluation episodes per run seed, ten seeds, 95% bootstrap intervals over seed-level means.

The whole loop runs on a CPU workstation; the language model is reached through OpenRouter. The cost is real: median wall-clock per Lava run is 33 minutes for Pinductor against 17 for POMDP Coder, and Pinductor's joint prompts consume roughly three times the tokens.

Median [interquartile range] minutes per run at N = 10, Appendix H.1.

What comes out is not a policy but four auditable functions. Whether they are right is measured next.