← Portfolio · P-01
Research · Continual Reinforcement Learning
Meta-learned neuromodulation for continual RL.
When the reward regime flips with no observable cue, how much does a trained meta-controller actually help — and where is the real bottleneck? A controlled instrument, not a demo.
Most RL systems break when the world changes.
Standard policies overfit a single training regime and collapse after distribution shifts. The goal: make adaptation a first-class behavior — and measure it honestly.
The environment is deliberately hostile: a gridworld where the rewarded goal flips every 100,000 steps with no observable cue. The agent can only infer that the world changed from unexpected rewards. Every run crosses seven hidden switches; success collapses at each one, and what you measure is the recovery.
The system under test is a nested learner. An inner Dyna-PPO agent learns continually — world model, imagined rollouts, prioritized replay, policy anchoring. An outer PPO “Brain” watches 19 summary signals of that training process and sets 7 plasticity levers plus an 8-dimensional neuromodulatory context code, online, while the inner agent learns.
hit-rate-80 on the benchmark — recovery to ≥80% success after hidden shifts
median steps to recover to 80% after a switch (replayed run)
trained-vs-init composite gain — p<0.01, n=32 eval seeds
headroom a zero-forgetting oracle leaves on the table
Watch a real run survive seven hidden switches.
This is genuine logged benchmark data, not an illustration — a late-training episode replayed from the experiment archive. Press play; every collapse is a hidden regime flip.
Meta-control helps. Memory decides.
The full control ladder — including the oracle rungs most published work never runs — relocates the bottleneck.
What held up
A trained Brain beats a matched random-init Brain by +0.029 composite (~5% relative, p<0.01, n=32 eval seeds per arm) — and the estimate converged as sample size grew, the signature of a real effect. The gain is purely mean-recovery: reliability (hit-rate-80) is identical between arms.
What didn’t
An earlier single-seed comparison suggested a far larger gap over a static baseline; the controls campaign traced that to a weak baseline and retired the claim. The neuromodulation pathway itself tested inert — even fed ground-truth regime information, it moved nothing. Three separate small-sample “effects” died under replication.
Where the bottleneck is
A zero-forgetting oracle that simply restores the right policy at each switch scores +0.25 above the controls — roughly 8× the trained Brain’s gain. Catastrophic forgetting here is a knowledge restoration problem, not a learning-dynamics problem. That finding redirects the whole research program toward memory.
A controlled instrument, built to be wrong in public.
Architecture
Inner learner: Dyna-PPO with a CNN actor-critic, a learned world model providing curiosity from prediction error, imagined rollouts, episodic replay prioritized toward older regimes, and policy anchoring at each detected shift. Outer “Brain”: a small MLP PPO agent whose environment is the inner learner’s training run — it observes 19 training signals and outputs 7 plasticity levers plus an 8-dim context code decoded through a frozen orthogonal projection into a feature-wise mask.
Discipline
Frozen benchmark, frozen scorer, single-variable matched contrasts, minimum n=8 with a convergence-vs-decay test before believing anything. The original paper proposed the architecture; the 2026 controls campaign hardened the claims — every number on this page is the audited one.
Why this matters for hiring
The headline isn’t a big number — it’s that the big number was checked. The control ladder (no-controller floor, oracle-timed and oracle-informed rungs, zero-forgetting ceiling) is the part most research skips, and it’s what turned an exciting-looking result into an honest, smaller one plus a much more useful finding about memory.
