The Memorial Bot Learns to Fight Back: From Behavioral Cloning to Style-Constrained Reinforcement Learning

The last post ended with the bot finally seeing the game correctly. This one is about what happened when it turned out that seeing correctly wasn't the bottleneck.

Imitation learning has a ceiling, and we hit it hard

Here's the uncomfortable arc of the past six weeks, compressed: we rebuilt the policy around action chunks — a VQ-VAE learned a 256-entry codebook of Alan's 16-frame motor primitives (his actual dash-ins, his magic series, his Shadow Blade timings), and a transformer learned to pick the next primitive from game state. Held-out token accuracy looked great. The behavioral-fidelity metrics looked great.

Live, it was a statue with good taste.

Four live sessions, four honest verdicts from the person holding the P2 stick: "not alan at all." It blocked a little, walked a little, attacked almost never. Three separate root causes, each a lesson:

The copycat lock. The decoder conditioned on its own previous chunk token, and that conditioning self-reinforced: previous token 236 predicted next token 236 at 98% probability, forever. One action, whole match. The literature calls this causal confusion; a February 2026 paper (Big Picture Policies) supplied the theory for why it's unavoidable at our scale — the space of possible histories grows exponentially, and 7,300 replays cannot cover it, so the model latches onto the one history feature that always correlates: itself.

The button-deletion bug class. A single hex constant — 0x3FF where 0xFFC belonged — silently stripped the MK and HK buttons from every emitted action, because two input-bit layouts (Fightcade's GGPO wire format and FBNeo's native bridge format) differ by a two-bit shift. This same class of bug shipped three times in different subsystems before a full-stack audit hunted down every copy. The bot literally could not press a third of its buttons, and no offline metric noticed, because the training data and the metrics lived in one layout and the emulator lived in the other.

Open-loop chunks can't block. Committing to 16 frames of pre-planned input means a ~270ms reaction floor in a game where blocking a jump-in gives you a 3–8 frame window. We tried a hand-coded threat-detector interrupt. It made the bot worse — because when we mined Alan's replays for how he responds to detected threats, he only "responds correctly" 1.2× more than baseline. Human defense is contextual, predictive, woven into positioning. You cannot bolt it on with an if-statement.

That last failure is what finally forced the pivot. You can't patch your way from a passive average to a fighter. The policy itself has to change, which means reinforcement learning — with everything we'd learned about how RL destroys the thing you're trying to preserve.

Reading ~34 papers so the RL run doesn't eat the style

The naive version of this pivot is well understood and fails predictably: fine-tune with PPO on damage-and-wins, and the policy drifts into whatever degenerate strategy beats the opponent, shedding every trace of the human it cloned. The entire point of the project dies in the reward function.

So before spending compute, we spent it on literature — three deep-research sweeps against arXiv, with a rule that no paper counts until its abstract is actually fetched and the claim verified against it (rate-limited verification agents get no benefit of the doubt; a couple of confident-sounding paper IDs from memory turned out not to say what they were "remembered" to say). Roughly 34 papers survived. A few genuinely changed the design:

• PostBC (Wagenmaker, Finn, Levine — Dec 2025) proved the failure we were about to walk into: standard behavioral cloning "can fail to ensure coverage over the demonstrator's actions, a minimal condition necessary for effective RL finetuning." Our prior cloned Alan's average, meaning his rare aggressive actions — the combos, the punishes, the things that win — may have fallen out of the policy's support entirely. RL can't reinforce what the policy never samples. This paper is why our loss anchors on data (advantage-weighted chunks of Alan's actual best moments) and not only on the collapsed prior.
• FASTER (Mar 2026) formalized the reaction-time problem: with committed action chunks, reaction latency is uniform over the execution horizon. The fix that transfers to our stack: predict 16 frames, execute 8, re-decide — with careful bookkeeping so the conditioning still looks like what the model saw in training. (A later audit found our first version of that bookkeeping silently degenerated to argmax decoding 94% of the time. Fixed, verified, humbling.)
• "KL-Regularized RL is Designed to Mode Collapse" (Oct 2025) reframed something we'd measured but not understood: our β-weighted KL leash to the prior — the intended style dial — didn't preserve style at any strength. High-β and low-β policies drifted from Alan equally. The paper's point is that at practical regularization strengths the objective specifies a single-mode target; collapse isn't a failure of optimization, it's what you asked for.
• Behavior-conditioned PPO (Dec 2025) supplied the replacement: don't leash the distribution, reward the measurable profile. Which is what we built.

There was also a negative result worth as much as any positive one: two papers independently killed the idea of adding curiosity bonuses (ICM) for exploration — intrinsic motivation demonstrably changes how the agent plays, which for a style-preservation project is disqualifying. We had an ICM implementation sitting in the repo, ready to wire in. The literature said don't. We didn't.

The RAM archaeology chapter, or: the byte that lied for two months

Sandwiched between the ML work was the kind of bug that only happens in emulator projects.

Our RAM map — reverse-engineered from thousands of Fightcade replay traces — recorded byte 0xFF4008 as the "fighting" flag: 0x99 when a match is live. Every trace agreed. The state features used it. The RL environment used it to detect match end.

Then the first arcade-mode training run produced episodes of eerily constant length, all ending around the 24-second mark. And when the operator captured a fresh arcade savestate, the byte read 0x92... then 0x93... I confidently declared it a "counter that saturates at 0x99" and patched in a band check. Wrong. The next run reset-looped at boot with the byte falling — 0x88, 0x87, 0x86.

The truth, settled by going back to the trace corpus and looking at the byte as a time series instead of a set of values: 0xFF4008 is the round timer, in binary-coded decimal, counting down from 99. It reads as a constant 0x99 in replay-derived savestates because those all snapshot near round start. The actual match-phase byte was its neighbor 0xFF4000 all along (0x08 = fighting, 0x04 = round intro, 0x12+ = menus) — verified against 867 frames where health actually changed. One mislabeled byte had quietly ended every arcade training match at "round timer crosses 90 seconds remaining," which is why run v1's episodes were all the same length, and why its win rate was really "who's ahead on health 10 seconds in."

The kicker: the game's actual timeout rule — whoever has the health lead when time expires wins — means our synthesized match outcomes were accidentally aligned with the real rules the whole time. Sometimes you get lucky in the exact shape of your bug.

Alan, as a vector

The piece of this chapter I'm most attached to: Alan's style is now a number the reward function can see.

Nine components, computed identically from his 400-replay sample and from the bot's live rollouts — attack-press rate, multi-button rate (his specials), approach rate, retreat rate, crouch rate, jump rate, idle rate, and the mean and spread of his spacing. His profile, measured: retreat 0.32 > approach 0.25, idle 0.28, mid-screen spacing 0.52. The patient, distance-controlling player everyone remembers, expressed as a target vector. The RL reward now includes a potential-based term that pays the bot for moving its own rolling profile toward his.

This replaced the dead KL dial, and it came with a satisfying validation: a 68-guide GameFAQs corpus from Alan's own era — mined and distilled into design constraints — independently names almost every component. "Defense, defense, defense" as the top human differentiator. Assist pacing. Throw discipline against blockers. "Repetition is punished." The 1998 strategy writers and the 2026 behavior vector agree on what skill looks like in this game.

Where it stands tonight

Run v2 is training as this posts. Full matches now (the timer bug is dead), a two-stage arcade CPU curriculum, the anchor and the style shaping both live in the loss. The bot sits around 50% against opponents that would have flattened the old clone, entropy healthy, no token-locks, style distance oscillating and — maybe, early — bending toward Alan.

The honest scoreboard: it is not Alan yet. It blocks more than it used to, presses buttons it literally couldn't press a month ago, and loses full matches it used to "win" by phantom timeout. Every one of those is progress of the unglamorous kind: the kind you only get by auditing your own work adversarially, believing live evidence over dashboards, and reading the literature before the compute bill instead of after.

The next milestones are written down and gated: harder curriculum states, a unified retrain that bakes in everything the research said (coverage-preserving pretraining, event-conditioning instead of the self-hypnotizing prev-token, the full 270-dimension perception), then self-play, then the only test with authority — the people who knew him, hands on the stick, saying whether the ghost in the machine moves like the man.

— Built with FBNeo, MLX, and an unreasonable number of verified arXiv abstracts. Part of the ongoing alanmargolies88 memorial bot series.