Counterpunch
A PPO agent that learns NES Punch-Out!! from raw pixels. After about 10 million steps it knocks out Glass Joe in roughly 16 seconds of game time and is starting to land punches on Von Kaiser.
A reinforcement learning agent for Punch-Out!! on the NES. stable-retro runs the emulator and stable-baselines3 trains a PPO agent with a CNN policy on 8 stacked 84×84 grayscale frames. The only other input is the reward, which is read out of RAM.
How it works
- Nine actions. NOOP, dodge left/right, block, body blows, head punches and the star uppercut, each mapped to a button combo.
- Reward from RAM. Health deltas, knockdowns, punches landed, stars earned and used, and a KO bonus. The RAM addresses were found and verified by hand, and the repo includes a written memory map and an HTML ROM map.
- Curriculum. Training is mostly Glass Joe, with a 33% chance of starting against Von Kaiser. The Von Kaiser save state is generated by letting a trained model beat Glass Joe, then capturing the moment the next fight starts.
- Eight frames, not four. Punch-Out!! is a game of tells, and four frames at a 4-frame skip only covers about 267 ms. Doubling the stack lets the agent see a whole wind-up before it commits.
Results so far
The first agent that worked (run 8) wins against Glass Joe about 95% of the time from 2M steps on. The current run is training against both opponents.
The research logs are worth reading for the failures. One run collapsed because of a reward term. Damage was double-counted when health reset after a knockdown. And the longest bug took days and came down to the NES ignoring a START press that only lasted one frame.
Running it
You need Python 3.10+, your own Punch-Out!! ROM, and preferably a CUDA GPU.
pip install -r requirements.txt
python3 scripts/make_state.py --match 1
python3 train.py
python3 evaluate.py --model models/<run>/best/best_model.zip --recordTraining records milestone videos at 1%, 25%, 50%, 75% and 100% of the run.
