The authors built a traffic-light control system using a language model that gets reinforcement-learning feedback from traffic simulators. Standard RL struggled because most actions barely change congestion (sparse, delayed rewards), so they added two tricks: a "reward hurdle" that filters out weak feedback signals by subtracting a threshold, and "uncertainty regularization" that pushes the model to be consistent across multiple attempts at the same decision. An 8-billion-parameter LLaMA model trained this way cut travel time by 75% and transferred to a different intersection layout without retraining.
Main takeaways:
- Language models doing RL on traffic control are unstable because most green-light timing changes produce tiny, delayed improvements in congestion
- Subtracting a calibrated threshold from rewards (the "hurdle") filters out noise and focuses learning on actions that matter
- Forcing the model to give consistent answers across sampled outputs (uncertainty regularization) stabilizes training
- A policy learned on one intersection generalized to a structurally different intersection (17% lower travel time, 39% shorter queues) with no fine-tuning
- The system keeps interpretability because it outputs natural-language explanations for its decisions