The authors solve the problem of LLMs making coordinate errors when building 3D structures by introducing "2.5-D decomposition": the LLM only plans in the horizontal (x-y) plane, and a deterministic executor computes all vertical (z) placement by tracking what's already stacked in each column. This eliminates an entire class of errors. On the Build What I Mean benchmark, GPT-4o-mini with this pipeline achieves 94.6% accuracy (within 3 points of the ceiling set by errors in understanding instructions), beating both GPT-4o (90.3%) and the best competing system (76.3%). The approach transfers directly to edge hardware (Nemotron on Jetson Thor) with no prompt changes.
Main takeaways:
- LLMs make systematic errors when generating 3D block placements; removing one dimension (vertical) from the LLM's output space and computing it deterministically fixes this
- GPT-4o-mini with 2.5-D decomposition achieves 94.6% structural accuracy, within 3 points of the ceiling and beating GPT-4o at 90.3%
- Controlled ablation shows 2.5-D decomposition accounts for 50.7 percentage points of accuracy improvement
- Approach transfers to edge hardware (Nemotron-3 120B on Jetson Thor) with no modifications, matching cloud performance at 94.5%
- Principle generalizes: remove deterministic dimensions from LLM output when physical constraints (gravity, assembly rules) fix degrees of freedom