We evaluated four frontier VLMs on 3D spatial reasoning derived from a production Level 4 AV stack. The best scored 55% — better than the 25% baseline for random guessing, yet still wrong on nearly half the questions — and every model’s reasoning trailed its answers. The field clusters in a narrow 49–55% band: public benchmarks may be saturating, but production-grade spatial data still isn’t.
Built in partnership with PlusAI, using their multi-million-mile driving catalog spanning the U.S., Europe, and APAC.
Every answer comes from calibrated sensor data: sub-meter distances, yaw in radians, verified object labels. Sourced from PlusAI’s production perception stack.
Each sample pairs an annotated front-camera image (numbered bounding boxes on detected objects) with a 4-way multiple-choice question. The model must infer depth, lateral position, heading, and object type from a single 2D frame.
This run spans 100 questions across 18 spatial categories — distance, ordering, heading, lateral position, object type, and embodied reasoning. Questions are based on production 3D scene graphs, ensuring metric precision and consistency, and are human-reviewed for accuracy.
Highway, urban, nighttime, construction. Multiple platforms, 2022–2025.
| Category | Task | Ground Truth |
|---|---|---|
identify_distance_long | Classify range: Close / Medium / Far | Metric distance (m) |
relative_distance_long | Compare two objects’ ranges | Distance pair (m) |
pick_closer | Which of two objects is nearer? | Distance pair (m) |
order_closest | Rank objects by distance from ego | Metric distances (m) |
identify_nearest_ahead | Nearest object on the forward axis | Forward projection (m) |
identify_position | Classify position (ahead-left, etc.) | Forward + lateral (m) |
relative_position | One object’s position relative to another | Forward + lateral (m) |
order_leftmost | Rank objects left-to-right in 3D | Lateral offset (m) |
identify_heading | Object heading in clock notation | Yaw angle (rad) |
relative_heading | Same, opposite, or perpendicular? | Yaw diff (deg) |
identify_type | Classify the object type | 3D annotation label |
embodied_distance | Reason about ego’s reach / collision | Trajectory + distance (m) |
Two scoring layers: choice accuracy (right letter?) and rationale match (right reasoning, verified by a judge model against metric ground truth). All four frontier models land between 49% and 55%; none pulls clear of the pack.
The categories carrying the most questions (n ≥ 5). Long-range distance is near-random for all four; position is the one bright spot.
| Category | Gemini 2.5 Pro | GPT 5.5 | Claude Opus 4.7 | Claude Sonnet 4.6 |
|---|---|---|---|---|
| Distance bin (long range) | 32% | 26% | 21% | 32% |
| Relative distance (long) | 46% | 23% | 31% | 38% |
| Heading (clock) | 46% | 69% | 62% | 62% |
| Relative heading | 25% | 38% | 50% | 38% |
| Position class | 80% | 70% | 90% | 90% |
| Relative position | 80% | 80% | 60% | 80% |
| Object type | 86% | 14% | 43% | 29% |
| Overall (100 q) | 55% | 52% | 49% | 49% |
Five patterns, each pointing to a specific, addressable gap in current VLM training.
All four models land near random on long-range distance binning — 21–32%. Monocular depth collapses past the near field; no model recovers metric range at distance.
Relative-heading sits at 25–50% across models. Estimating yaw from a single frame needs the 3D structure these VLMs don’t reconstruct.
Rationale match runs 5–9 points under choice accuracy for every model (Gemini 55 → 49.5%, Opus 49 → 41%). Right letters, shakier reasons.
The four models cluster in a tight 49–55% band; the largest don’t separate from the pack. Spatial competence needs targeted supervision, not more parameters.
Public spatial benchmarks are saturating — but on production-grade L4 ground truth the frontier tops out at 55% — better than the 25% random floor, yet still wrong on nearly half of every run. That distance is the opportunity: metric-grounded spatial data is exactly the training signal these models are still missing.
Every sample includes metric-grounded chain-of-thought rationales: the kind of spatial supervision current training corpora lack.
Step-by-step rationales with exact distances, yaw angles, object types. Directly usable for SFT.
Programmatic generation, with explicit human annotation, and expert-in-the-loop review, from a multi-year driving log archive. This run is 100; full runs produce thousands.
18 categories for precise capability profiling.
LiDAR-camera fusion, sub-meter precision. Deterministic, reproducible, no annotation noise.
Car, SUV, truck, bus, bike, pedestrian, barriers. 10m–200m+. Highway, urban, night, construction.
18 task variants, two-layer scoring, judge-model rationale grading. Ready to run.
Expert-curated, RL & SFT-ready spatial training data at the scale and quality frontier labs require.
Get in touch →
