Skip to content

Multi-camera BEV perception: Lift-Splat with DINOv2

I built a Lift-Splat bird's-eye-view network with a DINOv2 foundation-model backbone, module by module, and verified its six-camera geometry end to end on real nuScenes calibration.

Role
Architecture and geometry implementation
Setting
Architecture and camera geometry verified on real calibration; no trained-model metrics reported
Stack
  • PyTorch
  • DINOv2 ViT-S/14
  • Lift-Splat-Shoot
  • nuScenes devkit
Forward pass with the real calibration of six cameras forms a hexagon of frustums
Open full-size image in a new tab
Forward pass with real calibration: a hexagon of camera frustums, a hole inside the 4 m minimum depth and empty corners beyond 45 m. The maps show the geometry pipeline, not trained predictions.

Key results

frustum points per sample

125,952

frustum points per sample

6 × 41 depth bins × 16 × 32

BEV cell

0.5 m

BEV cell

200 × 200 grid, 100 m × 100 m

01

Problem & constraints

Camera-only BEV perception is attractive because cameras are cheap, but lifting 2D features into 3D requires per-pixel depth reasoning and an exact camera rig model.

02

My contribution

Architecture and geometry implementation

  • Implemented the Lift, Geometry and Splat modules with shape tests.
  • Replaced the image encoder with DINOv2 ViT-S/14 patch tokens.
  • Verified the geometry with the real nuScenes calibration of all six cameras.

03

System architecture

  1. 01 · Input

    • data

      6 cameras

      3×224×448

    • data

      Intrinsics + extrinsics

  2. 02 · Backbone

    • model

      DINOv2 ViT-S/14

      384×16×32 per camera

  3. 03 · Lift

    • process

      41 depth bins, 4–45 m

      softmax ⊗ 64-d context

  4. 04 · Splat

    • process

      scatter_add into 200×200

      0.5 m cells

  5. 05 · Heads

    • output

      Occupancy (3 classes)

    • output

      Detection heatmap

04

Results

nuScenes ground truth for the sample
Open full-size image in a new tab
The nuScenes ground truth the model will learn to predict.

05

Failures, limits & next steps

Next steps · what would prove it

  • Extension: rasterise drivable area, lanes and 3D box centres into BEV targets.
  • Extension: train with DINOv2 frozen, then unfreeze the last blocks.
  • What would prove it: BEV IoU and nuScenes mAP / NDS on v1.0-trainval.

06

Code & references

  • Philion & Fidler, Lift, Splat, Shoot, ECCV 2020.
  • Oquab et al., DINOv2, 2023.
  • Caesar et al., nuScenes, CVPR 2020 (CC BY-NC-SA 4.0).
Next: Infrared tracking

Contact

Want to go deeper on this project? I'm happy to walk through it.

Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.