Skip to content

Camera–LiDAR fusion on raw autonomous-vehicle data

I built LiDAR-to-camera projection for a 120° f-θ lens from raw NVIDIA AV files and gave every segmented object a metric range, over two 605-frame drives.

Role
Data pipeline, calibration chain, projection and fusion
Setting
Offline processing of recorded drives · research prototype
Stack
  • Python
  • NumPy
  • DracoPy
  • rosbags
  • ROS 2
  • YOLO11-seg
  • OpenCV
City clip: YOLO11x-seg masks, each labelled with the median LiDAR range inside the mask.

Key results

points per LiDAR spin

279,571

points per LiDAR spin

Draco-decoded, 10 Hz

f-θ polynomial projection

6-term

f-θ polynomial projection

with affine correction for the 120° lens

spins written to a ROS 2 bag

199

spins written to a ROS 2 bag

sensor_msgs/PointCloud2, without a ROS install

01

Problem & constraints

A camera-only segmenter knows what an object is but not how far away it is. LiDAR gives range but no semantics. Fusing them needs an exact calibration chain and a lens model that holds up at the edge of a 120° field of view.

The raw dataset is huge (the LiDAR chunk alone is about 32 GB) and ships Draco-compressed point clouds and parquet calibration tables rather than ready-made frames.

Can raw LiDAR and a distorted wide-angle camera be aligned well enough to give each segmented object a trustworthy range?

02

My contribution

Data pipeline, calibration chain, projection and fusion

  • Pulled only the chunks for one clip from a very large Hugging Face dataset and decoded Draco point clouds.
  • Streamed all 199 spins into a rosbag2 as PointCloud2 using rosbags only.
  • Chained quaternion extrinsics into 4×4 transforms (LiDAR → ego → camera) and implemented the polynomial f-θ projection.
  • Fused instance masks with projected points (median range per object) and rendered both clips.

03

System architecture

  1. 01 · Raw dataset

    • data

      Front-wide 120° video

      30 fps

    • data

      LiDAR parquet

      Draco spins @ 10 Hz

    • data

      Calibration parquet

      quaternions + f-θ poly

  2. 02 · Geometry

    • process

      LiDAR → ego → camera

      4×4 transforms

    • process

      f-θ projection

      pixels + depth

  3. 03 · Perception

    • model

      YOLO11x-seg

      instance masks

    • output

      ROS 2 bag

      PointCloud2

  4. 04 · Fusion

    • process

      Median depth inside mask

  5. 05 · Output

    • output

      Annotated frames / video

04

Method & key decisions

Calibrate through the ego frame

Camera and LiDAR extrinsics are both given relative to the vehicle, so points go LiDAR → ego → camera as one composed 4×4 transform.

A lens model for 120°

A pinhole model breaks down at the edges of a wide lens. The projection maps the ray angle θ through the dataset's 6-term polynomial, with the affine correction term, before converting to pixels.

Robust range per object

Each mask collects the LiDAR points that land inside it; the median is used so stray background points at mask edges do not dominate.

05

Evaluation

Dataset
NVIDIA PhysicalAI Autonomous Vehicles (gated)
Clips
City and highway, 605 frames each
Evaluation
Qualitative: projection alignment and per-object ranges; no ground-truth range metric

06

Results

Masks + range: The same frame with LiDAR points projected and coloured by depth, and with instance masks labelled by rangeLiDAR projectionMasks + range
Left: 86,952 LiDAR points projected through the f-θ model, coloured by depth. Right: YOLO11x-seg masks with the median LiDAR depth per object.
Highway clip, same pipeline.

07

Failures, limits & next steps

Timestamp units

Spin timestamps are microseconds; the original bag writer treated them as nanoseconds, so the bag spanned 19.8 ms instead of 19.8 s. Found while packaging and fixed.

The ego hood is segmented as a car

It shows up as CAR ≈ 3.6 m in every frame. The dataset's per-camera ego mask should filter it out; that is the first next step.

Camera/LiDAR sync by index

The camera runs at 30 fps and the LiDAR at 10 Hz; each spin is reused for three frames. Nearest-timestamp matching with motion compensation is the proper fix.

Next steps · what would prove it

  • Apply the ego mask.
  • Nearest-timestamp sync with ego-motion compensation.
  • Score ranges against 3D box annotations.

08

Code & references

DataNVIDIA PhysicalAI-AV is gated and governed by NVIDIA's dataset license. Not affiliated with or endorsed by NVIDIA. The repository is private until the license review is complete.

Contact

Want to go deeper on this project? I'm happy to walk through it.

Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.