Camera–LiDAR fusion on raw autonomous-vehicle data
I built LiDAR-to-camera projection for a 120° f-θ lens from raw NVIDIA AV files and gave every segmented object a metric range, over two 605-frame drives.
- Role
- Data pipeline, calibration chain, projection and fusion
- Setting
- Offline processing of recorded drives · research prototype
- Stack
- Python
- NumPy
- DracoPy
- rosbags
- ROS 2
- YOLO11-seg
- OpenCV
Key results
- points per LiDAR spin
279,571
points per LiDAR spin
Draco-decoded, 10 Hz
- f-θ polynomial projection
6-term
f-θ polynomial projection
with affine correction for the 120° lens
- spins written to a ROS 2 bag
199
spins written to a ROS 2 bag
sensor_msgs/PointCloud2, without a ROS install
01
Problem & constraints
A camera-only segmenter knows what an object is but not how far away it is. LiDAR gives range but no semantics. Fusing them needs an exact calibration chain and a lens model that holds up at the edge of a 120° field of view.
The raw dataset is huge (the LiDAR chunk alone is about 32 GB) and ships Draco-compressed point clouds and parquet calibration tables rather than ready-made frames.
Can raw LiDAR and a distorted wide-angle camera be aligned well enough to give each segmented object a trustworthy range?
02
My contribution
Data pipeline, calibration chain, projection and fusion
- Pulled only the chunks for one clip from a very large Hugging Face dataset and decoded Draco point clouds.
- Streamed all 199 spins into a rosbag2 as PointCloud2 using rosbags only.
- Chained quaternion extrinsics into 4×4 transforms (LiDAR → ego → camera) and implemented the polynomial f-θ projection.
- Fused instance masks with projected points (median range per object) and rendered both clips.
03
System architecture
01 · Raw dataset
data
Front-wide 120° video
30 fps
data
LiDAR parquet
Draco spins @ 10 Hz
data
Calibration parquet
quaternions + f-θ poly
02 · Geometry
process
LiDAR → ego → camera
4×4 transforms
process
f-θ projection
pixels + depth
03 · Perception
model
YOLO11x-seg
instance masks
output
ROS 2 bag
PointCloud2
04 · Fusion
process
Median depth inside mask
05 · Output
output
Annotated frames / video
04
Method & key decisions
Calibrate through the ego frame
Camera and LiDAR extrinsics are both given relative to the vehicle, so points go LiDAR → ego → camera as one composed 4×4 transform.
A lens model for 120°
A pinhole model breaks down at the edges of a wide lens. The projection maps the ray angle θ through the dataset's 6-term polynomial, with the affine correction term, before converting to pixels.
Robust range per object
Each mask collects the LiDAR points that land inside it; the median is used so stray background points at mask edges do not dominate.
05
Evaluation
- Dataset
- NVIDIA PhysicalAI Autonomous Vehicles (gated)
- Clips
- City and highway, 605 frames each
- Evaluation
- Qualitative: projection alignment and per-object ranges; no ground-truth range metric
06
Results
LiDAR projectionMasks + range07
Failures, limits & next steps
Timestamp units
Spin timestamps are microseconds; the original bag writer treated them as nanoseconds, so the bag spanned 19.8 ms instead of 19.8 s. Found while packaging and fixed.
The ego hood is segmented as a car
It shows up as CAR ≈ 3.6 m in every frame. The dataset's per-camera ego mask should filter it out; that is the first next step.
Camera/LiDAR sync by index
The camera runs at 30 fps and the LiDAR at 10 Hz; each spin is reused for three frames. Nearest-timestamp matching with motion compensation is the proper fix.
Next steps · what would prove it
- Apply the ego mask.
- Nearest-timestamp sync with ego-motion compensation.
- Score ranges against 3D box annotations.
08
Code & references
DataNVIDIA PhysicalAI-AV is gated and governed by NVIDIA's dataset license. Not affiliated with or endorsed by NVIDIA. The repository is private until the license review is complete.
Related projects
Next: Defect segmentationAutonomous Systems · Deep Learning
Multi-camera BEV perception: Lift-Splat with DINOv2
I built a Lift-Splat bird's-eye-view network with a DINOv2 foundation-model backbone, module by module, and verified its six-camera geometry end to end on real nuScenes calibration.

6 cameraslifted into one 200×200 bird's-eye-view grid
Deep Learning · Autonomous Systems
Pedestrian crossing intent with a vision-language model
I fine-tuned Qwen2-VL-2B with QLoRA to 100% valid JSON, then showed on held-out clips that its intent decision stayed at the majority baseline, traced why, and measured what a VLM does to a real-time tracker.
100%valid JSON after fine-tuning
Computer Vision · Edge Deployment
Infrared small-target detection & tracking
I built a detector and tracker for 3–7 pixel aerial targets in infrared video, then ported it from Python to C++/CUDA/TensorRT: 1.7 ms per frame, 9.3× faster, with tracks identical to the prototype.
1.7 msper 1280×1024 frame, enhance → detect → track
Contact
Want to go deeper on this project? I'm happy to walk through it.
Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.
