Pedestrian crossing intent with a vision-language model
I fine-tuned Qwen2-VL-2B with QLoRA to 100% valid JSON, then showed on held-out clips that its intent decision stayed at the majority baseline, traced why, and measured what a VLM does to a real-time tracker.
- Role
- Dataset construction, fine-tuning, held-out evaluation and the asynchronous architecture
- Setting
- Offline evaluation on dashcam clips · real-time architecture benchmarked on recorded video
- Stack
- Qwen2-VL-2B
- QLoRA (4-bit NF4)
- PEFT
- YOLO11s
- PyTorch
- Python threading
Key results
- valid JSON output
56.5% → 100%
valid JSON output
base vs fine-tuned, 131 held-out clips
- intent accuracy
77.1%
intent accuracy
always 'Crossing' scores 78.6%
- tracker p95 latency
20 → 296 ms
tracker p95 latency
VLM idle vs running on the same GPU
01
Problem & constraints
A planner benefits from knowing whether a pedestrian is about to cross, and a structured explanation makes that prediction auditable. Vision-language models can produce both, but they are slow and their accuracy has to be measured honestly against trivial baselines.
Does a small fine-tuned VLM actually learn crossing intent, and can it run beside a real-time tracker without hurting it?
02
My contribution
Dataset construction, fine-tuning, held-out evaluation and the asynchronous architecture
- Built a JAAD clip-and-prompt dataset: 655 clips × 15 frames with context prompts and JSON targets.
- Fine-tuned Qwen2-VL-2B with QLoRA (4-bit NF4, LoRA r = 16, α = 32, 5 epochs).
- Evaluated base vs fine-tuned against a majority baseline on 131 held-out clips and traced the causes of the result.
- Designed the asynchronous tracker + VLM architecture and logged telemetry over all 346 JAAD videos.
03
System architecture
01 · Offline
data
JAAD annotations + 346 videos
data
655 clips · 524 / 131 split
02 · Fine-tune
model
Qwen2-VL-2B QLoRA
NF4, r = 16
03 · Online · fast loop
model
YOLO11s tracker
persist, person class
04 · Online · slow loop
process
Queue (max 1)
model
VLM worker thread
≤ 40 new tokens
05 · Output
output
Logger thread → CSV
04
Evaluation
- Test set
- 131 held-out clips; 101 share a source video with training clips (a video-disjoint split is a next step)
- Baseline
- Always predict 'Crossing' (78.6%)
- Telemetry
- 346 videos, 79,344 frames, VLM and tracker on one RTX 5080
| Base Qwen2-VL-2B | QLoRA fine-tuned | |
|---|---|---|
| Valid JSON output | 56.5% | 100% |
| Accuracy (always 'Crossing' = 78.6%) | 53.4% | 77.1% |
| Balanced accuracy | 52.2% | 51.6% |
| Recall on 'Not Crossing' (28 clips) | 50.0% | 7.1% |
| p50 | p95 | p99 | |
|---|---|---|---|
| VLM idle | 17.9 ms | 20.4 ms | 28.1 ms |
| VLM running on the same GPU | 20.2 ms | 296 ms | 367 ms |
05
Results


06
Failures, limits & next steps
It learned the format, not the decision
The fine-tuned model predicts 'Crossing' for 125 of 131 clips. Causes traced in the notebook: only padding is masked in the loss, so ~1,440 video tokens and the prompt are training targets and the answer is ~5% of the loss; 73% of training clips are 'Crossing'; each clip is 0.5 s seen as 4 frames; the confidence score in the targets is random and the reasoning templated.
Next steps · what would prove it
- Mask the loss to the answer; rebalance or weight the intent token.
- Longer clips (1.5–2 s), predict intent first, take confidence from the token probability.
- Video-disjoint split.
- Give the VLM its own stream priority or device to protect tracker latency.
07
Code & references
- Rasouli, Kotseruba & Tsotsos, Are They Going to Cross?, ICCVW 2017 (JAAD; videos CC BY 4.0).
Related projects
Next: Wildfire smokeSensor Fusion · Autonomous Systems
Camera–LiDAR fusion on raw autonomous-vehicle data
I built LiDAR-to-camera projection for a 120° f-θ lens from raw NVIDIA AV files and gave every segmented object a metric range, over two 605-frame drives.
2 × 605frames fused end to end
Computer Vision · Autonomous Systems
Aerial multi-object tracking on VisDrone
I wrote a SORT tracker from scratch, ran it against BoT-SORT with camera-motion compensation on all 17 VisDrone test sequences, and caught an ID-leak bug through the output analysis.
201,931boxes tracked
Computer Vision · Deep Learning
Zero-shot wildfire smoke segmentation, scored against ground truth
I scored a Grounding DINO + SAM zero-shot pipeline against 744 annotated plumes, raised localisation from 36% to 51% through prompt design, and put the MobileSAM encoder on TensorRT at 4.3 ms.

Contact
Want to go deeper on this project? I'm happy to walk through it.
Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.