Skip to content

Pedestrian crossing intent with a vision-language model

I fine-tuned Qwen2-VL-2B with QLoRA to 100% valid JSON, then showed on held-out clips that its intent decision stayed at the majority baseline, traced why, and measured what a VLM does to a real-time tracker.

Role
Dataset construction, fine-tuning, held-out evaluation and the asynchronous architecture
Setting
Offline evaluation on dashcam clips · real-time architecture benchmarked on recorded video
Stack
  • Qwen2-VL-2B
  • QLoRA (4-bit NF4)
  • PEFT
  • YOLO11s
  • PyTorch
  • Python threading
YOLO11s tracking is the fast loop; the VLM runs in a background thread when a new pedestrian appears (status: COMPUTING).

Key results

valid JSON output

56.5% → 100%

valid JSON output

base vs fine-tuned, 131 held-out clips

intent accuracy

77.1%

intent accuracy

always 'Crossing' scores 78.6%

tracker p95 latency

20 → 296 ms

tracker p95 latency

VLM idle vs running on the same GPU

01

Problem & constraints

A planner benefits from knowing whether a pedestrian is about to cross, and a structured explanation makes that prediction auditable. Vision-language models can produce both, but they are slow and their accuracy has to be measured honestly against trivial baselines.

Does a small fine-tuned VLM actually learn crossing intent, and can it run beside a real-time tracker without hurting it?

02

My contribution

Dataset construction, fine-tuning, held-out evaluation and the asynchronous architecture

  • Built a JAAD clip-and-prompt dataset: 655 clips × 15 frames with context prompts and JSON targets.
  • Fine-tuned Qwen2-VL-2B with QLoRA (4-bit NF4, LoRA r = 16, α = 32, 5 epochs).
  • Evaluated base vs fine-tuned against a majority baseline on 131 held-out clips and traced the causes of the result.
  • Designed the asynchronous tracker + VLM architecture and logged telemetry over all 346 JAAD videos.

03

System architecture

  1. 01 · Offline

    • data

      JAAD annotations + 346 videos

    • data

      655 clips · 524 / 131 split

  2. 02 · Fine-tune

    • model

      Qwen2-VL-2B QLoRA

      NF4, r = 16

  3. 03 · Online · fast loop

    • model

      YOLO11s tracker

      persist, person class

  4. 04 · Online · slow loop

    • process

      Queue (max 1)

    • model

      VLM worker thread

      ≤ 40 new tokens

  5. 05 · Output

    • output

      Logger thread → CSV

04

Evaluation

Test set
131 held-out clips; 101 share a source video with training clips (a video-disjoint split is a next step)
Baseline
Always predict 'Crossing' (78.6%)
Telemetry
346 videos, 79,344 frames, VLM and tracker on one RTX 5080
Held-out test set (131 clips)
Base Qwen2-VL-2BQLoRA fine-tuned
Valid JSON output56.5%100%
Accuracy (always 'Crossing' = 78.6%)53.4%77.1%
Balanced accuracy52.2%51.6%
Recall on 'Not Crossing' (28 clips)50.0%7.1%
Tracker loop latency
p50p95p99
VLM idle17.9 ms20.4 ms28.1 ms
VLM running on the same GPU20.2 ms296 ms367 ms
The VLM (p50 824 ms per call) never blocks the tracker, but sharing the GPU inflates the tracker's tail latency about 14×.

05

Results

Held-out clip: model correctly predicts Not Crossing, with its structured output
Open full-size image in a new tab
Correct 'Not Crossing': one of only two correct answers for that class.
Held-out clip: model wrongly predicts Crossing
Open full-size image in a new tab
Typical error: the truth is 'Not Crossing', the model says 'Crossing' by default.

06

Failures, limits & next steps

It learned the format, not the decision

The fine-tuned model predicts 'Crossing' for 125 of 131 clips. Causes traced in the notebook: only padding is masked in the loss, so ~1,440 video tokens and the prompt are training targets and the answer is ~5% of the loss; 73% of training clips are 'Crossing'; each clip is 0.5 s seen as 4 frames; the confidence score in the targets is random and the reasoning templated.

Next steps · what would prove it

  • Mask the loss to the answer; rebalance or weight the intent token.
  • Longer clips (1.5–2 s), predict intent first, take confidence from the token probability.
  • Video-disjoint split.
  • Give the VLM its own stream priority or device to protect tracker latency.

07

Code & references

  • Rasouli, Kotseruba & Tsotsos, Are They Going to Cross?, ICCVW 2017 (JAAD; videos CC BY 4.0).
Next: Wildfire smoke

Contact

Want to go deeper on this project? I'm happy to walk through it.

Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.