Skip to content

Zero-shot wildfire smoke segmentation, scored against ground truth

I scored a Grounding DINO + SAM zero-shot pipeline against 744 annotated plumes, raised localisation from 36% to 51% through prompt design, and put the MobileSAM encoder on TensorRT at 4.3 ms.

Role
Pipeline, ground-truth scoring, prompt experiments and TensorRT export
Setting
Offline evaluation on still images from many cameras · encoder benchmarked on a workstation GPU
Stack
  • Grounding DINO
  • Segment Anything
  • MobileSAM
  • TensorRT
  • ONNX
  • PyTorch
Prompt-v2 output with masks and boxes, annotated plume in white
Open full-size image in a new tab
Prompt-v2 output (masks and boxes) with the annotated plume in white.

Key results

plumes localised

50.8%

plumes localised

IoU ≥ 0.5, 744 images

AP50

0.304

AP50

with cross-phrase NMS

MobileSAM encoder on TensorRT

4.31 ms

MobileSAM encoder on TensorRT

1024², mean over 698 runs, p99 4.38 ms

01

Problem & constraints

Early smoke detection matters most when plumes are small and faint, and labelled data for every camera and season does not exist. Foundation models promise text-prompted segmentation with no training; their claims need checking against ground truth.

02

My contribution

Pipeline, ground-truth scoring, prompt experiments and TensorRT export

  • Built the text → boxes → masks pipeline (Grounding DINO Swin-T → SAM ViT-H).
  • Scored detections against the 744 Pascal-VOC smoke boxes: recall, hit@IoU, precision, AP50.
  • Ran the prompt experiments and the cross-phrase NMS fix.
  • Exported the MobileSAM encoder to ONNX/TensorRT and timed it.
  • Removed an unsupported claim from the original demo video (see failures).

03

System architecture

  1. 01 · Input

    • data

      Lookout-camera frame

    • data

      Text prompt

      smoke + synonyms

  2. 02 · Ground

    • model

      Grounding DINO Swin-T

      box 0.25 / text 0.20

  3. 03 · Segment

    • model

      SAM ViT-H

      box prompts → masks

  4. 04 · Score

    • process

      vs GT boxes

      recall · precision · AP50

  5. 05 · Edge path

    • system

      MobileSAM → TensorRT

      4.31 ms / frame

04

Evaluation

Dataset
AI For Mankind HPWREN smoke boxes: 744 images, one annotated plume each
Hit
Smoke box with IoU ≥ 0.5 against the annotated plume
Timing
MobileSAM image encoder only, FP32 graph, TensorRT, RTX 5080
Prompt experiments (744 images)
Prompt v1Prompt v2 (+ synonyms)v2 + cross-phrase NMS
Images with any smoke detection59.3%77.7%77.7%
Plume localised (IoU ≥ 0.5)35.5%50.8%50.7%
Precision of smoke boxes57.4%24.2%47.3%
AP500.2550.2670.304
Synonyms find 15 points more plumes but box the same plume repeatedly; cross-phrase NMS recovers most of the precision.

05

Results

06

Failures, limits & next steps

Faint plumes

A faint plume is missed and the sky is boxed as 'haze' instead.

Tiny, distant plumes

Small plumes far from the camera are missed.

Whole-sky boxes

A box covering the entire sky swallows the plume.

A demo claim that did not hold

The original video drew a smoke 'trajectory' and overlaid '232 FPS'. The 744 images come from many cameras and days, so consecutive frames are unrelated, and 232 FPS was the MobileSAM encoder alone. The video is no longer presented as a result.

Next steps · what would prove it

  • A real HPWREN time-lapse sequence for temporal tracking.
  • End-to-end TensorRT pipeline including the grounding model.
  • Per-distance recall to quantify small-plume performance.

07

Code & references

  • Liu et al., Grounding DINO, 2023.
  • Kirillov et al., Segment Anything, 2023.
  • Zhang et al., MobileSAM, 2023.

DataSmoke annotations: AI For Mankind on public-domain HPWREN images, CC BY-NC-SA 4.0.

Next: BEV perception

Contact

Want to go deeper on this project? I'm happy to walk through it.

Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.