Skip to content

Infrared small-target detection & tracking

I built a detector and tracker for 3–7 pixel aerial targets in infrared video, then ported it from Python to C++/CUDA/TensorRT: 1.7 ms per frame, 9.3× faster, with tracks identical to the prototype.

Role
Design, implementation and evaluation: enhancement, detector, tracker, C++/CUDA port and validation
Setting
Research prototype · offline evaluation on recorded infrared sequences
Stack
  • PyTorch
  • YOLOv8
  • CUDA
  • TensorRT
  • C++17
  • Eigen
  • pybind11
  • CMake
Output of the C++ pipeline on held-out sequence Sample_5. The orange trail is the track; the inset is a 4× zoom on the target.

Key results

probability of detection

0.979

probability of detection

at 0.036 false alarms / Mpx · 12 held-out sequences, 846 frames

ID switches

0

ID switches

98% coverage, 1.27 px RMSE · 4 held-out sequences

GPU pipeline per frame

1.7 ms

GPU pipeline per frame

586 FPS, vs 15.9 ms in Python

signal-to-clutter gain

×3.5

signal-to-clutter gain

white top-hat vs raw frame

01

Problem & constraints

Infrared search-and-track systems watch the sky for aircraft and drones. At long range a target covers 3–7 pixels of a 1280×1024 frame and is only 10–16 grey levels brighter than cloud, haze and ground clutter.

A detector that misses it loses the track; one that fires on clutter floods the operator with false alarms. Either way the pipeline has to keep up with the camera.

Can a 3–7 pixel target be detected and tracked reliably in cluttered infrared video, fast enough to run on every frame?

Constraints

  • Target smaller than the stride of a standard detector's finest feature map (8 px).
  • Full-frame processing, no tiling, within a 33 ms budget at 30 fps.
  • The production port must reproduce the prototype's outputs, not just run faster.

02

My contribution

Design, implementation and evaluation: enhancement, detector, tracker, C++/CUDA port and validation

  • Designed the enhancement stage (dark-channel dehaze + white top-hat) and measured its signal-to-clutter gain.
  • Added a stride-4 (P2) head to YOLOv8n and chose the operating point on held-out full frames.
  • Built and tuned a constant-acceleration Kalman tracker with χ² gating and Hungarian assignment.
  • Ported every stage to C++17/CUDA/TensorRT with pybind11 bindings, then validated it stage by stage against the Python reference.

03

System architecture

  1. 01 · Input

    • data

      IR frame

      1280×1024 uint8

  2. 02 · Enhance · CUDA

    • process

      CudaPreprocessor

      dark-channel dehaze, guided filter, percentile stretch, white top-hat → 3-ch tensor

  3. 03 · Detect · TensorRT

    • model

      YOLOv8n-P2

      FP16 engine, GPU decode + NMS

  4. 04 · Track · C++

    • process

      MultiTargetTracker

      CA Kalman (Eigen), Mahalanobis gating, Hungarian

  5. 05 · Output

    • output

      Tracks + path prediction

    • system

      pybind11 parity tests

      vs Python prototype

Each frame is uploaded once and stays on the GPU until detections come back; the tracker runs on the CPU in 2 µs.

04

Method & key decisions

Enhancement before learning

The white top-hat keeps only bright features smaller than the structuring element, so clouds and buildings largely vanish. The raw, dehazed and top-hat maps are stacked as the detector's three input channels; mean signal-to-clutter ratio rises from 2.48 (raw) to 8.72 (top-hat).

A detection head at the target's scale

YOLOv8n's finest default head predicts on 8-pixel cells, already larger than the target. A stride-4 P2 head gives 4-pixel cells; the network stays fully convolutional so it runs on the whole frame.

Operating point chosen on held-out frames

At confidence 0.3 the detector finds 97.9% of targets with 0.047 false alarms per frame. The tracker can reject an isolated false alarm but cannot recover a target the detector never reports, so recall was favoured.

Noise model from the sensor, not from tuning

With measurement noise r = 1 px the constant-velocity filter produced up to 35 ID switches per sequence, because the detector's centre jitters by about 2 px. Setting r = 2 px, measured from the detector, gave one track per target. Constant acceleration then cut track error from 1.98 to 1.27 px, but its 30-frame forecast error was 5× larger (56 vs 11 px), so look-ahead prediction should use constant velocity.

05

Evaluation

Dataset
NUDT-SIRST-Seq (real infrared sequences)
Detection test
12 held-out sequences, 846 full 1280×1024 frames; hit = centre within 6 px (IRSTD convention)
Tracking test
4 held-out single-target sequences
Hardware
NVIDIA RTX 5080 (16 GB), CUDA 13, TensorRT 11
Timing scope
Upload → enhance → infer → decode/NMS → track, mean per frame; PNG decoding (8.9 ms) excluded
Latency per frame by stage
StagePython prototypeC++/CUDASpeed-up
Enhancement10.91 ms0.68 ms16×
Detection + decode/NMS4.83 ms0.96 ms5×
Tracking0.137 ms0.002 ms65×
Total15.9 ms1.7 ms9.3×
With PNG decoding included the CLI runs at 94 FPS; a live camera delivers raw frames.
Port validation against the Python prototype
StageCheckResult
Enhancement: raw, top-hatmax |difference|0 (bit-identical)
Enhancement: dehazedmax |difference|1 grey level on 0.38% of pixels
DetectionTensorRT FP16 vs PyTorch16 / 16 detections identical
TrackingC++ vs Python tracksagree to 2×10⁻¹³ px

06

Try it

Detection vs false alarms per frame (log) · measured

0.000.250.500.751.000.0010.010.1false alarms per frame
Detected
97.9%
False alarms
0.0473 /frame

Chosen operating point: the tracker can reject an isolated false alarm, but it cannot recover a target the detector never reports.

Real
Detection rate and false alarms measured at each threshold on 12 held-out sequences (846 full frames).
Scoring
A detection counts when its centre is within 6 px of the target (IRSTD convention).
Runs where
A lookup over measured results; no inference in the browser.

07

Results

Raw infrared frame of sky over a city with a red box marking the target region
Open full-size image in a new tab
Raw frame. The red box marks the target; it cannot be picked out at this scale.
Top-hat: Zoomed target patch before and after the top-hat filterRawTop-hat
Same patch before and after enhancement. Drag the divider, or focus it and use the arrow keys.

08

Failures, limits & next steps

Limitations

  • Evaluated on single-target sequences; crossings between targets are untested.
  • Detection is scored with a 6 px centre-distance rule rather than mask IoU.
  • Benchmark excludes image decoding; end-to-end latency from a live sensor is not yet measured.

Next steps · what would prove it

  • Multi-target sequences with crossings, reporting ID switches and IDF1.
  • Sensor-to-track latency from a live camera stream.
  • Profile the same pipeline on a Jetson-class device.

09

Code & references

DataNUDT-SIRST-Seq is not redistributed; frames shown are pipeline outputs for illustration.

Contact

Want to go deeper on this project? I'm happy to walk through it.

Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.