Infrared small-target detection & tracking
I built a detector and tracker for 3–7 pixel aerial targets in infrared video, then ported it from Python to C++/CUDA/TensorRT: 1.7 ms per frame, 9.3× faster, with tracks identical to the prototype.
- Role
- Design, implementation and evaluation: enhancement, detector, tracker, C++/CUDA port and validation
- Setting
- Research prototype · offline evaluation on recorded infrared sequences
- Stack
- PyTorch
- YOLOv8
- CUDA
- TensorRT
- C++17
- Eigen
- pybind11
- CMake
Key results
- probability of detection
0.979
probability of detection
at 0.036 false alarms / Mpx · 12 held-out sequences, 846 frames
- ID switches
0
ID switches
98% coverage, 1.27 px RMSE · 4 held-out sequences
- GPU pipeline per frame
1.7 ms
GPU pipeline per frame
586 FPS, vs 15.9 ms in Python
- signal-to-clutter gain
×3.5
signal-to-clutter gain
white top-hat vs raw frame
01
Problem & constraints
Infrared search-and-track systems watch the sky for aircraft and drones. At long range a target covers 3–7 pixels of a 1280×1024 frame and is only 10–16 grey levels brighter than cloud, haze and ground clutter.
A detector that misses it loses the track; one that fires on clutter floods the operator with false alarms. Either way the pipeline has to keep up with the camera.
Can a 3–7 pixel target be detected and tracked reliably in cluttered infrared video, fast enough to run on every frame?
Constraints
- Target smaller than the stride of a standard detector's finest feature map (8 px).
- Full-frame processing, no tiling, within a 33 ms budget at 30 fps.
- The production port must reproduce the prototype's outputs, not just run faster.
02
My contribution
Design, implementation and evaluation: enhancement, detector, tracker, C++/CUDA port and validation
- Designed the enhancement stage (dark-channel dehaze + white top-hat) and measured its signal-to-clutter gain.
- Added a stride-4 (P2) head to YOLOv8n and chose the operating point on held-out full frames.
- Built and tuned a constant-acceleration Kalman tracker with χ² gating and Hungarian assignment.
- Ported every stage to C++17/CUDA/TensorRT with pybind11 bindings, then validated it stage by stage against the Python reference.
03
System architecture
01 · Input
data
IR frame
1280×1024 uint8
02 · Enhance · CUDA
process
CudaPreprocessor
dark-channel dehaze, guided filter, percentile stretch, white top-hat → 3-ch tensor
03 · Detect · TensorRT
model
YOLOv8n-P2
FP16 engine, GPU decode + NMS
04 · Track · C++
process
MultiTargetTracker
CA Kalman (Eigen), Mahalanobis gating, Hungarian
05 · Output
output
Tracks + path prediction
system
pybind11 parity tests
vs Python prototype
04
Method & key decisions
Enhancement before learning
The white top-hat keeps only bright features smaller than the structuring element, so clouds and buildings largely vanish. The raw, dehazed and top-hat maps are stacked as the detector's three input channels; mean signal-to-clutter ratio rises from 2.48 (raw) to 8.72 (top-hat).
A detection head at the target's scale
YOLOv8n's finest default head predicts on 8-pixel cells, already larger than the target. A stride-4 P2 head gives 4-pixel cells; the network stays fully convolutional so it runs on the whole frame.
Operating point chosen on held-out frames
At confidence 0.3 the detector finds 97.9% of targets with 0.047 false alarms per frame. The tracker can reject an isolated false alarm but cannot recover a target the detector never reports, so recall was favoured.
Noise model from the sensor, not from tuning
With measurement noise r = 1 px the constant-velocity filter produced up to 35 ID switches per sequence, because the detector's centre jitters by about 2 px. Setting r = 2 px, measured from the detector, gave one track per target. Constant acceleration then cut track error from 1.98 to 1.27 px, but its 30-frame forecast error was 5× larger (56 vs 11 px), so look-ahead prediction should use constant velocity.
05
Evaluation
- Dataset
- NUDT-SIRST-Seq (real infrared sequences)
- Detection test
- 12 held-out sequences, 846 full 1280×1024 frames; hit = centre within 6 px (IRSTD convention)
- Tracking test
- 4 held-out single-target sequences
- Hardware
- NVIDIA RTX 5080 (16 GB), CUDA 13, TensorRT 11
- Timing scope
- Upload → enhance → infer → decode/NMS → track, mean per frame; PNG decoding (8.9 ms) excluded
| Stage | Python prototype | C++/CUDA | Speed-up |
|---|---|---|---|
| Enhancement | 10.91 ms | 0.68 ms | 16× |
| Detection + decode/NMS | 4.83 ms | 0.96 ms | 5× |
| Tracking | 0.137 ms | 0.002 ms | 65× |
| Total | 15.9 ms | 1.7 ms | 9.3× |
| Stage | Check | Result |
|---|---|---|
| Enhancement: raw, top-hat | max |difference| | 0 (bit-identical) |
| Enhancement: dehazed | max |difference| | 1 grey level on 0.38% of pixels |
| Detection | TensorRT FP16 vs PyTorch | 16 / 16 detections identical |
| Tracking | C++ vs Python tracks | agree to 2×10⁻¹³ px |
06
Try it
Detection vs false alarms per frame (log) · measured
- Detected
- 97.9%
- False alarms
- 0.0473 /frame
Chosen operating point: the tracker can reject an isolated false alarm, but it cannot recover a target the detector never reports.
- Real
- Detection rate and false alarms measured at each threshold on 12 held-out sequences (846 full frames).
- Scoring
- A detection counts when its centre is within 6 px of the target (IRSTD convention).
- Runs where
- A lookup over measured results; no inference in the browser.
07
Results

RawTop-hat08
Failures, limits & next steps
Limitations
- Evaluated on single-target sequences; crossings between targets are untested.
- Detection is scored with a 6 px centre-distance rule rather than mask IoU.
- Benchmark excludes image decoding; end-to-end latency from a live sensor is not yet measured.
Next steps · what would prove it
- Multi-target sequences with crossings, reporting ID switches and IDF1.
- Sensor-to-track latency from a live camera stream.
- Profile the same pipeline on a Jetson-class device.
09
Code & references
DataNUDT-SIRST-Seq is not redistributed; frames shown are pipeline outputs for illustration.
Related projects
Next: Cobot fault detectionSensor Fusion · Autonomous Systems
Camera–LiDAR fusion on raw autonomous-vehicle data
I built LiDAR-to-camera projection for a 120° f-θ lens from raw NVIDIA AV files and gave every segmented object a metric range, over two 605-frame drives.
2 × 605frames fused end to end
Computer Vision · Autonomous Systems
Aerial multi-object tracking on VisDrone
I wrote a SORT tracker from scratch, ran it against BoT-SORT with camera-motion compensation on all 17 VisDrone test sequences, and caught an ID-leak bug through the output analysis.
201,931boxes tracked
Computer Vision · Edge Deployment
Real-time surgical tool detection, from dataset to device link
I validated a surgical-tool detector at 0.963 mAP50 over 5 folds, deployed it as a 0.87 ms TensorRT engine behind a Qt panel and a C++ serial relay, and measured an honest cross-dataset drop to 0.17.

0.963mAP50 across 5 folds
Contact
Want to go deeper on this project? I'm happy to walk through it.
Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.
