Findings from the projects that changed a design decision, each traceable to an executed notebook. No publications yet; these are working notes.
01anomaly detection · evaluation
In-sample alarm limits lie
A PCA detector with 332 components set its own alarm limit so low that it would have flagged every normal test cycle.
An unsupervised detector needs an alarm limit, and the usual recipe is a high percentile of the training scores. With 332 PCA components fitted to 948 cycles, the model reconstructs its own training cycles almost perfectly, so their reconstruction error is tiny.
The 99th percentile of in-sample error was 43.8. That limit would flag 100% of unseen normal cycles. Scoring each training cycle with a PCA fitted on the other four fifths gave a limit of 185.5 and a 1.7% false-alarm rate, close to the 1% design target.
The lesson generalises: any threshold derived from scores a model produced on its own training data inherits that model's overfitting.
100% valid JSON, and an intent accuracy one point below always guessing 'Crossing'.
After QLoRA fine-tuning, Qwen2-VL-2B produced valid JSON on every held-out clip, up from 56.5%. Its accuracy of 77.1% looked respectable until it was compared with the majority class: always predicting 'Crossing' scores 78.6%.
The model predicted 'Crossing' for 125 of 131 clips. The training collator masked only padding, so about 1,440 video placeholder tokens and the prompt were training targets and the answer was roughly 5% of the loss. Add a 73/27 class imbalance and 0.5-second clips, and the cheapest way to lower the loss is to copy the format and the majority label.
Report a trivial baseline next to every accuracy, and check what the loss is actually optimising.
Set the filter's noise from the detector, not by tuning
One parameter, measured instead of guessed, took a tracker from 35 ID switches to zero.
A Kalman filter's measurement noise says how much to trust each detection. With r = 1 px, the filter treated the detector's normal 2-pixel centre jitter as a different object and split one target into dozens of tracks.
Measuring the detector's jitter and setting r = 2 px gave one unbroken track per target. A constant-acceleration model then tightened the track to 1.27 px RMSE, but its 30-frame forecast error grew 5× because it extrapolated a noisy acceleration on near-straight flight. The motion model has to match how the target moves.
Same settings, different background ratio: mask mAP50 from 0.25 to 0.41. A 2.2× larger model at 4× the pixels: no gain.
Two VisA runs with identical hyper-parameters differed only in how many defect-free background images they kept. Capping backgrounds at 10% of the defect count removed a 20-epoch stall at zero and lifted mask mAP50 by 16 points.
On MVTec AD, moving from YOLO11l-seg at 640 px to YOLO11x-seg at 1024 px with evolved hyper-parameters changed mask mAP50 from 0.819 to 0.823, within noise. When a model plateaus, inspect what it is being shown before scaling it.
A port is only faster if it gives the same answers
How the infrared pipeline's C++/CUDA port was checked against its Python prototype, stage by stage.
The C++ modules were exposed to Python through pybind11, so each stage could be fed the same input as the prototype and compared directly.
The raw and top-hat channels came out bit-identical; the dehazed channel differed by at most one grey level on 0.38% of pixels. The FP16 TensorRT engine matched PyTorch on 16 of 16 detections, and C++ and Python tracks agreed to 2×10⁻¹³ px. Only then does the 9.3× speed-up mean anything.
A cross-dataset test is the cheapest honest number a detector can report.
Five-fold cross-validation put the surgical-tool detector at 0.963 mAP50. The same model, unchanged, scored 0.167 on 2,811 laparoscopic frames from a different procedure.
Both numbers are true. Only the second tells you what happens when the model meets a new operating room, and it defines the next experiment: mixed-domain training, tested on a third procedure.
Every latency on this site, with how it was measured
Scopes differ (full pipeline vs encoder only vs model inference), so compare within a row, not across rows. GPU measurements: one RTX 5080, TensorRT 11.