Skip to content

Real-time surgical tool detection, from dataset to device link

I validated a surgical-tool detector at 0.963 mAP50 over 5 folds, deployed it as a 0.87 ms TensorRT engine behind a Qt panel and a C++ serial relay, and measured an honest cross-dataset drop to 0.17.

Role
Training, validation, deployment and system integration
Setting
Research prototype · simulated microcontroller over a virtual serial link (no physical device)
Stack
  • YOLO11n
  • TensorRT
  • ONNX
  • PySide6
  • C++17
  • POSIX termios
  • socat
Held-out endoscopic frame: ground truth in white, YOLO11n prediction in orange
Open full-size image in a new tab
Held-out validation frame. White: ground truth. Orange: prediction at the GUI's 0.4 threshold.

Key results

mAP50 on held-out split

0.937

mAP50 on held-out split

YOLO11n, 60 epochs, 640 px

5-fold mean mAP50

0.963

5-fold mean mAP50

± 0.034

TensorRT mean latency

0.87 ms

TensorRT mean latency

p99 1.03 ms, 3,451 runs, 640×640

cross-dataset mAP50

0.167

cross-dataset mAP50

laparoscopic m2cai16, no retraining

01

Problem & constraints

A tool detector in an endoscopy suite has to run in real time and pass its state to other equipment. Its in-domain accuracy means little if it collapses on a different procedure.

How accurate and how fast is a small detector on endoscopic video, and does that accuracy transfer to a new surgical domain?

02

My contribution

Training, validation, deployment and system integration

  • Converted Kvasir-Instrument to YOLO format and ran 5-fold cross-validation.
  • Compared YOLO11n against a YOLOv8n baseline.
  • Ran an unchanged cross-dataset test on 2,811 laparoscopic frames.
  • Exported to ONNX/TensorRT and profiled latency over 3,451 runs.
  • Built the PySide6 panel and a C++17 termios relay to a simulated microcontroller; wrote requirements and a test plan.

03

System architecture

  1. 01 · Data

    • data

      Kvasir-Instrument

      590 frames → 502 / 88

  2. 02 · Model

    • model

      YOLO11n

      60 epochs, 640 px

    • process

      5-fold CV

  3. 03 · Deploy

    • process

      ONNX → TensorRT

    • output

      PySide6 live panel

  4. 04 · Device link

    • system

      C++ status relay

      termios 115200 8N1

  5. 05 · Device

    • system

      Simulated MCU

      socat virtual UART

04

Evaluation

In-domain
Kvasir-Instrument, 88-image held-out split + 5-fold CV
Cross-domain
m2cai16-tool-locations: 2,811 frames, 3,929 boxes, 7 tool classes merged into one
Latency
trtexec, FP32 engine (TensorRT 11 strongly typed), 640×640, RTX 5080
mAP50 per fold, and on an unseen procedure
mAP50 per fold, and on an unseen procedure
Fold 10.993
Fold 20.903
Fold 30.979
Fold 40.989
Fold 50.948
m2cai16 (laparoscopic)0.167

The in-domain score does not transfer to a new procedure; mixed-domain training is the next experiment.

Detection quality
ModelPrecisionRecallmAP50mAP50-95
YOLO11n (final)0.9740.8790.9370.804
YOLOv8n (baseline)0.9120.9230.9290.818
YOLO11n, 5-fold mean ± std0.966 ± 0.0290.932 ± 0.0510.963 ± 0.0340.856 ± 0.039

05

Results

Held-out frame: tool found
Open full-size image in a new tab
In-domain: tool found.
Laparoscopic frame: one of two instruments found
Open full-size image in a new tab
Cross-dataset: one of two laparoscopic instruments found.
Laparoscopic frame: both instruments missed
Open full-size image in a new tab
Cross-dataset: both instruments missed.

06

Failures, limits & next steps

Missed guidewire

A thin guidewire in an over-exposed view gets no detection at all.

Extra box on the lumen

The tool is found at 0.92, but a second box at 0.11 lands on the dark lumen.

Limitations

  • Performance is in-domain only (cross-dataset mAP50 0.17).
  • FP32 engine; the GUI still serves the PyTorch model.
  • Camera-to-UART latency and the event log requirement are open in the test plan.

Next steps · what would prove it

  • Train on Kvasir + m2cai16, test on a third procedure.
  • FP16/INT8 engine served from the GUI.
  • Measure camera → UART byte latency.

07

Code & references

  • Jha et al., Kvasir-Instrument, MMM 2021.
  • Jin et al., m2cai16-tool-locations, WACV 2018.
Next: Aerial tracking

Contact

Want to go deeper on this project? I'm happy to walk through it.

Open to full-time engineering roles. If the problem is hard and the data is messy, I'm interested.