HoloTracker

Inspiration

Three team members came together for the Holoray Challenge at McHacks 26, experimenting with various object tracking approaches to solve real-time medical video tracking problems. The project explores multiple tracking methodologies to find the best fit for medical imaging applications.

What it does

HoloTracker is a real-time object tracking system designed for medical videos, specifically tested on laparoscopic surgery videos. The system provides tools to annotate medical videos with bounding boxes for training data, extracts video metadata including dimensions, frame rate, and duration, and tests multiple tracking algorithms across different branches including OpenCV-based trackers, OSTrack deep learning model, and custom implementations. It enables interactive pause/play annotation with mouse-driven bounding box selection and saves annotations in JSON format for downstream processing.

How we built it

We used Python with OpenCV for video processing and implemented a branch-based experimentation approach where different tracking methods were tested in parallel branches. The modular design includes separate utilities for metadata extraction and annotation. We built an interactive annotation tool with mouse event handlers for real-time bounding box drawing and created a structured data pipeline for handling Laparoscopic medical videos. The project compares classical computer vision trackers from OpenCV against modern deep learning approaches like OSTrack.

Challenges we ran into

Tracking surgical instruments in medical videos proved challenging due to blur, occlusion, and similar colors between instruments and tissue. We had to carefully compare performance across different algorithms including CSRT, KCF, MedianFlow, and deep learning models. Balancing accuracy with speed for real-time tracking applications was difficult. Creating reliable ground truth annotations for medical videos required significant effort. We also found that standard trackers trained on general videos often did not perform well on medical footage without domain adaptation.

Accomplishments that we're proud of

We built a complete annotation pipeline from scratch for medical videos and tested multiple tracking paradigms including classical computer vision and deep learning systematically. The modular codebase allows easy comparison between different trackers. We created a user-friendly annotation tool with an intuitive pause/draw/save workflow and successfully adapted the state-of-the-art OSTrack model for the medical imaging domain. All of this rapid prototyping was accomplished across multiple branches during the hackathon timeframe.

What we learned

Medical imaging presents unique challenges that standard tracking algorithms struggle with, particularly in surgical video contexts. We learned that annotation quality and good ground truth data are crucial for tracker evaluation. There are inherent trade-offs between speed and accuracy, as well as between classical and learning-based approaches. We gained experience with OpenCV's powerful video processing and tracker APIs. Using Git branches effectively for parallel experimentation improved our collaborative development workflow. We also developed domain knowledge about medical video characteristics including instrument appearance and camera motion patterns.

What's next for HoloTracker

We plan to conduct quantitative performance benchmarking to compare all tracker implementations. Adding confidence scoring and reliability metrics for tracking predictions is a priority. Expanding from single to multiple instrument tracking will make the system more practical. We want to optimize for real-time deployment on live surgical video streams and explore AR integration with HoloLens for augmented reality surgical guidance. Fine-tuning deep learning models on larger medical datasets and extracting surgical workflow insights from tracked instrument trajectories are longer-term goals.