hotfix extract_outlier_frames for cropped videos #3169
hotfix extract_outlier_frames for cropped videos #3169MMathisLab merged 6 commits intoDeepLabCut:mainfrom
Conversation
…re cropped in `extract_outlier_frames`. This commit fixes a bug that introduces a mismatch between label-coordinates and video, when extracting cropped outlier frames. When the `analyze_videos` step was performed using a cropping window. The predicted keypoints are stored in the crop-window space. In the previous implementation, the keypoints are ALWAYS converted back to reflect full-video coordinates by adding the left and top crop margins (introduced in PR DeepLabCut#2538). However the keypoints should only be converted back if the output frames are full-video, not when they are cropped. This commit fixes that by using the difference between the output margin and the margin stored in metadata.
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug where extract_outlier_frames incorrectly converts keypoint coordinates to full-frame coordinates when extracting cropped outlier frames. The fix adjusts keypoints by the difference between the analysis crop margins (from metadata) and output crop margins (from config's video_sets), allowing users to extract cropped frames with correctly aligned labels.
Key changes:
- Added helper function
_read_video_specific_cropping_marginsto read output crop configuration from video_sets - Modified coordinate transformation logic to account for the difference between analysis and output crop margins
- Added comment explaining the shifted coordinate behavior
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
|
May be related to : |
C-Achard
left a comment
There was a problem hiding this comment.
Looks good, I suspect this will not be a problem on the napari-DLC side for sure. Let's see how tests go; otherwise all good imo ! Thanks Jaap
This PR fixes a bug that introduces a mismatch between label-coordinates and video, when extracting cropped outlier frames.
bug description
When using cropping, the predicted keypoints produced with
analyze_videosare stored in crop-window coordinates. This is also reflected in the metadata.pickle. In the current implementation ofextract_outlier_frames, key-point coordinates are ALWAYS converted back to full-frame coordinates by adding the crop margin read from the metadata pickle file. (introduced in PR #2538)This should not always the intended behavior: e.g. when the user tries to extract cropped output frames for a next refinement iteration, the keypoints should also reflect cropped coordinates.
See this imageforum issue and potentially related github issues: #3060, #3130, #1979
Steps to reproduce:
analyze_videowith cropping enabled, e.g. :config.yamland specify cropping coordinates there.extract_outlier_frames:labeled-framesto check the labels.Omitting step 3:

Including step3:

After accepting proposed code change in this PR:

changed behavior:
extract_outlier_framesreads the fieldsvideo_setsin config and checks if cropping is configured. The keypoints are converted by the difference between margins in metadata.pickle and the output crop margin specified invideo_sets, instead of always adding the margins specified in the metadata.pickle.