Skip to content

hotfix extract_outlier_frames for cropped videos #3169

Merged
MMathisLab merged 6 commits intoDeepLabCut:mainfrom
deruyter92:jaap/hotfix_cropping_extract_outlier_frames
Jan 22, 2026
Merged

hotfix extract_outlier_frames for cropped videos #3169
MMathisLab merged 6 commits intoDeepLabCut:mainfrom
deruyter92:jaap/hotfix_cropping_extract_outlier_frames

Conversation

@deruyter92
Copy link
Collaborator

This PR fixes a bug that introduces a mismatch between label-coordinates and video, when extracting cropped outlier frames.

bug description
When using cropping, the predicted keypoints produced with analyze_videos are stored in crop-window coordinates. This is also reflected in the metadata.pickle. In the current implementation of extract_outlier_frames, key-point coordinates are ALWAYS converted back to full-frame coordinates by adding the crop margin read from the metadata pickle file. (introduced in PR #2538)

This should not always the intended behavior: e.g. when the user tries to extract cropped output frames for a next refinement iteration, the keypoints should also reflect cropped coordinates.

See this imageforum issue and potentially related github issues: #3060, #3130, #1979

Steps to reproduce:

  1. Create a new project with labeled data and train a network.
  2. Run analyze_video with cropping enabled, e.g. :
deeplabcut.analyze_videos(
    config_path,
    [video_path],
    shuffle=shuffle,
    trainingsetindex=trainingsetindex,
    cropping=[908, 1920, 0, 1080],
)
  1. Add the analyzed video to the video sets in config.yaml and specify cropping coordinates there.
video_sets: 
   path/to/video.mp4:
      crop: 908, 1920, 0, 1080
  1. Run extract_outlier_frames:
deeplabcut.extract_outlier_frames(
    config_path,
    [video_path],
    shuffle=shuffle,
    trainingsetindex=trainingsetindex,
    automatic=True,
)
  1. In Napari, open the created image folder inside labeled-frames to check the labels.

Omitting step 3:
image

Including step3:
image

After accepting proposed code change in this PR:
image

changed behavior:
extract_outlier_frames reads the fields video_sets in config and checks if cropping is configured. The keypoints are converted by the difference between margins in metadata.pickle and the output crop margin specified in video_sets, instead of always adding the margins specified in the metadata.pickle.

…re cropped in `extract_outlier_frames`.

This commit fixes a bug that introduces a mismatch between label-coordinates and video, when extracting cropped outlier frames.

When the `analyze_videos` step was performed using a cropping window. The predicted keypoints are stored in the crop-window space. In the previous implementation, the keypoints are ALWAYS converted back to reflect full-video coordinates by adding the left and top crop margins (introduced in PR DeepLabCut#2538). However the keypoints should only be converted back if the output frames are full-video, not when they are cropped. This commit fixes that by using the difference between the output margin and the margin stored in metadata.
@deruyter92 deruyter92 changed the title Hotfix extract_outlier_frames for cropped videos hotfix extract_outlier_frames for cropped videos Jan 8, 2026
@MMathisLab MMathisLab requested a review from Copilot January 9, 2026 14:46
@MMathisLab MMathisLab added the bug fix! fix for a real buggy one... label Jan 9, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug where extract_outlier_frames incorrectly converts keypoint coordinates to full-frame coordinates when extracting cropped outlier frames. The fix adjusts keypoints by the difference between the analysis crop margins (from metadata) and output crop margins (from config's video_sets), allowing users to extract cropped frames with correctly aligned labels.

Key changes:

  • Added helper function _read_video_specific_cropping_margins to read output crop configuration from video_sets
  • Modified coordinate transformation logic to account for the difference between analysis and output crop margins
  • Added comment explaining the shifted coordinate behavior

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@MMathisLab MMathisLab requested a review from C-Achard January 13, 2026 17:34
@C-Achard
Copy link
Collaborator

Copy link
Collaborator

@C-Achard C-Achard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I suspect this will not be a problem on the napari-DLC side for sure. Let's see how tests go; otherwise all good imo ! Thanks Jaap

@deruyter92 deruyter92 requested a review from MMathisLab January 15, 2026 12:38
Copy link
Member

@MMathisLab MMathisLab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@MMathisLab MMathisLab merged commit d6c0bb4 into DeepLabCut:main Jan 22, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug fix! fix for a real buggy one...

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants