Skip to content

Fix RF-DETR-Seg mask postprocessing for letterboxed input case#2001

Merged
mkaic merged 4 commits intomainfrom
fix/letterbox-postprocessing
Feb 13, 2026
Merged

Fix RF-DETR-Seg mask postprocessing for letterboxed input case#2001
mkaic merged 4 commits intomainfrom
fix/letterbox-postprocessing

Conversation

@mkaic
Copy link
Contributor

@mkaic mkaic commented Feb 13, 2026

What does this PR do?

Previously, box postprocessing worked properly with letterboxed inputs, but mask postprocessing wasn't properly accounting for / cropping out the black bars introduced by letterboxing. Now it does. The logic used is similar to what exists like 20 lines above for the un-letterboxing of the box predictions.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Testing

  • I have tested this change locally
  • I have added/updated tests for this change

Test details:
This turned out to be the correct fix (or, well, the first part of an even more correct fix) for what was addressed by this roboflow-train PR. My test case was the same — I saw squished masks before the change, and now the masks aren't squished anymore:

evaluated on main branch of inference:
letterbox_bug_annotated_masks_old_inference

evaluated on this branch of inference:
letterbox_bug_annotated_masks_old_inference

I also decided to check a dataset where the black bars are on the top and bottom instead of the sides, just to be sure.

evaluated on main branch of inference:
letterbox_bug_annotated_masks_old_inference

evaluated on this branch of inference:
letterbox_bug_annotated_masks_old_inference

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in hard-to-understand areas
  • My changes generate no new warnings or errors
  • I have updated the documentation accordingly (if applicable)

Additional Context

N/A

@mkaic mkaic marked this pull request as ready for review February 13, 2026 06:10
@mkaic mkaic enabled auto-merge (squash) February 13, 2026 16:37
@mkaic mkaic merged commit d8dfd0d into main Feb 13, 2026
51 of 52 checks passed
@mkaic mkaic deleted the fix/letterbox-postprocessing branch February 13, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants