Skip to content

Add fmpose3d functionality#3208

Merged
deruyter92 merged 5 commits intomainfrom
feat/add_fmpose3d
Feb 19, 2026
Merged

Add fmpose3d functionality#3208
deruyter92 merged 5 commits intomainfrom
feat/add_fmpose3d

Conversation

@deruyter92
Copy link
Copy Markdown
Collaborator

@deruyter92 deruyter92 commented Feb 12, 2026

Add FMPose3D integration for monocular 3D pose estimation

Add functionality for using https://github.com/AdaptiveMotorControlLab/FMPose3D to lift 2D predictions to 3D keypoints. (see paper)

Summary

  • Integrate FMPose3D into the DeepLabCut model zoo, enabling monocular 3D pose estimation via flow matching for both human (17 H36M joints) and animal (26 Animal3D joints) pipelines
  • Add a convenience wrapper get_fmpose3d_inference_api() that creates a ready-to-use inference object with automatic weight downloading from HuggingFace Hub
  • Add integration tests covering API initialization (lightweight) and full end-to-end 2D-to-3D inference (network-dependent)
  • Add a Jupyter notebook recipe to the docs cookbook showing end-to-end usage, two-stage workflows, and how to lift existing DLC 2D predictions to 3D

Details

Core integration (fmpose3d.py): A thin wrapper that constructs an FMPose3DConfig and returns an FMPose3DInference object. Supports both fmpose3d_humans and fmpose3d_animals model types, optional custom snapshot paths, device selection, and extra config kwargs.

Tests (test_fmpose_integration.py):

  • test_api_init — parametrized over both model types; verifies the API object has the expected interface (prepare_2d, pose_3d, predict). Runs on CPU with no network required.
  • test_prepare_2d_and_pose_3d / test_predict_end_to_end — full integration tests that download weights and run inference on the included example image. Skipped automatically when no network is available.

Recipe notebook (fmpose3d.ipynb): Covers three workflows:

  1. End-to-end human pose estimation (single predict call)
  2. End-to-end animal pose estimation
  3. Two-stage pipeline (separate 2D detection + 3D lifting)
  4. Lifting existing DLC 2D predictions (from analyze_images or analyze_videos) to 3D

@deruyter92 deruyter92 marked this pull request as ready for review February 13, 2026 17:04
Copy link
Copy Markdown
Collaborator

@C-Achard C-Achard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good !


# ---------------------------------------------------------------------------
# Integration: downloads weights and runs inference (needs network)
# ---------------------------------------------------------------------------
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a valid @pytest.mark.example where example implies it is end-to-end or slow etc

@deruyter92 deruyter92 merged commit 3b05a0b into main Feb 19, 2026
11 checks passed
@deruyter92 deruyter92 deleted the feat/add_fmpose3d branch February 19, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants