Update FMPose3D modelzoo integration#3221
Conversation
- Add fmpose3d_humans and fmpose3d_animals framework mapping to pytorch.
- Branch early in video_inference_superanimal(...) when model_name.startswith("fmpose3d").
- Route directly to _video_inference_fmpose3d(...)
Add _video_inference_fmpose3d(...) as a dedicated path.
Keep batch loop minimal in DLC:
- result_2d = api.prepare_2d(np.stack(frames))
- result_3d = api.pose_3d(result_2d.keypoints, result_2d.image_size)
DLC responsibilities only:
- iterate video frames/chunks,
- convert 2D results to DLC {"bodyparts": ...} layout,
- save .h5, .json, _3d.json, and optional labeled video.
C-Achard
left a comment
There was a problem hiding this comment.
Thanks, looks good overall. I generally agree with your point on separation of concerns, but probably the main concern for now is that it conforms to the modelzoo API
There was a problem hiding this comment.
Pull request overview
This PR integrates FMPose3D monocular 3D pose estimation into the DeepLabCut modelzoo pipeline by creating a dedicated inference path. It relocates FMPose3D functionality from deeplabcut/modelzoo/fmpose_3d to deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d and routes FMPose3D models through the video_inference_superanimal function with early branching. The PR standardizes outputs by writing both 2D predictions and 3D artifacts (*.h5 and *.json files) and provides an optional return flag for in-memory 3D DataFrames.
Changes:
- Relocated FMPose3D wrappers from
modelzoo/fmpose_3dtopose_estimation_pytorch/modelzoo/fmpose_3dwith enhanced inference pipeline - Added early routing in
video_inference_superanimalforfmpose3d_*models with dedicated inference function - Removed preliminary FMPose3D recipe notebook and updated documentation to include usage example in ModelZoo.md
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/pose_estimation_pytorch/modelzoo/test_fmpose_integration.py | Updated imports and added comprehensive unit tests with new pytest markers (unittest/functional) |
| pyproject.toml | Added pytest markers for fmpose3d, unittest, and functional test categories |
| docs/recipes/fmpose3d.ipynb | Deleted preliminary recipe notebook (reverting PR #3208) |
| docs/ModelZoo.md | Added FMPose3D usage example with video_inference_superanimal API |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/inference.py | New dedicated FMPose3D inference pipeline with 2D-to-3D conversion and dataframe generation |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/fmpose3d.py | Relocated FMPose3D API wrapper with model metadata definitions |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/init.py | Added module with refactoring markers for future keypoint/pandas migrations |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/README.md | Added brief FMPose3D overview documentation |
| deeplabcut/modelzoo/video_inference.py | Added early branching logic for fmpose3d models with parameter forwarding |
| deeplabcut/modelzoo/models_to_framework.json | Registered fmpose3d_humans and fmpose3d_animals as pytorch models |
| deeplabcut/modelzoo/fmpose_3d/fmpose3d.py | Deleted old location of FMPose3D wrapper |
| deeplabcut/modelzoo/fmpose_3d/init.py | Deleted old module initialization |
| _toc.yml | Removed fmpose3d recipe from table of contents |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/inference.py
Outdated
Show resolved
Hide resolved
|
< Note: this PR depends on the bleeding-edge code from the FMPose3D repository > Shall I make a new release of FMPose3D on pypi? |
|
Here I disagree:
Because it uses the 2D SA_quadruped data, or human data, for example, then the architecture is the 3D lifting pipeline... why not use this, which better adheres to the SA style? and keep naming consistent:
|
yes that would be great! |
|
~Note to self, @C-Achard, and @MMathisLab ~ Edit: a new version of FMPose3d is now published. The requirements are updated! ✅ |
Note: this PR depends on the bleeding-edge code from the FMPose3D repository. After release, the latest FMPose3D dependency needs to be specified in DeepLabCut before merging.
Summary:
This PR adds a dedicated FMPose3D inference path in the PyTorch modelzoo flow and wires early routing for fmpose3d_* models. It relocates FMPose-specific wrappers/inference helpers to pose_estimation_pytorch/modelzoo/fmpose_3d, standardizes outputs to include explicit 3D artifacts (*_3d.h5 / *_3d.json).
Motivation:
Where the first implementation of FMPose3D in #3208 was rather separated from the rest of the modelzoo pipeline, this PR aims to integrate it better in the existing modelzoo video inference pipeline.
Changes:
Revert preliminary implementation in Add fmpose3d functionality #3208:
Route video inference through video_inference_superanimal:
Add brief example usage in the existing superanimal docs
Example usage:
Additional considerations / limitations:
superanimal_name-> means 'dataset family' andmodel_name-> means 'model architecture'. For FMPose3D the 'dataset family' is different and therefore ignored.