Official repository for Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation paper 📄 selected for an Oral presentation at CVPR 2022.

We propose to exploit the synergy between images and 3D point clouds by learning to select the most relevant views for each point. Our approach uses the viewing conditions of 3D points to merge features from images taken at arbitrary positions. We reach SOTA results for S3DIS (74.7 mIoU 6-Fold) and on KITTI- 360 (58.3 mIoU) without requiring point colorization, meshing, or the use of depth cameras: our full pipeline only requires raw 3D scans and a set of images and poses.

• notebooks for manipulating multimodal data for S3DIS, ScanNet and KITTI-360, training and testing models and reproducing our papers' main results.
• pretrained weights from our best-performing model on S3DIS and KITTI-360
• wandb logs of our experiments

The following must be installed before installing this project.

• Anaconda3
• cuda >= 10.1
• gcc >= 7

All remaining dependencies (PyTorch, PyTorch Geometric, etc) should be installed using the provided installation script.

The code has been tested in the following environment:

• Ubuntu 18.04.6 LTS
• Python 3.8.5
• PyTorch 1.7.1
• CUDA 10.2, 11.2 and 11.4

To install DeepViewAgg, simply run ./install.sh from inside the repository.

• You will need to have sudo rights to install MinkowskiEngine and TorchSparse dependencies.
• ⚠️ Do not install Torch-Points3D from the official repository, or with pip.

This is not the official Torch-Points3D framework. This work builds on and modifies a fixed version of the framework and has not been merged with the official repository yet. In particular, this repository introduces numerous features for multimodal learning on large-scale 3D point clouds. In this repository, some TP3D-specific files were removed for simplicity.

The project follows the original Torch-Points3D framework structure.

├─ conf                    # All configurations live there
├─ notebooks               # Notebooks to get started with multimodal datasets and models
├─ eval.py                 # Eval script
├─ insall.sh               # Installation script for DeepViewAgg
├─ scripts                 # Some scripts to help manage the project
├─ torch_points3d
    ├─ core                # Core components
    ├─ datasets            # All code related to datasets
    ├─ metrics             # All metrics and trackers
    ├─ models              # All models
    ├─ modules             # Basic modules that can be used in a modular way
    ├─ utils               # Various utils
    └─ visualization       # Visualization
└─ train.py                # Main script to launch a training

Several changes were made to extend the original project to multimodal learning on point clouds with images. The most important ones can be found in the following:

• conf/data/segmentation/multimodal: configs for the 3D+2D datasets.
• conf/models/segmentation/multimodal: configs for the 3D+2D models.
• torch_points3d/core/data_transform/multimodal: transforms for 3D+2D data.
• torch_points3d/core/multimodal: multimodal data and mapping objects.
• torch_points3d/datasets/segmentation/multimodal: 3D+2D datasets (eg S3DIS, ScanNet, KITTI360).
• torch_points3d/models/segmentation/multimodal: 3D+2D architectures.
• torch_points3d/modules/multimodal: 3D+2D modules. This is where the DeepViewAgg module can be found.
• torch_points3d/visualization/multimodal_data.py: tools for interactive visualization of multimodal data.

Notebooks available very soon 🚨 🚧

The official documentation of Pytorch Geometric and Torch-Points3D are good starting points, since this project largely builds on top of these frameworks. For DeepViewAgg-specific features (i.e. all that concerns multimodal learning), the provided code is commented as much as possible, but hit me up 💬 if some parts need clarification.

We provide code to produce interactive and sharable HTML visualizations of multimodal data and point-image mappings:

Examples of such HTML produced on S3DIS Fold 5 are zipped here and can be opened in your browser.

• This implementation of DeepViewAgg largely relies on the Torch-Points3D framework, although not merged with the official project at this point.
• For datasets, some code from the official KITTI-360 and ScanNet repositories was used.

In case you use all or part of the present code, please include a citation to the following paper:

@inproceedings{robert2022dva,
  title={Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation},
  author={Robert, Damien and Vallet, Bruno and Landrieu, Loic},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022},
  url = {\url{https://github.com/drprojects/DeepViewAgg}}
}