This repository contains the official unified codebase for our work on Multi-Spectral Gaussian Splatting based on the papers Towards Integrating Multi-Spectral Imaging with Gaussian Splatting and Multi-Spectral Gaussian Splatting with Neural Color Representation.

📌 The repo supports two papers, which share the same implementation. The code will be released soon™ in this repository.

Josef Grün, Lukas Meyer, Maximilian Weiherer, Bernhard Egger, Marc Stamminger, Linus Franke

Lukas Meyer, Josef Grün, Maximilian Weiherer, Bernhard Egger, Marc Stamminger, Linus Franke

• 6. October 2025 – Code for Towards Multi-Spectral GS released! 🥳
• 1. October 2025 – Towards Multi-Spectral GS got accepted by VMV
• 31. August 2025 – Towards Multi-Spectral GS uploaded to arXiv
• 3. June 2025 – MS-Splatting uploaded to arXiv

If you use this repository, please cite the corresponding paper(s):

@inproceedings{gruen2025towards_msplatting,
  title     = {Towards Integrating Multi-Spectral Imaging with Gaussian Splatting},
  author    = {Grün, Josef and Meyer, Lukas and Weiherer, Maximilian and Egger, Bernhard and Stamminger, Marc and Franke, Linus},
  booktitle = {Proceedings of VMV 2025},
  year      = {2025},
  month     = {June},
  url       = {https://meyerls.github.io/towards_multi_spec_splat}
}

@article{meyer2025msplatting,
  title   = {Multi-Spectral Gaussian Splatting with Neural Color Representation},
  author  = {Meyer, Lukas and Grün, Josef and Weiherer, Maximilian and Egger, Bernhard and Stamminger, Marc and Franke, Linus},
  journal = {arXiv preprint arXiv:2508.14443},
  year    = {2025},
  url     = {https://arxiv.org/abs/2508.14443}
}

The following prerequisites are needed:

• python >= 3.8 (tested with 3.8.20)
• nerfstudio (tested with 1.1.5, see pyproject.toml and https://docs.nerf.studio/quickstart/installation.html)
• gsplat (tested with 1.4.0, see pyproject.toml)

Then download and install the MS-Splatting repository:

git clone https://github.com/j-gruen/MS-Splatting.git
cd MS-Splatting
pip install -e .
ns-install-cli

To prepare a multi-spectral dataset for training, run:

ns-process-mm-data images --data <DATA_PATH> --output-dir <OUTPUT_PATH>

This command uses colmap (has to be installed) to generate camera poses for all images. <DATA_PATH> must point to a directory containing the dataset images sorted into sub-folders for every multi-spectral image channel, for example:

data
├─ RGB
│  ├─ image1.png
│  ├─ image2.png
│  └─ ...
├─ MS_NIR
│  ├─ image1.tiff
│  ├─ image2.tiff
│  └─ ...
└─ ...

For more information, run ns-process-mm-data images --help.

The output of ns-process-mm-data adheres to the standard nerfstudio transforms.json, with the addition of the mm_channel property for every frame in the dataset, and an optional sparse_pc.ply pointcloud for Gaussian initialization. For more information and examples look at mmsplat_dataset.py and our datasets.

To train a multi-spectral model, ns-train can be used. Example:

ns-train mmsplat                 \
  --data <DATASET_PATH>        \
    --output-dir <OUTPUT_PATH>   \
    --max-num-iterations 120000  \
    mmsplat-dataparser           \
    --downscale-factor 0

For more information and available parameters, run ns-train mmsplat --help or look at the configurations in mmsplat_model.py, mmsplat_datamanager.py and mmsplat_dataparser.py.

To generate evaluation metrics and images, run

ns-eval
  --load-config <PATH/TO/config.yml>
  --output-path <OUTPUT_PATH/eval.json>
  --render-output-path <OUTPUT_IMAGES_PATH>

where --load-config points to the config.yml generated by ns-train.

Note that the default parameters of ns-train are not best out-of-the-box. Here is an example of the parameters used in the Joint-Optimized strategy from our VMV 2025 Paper:

ns-train mmsplat                                                               \
  --data <DATASET_PATH>                                                        \
  --output-dir <OUTPUT_PATH>                                                   \
  --max-num-iterations 120000                                                  \
  --pipeline.model.refine-every 300                                            \
  --pipeline.model.densification-strategy max_average                          \
  --pipeline.model.stop-split-at 60000                                         \
  --pipeline.model.pos-optim-delay-channels D 500 MS_G MS_R MS_RE MS_NIR 32000 \
  --pipeline.model.opacity-optim-delay-channels MS_G MS_R MS_RE MS_NIR 32000   \
  --pipeline.datamanager.delay-channels MS_G MS_R MS_RE MS_NIR 30000           \
  --pipeline.datamanager.channel-size D 3 MS_G MS_R MS_RE MS_NIR 1             \
  --pipeline.model.use-sh-channels D MS_G MS_R MS_RE MS_NIR                    \
  --pipeline.model.sh-degree 3                                                 \
  --pipeline.model.densification-pause-iterations 29000 32001                  \
  mmsplat-dataparser                                                           \
  --downscale-factor 0                                                         \
  --eval-mode json-list

A few more utilities like rendering and exporting are available. See the [project.scripts] section in pyproject.toml for more info.