This repository provides a ROS 2-based framework for semantic navigation using visual-language models.
It integrates segmentation models like LSeg and FC-CLIP within a Dockerized environment for reproducibility and ease of setup.

• GPU with support for nvidia/cuda:12.8.0
• At least 8 GB of VRAM (can be reduced by lowering the batch size)
• Docker
• NVIDIA Container Toolkit

Before building the Docker image, download the checkpoints for the segmentation models.

From the docker directory, run:

python3 download_checkpoint.py <PATH_TO_THIS_REPO>/docker
python3 download_cocopan_checkpoint.py

These scripts will automatically fetch the necessary pretrained weights for LSeg and FC-CLIP.

Inside build_docker.sh, set your preferred image name after the -t flag.
Then build the image using:

. build_docker.sh GITHUB_USERNAME GITHUB_EMAIL GITHUB_TOKEN

Note: The GitHub credentials are required to authenticate and access this repository during the build process.

After a successful build, edit run.sh to match the image name and tag used during the build, then run:

. run.sh

This repository is a ROS 2 package.
To use ROS 2 launch files and parameters, build the workspace with colcon:

colcon build --symlink-install
source install/local_setup.bash

Additionally, compile and source the ROS 2 message interface:

cd ros2_vlmaps_interfaces
colcon build --symlink-install
source install/local_setup.bash

You can add the source command to your ~/.bashrc for convenience.

ros2 launch visual_language_navigation build_map.launch.py

ros2 launch visual_language_navigation semantic_map_server.launch.py

If you already have a semantic map, launch the semantic_map_server.
If you need to build or update one, use the semantic_map_builder.
Then run the following nodes:

python3 visual_language_navigation/llm/nav_agent.py
python3 visual_language_navigation/llm/chat_agent.py

Main parameters are defined in params/mapping_params.yaml.
Key ones include:

• This branch allows choosing between LSeg and FC-CLIP as the segmentation backend.
• The Dockerfile provides a ready-to-use development environment for ease of setup.
• Ensure your GPU drivers and CUDA runtime are correctly configured before running.

TODO

This work has taken inspiration from VLMaps: https://github.com/vlmaps/vlmaps

@inproceedings{huang23vlmaps,
               title={Visual Language Maps for Robot Navigation},
               author={Chenguang Huang and Oier Mees and Andy Zeng and Wolfram Burgard},
               booktitle = {Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},
               year={2023},
               address = {London, UK}
}