GT-Craft is a framework for fast-prototyping geospatial-based digital twins in Unity 3D. It uses a streamed satellite map image and elevation data to create a virtual scene in Unity. From the map image, the object semantic is extracted by using DNN models, and this repo includes the map data streaming and segmentation phases of GT-Craft. For more detail, please check our paper.

The current implementation is developed and test on Ubuntu 22.04.

You need to get a key to use MS Bing Maps API from here. Then, save your key in MapSessionConfig.txt under the repo directory.

Map_Streaming_SemanticExtraction
 |- MapSessionConfig.txt
 |- scripts
 |-  ...

$ cd Map_Streaming_SemanticExtraction

# Pytorch Install or visit https://pytorch.org/get-started/locally/ and install pytorch by yourself
Map_Streaming_SemanticExtraction $ ./setup_cpu.sh   # Pytorch without GPU
Map_Streaming_SemanticExtraction $ ./setup_cu118.sh # Pytorch with NVIDIA GPU and CUDA

# Other Libraries
Map_Streaming_SemanticExtraction $ pip install -r requirements.txt

The datasets to train the semantic segmentation models for buildings and roads can be downloaded from the below links.

Massachusetts Roads Dataset

Massachusetts Buildings Dataset

Map_Streaming_SemanticExtraction
 |- MapSessionConfig.txt
 |- scripts
     |- config.json

You can set your target region by modifying the json file sample_data/map_patch.json.

cd scripts

python map_image_streamer.py      # streaming map image

# streaming elevation data for 1500x1500 image size
# as it takes too long, use the prepared elevation data `sample_data/elevation.json/bin` for first exploration.
# python elevation_streamer.py 1500

python elevation_visualizer.py

We share the pre-trained models' weights in this link. You need to download them under Map_Semantic_Extraction/pretrained_models.

You dataset directory should have the following structure. For testing, you can put your own map image under test directory.

YOUR_DATASET_PATH
  |- Buildings
      |- clabel_class_dict.csv
      |- metadata.csv
      |- png
      |- tiff
  |- Roads
      |- label_class_dict.csv
      |- metadata.csv
      |- tiff
  |- test

Then, you can run the following scripts for training the segmentation models.

cd scripts
python train_road.py
python train_building.py

After all models are prepared, you can validate the models' performance on your target region.

python validate_road_building.py

You can get the mask images of the object classes (road and building) in your target area.

python road_building_segmentation.py