The demo pipeline demonstrates how to utilize NanoSAM model to segment objects within a frame. Four dots are positioned on the frame to identify and separate objects. The dots appear in black, and each object is assigned a unique color (green, red, blue, yellow) with a gradient. The gradient is constructed from a series of masks produced by the model, with each mask being wider than the preceding one in relation to the point's position so that small objects close to the point will have a brighter color.
In addition, the pipeline demonstrates how to use a pyfunc element to handle cases where a model has custom inputs.
Preview:
Tested on platforms:
- Nvidia Turing
- Nvidia Jetson Orin family
git clone https://github.com/insight-platform/Savant.git
cd Savant
git lfs pull
./utils/check-environment-compatibleNote: Ubuntu 22.04 runtime configuration guide helps to configure the runtime to run Savant pipelines.
The demo uses models that are compiled into TensorRT engines the first time the demo is run. This takes time. Optionally, you can prepare the engines before running the demo by using the command:
# you are expected to be in Savant/ directory
./scripts/run_module.py --build-engines samples/nanosam/module/demo.ymlMask decoder engine has custom inputs which are not supported by the default engine builder. You need to do the previous step and then build a mask decoder engine with the following command:
# you are expected to be in Savant/ directory
# if x86
docker run --rm \
--gpus=all \
-v "$(pwd)/cache/models/nanosam:/opt/nanosam" \
--entrypoint bash ghcr.io/insight-platform/savant-deepstream-extra \
-c "/usr/src/tensorrt/bin/trtexec --onnx=/opt/nanosam/image_encoder/mobile_sam_mask_decoder.onnx \
--saveEngine=/opt/nanosam/mobile_sam_mask_decoder.engine \
--minShapes=point_coords:1x1x2,point_labels:1x1 \
--optShapes=point_coords:1x1x2,point_labels:1x1 \
--maxShapes=point_coords:1x10x2,point_labels:1x10"
# if Jetson
docker run --rm \
--runtime=nvidia \
-v "$(pwd)/cache/models/nanosam:/opt/nanosam" \
--entrypoint bash ghcr.io/insight-platform/savant-deepstream-l4t-extra \
-c "/usr/src/tensorrt/bin/trtexec --onnx=/opt/nanosam/image_encoder/mobile_sam_mask_decoder.onnx \
--saveEngine=/opt/nanosam/mobile_sam_mask_decoder.engine \
--minShapes=point_coords:1x1x2,point_labels:1x1 \
--optShapes=point_coords:1x1x2,point_labels:1x1 \
--maxShapes=point_coords:1x10x2,point_labels:1x10"Then you can run the demo:
# you are expected to be in Savant/ directory
# if x86
docker compose -f samples/nanosam/docker-compose.x86.yml up
# if Jetson
docker compose -f samples/nanosam/docker-compose.l4t.yml up
# open 'rtsp://127.0.0.1:554/stream/video' in your player
# or visit 'http://127.0.0.1:888/stream/video/' (LL-HLS)
# Ctrl+C to stop running the compose bundle