A Style-Transfer module for RealTime camera videos.

The trained-weights and data our system needs are stored here.

The weights rvm_mobilenetv3.pth should be put under /style_transfer/test/Model/, style_net-TIP-final.pth should be put under /mat/checkpoints/.

The codes are run on cuda=11.0, torch=1.9.0, torchvision=0.10.0

conda create -n <env_name> python=3.8
conda install cudatoolkit==11.0.3 -c conda-forge
conda install pytorch==1.9.0 torchvision==0.10.0  cudatoolkit=11.0 -c pytorch -c conda-forge
pip install -r requirement.txt

We built a new environment and test successfully with following steps.

conda create -n <env_name> python=3.8
conda activate <env_name>
pip install opencv-python
#### Install the pytorch, torchvision, and cudatoolkit you want. ####
conda install pytorch==1.9.0 torchvision==0.10.0 torchaudio==0.9.0 cudatoolkit=10.2 -c pytorch
#####################################################################
pip install scikit-image
pip install kornia

If you want to run the module:

python test.py

We use pyvirtualcam to create a virtual camera if you want to use this module on Google Meet.

The package works on Windows and Linux, but we only test on Ubuntu 20.04.3 LTS.

Install pyvirtualcam from PyPI with:

pip install pyvirtualcam

pyvirtualcam uses v4l2loopback virtual cameras on Linux.

Install v4l2loopback on Ubuntu with:

sudo apt install v4l2loopback-dkms

Create a virtual camera.

# Close the virtual devices
sudo modprobe -r v4l2loopback
# Create a virtual device (Google Meet)
sudo modprobe v4l2loopback video_nr=2 card_label="Virtual Camera" exclusive_caps=1
# Create a virtual device (Teams)
sudo modprobe v4l2loopback video_nr=2 card_label="Virtual Camera" exclusive_caps=0
# Check the devices
v4l2-ctl --list-devices

To use this module to send processed images to the virtual camera.

python test.py --virtual_camera

We use imageio, moviepy to collect the audio and infomation of the video, and tqdm is for checking the progress.

Install imageio, moviepy, and tqdm from PyPI with:

pip install imageio
pip install moviepy
pip install tqdm

Generate the styled video:

python test.py --process_video --input <input_video> --output <output_video>

This system mainly incorporated matting and style-transfer modules, which were forked from following implementations:

• Usability wrapper for ReReVST

• Robust Video Matting (RVM)

Here are the relevant papers:

• Consistent Video Style Transfer via Relaxation and Regularization

• Robust High-Resolution Video Matting with Temporal Guidance