Also known as atmospheric turbulance mitigation, warp stabilization, film shrink or VHS distortion fix, dewobble, dewiggle, detilt, rectification, heat haze removal. Can help with distortions from low bitrate compression or old codecs like MPEG2.

This is a partial implementation of the Turbulence Mitigation Transformer. (only distortion removal, no deblurring)

This does not do general video stabilization for shaky footage, only removes distortions within the frames. It is recommented to stabilize first if needed.

Check out stinkybread's comparisons here and here. (chrome recommended for smooth playback)

• pip install numpy (optional, only for pytorch backend)
• pytorch with cuda (optional, only for pytorch backend)
• vs-mlrt with tensorrt (optional, only for tensorrt backend)

Put the entire vs_undistort folder into your vapoursynth scripts folder.
Or install via pip: pip install -U git+https://github.com/pifroggi/vs_undistort.git

The PyTorch backend offers some extra control knobs and supports any CPU and Nvidia GPU, but is slower and requires more VRAM.

import vs_undistort
clip = vs_undistort.pytorch(clip, temp_window=10, tile_width=None, tile_height=None, overlap=None, scales=[True, True, True], interpolation="bilinear", device="cuda")

clip
Distorted clip. Must be in RGB format.

temp_window
Temporal window length. How many frames are grouped together and processed as a single chunk. Larger means higher VRAM requirements, but better temporal averaging and slower distortions can be removed. If this is too small, some distortions may not get removed, small jumps/hitches may be visible between windows and seams from tile size may become more obvious.

tile_width, tile_height (optional)
Size of tiles to split the frames into. Must be a multiple of 16. By default the full frame dimensions are used.
Larger means higher VRAM requirements, but better spatial averaging and larger distortions can be removed.

overlap (optional)
Overlap from one tile to the next. Use if seams between tiles are visible.

interpolation (optional)
Interpolation mode used for warping the frames.
Mode "bilinear" is a bit faster, but slightly blurry.
Mode "bicubic" is a bit slower and may oversharpen slightly, but no blur.

scales (optional)
Sets which distortion scales get fixed via scales=[True, True, True], which stands for [coarse, middle, fine]. Set one or more to False to disable them. This is an experimental feature and may get removed if it turns out to be useless.

device (optional)
Can be "cuda" to use with an Nvidia GPU, or "cpu". This will be extremely slow on CPU.

The TensorRT backend is much faster and requires less VRAM, but only supports bilinear interpolation and requires an Nvidia RTX GPU. On the first run, it will automatically build an engine, which takes a few minutes. Changing tile size or temporal window length will trigger rebuilding, but previously build engines are saved for later.

import vs_undistort
clip = vs_undistort.tensorrt(clip, temp_window=10, tile_width=None, tile_height=None, overlap=None, num_streams=1)

clip
Distorted clip. Must be in RGBH format.

temp_window
Temporal window length. How many frames are grouped together and processed as a single chunk. Larger means higher VRAM requirements, but better temporal averaging and slower distortions can be removed. If this is too small, some distortions may not get removed, small jumps/hitches may be visible between windows and seams from tile size may become more obvious.

tile_width, tile_height (optional)
Size of tiles to split the frames into. Must be a multiple of 16. By default the full frame dimensions are used.
Larger means higher VRAM requirements, but better spatial averaging and larger distortions can be removed.

overlap (optional)
Overlap from one tile to the next. Use if seams between tiles are visible.

num_streams (optional)
Number of parallel TensorRT streams. For high end GPUs higher can be faster, but requires more VRAM.