Jun Xiao, Zihang Lyu, Cong Zhang, Yakun Ju, Changjian Shui, Kin-Man Lam

The Hong Kong Polytechnic University, Nanyang Technological University, Vector Institute

Abstract: Image warping, a classic task in computer vision, aims to use geometric transformations to change the appearance of images. Recent methods learn the resampling kernels for warping through neural networks to estimate missing values in irregular grids, which, however, fail to capture local variations in deformed content and produce images with distortion and less high-frequency details. To address this issue, this paper proposes an effective method, namely MFR, to learn Multi-Frequency Representations from input images for image warping. Specifically, we propose a progressive filtering network to learn image representations from different frequency subbands and generate deformable images in a coarse-to-fine manner. Furthermore, we employ learnable Gabor wavelet filters to improve the model's capability to learn local spatial-frequency representations. Comprehensive experiments, including homography transformation, equirectangular to perspective projection, and asymmetric image super-resolution, demonstrate that the proposed MFR significantly outperforms state-of-the-art image warping methods. Our method also showcases superior generalization to out-of-distribution domains, where the generated images are equipped with rich details and less distortion, thereby high visual quality.

• Installation
• Prepare Datasets
• Pre-trained Models
• Train and Test
• Visual Results
• Citation

• Python 3.8, PyTorch >= 1.10.0
• Requirements: see requirements.txt
• Platforms: Ubuntu 20.04, cuda-11.3

We recommend using conda for installation:

conda env create --file environment.yaml
conda activate mfr

Our codes rely on the project of SRWarp. Please install it using the following commands.

git clone https://github.com/sanghyun-son/pysrwarp
cd pysrwarp
make

If your CUDA compatibility is sm86, modify cuda/Makefile before make.

vi cuda/Makefile

Homography transformation

• DIV2K: DIV2KW (isc), DIV2KW (osc)

• Benchmark datasets: Set5W (isc), Set5W (osc), Set14W (isc), Set14W (osc), B100W (isc), B100W (osc), Urban100W (isc), Urban100W (osc)

Symmetric-scale SR

1.1 Download benchmark datasets (e.g., Set5, Set14 and other test sets). 1.2 mkdir datasets, create a folder named 'datasets' 1.3 Cd to ./scripts and run generate_bicubic_asym_img.m in Matlab to prepare HR/LR images in your_data_path as belows:

The structure of the datasets are as follows:

datasets
├── benchmark
│   ├── B100
│   │   ├── HR
│   │   ├── LR_bicubic
│   │   ├── LR_bicubic_Transform
│   │   ├── LR_warp_in_scale
│   │   └── LR_warp_out_of_scale
│   ├── Set14
│   │   ├── HR
│   │   ├── LR_bicubic
│   │   ├── LR_warp_in_scale
│   │   └── LR_warp_out_of_scale
│   ├── Set5
│   │   ├── HR
│   │   ├── LR_bicubic
│   │   ├── LR_warp_in_scale
│   │   └── LR_warp_out_of_scale
│   └── Urban100
│       ├── HR
│       ├── LR_bicubic
│       ├── LR_warp_in_scale
│       └── LR_warp_out_of_scale
├── div2k
│   ├── DIV2K_train_HR
│   ├── DIV2K_train_LR_bicubic
│   ├── DIV2K_valid_HR
│   ├── DIV2K_valid_LR_bicubic
│   ├── DIV2K_valid_LR_warp_in_scale
│   └── DIV2K_valid_LR_warp_ouf_of_scale

Train: CUDA_VISIBLE_DEVICES=0,1 python train_warp.py --config configs/train/train_rrdb_mfr_warp.yaml --gpu 0,1

Test: bash ./scripts/test_scripts/test-benchmark-warp.sh {YOUR_MODEL_PATH} 0

Train: CUDA_VISIBLE_DEVICES=0 python train_sr.py --config configs/train/train_rcan_mfr.yaml --gpu 0

Test: bash ./scripts/test-benchmark-asym.sh {YOUR_MODEL_PATH} 0

If you find our work useful in your research, please consider citing our paper:

@inproceedings{xiao2024towards,
  title={Towards Progressive Multi-Frequency Representation for Image Warping},
  author={Xiao, Jun and Lyu, Zihang and Zhang, Cong and Ju, Yakun and Shui, Changjian and Lam, Kin-Man},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={2995--3004},
  year={2024}
}

This code is built on LETW and SRWarp. We thank the authors for sharing their codes.