This is an code implementation of NeurIPS2022 paper using mindspore(DropCov: A Simple yet Effective Method for Improving Deep Architectures (poster)), created by Qilong Wang, Mingze Gao and Zhaolin Zhang.

Post-normalization plays a key role in deep global covariance pooling (GCP) networks. In this paper, we for the first time show that effective post-normalization can make a good trade-off between representation decorrelation and information preservation for GCP, which are crucial to alleviate over-fitting and increase representation ability of deep GCP networks, respectively. Based on this finding, we propose a simple yet effective pre-normalization method for GCP (namely DropCov), which performs an adaptive channel dropout before GCP to achieve tradeoff between representation decorrelation and information preservation. The proposed DropCov improves performance of both deep CNNs and ViTs.

●OS：18.04
●CUDA：11.6
●Toolkit：mindspore1.9
●GPU:GTX 3090

●First, Install the driver of NVIDIA
●Then, Install the driver of CUDA
●Last, Install cudnn

create virtual enviroment mindspore conda create -n mindspore python=3.7.5 -y conda activate mindspore CUDA 10.1

conda install mindspore-gpu cudatoolkit=10.1 -c mindspore -c conda-forge

CUDA 11.1

conda install mindspore-gpu cudatoolkit=11.1 -c mindspore -c conda-forge

validataion

python -c "import mindspore;mindspore.run_check()"

Download and extract ImageNet train and val images from http://image-net.org/. The directory structure is the standard layout for the torchvision datasets.ImageFolder, and the training and validation data is expected to be in the train/ folder and val/ folder respectively:

/path/to/imagenet/
  train/
    class1/
      img1.jpeg
    class2/
      img2.jpeg
  val/
    class1/
      img3.jpeg
    class/2
      img4.jpeg

To evaluate a pre-trained model on ImageNet val with GPUs run:

CUDA_VISIBLE_DEVICES={device_ids}  python eval.py --data_path={IMAGENET_PATH} --checkpoint_file_path={CHECKPOINT_PATH} --device_target="GPU" --config_path={CONFIG_FILE} &> log &

You can run the main.py to train as follow:

mpirun --allow-run-as-root -n {RANK_SIZE} --output-filename log_output --merge-stderr-to-stdout python train.py  --config_path={CONFIG_FILE} --run_distribute=True --device_num={DEVICE_NUM} --device_target="GPU" --data_path={IMAGENET_PATH}  --output_path './output' &> log &

For example:

mpirun --allow-run-as-root -n 4 --output-filename log_output --merge-stderr-to-stdout python train.py  --config_path="./config/resnet50_imagenet2012_config.yaml" --run_distribute=True --device_num=4 --device_target="GPU" --data_path=./imagenet --output_path './output' &> log &

@inproceedings{wang2022nips,
  title={A Simple yet Effective Method for Improving Deep Architectures},
  author={Qilong Wang and Mingze Gao and Zhaolin Zhang and Jiangtao Xie and Peihua Li and Qinghua Hu},
  booktitle = {NeurIPS},
  year={2022}
}

Our code are built following GCP_Optimization, DeiT, Swin Transformer , thanks for their excellent work