Skip to content

tootouch/SID

BranchesTags

Repository files navigation

Detecting Adversarial Examples from Sensitivity Inconsistency of Spatial-Transform Domain

Pytorch re-implementation for "Detecting Adversarial Examples from Sensitivity Inconsistency of Spatial-Transform Domain". AAAI 2021

Run

run ./scripts/run_pipeline.sh

run_pipeline.sh

modelname_list="vgg19 resnet34"
adv_method_list=("DeepFool" "BIM" "CW" "CW" "FAB" "FGSM" "PGD" "PGD" "PGD")
adv_expname_list=("DeepFool" "BIM" "CW" "Low_CW" "FAB" "FGSM" "PGD" "Low_PGD1" "Low_PGD2")
dataname_list="CIFAR10 SVHN CIFAR100"


for modelname in $modelname_list
do
    for dataname in $dataname_list
    do
        # 1. train classifier
        bash run_classifier.sh $modelname $dataname 

        # 2. make adversarial examples
        for i in ${!adv_method_list[*]}
        do
            bash save_adv_samples.sh $modelname ${adv_method_list[$i]} ${adv_expname_list[$i]} $dataname
        done

        # 3. known attack
        for i in ${!adv_method_list[*]}
        do
            bash known_attack.sh $modelname ${adv_expname_list[$i]} $dataname
        done

        # 4. transfer attack
        bash run_transfer_attack.sh $modelname $dataname 

    done
done

Results

  • Model: ResNet34, VGG19

1. Adversarial Attacks

CIFAR10

  • VGG19
Adv Acc(%) Adv Acc(%) DWT # Success Images
DeepFool 1.3 84.73 7117
BIM 0 63.03 7213
CW 12.44 80.84 5993
Low_CW 52.96 87.62 2399
FAB 0.03 87.52 7240
FGSM 13.82 59.65 5872
PGD 0 64.92 7204
PGD_L2 0 65.4 7143
Low_PGD1 59.34 86.9 1879
Low_PGD2 15.96 84.4 5618
AutoAttack 0 68.12 7256
Square 0.81 81.66 7164
  • ResNet34
Adv Acc(%) Adv Acc(%) DWT # Success Images
DeepFool 5.59 90.66 6576
BIM 0.08 63.94 7022
CW 22.66 83.57 4852
Low_CW 58.79 91.86 1973
FAB 0 92.13 7021
FGSM 35.8 66.79 3825
PGD 0.06 67.95 7009
PGD_L2 0.43 66.59 6991
Low_PGD1 60.67 90.9 1704
Low_PGD2 14.4 88.23 5606
AutoAttack 0 70.23 7070
Square 0.88 86.27 6940

CIFAR100

  • VGG19
Adv Acc(%) Adv Acc(%) DWT # Success Images
DeepFool 2.9 57.6 2961
BIM 1.72 44.59 2950
CW 9.35 52.97 2245
Low_CW 30.82 59.3 1187
FAB 5.31 59.14 2865
FGSM 16.79 35.4 1826
PGD 1.32 45.68 2943
PGD_L2 2.07 47.33 2895
Low_PGD1 29.61 58.47 1190
Low_PGD2 9.72 56.72 2325
AutoAttack 0 47.42 3117
Square 2.45 51.7 2850
  • ResNet34
Adv Acc(%) Adv Acc(%) DWT # Success Images
DeepFool 11.49 69.42 3257
BIM 0.07 41.93 3811
CW 9.81 59.15 2890
Low_CW 31.38 68.75 1549
FAB 3.54 69.66 3587
FGSM 13.16 35.23 2610
PGD 0.08 44.93 3767
PGD_L2 0.23 46.04 3768
Low_PGD1 35.93 67.45 1202
Low_PGD2 8.45 63.67 2969
AutoAttack 0 49.85 3765
Square 0.45 60.86 3731

SVHN

  • VGG19
Adv Acc(%) Adv Acc(%) DWT # Success Images
DeepFool 1.4 54.58 23937
BIM 1.05 29.9 24015
CW 15.68 69.78 20208
Low_CW 74.88 92.11 4829
FAB 0.74 90.69 24116
FGSM 20.19 44.18 19039
PGD 1.26 31.61 23982
PGD_L2 1.11 24.12 24024
Low_PGD1 81.78 91.85 3085
Low_PGD2 52.86 83.1 10551
AutoAttack 0.44 34.92 24170
Square 2.98 82.6 23531
  • ResNet34
Adv Acc(%) Adv Acc(%) DWT # Success Images
DeepFool 4.28 78.8 23173
BIM 2.42 31.17 23615
CW 33.62 74.71 15495
Low_CW 76.69 93.47 4394
FAB 1.24 93.41 23938
FGSM 40.35 55.49 13760
PGD 2.74 33.89 23566
PGD_L2 2.74 24.23 23557
Low_PGD1 81.72 92.56 3142
Low_PGD2 49.66 83.99 11355
AutoAttack 0.43 38.36 24129
Square 4.81 81.37 23008

2. Known Attacks

CIFAR10

  • VGG19
AUROC(%) Detection Acc(%) #(train, dev, test)
DeepFool 91.76 86.14 (4528, 754, 2265)
BIM 99.79 98.27 (5245, 874, 2624)
CW 88.31 81.62 (3896, 648, 1952)
Low_CW 90.99 85.94 (1574, 261, 790)
FAB 97.3 92.56 (4629, 770, 2317)
FGSM 90.67 83.96 (4207, 700, 2108)
PGD 99.67 98 (5161, 859, 2585)
PGD_L2 99.74 97.76 (5157, 858, 2582)
Low_PGD1 80.74 74.68 (1252, 208, 630)
Low_PGD2 89.74 85.5 (3596, 598, 1801)
AutoAttack 99.64 97.86 (5102, 849, 2554)
Square 97.59 94.23 (4607, 766, 2306)
  • ResNet34
AUROC(%) Detection Acc(%) #(train, dev, test)
DeepFool 92.94 89.1 (4112, 684, 2059)
BIM 99.39 97.18 (5085, 846, 2546)
CW 90.28 84.19 (3184, 530, 1596)
Low_CW 82.58 79.17 (1267, 210, 637)
FAB 95.76 92.1 (4399, 732, 2204)
FGSM 94.89 88.94 (2863, 476, 1433)
PGD 99.28 96.39 (4935, 821, 2472)
PGD_L2 98.88 96 (5022, 836, 2514)
Low_PGD1 82.95 78.13 (1102, 183, 553)
Low_PGD2 90.07 85.16 (3519, 586, 1761)
AutoAttack 97.57 93.69 (4919, 818, 2462)
Square 95.32 91.77 (4374, 728, 2191)

CIFAR100

  • VGG19
AUROC(%) Detection Acc(%) #(train, dev, test)
DeepFool 80.4 73.61 (1893, 315, 949)
BIM 86.14 78.24 (1879, 312, 943)
CW 72.53 68.66 (1461, 243, 734)
Low_CW 59.76 59.38 (779, 129, 392)
FAB 76.26 70.87 (1836, 305, 920)
FGSM 63.77 62.82 (1247, 206, 627)
PGD 87.85 81.94 (1872, 310, 940)
PGD_L2 81.07 77.37 (1848, 306, 929)
Low_PGD1 55.68 58.55 (782, 129, 395)
Low_PGD2 66.83 67.21 (1483, 246, 745)
AutoAttack 80.02 73.1 (1969, 327, 988)
Square 77.73 72.67 (1859, 309, 932)
  • ResNet34
AUROC(%) Detection Acc(%) #(train, dev, test)
DeepFool 74.94 73.37 (2118, 351, 1064)
BIM 98.02 94.69 (2749, 457, 1378)
CW 78.52 74.3 (1936, 321, 970)
Low_CW 57.16 59.92 (1019, 169, 514)
FAB 76.46 71.85 (2320, 385, 1165)
FGSM 93.41 87.5 (1817, 302, 912)
PGD 97.23 94 (2652, 441, 1328)
PGD_L2 97.29 94.04 (2669, 444, 1338)
Low_PGD1 59.75 61.08 (794, 132, 399)
Low_PGD2 68.78 67.1 (1925, 320, 964)
AutoAttack 93.93 89.83 (2547, 423, 1277)
Square 85.4 79.39 (2406, 400, 1205)

SVHN

  • VGG19
AUROC(%) Detection Acc(%) #(train, dev, test)
DeepFool 95.4 89.52 (18605, 3100, 9304)
BIM 98.96 95.2 (24104, 4017, 12055)
CW 93.56 87.75 (14674, 2445, 7339)
Low_CW 90.12 84.41 (3358, 559, 1683)
FAB 96.75 92.39 (15473, 2578, 7739)
FGSM 92.78 86.19 (15777, 2628, 7891)
PGD 98.9 94.87 (23855, 3975, 11931)
PGD_L2 98.67 94.37 (25080, 4179, 12543)
Low_PGD1 78.78 72.43 (2203, 365, 1106)
Low_PGD2 88.59 81.52 (7886, 1313, 3948)
AutoAttack 98.94 94.77 (23558, 3925, 11782)
Square 98.3 94.83 (15747, 2624, 7876)
  • ResNet34
AUROC(%) Detection Acc(%) #(train, dev, test)
DeepFool 94.33 88.66 (16354, 2725, 8180)
BIM 99.57 97.43 (23573, 3927, 11790)
CW 90.29 84.11 (11686, 1947, 5845)
Low_CW 87.5 83.36 (3035, 504, 1523)
FAB 96.35 91.42 (15002, 2499, 7504)
FGSM 89.32 82.5 (12162, 2026, 6084)
PGD 99.49 97.21 (23148, 3857, 11576)
PGD_L2 98.75 94.97 (24687, 4114, 12347)
Low_PGD1 81.22 76.09 (2268, 377, 1136)
Low_PGD2 89.78 82.66 (8342, 1389, 4174)
AutoAttack 99.54 97.3 (22959, 3825, 11482)
Square 98.23 95.14 (15704, 2616, 7855)

3. Transfer Attacks

  • Row: Source
  • Column: Target

CIFAR10

  • VGG19
DeepFool BIM CW Low_CW FAB FGSM PGD PGD_L2 Low_PGD1 Low_PGD2 AutoAttack Square
DeepFool 91.76 61.91 90.83 90.47 89.7 89.07 62.33 60.69 84.43 84.71 62.17 89.83
BIM 65.76 99.79 81.7 41.25 43.87 82.32 99.78 99.81 63.43 96.45 99.79 32.8
CW 88.16 69.72 88.31 87.83 84.78 88.94 71.69 69.97 80.25 86.33 71.58 85.77
Low_CW 89.16 51.94 85.72 90.99 87.82 80.62 51.36 50.84 88.69 75.91 55.31 90.16
FAB 95.84 43.01 93.17 92.47 97.3 91.51 44.71 43.09 82.6 76.42 45.86 96.72
FGSM 84.7 72.84 88.67 81.27 79.18 90.67 75.32 73.41 74.65 84.2 73.15 83.41
PGD 66.58 99.82 81.71 42.17 45.08 84.06 99.67 99.78 62.75 96.62 99.77 36.41
PGD_L2 67.39 99.83 84.48 46.9 48.83 85.01 99.77 99.74 69.58 96.98 99.81 36.83
Low_PGD1 77.68 69.07 80.04 82.66 77.93 77.01 69.56 68.64 80.74 81.79 71.75 76.51
Low_PGD2 77.36 90.23 83.99 71.57 68.34 83.23 90.06 90.25 76.74 89.74 90.26 66.8
AutoAttack 66.91 99.72 82.04 38.63 43.62 85.85 99.66 99.7 63.95 95.72 99.64 38.06
Square 95.91 47.4 94.39 96.43 96.83 92.97 48.66 46.95 87.56 84.2 48.42 97.59
  • ResNet34
DeepFool BIM CW Low_CW FAB FGSM PGD PGD_L2 Low_PGD1 Low_PGD2 AutoAttack Square
DeepFool 92.94 73.1 89.64 86.91 94.01 84.39 75.35 71.03 87.59 91.4 74.82 90.98
BIM 62.13 99.39 83.62 56.03 60.76 76.35 99.32 99.45 66 90.74 99.33 45.59
CW 87.24 81.21 90.28 81.74 84.14 88.61 80.69 79.03 83.92 87.26 82.65 86.05
Low_CW 84.41 62.76 84.19 82.58 85.55 72.83 67.99 58.3 88.52 84.09 63.57 85.86
FAB 88.69 74.51 86.99 81.15 95.76 84.35 79.18 76.58 78.78 90.76 80.11 86.79
FGSM 86.49 82.09 92.81 83.56 82.92 94.89 81.67 81.75 81.56 86.81 83.67 85.67
PGD 59.23 99.42 82.57 57.65 56.27 75.99 99.28 99.36 64.2 90.35 99.46 44.45
PGD_L2 65.7 99.1 83.43 61.45 64.05 73.18 99.04 98.88 67.84 91.59 98.81 54.11
Low_PGD1 77.44 70.08 82.17 79.2 79.07 75.33 77.02 69.89 82.95 86.42 74.85 81.79
Low_PGD2 85.74 92.84 90.4 80.67 86.79 85.36 91.48 92.35 85.76 90.07 92.79 82.17
AutoAttack 66.04 98 82.31 61.45 58.13 79.81 97.69 98.18 64.44 89.35 97.57 51.35
Square 92.02 63.15 93.01 86.4 94.09 85.05 68.54 62.23 87.73 91.03 68.51 95.32

CIFAR100

  • VGG19
DeepFool BIM CW Low_CW FAB FGSM PGD PGD_L2 Low_PGD1 Low_PGD2 AutoAttack Square
DeepFool 80.4 52.95 71.77 81.31 82.27 68.61 51.34 47.03 72.65 70.04 51.3 77.77
BIM 46.63 86.14 57.88 53.55 41.39 61.81 90.2 84.8 56.72 73.45 80.44 42.52
CW 75.26 72.3 72.53 69.18 72.6 67.45 65.02 64.75 66.54 71.74 64.16 68.25
Low_CW 83.81 52.64 83.21 59.76 78.18 60.5 52.16 54.37 62.42 72.59 52.83 65.95
FAB 84.15 53.52 72.35 74.6 76.26 58.67 54.37 50.09 65.21 66.54 53.59 76.95
FGSM 72.34 61.55 71.93 67.63 71.73 63.77 62.16 59.52 63.21 64.98 58.82 71.91
PGD 44.98 89.12 51.08 45.1 40.51 56.23 87.85 85.88 52.43 66.13 81.35 44.49
PGD_L2 41.47 89.16 55.15 43.89 38.49 49.78 91.12 81.07 59.44 70.08 79.57 41.95
Low_PGD1 76.1 57.68 73.99 75.12 77.17 61.2 62.45 68.64 55.68 75.67 55.72 67.65
Low_PGD2 64.91 77.37 64.82 62.86 59.45 60.75 73.54 70.8 70.4 66.83 66.56 60.75
AutoAttack 49.88 89.22 58.27 45.34 42.91 61.9 88.54 83.86 54.29 65.01 80.02 45.25
Square 81.28 51.62 78.49 75.29 83.78 65.03 55.67 51.96 68.4 70.97 55.75 77.73
  • ResNet34
DeepFool BIM CW Low_CW FAB FGSM PGD PGD_L2 Low_PGD1 Low_PGD2 AutoAttack Square
DeepFool 74.94 57.48 75.73 73.52 83.33 72.89 54.86 58.38 67.29 67.29 59.76 79.52
BIM 31.34 98.02 55.44 38.47 28.08 50.25 95.76 97.58 31.97 72.02 93.13 39.29
CW 77.76 72.18 78.52 76.02 74.37 79.36 68.92 71.47 69.53 73.97 68.14 73.56
Low_CW 78.28 72.04 84.22 57.16 78.3 64.95 60.15 67.52 71.12 72.49 64.14 73.62
FAB 78.39 58.57 76.55 66.73 76.46 74.02 58.53 62.44 65.64 71.09 61.52 78.15
FGSM 77.61 71.77 88.83 70.72 79.05 93.41 68.76 70.19 65.67 69.9 77.14 84.17
PGD 26.7 98.19 57.63 36.64 31.42 52.43 97.23 99.56 34.39 75.72 95.32 37.67
PGD_L2 28.46 98.88 51.37 35.31 25.86 46.27 95.76 97.29 29.97 73.22 92.96 32.25
Low_PGD1 72.63 72.77 80.75 72.64 64.63 65.4 66.8 73.77 59.75 81.46 73.27 70.26
Low_PGD2 64.37 83.87 79.34 68.54 67.23 68.19 80.81 82.5 60 68.78 85.91 68.7
AutoAttack 28.99 98 59.05 37.38 32.47 61.76 98.28 99.2 31.78 72.65 93.93 47.21
Square 81.09 66.76 75.13 70.5 80.85 74.05 66.64 72.78 55.91 74.63 72.1 85.4

SVHN

  • VGG19
DeepFool BIM CW Low_CW FAB FGSM PGD PGD_L2 Low_PGD1 Low_PGD2 AutoAttack Square
DeepFool 95.4 35.78 95.32 94.09 94.48 95.07 33.91 33.6 90.72 82.15 39.71 93.9
BIM 32.3 98.96 31.01 22.1 34.79 31.1 99.05 98.99 55.46 91.36 99.06 16.17
CW 93.02 40.19 93.56 92.91 92.86 93.21 38.05 37.82 90.77 85.13 44.09 92.19
Low_CW 86.67 47.91 87.49 90.12 86.96 84.54 44.6 43.49 87.35 78.68 50.73 89.86
FAB 95.67 40.36 95.64 95.64 96.75 95.33 38.83 37.38 91.82 83.18 44.95 96.3
FGSM 92.08 38.91 92.23 90.46 91.06 92.78 36.37 35.97 87.41 80.92 42.57 89.7
PGD 33.59 98.85 32.33 23.09 35.95 32.42 98.9 98.92 55.99 91.46 98.96 17.7
PGD_L2 31.62 98.65 28.87 20.87 32.87 29.1 98.69 98.67 54.7 91.03 98.66 16.58
Low_PGD1 72.79 63.26 74.54 76.43 70.4 69.07 59.46 62.38 78.78 77.47 64.46 71.02
Low_PGD2 76.74 77.22 79.7 75.42 76.82 77.04 77.11 77.54 83.25 88.59 78.72 71.01
AutoAttack 38.41 98.85 37.05 25.45 40.35 37.31 98.96 98.89 58.59 91.69 98.94 19.92
Square 96.99 33.77 96.93 97.95 97.11 95.84 29.17 30.33 94.99 87.48 34.8 98.3
  • ResNet34
DeepFool BIM CW Low_CW FAB FGSM PGD PGD_L2 Low_PGD1 Low_PGD2 AutoAttack Square
DeepFool 94.33 53.61 93.78 89.5 91.69 91.08 55.08 57.89 86.06 77.35 52.81 92.63
BIM 34.16 99.57 46.25 23.13 24.32 55.06 99.63 99.59 55.35 94.76 99.56 15.03
CW 88.69 58.87 90.29 86.35 86.55 88.29 61.13 62.77 84.75 79.73 59.11 86.49
Low_CW 87.76 54.83 87.18 87.5 87.62 80.5 57.36 54.97 87.41 80 57.78 89.15
FAB 96.59 56.24 95.28 93.84 96.35 92.54 58.97 57.76 88.04 78.36 56.18 97.06
FGSM 85.22 60.89 87.28 81.43 83.11 89.32 62.84 63.46 78.65 74.61 59.8 83.29
PGD 35.69 99.55 48.53 24.24 25.97 55.67 99.49 99.51 55.93 94.37 99.4 16.04
PGD_L2 34.54 98.55 44.23 24.16 25.42 52.18 98.56 98.75 53.97 92.27 98.27 16.5
Low_PGD1 79.42 66.12 81.67 80.15 76.53 75.8 64.34 66.59 81.22 81.41 68.53 75.92
Low_PGD2 73.31 84.54 80.64 70.08 68.54 76.95 84.6 85.39 81.43 89.78 84.51 64.15
AutoAttack 38.17 99.61 50.6 23.6 26.74 57.42 99.55 99.55 55.86 94.62 99.54 16.53
Square 97.49 56.05 96.95 96.46 96.03 94.34 59.29 57.35 93.28 83.35 57.63 98.23

Citations

@article{kim2020torchattacks,
  title={Torchattacks: A pytorch repository for adversarial attacks},
  author={Kim, Hoki},
  journal={arXiv preprint arXiv:2010.01950},
  year={2020}
}

About

pytorch reimplementation for Detecting Adversarial Examples from Sensitivity Inconsistency of Spatial-Transform Domain

Resources

Readme

License

Apache-2.0 license
Activity

Stars

11 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages