CIFAR-10 ImageNet-1K CheXpert EyePACS

CIFAR-10 without public data

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Differentially Private Learning Needs Better Features (or Much More Data) (code) ICLR 2021 ScatterNet + Linear 3 1e-5

67.0%

 Opacus RDP
Differentially Private Learning Needs Better Features (or Much More Data) (code) ICLR 2021 ScatterNet + CNN 3 1e-5

69.3%

 Opacus RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 1 1e-5

56.8%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 2 1e-5

65.9%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 4 1e-5

73.5%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 8 1e-5

81.4%

 TFP RDP
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 3 1e-05

55.18% AUC

 Torch PRV

CIFAR-10 with ImageNet 1k

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 1 1e-5

94.7%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 2 1e-5

95.4%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 4 1e-5

96.1%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 Wide-ResNet 8 1e-5

96.7%

 TFP RDP

CIFAR-10 with anything goes

Method Venue Public Data Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 0 0

99.75%

 N/A
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 0 0

87.14% AUC

 N/A

ImageNet without public data

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 NF-ResNets 8 8e-7

32.4%

 TFP RDP

ImageNet with public DataComp-1B

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Reproducible scaling laws for contrastive language-image learning (code) CVPR, 2023 ViT-B/32 0 0

72.8%

 N/A
Reproducible scaling laws for contrastive language-image learning (code) CVPR, 2023 ViT-B/16 0 0

73.5%

 N/A
Reproducible scaling laws for contrastive language-image learning (code) CVPR, 2023 ViT-L/14 0 0

79.2%

 N/A

ImageNet with anything goes

Method Venue Public Data Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 JFT4B NF-ResNet 1 8e-7

84.4%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 JFT4B NF-ResNet 2 8e-7

85.6%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 JFT4B NF-ResNet 4 8e-7

86.0%

 TFP RDP
Unlocking High-Accuracy Differentially Private Image Classification through Scale (code) arXiv, Apr. 2022 JFT4B NF-ResNet 8 8e-7

86.7%

 TFP RDP
Reproducible scaling laws for contrastive language-image learning (code) CVPR, 2023 LAION-2B ViT-L/14 0 0

75.3%

 N/A
Reproducible scaling laws for contrastive language-image learning (code) CVPR, 2023 LAION-2B ViT-H/14 0 0

78.0%

 N/A
Reproducible scaling laws for contrastive language-image learning (code) CVPR, 2023 LAION-2B ViT-G/14 0 0

80.1%

 N/A

CheXpert without public data

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 1 1e-06

78.16% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 3 1e-06

79.15% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 5 1e-06

79.16% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 8 1e-06

79.68% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 1 1e-06

77.43% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 3 1e-06

77.95% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 5 1e-06

78.18% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 8 1e-06

78.30% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 1 1e-06

76.87% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 3 1e-06

77.79% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 5 1e-06

77.92% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 8 1e-06

78.75% AUC

 Torch PRV

CheXpert with ImageNet 1k

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 1 1e-06

78.46% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 3 1e-06

79.40% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 5 1e-06

80.98% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 8 1e-06

82.62% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 1 1e-06

74.95% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 3 1e-06

75.29% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 5 1e-06

75.31% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 8 1e-06

75.52% AUC

 Torch PRV

CheXpert with anything goes

Method Venue Public Data Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Unlocking Accuracy and Fairness in Differentially Private Image Classification (code) arXiv, Dec. 2022 ImageNet-21K NFNet-F0 0.5 4.478e-06

84.9% AUC

 TFP RDP
Unlocking Accuracy and Fairness in Differentially Private Image Classification (code) arXiv, Dec. 2022 ImageNet-21K NFNet-F0 1 4.478e-06

86.3% AUC

 TFP RDP
Unlocking Accuracy and Fairness in Differentially Private Image Classification (code) arXiv, Dec. 2022 ImageNet-21K NFNet-F0 2 4.478e-06

87.5% AUC

 TFP RDP
Unlocking Accuracy and Fairness in Differentially Private Image Classification (code) arXiv, Dec. 2022 ImageNet-21K NFNet-F0 4 4.478e-06

88.4% AUC

 TFP RDP
Unlocking Accuracy and Fairness in Differentially Private Image Classification (code) arXiv, Dec. 2022 ImageNet-21K NFNet-F0 8 4.478e-06

89.2% AUC

 TFP RDP
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 0 0

59.11% AUC

 N/A
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 1 1e-06

80.63% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 3 1e-06

81.80% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 5 1e-06

82.25% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 8 1e-06

82.27% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 0 0

49.75% AUC

 N/A
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 1 1e-06

77.28% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 3 1e-06

78.21% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 5 1e-06

78.33% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 8 1e-06

78.54% AUC

 Torch PRV

EyePACS without public data

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 1 1e-05

55.02% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 3 1e-05

57.1% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 5 1e-05

57.13% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + CNN 8 1e-05

57.34% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 1 1e-05

55.59% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 3 1e-05

57.29% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 5 1e-05

57.44% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) ScatterNet + Linear 8 1e-05

57.65% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 1 1e-05

55.51% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 3 1e-05

56.49% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 5 1e-05

56.83% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet 8 1e-05

57.79% AUC

 Torch PRV

EyePACS with ImageNet 1k

Method Venue Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 1 1e-05

69.34% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 3 1e-05

79.21% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 5 1e-05

79.84% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Full) 8 1e-05

80.78% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 1 1e-05

67.73% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 3 1e-05

68.68% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 5 1e-05

68.91% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) Wide-ResNet (Final Layer) 8 1e-05

69.10% AUC

 Torch PRV

EyePACS with anything goes

Method Venue Public Data Model Epsilon (ε) Delta (δ) Accuracy Accountant Verification
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 0 0

50.73% AUC

 N/A
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 1 1e-05

65.47% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 3 1e-05

70.30% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 5 1e-05

71.74% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) LAION-2B Vit-G/14 + TLNN 8 1e-05

72.3% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 0 0

50.73% AUC

 N/A
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 1 1e-05

65.12% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 3 1e-05

67.89% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 5 1e-05

69.22% AUC

 Torch PRV
Rethinking Benchmarks for Differentially Private Image Classification (code) NeuralIPS, 2024 (Submitted) WebImageTex Vit-B/16 + TLNN 8 1e-05

69.84% AUC

 Torch PRV