Demographic Differences in AI-Generated Face Detection: An Operational Evaluation

Errors in identity verification can affect access to employment, financial services, and age-restricted goods and content, making demographic disparities in detection performance particularly consequential. We build on previous work and evaluate ten generated-image detectors across demographic groups and image generators. We further examine their robustness to lossy JPEG compression under production conditions and evaluate skin-tone labeling by vision–language models (VLMs).

Abstract

AI-generated portraits present a growing fraud risk for identity-verification platforms. Reliable deployment therefore requires understanding how detector performance varies across both demographic groups and image-generation methods. We evaluated ten generated-image detectors on 9,000 bona fide and 9,000 generated images from three generators, balanced across perceived gender presentation and Monk skin-tone bands. At fixed thresholds calibrated to target a pooled 1% false-positive rate, Persona deepfake v3 and AI-generated Image Detector (AIDE) achieved false-negative rates of 3.3% and 8.7%, respectively. However, performance varied across demographic cohorts and generators, with relative detector performance reversing under some conditions. Robustness to image processing also differed sharply: Under JPEG compression at quality 92 and unchanged detector thresholds, AIDE’s false-negative rate rose from 8.7% to 97.6%, while Persona’s changed from 3.3% to 3.4%. These results demonstrate that aggregate performance can obscure substantial variation across populations, generators, and deployment conditions.