Sex classification from hand X-ray images in pediatric patients: How zero-shot Segment Anything Model (SAM) can improve medical image analysis

Abstract

The potential to classify sex from hand data is a valuable tool in both forensic and anthropological sciences.This work presents possibly the most comprehensive study to date of sex classification from hand X-ray images.The research methodology involves a systematic evaluation of zero-shot Segment Anything Model (SAM) inX-ray image segmentation, a novel hand mask detection algorithm based on geometric criteria leveraginghuman knowledge (avoiding costly retraining and prompt engineering), the comparison of multiple X-rayimage representations including hand bone structure and hand silhouette, a rigorous application of deeplearning models and ensemble strategies, visual explainability of decisions by aggregating attribution mapsfrom multiple models, and the transfer of models trained from hand silhouettes to sex prediction of prehistorichandprints. Training and evaluation of deep learning models were performed using the RSNA Pediatric BoneAge dataset, a collection of hand X-ray images from pediatric patients. Results showed very high effectivenessof zero-shot SAM in segmenting X-ray images, the contribution of segmenting before classifying X-ray images,hand sex classification accuracy above 95% on test data, and predictions from ancient handprints highlyconsistent with previous hypotheses based on sexually dimorphic features. Attention maps highlighted thecarpometacarpal joints in the female class and the radiocarpal joint in the male class as sex discriminanttraits. These findings are anatomically very close to previous evidence reported under different databases,classification models and visualization techniques.