Beyond One-Way Mapping: Linking Model-Brain Asymmetry to Behavioral Predictions in Visual Object Recognition

Advancements in artificial neural networks (ANNs) have yielded object recognition models that closely mimic the primate ventral visual pathway. Traditional evaluation metrics focus mainly on how well ANN units predict neural activity, often overlooking the bidirectional nature of this relationship. In this study, we investigate the symmetry of predictive relationships between ANN components and inferior temporal (IT) neurons and explore its implications for aligning computational models with primate behavior. We conducted large-scale neural recordings from 288 sites across the IT cortex in two macaques engaged in 45 binary object discrimination tasks using 1,320 naturalistic images. Human behavioral data were collected from 80 participants, achieving an image-level reliability of 0.89. Our analysis revealed significant asymmetries in the bidirectional predictive relationships between ANN units and neural responses. By employing linear regression and centered kernel alignment (CKA), we tagged two classes of ANN units: “best” units (top 10th percentile explained variance, EV) demonstrated significantly higher CKA values compared to all units (p<0.0001) while “worst” units (bottom 10th percentile EV) showed significantly lower CKA values. This asymmetry was consistent across multiple architectures, including Vision Transformer (ViT), ResNet50v2, Inception-v3, and AlexNet. Crucially, we found that the “best” ANN units more accurately predicted both human and macaque object discrimination performance compared to the “worst” units (p<0.05, permutation test). This relationship also remained consistent across different object categories. Interestingly, monkey IT neurons identified as “best” units as predicted by other monkey ITs also demonstrated a similar enhanced prediction of human behavior, suggesting potential shared neural mechanisms across species. Taken together, our findings underscore that developing human-like object recognition in ANNs requires optimizing neural prediction accuracy and jointly ensuring representational symmetry with biological systems.

Recommended citation: Muzellec, S. & Kar, K. (2025). Beyond One-Way Mapping: Linking Model-Brain Asymmetry to Behavioral Predictions in Visual Object Recognition. NETI Workshop.