Learning Sparse Semantic-Cortical Atoms for Multisubject Naturalistic fMRI Encoding

Alicia Zeng

NeurIPS 2026, oral · Sydney

PDF OpenReview NeurIPS The 2025 paper

Four rows, one per atom: food and meals, family members and relatives, numerical measurements, and religion and identity. Each row shows the atom’s brain decoder map on the flattened cortex of three subjects and the fsaverage mean, six story excerpts with the words that drove the atom highlighted, and a summary of its AutoInterp AUC, semantic coherence, best Neurosynth term and Yeo7 networks. — **Fig. 1: Four example semantic-cortical atoms.** Each row is one atom: its map in three subjects and on the average brain, the story moments that drive it, and how it scores. From the manuscript.
The manuscript’s legend, verbatim

**Figure 1: Four example semantic-cortical atoms.** Each row shows brain decoder maps for three example subjects together with the strongest activating TRs and surrounding context. The summary panels show the mean fsaverage map, AutoInterp AUC, semantic coherence z, the best-matching Neurosynth term and brain-map correlation, and significant Yeo7 network affiliations. Food and meals (a18) strongly activates posterior temporal cortex and lateral prefrontal cortex; family members (a23) and religion and identity (a10) activate medial prefrontal cortex and temporoparietal regions; and numerical measurements (a8) strongly activates right intraparietal sulcus and lateral prefrontal cortex, consistent with prior cognitive neuroscience findings. Because brain decoders are fit independently for each subject, any cross-subject consistency must arise from the common encoder. Yeo7 networks: DMN = Default Mode, FPN = Frontoparietal, DAN = Dorsal Attention, SMN = Somatomotor.

Semantic selectivity is fundamentally non-identifiable in high-dimensional correlated feature spaces.

Written for this page from the paper’s abstract. The paper is on OpenReview and is presented as an oral at NeurIPS 2026 in Sydney, with a poster in Poster Session 2. The PDF here and Fig. 1 are the manuscript as submitted, line-numbered and without author names; the camera-ready will replace them.

The short version

Encoding models built on language-model features predict cortical responses to natural speech well, and they remain hard to read. Three things stand in the way: selectivity is not identifiable in a correlated, high-dimensional feature space; the organisation of cortex stays implicit in voxelwise weight maps; and correspondence across people is handled only after the model is fit. This paper puts all three inside one model.

A multisubject coupled sparse autoencoder reconstructs the language-model features and predicts each subject’s brain responses at the same time. What it learns is a sparse basis of semantic-cortical atoms: each atom is a direction in meaning and, by construction, the pattern of cortical response that goes with it, in every subject at once. Semantic structure, cortical organisation and cross-subject correspondence are therefore explicit in the fitted object rather than recovered from it afterwards.

88%of a matched dense-feature ridge model’s prediction performance, retained
50atoms in the learned basis
k = 5atoms active per time point

In naturalistic story-listening fMRI, that basis keeps 88% of the prediction performance of the dense model with fifty atoms and five active per TR. Against matched two-stage baselines, which learn the basis first and fit the brain second, joint training gives the strongest overall trade-off between prediction and interpretability. The atoms are semantically coherent; they give more reproducible cross-subject maps than standard post hoc encoding-model analyses; they align with large-scale cortical structure from the Yeo 7 networks and from Neurosynth; and they transfer better than ridge to held-out subjects when data are scarce.

It is the successor to the 2025 paper, which unpacked the embedding into sparse concept atoms first and fit the encoding model second. Here the two steps are one, and the subjects are modelled together rather than one at a time.

Abstract

Naturalistic language encoding models based on large language model features can predict cortical responses well, but their learned representations remain difficult to interpret. Three challenges are central: semantic selectivity is fundamentally non-identifiable in high-dimensional correlated feature spaces, cortical organization remains implicit in voxelwise weight maps, and cross-subject structure is usually addressed only after model fitting. We introduce a multisubject coupled sparse autoencoder that jointly reconstructs language-model features and predicts subject-specific brain responses, learning a sparse basis of semantic-cortical atoms that makes semantic structure, cortical organization, and cross-subject correspondence explicit within a single model. In naturalistic story-listening fMRI, the model retains 88% of matched dense-feature ridge brain prediction performance using 50 atoms and k = 5 active atoms per TR. Relative to matched two-stage baselines, joint training yields the strongest overall tradeoff between prediction and interpretability. The learned atoms are semantically coherent and interpretable, recover more reproducible cross-subject brain maps than standard post hoc encoding-model analyses, align with large-scale cortical structure from Yeo 7 and Neurosynth, and transfer better than ridge to held-out subjects in the low-data regime. This framework provides a more interpretable basis for cognitive neuroscience analyses of semantic-to-brain encoding by linking semantic features to corresponding cortical response patterns across subjects.

BibTeX

@inproceedings{zeng2026atoms,
  title     = {Learning Sparse Semantic-Cortical Atoms for Multisubject
               Naturalistic {fMRI} Encoding},
  author    = {Zeng, Alicia},
  booktitle = {Advances in Neural Information Processing Systems 39 (NeurIPS 2026)},
  year      = {2026},
  note      = {Oral},
  url       = {https://openreview.net/forum?id=YcEZoSqrxa}
}