Population aging has prompted considerable interest in identifying modifiable factors that may help protect the brain and its functions. Collectively, epidemiological studies show that leisure activities with high mental and social demands are linked with better cognition in old age. The extent to which socio-intellectual activities relate to the brain's structure is, however, not yet fully understood. This systematic review and meta-analysis summarizes magnetic resonance imaging studies that have investigated whether cognitive and social activities correlate with measures of gray and white matter volume, white matter microstructure and white matter lesions. Across eighteen included studies (total n = 8429), activity levels were associated with whole-brain white matter volume, white matter lesions and regional gray matter volume, although effect sizes were small. No associations were found for global gray matter volume and the evidence concerning white matter microstructure was inconclusive. While the causality of the reviewed associations needs to be established, our findings implicate socio-intellectual activity levels as promising targets for interventions aimed at promoting healthy brain aging.