Various studies have reported an association between physical activity and grey matter volumes. Some studies have suggested that this relationship may be moderated by sex, yet the direction is still under debate. Focusing on hippocampus and dorsolateral prefrontal cortex (dlPFC), we tested whether the association between regional grey matter volumes and self-reported physical activity differs between women and men. We examined this interaction in five European cohorts from the Lifebrain consortium (n = 1809; age range: 18-88 years). Effect sizes were first determined by linear models run separately for each cohort, then pooled across datasets in a random-effects meta-analysis. Contrary to our hypotheses, there was no evidence of a relationship between physical activity and hippocampal or dlPFC volumes, nor was there a moderation by sex. Our null findings raise the question of whether self-report questionnaires of physical activity, which commonly feature in big datasets, are sufficiently sensitive to capture a-presumably modest-association between physical activity levels and grey matter outcomes. We conclude that the reliance on self-report questionnaires of physical activity is sub-optimal for brain-behaviour analyses.