Increasing age and carrying an APOE ε4 allele are well established risk factors for Alzheimer's disease (AD). The earlier age of onset of AD observed in ε4-carriers may reflect an accelerated aging process. We recently reported that APOE genotype modulates brain function decades before the appearance of any cognitive or clinical symptoms. Here we test the hypothesis that APOE influences brain aging by comparing healthy ε4-carriers and non-carriers, using the same imaging protocol in distinct groups of younger and older healthy volunteers. A cross-sectional factorial design was used to examine the effects of age and APOE genotype, and their interaction, on fMRI activation during an encoding memory task. The younger (N=36; age range 20-35; 18 ε4-carriers) and older (35 middle-age/elderly; age range 50-78 years; 15 ε4-carriers) healthy volunteers taking part in the study were cognitively normal. We found a significant interaction between age and ε4-status in the hippocampi, frontal pole, subcortical nuclei, middle temporal gyri and cerebellum, such that aging was associated with decreased activity in e4-carriers and increased activity in non-carriers. Reduced cerebral blood flow was found in the older ε4-carriers relative to older non-carriers despite preserved grey matter volume. Overactivity of brain function in young ε4-carriers is disproportionately reduced with advancing age even before the onset of measurable memory impairment. The APOE genotype determines age-related changes in brain function that may reflect the increased vulnerability of ε4-carriers to late-life pathology or cognitive decline.