Episodic memory is a core feature of Alzheimer's disease (AD) and mild cognitive impairment (MCI). Impaired episodic memory in AD results from the dysfunction of an integrated network and involves both gray and white matter pathologies. We explored the neural correlates of episodic memory in AD, MCI and healthy aging by correlating a measure of episodic memory with hippocampal volume and fractional anisotropy (FA) and mean diffusivity (MD) of the cingulum and fornix. Episodic memory was associated with hippocampal volume and MD of the cingulum and fornix. In contrast, there were fewer significant associations between episodic memory and FA. These findings support a relationship between episodic memory and hippocampal circuitry, and suggest that MD is a more sensitive marker of decreased white matter integrity in the study of AD and MCI than FA. Furthermore, MD was significantly associated with hippocampal volume, indicating that white matter pathology is not completely independent of gray matter pathology. However, the pattern of diffusivity differences in AD and MCI implies a more complex pathology than simply Wallerian degeneration.