The functional organization of the frontal cortex is dynamic. Age-related increases in frontal functional responses have been shown during various cognitive tasks, but the cross-sectional nature of most past studies makes it unclear whether these increases reflect reorganization or stable individual differences. Here, we followed 130 older individuals’ cognitive trajectories over 20–25 years with repeated neuropsychological assessments every 5th year, and identified individuals with stable or declining episodic memory. Both groups displayed significant gray matter atrophy over 2 successive magnetic resonance imaging sessions 4 years apart, but the decline group also had a smaller volume of the right hippocampus. Only individuals with declining memory demonstrated increased prefrontal functional responses during memory encoding and retrieval over the 4-year interval. Regions with increased functional recruitment were located outside, or on the borders of core task-related networks, indicating an expansion of these over time. These longitudinal findings offer novel insight into the mechanisms behind age-associated memory loss, and are consistent with a theoretical model in which hippocampus atrophy, past a critical threshold, induces episodic-memory decline and altered prefrontal functional organization.