Emerging ideas of brain function emphasize the context-dependency of regional contributions to cognitive operations, where the function of a particular region is constrained by its pattern of functional connectivity. We used functional magnetic resonance imaging to examine how modality of input (auditory or visual) affects prefrontal cortex (PFC) functional connectivity for simple working memory tasks. The hypothesis was that PFC would show contextually dependent changes in functional connectivity in relation to the modality of input despite similar cognitive demands. Participants were presented with auditory or visual bandpass-filtered noise stimuli, and performed 2 simple short-term memory tasks. Brain activation patterns independently mapped onto modality and task demands. Analysis of right ventral PFC functional connectivity, however, suggested these activity patterns interact. One functional connectivity pattern showed task differences independent of stimulus modality and involved ventromedial and dorsolateral prefrontal and occipitoparietal cortices. A second pattern showed task differences that varied with modality, engaging superior temporal and occipital association regions. Importantly, these association regions showed nonzero functional connectivity in all conditions, rather than showing a zero connectivity in one modality and nonzero in the other. These results underscore the interactive nature of brain processing, where modality-specific and process-specific networks interact for normal cognitive operations.