The brain is continuously active, even without external input or task demands. This so-called resting-state activity exhibits a highly specific spatio-temporal organization. However, how exactly these activity patterns map onto the anatomical and functional architecture of the brain is still unclear. We addressed this question in the human visual cortex. We determined the representation of the visual field in visual cortical areas of 44 subjects using fMRI and examined resting-state correlations between these areas along the visual hierarchy, their dorsal and ventral segments, and between subregions representing foveal versus peripheral parts of the visual field. We found that retinotopically corresponding regions, particularly those representing peripheral visual fields, exhibit strong correlations. V1 displayed strong internal correlations between its dorsal and ventral segments and the highest correlation with LGN compared with other visual areas. In contrast, V2 and V3 showed weaker correlations with LGN and stronger between-area correlations, as well as with V4 and hMT+. Interhemispheric correlations between homologous areas were especially strong. These correlation patterns were robust over time and only marginally altered under task conditions. These results indicate that resting-state fMRI activity closely reflects the anatomical organization of the visual cortex both with respect to retinotopy and hierarchy.