Processing units are interconnected in the visual system, where a sensory organ and downstream cortical regions communicate through hierarchical connections, and local sites within the regions communicate through horizontal connections. In such networks, neural activities at local sites are likely to influence one another in complex ways and thus are intricately correlated. Recognizing the functional importance of correlated activity in sensory representation, spontaneous activities have been studied via diverse local or global measures in various time scales. Here, measuring functional magnetic resonance imaging (fMRI) signals in human early visual cortex, we explored systematic patterns that govern the correlated activities arising spontaneously. Specifically, guided by previously identified biases in anatomical connection patterns, we characterized all possible pairs of gray matter sites in 3 relational factors: “retinotopic distance,” “cortical distance,” and “stimulus tuning similarity.” By evaluating and comparing the unique contributions of these factors to the correlated activity, we found that tuning similarity factors overrode distance factors in accounting for the structure of correlated fMRI activity both within and between V1, V2, and V3, irrespective of the presence or degree of visual stimulation. Our findings indicate that the early human visual cortex is intrinsically organized as a network tuned to the stimulus features.