The object selectivity of nearby cells in inferior temporal (IT) cortex is often different. To elucidate the relationship between columnar organization in IT cortex and the variability among neurons with respect to object selectivity, we used optical imaging technique to locate columnar regions (activity spots) and systematically compared object selectivity of individual neurons within and across the spots. The object selectivity of a given cell in a spot was similar to that of the averaged cellular activity within the spot. However, there was not such similarity among different spots (>600 μm apart). We suggest that each cell is characterized by 1) a cell-specific response property that cause cell-to-cell variability in object selectivity and 2) one or potentially a few numbers of response properties common across the cells within a spot, which provide the basis for columnar organization in IT cortex. Furthermore, similarity in object selectivity among cells within a randomly chosen site was lower than that for a cell in an activity spot identified by optical imaging beforehand. We suggest that the cortex may be organized in a region where neurons with similar response properties were densely clustered and a region where neurons with similar response properties were sparsely clustered.