National calls to improve the performance and persistence of students from historically underrepresented backgrounds in science have led to a surge of research on inclusive, evidence-based teaching methods. Less work has revealed the effects of diversity support initiatives that improve campus climate and community cohesion. Here, we examine whether participation in the Biology Scholars Program (BSP) at Cornell University—a diversity support program at a prominent university—affects underrepresented racial minority (URM) student performance. We found that BSP participants are less academically prepared when they enter college but typically have GPAs similar to those of their non-BSP counterparts at graduation, thereby closing achievement gaps. Although the BSP appears to help URM students, we cannot assert that the BSP alone is responsible for these effects; future work should isolate effective strategies that contribute to student success. In response to these results, we lay out strategies that support programs could implement to maximize positive impacts.