Objective: The advent of computerized assessments such as the ImPACT has led to increasingly widespread use of neuropsychological assessments in concussion management. The shift from paper-and-pencil to computerized assessments has allowed for increased access to evaluations with decreased cost and personnel needs; however, computerized assessments have not been directly compared to paper-and-pencil measures to establish comparable sensitivity to cognitive changes. The present study compares rates of decline on traditional paper-and-pencil tasks to rates of decline on the ImPACT. Method: 58 participants from a university-based sports concussion program received baseline and post-concussion assessments including both paper-and-pencil and ImPACT measures. All post-concussion evaluations were performed within 1 week post-injury. Change in performance from baseline to post-concussion was evaluated for all measures using reliable change indices. Results: 24.1% of athletes showed a reliable decline on at least one ImPACT index (Verbal Memory, Visual Memory, Reaction Time, and Visuomotor Speed) compared to 46.6% on at least one of four paper-and-pencil measures (HVLT-R Total Recall, BVMT-R Total Recall, Stroop Word Reading, and Symbol Digit Modalities Test). 25.9% of athletes showed a decline on at least one paper-and-pencil measure without showing a decline on any ImPACT indices. Conclusion: Our findings suggest that the ImPACT is less sensitive than paper-and-pencil measures in assessing post-concussion cognitive changes. The recognition-only format of the memory tests and the brief screening nature of the ImPACT may contribute to its reduced sensitivity, though other interpretations are possible.