The peer-review process is an invaluable service provided by the professional community, and it provides the critical foundation for the advancement of science. However, there is remarkably little systematic guidance for individuals who wish to become part of this process. This paper, written from the perspective of reviewers and editors with varying levels of experience, provides general guidelines and advice for new reviewers in neuropsychology, as well as outlining benefits of participation in this process. It is hoped that the current information will encourage individuals at all levels to become involved in peer-reviewing for neuropsychology journals.
On Becoming a Peer Reviewer for a Neuropsychology Journal
The peer-review process of professional journal publishing is as important to the scientific enterprise as developing reliable and valid measures, well-characterized samples, and appropriate statistical techniques. However, many professionals have limited involvement in reviewing manuscripts for scientific publication. Whether one is well grounded by scientific training or not, beginning involvement in the review of journal manuscripts typically occurs with little to no guidance. In addition, empirical evaluation of the peer-review process has in some instances revealed a disappointing level of agreement between peer reviewers (e.g., Rothwell & Martyn, 2000). Probable factors in producing low reviewer agreement are the general lack of direction provided to reviewers, who are often left to “figure it out” on their own, as well as a disproportionate number of reviewers who are less experienced and more junior in their careers. The current article will review some benefits of being an ad hoc reviewer for journals, and outline some points to keep in mind while conducting reviews.
Why Become an Ad Hoc Reviewer?
There are many reasons to serve as an ad hoc reviewer for scientific journals. It may seem a truism that volunteerism is its own reward, but nevertheless the following is a list of specific benefits that can accrue to reviewers:
Staying current with the literature. By evaluating manuscripts, the reviewer is exposed to the most recent, cutting edge research in the field, even before it is published. As one reads a manuscript, one takes in concise summaries of relevant literature in the Introduction section, new procedures detailed in the Methods section, and the latest findings in the Results section. Although not all submissions will be published, reviewers are privy to data that few others have seen.
Developing your professional diversity. If your work primarily involves conducting clinical assessments or interventions, then reviewing manuscripts might provide an outlet for your “researcher side.” If you primarily conduct basic science research, then reviewing clinically relevant findings might widen the scope of your own work or viewpoint. If you primarily teach, then reviewing might expand your breadth of the basics for what you teach, as well as offer teaching opportunities for your students as co-reviewers (discussed subsequently).
Shaping the field. By commenting on the work of your peers, you can sometimes guide the focus of a particular paper. Your editorial suggestions could improve methodologies and points of view within neuropsychology.
Taking advantage of an opportunity to provide service to the scientific community. At some academic institutions, one way of demonstrating scientific productivity and involvement in your discipline is through service in peer review of manuscripts related to your areas of investigation. In fact, being asked to be a peer reviewer represents recognition of your own work, which has attracted the attention of the journal's editorial board. Frequent involvement in publishing your own work and peer reviewing manuscripts for the same journal can lead to an invitation to serve on the editorial board, which is widely considered to be a distinctive recognition of your own scientific efforts. This is true whether one works in an academic setting or in a private practice setting. Interestingly, clinical neuropsychology is one of the few healthcare specialties in which private practitioners are relatively frequently involved in peer-reviewed journal publishing.
Giving back to the field. Many neuropsychology journals are affiliated with professional organizations. For example, Archives of Clinical Neuropsychology is the official journal of the National Academy of Neuropsychology, The Clinical Neuropsychologist is the official journal of the American Academy of Clinical Neuropsychology, The Journal of the International Neuropsychological Society is the official journal of the International Neuropsychological Society, and Applied Neuropsychology is the official journal of the American College of Professional Neuropsychology. By participating in the review process, you are sharing your individual skills and professional perspective with peers in your specialty, including those who constitute a much broader readership than the membership organization, such as psychologists who are not neuropsychologists and individuals outside our discipline altogether, such as physicians.
Improving the quality of your own work. Just as reading recently published studies provides new avenues for one's own studies, reviewing manuscripts submitted for publication can provide ideas about the methods being used and research questions being investigated by peers with similar interests to one's own. Though reviewers must be careful to avoid intellectual plagiarism, there is potential for learning about new techniques that might apply to your existing line of research.
How Does One Become an Ad Hoc Reviewer?
If you are interested in reviewing manuscripts for a journal, but do not know where to begin, there are many ways to become involved in the process. These fundamental ideas may require patience on your part, but will improve your chances of being given an opportunity.
Directly contact journal editors. A fact known all too well to journal editors is that it can be quite difficult at times to find suitable reviewers who will be able to provide a timely review of a manuscript, in part because the most skilled and experienced reviewers are often the busiest. Editors therefore need a long list of potential reviewers, and often are looking for names to add to that list. This remains true, even in the age of electronic databases to which editors have access. A brief email stating your name, basic qualifications, and areas of interest is usually sufficient. Most editors will be very receptive to anyone expressing interest in reviewing articles and excited at the prospect of another resource!
Talk to your peers. It is likely that some of your colleagues are already involved in the review process and can provide you with contact information. Alternatively, your peers can suggest your name as a potential reviewer to the editor. Finally, you could offer to co-review a paper with a peer who is already an ad hoc reviewer. This could give you a chance to “prove yourself” to the editor. If you want to co-review a manuscript, then you should contact the editor about this in advance, as submitted manuscripts are confidential outside the review process and a co-reviewer would usually receive an acknowledgement in one of the issues of the journal.
Publish. If you produce research, then eventually someone will “cold call” you and ask for your opinion on a manuscript. However, this latter method may take some time and is not very efficient.
What Does One Do as an Ad Hoc Reviewer?
If you were interested in reviewing, have contacted an editor, and received your first manuscript for review, how should you approach the assignment? As with writing a paper or grant, evaluating a patient, or teaching a course, there is no right or wrong way to review a paper, but there are some basic guidelines one might follow.
Before thinking of specific review suggestions, we suggest first taking a mental step backward to reflect on the bigger picture of the task at hand. It may help to consider that a colleague has taken a great deal of time and energy to conceptualize a problem, identified data that can address the problem, gathered and analyzed the data, and finally written down all the relevant information related to the specific project at hand. Whether all of these steps were effective and led to a publishable paper or not, each of the individuals who is willing to undertake this research activity does so with the belief that new and important information is being gathered for sharing with peers. It is therefore the general goal of a peer reviewer to be helpful to the author(s), even if the review identifies numerous issues that may ultimately prevent publication. Ideally, there is a spirit or tone set in the reviewing process, which is devoid of bias, competition or envy, arbitrariness, and harshness. Stated more positively, as it was first established by the Royal Society of England in the seventeenth century, effective peer review as a component of scientific journal publishing has been conceptualized as a professional consultation that is delivered in a respectful and timely manner on an intellectual matter, with the process being confidential (cf. Moore, 2005). The peer reviewer is serving as a consultant to the author(s) and with the editor, and is essential to the integrity of the scientific publishing process in that an editor cannot be a content expert in all fields (Moore, 2005). Therefore, the duties of a peer reviewer are not to be viewed lightly.
Some reviewers prefer to start with a brief introductory paragraph that summarizes the article and its major findings. This section might also highlight any broad strengths and weaknesses of the manuscript. Although not required, this approach has the advantage of assuring the editor and author that you read the article and understand its main points. Alternatively, other reviewers prefer to skip this step and get right to the critical comments. The key throughout is to evaluate the manuscript, and not simply annotate it. The authors, in particular, already know what they did and what their study was about, and long detailed summaries are not helpful. The reviewer's comments should be evaluative.
Reviews can be structured based on the sections of a typical manuscript (e.g., Introduction, Methods, Results, Discussion), and relevant comments are only made using the outline of those sections. Alternatively, reviews can lay out comments in order of importance (e.g., most important to least important), perhaps with headings that identify “major” and “minor” points for the authors to consider. Some form of structure, if only to present comments in the order of the manuscript pages is better, than presenting comments in a haphazard manner. More structure usually allows the editor and author to follow your reasoning and act accordingly (e.g., make a reasoned decision on the manuscript).
Below are some suggestions to consider when reviewing specific sections of a manuscript.
Does the introduction begin generally, and then become more focused? Bem (1987) noted that an empirical article often has an hourglass shape, starting broadly, but narrowing its scope as the Introduction moves to the Methods section. The Results section is also narrow, but gradually widens its scope throughout the Discussion section.
Is the relevant and most recent literature cited and reviewed? Although classic studies in neuropsychology might set the stage, more contemporary studies usually provide more relevant information. For example, a 1975 article using the WAIS might have been relevant in a prior era, but a 2005 article using the WAIS-III is more relevant to the present reader.
Are all critical topics and/or questions adequately covered in this section? Do gaps exist in the authors' line of reasoning? Does this section “flow”?
Is the length appropriate? Is it too long and does it cover unnecessary background information? Many inexperienced authors write lengthy, dissertation style introductions, which take up valuable journal space and thereby may unwittingly put the article's acceptance into some jeopardy. A reviewer can help the author by recommending the introduction be tightened and condensed. Conversely, some Introductions can be too brief, leaving an uninformed reader without any context.
Is a specific purpose of the study stated?
Are specific hypotheses clearly stated? And are the hypotheses properly motivated by the background provided in the Introduction?
Materials and Methods
Methods sections are generally quite specific and detailed, and it is difficult to comment on all the issues and possible problems that a reviewer might encounter. This is where identifying key issues is quite important. Some guidelines for reviewing this section of paper:
Is the sample appropriately/adequately described? To utilize research findings, readers need to know who the sample was, what were the recruitment procedures, how were participants assigned to groups. Some information about demographic characteristics should be reported (e.g., means, standard deviations, ranges). Age, education, and sex might be most relevant in some studies, whereas Glasgow Coma Scale and length of loss of consciousness might be most relevant in others. Given the increasing diversity of the population, information regarding race/ethnicity and primary language are oftentimes necessary.
Was approval received from the local Institutional Review Board? Was informed consent obtained from each participant?
What were the methods of data collection? Are methods adequately described so that the study could be replicated? For example, it is more informative to indicate that “age-corrected standard scores from the test manual of the California Verbal Learning Test – II were used” than “the California Verbal Learning Test – II was used.”
If an intervention was used, is it also adequately described so that readers can understand what was done? Sometimes a citation to another published article is sufficient; sometimes it is not.
What are the statistical analyses utilized? Are the dependent variables clearly stated? Are the statistics appropriate for the questions? Are relevant covariates considered? Would non-parametric tests be more appropriate? Are there sample size/power concerns (e.g., too many analyses and/or no alpha correction)? Would other analytic techniques better answer the same question? Should the author(s) drop/add any analyses?
Are the necessary results presented? Are the relevant statistical values and degrees of freedom reported, along with p-values? If appropriate, are effect sizes reported?
Is the presentation of the results understandable to someone who does not do research in this area? Within the body of the text, subheadings might make results easier to read and understand. Within tables, clear column and row headings can increase the value of the table.
Are figures and tables appropriately titled? Do the authors use notes that are clear and informative? Are all abbreviations used within figures and tables defined in notes, such that readers will not have to search back through text to grasp their meaning?
Do the authors include analyses that were not discussed in the Introduction and Method sections? Conversely, are analyses missing that were mentioned earlier in the paper?
Do the authors commit some of the “deadly sins” outlined by Millis (2003)? In this paper, Millis highlights some common statistical errors made in neuropsychology manuscripts (e.g., multiple comparisons, low power, ignoring missing data), as well as provides guidance for correcting these errors.
Does the manuscript need an expert statistical review? Some statistical analyses can be quite challenging for the average reviewer, and it is appropriate to let the editor know that you do not have the expertise to fully comment on results. Consider that if these results are too complex for you as the reviewer, then they may also be too complex for the typical journal reader, in which case the authors must do a better job of explaining their analyses.
In the Discussion section, the results are often summarized, integrated, and put into context with the existing literature.
Although the primary sections of the manuscript associated with the preceding points are most important to evaluate when reviewing a manuscript, reviewers are also expected to comment on other components of the manuscript, if they are relevant to the question of publishability. The following are common points on which the journal editor will appreciate guidance from a reviewer.
Are the conclusions supported by the findings? Some authors tend to over-interpret their findings. In a manner of speaking, data are specific to that single study, and they may have limited relevance outside of that single study. As such, data should not be stretched to topics or levels of meaning for which they are not relevant. Reviewers should offer constructive suggestions to the authors for correcting over-interpretation of data.
Are the findings generalized to the appropriate populations and/or settings? For example, a finding that symptom validity testing incorrectly classified children with Learning Disorders should not necessarily be extended to adults with Learning Disorders or even children with other development disorders.
Are the results put into context with the existing body of literature? If they stand apart from other studies, do the authors discuss why this might be?
Do the authors simply restate the Results section? Although authors often highlight certain findings in this section, they should move beyond a restating of the findings and actually discuss and integrate their findings.
Are new results and/or data presented that have not been discussed elsewhere? Typically, results should be presented in the Results section, not the Discussion section.
Do the authors provide a conceptual framework and/or presentation for how and when to use the findings? Finding a statistically significant result does not necessarily answer the “so what?” question. What is gained by the completion and reporting of this study?
Are the appropriate caveats and limitations to the article mentioned and discussed?
Are future research directions provided?
Has the manuscript been carefully prepared in the appropriate style of the journal (e.g., APA writing style)?
Does the theme/content of the manuscript fit with the aims of the journal?
Does the manuscript address something new and add something to the existing literature?
How adequate is the general writing style (e.g., grammar, punctuation, and readability)? When recommendations are made for correcting or improving the writing, it is best to be very specific. That is, rather than suggest that the authors improve their writing and remove errors, it is much better to identify the page and line in which the problems can be found, and to suggest the solution.
Is there an Acknowledgment section? Does it mention the funding agency? What was the role of the funding agency in the project? Were possible conflicts of interest of the authors noted to the reader?
Is the Reference section correctly formatted for the specific journal? Are all references mentioned in text provided in the reference section and vice versa? Are there errors in specific references? If references are recommended by a reviewer, it is best to provide the author the complete citation, to avoid confusion. If at all possible, avoid suggesting that the authors cite your own work; this is viewed as self-promotion and to be avoided, unless the citation in question is truly seminal and the omission would substantially weaken the instructive nature of the article for the reader.
Other Tips on Reviewing
Now that you have completed your initial review, you might wonder how to hone your new craft. Below are some suggestions for becoming a more refined ad hoc reviewer.
The pros and cons of the peer review process in scientific publishing have been discussed at great length in numerous journals outside of neuropsychology. To be sure, there are both detractors (Pravinkumar, 2003), who question whether peer review should be bypassed, and those who choose to improve upon the prevailing practices of peer review (Ray, 2002). We agree with Ray, who among numerous other points, has indicated that editorial decisions may be weakened if peer reviews are imprecise and variable in quality. He makes a recommendation with which we agree, and which is in concert with the suggestions we have provided.
As in all areas of science, new “fresh-perspectives” are always needed in neuropsychology and the current article is intended to provide general guidance to those wishing to become new reviewers for their professional journals. As pointed out earlier, the review process can offer personal and professional benefits to those willing to undertake this invaluable task. We trust that this article will encourage neuropsychologists who have never embarked in this process to consider joining and to persuade those who have reviewed in the past, but no longer do so, to rejoin the process.
Peer reviewers, the bedrock of medical journal objectivity, require more training and experience. One simple solution might be for editors to provide them with a short review of best practices along with the checklist of core elements to consider. (Ray, 2002, p.772)
Provide critiques in a timely manner. Try to adhere to deadlines. If you are late on a review, it slows down the process and prevents the editors from achieving a timely decision on the manuscript, and the authors from receiving timely feedback. If you will be late with the review, contact the editor to give an estimate of when it will be completed.
Reviewing takes practice. Similar to writing papers, interviewing patients, and preparing lectures, do not expect that your initial review will be your best work. However, it is likely that reviewing more papers will make you better at it.
Readily ask for advice/guidance. You do not need to be an expert on every topic to be a good reviewer. Know your areas of expertise within neuropsychology (e.g., specific tests, specific disorders) and research (e.g., study design, statistics). When the manuscript exceeds those areas, do not be afraid to consult with others. You could ask colleagues for their thoughts (without divulging the entire manuscript). You could refer to the literature to see how others have addressed similar problems. You can let the editor know that certain aspects of the manuscript fall outside your knowledge base, and, in some instances, you should decline the solicitation to be a reviewer if the topic is not sufficiently within your knowledge and expertise. The boundary on when to decline a topic based on lack of relevant knowledge is perhaps best illuminated by asking yourself the question, “If I was the author, would I want a reviewer who has only my degree of knowledge to pass judgment on my hard work?”
Learn from the other reviewers. Once you submit your review, most journals will allow you to see the other completed reviews. Use this as opportunity to see if your specific comments and ultimate recommendation for the disposition of the paper align with the reviews of potentially more experienced reviewers, and learn from the issues brought up by other reviewers that you may have missed.
Cover the entire manuscript. Try to find strengths and weaknesses in all components of the submission. Resist the urge to stop reviewing the manuscript because you found a “fatal flaw” in the Introduction or Methods sections. Your feedback to the authors may lead the editor to reject the manuscript, but the feedback itself can still be very valuable to the authors in their future research endeavors. It is easy to be critical; it is better but more challenging to help the authors improve their paper by adopting a constructive tone to your critique.
Provide helpful comments to the author(s). Reviewers can help shape manuscripts (and ultimately the field). By providing constructive and concrete comments, a reviewer can provide direction that assists the author in building a better manuscript. Unclear suggestions (e.g., “statistical analyses are wrong”) provide no real instruction to the authors when revising their work. In a survey of corresponding authors of a psychology journal, Nickerson (2005) found authors want specific information on problems in their manuscripts and concrete suggestions to improve those problems.
At all costs, avoid tirades and unnecessary negativity in reviews. There is no place in a review for personal attacks or vendettas of any sort. If you disagree with the author, then state so in a constructive manner. Try to help the author.
Avoid statements about your recommendations for acceptance or rejection of the manuscript. Most editors will prefer that the reviewers' comments to the authors focus on the strengths and weaknesses of the submission, rather than indicating whether the reviewer thinks the paper should be published. If one reviewer is recommending publication and others are not, this can be confusing and frustrating to the authors. You can indicate your enthusiasm/degree of concern about the work, but reserve comments such as “I think this paper should definitely be published” for the section of confidential communication to the editor.
Unpublished manuscripts are confidential. Until a manuscript is accepted and “in press,” you cannot reference it. Refrain from contacting authors or letting them know in any manner that you reviewed their work. Do not share the findings with your peers. A corollary of this point is that you, as the reviewer, also have anonymity. Editorial staff will not divulge who reviewed specific manuscripts.
Signing reviews remains a controversial practice in the review process. In favor of this practice, it makes the process more transparent (e.g., authors know how specific individuals feel about their work). This may carry more weight in the revision process, especially if the reviewers are more senior members of the field. Conversely, confidentiality is removed, which may affect critiques. Reviewers should check with journal editors about their preferences for signing reviews.
Conflict of Interest
The authors of this article hold editorial positions at several neuropsychology journals, including Archives of Clinical Neuropsychology (KD, HW, RJM), Applied Neuropsychology (CRR), Journal of Clinical and Experimental Neuropsychology (WGvG, DT), and The Clinical Neuropsychologist (JJS).
We would like to gratefully acknowledge the contributions of the Guest Action Editor, Arthur MacNeill Horton, Jr., Ed. ABPP, ABPN, and the anonymous reviewers of this manuscript.