Impact of peer review audit on occupational health report quality

Background In a previous report, we described the implementation of a formal process for peer review of occupational health (OH) reports and a method of assessment of the outcomes of this process. The initial audit identified that 27% of OH reports required modifications. Aims To assess formally, following implementation of this process, if changes in practice had occurred, i.e. whether fewer deficiencies were being identified in reports. Methods We repeated a prospective internal audit of all peer reviewed OH reports between September and November 2011. We used an abbreviated assessment form, based on questions 4–8 and 10–12 of the modified SAIL (Sheffield Assessment Instrument for Letters), with four possible outcomes: no action, no changes made to report following discussion with author, changes made without discussion with author and changes made following discussion with author. Results One hundred seventy-three reports by 10 clinicians were audited. The audit identified a 13% reduction in OH reports requiring modifications (from 27 to 14%) compared with the previous cycle. Where modifications were required, 8% of these were related to minor typographical, spelling and grammar errors and 6% were for more complex reasons. Implementation of this process also produced a reduction in clinical complaints about OH reports from customers, from three in the preceding year to none 2 years later. Conclusions Peer review improved the standard of OH reports and was associated with a reduction in customer complaints about reports.


Introduction
The role of peer review (i.e. structured evaluation of clinical work by colleagues in the same field) is expanding within clinical practice [1] as part of clinical governance and quality improvement [2,3] and with revalidation now established in the UK [4]. In a previous article [5], the importance of occupational health (OH) reports to managers and human resource professionals was highlighted, with issues relating to reports being identified as one of the commonest causes of complaints and customer dissatisfaction. A formal process for peer review of OH reports for selected customers was implemented and a method of assessing the outcomes of this process established. The initial audit identified that 27% of OH reports required modifications. Eighteen per cent of these related to minor errors, while in 9%, there were more complex reasons. Although the clinicians involved cited the process as a valuable educational tool and reported change in their practice, this had not been formally assessed. The purpose of this audit cycle was to assess formally, following implementation of a peer review process, whether changes in practice had in fact occurred, i.e. whether fewer deficiencies were being identified in reports, with a resulting improvement in the overall standard of reports.

Methods
A prospective internal audit of all peer reviewed OH reports was repeated between September and November 2011. As previously, we used an abbreviated assessment form, based on questions 4-8 and 10-12 of the modified SAIL (Sheffield Assessment Instrument for Letters) [6,7]. Four key aspects of reports (administrative, response, professional issues and clarity) were graded on a 3-point rating scale (below expected, satisfactory and above expected) or were judged 'not applicable'.
The peer review process detailed in the original article [5] was unchanged. One of four possible outcome options remained, i.e. no action, no changes made to report following discussion with author, changes made without discussion with author or changes made following discussion with author.
The completed forms were collated and analysed via the Survey Monkey™ online tool (SurveyMonkey.com, LLC, California, USA).
As this was a service audit involving the analysis of anonymized data from the outcome of the peer review process, ethical approval was not required. Management approval to undertake this work was obtained.

Results
Eight reviewers (four consultant occupational physicians, one speciality registrar and three senior nurses) peer reviewed 173 reports by 10 clinicians over the 3-month audit period. The 10 clinicians whose reports were peer reviewed comprised the eight peer reviewers, one sessional doctor and an OH adviser. Each peer reviewer assessed a number of different clinicians' reports and each clinician had reports assessed by a variety of reviewers. On this occasion, 14% of reports required modifications, compared with 27% previously. One year on, 86% of reports required no change, an increase from 71% in the previous year. The Fisher's exact test demonstrated that the improvement in the standard of reports (Table 1) was statistically significant (P < 0.01). Where modifications were required, 8% of these were the result of minor errors and 6% for more complex reasons. Examples of issues identified are detailed in Table 1. The minor errors (typographical, spelling and grammar) were unchanged and remained the key reason for modifying reports. Among more complex reasons, reports not being clear and understandable to the intended readership and failure to address all the referrer's questions remained areas for improvement. There was a substantial improvement in review arrangements being made clear in reports, with a 100% satisfactory score.
In addition to the audit process, internal consistency between the eight reviewers was formally assessed using 10 randomly selected reports that were separately assessed by each reviewer. Fleiss's kappa statistic, a measure of agreement between three or more observers when the response is a group or category, was used for the calculation and the results are presented in Table 2. The majority of questions demonstrated fair or higher levels of agreement and only 2 out of the 11 had less than chance agreement between reviewers.

Discussion
The repeat audit identified a 13% reduction (from 27 to 14%) in OH reports requiring modifications. This may be related to numerous factors: a desire by the clinician to meet the standards, active feedback by the reviewers on specific areas for improvement and/ or knowledge that the report would be reviewed by a peer.
This finding supports the evidence that medical audit feedback together with educational measures has some success in changing practice [9,10], particularly when delivered frequently and with specific suggestions for improvement [10]. The main strengths of this audit were the use of a validated assessment tool and the simplicity of the process. Internal consistency between reviewers was also addressed. This identified a good level of agreement for most questions and identified potential areas (e.g. adherence to contractual, ethical and legal boundaries) for reviewer training to improve our process further.
There were some concerns that peer review would breed complacency among clinicians, in the knowledge that their report would be 'double checked' and any deficiencies identified and actioned by the reviewer before being sent. We did not find any evidence to suggest this happens in practice. We intend to repeat the audit at intervals to assess if changes in practice are maintained. In time we hope that even fewer deficiencies will be identified and that peer review of samples, rather than of all reports, can be implemented.
Notably there were no complaints relating to the content of OH reports for these customers 2 years after the introduction of peer review compared with several the year before implementation. We will continue to monitor this. Customer feedback on OH reports since the process was formalized has also been positive although further work is required in terms of a formal survey.
We conclude that this peer review process not only improved the standard of OH reports but was also associated with a reduction in clinical complaints related to reports.

Key points
• In this study, peer review was a useful tool in improving the standard of occupational health reports.
• Its establishment has also led to a reduction in customer complaints.
• Inter-observer evaluation is an important factor in the interpretation of results and can highlight potential areas for reviewer training and process improvement.

A memorandum prepared by the Education Panel and approved by Council for submission to the General Medical Council.
OccupatiOnal medicine is the practice of clinical, scientific and other skills in preventive medicine with reference to occupation and the working environment.
Since the latter is the environment in which most people spend a large part of their time, all students should be taught to appreciate the importance of its possible effects on the health of those at work. Occupation is of significance to the practice of medicine in several ways. The nature of a man's work may be wholly or partly responsible for his illness, and to arrive at a correct diagnosis promptly, the doctor must be aware of these possibilities. More commonly the patient's work is important in the management of his illness; the economic and social disturbance of time off work, the need for complete rehabilitation in the shortest period, the possibility or desirability of return to a former job, and consideration of suitable alternative work where there is some residual disability, should be constantly in the doctor's mind. Furthermore, a doctor, whether engaged in public health practice or not, should appreciate the influence of occupation on the health and well-being of a community.
Some of the skills of occupational medicine are common to medicine as a whole and the undergraduate will not require special training within the occupational setting; for example, the treatment of accidents, which is a part of occupational medicine, will be included in the general training of students. Much of the extra knowledge required by occupational medical officers, such as the recognition of dangerous environments and the management and prevention of diseases caused by them, is properly provided by postgraduate teaching. The undergraduate period of training should not be used to train embryo industrial medical officers any more than other types of specialist.
The following recommendations are made for the teaching of occupational medicine to undergraduate students: 1 Ideally there should be frequent reference to the occupational aspects of disease in all clinical teaching.
2 Since it is too casual an approach to the teaching of a subject to leave it to occasional references by different departments, a positive attempt must be made to inculcate in the student an understanding of the significance of the patient's environment.
3 Properly organized works visits are valuable to students in demonstrating the conditions of work in selected industries, emphasizing some of the occupational hazards and showing the special type of medical organization required in industry. 4 An additional means of teaching the subject is through the medium of the socio-medical case conferences which are conducted in hospitals by many departments of social and preventive medicine, in conjunction with the consultant staff.
In summary, the aim of these activities should not be instruction in occupational medicine as such, and the subject must not be presented so that it is an extra dissociated from the rest of the undergraduate course. The teaching must clearly be a part of the general education of students in the role of environment in the causation and management of ill health.