Survey using incognito standardized patients shows poor quality care in China’s rural clinics

Abstract

Over the past decade, China has implemented reforms designed to expand access to health care in rural areas. Little objective evidence exists, however, on the quality of that care. This study reports results from a standardized patient study designed to assess the quality of care delivered by village clinicians in rural China. To measure quality, we recruited individuals from the local community to serve as undercover patients and trained them to present consistent symptoms of two common illnesses (dysentery and angina). Based on 82 covert interactions between the standardized patients and local clinicians, we find that the quality of care is low as measured by adherence to clinical checklists and the rates of correct diagnoses and treatments. Further analysis suggests that quality is most strongly correlated with provider qualifications. Our results highlight the need for policy action to address the low quality of care delivered by grassroots providers.

Physician quality, China, standardized patients

KEY MESSAGES

• We used standardized patients to measure the quality of health care in rural China.

• Clinicians completed only 18% of items on a checklist of recommended questions and exams.

• Only 26% of clinician diagnoses were correct.

• Clinicians provided medicine that was unnecessary/harmful in 64% of the interactions.

Introduction

Over the past decade, China has implemented comprehensive health system reforms and invested heavily in providing health care for the more than 600 million people who live in rural areas. Between 2000 and 2010 total health expenditure per capita in rural areas more than doubled, from 116 yuan ($18.4) per capita to 326 yuan ($51.7) per capita, mostly due to increased government investment (Yip et al. 2012). The majority of this investment has been aimed at expanding access to health care by increasing coverage under new public health insurance programmes (Eggleston 2012). By the end of 2010 around 96% of rural residents had insurance coverage (Liang et al. 2012).

Although China has undoubtedly made great strides in improving access to health care in rural areas since the start of reforms, a fundamental question that remains is whether reforms have led to actual gains in population health outcomes (Wagstaff et al. 2009; Babiarz et al. 2010; Liang et al. 2012). Even with greater access to health care, health outcomes may not improve if the quality of care is poor. Government attention has recently shifted towards improving health care services; however, little evidence exists on the quality of care currently delivered by providers in China’s rural areas (Eggleston et al. 2008; Yip et al. 2012).

The primary goal of this study is to provide objective evidence on the quality of primary care delivered by village clinicians in China’s rural areas. We provide estimates of the quality of care based on interactions between village clinicians and incognito standardized patients (SPs). The SP approach has long been considered the gold standard for assessing clinical practice in developed settings (Glassman et al. 2000; Peabody et al. 2000; Rethans et al. 2007; Mohanan et al. 2011; Das et al. 2012), and has recently begun to be applied in developing settings as well (Das et al. 2012). In our study, SPs were individuals recruited from the local community and trained to consistently and covertly present cases of common illnesses to clinicians. We use data collected during SP–provider interactions to evaluate providers on three main dimensions of care quality: (1) process quality (i.e. how closely clinicians adhere to disease-specific lists of recommended questions and physical exams), (2) the accuracy of diagnoses and (3) the appropriateness of treatment. In addition to describing the quality of care provided by village clinicians, we compare these results to a smaller sample of SP interactions with township health centre (THC) clinicians (the next level up in the tiered rural health care system). We also use additional data collected on providers prior to SP visits to assess how provider characteristics are correlated with different dimensions of quality as well as with fees charged for drugs and services. Finally, to provide additional perspective, we compare our results with results from a similar SP study of the quality of health care providers in rural and urban India.

Village clinicians and China’s rural health care reforms

Village clinicians are the front line of the rural health system in China. At the bottom tier of a three-tiered hierarchy, village clinicians provide primary care and public health services to China’s rural population under the management of THCs (the middle tier) (Babiarz et al. 2012). Although China does not impose gatekeeping requirements, the majority of rural patients is first seen in village clinics, often the only source of medical care in remote villages (Babiarz et al. 2010; Li et al. 2013). In fact, the rural insurance system is set up to encourage patients to first contact their village clinician for treatment for a specified set of basic health conditions. County hospitals at the top tier of the rural health system provide relatively expensive, specialized care (Babiarz et al. 2012).

Expanding the scope of public health service provision through village providers is a primary focus of China’s newest wave of health care reforms (Chen 2009; Yip et al. 2012). As part of these reforms, the government provides funding of 25 yuan ($3.97) per head for village providers to deliver a defined package of public health services to the population in their catchment areas. This payment is on top of all existing wage and salary payments, and is meant to cover the additional labour involved in providing vaccinations, time spent administering the rural health insurance system inside the village and other public health services. In terms of primary health care service provision, in the long-run, reforms envision village clinicians serving as gatekeepers to upper tiers of the system (Yip and Hsiao 2009; Yip et al. 2012).

Although village clinicians are shouldering an increased role as part of China’s rural health system reforms, there is currently little objective evidence on the quality of care they provide (Eggleston et al. 2008). Indeed, some scholars have voiced concerns that primary care providers in rural areas lack the education and skills necessary to address the country’s future health challenges (Eggleston et al. 2008). Non-communicable diseases are already widespread in both rural and urban areas (Yang et al. 2008, 2010). Heart conditions, cancer and cerebrovascular disease are now the leading causes of death in rural China (He et al. 2005). At the same time, the burden of many infectious diseases—such as tuberculosis and hepatitis B—remains significant (Wang et al. 2008). It is unclear whether a minimally educated village clinician has the training required to tackle this range of sophisticated medical conditions.

Another often-voiced concern is that primary health care providers have strong incentives for unnecessary care and waste (Eggleston et al. 2008; Yip and Hsiao 2009; Yip et al. 2010; Bloom, 2011; Wang et al. 2011a,b). Until recently, village clinicians were fully reliant on sales of services and drugs to clients. In recent years, in most villages, village clinicians receive a nominal base salary from the government. China’s government is looking to gradually change this (as evidenced by the new per capita public health payments to village clinicians described earlier), but for now it is safe to say that village clinicians are almost fully private practitioners. They are best thought of as self-employed clinicians who live mostly off of their earnings in the village. Village clinicians’ heavy reliance on drug sales and user fees for income paired with a price schedule distorted away from basic care and prevention have led to escalating patient costs and inappropriate treatment (Eggleston et al. 2008; Yip and Hsiao 2008; Yip et al. 2010). Studies have shown, for example, that antibiotics are widely overprescribed (Reynolds and McKee 2009; Currie et al. 2011).

Are China’s rural clinicians prepared for the unique and daunting epidemiological landscape the country faces? Will the patients that utilize the services of these providers end up with improved health outcomes? Although some have voiced concerns that the answer to these questions is no, others argue that the quality of rural clinicians is sufficiently high as not to be of first order importance, and that reforms should first focus on other aspects of the health care system.

Materials and methods

Survey sample and facility surveys

Our study was conducted in six counties located in southern Shaanxi province. Shaanxi province is a western inland province with a high number of poor, rural counties. The per capita rural income and average life expectancy in Shaanxi are among the lowest in all of China, both well below the national average (CNBS 2011). From each of our six sample counties, we randomly selected two townships (the middle level of administration between the county and the village) and three villages from each township. Within each village a census was taken of all clinics designated for reimbursement under the New Cooperative Medical Scheme, the major public insurance programme in rural areas. From this list, one clinic per village was selected at random. [Most villages (67%) had only one eligible village clinic, in which case that clinic was automatically selected for participation in the study. In the remaining villages with two or more eligible clinics (only 8% of villages had more than two), we randomly selected one clinic for the study.] In all, the study sample consisted of 36 village clinics and 12 THCs and was representative of a catchment area serving approximately 2.35 million individuals.

Following sample selection, an initial facility survey was conducted in November 2012. The facility survey included questions on clinic infrastructure, patient load and disease frequency. A separate module collected information on clinicians employed at each clinic, including demographic characteristics, educational attainment, medical qualifications and training, and experience.

This study received ethical approval from the Stanford University Institutional Review Board (IRB) on 19 November 2012 (Protocol ID 25904). The IRB protocol allowed us to record SP interactions using a concealed device (discussed below), provided that we contacted providers in the sample at some point prior to SP visits and obtained their verbal consent to participate in the study. To obtain consent, clinicians were called approximately 4 months prior to the actual SP visits and asked if they agreed to be recorded by an individual posing as a patient. All of the clinicians asked gave their consent.

Standardized patients

The quality of care provided by rural clinicians was assessed based on interactions between rural clinicians and incognito SPs. SPs were individuals recruited from the local community and trained to present consistent disease cases. Using these interactions, we assessed clinicians on process quality, correctness of diagnoses and appropriateness of treatment.

The SP approach (also commonly referred to as ‘simulated’ patients) is considered the gold standard for assessing clinical practice (Glassman et al. 2000; Peabody et al. 2000; Rethans et al. 2007; Mohanan et al. 2011; Das et al. 2012). This approach has distinct advantages over other methods commonly used to measure provider quality (Woodward et al. 1985; Peabody et al. 2000; Das et al. 2012). First, because clinicians are unaware that they are being evaluated, assessments using SPs are not biased by changes in clinician behaviour due to observation alone (or ‘Hawthorne Effects’) (Woodward et al. 1985; Glassman et al. 2000; Peabody et al. 2000; Leonard and Masatu 2010). Second, this method measures ‘actual’ clinical practice as opposed to clinician knowledge (as is measured using clinical vignettes). Third, the SP approach (particularly when interactions are recorded) is less subject to recall bias than are assessments based on patient exit interviews, for example. Finally, because patients and cases are common and standardized across providers, the SP approach allows for comparisons across providers of different types and in different locations. Although SPs have long been used in western medical schools and in evaluating providers in developed countries, we are aware of only one other study that has used the SP approach to measure the quality of primary care providers in a developing country (India—Das et al. 2012).

Recruitment and training of SPs

To ensure that SPs were similar to patients typically seen by clinicians (in language, mannerisms, dress, etc.), SPs were recruited from the sample counties (though not from the specific villages included in the sample). A total of four SPs, two male and two female, were selected to participate in the study based on initial interviews and demographic match to the disease cases used in the study (discussed below).

Selected SPs were trained in a classroom setting for 4 days (32 h) by a team consisting of researchers and consulting medical professionals. Classroom training focused on preparing SPs to represent their assigned disease cases to providers in a consistent and unsuspicious manner. Medical professionals discussed the symptoms of diseases to be portrayed and the typical behaviour and presentation of real patients afflicted with the disease. Following classroom instruction, the SPs went through extensive field rehearsals in rural areas with clinicians who volunteered to assist with the study.

Cases presented by SPs

Each SP was trained to present a case of either dysentery or unstable angina. (For the dysentery case, female SPs presented the case of a child who was not present.) These diseases were selected to match those diseases used in the Das et al. (2012) study of rural India. Using the same diseases in each study allows for comparisons between rural China and rural India. These diseases are appropriate for the SP methodology as (1) there are no obvious physiological symptoms and (2) there is low risk that SPs would be exposed to invasive procedures or tests.

These diseases are also highly relevant to the current and future disease burden in rural China. Although deaths from dysentery have declined dramatically in recent years, it remains prevalent in many rural areas (Wang et al. 2006). In 2006, dysentery was the third highest notified communicable disease after tuberculosis and hepatitis B (Wang et al. 2008). The burden of heart disease is on the rise and is now a leading cause of death in rural China (He et al. 2005). The ability of rural clinicians to correctly diagnose and treat cases of dysentery and angina, therefore, provides an indication of how well rural providers are prepared to address both China’s ‘future’ prevalence of non-communicable disease and the infectious diseases of China’s ‘past’.

The SPs were trained on and followed scripts for each disease. Each script included disease symptoms and history as well as a detailed background story for each case. These scripts were based on those used in a previous study in India and were adapted for use in China by the research team, which included consulting doctors from local hospitals (Das et al. 2012). In designing case scripts, the intention was to make diagnosis relatively easy but to not lead clinicians or reveal information that could be used in assessment before being prompted by the clinician himself.

SP visits

SPs visited sample providers in late January 2013. Two SPs independently visited each provider in the sample (one presenting each disease case). One of the two SPs trained in each case was randomly assigned to each provider to reduce the potential for any remaining differences in individual SP presentations to bias comparisons across providers. Upon entering each clinic, the SPs were seen by whoever would have seen them had they been a regular patient (they made no attempt to be seen by specific clinicians). As, according to our data, only one clinician was employed in the large majority of clinics (78%), this typically was not an issue.

We used three methods to collect data from SP–clinician interactions. First, SPs wore a concealed recording device. This allowed us to accurately score interactions without relying on the SP’s ability to recall details. Second, SPs were administered a case-specific ‘debriefing survey’ upon exiting clinics. This survey covered the interaction with the clinician as well as the SP’s own impressions of the providers and any additional observations made by the SP that they thought relevant but not captured on the audio recording. Finally, to collect information on drugs dispensed and fees charged, SPs were directed to purchase any and all medications prescribed (which are sold by providers in China) and to pay any additional fees charged by the provider.

Quality measures

We evaluated clinicians on three dimensions of quality observable from their interaction with the SPs: process quality, the accuracy of diagnoses and the appropriateness of treatment. We assessed process quality by grading recorded interactions against ‘clinical checklists’ of recommended questions to be asked of the patient and any physical exams to be performed (full checklists for each case are given in the Supplementary Appendix). These clinical checklists were identical to those used in the Das et al. (2012) study of quality of care in rural India. In adapting them to the Chinese setting, we checked the checklists against the Chinese national treatment guidelines; finding no discrepancies, and finding the checklists to be generally applicable to the rural Chinese clinical setting, we left the checklists unchanged. The clinical checklist for the unstable angina case included 22 ‘question’ items and 5 ‘exam’ items. The dysentery list consisted of 19 ‘question’ items. There were no ‘exam’ items on the dysentery list as the child was not present. A subset of items on each checklist are considered ‘essential’. These are items that were determined to be necessary for any prudent clinician to be able to make a basic diagnosis and protect a patient from harm. Six items on the angina checklist were deemed essential; four items on the dysentery checklist were deemed essential.

Diagnoses and treatments were assessed based on predetermined standards of correctness (full standards given in the Supplementary Appendix). Diagnoses were classified as ‘correct’, ‘partially correct’ or ‘incorrect’. To ensure that diagnoses were given for each interaction, SPs were instructed to ask clinicians directly at the conclusion of the visit if a diagnosis had not already been volunteered. Treatments were similarly judged against a predetermined definition of appropriate treatments. Treatments were deemed ‘correct’ if clinicians dispensed any one of the ‘correct’ medications. For the unstable angina case, treatments were also considered correct if the clinician referred the SP to an upper level provider.

Results

Village provider characteristics

As part of the facility survey we asked village clinicians how frequently they saw cases of diarrhoea and heart problems (Table 1). On average, they reported seeing 1.17 cases of diarrhoea and 0.83 cases of heart problems in the previous 2 weeks. We also asked village clinicians whether they felt their clinic had the necessary equipment to treat ‘moderate’ and ‘serious’ cases of diarrhoea and heart problems. For diarrhoea, most providers (88.9%) said that they had the necessary equipment to treat moderate cases and 5.6% said that they could treat serious cases. Another 5.6% said that they did not have the necessary equipment to treat the disease. For heart problems, slightly under half (47.2%) said that they had the necessary equipment to treat moderate cases. None thought that they had the necessary equipment to treat severe cases. As providers at the lowest tier of China’s health system, if village clinicians encounter diseases that they cannot treat, they are supposed to refer patients to the THCs (the next tier).

Table 1

Clinic statistics on diseases used for SP cases

	Diarrhoeaa	Heart problems
	Mean (SD)b or %	Mean (SD)b or %
Number of patients seen in past 2 weeks with diarrhoea and heart problems	1.17 (1.84)	0.83 (1.4)
Clinic has the necessary equipment to treat
Serious cases	5.6%	0%
Moderate cases	88.9%	47.2%
Does not have the necessary equipment	5.6%	52.8%

Source: Survey of providers conducted approximately 6 months before SP visit.

^aProviders were asked about frequency of ‘diarrhoea’ and ‘heart problems’, not dysentery and angina specifically.

^bSD: Standard deviation.

Village clinicians had low levels of general education and formal medical qualifications (Table 2). Only 20% completed academic high school or vocational college, and none completed academic college. The highest degree completed by the majority of clinicians (60.5%) was vocational high school. The majority (84.3%) of village clinicians held only the most basic certificate required to practice medicine in rural areas: a ‘Rural Physician’ certificate. The remaining 15.7% had higher certifications: either an ‘Assistant Practising Physician’ certificate or a ‘Practising Physician’ certificate. This pattern of education and medical qualifications is roughly similar to that found in larger surveys of village clinicians in China’s western regions (Li et al. 2013).

Table 2

Medical provider characteristics

	Village clinics (n = 36)	Township health centresa (n = 12)
	Mean (SD) or %	Mean (SD) or %
Clinician characteristics
N	45	24
Age (years)	45.85 (10.87)	43.78 (8.565)
Male	73.3%	83.3%
Experience (years)	23.33 (9.523)	20.89 (8.575)
Education
Middle school	19.5%	0%
Vocational high school	60.5%	41.7%
Vocational college	10%	40.3%
General high school	10%	4.2%
College	0%	13.9%
Qualification
Practising Physician	10%	66.7%
Assistant Practising Physician	5.7%	33.3%
Rural Physician	84.3%	0%
Hours worked per day	9.943 (4.311)	—
Base salary (yuan)	351.2 (377.7)	—
Total income (yuan)	1355.2 (1400.7)	—
Facility characteristics
N	36	12
Patients per week	43.69 (31.69)	152.6 (141.5)
Value of medical instruments (10 000 yuan)	1.097 (1.265)	34.29 (31.59)
Stethoscope	100%	—
Sphygmomanometer	100%	—
Sterilization pot	97.2%	—
Thermometer	100%	—
Ultraviolet (UV) sterilizer	100%	—
Debridement kit	97.2%	—
Tongue depressor	88.9%	—
Suction device	16.7%	—
Oxygen bag	52.8%	—

Source: Survey of providers conducted approximately 6 months before SP visit.

^aDetailed questions on medical training and equipment were not asked of township providers.

Also of note is that only a fraction of the village clinicians’ total income is derived from his/her base salary. On average, only 351 yuan ($55) per month of a total income of 1355 yuan ($215) was not from the sales of drugs or services rendered. Only 59% of village clinicians had any base salary at all; among these, the average base salary was 603 yuan ($96) out of a total income of 1601 yuan ($254). As noted earlier, this is the prevailing pattern of compensation among medical providers in China’s villages and is the source of incentives to over-prescribe drugs and not refer patients to other tiers of the health system (Eggleston et al. 2008; Yip et al. 2010).

Our survey data show that village clinics have the medical equipment needed for basic examinations. All of the clinics were stocked with at least one stethoscope, sphygmomanometer (blood pressure meter) and thermometer—three pieces of equipment that are needed for the recommended exams on the clinical checklist we used to score clinicians on process quality for angina. Due to low ownership rates of electrocardiogram (EKG) recording devices, the EKG item on the checklist was scored based on whether clinicians referred patients to another provider for the test.

SP interactions and provider quality

Given our sample of 36 village clinics, there were a total of 72 potential interactions with village clinicians across the two diseases. At the time of the SP visits, village clinicians were absent in five instances for the dysentery case and six instances for the angina case. The rate of absenteeism was thus around 15%—lower than that found in many other developing countries (Chaudhury et al. 2006). It is also important to note that village clinicians in China often make house calls and perform public health duties outside of the clinic, so absence at the time of the unannounced visit is not necessarily a sign of shirking.

The detection rate of the SPs was assessed in the post-interaction debriefing survey and found to be extremely low. In only two instances (two interactions for the angina case), was there any suspicion that the clinician might have detected the SP. We excluded these two instances from the analysis. To further test whether suspicion of the SPs may have influenced clinician performance, we examined the correlation between the probability that SPs were ‘correctly’ treated and (1) time spent on diagnosis and (2) the percentage of checklist items completed. We find both to be positive. Assuming that clinicians would be more likely to treat ‘incorrectly’ if they believed that the patient was faking their condition, this is the opposite of what one would expect if obtaining more information from an SP raised suspicion that the SPs were genuine patients.

After excluding the two cases where SPs were plausibly detected plus the 11 cases in which the village doctor was absent, the final sample contained 59 total interactions with village clinicians across the two diseases.

Process quality

Unannounced SP visits revealed the overall quality of care provided in village clinics to be poor. Although the average length of each SP interaction with providers was 7.2 min, the majority of this time (3.6 min) was spent filling prescriptions (Table 3). On average, clinicians only spent 1.6 min consulting with patients. During these consultations, village clinicians asked 18% of the recommended questions on average. For the unstable angina case, only 15% of the recommended exams were performed. Of checklist items deemed essential to make a proper diagnosis and protect patients from harm, clinicians addressed only 36% on average. No village clinicians addressed all essential items.

Table 3

Standardized patient measures of provider quality^a

	Village clinics			Township health centres
	All (n = 59)	Angina (n = 28)	Dysentery (n = 31)	All (n = 23)	Angina (n = 12)	Dysentery (n = 11)
	Mean (SE)e	Mean (SE)e	Mean (SE)e	Mean (SE)e	Mean (SE)e	Mean (SE)e
Time
Total time (min)	7.24	8.57	6.05	10.52	15.08	5.54
	(1.05)	(1.83)	(1.12)	(2.29)	(3.77)	(1.55)
Wait time (min)	1.72	2.57	0.96	1.17	1.12	1.23
	(0.67)	(1.34)	(0.36)	(0.44)	(0.40)	(0.83)
Diagnosis time (min)	1.61	2.18	1.09	5.06	8.56	1.25
	(0.19)	(0.33)	(0.16)	(1.34)	(2.12)	(0.19)
Prescription time (min), if given	3.63	3.82	3.51	4.10	4.22	3.95
	(0.73)	(0.85)	(1.07)	(0.97)	(0.80)	(2.05)
Process quality
Number of recommended questions asked	3.73	4.11	3.39	4.78	6.00	3.45
	(0.28)	(0.36)	(0.42)	(0.47)	(0.63)	(0.47)
% recommended questions asked	18.23	18.67	17.83	22.92	27.27	18.18
	(1.38)	(1.66)	(2.19)	(2.10)	(2.85)	(2.50)
Number of recommended exams performed	0.89	0.89	—	1.67	1.67	—
	(0.22)	(0.22)	—	(0.26)	(0.26)	—
% recommended exams performedb	14.88	14.88	—	27.78	27.78	—
	(3.67)	(3.67)	—	(4.27)	(4.27)	—
% recommended questions and exams	17.84	17.86	17.83	22.98	27.38	18.18
	(1.41)	(1.77)	(2.19)	(2.05)	(2.69)	(2.50)
% essential questions and exams	36.44	32.14	40.32	43.48	37.50	50.00
	(2.40)	(2.96)	(3.61)	(3.18)	(3.63)	(4.77)
Diagnosesc
Diagnosis correct (0/1)	0.26	0.25	0.27	0.52	0.58	0.45
	(0.06)	(0.08)	(0.08)	(0.11)	(0.15)	(0.16)
Diagnosis partially correct (0/1)	0.33	0.32	0.33	0.13	0.25	0.00
	(0.06)	(0.09)	(0.09)	(0.07)	(0.13)	(0.00)
Treatment
Treatment correct or partially correctc (0/1)	0.53	0.61	0.45	0.52	0.67	0.36
	(0.07)	(0.09)	(0.09)	(0.11)	(0.14)	(0.15)
Medications dispensed (0/1)	0.75	0.61	0.87	0.48	0.50	0.45
	(0.06)	(0.09)	(0.06)	(0.11)	(0.15)	(0.16)
Number of medications dispensed, if any	2.07	2.35	1.89	2.55	2.17	3.00
	(0.14)	(0.24)	(0.16)	(0.37)	(0.60)	(0.32)
Medication correct, if dispensed (0/1)	0.36	0.12	0.52	0.45	0.17	0.80
	(0.07)	(0.08)	(0.10)	(0.16)	(0.17)	(0.20)
Referral to other providerd (0/1)	0.37	0.61	0.16	0.48	0.67	0.27
	(0.06)	(0.09)	(0.07)	(0.11)	(0.14)	(0.14)
Referred to county provider, if referral (0/1)	0.41	0.47	0.20	0.91	0.88	1.00
	(0.11)	(0.12)	(0.20)	(0.09)	(0.13)	(0.00)
Referred to township provider, if referral (0/1)	0.27	0.24	0.40	—	—	—
	(0.10)	(0.11)	(0.24)	—	—	—
Suggested follow up visit (0/1)	0.19	0.04	0.32	0.61	0.33	0.91
	(0.05)	(0.04)	(0.09)	(0.10)	(0.14)	(0.09)
Fees
Total fees charged (yuan)	11.57	14.46	8.96	29.56	32.02	25.12
	(1.83)	(3.60)	(1.16)	(7.16)	(10.24)	(9.02)

Source: Authors’ analysis.

^aUnit of observation is the SP–provider interaction. Doctors were absent in five instances for the dysentery case and six instances for the angina case. Two angina interactions with village doctors were excluded due to suspicion that the doctor detected the SP.

^bNo exams recommended for dysentery case as child was not present.

^cSee Supplementary Appendix for classification of correct, partially correct, and wrong diagnoses and treatment.

^dReferrals to places other than the county or township provider were typically to a ‘larger hospital’.

^eSE: Standard error in parentheses.

In looking at the frequency of individual questions asked and exams performed (Figures 1 and 2), the emphasis appears to be on collecting the information needed to provide medicine. In the dysentery case, 97% of clinicians asked the age of the child—the main basis for prescription of medication—whereas the remaining questions were asked around half of the time or less. Only 10% of clinicians asked about the frequency of urination, an indication of dehydration that is supposed to be one of the primary concerns of a provider in cases of dysentery.

Figure 1

Adherence to clinical checklist, dysentery case Source: Authors’ analysis. Notes: Figure shows means and 95% confidence intervals for clinician adherence to dysentery case clinical checklist items. Items with asterisks (*) are considered essential.

Figure 2

Adherence to clinical checklist, unstable angina case Source: Authors’ analysis. Notes: Figure shows means and 95% confidence intervals for clinician adherence to dysentery case clinical checklist items. Items with asterisks (*) are considered essential. EKG refers to either performance of an electrocardiogram or referral for an electrocardiogram.

Clinicians asked unstable angina SPs about pain location in almost all interactions yet asked other checklist items and conducted exams infrequently. Few clinicians (only 14%) asked about pain radiation, a clear symptom of stable or unstable angina. Further, although considered essential, and even though all providers had the necessary equipment, only 11% of clinicians took the blood pressure of the SPs.

Diagnoses

Given the low level of process quality, it is unsurprising that the quality of diagnoses and treatments was also poor. Diagnoses were volunteered in 71% of the angina cases and 32% of the dysentery cases. A fully correct diagnosis was made in only 26% of the interactions (Table 3). Diagnoses were completely incorrect in 41% of interactions. The most common incorrect diagnosis was ‘indigestion’ for the dysentery case and ‘sprain’ for the angina case.

Treatments

Across the two cases, treatments were deemed correct or partially correct in 53% of interactions (Table 3). Note, though, that treatments are judged correct or partially correct if they include any of the correct medications or if clinicians gave a referral in the angina case. This number, therefore, may be a poor indicator of treatment quality.

A better indication of appropriate treatment is perhaps given by the overall appropriateness of the medications dispensed. Village clinicians dispensed medications in 75% of interactions (61% of angina interactions and 87% of dysentery interactions). When medications were prescribed, on average, each village clinician prescribed 2.07 medications. In the cases where drugs were dispensed, consulting physicians determined that they were either unnecessary or harmful 64% of the time (see Supplementary Table 2 for standards for correct and incorrect treatments).

Failure of clinicians to refer patients was not entirely due to misdiagnosis. For the angina case (and heart problems more generally), for example, correct treatment dictates that patients are referred to higher levels of the rural health system. However, of clinicians who correctly or partially correctly diagnosed unstable angina, 25% failed to refer patients.

Patient costs

Total fees charged by village clinicians (including medication and clinic fees) were 11.5 yuan ($1.85) on average. Angina SPs were charged 14.5 yuan ($2.30) on average and dysentery SPs were charged 9 yuan ($1.43). Village providers typically do not provide itemized receipts so obtaining a detailed breakdown of fees charged was not possible. The post-interaction debrief survey of SPs revealed that most of the fees charged were for drug sales.

Correlates of village provider quality and patient costs

The summary of quality discussed earlier hides meaningful differences across individual providers in our sample. For instance, while on average village clinicians asked 18.2% of recommended questions on average, clinicians at the 10th percentile asked 5.3% of recommended questions whereas clinicians at the 90th percentile asked 36.8% of recommended questions. In order to ascertain what might account for these differences in provider quality, in this section we present analyses that consider correlations between provider quality and individual characteristics such as physician qualifications and financial incentives.

Figure 3 shows results from multiple linear regressions for the different dimensions of provider quality and for fees charged during SP visits (full specification details are in the Supplementary Appendix). By far, the factors most strongly correlated with dimensions of quality were clinician education and qualifications. Village clinicians with ‘Practising Physician’ qualifications, for example, addressed 12.3% more recommended checklist items than those with lower levels of qualification. Clinicians with an upper secondary degree were 45 percentage points more likely to give a correct diagnosis and nearly 30 percentage points more likely to offer an appropriate treatment.

Figure 3

Marginal effects on process quality, diagnosis, treatment and patient costs in village clinics Source: Authors’ analysis. Notes: Figure graphs the marginal effects and 95% confidence intervals of village clinician experience (in years), education (whether the clinician has an upper secondary degree or higher), qualification (whether the clinician has a ‘Practising Physician’ certificate), and compensation (whether the clinician has a base salary) on dimensions of quality and fees charged during standardized patient visits. For clinics with more than one practicing clinician, the average of clinician characteristics was used. Marginal effects on each dimension of quality and cost were estimated in separate regressions. Outcome variables for each regression were as follows: percentage of checklist items addressed by clinicians were rescaled to be between 0 and 1 (so 1 is 100%); correct diagnosis and treatment are 1 if clinicians gave a correct diagnosis and 0 otherwise; and patient costs are expressed in 100 yuan. Regressions for the percentage of checklist items and patient costs were estimated using ordinary least squares and correct diagnosis and treatment were estimated using probit regressions. Each regression also controls for clinician age and gender, the total value of the provider’s medical instruments, patient load (measured as the number of patients in the clinic at the time of the SP visit) the SP case (dysentery or unstable angina) and country fixed effects.

Apart from clinician education and qualifications, few other factors were correlated significantly with provider quality. Although these estimates should be interpreted with caution (given that our small sample size may prevent us from detecting these differences), correlations with some other variables are nevertheless instructive. Of particular interest, given current policy discussions, is the relationship between a clinician’s base pay (pay not tied to drug or service sales) and the quality of services provided and costs charged by providers. Specifically, separating clinician pay from drug sales may reduce over-prescription of drugs and inappropriate treatment, but may also weaken clinician incentives. While not statistically significant, clinicians with a base salary (of any amount) charge patients about 5 yuan (55%) less in total fees. Point estimates for having base pay are positive for process quality and the probability of correct treatment, but negative for the probability of a correct diagnosis.

Comparison to THCs

While based on a small sample of interactions (23 between the two diseases), we found the quality of primary care provided in THCs to be better than that provided in village clinics, but still poor generally. Possibly due to higher levels of education and qualifications among clinicians in THCs, process quality and the accuracy of diagnoses were better than that provided by village clinicians (Table 3). This improvement in measured process quality and diagnoses, however, was mainly due to improved scores on the angina case. For this case, THC clinicians asked 8% more of the recommended questions and performed 13% more of the recommended exams. In the dysentery case, township clinicians asked approximately the same percentage of recommended questions as village clinicians. Of those questions asked, however, a higher proportion was considered essential. Still, although they performed better than village clinicians, township clinicians only addressed 38% of essential items for angina and 50% for dysentery.

In addition to better process quality, THC clinicians were also more likely to provide a correct diagnosis. In total, diagnoses were volunteered by the THC clinicians in 83% of the angina cases and 36% of the dysentery cases. These rates are slightly higher than that among village clinicians. Overall, 52% of diagnoses were correct, twice the rate of village clinicians. The largest difference was for the angina case; 58% of township clinicians correctly identified this disease compared with 25% of village clinicians.

Although surprising given better scores on other quality dimensions, the appropriateness of treatments given by THC clinicians was comparable to village clinicians. THC clinicians gave correct treatments for the angina case 6% more often than village clinicians, but 9% less often for the dysentery case. Perhaps most strikingly, THCs were significantly less likely to dispense medicine. Although village clinicians dispensed at least one medication in 75% of interactions, THC clinicians gave drugs to only 48% of the patients. The largest difference for the rate of prescription was in the dysentery case where village clinicians dispensed medicine 42% more often. Moreover, when THC clinicians did prescribe medicine, they were more likely than village clinicians to do so correctly. In the dysentery case, for example, the medicines prescribed by THC clinicians were correct nearly 30% more often. This suggests that THC clinicians were less likely to dispense unnecessary medications.

Using multivariate regression analysis (full specification and results shown in Supplementary Table A6), we find that THC clinicians address around 5% more of the recommended questions and physical exams (P < 0.05). They are also around 22 percentage points more likely to give a correct diagnosis (P < 0.05). Once clinician and medical qualifications are adjusted for, however, there is no significant difference between the process quality provided at THCs and village clinics.

Discussion

Interactions between incognito SPs and village clinicians showed the quality of care provided in China’s rural clinics to be poor. On average, village clinicians completed only 18% of the items on a clinical checklist of recommended questions and physical exams, only 26% of clinician diagnoses were fully correct, and clinicians provided medicine that was unnecessary or harmful in 64% of the interactions.

The factors most strongly correlated with the quality of care were the educational attainment and medical qualification of clinicians. These factors also appear to explain a substantial portion of the difference in quality observed between village clinics and THCs at the next tier in the rural health system. THC clinicians performed significantly better than village clinicians; however, the quality of care in THCs was also found to be low.

The results we found for village clinicians in China are comparable to those found in a recent study that used the same SP methodology to measure the quality of care provided by rural providers in Madhya Pradesh, one of the poorest states in India (Das et al. 2012). In that study, it was found that the average percentage of ‘essential’ clinical checklist items completed by rural providers was 34% (for dysentery and unstable angina cases). In our study—which closely replicated the Indian study, down to the same list of ‘essential’ checklist items used to measure process quality—we found that village clinicians completed 36% of these on average. In terms of diagnoses (for which assessment criteria were comparable), the study in India found that 56% of providers gave correct or partially correct diagnoses, compared with 59% in our sample. It is important to note, however, that the methodology for obtaining a diagnosis from providers varied slightly across the two studies: in the India study, diagnoses were obtained only from providers who volunteered one without prompting from the SP, whereas in the China study SPs were instructed to prompt providers for a diagnosis if none was volunteered. The study in India found a much lower rate of ‘correct’ treatments (22% compared with 53% in our study); however, these figures are also less comparable due to differences in standards used to assess ‘correct’ and ‘incorrect’ treatments.

That the quality of care we find in our sample in China is similar to that found among rural providers in Madhya Pradesh is surprising considering that our study area is much more developed economically. In 2011, per capita gross domestic product (GDP) was $583 in Madhya Pradesh and $3179 in Shaanxi, the province in China where our study was conducted. Moreover, 67% of the clinicians in the Indian study had no medical qualifications at all whereas those in our sample all had at least a ‘Rural Physician’ qualification.

Establishing the exact source of the low quality of health care in rural China is beyond the scope of this study. However, one possible reason might be related to the state of the rural labour market today. Because of the low incomes of villagers in our study areas, the fees that village clinicians can reasonably charge patients are low. Moreover, the government only pays a small salary (and this has only started in recent years). Given these realities, only low-skilled workers will select themselves into the rural health sector. The low human capital of village clinicians (less than half have completed any form of secondary education—Babiarz et al., 2013) supports this notion. Without strong financial incentives, individuals have little reason to be attracted to the field and, once they are in it, may have little reason to work diligently.

Another possible explanation for the surprising combination of low quality of care and high per capita GDP in rural China may stem from the perverse financial incentives facing providers. In rural China, as has already been described, providers depend on large profits from drug sales to supplement their nominal base salary (Eggleston et al. 2008; Yip and Hsiao 2008; Yip et al. 2010). Currie et al. (2011) have shown that physician induced demand is in part responsible for the high rates of unnecessary antibiotic prescriptions in China. A more efficient system would regulate profits from drug sales so as to discourage over-prescription of unnecessary medications; indeed, recent policy pronouncements indicate that this is the direction in which China seems to be moving (Chen 2009; Yip et al. 2012).

As with other studies using SPs, our study faces a number of limitations. First, the cases that SPs can present to physicians are necessarily limited to diseases with no obvious physiological symptoms and for which there is no risk of invasive procedures to the SP. Thus, quality measured based on these diseases may not be representative of the broader spectrum of diseases treated by clinicians.

Second, the criteria on which the quality of care is judged in SP studies are based on a western model of patient care. This may be of particular concern given that around 15% of clinicians in western China specialize in traditional Chinese medicine (7.7% nationally) (Li et al. 2013). Note, however, that—regardless of specialty—national guidelines for diagnosing and treating are consistent across providers and adhere to international standards of western medical treatment (Fang et al. 1998; Chinese Society of Cardiovascular Diseases of Chinese Medical Association 2007).

Third, our study focuses on actual clinical practice, not on clinicians’ knowledge of best practices. There is therefore no way to determine whether the poor quality of care observed in our sample clinics is due to lack of clinician knowledge, or to a gap between clinician knowledge and practice, or to a combination of the two. Other researchers (Das and Hammer, 2007; Das et al. 2008) have found that the ‘know-do gap’ is a serious contributor to the low quality of health care in developing settings, but further research is required to determine whether this is the case in rural China specifically.

An additional limitation of our study is size and representativeness of our sample. Our sample was large enough to give informative estimates of the quality of care provided among village clinicians and to detect differences between providers; however, a larger sample would make these estimates more precise. Future studies should also draw their samples in such a way as to represent clinicians over a wider geographical area.

Conclusion

Our results raise doubts that village clinicians are currently prepared to shoulder the increased role intended by reforms. Although our results do not speak to the ability of village clinicians to effectively deliver more general public health services (vaccinations, health education, etc.), they do suggest that village clinicians may not be able to serve as effective primary first points of contact for ill patients. If village clinicians are unable to effectively diagnose and treat common diseases, reforms that aim to reduce patient loads at upper tiers by encouraging or mandating patients to first visit village clinicians are premature. This is particularly true given the emergence of non-communicable diseases as prominent public health problems in rural China. That these diseases have become prevalent at a time when the burden of many infectious diseases remains significant means that rural clinicians need to be prepared to deal with a wide array of health problems.

What is clear from this study is that rural clinicians are not yet prepared to act as the front line fighters in China’s primary care system. If recent reforms that expand access to rural health care are to lead to actual gains in population health, more attention should be paid to improving the quality of care delivered by the rural health system. Poor quality care may dampen the effect of expanded access on health outcomes not only due to worse outcomes for those patients that seek care but also because it may dissuade individuals from utilizing the health system in the first place. Recent government initiatives that focus on delivery are a step in the right direction, but more needs to be known about current quality of care delivered and policy approaches to improve that care. Little is known, for example, about to what degree lack of clinician training or misaligned incentives affect the care that patients receive. China has recently made enormous strides in providing access to health care for rural residents; however, efforts focused on improving the quality of care delivered by grassroots providers will be essential for reforms to lead to actual gains in health. Further research is needed in order to better measure the scope of this problem, and to identify specific policy interventions that can improve quality of care in these areas.

Supplementary Data

Supplementary data are available at HEAPOL online.

Acknowledgements

The authors are grateful to Grant Miller and Manoj Mohanan for their invaluable advice in the early planning stages of this study. They would also like to offer a special thank you to John Kennedy, who kindly let us borrow his sampling frame for this survey, and without whom this study would have been impossible. Lastly, Sean Sylvia would like to thank the Stanford Center for International Development (SCID) for their support.

Conflict of interest statement: None declared.