A comparable yardstick: adjusting for education bias in South African health system responsiveness ratings

Abstract

Health-system responsiveness (HSR) measures the experience of health-system users in terms of the non-clinical aspects of the health system. This has been operationalized as a measurable construct in multiple surveys and studies. According to the literature, reporting behaviour may vary systematically across socio-demographic characteristics. In this study we explore the association between education levels and reporting behaviour in terms of HSR in South Africa using data from the World Health Organization Study on Global Ageing and Adult Health for South Africa (WHO SAGE) conducted in 2007 and 2008. We consider the reporting behaviour of 1499 adults aged 50 and older in terms of the reported HSR for their most recent outpatient provider visit during the preceding 12 months. More specifically, we explore whether there are systematic biases in reporting behaviour by education levels and other socio-economic covariates through the use of data from anchoring vignettes. These are questions depicting hypothetical HSR scenarios which provide a fixed benchmark for comparing individuals’ own HSR ratings and identifying potential reporting biases. Using a hierarchical-ordered probit model in regression analysis, we found large differences in HSR ratings between the lowest and highest education groups after adjusting for reporting bias using the anchoring vignettes. This finding holds across all seven HSR domains captured in the WHO SAGE dataset. In the most extreme case, individuals with no education are likely to underreport poor HSR by between 2.6 and 9.4% percentage points compared with individuals with secondary schooling or higher. Policy-makers need to take cognizance of potential reporting biases in HSR ratings and make the necessary adjustments to obtain data that are as true and accurate as possible. The need for this is especially acute in a country such as South Africa with large socio-economic inequalities and disparities in access to healthcare.

: Health system responsiveness, vignettes, reporting behaviour, reporting heterogeneity, socio-economic determinants, South Africa

Key Messages

• Using data from the WHO’s SAGE dataset, we demonstrate the presence of reporting bias by education level in South Africa. Individuals in the lowest education categories report more negatively on health system responsiveness (HSR) than individuals in the highest education categories on responsiveness. After adjusting for possible reporting biases, individuals at the lowest education levels report even more negatively on HSR.

• The need for policy-makers to be aware of and adjust for such reporting biases is especially acute in the context of developing countries which may be characterized by extreme socio-economic and healthcare access inequalities. Failure to do so may further entrench such inequities.

Introduction

Client satisfaction, or the inter-personal experiences between clients and healthcare providers, has long been viewed as an important factor in understanding the quality of healthcare (Donabedian, 2005). However, the World Health Organization (WHO) has only from 2000 explicitly recognized the importance of the interaction between clients and the health system and termed this ‘responsiveness’ (De Silva and Valentine 2000; Houweling et al. 2001; Valentine et al. 2003). Health-system responsiveness (HSR) differs from client satisfaction in that it is mostly focused on measuring the non-clinical aspects of the health system (De Silva and Valentine 2000). The concept of HSR has typically been operationalized as a measurable construct consisting of eight domains: clarity of information (communication); dignity; involvement in decision-making (autonomy); confidentiality; prompt attention; quality of basic amenities; access to social support networks and choice of care provider (Gostin et al. 2003). Questions on individuals’ experiences in these responsiveness areas are generally asked relative to a five-point ordinal scale, with (1) indicating a ‘very good’ experience and (5) a ‘very bad’ experience.

HSR has been identified as one of three goals in achieving or striving towards health-system performance (Murray and Frenk 2000). HSR is also important for health system users’ satisfaction with the system (Gostin et al. 2003). This, in turn, may influence health-seeking behaviour and adherence to medical treatment (Dang et al. 2013). It is therefore important for policy-makers to obtain as true and accurate HSR ratings as possible allowing for HSR to be tracked and improved over time.

The issue of variation in reporting behaviour in HSR rating, i.e. of survey respondents using differential scales against which they benchmark their experienced HSR (Murray et al. 2001), has been identified across multiple responsiveness studies (e.g. Rice et al. 2012; Sirven et al. 2012; Malhotra and Do 2013; Valentine et al. 2015). Reporting behaviour in HSR may vary across countries (Rice et al. 2012; Sirven et al. 2012; Valentine et al. 2015); across different domains (Sirven et al. 2012); and, across individual-level characteristics of survey respondents (Malhotra and Do 2013).

One individual-level characteristic which may influence responsiveness experiences and, as a result, reporting behaviour, is level of education. Education is likely to matter for HSR ratings in that individuals with lower levels of education may be less aware of their rights and entitlements from the health system (Delgado Gallego and Vázquez-Navarrete 2013). This, in turn, may be associated with overly positive HSR ratings.

South Africa is at a unique moment in the development of its health system where a National Health Insurance (NHI) Scheme has been proposed for which draft legislation is currently being developed (National Department of Health, Republic of South Africa 2015). The creation of a more responsive health system is part of the reasoning for transitioning to universal health coverage (UHC) through NHI in South Africa (National Department of Health, Republic of South Africa 2015). A National Office of Health Standards Compliance has therefore been established to monitor the performance of healthcare facilities (Republic of South Africa 2013). Ensuring a responsive health system is integral to both NHI and the improvement of healthcare performance in South Africa.

A previous study which examined variation in the HSR experiences of South African individuals by socio-economic and demographic characteristics did not find education to play a large role in HSR reporting (Peltzer and Phaswana-Mafuya 2012). This is unexpected given the country’s highly polarized and unequal health system (McIntyre et al. 2007). Within this system, access to private care is mostly determined by health-insurance coverage obtained via employers in the labour market (Ramjee et al. 2014). Employed South Africans have higher education levels than those not employed or not economically active (Statistics South Africa 2016). Individuals with lower education levels may therefore have different health-system experiences, even though the ratings of these experiences may appear the same as those of individuals with higher education levels.

Variation in reporting behaviour in HSR ratings across individual-level characteristics can be identified by using anchoring vignettes that mimic HSR questions. An anchoring vignette is a data-collection tool which describes the experience of a fixed hypothetical person (King et al. 2004). Data from the anchoring vignette responses allow researchers to control and adjust for differences in reporting behaviour between sub-groups and across persons as they provide an objective benchmark for comparing respondent experiences (King et al. 2004). This approach has been more frequently applied to self-reported health measures (Bago d’Uva et al., 2008a,b; Peracchi and Rossetti 2008; Grol-Prokopczyk 2011; Dowd and Todd 2011; Guindon and Boyle 2012; Zhang et al. 2015), but has also been used in self-reported HSR (Rice et al. 2008, 2009, 2010; Sirven et al. 2012; Malhotra and Do 2013).

In this study, we tested and report on the level of reporting heterogeneity present by education level amongst users of South African outpatient providers who evaluated their experiences with HSR. This was done through the adjustment of responsiveness reporting data from the WHO Study on Global Ageing and Adult Health for South Africa (WHO SAGE) (WHO 2008; Kowal et al. 2012) with data from anchoring vignettes collected in the same survey. The establishment of the presence of variation in reporting behaviour is of particular importance given high levels of overall socio-economic inequality in South Africa (Leibbrandt et al. 2011) but, also more specifically, high levels of education inequality (Van der Berg 2007).

Methods

Data

The data used in this analysis are from the WHO SAGE, collected for South Africa during 2007 and 2008. The study included individuals aged 50 and older, was nationally representative, and consists of about 3800 observations (Phaswana-Mafuya et al. 2012). The particular focus of the analysis is on individuals who reported accessing outpatient services, defined as healthcare visits or consultations that do not entail an overnight stay. More specifically, respondents were asked to rate the HSR of their most recent outpatient-provider visit during the preceding 12 months. The type of healthcare-provider visits analysed included visits to a medical doctor, nurse/midwife, dentist, physiotherapist, traditional healer and pharmacist. Whereas the majority of outpatient visits in the South African private sector are likely to be with medical doctors and the other mentioned practitioners (excluding nurses), most outpatient consultations in the public sector are likely to be at primary healthcare facilities mainly staffed by nurses.

The total sample of respondents who visited a healthcare facility in the preceding year consisted of 1846 individuals (49% of the sample), of which 14% (n = 264) were excluded because they did not self-report their race. This omission is not statistically different across groups, other than for the highest education category who were 4 percentage points (p.p.) more likely than the other education groups to not report their race group. There is no statistically significant relationship between not reporting race and responsiveness ratings or vignettes. A further 83 observations were dropped due to statistically random missing variables across other socio-demographic variables and vignettes. This left analysable data for a sub-sample of 1499 individuals.

Variables

HSR ratings

The WHO SAGE dataset captured patient-evaluated HSR ratings. The WHO structured the health-responsiveness scale (as opposed to the usual patient-satisfaction scales) in such a way that a respondent is able to report on their actual experience of the health system without their expectations of the system affecting their reporting (Murray et al. 2001). This is done by focusing on relatively objective measures of responsiveness, such as waiting time and cleanliness. Despite this, reporting differences and expectations influencing responses may still occur since responsiveness is measured using a Likert-type scale. Any responses to questions answered relative to such a scale are still subjective in nature.

Although HSR was originally conceptualized as consisting of eight domains (Gostin et al. 2003), the WHO SAGE dataset measures only seven of these. All domains consider non-clinical system interaction, including structural and interpersonal interactions (Valentine et al. 2003). The structural interaction domains included in the WHO SAGE HSR questions are whether patients received prompt attention, provider choice and quality of basic amenities. The interpersonal interaction domains include the dignity with which patients were treated, the clarity of information provided, patients’ autonomy in any decision making and the confidentiality with which their case file was treated. Respondents were asked to rate the HSR of their outpatient interaction with regards to these domains on a scale from 1 to 5, where 1 is ‘very good’, 2 is ‘good’, 3 is ‘moderate’, 4 is ‘bad’ and 5 is ‘very bad’.

HSR vignettes

The WHO also asked respondents to rate the experiences of hypothetical persons described in anchoring vignettes. These anchoring vignettes describe various possible interactions with the health system, and serve as a useful data-collection tool as they represent a fixed scenario that can be used to benchmark self-reported ratings. One example of an anchoring vignette relating to the domain ‘prompt attention’ is as follows:

[Stan] broke his leg. It took an hour to be driven to the nearest hospital. He was in pain but had to wait an hour for the surgeon, and was only operated on the next day.

How would you rate the amount of time [Stan] waited before being attended to?

These scenarios were described for each of the seven domains of non-clinical interaction (the full set of vignettes can be accessed at http://www.who.int/healthinfo/sage/cohorts/en/index2.html). Each respondent was asked to evaluate the experience of the hypothetical people in these vignettes on a scale from 1 to 5 (ranging from ‘very good’ to ‘very bad’). Any systematic variation in the way that respondents with different levels of education evaluated the vignettes is an indication that they were using different reporting scales.

Socio-demographic and healthcare utilization variables

Respondents were categorized into one of four education groups, created to match those of an earlier study on patient experiences in South Africa using the WHO SAGE dataset (Peltzer and Phaswana-Mafuya 2012). These four categories are 1 ‘No education’, 2 ‘Less than primary education’ (these individuals did not complete primary schooling), 3 ‘Primary schooling completed’ and 4 ‘Secondary education or more’ (a category encompassing all education higher than primary schooling).

The other socio-demographic covariates considered in the analysis were respondents’ age, wealth, binary gender, self-identified race group (as defined by Statistics South Africa and including four groups: Black African, White, Coloured and Indian/Asian) and urban residence. Respondents’ wealth status was controlled for by using a variable which was a composite measure of a respondent’s assets, dwelling and access to municipal services such as clean water (He et al. 2012). Further factors which may affect respondents’ ratings of HSR include whether they accessed a private or public healthcare facility, whether they self-reported experiencing poor or very poor health and travel time to the health facility.

Statistical analysis

Summary statistics of the socio-demographic variables (aggregated by educational status) and the average HSR ratings for the sample of respondents included in the analysis are provided in Table 1 and Figure 1. The differences in reporting behaviour in the ratings of HSR are calculated by comparing the results from an ordered probit model to a hierarchical-ordered probit (HOPIT) model (King et al. 2004). The ordered probit model determines the correlation of various socio-demographic characteristics (including education levels) to users’ own HSR ratings. This provides estimates of the relationship between education and HSR before taking into account that individuals evaluate their experiences differently.

Table 1.

Summary statistics of socio-demographic status and healthcare utilization factors

	Total sample (n = 1499)		No education (n = 340)		Less than primary (n = 369)		Primary (n = 403)		Secondary or more (n = 389)
	Mean	(SE)	Mean	(SE)	Mean	(SE)	Mean	(SE)	Mean	(SE)
Education
No education	0.20	(0.40)
Less than primary	0.24	(0.43)
Primary	0.27	(0.44)
Secondary or more	0.29	(0.46)
Wealth quintile
Quintile 1	0.18	(0.39)	0.32	(0.47)	0.21	(0.40)	0.22	(0.41)	0.04	(0.20)
Quintile 2	0.17	(0.38)	0.26	(0.44)	0.22	(0.41)	0.14	(0.35)	0.10	(0.31)
Quintile 3	0.20	(0.40)	0.21	(0.41)	0.22	(0.41)	0.21	(0.41)	0.16	(0.36)
Quintile 4	0.22	(0.42)	0.15	(0.36)	0.22	(0.41)	0.25	(0.43)	0.26	(0.44)
Quintile 5	0.22	(0.42)	0.06	(0.24)	0.14	(0.35)	0.18	(0.39)	0.44	(0.50)
Female	0.65	(0.48)	0.65	(0.48)	0.68	(0.47)	0.64	(0.48)	0.62	(0.49)
Age	62.49	(9.01)	64.51	(9.43)	62.11	(9.09)	62.46	(8.75)	61.44	(8.70)
Urban	0.68	(0.47)	0.44	(0.50)	0.61	(0.49)	0.81	(0.40)	0.77	(0.42)
Race
African Black	0.72	(0.45)	0.86	(0.35)	0.83	(0.38)	0.78	(0.41)	0.48	(0.50)
White	0.08	(0.27)	0.00	(0.00)	0.00	(0.06)	0.03	(0.18)	0.24	(0.43)
Coloured	0.15	(0.36)	0.09	(0.29)	0.13	(0.34)	0.15	(0.36)	0.21	(0.41)
Asian/Indian	0.04	(0.21)	0.05	(0.21)	0.04	(0.19)	0.03	(0.17)	0.07	(0.25)
Utilized private healthcare	0.25	(0.43)	0.18	(0.39)	0.22	(0.42)	0.16	(0.36)	0.39	(0.49)
Poor self-reported health	0.22	(0.42)	0.32	(0.47)	0.24	(0.43)	0.23	(0.42)	0.14	(0.34)
Travel time > 1 h	0.07	(0.26)	0.08	(0.27)	0.08	(0.28)	0.03	(0.18)	0.10	(0.30)

Note: Descriptive statistics have been calculated using a population weight; SE, standard error.

Source: Calculations from WHO SAGE (2008).

Figure 1.

Distribution of reported responsiveness ratings by HSR domain. Note: Descriptive statistics have been calculated using a population weight. Source: Calculations from WHO SAGE (2008)

In contrast to the ordered probit model, the HOPIT model is used to estimate the relationship between various socio-demographic characteristics (including education levels) and self-reported HSR while taking into account differences in reporting behaviour (as established by their rating of the vignettes). The model consists of two components, the ‘reporting behaviour equation and the HSR equation’, which are calculated jointly for efficiency (Bago d’Uva et al. 2008b).

The first, ‘reporting behaviour’ component utilizes the vignettes, and calculates how the position of the cut-points in the Likert-type scale vignette variable varies based on individual-level characteristics (such as education, wealth, gender and the other socio-demographic and healthcare utilization factors listed in the previous section). Variations in the rating of the responsiveness experienced by the vignette are deemed to be due to differences in reporting style, and a statistical test is performed to establish whether these variations are statistically significant. In this step, for instance, one would be able to establish whether an individual with no education evaluates the responsiveness of the vignette significantly differently from a person with secondary education or more.

In the second component of the model (the ‘HSR’ component), the direction and size of reporting behaviour variations as established in the ‘reporting behaviour component’ are imposed onto the individual’s evaluation of HSR. This is done by imposing the cut-points from the ‘reporting behaviour component’ onto the self-reports, thereby removing reporting differences and providing more unbiased estimates of the individual’s experience of the health system. The analysis was conducted for all seven HSR domains. Statistical significance was always determined at an alpha $\left(\alpha \right)$of P < 0.05. All statistical analyses were performed using Stata 13 (StataCorp 2013).

One limitation of the WHO SAGE vignettes section is that it contained only one vignette per domain. Since more vignettes are required in order to draw robust conclusions about differing reporting styles, the approach from a similar study in India (Malhotra and Do 2013) is used to deal with this limitation. The authors included the vignettes from all seven health domains into each HOPIT estimation, arguing that one of the assumptions of the HOPIT model is that respondents rate the vignettes for all the domains in the same manner. This approach assumes that all domains represent one theme, namely HSR. Similar tests to those done by the authors from the earlier study (Malhotra and Do 2013) were performed to confirm this latent theme, which established that the domains were all significantly correlated (P < 0.001), and had high internal consistency (Cronbach’s alpha of 0.92).

Results

The final sample analysed consists of 1499 individuals, of which 20% (n = 340) had no education, 24% (n = 367) had less than primary education, 27% (n = 403) had primary education and 29% (n = 389) secondary education or more (Table 1). Respondents in the sample were on average 62 years of age, and a larger portion were female (65%), living in an urban area (68%) and African Black (72%). Twenty five percent (n = 368) of the sample reported that their last healthcare visit was at a private-health facility, and 7% (n = 109) reported travelling more than an hour to reach the facility. When asked to evaluate their own health, 22% (n = 337) of individuals reported having poor or very poor health.

A disaggregation of the sample means by education levels reveal that those with no education were more likely to fall within the lower-wealth quintiles, be African Black, live in a rural area, utilize public-health facilities (as opposed to private) and reported worse health compared with respondents with higher education levels.

Figure 1 summarizes the reported ratings in HSR for each domain. Across all domains, respondents were most likely to rate the level of responsiveness as ‘Good’, and less likely to choose the options ‘Bad’ or ‘Very bad’.

The results from the regression analyses are presented in Table 2, which shows the marginal effect of education status on rating the responsiveness of a domain as ‘moderate’ to ‘very poor’ (or not good) before (Ordered probit) and after (HOPIT) controlling for reporting differences. By comparing the two sets of results, we are able to gauge the magnitude and direction of reporting differences in reported ratings. The highest education category, secondary education or more was selected as the reference category. Therefore, all effects are interpreted as how less-educated respondents rated responsiveness relative to respondents in the highest education category.

Table 2.

The relationship between responsiveness ratings and education status, before and after correcting for differences in reporting behaviour

		Before correction		After correction
		Marginal effect (n = 1499)	(95% C.I.)	Marginal effect (n = 1499)	(95% CI)
Prompt attention
	No education	0.08	(0.04–0.13)*	0.11	(0.05–0.17)*
	Less than primary	0.06	(0.02–0.10)*	0.08	(0.02–0.14)*
	Primary	0.06	(0.02–0.10)*	0.06	(0.00–0.12)*
	Secondary or more	(Reference)		(Reference)
Dignity
	No education	0.06	(0.03–0.09)*	0.13	(0.06-0.19)*
	Less than primary	0.03	(0.00–0.05)*	0.05	(-0.02-0.11)
	Primary	0.02	(0–0.002–0.05)	0.01	(-.05-0.07)
	Secondary or more	(Reference)		(Reference)
Clarity of information
	No education	0.06	(0.04–0.10)*	0.13	(0.06-0.20)*
	Less than primary	0.03	(0.00–0.06)*	0.06	(-0.01-0.12)
	Primary	0.03	(0.00–0.06)*	0.03	(-0.03-0.10)
	Secondary or more	(Reference)		(Reference)
Autonomy
	No education	0.07	(0.03–0.11)*	0.11	(0.04-0.18)*
	Less than primary	0.04	(0.00–0.07)*	0.06	(-0.01-0.13)
	Primary	0.04	(0.00–0.07)*	0.04	(-0.02-0.10)
	Secondary or more	(Reference)		(Reference)
Confidentiality
	No education	0.05	(0.03–0.11)*	0.14	(0.07-0.18)*
	Less than primary	0.03	(0.01–0.07)*	0.07	(0.00-0.13)*
	Primary	0.02	(–0.00–0.07)	0.03	(-0.04-0.10)
	Secondary or more	(Reference)		(Reference)
Choice
	No education	0.07	(0.04–0.10)*	0.14	(0.07-0.21)*
	Less than primary	0.04	(0.01–0.06)*	0.07	(0.01-0.14)*
	Primary	0.03	(–0.00–0.06)	0.03	(-0.04-0.09)
	Secondary or more	(Reference)		(Reference)
Quality of basic amenities
	No education	0.01	(0.00–0.02)*	0.11	(0.04-0.17)*
	Less than primary	0.01	(–0.00–0.01)	0.05	(-0.02-0.11)
	Primary	0.01	(–0.00–0.01)*	0.05	(-0.01-0.11)
	Secondary or more	(Reference)		(Reference)

Note: these are the marginal effects from the ordered probit (before correction) and a HOPIT (after correction), controlling for wealth quintile, female gender, age, urban residency, race, whether they utilized private healthcare, their self-reported health status and travelling time. Source: Calculations from WHO SAGE (2008).

All marginal effects marked * have P < 0.05; CI, confidence interval.

Across all domains, respondents with no education or less than primary education were likely to underestimate poor responsiveness, i.e. provide overly positive ratings, compared with respondents with secondary education or more. For instance, individuals with no education were 6 p.p. more likely to rate HSR for the dignity domain as ‘Moderate’ to ‘Very poor’ compared with individuals with secondary education or more. Once the vignette ratings were applied using the HOPIT estimator, this difference increased to 13 p.p. Reporting tendencies therefore led to an underestimation of poor responsiveness by 7 p.p. for this group. The size of the changes in rating before and after correction are summarized in Figure 2.

Figure 2.

Changes in ‘moderate to very poor’ responsiveness rating after correction (relative to secondary education or more). Source: Calculations from WHO SAGE (2008)

The largest changes are visible when the least educated (no schooling) are compared with the most educated (secondary schooling or more). Correcting for reporting is associated with increases of between 2.6 and 9.4 p.p. in reporting ‘moderate to very poor’ responsiveness across domains by those with no schooling. Reporting differences by those with less than primary school are similar, but smaller in magnitude (between 1.8 and 4.1 p.p. increases). The differences between those with primary schooling and secondary education or more, are less pronounced: for five of the seven domains (prompt attention, clarity of information, autonomy, confidentiality and choice) the shifts are smaller than 1 p.p. When asked to rate the differences in dignity, those with primary schooling are 1.1 p.p. less likely to report ‘moderate or very poor’ responsiveness compared with those with secondary schooling or more. In only one domain, the quality of basic amenities, do we observe a larger shift (4 p.p.) in the likelihood that respondents with primary school education will report worse responsiveness after controlling for possible reporting biases. The statistical significance of these changes is reported in Table 2.

Discussion

Large differences in the responsiveness ratings between the lowest and highest education groups were found after adjusting for reporting bias using the anchoring vignettes. This finding holds across all seven responsiveness domains captured in the WHO SAGE dataset. The adjustment of reported responsiveness ratings with anchoring vignette data was associated with large increases in the likelihood of the lowest education group (relative to the highest group) providing a negative rating of responsiveness. This result is robust to different specifications of education categories and the use of different comparison groups. This implies that health-system users of outpatient services in the lowest education groups in South Africa were overly positive in their initial (self-reported) ratings of responsiveness relative to highest education groups. Comparisons of negative responsiveness ratings or similar measurement-tool ratings across education groups is likely to be biased and under captured if one does not take into account differences in reporting behaviour between these groups.

A study using WHO SAGE data for India also found education to be an important and significant correlate of variation in responsiveness reporting behaviour (Malhotra and Do 2013). Not only were the initial self-reported responsiveness of the lowest education group in this analysis more negative than that of the highest, but this disparity increased after controlling for reporting bias using anchoring vignettes. Reporting heterogeneity by education level has also been found to play a role in self-reported health status (Bago d’Uva et al. 2008b, 2011). In the case of education and self-reported health, this has led to an under-estimation of inequalities in health by education status (Bago d’Uva et al. 2008b, 2011).

The results show the relative differences in HSR between sub-groups. One limitation of this method is the inability to conclude whether this effect is driven by more-educated individuals overstating poor HSR, or by less-educated individuals underestimating poor HSR. Rather, it is possible to conclude that comparison of more to less educated individuals is biased towards the less educated, and is likely to underestimate the gap in system responsiveness received by persons of varying socio-economic status.

Another limitation of the current analysis is that it is limited to the population aged 50 and older.

Some of the education effects we find may be specific to the older cohort of individuals captured in the WHO SAGE sample. It will be necessary to investigate reporting heterogeneity amongst younger individuals once this data becomes available. Furthermore, the wording in WHO SAGE is such that individuals were asked to rate their outpatient experience in the preceding 12 months, which introduces the possibility of recall bias (Malhotra and Do 2013).

Educational attainment (both average years and completion of schooling) has much improved during the last twenty years in South Africa (Statistics South Africa 2015). However, our analysis did not control for quality of education. Recent research (Spaull 2013) has found large inequalities in access to quality education in South Africa. Given the finding that more vulnerable sub-groups tend to underestimate poor responsiveness, one would expect that differentials in quality of education would exacerbate the reporting differences even further.

However, education is not the only factor likely to affect reporting behaviour with regards to HSR in South Africa. Variation in reporting behaviour in responsiveness in South Africa has been shown to be associated with socio-economic status or wealth (Burger et al. 2016). This analysis found that individuals living in urban areas or utilizing private healthcare were excessively negative in their rating of responsiveness relative to their counterparts. This further emphasizes the importance of socio-economic status as a source of variation in ratings behaviour.

This analysis does not reveal the channels through which the relationship between education and reporting behaviour are likely to operate. Although it is possible to anticipate (intuitively) that education exposes individuals to greater knowledge about their rights and entitlements from the health system, it is possible that there are also other channels at work. Further investigation and possibly qualitative research will be required to isolate these.

Finally, it is also important to keep in mind that vignettes are a work in progress and require testing and validation. The extent to which vignettes are able to control for reporting differences depends partly on the extent to which they are able to capture different reporting styles. Their success depends on two assumptions: that the perception of the system responsiveness depicted by a vignette remains invariant across individuals (vignette equivalence) and that individuals apply the same reporting scale with which they evaluate their own health-system experience to the evaluation of the vignette (response consistency) (King et al. 2004). These assumptions are necessary for the first and second component of the HOPIT model respectively (King et al. 2004). Although they have been tested more extensively for use in self-reported health, the testing has been limited for vignettes in HSR (Rice et al. 2011) and, specifically, the WHO SAGE dataset. More work is thus needed to establish the reliability of vignettes in adjusting HSR ratings for subjectivity and reporting bias.

There have been recommendations that reporting on responsiveness should become more transparent with regard to biases in reporting and the need for explicit adjustment of ratings for these biases (Valentine et al. 2015). This is even more important in the context of a developing country with a highly diverse population such as South Africa. Statistics South Africa collects annual data on health system experiences and satisfaction through the General Household Survey (GHS). It would be possible to include anchoring vignette questions in the GHS questionnaire, or at least administer anchoring vignettes to a sub-sample of its respondents. It will, however, be important to ensure that any vignettes included have been adjusted to the South African context to ensure validity.

A secondary implication of our analysis is that HSR ratings should be used with caution where policy-makers are tempted to use them as an inexpensive measure of quality of healthcare, as it typically done through routine health system surveys. Rather, other more objective measures such as those obtained through administrative data or through other methods such as mystery client visits (Burger et al. 2016) should be identified and used. Where no other data are available, policy-makers need to (at a minimum) conduct subgroup analysis of responsiveness rating data. It may be useful to subject the ratings of sub-groups to a test of plausibility: is it likely that low-education users could have experienced better or higher HSR than high-education users?

Policy-makers should be made aware of the pitfalls of using HSR ratings at face value. Failure to do so will mean that the value of this potentially useful tool to improve the performance of the health system will never be fully unlocked.

Acknowledgements

L.R. thanks Health Systems Global for funding provided to present this research at the Fourth Global Symposium in Health Systems Research in Canada, and the Symposium’s audience for valuable feedback. A.S. acknowledges funding from the National Research Foundation (NRF) for her post-doctoral fellowship which enabled the co-authoring of this article. The authors would also like to thank Teresa Bago d’Uva for valuable inputs regarding anchoring vignettes.

Funding

A.S. acknowledges funding from the National Research Foundation (NRF) for her post-doctoral fellowship which enabled the co-authoring of this article. L.R. received funding from the Margaret McNamara Education Grant during her doctoral studies which enabled the co-authoring of this article.

Ethical approval

Access to the dataset was granted by WHO SAGE. SAGE received ethical approval from the World Health Organization’s Ethical Review Committee and national ethical approval from the Human Sciences Research Council Ethics committee in South Africa.

Conflict of interest statement. None declared.