Validity of self-reported utilization of physician services: a population study

Abstract

Background: Health care utilization is of central interest in epidemiology, and most of the studies rely on self-report. The objectives of this study were to assess the validity of self-reported utilization of general practitioner and specialist physician by correlating self-reported utilization with registered services utilization, and to determine the factors related to that validity. Methods: The 1997 Belgian National Health Interview Survey (BNHIS) was linked with registered medical utilization data provided by the Belgian Health Insurance Funds. Valid information on general practitioner and specialist physician utilization during the past 2 months was found for 5869 participants at the BNHIS who were aged ≥25 years. Intra-class correlation coefficients were used to determine the rate of agreement, and multinomial logistic regression to model factors influencing under- and over-reporting. Results: The results demonstrated a substantial agreement between the self-reported and registered general practitioner contacts, and only a minor bias was found towards under-reporting. There was no significant difference between mean self-reported and registered specialist physician utilization, but the agreement was rather moderate. Gender, age, country of birth, self-rated health, number of chronic illnesses, having functional limitations and having mental health problems, were associated with under- and/or over-reporting. Conclusion: Studies that aim to compare the utilization of different socio-demographic groups have to take into account that the reporting errors vary by respondents characteristics.

Introduction

Information on health care utilization is of central interest in many areas of epidemiology and health services research. In most studies, this information is obtained by asking the respondent about her/his utilization using a standardized questionnaire. Questionnaires are often the only option to obtain this information in a general population, and they also permit simultaneous collection of several types of health-related information of the same person. Nevertheless, questions may arise about the validity of self-reported health care utilization. If the reporting errors vary by respondent’s characteristics, such as age, gender, socio-economic status or health status, systematic biases could be introduced into the studies that compare the utilization of groups that differ on these characteristics.¹ Knowledge of the degree and direction of reporting errors and the correlates of such error would therefore be useful when self-reported utilization data are used.²

Yet, only a few studies have focused on the accuracy of self-reported utilization and on the factors that may influence it.^3,4 In their systematic review on the relationship between respondents’ demographics (age, education, ethnicity, gender, health and socio-economic status) and the accuracy of self-report, Bhandari and Wagner³ concluded that the results were mixed. When significant associations between demographic factors and self-report inaccuracy were reported, the association was most consistent for age, with older people under-reporting.³ Other studies emphasize the importance of the patient’s actual utilization rate as one of the strongest predictors of reporting error, as people tend to forget some visits when the number of visits increase.^1–3,5–8 As the rate of utilization is also correlated with many other characteristics of the respondent, such as age, research about the influence of those characteristics on the accuracy of self-report should include the actual utilization of the person into the analysis because it has the potential to act as an important confounder.

Yet, most previous research has focused on specific populations such as elderly,^6,7,9 AIDS patients¹⁰ or individuals with alcohol use disorders,¹¹ or on the use of specific types of health care such as mental health services.^12,13 Other studies were restricted to the population of a city¹⁴ or a specific health care centre in an urban region.⁸ Furthermore, many of these studies had sample sizes that were too small to explore the impact of factors in-depth that might possibly influence the reporting errors.

A study by Yu et al.⁴ is the first, and to our knowledge the only study, that used a national representative sample to examine the agreement between self-reported and health insurance claims data on the utilization of the whole spectrum of health care services in Taiwan, and to determine factors that have an influence on the agreement. They concluded that agreement between self-reported utilization and insurance claims was fairly good in a general population.⁴ However, a limitation of their study is that they do not make a distinction between under-reporters and over-reporters,³ although under- and over-reporting seemed to be influenced by different factors.⁸ Because this study was not repeated in other contexts, no information is available on whether their results also hold in other populations and cultures.

Therefore, the purpose of this study was to assess the validity of self-reported utilization of general practitioner and specialist physician, by correlating the self-reported utilization with registered services utilization, and to determine factors related to the validity.

Methods

Subjects and data collection

For this study, a database constructed in 2003 was used that linked the 1997 Belgian National Health Interview Survey (BNHIS) with registered medical utilization data provided by the Belgian Health Insurance Funds.

The BNHIS is a cross-sectional survey based on a representative sample of the non-institutionalized population residing in Belgium. Sampling was based on a combination of stratification, multistage sampling and clustering. Stratification was done at the regional level and at the level of the provinces, clustering within the municipalities and within the households. The total sample size was 10 221 individuals. Information on the health status, lifestyle and background characteristics was collected through a face-to-face interview carried out at home and through the completion of a written questionnaire. Information at the level of the household was obtained by means of a household questionnaire comprising mainly validated instruments, often based on WHO recommendations.¹⁵ The method of the BNHIS has been published previously.^15–18

In the Belgian fee-for-services system, patients pay the fee in advance for their medical care and get refunded afterwards by the National Health Insurance Funds. In this context, the Health Insurance Funds register the data on all medical utilization that is subjected to a refund. The registration also contains information about the date of the contact.

To protect the privacy of the participants, the linking of the two databases was carried out by the Crossroads Bank for Social Security, a governmental organization in charge of promoting information security and privacy protection, and delivering integrated statistical information to the politicians and researchers to support the social policy. The respondent’s identification number in the State Register was used to link both the databases. For some respondents, no identification number was found which made the linking of their data impossible. Therefore, the linked database contained information about 9184 individuals. Additionally, to protect the privacy of the respondents, not all information from the BNHIS was available in the linked database, especially information that could identify the individual, such as their date of birth.

A group of 1094 individuals, mainly self-employed respondents or their relatives, were excluded because the National Health Insurance Funds do not have valid data on their health care utilization, as they were not covered by the National Health Insurance Funds for outpatient medical care at the time of the survey.¹⁹ Furthermore, because educational attainment was an important independent variable in this study, only respondents of ≥25 years were used for this study (n = 5869).

For the statistical analysis, only cases with complete information on all variables included in the multivariate analysis were used.

This study was approved by the Ethics Committee of the Ghent University Hospital (Ref. 2001/64).

Variables of interest

To compare the self-reported health care use and the registered health care use, information on the number of contacts with general practitioner (GP) and outpatient specialist physician (SP) during the past 2 months from both the databases were used.

All the other variables derive from the BNHIS. Age was included in this study in six categories (see table 1). For privacy reasons, no more detailed information on age was available. To determine the educational level, the highest diploma of the respondent was used. To calculate the variable equivalent income, a method was applied on the household income that takes the size and the composition of the households into account²⁰. This variable was further reduced into three categories (<750€, 750€–1000€ and >1000€/month). As an indicator for ethnicity, country of birth (Belgium or not Belgium) was used.

Table 1

ICC, and comparison between self-reported and registered contact with GP for total population and for subgroups

	ICC (95% CI)	Mean (SD) Self-reported GP contact	Mean (SD) registered GP contact	t-test P-value	Accurate (%)	Under-reporting (%)	Over-reporting (%)	Cohen’s Kappaa
Total sample (n = 5128)	0.66 (0.64–0.67)	0.89 (1.48)	0.94 (1.56)	0.018	64	19	17	0.65
Gender
Male (n = 2445)	0.61 (0.58–0.63)	0.72 (1.16)	0.74 (1.31)	0.278	70	16	14	0.66
Female (n = 2683)	0.67 (0.65–0.69)	1.06 (1.70)	1.11 (1.74)	0.032	59	22	19	0.62
Age (years)
25–34 (n = 1169)	0.46 (0.41–0.51)	0.57 (1.27)	0.57 (1.03)	0.976	71	15	14	0.61
35–44 (n = 1084)	0.62 (0.59–0.66)	0.61 (1.16)	0.56 (1.12)	0.074	71	13	16	0.59
45–54 (n = 888)	0.68 (0.65–0.72)	0.75 (1.18)	0.77 (1.30)	0.624	68	16	16	0.66
55–64 (n = 778)	0.51 (0.45–0.56)	0.94 (1.23)	0.99 (1.55)	0.379	59	20	21	0.63
65–74 (n = 748)	0.54 (0.49–0.59)	1.35 (1.57)	1.44 (1.66)	0.136	55	25	20	0.63
75+ (n = 461)	0.79 (0.75–0.83)	1.87 (2.47)	2.22 (2.64)	<0.001	43	40	18	0.53
Educational level
No or primary (n = 1099)	0.75 (0.72–0.78)	1.39 (2.03)	1.54 (2.20)	0.001	55	27	18	0.69
Lower secondary (n = 1147)	0.62 (0.59–0.66)	0.87 (1.19)	0.97 (1.52)	0.005	62	21	17	0.62
Higher secondary (n = 1451)	0.51 (0.47–0.54)	0.76 (1.36)	0.72 (1.21)	0.283	67	16	17	0.63
Higher (n = 1431)	0.55 (0.51–0.58)	0.64 (1.18)	0.63 (1.14)	0.767	71	14	15	0.60
Equivalent income
<750 (n = 1262)	0.76 (0.74–0.78)	1.10 (1.80)	1.16 (1.92)	0.091	59	22	19	0.63
750–1000 (n = 1320)	0.54 (0.51–0.58)	1.02 (1.58)	1.06 (1.62)	0.246	61	21	17	0.66
>1000 (n = 2546)	0.62 (0.60–0.64)	0.74 (1.20)	0.77 (1.29)	0.182	67	17	16	0.64
Country of birth
Belgium (n = 4461)	0.66 (0.64–0.67)	0.91 (1.54)	0.96 (1.63)	0.006	64	20	17	0.65
Other (n = 667)	0.63 (0.57–0.68)	0.73 (0.94)	0.64 (0.91)	0.036	68	12	19	0.63
Self-rated health
(Very) good (n = 3661)	0.61 (0.59–0.63)	0.57 (0.94)	0.65 (1.14)	<0.001	70	17	13	0.64
Fair/(very) bad (n = 1467)	0.62 (0.59–0.65)	1.76 (2.12)	1.69 (2.14)	0.214	47	26	27	0.57
Number of chronic illnesses
0 (n = 1935)	0.71 (0.68–0.73)	0.41 (0.86)	0.50 (1.04)	<0.001	77	14	9	0.64
1 (n = 1337)	0.74 (0.72–0.76)	0.84 (1.50)	0.95 (1.72)	<0.001	64	20	16	0.64
2 or more (n = 1856)	0.53 (0.50–0.57)	1.49 (1.76)	1.43 (1.75)	0.118	49	25	26	0.57
Limitations
No (n = 3925)	0.65 (0.63–0.67)	0.61 (0.96)	0.71 (1.22)	<0.001	69	18	13	0.63
Yes (n = 1203)	0.60 (0.56–0.63)	1.87 (2.27)	1.72 (2.19)	0.011	47	25	29	0.59
GHQ12-score
0–1 (n = 3471)	0.70 (0.68–0.71)	0.73 (1.26)	0.80 (1.42)	<0.001	67	18	15	0.65
2–12 (n = 1657)	0.59 (0.56–0.62)	1.27 (1.81)	1.23 (1.78)	0.414	58	21	21	0.64

a: Based on yes or no contact. Bold indicates P < 0.05.

Self-rated health was measured with the question ‘How is your health in general?’. The respondents had to select one of the five closed-ended answering categories (‘very good’, ‘good’, ‘fair (reasonable)’, ‘bad’ and ‘very bad’). To assess the number of chronic diseases, the respondent was asked to report for a list of 34 chronic conditions whether he or she had suffered from it during the past 12 months. For the analysis in this study, the number of conditions was added and regrouped into three categories (‘0’, ‘1’, ‘2 or more’). An indicator about the functional limitations as a consequence of chronic conditions or diseases was also available, and it grouped the population into two groups: those with and those without functional limitations. As a measure of current mental health, the General Health Questionnaire-12 (GHQ-12) was used.²¹ This instrument asks with 12 questions whether the respondent has experienced a particular symptom or behaviour recently. The score ranges from 0 to 12. A score of 2 or more is an indicator for possible mental health problems.

Data analysis

Respondents were grouped into three categories: under-reporters, over-reporters and accurate reporters. The last category contains those respondents for whom self-reported and registered contacts matched perfectly. To determine the rate of agreement between the number of self-reported and registered contacts with GP and SP, intra-class correlation coefficients (ICCs), a two-way mixed model²² with measures of absolute agreement, were used. Agreement for dichotomized variables (there was a contact or there was no contact) was determined with Cohen’s Kappa. The benchmark for evaluating the strength of the ICC and Cohen’s Kappa was based on Landis and Koch.²³ They considered values of ≥0.81 as almost perfect, values ranging from 0.61 to 0.80 represent substantial agreement, values from 0.41 to 0.60 signify moderate agreement and values <0.41 signify poor to fair agreement. A systematic difference between the self-reported and registered utilization data was examined by comparing the mean number of contacts and by using the paired sample t-test.

Multinomial logistic regression was used to model factors influencing under- and over-reporting. To assess the impact of the actual utilization rate of the respondents as a confounder, models with and without number of registered contacts were estimated.

All analyses were done using the SAS Statistical Software, in accordance with the guidelines for analysis that takes the sampling design of the BNHIS into account.²⁴

Results

Table 1 provides information on the distribution of the population by the different variables used in this study. For the analysis with GP contact, information of 5128 cases was available. People dropped out of the analysis especially because information was missing on self-reported contact with GP (n = 47), self-rated health (n = 362), GHQ-12-score (n = 187), education (n = 87) and income (n = 236). Those who were not included in the analysis because of missing information compared with those who did, differed significantly on the mean number of registered GP contacts: 1.26 vs. 0.94 (P < 0.001).

For the analyses that focus on the validity of self-reported SP utilization, information on 5119 cases are available (table 2). The reasons for the dropout were to a large extent the same as for GP utilization; however, those who dropped out differed not significantly on the mean number of registered SP contact with those included in the analyses. This was 0.39 for the group that dropped out and 0.37 for the group that was included in the analyses (P = 0.435).

Table 2

ICC and comparison between self-reported and registered contact with SP for total population and for subgroups

	ICC (95% CI)	Mean (SD) Self-reported SP contact	Mean (SD) registered SP contact	t-test P -value	Accurate (%)	Under-reporting (%)	Over-reporting (%)	Cohen’s Kappaa
Total sample (n = 5119)	0.45 (0.42–0.47)	0.35 (1.05)	0.37 (0.87)	0.210	80	11	8	0.62
Gender
Male (n = 2439)	0.42 (0.39–0.45)	0.27 (0.94)	0.29 (0.76)	0.359	84	9	6	0.62
Female (n = 2680)	0.45 (0.42–0.48)	0.42 (1.15)	0.44 (0.95)	0.384	77	13	10	0.61
Age (years)
25–34 (n = 1166)	0.45 (0.40–0.49)	0.32 (0.93)	0.33 (0.84)	0.570	84	9	6	0.67
35–44 (n = 1081)	0.55 (0.50–0.59)	0.32 (0.83)	0.27 (0.74)	0.050	83	8	10	0.60
45–54 (n = 888)	0.63 (0.58–0.66)	0.37 (1.02)	0.35 (0.94)	0.617	81	10	9	0.60
55–64 (n = 775)	0.66 (0.62–0.70)	0.36 (0.79)	0.46 (0.97)	<0.001	78	14	8	0.63
65–74 (n = 748)	0.17 (0.10–0.24)	0.46 (1.77)	0.49 (0.90)	0.689	74	16	10	0.59
75+ (n = 461)	0.60 (0.54–0.66)	0.31 (0.66)	0.40 (0.83)	0.004	78	14	8	0.56
Educational level
No or primary (n = 1098)	0.25 (0.19–0.30)	0.39 (1.56)	0.38 (0.88)	0.943	79	13	8	0.63
Lower secondary (n = 1142)	0.62 (0.58–0.65)	0.33 (0.79)	0.38 (0.92)	0.018	79	12	9	0.58
Higher secondary (n = 1451)	0.52 (0.48–0.56)	0.34 (0.96)	0.36 (0.87)	0.342	82	10	8	0.61
Higher (n = 1428)	0.56 (0.52–0.59)	0.36 (0.83)	0.36 (0.82)	0.958	81	11	9	0.64
Equivalent income
<750 (n = 1258)	0.32 (0.27–0.37)	0.34 (1.23)	0.36 (0.83)	0.649	80	11	9	0.58
750–1000 (n = 1320)	0.44 (0.39–0.48)	0.37 (1.18)	0.41 (0.91)	0.229	79	13	8	0.61
>1000 (n = 2541)	0.54 (0.51–0.57)	0.35 (0.87)	0.35 (0.87)	0.595	81	10	8	0.64
Country of birth
Belgium (n = 4451)	0.46 (0.44–0.48)	0.35 (1.07)	0.37 (0.89)	0.171	81	11	8	0.62
Other (n = 668)	0.29 (0.20–0.37)	0.40 (0.91)	0.39 (0.70)	0.822	77	12	11	0.53
Self-rated health
(Very) good (n = 3652)	0.54 (0.52–0.57)	0.27 (0.76)	0.30 (0.77)	0.011	84	10	7	0.62
Fair/(very) bad (n = 1467)	0.34 (0.30–0.39)	0.58 (1.54)	0.56 (1.06)	0.700	71	16	13	0.59
Number of chronic illnesses
0 (n = 1930)	0.56 (0.53–0.59)	0.17 (0.55)	0.22 (0.64)	<0.001	88	8	5	0.59
1 (n = 1336)	0.51 (0.47–0.55)	0.34 (0.93)	0.37 (0.83)	0.225	80	12	8	0.63
2 or more (n = 1853)	0.37 (0.33–0.41)	0.56 (1.06)	0.54 (1.43)	0.564	72	15	13	0.59
Limitations
No (n = 3917)	0.56 (0.54–0.58)	0.27 (0.78)	0.31 (0.81)	<0.001	83	10	6	0.60
Yes (n = 1202)	0.29 (0.24–0.35)	0.64 (1.63)	0.57 (1.01)	0.124	71	14	15	0.61
GHQ12-score
0–1 (n = 3467)	0.58 (0.57–0.60)	0.28 (0.72)	0.32 (0.83)	<0.001	83	10	7	0.62
2–12 (n = 1652)	0.33 (0.29–0.38)	0.52 (1.53)	0.47 (0.94)	0.199	75	13	11	0.60

a: Based on yes or no contact. Bold indicates P < 0.05.

The results of the ICCs and Kappa’s show that for the total population, and for most subgroups, the agreement between self-reported and registered contact with GP was substantial, although for some subgroups, the agreement was moderate (table 1). The average difference between reported and registered GP utilization in the two months prior to the interview was 0.05 contacts for the total population. In most subgroups, there was an under-reporting of GP contacts. The largest difference was found in the oldest age group. For two subgroups, ‘individuals with a limitation’ and ‘individuals born outside Belgium’, a statistical significant over-reporting of GP utilization was stated. These results are also reflected in the percentages of subjects who under- and over-reported.

The ICCs of the agreement between the self-reported and registered SP contacts is provided in table 2. They disclosed that the agreement for the total sample and for most subgroups was moderate or rather poor. Yet, for the total sample, on average, 80% of the respondents reported the number of contacts with SP correct, varying between 71 and 88% in the subgroups. The values of the Kappa’s, calculated for dichotomized variables, were most of the time around 0.60, indicating moderate agreement. A statistical significant difference between the mean self-reported and mean registered SP contact was found for eight subgroups: the age group ‘35–44’ was over-reporting their contacts with SP; the age groups ‘55–64’, ‘75+’ and the subgroups with a degree of lower secondary education, (very) good self-rated health, without chronic illnesses, without limitations and a score on the GHQ12 of 0 or 1, were inclined to under-reporting.

Factors that predicted under-reporting of GP contacts, taking number of registered contacts into account, were having (very) good health, having no limitations and having a GHQ-12-score of 0 or 1 (table 3). The results also suggested that there was some under-reporting at higher numbers of GP contacts. Over-reporting was associated with being female, worse self-rated health, a greater number of chronic illnesses and having limitations.

Table 3

Multinomial logistic regression predicting under-reporting and over-reporting of self-reported contact with GP

	Number of registered contacts not included in the analysis		Number of registered contacts included in the analysis
	Under-reporting	Over-reporting	Under-reporting	Over-reporting
	OR (95% CI)	OR (95% CI)	OR (95% CI)	OR (95% CI)
Gender
Male	1	1	1	1
Female	1.41 (1.20–1.64)	1.31 (1.11–1.54)	1.19 (0.98–1.45)	1.30 (1.10–1.53)
Age (years)
25–34	1	1	1	1
35–44	0.75 (0.59–0.97)	0.95 (0.74–1.22)	0.88 (0.65–1.19)	0.96 (0.74–1.23)
45–54	0.95 (0.73–1.22)	0.94 (0.72–1.23)	0.81 (0.59–1.10)	0.96 (0.73–1.25)
55–64	1.16 (0.89–1.52)	1.21 (0.92–1.60)	1.02 (0.73–1.42)	1.19 (0.91–1.57)
65–74	1.47 (1.12–1.93)	1.14 (0.85–1.53)	0.78 (0.55–1.09)	1.12 (0.83–1.50)
75+	2.64 (1.97–3.53)	1.15 (0.82–1.63)	1.00 (0.68–1.46)	1.12 (0.79–1.59)
Educational level
No or primary	1	1	1	1
Lower secondary	0.95 (0.77–1.18)	1.13 (0.89–1.44)	1.23 (0.93–1.63)	1.12 (0.87–1.43)
Higher secondary	0.79 (0.63–0.99)	1.25 (0.98–1.59)	1.03 (0.77–1.37)	1.23 (0.96–1.57)
Higher	0.68 (0.53–0.88)	1.20 (0.92–1.57)	0.89 (0.65–1.23)	1.19 (0.91–1.55)
Equivalent income
<750	1	1	1	1
750–1000	0.94 (0.77–1.16)	0.96 (0.77–1.20)	0.93 (0.71–1.22)	0.96 (0.77–1.20)
>1000	0.97 (0.79–1.18)	0.96 (0.78–1.19)	0.93 (0.72–1.21)	0.94 (0.76–1.17)
Country of birth
Belgium	1	1	1	1
Other	0.57 (0.42–0.78)	1.09 (0.83–1.43)	0.79 (0.54–1.15)	1.08 (0.82–1.42)
Self-rated health
(Very) good	1	1	1	1
Fair/(very) bad	1.24 (1.01–1.52)	1.66 (1.34–2.05)	0.73 (0.55–0.96)	1.61 (1.30–1.99)
Number of chronic illnesses
0	1	1	1	1
1	1.43 (1.18–1.74)	1.84 (1.47–2.30)	1.04 (0.81–1.32)	1.82 (1.46–2.27)
2 or more	1.72 (1.40–2.11)	2.76 (2.20–3.45)	1.00 (0.77–1.30)	2.69 (2.15–3.37)
Limitations
No	1	1	1	1
Yes	1.12 (0.91–1.38)	1.61 (1.30–1.98)	0.53 (0.40–0.70)	1.58 (1.28–1.95)
GHQ12-score
0–1	1	1	1	1
2–12	1.16 (0.98–1.37)	1.12 (0.94–1.34)	0.80 (0.64–0.99)	1.14 (0.96–1.36)
Number of registered contacts
Extra contact	na	na	4.38 (3.95–4.85)	1.10 (1.00–1.22)

na: not applicable. Bold indicates P < 0.05.

After controlling for number of registered contacts, a significantly greater likelihood of under-reporting SP contacts was observed in the oldest age groups, for people without limitations and for having more registered SP contacts (table 4). Being female, being between ‘35–45’, having a greater number of chronic illnesses, having limitations and being born outside Belgium were all significant predictors of over-reporting. The number of registered contacts was also associated with over-reporting, although the magnitude of the association was smaller than for under-reporting.

Table 4

Multinomial logistic regression predicting under-reporting and over-reporting of self-reported contact with SP

	Number of registered contacts not included in the analysis		Number of registered contacts included in the analysis
	Under-reporting	Over-reporting	Under-reporting	Over-reporting
	OR (95% CI)	OR (95% CI)	OR (95% CI)	OR (95% CI)
Gender
Male	1	1	1	1
Female	1.35 (1.12–1.63)	1.50 (1.20–1.86)	1.08 (0.84–1.40)	1.42 (1.14–1.78)
Age (years)
25–34	1	1	1	1
35–44	0.79 (0.58–1.07)	1.34 (0.97–1.86)	1.02 (0.66–1.57)	1.41 (1.01–1.96)
45–54	1.09 (0.81–1.48)	1.26 (0.89–1.79)	1.47 (0.96–2.25)	1.28 (0.90–1.82)
55–64	1.48 (1.09–2.02)	1.00 (0.68–1.47)	1.59 (1.03–2.46)	0.97 (0.66–1.44)
65–74	1.72 (1.25–2.36)	1.22 (0.82–1.79)	2.15 (1.39–3.33)	1.16 (0.78–1.71)
75+	1.28 (0.89–1.85)	0.82 (0.52–1.30)	1.90 (1.16–3.13)	0.83 (0.52–1.32)
Educational level
No or primary	1	1	1	1
Lower secondary	1.23 (0.94–1.61)	1.25 (0.91–1.61)	1.17 (0.81–1.70)	1.23 (0.89–1.69)
Higher secondary	1.19 (0.90–1.56)	1.24 (0.90–1.72)	0.97 (0.66–1.43)	1.17 (0.85–1.63)
Higher	1.39 (1.03–1.88)	1.51 (1.07–2.14)	1.11 (0.73–1.69)	1.41 (0.99–1.99)
Equivalent income
<750	1	1	1	1
750–1000	1.25 (0.98–1.61)	1.04 (0.77–1.40)	1.27 (0.90–1.78)	1.01 (0.75–1.36)
>1000	1.03 (0.81–1.32)	1.17 (0.88–1.54)	0.91 (0.64–1.28)	1.14 (0.86–1.51)
Country of birth
Belgium	1	1	1	1
Other	1.17 (0.85–1.61)	1.50 (1.08–2.10)	1.14 (0.74–1.74)	1.47 (1.05–2.06)
Self-rated health
(Very) good	1	1	1	1
Fair/(very) bad	1.37 (1.07–1.74)	1.29 (0.98–1.70)	1.21 (0.86–1.69)	1.25 (0.95–1.66)
Number of chronic illnesses
0	1	1	1	1
1	1.46 (1.15–1.86)	1.57 (1.16–2.13)	1.03 (0.75–1.43)	1.50 (1.10–2.03)
2 or more	1.58 (1.23–2.03)	2.48 (1.84–3.34)	0.89 (0.63–1.25)	2.31 (1.71–3.12)
Limitations
No	1	1	1	1
Yes	1.03 (0.81–1.32)	1.73 (1.33–2.27)	0.64 (0.45–0.91)	1.61 (1.23–2.11)
GHQ12-score
0–1	1	1	1	1
2–12	1.22 (1.01–1.49)	1.26 (1.01–1.58)	1.04 (0.79–1.37)	1.20 (0.96–1.51)
Number of registered contacts
Extra contact	na	na	10.16 (8.64–11.94)	2.23 (1.88–2.65)

na: not applicable. Bold indicates P < 0.05.

Discussion

The purpose of this study was to determine the validity of self-reported GP and SP utilization and to assess factors related with the validity.

The results demonstrated a substantial agreement between the self-reported and registered GP contacts in the last two months and only a minor bias towards under-reporting. The under-reporting of self-reported health care utilization is supported in numerous studies, focusing on the different types of health care services.^1,5,7,25 On the contrary, Bellon et al.⁸ observed a net tendency to over-report the actual number of visits. This could be attributed to the composition of their sample, with an oversample of participants with health problems, or the fact that their study was limited to scheduled visits and that patients probably did not distinguish between the visits that were being investigated and unplanned visits.⁸

When observing the high percentages of accurate respondents, it seems that there is a good agreement between the self-reported and registered SP utilization. However, the ICCs showed that the agreement was rather moderate. There was no significant difference between mean self-reported and registered SP utilization.

If all the respondents under- or over-report their utilization by a constant amount, then the estimate of the correlation between the utilization and other variables would not be seriously affected. Yet, under- or over-reporting is a major problem if the bias is related to a variable of interest. The results of this study indicate that certain types of respondents are much more likely to under- or over-report.

The number of registered contacts was an important determinant of not reporting care use accurately: the more contacts, the greater is the bias towards both under-reporting and over-reporting, with the magnitude of the former association being clearly higher. These findings are in agreement with other studies.⁸ This finding is not surprising: the more frequently an individual uses health care services, the greater the opportunity for inaccuracy.² The differences in the results between the models with and without the number of registered contacts indicate that it is important to include the actual utilization rate into research that tries to determine the factors that influence the accuracy of self-report.

Health status was also associated with inaccuracy. People with good self-reported health, no limitations and better mental health were more inclined to under-report. Worse self-rated health, having chronic illnesses or limitations increase the likelihood of over-reporting. This result was already demonstrated in other studies.^2,8 Bellon et al.⁸ suggests that individuals with health problems use many different types of health care services and might confuse those services with physician visits, which explains their over-reporting. Another explanation that is put forward is that those who are most concerned about their health would over-report their use, as they might exaggerate their utilization in their own minds.² Over-reporting may also be the result of forward telescoping. Forward telescoping occurs when events from the past are recalled as taking place more recently than they did,^3,26 and persons with health problems may telescope their use more than those without health problems.

The hypothesis that persons who are more concerned with their health would over-report, offers also an explanation for the finding that women, who are generally more focused on health than men,^27,28 over-report both GP and SP utilization.

Other studies have reported an association between increasing age and greater frequency of under-reporting and hypothesize the effect of being more forgetful.^2,5,9 This finding was confirmed by this study for SP contacts but not for GP contacts.

Educational level and income had no influence on the disagreement between self-reported and registered data, which is congruent with the results of Yu et al.⁴ and Reijneveld and Stronks.¹⁴

This study found that respondents born outside Belgium had a greater likelihood to over-report their SP utilization. There is only limited information available on the cross-cultural validity of self-reporting health care utilization.²⁹ Yet, language problems, knowledge of the health care organization and cultural differences might cause a culturally determined information bias regarding self-reported health care utilization. Reijneveld²⁹ found in a study about the cross-cultural validity of self-reported use of hospitalization, physiotherapy and drugs prescription that concordance between self-reported use and registered data is lower among non-native respondents, although mostly without statistical significance, and that there was no systematic impact on the estimates of ethnic differences in health care utilization but adds measurement error to such comparisons.

The results of this study are in line with the previous research, but, whereas most of the previous studies were restricted to specific populations and many had small sample sizes, this study was the first to use a national sample, focusing on commonly used health care services in a Western European country.

In Belgium, at the time of the data collection, ∼99% of the population was covered by the National Health Insurance Funds.¹⁹ However, a limitation of this study is that most of the self-employed people, and some of their relatives, are excluded from the analysis, as this group was not covered by the National Health Insurance Funds in the same way as the other respondents.

It was also not possible to link all the participants at the BNHIS with the registered medical utilization data provided by the Belgian Health Insurance Funds, but there are no indications that this could have introduced bias.

The recall period used in this study was 2 months. A recall period of 1, 2, 3 and 6 months or 1 year are frequently used in health surveys. The literature provides no evidence on the optimal period, but Bhandari and Wagner³ demonstrated that inaccuracy increases with longer recall periods. Future studies are needed to confirm the conclusions from this study for other recall periods.

The BNHIS allowed a proxy interview for some questions in specific situations, and the proxy interview was mandatory when the person to be interviewed was very sick or cognitively impaired.¹⁶ Such a proxy interview was also allowed for the questions about physician utilization. For the objectives of this study, it was appropriate to include only respondents who answered the questions by themselves. Unfortunately, in the linked database, no information was available whether the answers were given by the respondent or by a proxy. Because a proxy interview was not permitted for some of the other variables included in this study, such as self-rated health and the GSQ-12, proxy interviews were excluded as a consequence of limiting our analysis to cases with complete information on all variables included in the multivariate analysis. The high number of missing information on the variables self-rated health and GSQ-12 was probably caused because a proxy interview was not allowed for those questions.

This study indicated that dropout owing to missing information caused bias. Those who were not included had substantial more contact with the GP than those who were included. This demonstrated maybe the most important threat for the external validity of self-reported health services utilization. Those who experience illnesses and declines in physical functions and who use health care services the most, may not only fail to accurately report the use of health care services, but they may fail to participate, partial or completely, at a health survey. Research about the effect of non-response on estimates of health care utilization concluded that different types of non-response had different bias effects but that illness is the most important contributor to non-response bias, although the general validity of the results may not be threatened.³⁰

To conclude, the implications of these results for research on the utilization of physician services using self-reported information depend on the objectives of the research at issue. The frequency of contact with a GP will be under-estimated on a population level, especially for some sub-populations. The population estimate of SP visits will be accurate. The results of the analysis with the ICCs showed that self-reported GP utilization is valid to discriminate between the subjects in the sample, but the self-reported SP utilization is not. Furthermore, studies that aim to compare the utilization of different socio-demographic groups have to take into account that reporting error varies by respondent’s characteristics.

Conflicts of interest: None declared.

Key points

• This study examined factors that determined under- and over-reporting of self-reported utilization of general practitioner and specialist physician services by linking a national health interview survey with data provided by Health Insurance Funds.

• The implications of the results for research on utilization of physician services using self-reported information depend on the objectives of the research at issue.

• The population estimate of the contact with general practitioners will be under-estimated; the estimate of specialist physicians will be accurate.

• Studies that aim to compare the utilization of different socio-demographic groups have to take into account that reporting error varies by respondents characteristics.