https://orcid.org/0000-0001-9334-8230
Abstract
The burden of sexually transmitted infections (STIs) continues to increase in developing countries like China, but the access to STI care is often limited. The emergence of direct-to-consumer (DTC) telemedicine offers unique opportunities for patients to directly access health services when needed. However, the quality of STI care provided by telemedicine platforms remains unknown. After systemically identifying the universe of DTC telemedicine platforms providing on-demand consultations in China in 2019, we evaluated their quality using the method of unannounced standardized patients (SPs). SPs presented routine cases of syphilis and herpes. Of the 110 SP visits conducted, physicians made a correct diagnosis in 44.5% (95% CI: 35.1% to 54.0%) of SP visits, and correctly managed 10.9% (95% CI: 5.0% to 16.8%). Low rates of correct management were primarily attributable to the failure of physicians to refer patients for STI testing. Controlling for other factors, videoconference (vs SMS-based) consultation mode and the availability of public physician ratings were associated with higher-quality care. Our findings suggest a need for further research on the causal determinants of care quality on DTC telemedicine platforms and effective policy approaches to promote their potential to expand access to STI care in developing countries while limiting potential unintended consequences for patients.
We identified all direct-to-consumer (DTC) telemedicine platforms and used the standardized patient (SP) method to evaluate the quality of care provided.
Physicians made a correct diagnosis in 44.5% of SP visits, and correctly managed 10.9%.
Low rates of correct management were primarily attributable to the failure of physicians to refer patients for sexually transmitted infection (STI) testing.
Introduction
The burden of sexually transmitted infections (STIs) continues to increase globally (Zheng et al., 2022), especially in developing countries like China (Dong et al., 2020; Du et al., 2022). In 2019, syphilis ranked third highest infectious disease in China, following viral hepatitis and tuberculosis (National Health Commission of the People’s Republic of China, 2020). From 2009 to 2019, the incidence of syphilis rapidly increased from 24.7 per 100 000 people to 38.4 per 100 000 people (National Health Commission of the People’s Republic of China, 2020), calling for strong attention for the effective provision of syphilis and other STIs care services. At the same time, a large proportion of STIs remain undiagnosed and untreated (Conley et al., 2015; UNAIDS, 2022), due to negative attitudes towards STIs and social stigma that prevent infected individuals from seeking care (WHO, 2011; UNAIDS, 2014; Yan et al., 2019; Smith et al., 2020). Undiagnosed STIs are likely to be significantly larger in underdeveloped areas with limited accessibility to quality-ensured STI care services.
Due to its convenience, privacy and low cost, direct-to-consumer (DTC) telemedicine provides a new and rapidly evolving model to provide medical consultations particularly for population at stakes. While telemedicine has been in use since the early 2000s and has increased rapidly in many high and middle-income countries since, it has primarily been used where there is an established relationship between the provider or health facility and the patient (Barnett et al., 2018). The distinguishing feature of DTC telemedicine platforms, which have emerged more recently, is that they facilitate the direct matching of patients with providers where there is no prior relationship due to geographical barriers (Elliott and Yopes, 2019; Fogel and Kvedar, 2019; Cohen et al., 2020). Patients are able to request consultations through these platforms directly on their computer and mobile device, most often choosing from a list of available providers. This relaxes geographical constraints on the providers to whom patients have access and to whom providers can offer services, and has potential to change the nature of medical consultations in a number of other dimensions that have unknown effects on quality.
The fast increase in internet users in China (from 22.7% in 2008 to 75.6% in 2022) well positions its development of telemedicine platforms (Statista, 2023). In November 2019, about 46.9 million people were recorded active DTC telemedicine users (i.e. seeking online medical advice, purchasing medications or scheduling appointments), and the number increased to 54.8 million in November 2020 (Cheng et al., 2022). In addition, the Chinese government committed to develop the internet ecosystem to address persistent problems within its health system, and improve the quality and efficiency of healthcare delivery systems (Yip et al., 2019). For example, widespread mistrust in the quality of health professionals at primary care drove patients to visit higher-level hospitals, exacerbating problems of overcrowding and long waiting lines (Yip and Hsiao, 2014). DTC telemedicine provides the convenience of a virtual appointment with doctors without having to visit overcrowded public tertiary hospitals. It may also alleviate geographical disparities in access to a high-quality health workforce (Li et al., 2018).
Since social stigma is a significant barrier to STI care, the public health implications of the emergence and growth of DTC telemedicine platforms may be particularly large for STI control. STI patients often seek healthcare services on DTC telemedicine platforms, either via teleconferencing or text messages, for the abovementioned advantages (Muessig et al., 2015). Next to mental health, STI-related conditions have been shown to represent the largest proportion of consultations on some platforms (Lv et al., 2019). Although DTC platforms have potential to dramatically expand access to medical expertise for STIs and improve rates of diagnosis and correct management (Resneck et al., 2016), the quality of care provided to patients presenting with STI symptoms on these platforms is unknown. Failure to understand the quality of care provided by DTC telemedicine platforms may lead to ineffective and inefficient policy interventions with the aim to strengthen health system as a whole.
To fill this gap, this study aims to evaluate the quality of STI care services provided on DTC telemedicine platforms in China using standardized patients (SPs). SPs are actors trained to consistently present symptoms to healthcare providers (Das et al., 2012; Sylvia et al., 2015). As the gold-standard method to measure the quality of primary care, observing clinical interactions via unannounced visits by SPs has been used in a growing number of studies (Kwan et al., 2019) and are less likely to be subject to recall bias.
Institutional context
A growing number of DTC telemedicine platforms have emerged in China, akin to Teladoc in the USA, DocOnline in India and Halodoc in Indonisia. These platforms serve a distinct role as third-party intermediaries to facilitate interactions between patients and physicians (Cheng et al., 2022), using either teleconference or text-based interactions. Teleconference visits involve real-time (synchronous) bi-directional video conferencing while SMS/text-based encounters involve messages exchanged asynchronously between providers and patients over hours or days (Elliott and Yopes, 2019). These platforms are often established by third-party (private) companies, although a growing number of public hospitals have extended to provide online healthcare services during the COVID-19 pandemic (Han et al., 2020). For example, Haodf.com, WeDoctor, Chunyu Doctor and Ping’an Good Doctor are four leading third-party telemedicine platforms, accounting for over 60% of active users of the top 10 DTC telemedicine platforms in China.
While several leading platforms are run by private entities, public hospital physicians form the core of DTC telemedicine physician workforce (Cheng et al., 2022). This is known as ‘online dual practice’ (Xu et al., 2022). Moreover, patient visits on these platforms are very similar to those in public hospitals, since both telemedicine platforms and public hospitals use a fee-for-service payment system in China. Generally, patients can choose any physician freely and pay a fixed consultation fee to initiate a consultation. Via the consultation, physicians make diagnoses and recommend tests and drugs to be prescribed based on his/her judgement of the patient’s condition (Si et al., 2023).
Methods
Study design
This study involved two phases of data collection. First, we conducted a systematic search for online hospitals in August–October 2019 using a pre-defined list of 21 search terms (i.e. internet hospitals, telemedicine and e-health), to include all DTC telemedicine platforms providing on-demand consultations within China at that time. After the systematic search for the universe of DTC telemedicine platforms providing on-demand consultations in China in 2019 (Xue et al., 2021), we worked with SPs to evaluate the quality of STIs care provided by all identified DTC telemedicine platforms For analysis in the study, we included successful SP visits for all identified DTC telemedicine platforms and excluded SP visits that were unsuccessful or had incomplete quality information.
Standardized patients
We recruited 11 male SPs from Sun Yat-Sen University in Guangzhou China in October 2019. SPs presented disease cases depicting routine (uncomplicated) syphilis and herpes. These two STIs were chosen due to their prevalence and public health importance in China as well as their suitability for the SP method given plausible lack of obvious physiological symptoms that could be ascertained.
SPs were trained to present their chief complaint consistently (i.e. ‘Doctor, I had something growing down there and I’m worried it might affect my girlfriend.’ for syphilis, and ‘Doctor, recently I have been feeling uncomfortable down there, the skin was broken, and a bit painful.’ for genital herpes.) and then answer any question asked by physicians following scripts (see both Chinese and English versions in Supplement 1).
It should be noted that the visible symptoms for syphilis had gone away before the e-consultation, consistent with the progression of syphilis. For genital herpes, SPs consistently refused the request of showing the genital because SPs were not in a good place to do that. For any question asked by physicians, SPs answered them following set scripts (Supplement 1). These scripts were developed to ensure that the information presented and language used were consistent across all interactions and training-emphasized standardization across SPs.
SP visits
For each disease case, SPs conducted e-consults via teleconference and SMS/text (when available). SPs were assigned to platforms and video or SMS/text-based visits randomly. On each platform, SPs booked e-consults with providers following the process of a typical patient as closely as possible (i.e. generally in the order of selecting physicians recommended by platforms). If provided a choice of physician speciality, physicians from the department of dermatology or venereology were given priority, and general practitioners were visited as an alternative if there was no dermatologist or venereologist. When presented with a list of providers, the first physician listed by the platform was chosen.
The information collected from each interaction included the consultation price, timing of the encounter, any questions asked by providers, recommendations made including referral, drugs prescribed, and other further guidance offered to patients. In addition, enumerators recorded the professional rank and gender of physicians, but did not record any identifying information on individual physicians.
Similar to previous studies (Das et al., 2012; Sylvia et al., 2015), we assigned SPs to visit all eligible DTC platforms to provide descriptive evidence on the quality of care. For video-based platforms, SPs both presenting syphilis and herpes randomly visited them on a workday, and were required to do a second-round visit one week later after the initial consultation. It is noted that SPs rarely visited the same physician across the two rounds as the DTC platforms have many physicians online simultaneously. For text-based platforms, only SPs presenting syphilis were assigned to visit these platforms once. A total of 112 SP visits were successful and 2 visits were excluded from our analysis due to incomplete interaction and price information.
Quality control
Quality control in the study included: (1) We included two assistants to conduct platform screening and selection and cross-validated the results; (2) SPs received the same intensive training and followed the same scripts; (3) to assure the validity of data collection, clinical encounters between SPs and providers were observed by trained enumerators who entered data following a set form (i.e. computer-assisted process to assure the high quality of data entry). The encounter information was further double checked by the SP after his interaction with a physician.
Ethical approval
Ethical approvals were obtained from the institutional review boards (IRBs) of the University of North Carolina (protocol number: 257 307) and Guangdong Second Provincial General Hospital (protocol number: 2019-ITT-1008-02). The informed consent was waived given that only de-identified information on physicians was collected and the study posed minimal risk. SPs paid for the consultation and all drugs prescribed.
Healthcare quality
We focus on one aspect of quality—the degree to which patients receive timely and accurate diagnosis and evidence-based treatments for their conditions (Das et al., 2018). The quality of care was evaluated based on guidelines for diagnosis and treatment of syphilis and herpes published by the National Center for STD Control of China, Venereology Group (Center for STD Control, Chinese Center for Disease Control and Prevention, 2014; National Center for STD Control CC for DC, Prevention; Venereology Group CDA, 2020). Generally, using appropriate history-taking questions and tests for SP cases should lead to correct and unambiguous diagnosis and treatment. We report four metrics regarding physician diagnostic skills and disease management to reflect the quality of STI care services.
Adherence to a checklist of recommended questions and exams
For analysis, this is coded as a continuous variable ranging from 0 to 1, defined as the proportion of recommended questions and exams recommendations based on national standards in China and international standards for each disease case (Center for STD Control, Chinese Center for Disease Control and Prevention, 2014; National Center for STD Control CC for DC, Prevention; Venereology Group CDA, 2020). The details regarding the check list can be found in Supplement 1. The distribution of each specific item in the study has been displayed in Figure S1.
Adherence to an ‘essential’ checklist
This is a subset of items from the longer checklist of recommended questions and exams deemed as being essential to properly diagnose each disease case. For analysis, this is coded as a continuous variable ranging from 0 to 1, defined as the proportion of ‘essential’ questions and exams completed (Center for STD Control, Chinese Center for Disease Control and Prevention, 2014; National Center for STD Control CC for DC, Prevention; Venereology Group CDA, 2020).
Correct diagnosis (0/1)
The diagnosis provided by consulting physicians was classified as ‘correct’, ‘partially correct’ or ‘incorrect’. Please note that in the study the correct diagnosis is ‘preliminary’ as it was not confirmed by STI test. Classification details can be found in Supplement 2 (Center for STD Control, Chinese Center for Disease Control and Prevention, 2014; National Center for STD Control CC for DC, Prevention; Venereology Group CDA, 2020).
Correct management (0/1)
The primary metric of interest is the correct management of disease. This is coded as a binary variable following pre-specified definitions based on standards for each disease case. Correct disease management was defined as a joint decision of a referral to an in-person hospital, advice of STI test and no drug prescription (Center for STD Control, Chinese Center for Disease Control and Prevention, 2014; National Center for STD Control CC for DC, Prevention; Venereology Group CDA, 2020). Classification details and its sub-items can be found in Supplement 2.
Statistical analysis
Data were summarized for the full sample and separated into two subsamples: video-based visits and text-based visits. First, we present a descriptive analysis of the quality metrics (for each disease). We report the number (No.) and percent of visits (%) for binary variables and the mean and standard deviation (SD) for continuous variables.
Second, we used a decision tree analysis to enable a thorough mapping of all patient management decisions. This classifies combinations of management components and provides graphical representation classifying patient management outcomes in detail (Chang and Chen, 2009).
Third, we examined the association between quality metrics and all observable hospital and physician characteristics.
Public rating: Denotes the presence of public ratings of physicians on DTC telemedicine platforms. Patients are increasingly using physician review information to find ‘a good physician’ (Okike et al., 2016).
Medical appointment: Whether a medical appointment was necessary for the SP visit and scheduled in advance. Medical appointment scheduling is considered the starting point of most non-urgent healthcare services, which supports active involvement of patients with identified positive effects (Zhao et al., 2017).
Consultation fee: Consultation fee measured in Chinese Yuan (CNY), as physicians can set the price of consultation on DTC platforms based on their own perceptions and preference of ‘competence’.
Physician gender: Growing evidence supports that female physician perform better than their male counterparts (Bertakis et al., 2009; Tsugawa et al., 2017).
Specialist vs Generalist: Physician was affiliated to a general department or a specialty medicine department. Whether care for patients is more consistent with national guidelines for specialist rather than generalist shows very mixed results in other contexts (Diette et al., 2001; Landon et al., 2005).
Physician’s professional rank: Physicians with more experience are generally believed to have accumulated knowledge and skills during years in practice and therefore to deliver higher-quality care though empirical evidence is mixed (Choudhry et al., 2005). In this study, senior physicians are identified as those with a rank of chief (or associate) physician as rank in the Chinese hospital system closely tracks experience.
We used generalized linear models for continuous outcomes and probit regressions for binary outcomes to identify the conditional correlations between quality metrics and observed hospital and physician characteristics. We presented the adjusted association (the coefficient on the indicator) and 95% confidence intervals adjusted for clustering at the platform level. Average marginal effects are reported for all regressions. All analyses were performed using Stata 16.1 (Stata Corporation, College Station, TX).
Results
DTC telemedicine platforms in China
We identified 949 potential platforms. Then, 54 duplicates were removed, and in a pilot test, we found that 859 platforms failed to provide healthcare consultations (Xue et al., 2021). This yielded a final list of 36 DTC telemedicine platforms (Table S1). Platforms offered two modes of telemedicine consultations via synchronous video-conference (N = 29) and/or asynchronous text- (SMS-) messages exchanging over hours or days (N = 33).
Quality of STI care service
Of the 110 visits, 70 (63.6%) were for syphilis and 40 (36.4%) were for herpes. In addition, 63 of these visits (57.3%) involved physicians with public rating information, 60 visits (54.6%) required an appointment made in advance and the average consultation fee for each visit was 71.6 RMB ($11.2 US dollars, SD = 9.8). As for physician characteristics, 66 visits (60.0%) had male physicians, 12 visits (10.9%) had physicians from general medicine and 59 visits (53.6%) had physicians with a professional rank of chief (or associate) physician (Table 1).
Disease, platform and physician characteristics of SP visits (N = 110)
| Video-based | Text-based | Total | |||||
|---|---|---|---|---|---|---|---|
| No. | % | No. | % | No. | % | ||
| Disease presented | Syphilis | 45 | 52.9 | 25 | 100.0 | 70 | 63.6 |
| Herpes | 40 | 47.1 | 0 | 0.0 | 40 | 36.4 | |
| Platform characteristic | |||||||
| Public rating | No | 37 | 44.1 | 10 | 38.5 | 47 | 42.7 |
| Yes | 47 | 56.0 | 16 | 61.5 | 63 | 57.3 | |
| Medical appointment | No | 26 | 31.0 | 24 | 92.3 | 50 | 45.5 |
| Yes | 58 | 69.1 | 2 | 7.7 | 60 | 54.6 | |
| Consultation fee (CNY), mean (S.D.) | 80.24 | (65.60) | 43.67 | (42.14) | 71.60 | (62.67) | |
| Physician characteristics | No. | % | No. | % | No. | % | |
| Physician gender | Female | 34 | 40.5 | 10 | 38.5 | 44 | 40.0 |
| Male | 50 | 59.5 | 16 | 61.5 | 66 | 60.0 | |
| Affiliated department | Specialized | 72 | 85.7 | 26 | 100.0 | 98 | 89.1 |
| General | 12 | 14.3 | 0 | 0.0 | 12 | 10.9 | |
| Professional rank | Resident | 40 | 47.6 | 11 | 44.0 | 51 | 46.4 |
| (Deputy) Director | 45 | 52.9 | 14 | 56.0 | 59 | 53.6 | |
| Video-based | Text-based | Total | |||||
|---|---|---|---|---|---|---|---|
| No. | % | No. | % | No. | % | ||
| Disease presented | Syphilis | 45 | 52.9 | 25 | 100.0 | 70 | 63.6 |
| Herpes | 40 | 47.1 | 0 | 0.0 | 40 | 36.4 | |
| Platform characteristic | |||||||
| Public rating | No | 37 | 44.1 | 10 | 38.5 | 47 | 42.7 |
| Yes | 47 | 56.0 | 16 | 61.5 | 63 | 57.3 | |
| Medical appointment | No | 26 | 31.0 | 24 | 92.3 | 50 | 45.5 |
| Yes | 58 | 69.1 | 2 | 7.7 | 60 | 54.6 | |
| Consultation fee (CNY), mean (S.D.) | 80.24 | (65.60) | 43.67 | (42.14) | 71.60 | (62.67) | |
| Physician characteristics | No. | % | No. | % | No. | % | |
| Physician gender | Female | 34 | 40.5 | 10 | 38.5 | 44 | 40.0 |
| Male | 50 | 59.5 | 16 | 61.5 | 66 | 60.0 | |
| Affiliated department | Specialized | 72 | 85.7 | 26 | 100.0 | 98 | 89.1 |
| General | 12 | 14.3 | 0 | 0.0 | 12 | 10.9 | |
| Professional rank | Resident | 40 | 47.6 | 11 | 44.0 | 51 | 46.4 |
| (Deputy) Director | 45 | 52.9 | 14 | 56.0 | 59 | 53.6 | |
Note: Data are No. (%) for binary variables and Mean (SD) for continuous variables. Exchange rate, 6.37 CNY ≈ 1 US dollar.
Disease, platform and physician characteristics of SP visits (N = 110)
| Video-based | Text-based | Total | |||||
|---|---|---|---|---|---|---|---|
| No. | % | No. | % | No. | % | ||
| Disease presented | Syphilis | 45 | 52.9 | 25 | 100.0 | 70 | 63.6 |
| Herpes | 40 | 47.1 | 0 | 0.0 | 40 | 36.4 | |
| Platform characteristic | |||||||
| Public rating | No | 37 | 44.1 | 10 | 38.5 | 47 | 42.7 |
| Yes | 47 | 56.0 | 16 | 61.5 | 63 | 57.3 | |
| Medical appointment | No | 26 | 31.0 | 24 | 92.3 | 50 | 45.5 |
| Yes | 58 | 69.1 | 2 | 7.7 | 60 | 54.6 | |
| Consultation fee (CNY), mean (S.D.) | 80.24 | (65.60) | 43.67 | (42.14) | 71.60 | (62.67) | |
| Physician characteristics | No. | % | No. | % | No. | % | |
| Physician gender | Female | 34 | 40.5 | 10 | 38.5 | 44 | 40.0 |
| Male | 50 | 59.5 | 16 | 61.5 | 66 | 60.0 | |
| Affiliated department | Specialized | 72 | 85.7 | 26 | 100.0 | 98 | 89.1 |
| General | 12 | 14.3 | 0 | 0.0 | 12 | 10.9 | |
| Professional rank | Resident | 40 | 47.6 | 11 | 44.0 | 51 | 46.4 |
| (Deputy) Director | 45 | 52.9 | 14 | 56.0 | 59 | 53.6 | |
| Video-based | Text-based | Total | |||||
|---|---|---|---|---|---|---|---|
| No. | % | No. | % | No. | % | ||
| Disease presented | Syphilis | 45 | 52.9 | 25 | 100.0 | 70 | 63.6 |
| Herpes | 40 | 47.1 | 0 | 0.0 | 40 | 36.4 | |
| Platform characteristic | |||||||
| Public rating | No | 37 | 44.1 | 10 | 38.5 | 47 | 42.7 |
| Yes | 47 | 56.0 | 16 | 61.5 | 63 | 57.3 | |
| Medical appointment | No | 26 | 31.0 | 24 | 92.3 | 50 | 45.5 |
| Yes | 58 | 69.1 | 2 | 7.7 | 60 | 54.6 | |
| Consultation fee (CNY), mean (S.D.) | 80.24 | (65.60) | 43.67 | (42.14) | 71.60 | (62.67) | |
| Physician characteristics | No. | % | No. | % | No. | % | |
| Physician gender | Female | 34 | 40.5 | 10 | 38.5 | 44 | 40.0 |
| Male | 50 | 59.5 | 16 | 61.5 | 66 | 60.0 | |
| Affiliated department | Specialized | 72 | 85.7 | 26 | 100.0 | 98 | 89.1 |
| General | 12 | 14.3 | 0 | 0.0 | 12 | 10.9 | |
| Professional rank | Resident | 40 | 47.6 | 11 | 44.0 | 51 | 46.4 |
| (Deputy) Director | 45 | 52.9 | 14 | 56.0 | 59 | 53.6 | |
Note: Data are No. (%) for binary variables and Mean (SD) for continuous variables. Exchange rate, 6.37 CNY ≈ 1 US dollar.
Overall, physicians performed 16.8% (95% CI: 15.2% to 18.5%) of the recommended diagnostic checklist items and 30.5% (95% CI: 27.1% to 33.9%) of essential checklist items for all SP visits. The details about checklist distribution for the two diseases are displayed in Figure 1. Physicians made a ‘preliminary’ correct diagnosis in 44.5% (95% CI: 35.1% to 54.0%) of SP visits. Patients were fully correctly managed given the disease cases in 10.9% (95% CI: 5.0% to 16.8%) of visits and partially correctly managed in 39.1% (95% CI: 29.8% to 48.4%). Physicians recommended a referral to in-person hospital in 46.4% (95% CI: 36.9% to 55.8%) of the SP interactions, recommended an STI test in 15.5% (95% CI: 8.6% to 22.3%) and prescribed drugs in 20.0% (95% CI: 12.4% to 27.6%). However, 72.7% of SP visits with drugs prescribed were unnecessary. The results grouped by diseases were not more encouraging although we did find some variations (Table S2).
The distribution of each specific item in the checklist
We find that physician’s performance of STI care via video-based consults was superior to that via text-based consults. The comparison of specific metrics on physician’s diagnostic skills and disease management is presented in Figure 2. Physicians via video-based visits completed significantly more checklist items (18.6% vs 10.8%; P < 0.001) and the essential checklist items (32.6% vs 23.4%; P = 0.03). This contributed to significantly higher rates of correct diagnosis (54.1% vs 12.0%; P < 0.001) compared to their counterpart physicians via text-based visits. However, physicians consulting patients via video-based consults had a higher probability of prescribing drugs than those via text-based consults (24.7% vs 4.0%; P =0.02). We did not find any significant differences in rates of correct or partially correct disease management across video- and text-based consults (i.e. referral and recommending a STI test).
The quality of STIs care by delivery model
A decision tree analysis was used to explore the components of disease management in more detail (Figure 3). SPs were fully correctly managed in 12 visits, meaning that they were referred to an in-person hospital, advised to take an STI test and not given any drug prescription. We found that more than half of SP visits did not recommend a referral to in-person hospitals, of which most (93.2%, or 55 out of 59) were not recommended an STI test. Even when a referral was made, few patients were recommended to take an STI test explicitly (25.5%, or 13 out of 51). The large majority of patients (80.0%, or 88 of 110) were not prescribed drugs; however, of the 20% where drugs were prescribed, clinicians prescribed only the correct drug in just 27% of these interactions. The rate of any drug prescription was comparable whether or not the patient was also referred to an in-person facility.
Decision tree of disease management
Correlates of STI care quality
Correlates of STI care quality were evaluated using regression models for four outcomes of primary interest: the proportion of checklist items completed, the proportion of essential checklist items completed, fully correct diagnosis and fully correct management (Figure 4, upper panel). First, video-based (rather than text-based) visits were significantly associated with 10.5% (95% CI: 5.8% to 15.2%; P < 0.001) higher adherence to guideline checklist items, and syphilis cases (rather than herpes) saw 5.2% (95% CI: 2.4 % to 7.9%; P = 0.01) higher performance than herpes visits. This pattern was similar when we focused on physicians’ completion of essential checklist items, except that physicians from specialized departments (rather than general medicine) completed 11.8% (95% CI: 1.9% to 21.6%; P = 0.02) more essential checklist items. Second, herpes visits were significantly associated with a 49.0 (95% CI: 29.8 to 68.3; P < 0.001) percentage-point (pp) increase in receiving a correct diagnosis than syphilis visits. In addition, physicians with publicly posted ratings on the platform were 17.9 (95% CI: 3.0 to 32.7; P = 0.02) pp more likely to provide a correct diagnosis than those without. Third, physicians requiring medical appointment in advance were 12.1 (95% CI: 2.4 to 21.8; P = 0.01) pp more likely to correctly manage patients than those who did not.
Correlates of diagnostic skills and disease management
Disease management was further evaluated using more specific and lenient metrics (Figure 4, lower panel). First, video-based visits were associated with a 26.8 (95% CI: 0.2 to 53.3; P = 0.05) pp increase in partially correct management and this was mainly driven by a 25.4 (95% CI: 1.1 to 49.8; P = 0.04) pp increase in receiving a referral compared to text-based visits. In addition, syphilis patients were 31.7 (95% CI: 1.2 to 51.4; P = 0.002) pp more likely to receive a referral than herpes visits. Second, visits with an appointment were 11.4 (95% CI: 1.6 to 21.2; P = 0.02) pp more likely to be recommended an STI test than those without. Third, herpes visits, visits with medical appointment and visits involving physicians from general medicine were associated with 16.1 (95% CI: 5.3 to 26.9; P = 0.003), 21.7 (95% CI: 6.7 to 36.6; P = 0.005) and 21.7 (95% CI: 3.7 to 39.7; P = 0.02) pp higher probability of being prescribed drugs, respectively. Female physicians were 13.3 (95% CI: 2.6 to 24.1; P = 0.02) percentage points more likely to provide any drug prescription.
We performed four sets of additional analysis to examine the reliability of our results. First, we performed analysis of collinearity diagnostics to address the concern of collinearity of covariates (Table S5). Second, by introducing patient or platform fixed effects, we checked the robustness of our results although we cannot get much observable information about physicians and platforms (Table S6 and Table S7). Third, we excluded physical exams and STI tests from the checklist (Table S8). Fourth, we defined physical exams and STI tests as ‘done’ if the doctor suggested the patient go to a hospital (Table S9). None of them altered our results substantially. Finally, we compared the first-round and second-round visits and found no significant differences (Table S10).
Discussion
The emergence of the direct-to-consumer telemedicine industry calls for strong attention in the quality evaluation and monitoring, especially in developing countries where these service are becoming prominent (Elliott and Yopes, 2019; Hollander and Carr, 2020). This study aimed to fill this gap by evaluating the quality of STI care services provided by DTC telemedicine platforms in China. Our study extended the existing literature by systematically identifying DTC telemedicine platforms providing consultations, and using the unannounced SP method to rigorously evaluate the quality of care provided to patients presenting with STI symptoms. We found that patients were given a ‘preliminary’ correct diagnosis in only 44.5% of consults and only partially correctly managed in 50.0%. As a point of comparison with in-person consultations, in a separate study using unannounced SPs presenting the same syphilis case, we found that providers in urban STI care clinics recommended syphilis testing in 84.6% of interactions (vs 15.5% here). Of course, many patients lack easy access to in-person STI care in urban areas.
Independent of how the quality of consultations for patients presenting with STI symptoms compare to in-person visits, the low quality of care that we find on DTC platforms is of concern. Particularly in areas where STI patients lack easy access to in-person care or in-person care is costly or inconvenient, these platforms offer potential to dramatically expand access to quality medical advice and address low rates of diagnosis and treatment at the population level. Further research is needed to understand the factors that affect the care quality on these platforms It is likely that the factors contributing to the quality of consultations differ from those in more traditional in-person clinical settings, depending on the composition of providers participating on these platforms and whether incentives stemming from platform design and remuneration are aligned with high-quality care (Xue et al., 2021).
For STIs, a particular concern is that physicians consulting on DTC telemedicine platforms seemed unwilling to refer patients to in-person hospitals for STI tests. It is reasonable to argue that physicians may not feel it necessary to recommend patients to do genital exam given the virtual nature of visits. However, it can be less of a concern because our further analysis showed that, after the e-consultation as an indication physicians’ preference of suggesting, about 40% of physicians (44/110) initiatively gave patients suggestions to manage the diseases. This is in contrast to an extremely low rate of suggesting a STI test (15.5%). We find this to be the case even though a larger proportion of physicians provide a ‘preliminary’ correct diagnosis. It is likely that this is at least in part due to the nature of ‘gig’ work on platforms such as these: Physicians are independent contractors and are paid based on a set percentage of their earnings from consultation fees (Cheng et al., 2022). Physician’s income is not strongly linked to performance, including the appropriate referral or recommendation of STI testing.
Although public ratings may provide some incentives for physicians to provide higher quality care, it is difficult for patients to accurately assess the quality of care they receive, and ratings can be weakly correlated with objective performance as a result (Dulleck et al., 2011). We do find that physicians on platforms with public ratings are more likely to provide accurate diagnoses, but more research is needed to determine whether public ratings have a causal effect on the quality of care provided. In other contexts, reputation systems, such as hospital report cards, have been shown to improve care quality (Tadelis, 2016).
Overall, our results suggest a need for further research into how interventions may affect the quality of care provided through DTC telemedicine platforms in China and other, either developed or developing, countries that share the same aspiration to strengthen their health systems The emergence of these platforms represents an opportunity to dramatically expand access to quality medical advice, yet this opportunity may be squandered unless these platforms are properly regulated and incentivized to design their processes to promote better patient care. In contrast to in-person facilities, online platforms have a number of novel mechanisms to influence care quality through how providers are admitted to the platform, how they are enumerated, how quality data are collected and publicly reported and what information is provided to physicians to support consultations.
Several limitations in the study should be acknowledged. First, it is noted that only two cases presenting STIs were used in unannounced SP visits and we were measuring the ‘true’ quality of healthcare regarding STIs since no SPs reported any indication that they were suspected as fake patients. While these two STIs (syphilis and herpes) represent two most common STIs and are each of major public health concern, it is possible that the quality of care for patients presenting with symptoms of other STIs differs. Second, the use of only two cases leads to a moderately small sample size in the study, although the sample is big enough compared to similar SP study and was sufficient to answer the research questions (Das et al., 2012; Sylvia et al., 2015). Third, we only included male SPs in the study, though results may differ for female patients (Si et al., 2019). Fourth, we did not identify many characteristics of physician (i.e. education level and employment status) and the ownership of platform as well in the analysis. The absence of these variables can potentially bias the coefficient estimations for other observed physician and platform characteristics in the regression analysis. However, it would not impact our inference on healthcare quality metrics. Fifth, the cross-sectional design of SP study prevented us from making causal inferences. Finally, this study was conducted before the outbreak of the COVID-19 pandemic; the landscape of DTC telemedicine platforms has evolved rapidly in China and elsewhere. Subsequent studies examining the post-pandemic state and quality provided through DTC platforms will be needed.
In conclusion, we report novel evidence on the quality of e-consultations for patients presenting with symptoms of STIs on DTC telemedicine platforms in China. DTC telemedicine has the potential to expand access to high-quality healthcare services, especially for residents in underdeveloped areas. Our findings, however, suggest a need for policy attention to how best to promote high-quality STI care on e-consultation platforms such as these. As for policy makers in developing and developed countries, exploring novel mechanisms to effectively integrate DTC telemedicine platforms and in-person care would potentially introduce significant gains for both patients and health systems as a whole. To better inform these policies, future research should explore the causal effects of platform design features on the quality of care.
Supplementary data
Supplementary data is available at HEAPOL Journal online.
Data Availability
Data and code are available for download on the Harvard Dataverse at https://doi.org/10.15139/S3/LKMEEW.
Funding
This study was supported by the Key-Area Research and Development Program of Guangdong Province (No. 2020B0101130020).
Author Contributions
Yafei Si: Analysis, Writing – Original Draft; Hao Xue: Conceptualization, Investigation, Writing – Reviewing & Editing; Huipeng Liao: Analysis, Investigation, Writing – Reviewing & Editing; Yewei Xie: Analysis, Writing – Reviewing & Editing; Dong Xu: Methodology, Investigation, Writing – Review & Editing; Kumi Smith: Methodology, Writing – Review & Editing; Winnie Yip: Conceptualization, Writing – Review & Editing; Weibin Cheng: Conceptualization, Investigation, Writing – Review & Editing; Junzhang Tian: Conceptualization, Investigation, Writing – Review & Editing; Weiming Tang: Conceptualization, Funding Acquisition, Methodology, Writing – Original Draft; Sean Sylvia: Conceptualization, Funding Acquisition, Methodology, Investigation, Writing – Original Draft.
Reflexivity statement
The authors include three females and eight males and span multiple levels of seniority. While three of the authors specialize in health care quality and health policy in China and West Pacific, the rest includes epidemiologists, economists and medical staffs with expertise in sextually transmitted diseases, health economics and global health in the Africa and Asia. All authors have extensive experience conducting fieldwork in developing countries, especially in China and India.
Ethical approval.
Ethical approvals were obtained from the institutional review boards (IRBs) of the University of North Carolina (protocol number: 257307) and Guangdong Second Provincial General Hospital (protocol number: 2019-ITT-1008-02). Informed consent was waived given that only de-identified information on physicians was collected and the study posed minimal risk. No compensation was needed for physicians because SPs paid for the consultation and all drugs prescribed.
Conflict of interest.
The authors have no conflicts of interest to declare.
Notes
A preprint version of the paper is available online, https://preprints.jmir.org/preprint/44190, but the paper is not under consideration for publication elsewhere. The scripts were originally developed in Chinese and we have provided the Chinese and English version of the scripts in response to the referee’s comment.
