https://orcid.org/0000-0003-4213-4405
Abstract
We evaluated the usability of mobile COVID-19 contact tracing apps, especially for individuals with barriers to communication and limited digital literacy skills. We searched the Apple App Store, Google Play, peer-reviewed literature, and lay press to find contact tracing apps in the United States. We evaluated apps with a framework focused on user characteristics and user interface. Of the final 26 apps, 77% were on both iPhone and Android. 69% exceeded 9th grade readability, and 65% were available only in English. Only 12% had inclusive illustrations (different genders, skin tones, physical abilities). 92% alerted users of an exposure, 42% linked to a testing site, and 62% linked to a public health website within 3 clicks. Most apps alert users of COVID-19 exposure but require high English reading levels and are not fully inclusive of the U.S. population, which may limit their reach as public health tools.
INTRODUCTION
In the United States (U.S.), contact tracing has been performed with localized, sometimes disjointed, approaches. Manual contact tracing, in which trained personnel conduct interviews with those who have tested positive for the virus (“cases”), requires a large workforce and cannot keep pace with the mounting number of COVID-19 infections.1–3 Therefore, some states introduced contact tracing mobile applications (apps) to augment their contact tracing efforts.4–6 It is estimated that these apps must be used by nearly 60% of the population to reduce the spread of the virus, though any use at all helps identify people with potential exposure (“contacts”).7 Adoption of contact tracing apps is relatively high in Qatar and Iceland (91% and 40%, respectively).5 In the U.S. and other countries, however, major issues with app adoption revolve around engagement, privacy concerns, cybersecurity, and accuracy.6,8–16
Many of these barriers are inextricably linked with the digital divide in the U.S., from structural to individual levels.6,8–16 At the structural level, there are clear disparities in smartphone ownership and broadband/high speed Internet access.17 At the individual level, even among those with existing devices and sufficient Internet access, digital literacy skills and app use vary among consumers. Experts define digital engagement as “how well users can use a product to achieve their goals and how satisfied they are with that process.”18 Therefore, it is clear that contact tracing apps can only achieve widespread use if they are usable by diverse populations, especially considering the disproportionate number of COVID-19 infections in low-income and minority populations.
Several studies have assessed national contact tracing apps, along with COVID symptom tracking or informational apps.5,9,19–23 These studies evaluated apps with various frameworks in the following general categories: accessibility, functionality, engagement, aesthetics, and inclusion of resources and information.5,9,19–23 They were not specific to common health communication barriers and diversity or inclusion concepts, which were our focus here. Other usability frameworks, such as Nielsen’s or Kientz’s, similarly encompass these broad, subjective categories.24,25 No study to our knowledge has evaluated COVID-19 contact tracing apps in the US with objective measures specifically focused on health communication barriers or diversity of end users.
OBJECTIVE
We present an evaluation framework with objective measures of usability with respect to health communication and diversity/inclusivity. We use this framework to evaluate COVID-19 contact tracing apps available in the US. Finally, we present suggestions for how app developers can improve the fit between apps and user needs.
MATERIALS AND METHODS
Sample
To identify COVID-19 contact tracing apps, we searched the Apple App Store and Google Play using search terms “covid-19,” “coronavirus,” “exposure notification,” and “contact tracing.” To be thorough, we also searched for publications in PubMed and medRxiv using “(covid* OR coronavirus OR contact tracing) AND (app OR apps)”. Finally, we searched mass media publications using a Google Search with the search terms “covid app,” “contact tracing app,” and “coronavirus exposure notification.” However, all the apps we evaluated came from app store searches. We searched between October 6 and November 12, 2020.
We excluded apps for the following reasons: developed with a target audience outside the US, only included mapping of population-level COVID-19 information (no individual contact tracing functionality), restricted to members of a specific institution, or required scanning physical Quick Response (QR) codes. Apps requiring an e-mail to register were also excluded, given privacy concerns. Cost was not a factor, as all remaining identified apps were free to use.
Measures
This study focused on usability within 2 domains of the FDA Applying Human Factors and Usability Engineering to Medical Devices guidance framework: 1) “user characteristics” with careful expansion within this domain to define categories related to health communication, such as readability and language; and 2) “user interface” with a focus on specific elements in this domain related to audiovisual information (including racial and gender diversity within visual elements) and functionality/logic of information presented (such as number of clicks and provided resources).26 The “user environment” within this FDA framework was not evaluated specifically, as we limited our evaluation to upfront engagement in the apps rather than completion of the contact tracing process.26
Focusing on “user characteristics” and “user interface and functionality” domains of the FDA framework for this analysis allowed us to narrow in on elements related to accessibility and inclusivity; we anticipate that other research may examine additional usability aspects, such as type of exposure notification. More specifically, we combined this FDA framework with concepts from multiple published studies outlining health communication and diversity barriers related to technology use, resulting in a 14-item evaluation framework (see Table 2 for details).5,9,19–26,31–36 Five of these items were more closely related to “user characteristics,” such as readability, while the remaining items and our clickability evaluation were more related to “user interface” and functionality.
Evaluation framework and results for the apps
| Feature | Number of apps with feature, N (%) | List of apps with feature (identified by letter, seeTable 1) | Definition | Rationale |
|---|---|---|---|---|
| USER CHARACTERISTICS | ||||
| Readability | Readability is defined by the Flesch-Kincaid Grade Level. This is a validated grade-level readability of texts, based on sentence length and number of syllables.27 It was calculated overall from home screens and simplified privacy policy sections (comprehensive privacy policies were excluded) using Microsoft Word. Relevant categories are: <6th grade; 6-9th grade; >9th grade. | The average reading level in the United States (U.S.) is 7th-8th grade, and 20% of adults cannot read above a 5th grade level.23 Though some of these apps had home screens with few sentences or words, their explanations of how the apps work and summaries of the privacy policy were well above 5th grade level. | ||
| <6th grade | 0 (0) | None | ||
| 6th-9th grade | 8 (31) | A, E, K, M, P-R, Z | ||
| 9th-12th grade | 18 (69) | B-D, F-J, L, O, S-Y | ||
| Language options | Apps are available in English as well as other languages. Relevant categories are: English only; English and Spanish; 3+ languages. | 4.9% of U.S. citizens over age 18 speak English less than “very well.”28 For these apps to have high usage rates and be efficacious, they must not exclude those who primarily speak a language other than English or cannot read English, especially given the high literacy levels already required to understand the apps. | ||
| English only | 17 (65) | C, D, F-H, J, K, M-P, T-Y | ||
| English and Spanish | 4 (15) | B, E, I, Z | ||
| 3+ languages | 5 (19) | A, L, Q-S | ||
| Consistent terminology | 13 (50) | B-D, F-H, J, O, P, S, W-Y | Each concept is linked to 1 word (as opposed to switching between terms, such as “coronavirus” and “COVID-19” or “alerts” and “notifications”). | Consistent wording aids in understanding and remembering information. |
| Registration not required | 22 (85) | A-J, L, N-P, S-Z | Email, phone number, name, address, date of birth, and other personal identifiers are not required to use the app. | People should be able to participate in contact tracing efforts without having to share their personal information. |
| iPhone and Android compatible | 20 (77) | A-S, Z | Apps are available on both iPhone and Android. | Apps will work best when they include as many users as possible. Limiting an app to a certain device and a certain operating system (not addressed here) limits the potential number of users and therefore its potential efficacy. |
| USER INTERFACE and FUNCTIONALITY | ||||
| Audio or video option | 2 (8) | K, S | Audio or video is available to explain how the app works and/or to help navigate the app. | Apps should include audio or video options to assist those with communication barriers (eg, visual impairment, limited literacy) use the app. |
| Illustrated instructions | 6 (23) | A, B, D, H, P, Z | Instructions have illustrations alongside them. | Illustrations help convey key points.29,30 |
| Inclusive illustrations | 3 (12) | A, D, Z | Illustrations or videos, if included, have various skin tones and genders represented. | Users should feel represented and valued in the app. No apps included illustrations of people with different physical abilities, so we only evaluated for skin tone and gender. |
| Feature to explain how exposure alerts work | 20 (77) | A-K, O, S-Z | There is a “How It Works” or similar section of the app that can be accessed from the home page. | Providing a “How it Works” section accessible from the home page allows users to re-visit this information after first use. This feature may help build user trust.15 |
| Directly alerts user of an exposure | 24 (92) | A-L, S-Z | The app alerts the user of a potential exposure using Exposure Alerts rather than solely collecting data. | Part of contact tracing includes alerting contacts of their exposure. If the app only collects information and sends it to the associated Health Department, without alerting contacts automatically, there is a time lag between when a contact could be alerted of an exposure and when the contact is finally alerted. |
| Links to a COVID testing site | 11 (42) | A, B, D, E, I, K, M, O-R | The app helps the user find a testing site. | In the event of an exposure, the user should be able to quickly find a testing site. |
| Links to comprehensive Privacy Policy | 25 (92) | Within app: J, R Externally linked: A-P, R-Z | The Privacy Policy is accessible from the app without requiring the user to search for it online. | The Privacy Policy should be clearly displayed within the app. Ideally, the Privacy Policy is described in simple writing within the app. The comprehensive policy could be an external link available from the app. |
| Links to the local Health Department website | 16 (62) | C, D, F, G, J, M-O, Q, R, U-Z | The app provides a link to the user’s local Health Department website. | The user can access further guidance and resources (for example, social support for quarantine). |
| Links to social support for quarantine | 0 (0) | None | The app links the user to social support services, if the user requests, for quarantine or isolation support. | Users are equipped to help reduce the spread of COVID-19 if they have access to the tools they need to quarantine after an exposure alert. |
| Feature | Number of apps with feature, N (%) | List of apps with feature (identified by letter, seeTable 1) | Definition | Rationale |
|---|---|---|---|---|
| USER CHARACTERISTICS | ||||
| Readability | Readability is defined by the Flesch-Kincaid Grade Level. This is a validated grade-level readability of texts, based on sentence length and number of syllables.27 It was calculated overall from home screens and simplified privacy policy sections (comprehensive privacy policies were excluded) using Microsoft Word. Relevant categories are: <6th grade; 6-9th grade; >9th grade. | The average reading level in the United States (U.S.) is 7th-8th grade, and 20% of adults cannot read above a 5th grade level.23 Though some of these apps had home screens with few sentences or words, their explanations of how the apps work and summaries of the privacy policy were well above 5th grade level. | ||
| <6th grade | 0 (0) | None | ||
| 6th-9th grade | 8 (31) | A, E, K, M, P-R, Z | ||
| 9th-12th grade | 18 (69) | B-D, F-J, L, O, S-Y | ||
| Language options | Apps are available in English as well as other languages. Relevant categories are: English only; English and Spanish; 3+ languages. | 4.9% of U.S. citizens over age 18 speak English less than “very well.”28 For these apps to have high usage rates and be efficacious, they must not exclude those who primarily speak a language other than English or cannot read English, especially given the high literacy levels already required to understand the apps. | ||
| English only | 17 (65) | C, D, F-H, J, K, M-P, T-Y | ||
| English and Spanish | 4 (15) | B, E, I, Z | ||
| 3+ languages | 5 (19) | A, L, Q-S | ||
| Consistent terminology | 13 (50) | B-D, F-H, J, O, P, S, W-Y | Each concept is linked to 1 word (as opposed to switching between terms, such as “coronavirus” and “COVID-19” or “alerts” and “notifications”). | Consistent wording aids in understanding and remembering information. |
| Registration not required | 22 (85) | A-J, L, N-P, S-Z | Email, phone number, name, address, date of birth, and other personal identifiers are not required to use the app. | People should be able to participate in contact tracing efforts without having to share their personal information. |
| iPhone and Android compatible | 20 (77) | A-S, Z | Apps are available on both iPhone and Android. | Apps will work best when they include as many users as possible. Limiting an app to a certain device and a certain operating system (not addressed here) limits the potential number of users and therefore its potential efficacy. |
| USER INTERFACE and FUNCTIONALITY | ||||
| Audio or video option | 2 (8) | K, S | Audio or video is available to explain how the app works and/or to help navigate the app. | Apps should include audio or video options to assist those with communication barriers (eg, visual impairment, limited literacy) use the app. |
| Illustrated instructions | 6 (23) | A, B, D, H, P, Z | Instructions have illustrations alongside them. | Illustrations help convey key points.29,30 |
| Inclusive illustrations | 3 (12) | A, D, Z | Illustrations or videos, if included, have various skin tones and genders represented. | Users should feel represented and valued in the app. No apps included illustrations of people with different physical abilities, so we only evaluated for skin tone and gender. |
| Feature to explain how exposure alerts work | 20 (77) | A-K, O, S-Z | There is a “How It Works” or similar section of the app that can be accessed from the home page. | Providing a “How it Works” section accessible from the home page allows users to re-visit this information after first use. This feature may help build user trust.15 |
| Directly alerts user of an exposure | 24 (92) | A-L, S-Z | The app alerts the user of a potential exposure using Exposure Alerts rather than solely collecting data. | Part of contact tracing includes alerting contacts of their exposure. If the app only collects information and sends it to the associated Health Department, without alerting contacts automatically, there is a time lag between when a contact could be alerted of an exposure and when the contact is finally alerted. |
| Links to a COVID testing site | 11 (42) | A, B, D, E, I, K, M, O-R | The app helps the user find a testing site. | In the event of an exposure, the user should be able to quickly find a testing site. |
| Links to comprehensive Privacy Policy | 25 (92) | Within app: J, R Externally linked: A-P, R-Z | The Privacy Policy is accessible from the app without requiring the user to search for it online. | The Privacy Policy should be clearly displayed within the app. Ideally, the Privacy Policy is described in simple writing within the app. The comprehensive policy could be an external link available from the app. |
| Links to the local Health Department website | 16 (62) | C, D, F, G, J, M-O, Q, R, U-Z | The app provides a link to the user’s local Health Department website. | The user can access further guidance and resources (for example, social support for quarantine). |
| Links to social support for quarantine | 0 (0) | None | The app links the user to social support services, if the user requests, for quarantine or isolation support. | Users are equipped to help reduce the spread of COVID-19 if they have access to the tools they need to quarantine after an exposure alert. |
Evaluation framework and results for the apps
| Feature | Number of apps with feature, N (%) | List of apps with feature (identified by letter, seeTable 1) | Definition | Rationale |
|---|---|---|---|---|
| USER CHARACTERISTICS | ||||
| Readability | Readability is defined by the Flesch-Kincaid Grade Level. This is a validated grade-level readability of texts, based on sentence length and number of syllables.27 It was calculated overall from home screens and simplified privacy policy sections (comprehensive privacy policies were excluded) using Microsoft Word. Relevant categories are: <6th grade; 6-9th grade; >9th grade. | The average reading level in the United States (U.S.) is 7th-8th grade, and 20% of adults cannot read above a 5th grade level.23 Though some of these apps had home screens with few sentences or words, their explanations of how the apps work and summaries of the privacy policy were well above 5th grade level. | ||
| <6th grade | 0 (0) | None | ||
| 6th-9th grade | 8 (31) | A, E, K, M, P-R, Z | ||
| 9th-12th grade | 18 (69) | B-D, F-J, L, O, S-Y | ||
| Language options | Apps are available in English as well as other languages. Relevant categories are: English only; English and Spanish; 3+ languages. | 4.9% of U.S. citizens over age 18 speak English less than “very well.”28 For these apps to have high usage rates and be efficacious, they must not exclude those who primarily speak a language other than English or cannot read English, especially given the high literacy levels already required to understand the apps. | ||
| English only | 17 (65) | C, D, F-H, J, K, M-P, T-Y | ||
| English and Spanish | 4 (15) | B, E, I, Z | ||
| 3+ languages | 5 (19) | A, L, Q-S | ||
| Consistent terminology | 13 (50) | B-D, F-H, J, O, P, S, W-Y | Each concept is linked to 1 word (as opposed to switching between terms, such as “coronavirus” and “COVID-19” or “alerts” and “notifications”). | Consistent wording aids in understanding and remembering information. |
| Registration not required | 22 (85) | A-J, L, N-P, S-Z | Email, phone number, name, address, date of birth, and other personal identifiers are not required to use the app. | People should be able to participate in contact tracing efforts without having to share their personal information. |
| iPhone and Android compatible | 20 (77) | A-S, Z | Apps are available on both iPhone and Android. | Apps will work best when they include as many users as possible. Limiting an app to a certain device and a certain operating system (not addressed here) limits the potential number of users and therefore its potential efficacy. |
| USER INTERFACE and FUNCTIONALITY | ||||
| Audio or video option | 2 (8) | K, S | Audio or video is available to explain how the app works and/or to help navigate the app. | Apps should include audio or video options to assist those with communication barriers (eg, visual impairment, limited literacy) use the app. |
| Illustrated instructions | 6 (23) | A, B, D, H, P, Z | Instructions have illustrations alongside them. | Illustrations help convey key points.29,30 |
| Inclusive illustrations | 3 (12) | A, D, Z | Illustrations or videos, if included, have various skin tones and genders represented. | Users should feel represented and valued in the app. No apps included illustrations of people with different physical abilities, so we only evaluated for skin tone and gender. |
| Feature to explain how exposure alerts work | 20 (77) | A-K, O, S-Z | There is a “How It Works” or similar section of the app that can be accessed from the home page. | Providing a “How it Works” section accessible from the home page allows users to re-visit this information after first use. This feature may help build user trust.15 |
| Directly alerts user of an exposure | 24 (92) | A-L, S-Z | The app alerts the user of a potential exposure using Exposure Alerts rather than solely collecting data. | Part of contact tracing includes alerting contacts of their exposure. If the app only collects information and sends it to the associated Health Department, without alerting contacts automatically, there is a time lag between when a contact could be alerted of an exposure and when the contact is finally alerted. |
| Links to a COVID testing site | 11 (42) | A, B, D, E, I, K, M, O-R | The app helps the user find a testing site. | In the event of an exposure, the user should be able to quickly find a testing site. |
| Links to comprehensive Privacy Policy | 25 (92) | Within app: J, R Externally linked: A-P, R-Z | The Privacy Policy is accessible from the app without requiring the user to search for it online. | The Privacy Policy should be clearly displayed within the app. Ideally, the Privacy Policy is described in simple writing within the app. The comprehensive policy could be an external link available from the app. |
| Links to the local Health Department website | 16 (62) | C, D, F, G, J, M-O, Q, R, U-Z | The app provides a link to the user’s local Health Department website. | The user can access further guidance and resources (for example, social support for quarantine). |
| Links to social support for quarantine | 0 (0) | None | The app links the user to social support services, if the user requests, for quarantine or isolation support. | Users are equipped to help reduce the spread of COVID-19 if they have access to the tools they need to quarantine after an exposure alert. |
| Feature | Number of apps with feature, N (%) | List of apps with feature (identified by letter, seeTable 1) | Definition | Rationale |
|---|---|---|---|---|
| USER CHARACTERISTICS | ||||
| Readability | Readability is defined by the Flesch-Kincaid Grade Level. This is a validated grade-level readability of texts, based on sentence length and number of syllables.27 It was calculated overall from home screens and simplified privacy policy sections (comprehensive privacy policies were excluded) using Microsoft Word. Relevant categories are: <6th grade; 6-9th grade; >9th grade. | The average reading level in the United States (U.S.) is 7th-8th grade, and 20% of adults cannot read above a 5th grade level.23 Though some of these apps had home screens with few sentences or words, their explanations of how the apps work and summaries of the privacy policy were well above 5th grade level. | ||
| <6th grade | 0 (0) | None | ||
| 6th-9th grade | 8 (31) | A, E, K, M, P-R, Z | ||
| 9th-12th grade | 18 (69) | B-D, F-J, L, O, S-Y | ||
| Language options | Apps are available in English as well as other languages. Relevant categories are: English only; English and Spanish; 3+ languages. | 4.9% of U.S. citizens over age 18 speak English less than “very well.”28 For these apps to have high usage rates and be efficacious, they must not exclude those who primarily speak a language other than English or cannot read English, especially given the high literacy levels already required to understand the apps. | ||
| English only | 17 (65) | C, D, F-H, J, K, M-P, T-Y | ||
| English and Spanish | 4 (15) | B, E, I, Z | ||
| 3+ languages | 5 (19) | A, L, Q-S | ||
| Consistent terminology | 13 (50) | B-D, F-H, J, O, P, S, W-Y | Each concept is linked to 1 word (as opposed to switching between terms, such as “coronavirus” and “COVID-19” or “alerts” and “notifications”). | Consistent wording aids in understanding and remembering information. |
| Registration not required | 22 (85) | A-J, L, N-P, S-Z | Email, phone number, name, address, date of birth, and other personal identifiers are not required to use the app. | People should be able to participate in contact tracing efforts without having to share their personal information. |
| iPhone and Android compatible | 20 (77) | A-S, Z | Apps are available on both iPhone and Android. | Apps will work best when they include as many users as possible. Limiting an app to a certain device and a certain operating system (not addressed here) limits the potential number of users and therefore its potential efficacy. |
| USER INTERFACE and FUNCTIONALITY | ||||
| Audio or video option | 2 (8) | K, S | Audio or video is available to explain how the app works and/or to help navigate the app. | Apps should include audio or video options to assist those with communication barriers (eg, visual impairment, limited literacy) use the app. |
| Illustrated instructions | 6 (23) | A, B, D, H, P, Z | Instructions have illustrations alongside them. | Illustrations help convey key points.29,30 |
| Inclusive illustrations | 3 (12) | A, D, Z | Illustrations or videos, if included, have various skin tones and genders represented. | Users should feel represented and valued in the app. No apps included illustrations of people with different physical abilities, so we only evaluated for skin tone and gender. |
| Feature to explain how exposure alerts work | 20 (77) | A-K, O, S-Z | There is a “How It Works” or similar section of the app that can be accessed from the home page. | Providing a “How it Works” section accessible from the home page allows users to re-visit this information after first use. This feature may help build user trust.15 |
| Directly alerts user of an exposure | 24 (92) | A-L, S-Z | The app alerts the user of a potential exposure using Exposure Alerts rather than solely collecting data. | Part of contact tracing includes alerting contacts of their exposure. If the app only collects information and sends it to the associated Health Department, without alerting contacts automatically, there is a time lag between when a contact could be alerted of an exposure and when the contact is finally alerted. |
| Links to a COVID testing site | 11 (42) | A, B, D, E, I, K, M, O-R | The app helps the user find a testing site. | In the event of an exposure, the user should be able to quickly find a testing site. |
| Links to comprehensive Privacy Policy | 25 (92) | Within app: J, R Externally linked: A-P, R-Z | The Privacy Policy is accessible from the app without requiring the user to search for it online. | The Privacy Policy should be clearly displayed within the app. Ideally, the Privacy Policy is described in simple writing within the app. The comprehensive policy could be an external link available from the app. |
| Links to the local Health Department website | 16 (62) | C, D, F, G, J, M-O, Q, R, U-Z | The app provides a link to the user’s local Health Department website. | The user can access further guidance and resources (for example, social support for quarantine). |
| Links to social support for quarantine | 0 (0) | None | The app links the user to social support services, if the user requests, for quarantine or isolation support. | Users are equipped to help reduce the spread of COVID-19 if they have access to the tools they need to quarantine after an exposure alert. |
Analysis
One author (SB) primarily evaluated apps using an iPhone 7. A second coder (SL) used an iPhone 8 to independently assess 5 (20%) of the apps to ensure reproducibility of coding. We downloaded 50% of the apps evaluated on the iPhone onto an Android (Samsung Galaxy J3) to verify the same functionalities and appearances on Android as on iPhone (with no differences to note, data not shown).
We scored apps in each category and recorded the data in Microsoft Excel. For the majority of coding, we categorized apps either dichotomously as yes/no (eg, app has audio or visual components) or into discrete groups (eg, readability level at <6th grade, 6th–9th grade, or >9th grade). For 6 prespecified functions, we recorded the number of clicks to reach the target feature or content. Finally, we selected 3 apps that provided concrete examples of several usability features that could be considered in future app designs: 1) written at lower reading levels; 2) used racially and gender diverse visual features; 3) included audiovisual components.
RESULTS
The initial search yielded 54 apps related to contact tracing in the U.S. After applying our exclusion criteria, we evaluated 26 apps (Figure 1). The coding was highly consistent between coders, with a range of Cohen’s Kappa between the categories of 0.95 to 1.0. State and county government-affiliated entities created 19/26 (73%) of the apps, but, notably, Apple and Google jointly created the Exposure Notification Express system, which they embedded in the operating systems without a corresponding app on iPhone (Table 1, Figure 2). Most apps (20/26, 77%) were available on both iPhone and Android (Table 1, Table 2). Public health departments in some regions (CO, DC, CA, MD, CT, and WA) only provided Android apps, but all of those states were also available in the Exposure Notifications Express system in iPhone Settings (Table 1).
Process of selecting apps for evaluation.
Affiliation of app developer.
The 26 contact tracing apps in the United States that were evaluated
| Letter | Application (device on which it was evaluated) | Developer | Update Date | iPhone rating (reviews) | Google Play rating (reviews) | Google Play # of downloads |
|---|---|---|---|---|---|---|
| A | COVID Alert NY (iPhone) | New York State Department of Health | 10/7/20 | 4.8 (408) | 4 (751) | 100 000+ |
| B | COVID Alert NJ (iPhone) | New Jersey Office of Information Technology | 10/14/20 | 3.8 (214) | 3.9 (296) | 100 000+ |
| C | COVIDWISE (VA) (iPhone) | Virginia Department of Health | 8/27/20 | 4.7 (823) | 3.7 (834) | 100 000+ |
| D | SlowCOVIDNC (iPhone) | NC Department of Health and Human Services | 10/7/20 | 4.8 (176) | 3.9 (314) | 100 000+ |
| E | COVID Alert Pennsylvania (iPhone) | Commonwealth of Pennsylvania | 10/1/20 | 4.4 (213) | 4.0 (539) | 100 000+ |
| F | Care19 Alert (ND and WY) (iPhone) | ProudCrowd, LLC | 10/24/20 | 4.3 (20) | 3.3 (46) | 10 000+ |
| G | COVID Trace Nevada (iPhone) | Nevada Division of Public and Behavioral Health | 10/2/20 | 3.2 (106) | 3.6 (96) | 10 000+ |
| H | Covid Watch Arizona (iPhone) | Arizona Department of Health Services | 10/14/20 | 4.3 (54) | 4.0 (28) | 5000+ |
| I | COVID Alert DE (iPhone) | Delaware Division of Public Health | 10/1/20 | 4.4 (24) | 4.2 (59) | 10 000+ |
| J | GuideSafe (Alabama) (iPhone) | Alabama Department of Public Health | 9/27/20 | 4.6 (80) | 3.4 (126) | 50 000+ |
| K | Citizen SafePass (San Joaquin County) (iPhone) | sp0n, Inc. | 9/27/20 | 2.5 (106) | (0) | 50+ |
| L | PathCheck SafePlaces (Haiti, Guam, Puerto Rico, Teton County WY) (iPhone) | Path Check, Inc. | 8/27/20 | 4.5 (55) | 4.1 (70) | 10 000+ |
| M | COVID Control (iPhone) | Johns Hopkins Mobile Medicine | 8/27/20 | 4.6 (30) | 3.2 (20) | 1000+ |
| N | CheckCOVID (iPhone) | University of Nebraska Medical Center | 10/7/20 | 4.5 (74) | (1) | 1000+ |
| O | MI COVID Alert (iPhone) | State of Michigan Department of Technology | 10/21/20 | 4.9 (64) | 4.7 (35) | 5000+ |
| P | SC Safer Together (iPhone) | Medical University of South Carolina | 11/13/20 | 4.7 (7) | (0) | 5+ |
| Q | CombatCOVID MDC (iPhone) | County Government of Miami-Dade County, Florida | 10/26/20 | 3.4 (48) | 4.5 (23) | 1000+ |
| R | CombatCOVID PBC (iPhone) | County Government of Palm Beach County, Florida | 9/27/20 | 3.5 (17) | 3.7 (37) | 1000+ |
| S | Exposure Notifications Expressa (PA, NY, MI, VA, NV, AL, ND, WY, DC, NC, CO, MD, CT) (iPhone) | Apple/Google | 9/1/20 | – | - | – |
| T | CO Exposure Notifications (Android) | Colorado Department of Public Health & Environment | 10/23/20 | – | 4.6 (163) | 50 000+ |
| U | DC CAN (Android) | Government of the District of Columbia | 10/23/20 | – | 4.2 (35) | 5000+ |
| V | CA Notify (Early Access) (Android) | California Department of Technology | 11/10/20 | – | (0) | 5000+ |
| W | MD COVID Alert (Android) | Maryland Department of Health | 11/10/20 | – | 4.0 (91) | 50 000+ |
| X | COVID Alert CT (Android) | Connecticut Department of Public Health | 11/9/20 | – | 4.0 (66) | 10 000+ |
| Y | WA Notify (Early Access) (Android) | Washington Department of Health | 10/29/20 | – | (0) | 100+ |
| Z | AlohaSafe Alert (iPhone) | Hawaii State Department of Health | 11/12/20 | 5 (5) | (0) | 100+ |
| Letter | Application (device on which it was evaluated) | Developer | Update Date | iPhone rating (reviews) | Google Play rating (reviews) | Google Play # of downloads |
|---|---|---|---|---|---|---|
| A | COVID Alert NY (iPhone) | New York State Department of Health | 10/7/20 | 4.8 (408) | 4 (751) | 100 000+ |
| B | COVID Alert NJ (iPhone) | New Jersey Office of Information Technology | 10/14/20 | 3.8 (214) | 3.9 (296) | 100 000+ |
| C | COVIDWISE (VA) (iPhone) | Virginia Department of Health | 8/27/20 | 4.7 (823) | 3.7 (834) | 100 000+ |
| D | SlowCOVIDNC (iPhone) | NC Department of Health and Human Services | 10/7/20 | 4.8 (176) | 3.9 (314) | 100 000+ |
| E | COVID Alert Pennsylvania (iPhone) | Commonwealth of Pennsylvania | 10/1/20 | 4.4 (213) | 4.0 (539) | 100 000+ |
| F | Care19 Alert (ND and WY) (iPhone) | ProudCrowd, LLC | 10/24/20 | 4.3 (20) | 3.3 (46) | 10 000+ |
| G | COVID Trace Nevada (iPhone) | Nevada Division of Public and Behavioral Health | 10/2/20 | 3.2 (106) | 3.6 (96) | 10 000+ |
| H | Covid Watch Arizona (iPhone) | Arizona Department of Health Services | 10/14/20 | 4.3 (54) | 4.0 (28) | 5000+ |
| I | COVID Alert DE (iPhone) | Delaware Division of Public Health | 10/1/20 | 4.4 (24) | 4.2 (59) | 10 000+ |
| J | GuideSafe (Alabama) (iPhone) | Alabama Department of Public Health | 9/27/20 | 4.6 (80) | 3.4 (126) | 50 000+ |
| K | Citizen SafePass (San Joaquin County) (iPhone) | sp0n, Inc. | 9/27/20 | 2.5 (106) | (0) | 50+ |
| L | PathCheck SafePlaces (Haiti, Guam, Puerto Rico, Teton County WY) (iPhone) | Path Check, Inc. | 8/27/20 | 4.5 (55) | 4.1 (70) | 10 000+ |
| M | COVID Control (iPhone) | Johns Hopkins Mobile Medicine | 8/27/20 | 4.6 (30) | 3.2 (20) | 1000+ |
| N | CheckCOVID (iPhone) | University of Nebraska Medical Center | 10/7/20 | 4.5 (74) | (1) | 1000+ |
| O | MI COVID Alert (iPhone) | State of Michigan Department of Technology | 10/21/20 | 4.9 (64) | 4.7 (35) | 5000+ |
| P | SC Safer Together (iPhone) | Medical University of South Carolina | 11/13/20 | 4.7 (7) | (0) | 5+ |
| Q | CombatCOVID MDC (iPhone) | County Government of Miami-Dade County, Florida | 10/26/20 | 3.4 (48) | 4.5 (23) | 1000+ |
| R | CombatCOVID PBC (iPhone) | County Government of Palm Beach County, Florida | 9/27/20 | 3.5 (17) | 3.7 (37) | 1000+ |
| S | Exposure Notifications Expressa (PA, NY, MI, VA, NV, AL, ND, WY, DC, NC, CO, MD, CT) (iPhone) | Apple/Google | 9/1/20 | – | - | – |
| T | CO Exposure Notifications (Android) | Colorado Department of Public Health & Environment | 10/23/20 | – | 4.6 (163) | 50 000+ |
| U | DC CAN (Android) | Government of the District of Columbia | 10/23/20 | – | 4.2 (35) | 5000+ |
| V | CA Notify (Early Access) (Android) | California Department of Technology | 11/10/20 | – | (0) | 5000+ |
| W | MD COVID Alert (Android) | Maryland Department of Health | 11/10/20 | – | 4.0 (91) | 50 000+ |
| X | COVID Alert CT (Android) | Connecticut Department of Public Health | 11/9/20 | – | 4.0 (66) | 10 000+ |
| Y | WA Notify (Early Access) (Android) | Washington Department of Health | 10/29/20 | – | (0) | 100+ |
| Z | AlohaSafe Alert (iPhone) | Hawaii State Department of Health | 11/12/20 | 5 (5) | (0) | 100+ |
Exposure Notifications Express refers to the iPhone-integrated system that can be accessed from iPhone Settings. The listed states enabled this method for contact tracing. The corresponding system on Android has generated apps, which were evaluated separately (see T–Y). Therefore, some apps on Android do not have corresponding apps on iPhone.
The 26 contact tracing apps in the United States that were evaluated
| Letter | Application (device on which it was evaluated) | Developer | Update Date | iPhone rating (reviews) | Google Play rating (reviews) | Google Play # of downloads |
|---|---|---|---|---|---|---|
| A | COVID Alert NY (iPhone) | New York State Department of Health | 10/7/20 | 4.8 (408) | 4 (751) | 100 000+ |
| B | COVID Alert NJ (iPhone) | New Jersey Office of Information Technology | 10/14/20 | 3.8 (214) | 3.9 (296) | 100 000+ |
| C | COVIDWISE (VA) (iPhone) | Virginia Department of Health | 8/27/20 | 4.7 (823) | 3.7 (834) | 100 000+ |
| D | SlowCOVIDNC (iPhone) | NC Department of Health and Human Services | 10/7/20 | 4.8 (176) | 3.9 (314) | 100 000+ |
| E | COVID Alert Pennsylvania (iPhone) | Commonwealth of Pennsylvania | 10/1/20 | 4.4 (213) | 4.0 (539) | 100 000+ |
| F | Care19 Alert (ND and WY) (iPhone) | ProudCrowd, LLC | 10/24/20 | 4.3 (20) | 3.3 (46) | 10 000+ |
| G | COVID Trace Nevada (iPhone) | Nevada Division of Public and Behavioral Health | 10/2/20 | 3.2 (106) | 3.6 (96) | 10 000+ |
| H | Covid Watch Arizona (iPhone) | Arizona Department of Health Services | 10/14/20 | 4.3 (54) | 4.0 (28) | 5000+ |
| I | COVID Alert DE (iPhone) | Delaware Division of Public Health | 10/1/20 | 4.4 (24) | 4.2 (59) | 10 000+ |
| J | GuideSafe (Alabama) (iPhone) | Alabama Department of Public Health | 9/27/20 | 4.6 (80) | 3.4 (126) | 50 000+ |
| K | Citizen SafePass (San Joaquin County) (iPhone) | sp0n, Inc. | 9/27/20 | 2.5 (106) | (0) | 50+ |
| L | PathCheck SafePlaces (Haiti, Guam, Puerto Rico, Teton County WY) (iPhone) | Path Check, Inc. | 8/27/20 | 4.5 (55) | 4.1 (70) | 10 000+ |
| M | COVID Control (iPhone) | Johns Hopkins Mobile Medicine | 8/27/20 | 4.6 (30) | 3.2 (20) | 1000+ |
| N | CheckCOVID (iPhone) | University of Nebraska Medical Center | 10/7/20 | 4.5 (74) | (1) | 1000+ |
| O | MI COVID Alert (iPhone) | State of Michigan Department of Technology | 10/21/20 | 4.9 (64) | 4.7 (35) | 5000+ |
| P | SC Safer Together (iPhone) | Medical University of South Carolina | 11/13/20 | 4.7 (7) | (0) | 5+ |
| Q | CombatCOVID MDC (iPhone) | County Government of Miami-Dade County, Florida | 10/26/20 | 3.4 (48) | 4.5 (23) | 1000+ |
| R | CombatCOVID PBC (iPhone) | County Government of Palm Beach County, Florida | 9/27/20 | 3.5 (17) | 3.7 (37) | 1000+ |
| S | Exposure Notifications Expressa (PA, NY, MI, VA, NV, AL, ND, WY, DC, NC, CO, MD, CT) (iPhone) | Apple/Google | 9/1/20 | – | - | – |
| T | CO Exposure Notifications (Android) | Colorado Department of Public Health & Environment | 10/23/20 | – | 4.6 (163) | 50 000+ |
| U | DC CAN (Android) | Government of the District of Columbia | 10/23/20 | – | 4.2 (35) | 5000+ |
| V | CA Notify (Early Access) (Android) | California Department of Technology | 11/10/20 | – | (0) | 5000+ |
| W | MD COVID Alert (Android) | Maryland Department of Health | 11/10/20 | – | 4.0 (91) | 50 000+ |
| X | COVID Alert CT (Android) | Connecticut Department of Public Health | 11/9/20 | – | 4.0 (66) | 10 000+ |
| Y | WA Notify (Early Access) (Android) | Washington Department of Health | 10/29/20 | – | (0) | 100+ |
| Z | AlohaSafe Alert (iPhone) | Hawaii State Department of Health | 11/12/20 | 5 (5) | (0) | 100+ |
| Letter | Application (device on which it was evaluated) | Developer | Update Date | iPhone rating (reviews) | Google Play rating (reviews) | Google Play # of downloads |
|---|---|---|---|---|---|---|
| A | COVID Alert NY (iPhone) | New York State Department of Health | 10/7/20 | 4.8 (408) | 4 (751) | 100 000+ |
| B | COVID Alert NJ (iPhone) | New Jersey Office of Information Technology | 10/14/20 | 3.8 (214) | 3.9 (296) | 100 000+ |
| C | COVIDWISE (VA) (iPhone) | Virginia Department of Health | 8/27/20 | 4.7 (823) | 3.7 (834) | 100 000+ |
| D | SlowCOVIDNC (iPhone) | NC Department of Health and Human Services | 10/7/20 | 4.8 (176) | 3.9 (314) | 100 000+ |
| E | COVID Alert Pennsylvania (iPhone) | Commonwealth of Pennsylvania | 10/1/20 | 4.4 (213) | 4.0 (539) | 100 000+ |
| F | Care19 Alert (ND and WY) (iPhone) | ProudCrowd, LLC | 10/24/20 | 4.3 (20) | 3.3 (46) | 10 000+ |
| G | COVID Trace Nevada (iPhone) | Nevada Division of Public and Behavioral Health | 10/2/20 | 3.2 (106) | 3.6 (96) | 10 000+ |
| H | Covid Watch Arizona (iPhone) | Arizona Department of Health Services | 10/14/20 | 4.3 (54) | 4.0 (28) | 5000+ |
| I | COVID Alert DE (iPhone) | Delaware Division of Public Health | 10/1/20 | 4.4 (24) | 4.2 (59) | 10 000+ |
| J | GuideSafe (Alabama) (iPhone) | Alabama Department of Public Health | 9/27/20 | 4.6 (80) | 3.4 (126) | 50 000+ |
| K | Citizen SafePass (San Joaquin County) (iPhone) | sp0n, Inc. | 9/27/20 | 2.5 (106) | (0) | 50+ |
| L | PathCheck SafePlaces (Haiti, Guam, Puerto Rico, Teton County WY) (iPhone) | Path Check, Inc. | 8/27/20 | 4.5 (55) | 4.1 (70) | 10 000+ |
| M | COVID Control (iPhone) | Johns Hopkins Mobile Medicine | 8/27/20 | 4.6 (30) | 3.2 (20) | 1000+ |
| N | CheckCOVID (iPhone) | University of Nebraska Medical Center | 10/7/20 | 4.5 (74) | (1) | 1000+ |
| O | MI COVID Alert (iPhone) | State of Michigan Department of Technology | 10/21/20 | 4.9 (64) | 4.7 (35) | 5000+ |
| P | SC Safer Together (iPhone) | Medical University of South Carolina | 11/13/20 | 4.7 (7) | (0) | 5+ |
| Q | CombatCOVID MDC (iPhone) | County Government of Miami-Dade County, Florida | 10/26/20 | 3.4 (48) | 4.5 (23) | 1000+ |
| R | CombatCOVID PBC (iPhone) | County Government of Palm Beach County, Florida | 9/27/20 | 3.5 (17) | 3.7 (37) | 1000+ |
| S | Exposure Notifications Expressa (PA, NY, MI, VA, NV, AL, ND, WY, DC, NC, CO, MD, CT) (iPhone) | Apple/Google | 9/1/20 | – | - | – |
| T | CO Exposure Notifications (Android) | Colorado Department of Public Health & Environment | 10/23/20 | – | 4.6 (163) | 50 000+ |
| U | DC CAN (Android) | Government of the District of Columbia | 10/23/20 | – | 4.2 (35) | 5000+ |
| V | CA Notify (Early Access) (Android) | California Department of Technology | 11/10/20 | – | (0) | 5000+ |
| W | MD COVID Alert (Android) | Maryland Department of Health | 11/10/20 | – | 4.0 (91) | 50 000+ |
| X | COVID Alert CT (Android) | Connecticut Department of Public Health | 11/9/20 | – | 4.0 (66) | 10 000+ |
| Y | WA Notify (Early Access) (Android) | Washington Department of Health | 10/29/20 | – | (0) | 100+ |
| Z | AlohaSafe Alert (iPhone) | Hawaii State Department of Health | 11/12/20 | 5 (5) | (0) | 100+ |
Exposure Notifications Express refers to the iPhone-integrated system that can be accessed from iPhone Settings. The listed states enabled this method for contact tracing. The corresponding system on Android has generated apps, which were evaluated separately (see T–Y). Therefore, some apps on Android do not have corresponding apps on iPhone.
Overall, 18/26 (69%) were above 9th grade readability and 17/26 (65%) were available only in English (Table 2). Nineteen percent (5/26) were available in more than 3 languages, including Spanish (Table 2). Eighty-five percent (22/26) did not require users to input a phone number to sign up (Table 2). Regarding user interface and functionality, the vast majority of apps directly alerted users of an exposure and explained how their alert system worked, but only 8/26 (31%) included videos or illustrations to do so (Table 2). Very few (3/26, 12%) included illustrations with diverse representations (different genders, skin tones, or physical abilities) (Table 2). Though less than half provided links to find physical testing locations, 16/26 (62%) linked to a local or state health department website within 3 clicks (Table 2, Figure 3). Eighty-five percent (22/26) verified a user-reported positive test with local health authorities (data not shown). None of the apps provided direct links to social support services or resources for quarantining (Table 2).
Clickability of apps for various contact tracing-related functions.
Note: Any potential information that would have been made available after receiving an exposure alert or submitting a positive test notification was not evaluated in our study.
Case studies
SlowCOVIDNC
SlowCOVIDNC was launched by the North Carolina Department of Health and Human Services in September 2020. It is free and does not prompt for user registration. The app runs in the background of a user’s phone and, as explained within the app, does not collect any personally identifiable information because it uses Bluetooth rather than location sensing. Using a specific, illustrated example of two people meeting in a grocery store, the app explains its utilization of Bluetooth token exchanges to remember interactions between users exceeding 10–20 minutes. If a user tests positive and reports it in the app, the Department confirms the case before the app alerts users of an exposure. All app illustrations represent a spectrum of skin tones and of genders. However, readability level of the “How it Works” explanation is 9.1, and the app is only available in English.
AlohaSafe Alert
AlohaSafe Alert was launched by the Hawaii State Department of Health in November 2020. It similarly does not require registration, uses Bluetooth sensing, and confirms cases before alerting contacts of exposures. When first opening the app, users choose between Spanish and English. The welcome illustration represents different skin tones and genders. A concise explanation of how the app works has a readability level of 6.0. Part of this explanation is: “In the event of an encounter, your data and information remain anonymous. The app doesn’t store any personal data. Only random IDs are exchanged. These are deleted after 14 days.” Within the app, however, there is some inconsistent wording, as “exposure,” “detection,” “alert,” and “report” are used interchangeably. The app also links to the Hawaii Department of Health website, but it does not clearly present a link to access testing or provide user feedback.
Exposure Notifications Express
Exposure Notifications Express was launched by Apple and Google in September 2020. It is not an app on the iPhone, rather a system that can be enabled from Settings. (On Android, Google creates apps corresponding to the states that opt-in to their system, and those apps were evaluated separately). Since this system is integrated with the iPhone, the user can use any language available on the iPhone and enable VoiceOver for low vision. Unfortunately, there are no illustrations, and the welcome message readability, depending on the state, is around 11. This system does not provide links to more information or testing sites.
DISCUSSION
Overall, most contact tracing apps included basic functionalities, such as alerting users of exposures with few clicks. The most room for improvement was in inclusivity for potential users. First, while the average reading level in the U.S. is 7th–8th grade and 20% of adults cannot read above a 5th grade level,23 the readability of contact tracing apps (even excluding the comprehensive privacy policies) was higher and potentially less accessible for the general population. High readability levels have also been reported for privacy policies of other contact tracing apps.23 Second, 30.6% of U.S. citizens over age 18 who speak a language other than English speak English less than “very well,”28 yet only 35% of apps were accessible in languages other than English. Third, audiovisual features have the potential to enhance understanding and improve recall29,30; these were infrequent within the contact tracing apps. Finally, none of the apps in this study provided direct links to social support services that individuals may need in the event of an exposure alert.
Prior studies evaluated contact tracing apps; however, this was mostly done outside the U.S. using subjective rating scales, such as the Mobile App Rating Scale or the System Usability Score.5,9,19–23,32 Our more objective rating system with a specific emphasis on health communication and inclusivity may therefore offer new insights, such as providing novel data on audiovisual features, readability, and inclusivity of graphics presented (eg, people of different races, genders, and abilities).
Our study has limitations. First, we acknowledge that these apps were designed within tight time frames, and developers might not have been able to implement all features to date. We also did not use broader usability metrics identified in other research,24–26,31,32 since our intention was to focus on health communication and diverse end users more explicitly. Future work is needed to replicate and/or expand our list of inclusivity and functionality criteria, as this paper is not intended to validate but rather to enumerate and document the various domains to consider in this space. In addition, we did not fully interact with the exposure alert systems because we did not come in contact with cases during the study (ie, we were unable to evaluate further links or information that may become available to users in the event of an exposure alert). We also only evaluated the apps in English. Finally, it is possible that the apps have changed since our evaluation due to the iterative nature of development. Despite these limitations, our suggestions are still valid for future app development.
CONCLUSION
For contact tracing apps to be maximally effective, they must be usable and accessible to the population they aim to serve, including those with low digital literacy and different backgrounds. Our findings present concrete features and categories for developers to consider in current and future apps (for contact tracing and beyond). Further, our work builds upon existing standards for accessibility of digital health to reach diverse end users, such as those developed by the Agency for Healthcare Research and Quality,18 The Commonwealth Fund,19 the FDA,26 Xcertia,35 and the National Academy of Medicine.36 Moving forward, developers should routinely reference these standards to increase usability of apps and implementation guidelines into real-world practice.
FUNDING
Dr. Sarkar is supported through a grant from the National Cancer Institute (K24CA212294). Dr. Lyles is supported through a UCSF mid-career faculty award, “Bolstering the pipeline to achieve health equity.” Dr. Sarkar and Dr. Lyles are jointly funded by the California Health Care Foundation (30352). The content is solely the responsibility of the authors and do not necessarily represent the official views of the study funders.
AUTHOR CONTRIBUTIONS
SB helped design the study and develop the usability framework and then searched for apps, led coding, and wrote the manuscript. SL double-coded apps and helped develop the framework, edited the manuscript, and coordinated group members. MN helped design the search strategy and edit the manuscript. US and CL conceived of the study, helped develop the framework, reviewed coding, and edited the manuscript.
DATA AVAILABILITY STATEMENT
Data available from the authors upon request.
CONFLICT OF INTEREST STATEMENT
None declared.
REFERENCES
Fitzhugh Mullan Institute for Health Workforce Equity, The George Washington University. Contact Tracing Workforce Estimator. Updated December
The Local (Swiss News). More than 1.6 million Swiss have downloaded Covid tracing app. Updated July 9,
Pew Research Center, Internet & Technology. Internet/Broadband Fact Sheet. Updated
Nielsen Norman Group. 10 Usability Heuristics for User Interface Design. Updated November
Center for Devices and Radiological Health. Applying Human Factors and Usability Engineering to Medical Devices: Guidance for Industry and Food and Drug Administration Staff. Updated February 2016. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/applying-human-factors-and-usability-engineering-medical-devices. Accessed December 1–14, 2020 and May 1–14, 2021.
United States Census Bureau. Language Spoken at Home. TableID: S1601. Updated
Usability.gov: Improving the User Experience. Updated December 2020. https://www.usability.gov/what-and-why/usability-evaluation.html Accessed October 1–14, 2020 and May 1–5, 2021.
Xcertia. mHealth App Guidelines: 2019 Board Approved Xcertia Guidelines. Issued August 12, 2019. https://www.himss.org/sites/hde/files/media/file/2020/04/17/xcertia-guidelines-2019-final.pdf Accessed December 21, 2020.
Author notes
Dr. Urmimala Sarkar and Dr. Courtney Lyles served as joint last authors.
