Veterans’ response to an automated text messaging protocol during the COVID-19 pandemic

Abstract The US Department of Veterans Affairs (VA) is using an automated short message service application named “Annie” as part of its coronavirus disease 2019 (COVID-19) response with a protocol for coronavirus precautions, which can help the veteran monitor symptoms and can advise the veteran when to contact his or her VA care team or a nurse triage line. We surveyed 1134 veterans on their use of the Annie application and coronavirus precautions protocol. Survey results support what is likely a substantial resource savings for the VA, as well as non-VA community healthcare. Moreover, the majority of veterans reported at least 1 positive sentiment (felt more connected to VA, confident, or educated and/or felt less anxious) by receiving the protocol messages. The findings from this study have implications for other healthcare systems to help manage a patient population during the coronavirus pandemic.


INTRODUCTION
Robust evidence supports the value of integrating SMS/textmessaging interventions into healthcare delivery and population health practice. 1 The US Department of Veterans Affairs (VA) Office of Connected Care developed and implemented an automated short-message service (SMS) application for guided patient selfmanagement, named Annie. 2 Developed by the VA, Annie is modeled after a system used by the National Health Service in England called Flo. The platform is hosted on Amazon Web Services Veterans Affairs Enterprise Cloud -Mobile Applications Platform. Annie makes use of the same shared services and databases VA health apps utilize, but it is not directly connected to the Computerized Patient Record System or My HealtheVet, VA's patient health record and patient portal, respectively. Within Annie, clinicians can assign protocols to patients and view individual veteran responses. Veterans use SMS text to exchange messages with Annie or they can log in to exchange messages and see graphs of their readings.
Annie can deliver educational and motivational messages to veterans' mobile phones, send reminders for health tasks, and query about their current health status. This can encourage veterans to adopt healthy habits and monitor progress toward wellness goals.
The flexible design and branching logic allow clinical teams to develop protocols tailored to a variety of health conditions. Annie enables clinical teams to assign protocols to patients based on clinical and social conditions as well as user preferences; clinical teams may or may not review responses. In addition, for certain protocols, veterans can self-enroll. Currently, there are over 170 clinical protocols and around 19 000 Annie users. The VA used Annie as part of its coronavirus disease 2019 (COVID-19) response, adding several protocols, including a coronavirus precautions protocol (CPP).
After the first known cases of COVID-19 were diagnosed in the US in February 2020, many healthcare organizations, including VA facilities, were overwhelmed with calls seeking guidance. Within the VA, call center volume increased by 21% from February (3 488 478 total calls) to March (4 217 229 total calls). 3 Similarly, the VA saw a sharp increase in secure messages. In February 2020, the number of secure messages sent and received by veterans ranged from a low of 354 832 (week of Feb 16) to 413 084 (week of March 1). In the following weeks, secure messaging continued to increase to a high of 622 858 the week of March 15. 4 On March 4, 2020 a request was made through the Veterans Health Administration Emergency Management Coordination Cell to develop an Annie protocol to assist veterans during COVID-19 response efforts. A team of clinical home telehealth (remote monitoring) and Annie experts convened on March 5 for a one-time, virtual advisory meeting to shape 2 protocols for COVID-19 support: CPP and Isolation/Quarantine. Protocol development and testing occurred March 6-9 and were presented to Emergency Management Coordination Cell clinical subject matter experts on March 10. Expert concurrence was obtained March 11 and Veterans Health Administration senior leadership approved the protocol for use nationally on March 13. It was made available the same day.
The CPP ( Figure 1) is a 2-month protocol that delivers educational content on coronavirus precautions, guides veterans in monitoring their temperature and other clinical symptoms, and advises them to call their care team or a nurse triage line when they cross certain clinical thresholds. For the first 2 weeks the protocol was offered, check-ins were sent daily. Currently, messages include wellness questions sent Monday, Wednesday, Friday, Sunday ("Annie here. Are you feeling well today?") and educational tips (eg, "Annie again. If you are ill, use a separate bedroom and bathroom if possible.") sent Monday, Wednesday, and Friday. Annie always identifies herself, eg, "Annie here." Veterans can self-enroll in this protocol or be added by a member of their care team. The Office of Connected Care reached out to veterans across the country encouraging them to enroll in Annie and subscribe to the CPP, and around 4200 self-enrolled. As the number grew, we decided to invite CPP subscribers to provide feedback on Annie's utility and value to them during this period.

MATERIALS AND METHODS
A subset of the authors designed a set of 9 questions to assess the respondents' geographic distribution, self-reported clinical status, reactions to the Annie CPP messages, and actions taken or not taken because of the messages they received. The questions included multiple formats in which the respondents could choose all responses that applied, as well as open-ended questions allowing free-text responses. The questions were loaded into SurveyMonkey and made available via internet link. On April 22, 2020, a broadcast message was sent to 11 630 users of the Annie application who had that feature enabled, which is 81% of all Annie users. At that time, there were approximately 4400 active users of Annie's CPP (96% self- enrolled; the other 4% enrolled by a VA staff member). Data was exported from SurveyMonkey, April 23, 2020, 24 hours after the link was sent and transferred to an Excel spreadsheet for analysis. Data were summarized with descriptive statistics.
Questions that included open-ended responses were analyzed using an inductive coding approach 5,6 by a single team member (ie, allowing codes to emerge from the data rather than using a predetermined coding scheme) with an independent audit of all coding by a coauthor. This type of auditing procedure by a second person is considered an acceptable alternative to using independent coders for ensuring validity of the analysis. 7 Questions raised by the audit were resolved by consensus calls as needed. For 2 questions, a secondary analysis of the content for each primary code was warranted based on the complexity of the responses to further reveal subpatterns. For example, for the primary code "Connected" for the question "What about Annie's messages was helpful for you?" the first author then summarized the different types of subpatterns for "Connected" as subpatterns under this primary code.

RESULTS
Of the 11 630 veterans who received an invitation to participate, 1567 (13.5%) went to the SurveyMonkey TM website. Of those, 188 (12.0%) reported that they were not subscribed to the CPP Annie protocol, so were excluded from analysis. Of the remaining 1379 who were eligible, 245 (17.8%) did not complete the survey, but we retained partial responses; 1134 veterans responded to all questions. This response was from a total of 4400 veterans who were actually subscribed to the CPP protocol, so the calculated response rate is 25.8% (1134/4400). Table 1 shows the questions that the respondents answered, along with a summary of the results for each question. Responses were received from veterans in all 50 states plus the District of Columbia and Puerto Rico. Five of the 6 most highly represented states in terms of ratio of respondents to veterans living in that state were within a single VA network (VA Midwest Health Care Network: Minnesota, South Dakota, North Dakota, Iowa, and Nebraska) that was an early adopter and promoter of Annie. Following Table 1 is further analysis of veterans' free-text responses to open-ended questions and when they selected the "other" option for various questions.
Why did you choose to subscribe to Annie's coronavirus precautions protocol messages?
Frequent free-text responses in the "Other" field for the question "Why did you choose to subscribe to Annie's Coronavirus Precautions protocol messages?" included reasons such as curiosity, health management, and simply to stay informed.
What about Annie's messages was helpful for you?
This was an open-ended question. Table 2 shows a summary of the qualitative analysis conducted on the 808 free-text responses.
A secondary analysis of the categories revealed subpatterns for the "Connected" category. These subpatterns included: • Veterans liked a feeling of contact with another person or entity (Annie). (49) • Many veterans specifically mentioned liking the daily contact.
(35) • Veterans simply liked feeling connected in general. (15) • A dozen veterans specifically mentioned feeling connected with the VA. (12) • Seven veterans liked that their status was being checked on. (7) • Six veterans specifically mentioned that the messages made them feel "not alone." (6) Why were Annie's messages not helpful to you?
This open-ended question received 61 responses. Most of those responses were coded as "Not enough information" (34); that is, they felt that the Annie messages did not provide enough information to be valuable to the user or that they shared repeated, common knowledge. Others reasons included "Too many messages" (3), the user did not receive a reply from Annie (3), the user did not need the help (3), and the user was limited in how they send messages to Annie (4). For example, user was unsure how to respond to the question "Are you feeling well today?" because ". . .you can have different ways of not feeling well." Other responses were one-offs or had insufficient information to be coded.
Did you take any of these actions as a result of Annie's coronavirus precautions messages?
Frequent free-text responses in the "Other" field mostly included users reporting that they were not receiving messages at all or messages that required action of them (13). Other responses had insufficient information in their response to be coded.
Would you have taken any of the following actions if you were not receiving Annie's coronavirus precautions messages?
The most frequent type of free-text response in the "Other" field included veterans reporting that they would have sought information from alternative information sources such as the Centers for Disease Control (5), the World Health Organization (1), news outlets (4), a health department (2), medical professionals (2), or the internet in general (6). Other responses had insufficient information in their response to be coded.

What impact have Annie's coronavirus precautions messages had on you?
Although most "Other" responses had insufficient information to be coded, a couple of users had a critique of the messages, the Annie application, or the process to sign up (2). Others simply indicated that they use Annie as an information source (3).

DISCUSSION
Survey findings support that the use of Annie's CPP resulted in what is likely a substantial resource savings for the VA, as well as non-VA community healthcare. Of the 1134 veterans who answered the question "Would you have taken any of the following actions if you were not receiving Annie's Coronavirus Precautions messages?", 532 (47%) report they would have reached out to VA in at least 1 way (secure message, phone call, visit, or some combination of those). In addition, 159 (14%) veterans reported they would have interacted with community (non-VA) care (called or sought care). Of the 1134 veterans who answered the question "What impact have Annie's coronavirus precautions messages had on you?", 172 (15%) reported that they opted not to interact (message, call, or visit) with a healthcare provider (VA or community) because Annie provided the information they sought. Moreover, 124 (11%) responses mentioned a sense of connection as something that was helpful in Annie's messages. This is important when feelings of isolation due to lockdowns and closures begin to set in: the landing page for Mental Health Apps on mobile.va.gov saw an 8476% increase in pageviews from March to April. 8 Additionally, 879 (76%) of veterans reported at least 1 positive sentiment (felt more connected to VA, confident, or educated and/or felt less anxious) as 1 impact of CPP. This is in line with previous studies on Annie. 9 The over 800 open-ended responses for the question "What about Annie's messages was helpful for you?" also support the positive sentiment findings (Table 2).

CONCLUSION
The findings from this study have implications for other healthcare systems. The use of an automated SMS/text service to help manage a patient population during the COVID-19 pandemic is a costeffective way to simultaneously encourage patient monitoring and provide updated information while demonstrating potential healthcare resource savings. In addition, because those who subscribed to the CPP messages were overwhelmingly positive about receiving them on a regular basis, we have shared the messages and flow map should any healthcare system want to use them as the country continues to navigate through the COVID-19 pandemic.
These findings should be interpreted within the context of certain limitations. First, as guidance is evolving, some changes should be expected to the protocol itself. Around 600 Veterans had opted out of receiving Annie CPP messages prior to our survey, and so those Veterans are not represented in our sample. However, that is still a small percentage of the active subscribers. Also, we did not collect data on how long each veteran has been subscribed to the CPP messages. The length of time a veteran has been receiving the messages may have influenced their responses. Lastly, a couple of free-text responses noted that responders were not receiving messages. Since completion was on a "self-selection" basis, it is possible that a few responses were not based on experience with CPP. However, that is true of many survey opportunities. Despite these limitations, the survey seems to suggest that the Annie CPP has had a substantial positive impact on the veteran population.

AUTHOR CONTRIBUTIONS
KLF, JJM, NRW, BV, JH, JJS conceived and designed the study. JH and BV were responsible for the survey data collection. BL, JH, JJM, JJS, and JMR contributed to the analysis and interpretation of the survey data. BV, BL, and NRW provided protocol information such as history and usage statistics. All authors revised the manuscript critically for important intellectual content. JJS had principal responsibility for drafting the manuscript. All authors approved the final version.