Enhanced surveillance of hospitalized COVID-19 patients in Europe: an evaluation of the I-MOVE-COVID-19 surveillance network

Abstract Background A pre-existing, well-established European influenza surveillance network known as I-MOVE enabled the rapid implementation of a European multi-country COVID-19 hospital surveillance network for surveillance of hospitalized COVID-19 cases in early 2020. This network included 257 hospitals in 11 surveillance sites across nine countries. We aimed to identify whether the surveillance objectives were relevant to public health actions, whether the surveillance system met its objectives, where and how shortcomings could be improved, and whether the system was sustainable. Methods We identified six key attributes (meeting objectives, usefulness, timeliness, data quality, simplicity and sustainability) to assess, using Centers for Disease Control and Prevention’s evaluation framework. We analyzed pooled datasets, held interviews and group discussions with 10 participating and coordinating sites and gathered feedback through web surveys. Results There was overall agreement that the surveillance objectives had been met and being involved in a network of European partners had additional important benefits for stakeholders. While the publication of the outputs was not always sufficiently timely, data submission processes were considered straightforward and the key surveillance variables (age, sex, hospital admission and mortality data) were complete. The main challenges were identified as the collection of the large number of variables, limited available human resources and information governance and data protection laws. Conclusions I-MOVE-COVID-19 delivered relevant and accurate data supporting the development and implementation of COVID-19 surveillance. Recommendations presented here identify learning opportunities to support preparedness and surveillance response for future pandemics. The applied evaluation framework in this study can be adapted for other European surveillance system evaluations.


I
n response to the emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), many European countries established comprehensive multi-level surveillance systems for COVID-19 cases, to provide critical information for public health decision-making. 1 The I-MOVE-COVID-19 hospital surveillance network was established by the I-MOVE-COVID-19 Consortium (founded in February 2020 and coordinated by Epiconcept 2,3 ).This was an expansion of the multi-country Influenza-Monitoring Vaccine Effectiveness (I-MOVE) network founded in 2007 to measure influenza vaccine effectiveness (VE) in Europe. 4Having this pre-existing, wellestablished European platform facilitated the development and implementation of COVID-19 hospital surveillance across surveillance sites, as they were able to build on existing common protocols, expand existing information governance procedures and re-train staff already familiar with the surveillance activities.
The surveillance objectives were to (i) describe clinical and epidemiological characteristics of hospitalized COVID-19 SARI patients, (ii) describe the virological characteristics of SARS-CoV-2 in these patients, (iii) improve understanding of severe disease progression to guide patient management and public health response, (iv) strengthen COVID-19 preparedness through hospital surveillance, (v) describe severe COVID-19 cases by sex, age group and risk/protective factors; and (vi) describe in-hospital COVID-19 deaths. 5hese objectives were established in the early days of the COVID-19 pandemic in a rapidly changing context with much uncertainty as to how the situation would develop.It was envisaged that pooling data from each country, thereby providing a greater sample size, would allow for a more accurate and representative description of hospitalized COVID-19 cases.Surveillance bulletins were prepared quarterly, and regular network meetings were held to share experiences on the implementation processes and developments across each participating site's surveillance systems.
Any surveillance system, and particularly new systems, should be evaluated to identify possible improvements in performance and the overall public health response.Our evaluation objectives were (i) to identify whether the surveillance objectives were relevant to public health action, (ii) to assess the surveillance objectives and areas for improvement, (iii) to evaluate the sustainability of the surveillance system and finally, (iv) to consider more efficient or alternative routes to achieve the surveillance objectives.We report the results of this comprehensive evaluation of the European COVID-19 hospital surveillance system and provide recommendations to improve the current system and to support the planning and implementation of future enhanced surveillance activities.

COVID-19 hospital surveillance network
Results were collected from 11 surveillance sites, i.e. groups of hospitals and public health institutes within specific European regions that submitted the surveillance data collectively.A total of 257 participating hospitals across nine European countries were included in I-MOVE-COVID-19 (Albania, Belgium, England, France, Lithuania, Portugal, Romania, Scotland and Spain).Spain and France had two participating sites in different regions.With the exceptions of hospitals in England and Scotland-where surveillance was nationwide and register-based via data linkage of routine hospital and infection datasets-an average of three hospitals were included per site.All 23 sentinel hospitals collected COVID-19 hospital data through questionnaire-based surveillance, either on paper or electronically.Every quarter, the participating sites securely submitted their casebased surveillance data to Epiconcept.There were 105 separate variables requested for data submission covering patient demographics, hospital records, severity indexes, risk factors, SARS-CoV-2 presenting symptoms, laboratory test results and clinical information.The data were cleaned and pooled before sharing with Public Health Scotland (PHS), who together with Epiconcept led on analysis (figure 1).Analytical results were presented in quarterly surveillance bulletins that were published on the I-MOVE-COVID-19 website 3 and used to improve understanding of I-MOVE's objectives.No ethical approval was required for this evaluation as existing nonidentifiable data were used.

Evaluation framework
The framework for this evaluation was adapted from the 'Updated Guidelines for Evaluating Public Health Surveillance Systems' of the Centers for Disease Control and Prevention (CDC). 6The surveillance system was assessed against six attributes tailored to suit the scope and the specific evaluation objectives including the achievement of surveillance objectives, usefulness, timeliness, data quality, simplicity and sustainability.Indicators to measure these attributes were defined using adapted CDC and European Centre for Disease Prevention and Control (ECDC) guidelines 6,7 (table 1).

Data collection
The evaluation was performed unblinded by three internal evaluators from PHS, part of the I-MOVE-COVID-19 hospital surveillance network, and data were collected through a mixed-method approach using quantitative and qualitative data collection methods, including document review and stakeholder consultation (survey and semistructured group discussions and interviews).Table 1 shows the data collected for each indicator, reviewed between October and December 2021.

Document review
All documents and systems related to the surveillance system were identified and reviewed.These included the protocol for I-MOVE-COVID-19 hospital phased surveillance, documents describing standard operating procedures, surveillance bulletins, scientific posters presented at scientific conferences, network meeting agendas and meeting minutes.

Survey
The indicators of each attribute were used to formulate survey questions that were administered online via LimeSurvey. 8The survey was piloted and adapted accordingly prior to its administration.The survey link was sent to all network members from the participating sites.
Respondents were asked to identify their role in the surveillance system but otherwise remained anonymous.

Group discussions/interview
Depending on participant availability, either semi-structured interviews or group discussions were held with the sites, to allow a blend of closed-and open-ended questions, accompanied by follow-up questions.The questions were prepared in advance using the attributes and indicators under surveillance and the key discussion points were shared with the interviewees in advance of the meeting.The sessions were held virtually by the authors using Microsoft Teams between November 2021 and January 2022.Interview notes were recorded on a structured interview template according to each attribute and coded into the most commonly mentioned themes within each indicator by two researchers.

Analysis of the datasets
Pooled datasets from six data collections with data up to 31 December 2021 were used to derive quantitative indicators (table 1).Timeliness of the surveillance system was measured by calculating the number of days between key steps in the surveillance process; specifically, the time between the initial date of analysis and the median date of admission, and the publication date of the surveillance bulletins.The data quality of the surveillance system was measured by calculating the level of completeness of the collected variables, the proportion of reported cases that fit the case definition 9 and the number of unanalyzed variables.All analyses were performed using Microsoft Excel.

Document review
1][12][13][14][15][16] All network meeting agendas and meeting minutes of the hospital surveillance system were identified and included in the document review.

Semi-structured group discussions and interviews and web survey
There were two interviews with individuals from two sites and four group discussions.Overall, all countries and 9 out of 11 participating sites were represented.The link to the web survey was sent to all stakeholders in the network and ten completed surveys were received.The roles of the respondents were varied (diverse and multiple roles could be selected): regional and national public health institute-based network members (n ¼ 3), co-ordinators (n ¼ 3),

Evaluation of system attributes
The achievement of surveillance objectives Respondents generally indicated that the surveillance system met all six of its objectives.From the 10 completed surveys received, 8 responded to six parts of this question.Most questions (42/48) were answered by the respondents in (strong) agreement that the surveillance system met its objectives.Five responses were neutral, and only one answer disagreed that the surveillance system met all its objectives.

Data collection
Forty-two percent (61/145) of the requested variables as defined in the study protocol and collected for surveillance purpose (table 3) were selected by at least one respondent as unnecessary for surveillance purposes.They also commented on variables which could be omitted for other reasons, such as postcode which could lead to patient identification, clinical characteristics that did not align with typical epidemiological surveillance, and hospital ward and patient test/scan results as these were free text complicating comparability.
Six out of nine interviewed sites reported that too many variables were required, resulting in high levels of missing data for some variables that were not considered core.It was suggested that the number of variables should be determined according to variable completeness levels, and from both the interviewees and respondents, a common view was that if timely data collection is desired then only essential variables, such as patient demographics, hospital information (e.g.hospital admission and discharge dates)and severity indicators should be collected (table 3).

Added value of participation in a European surveillance network
Stakeholders identified that a key strength of this surveillance system was that it involved a network of European partners.Having a network supported the understanding of developments across different sites, strengthened relationships with stakeholders, supported advocacy for national COVID-19 hospital surveillance and helped to attract funding from other sponsors to support the surveillance activities.Collaboration within the network allowed sites to automate data flow processes which was recognized and highly praised at the national governmental level.
The quarterly publication of the surveillance data and network meetings were generally perceived positively.This allowed sites to raise awareness of the importance of collecting this data, thereby supporting the implementation and development of their surveillance.

Number of days between key steps in the surveillance process
An average of 39 days (range: 15-63 days) was recorded between the pooled dataset being sent for analysis and publication date (table 2).Across surveillance bulletins, the median number of days between mean admission date and publication was 132 days (table 2).On average, the median date of dataset submission was within 1 day of the agreed submission deadline (table 2).

Timeliness and frequency of system reporting (data dissemination) to meet surveillance objectives
Half of the respondents considered the frequency of the publications sufficient to meet the surveillance objectives, although it was suggested that more regular, short, and rapid communications could have been combined with lengthier in-depth surveillance bulletins published less frequently.However, it was also acknowledged that more frequent data submissions would have caused an increased burden on an already-strained workforce.

Data quality
Key surveillance variables (age, sex, hospital admission and mortality data) had completion rates of 100%.Patient's type of residence, healthcare worker status, symptom onset date and most chronic conditions had completion rates between 30% and 69%, whereas patient's smoking status, pregnancy status, three chronic conditions (anaemia, stroke and tuberculosis), and all symptom data had completion rates <25% (table 3).The overall completion rate of the variables varied by site and decreased over time.Data from England and Scotland were the main contributors to this decrease as they supplied more data and their data completion for nonessential surveillance variables decreased over time.An average of 14% (n ¼ 9611/69 734) of cases that did not meet the case definition were excluded.The proportion of analyzed variables in the surveillance bulletins varied between 43% and 63%.

Stakeholder opinion of data quality
While most interviewees performed regular data management activities to ensure better data quality, such as variable recoding, deduplication, cross-validation and translation to English at a national level, several sites indicated that these were completed by the hospital sites directly.Generally, interviewees indicated that data quality improved over time.One interviewee suggested introducing Notes: No formal deadline was given for the first data submission; therefore, the number of days that data were submitted prior to the agreed submission deadline could not be calculated for surveillance bulletin 1. Minus figures refers to days submitted after the agreed submission deadline.conditions and rules during automated data collection to facilitate high data quality and improved completeness levels.

Data collection
Several challenges were identified by respondents around the simplicity of data collection, including information governance processes and following data protection laws, obtaining clearance from data protection bodies or patients, and developing automated data collection approaches.Respondents and interviewees suggested that more frequent updates of the surveillance protocols and training sessions for new staff would be helpful.

Data collation
The collation of data was deemed straightforward, although respondents expressed challenges such as the lack of automation in some sites, high workload and the large number of requested clinical variables (e.g.chest X-ray findings, ECG findings).

Data submission
Preparing the data for the submission process was considered straightforward by respondents and interviewees.This process took between 2 and 30 days with an average of 11.5 days.Four out of nine sites reported during the interviews that data submission deadlines were challenging due to competing deadlines, lack of human resources, compliance with information governance procedures or delayed data collection in hospitals.

Sustainability
Sites' stances differed regarding the continuation of COVID-19 surveillance post the I-MOVE-COVID-19 project.Sites intending to continue their COVID-19 hospital surveillance will do so either through their national surveillance programme or through other integrated European respiratory surveillance networks such as the 'Vaccine Effectiveness, Burden and Impact Studies' (VEBIS), 17 which also includes E-SARI-NET, the European SARI surveillance network.
For sites not part of these networks, continuity of the surveillance will depend on the development of the pandemic.

Discussion
This evaluation of the multi-country European hospital surveillance system for COVID-19, focused on the description of the system and assessment of the surveillance attributes.Survey respondents were in overall agreement that the surveillance system met its objectives.

Usefulness
This rapidly deployed European surveillance network was an important enhancement to existing surveillance mechanisms used in Europe.Sharing and pooling of European data allowed questions to be answered that individual countries could not answer efficiently alone, particularly where data is scarce (e.g.information on chronic conditions). 18Being involved in a European network was also considered extremely valuable, as it assisted individual sites to strengthen national surveillance and supported relationship building with different national and international stakeholders.Cross-country cooperation and data sharing have been recognized to support the centralization of efforts, disseminate information and better prepare and respond to global health challenges. 19,20Combining the expertise and resources of I-MOVE influenza, an existing European multidisciplinary network, to include COVID-19 has shown that European surveillance systems can be adapted in a timely and flexible way suggesting potential for adaptability to also include other pathogens.
While the experience and infrastructure from this surveillance network could form a foundation for future emerging pathogens, it is acknowledged that the adaptability of the surveillance system would still depend on various factors such as the nature of the pathogen, the availability of data, and coordination between different stakeholders.Many hospitals and clinical staff were overburdened during the pandemic, so the collection of the large number of variables required at the start of the pandemic to better understand this novel virus in a hospital setting was challenging.Data collection should be designed to meet the information needs of public health decision-makers, the public and health workers. 21It is key that the benefits of collecting additional variables are balanced against the potentially increased burden on clinical staff.A minimum dataset for integrated influenza/SARS-CoV-2 sentinel surveillance is recommended to support the data quality and sustainability of future surveillance systems. 21,22

Timeliness
The publication of the bulletins was not timely enough to influence public health actions.However, most sites shared their own data prior to publication to inform decision-makers and ensure an adequate response.It is acknowledged that improved timeliness of the reporting of pooled and therefore more powerful analyses could have influenced wider public health actions but it was also important not to compromise the validity and quality of the data, and to burden sites with additional data collection and submissions.The development of automated data submission systems linked to electronic databases would address this problem, but the development of such systems across Europe is heterogeneous. 23

Data quality
Basic demographic data generally had high completeness levels, whereas more specific patient characteristics were less complete.While some issues stemmed from clinicians being overwhelmed by the pressures of the pandemic, other reasons for incomplete data collection included difficulty obtaining ethical permission for certain variables, patients' unwillingness to disclose sensitive information 21 and symptom data not always being routinely reported in hospital settings.While sites that collected data through a sentinel questionnaire-based approach generally reached higher levels of data completeness, sites that collected their data through linkages of national registers accrued more cases.These sites supplying more data submitted fewer non-essential surveillance variables, resulting in lower variable completeness.The World Health Organization (WHO) suggests that good quality data with timely reporting, even from fewer sites, are more useful than a large volume of poor-quality data not reported in a timely manner. 22It is therefore key that sites collecting data on more cases assess whether they can be effectively managed, monitored and sustained, while considering the increased human and financial resources, technical and operational assistance that may be required. 22

Simplicity
While the operation of the COVID-19 hospital surveillance was generally perceived as straightforward, the main challenges were associated with data collection processes.The collection of COVID-19specific hospital data has been previously identified as an additional burden for clinical staff. 24To facilitate the collation and analysis of data at the national level, countries may consider establishing or strengthening electronic data platforms that link epidemiological and virologic data to sentinel and non-sentinel surveillance systems and are accessible to stakeholders. 22Flexible electronic systems should support data collection and transfer in the changing COVID-19 pandemic situations as well as in other emerging outbreaks. 25

Sustainability
The sites took different positions on the continuation of COVID-19 hospital surveillance, with some planning to continue and others depending on the pandemic's development.ECDC and WHO recommend integrated respiratory surveillance of COVID-19, influenza and other respiratory pathogens. 21,22While most European countries have established comprehensive surveillance systems for COVID-19 with a large proportion reporting all positive cases regardless of indication for testing, European countries are now being encouraged to transition from emergency surveillance to more sustainable and objective-driven surveillance systems. 21As a result, integrated European respiratory surveillance networks that include COVID-19 hospital surveillance have been developed. 17

Limitations
This evaluation has some limitations.The variance of the different types of surveillance systems (sentinel vs. national surveillance), as well as the discrepancies in datasets, number of collected variables and completion rates, made comparisons challenging.The evaluation was performed during an emergency response, which may have reduced stakeholder participation, thereby reducing representativeness from participating sites.For example, due to staffing and resource issues, it was not possible to arrange separate one-to-one interviews with all stakeholders from all participating sites which would have provided the most comprehensive picture, but we were able to arrange either group discussions or interviews with representatives from all countries and nine out of 11 sites.In contrast, the response rate of the survey was low, despite reminder e-mails being sent to the network in attempts to improve the response rate.Responses to certain questions of the survey may also have reflected the respondent's perception of the current system within their site or specific area of work and may not necessarily provide a general view of the overall participating site.Although facilitators attempted to mitigate dominance bias, responses may have been biased towards participants with strong opinions.The evaluation was performed by internal evaluators, which enabled a strong understanding of the system, an extended period of participant observation, and the ability to make ongoing improvements throughout the implementation.However, this may have introduced confirmation bias and inhibited their ability to act purely as objective outsiders.

Recommendations
Our evaluation has generated the following recommendations specific to any future deployment of an integrated respiratory surveillance system:

Surveillance methods
• Increase the timeliness of reporting data to support public health actions in a timely fashion.• Reduce the minimum mandatory dataset in collaboration with participating sites to enable increased data completeness and improve data quality.• Consider introducing a uniform template which is consistent between data submissions to harmonize the order and coding of variables.• Further reduce the substantial inter-and intra-country differences in surveillance methods to ensure homogeneity of the scope, focus, objectives, methodology, resources and reporting across different regions and countries.

Communication
• Continue further data analysis to identify preventive and risk factors with the available data collected through the surveillance and share results with the network and scientific community.

Sustainability
• Monitor outbreak-related workload issues to ensure the sustainability of the surveillance.

Table 1
Methods to evaluate the surveillance system attributes and its set key indicators I-MOVE-COVID-19 surveillance network evaluation 183 data managers (n ¼ 2), hospital-based network members (n ¼ 2), university-based network members (n ¼ 2) and laboratory expert (n ¼ 1).

Table 2
Number of days between key steps to assess timeliness in the surveillance system, by surveillance bulletin, I-MOVE-COVID-19 hospital surveillance network, Europe, 2020-21

Table 3
Average completion rates across all participating sites of the I-MOVE COVID-19 hospital surveillance network and data submissions of variables analyzed and included in the surveillance bulletins, by category only if height, weight and BMI not collected; can be calculated) 60Notes: BMI, body mass index; ECMO, extracorporeal membrane oxygenation; HDU, high-dependency unit; ICU, intensive care unit.I-MOVE-COVID-19 surveillance network evaluation 185 • Consider lessons learned from the I-MOVE-COVID-19 hospital surveillance network when data submission continues under other networks (VEBIS) to ensure sustainability.