PHIRE (Public Health Innovation and Research in Europe): methods, structures and evaluation

Research in Europe (PHIRE), building on previous European collabora- tive projects, was developed to assess national uptake and impacts of European public health innovations, to describe national public health research programmes, strategies and structures and to develop participation of researchers through the organizational structures of the European Public Health Association (EUPHA). This article describes the methods used. Methods: PHIRE was led by EUPHA with seven partner organisations over 30 months. It was conceived to engage the organisation of EUPHA – working through its thematic Sections, and through its national public health associations – and assess innovation and research across 30 European countries. Public health research was defined broadly as health research at population and organisational level. There were seven Work Packages (three covering coordination and four for technical aspects) led by partners and coordinated through management meetings. Results: Seven EUPHA Sections identified eight innovations within the projects funded by the Public Health Programme of the European Commission Directorate for Health and Consumers. Country informants, identified through EUPHA thematic Sections, reported on national uptake of the innovations in eight public health projects supported by the European Union Public Health Programme. Four PHIRE partners, each taking a regional sector of Europe, worked with the public health associations and other informants to describe public health research programmes, calls and systems. A classification was created for the national public health research programmes and calls in 2010. The internal and external evaluations were supportive. Conclusions: PHIRE described public health innovations and research across Europe through national experts. More work is needed to conceptualize and define public health ‘innovations’ and to develop theories and methods for the assessment of their uptake and impacts at country and cross-country levels. More attention to methods to describe and assess national public health research programmes, strategies and structures – contributing to development of the European Research Area. of public health in assessed public the Public

Introduction: Public Health Innovation and Research in Europe (PHIRE), building on previous European collaborative projects, was developed to assess national uptake and impacts of European public health innovations, to describe national public health research programmes, strategies and structures and to develop participation of researchers through the organizational structures of the European Public Health Association (EUPHA). This article describes the methods used. Methods: PHIRE was led by EUPHA with seven partner organisations over 30 months. It was conceived to engage the organisation of EUPHA -working through its thematic Sections, and through its national public health associations -and assess innovation and research across 30 European countries. Public health research was defined broadly as health research at population and organisational level. There were seven Work Packages (three covering coordination and four for technical aspects) led by partners and coordinated through management meetings. Results: Seven EUPHA Sections identified eight innovations within the projects funded by the Public Health Programme of the European Commission Directorate for Health and Consumers. Country informants, identified through EUPHA thematic Sections, reported on national uptake of the innovations in eight public health projects supported by the European Union Public Health Programme. Four PHIRE partners, each taking a regional sector of Europe, worked with the public health associations and other informants to describe public health research programmes, calls and systems. A classification was created for the national public health research programmes and calls in 2010. The internal and external evaluations were supportive. Conclusions: PHIRE described public health innovations and research across Europe through national experts. More work is needed to conceptualize and define public health 'innovations' and to develop theories and methods for the assessment of their uptake and impacts at country and cross-country levels. More attention to methods to describe and assess national public health research programmes, strategies and structures -contributing to development of the European Research Area.

Introduction
European countries seek to ensure the health of their citizens, and health has been included within the European Union (EU) treaties since 1992. The European Commission Directorate for Health and Consumers created an annual funding programme, the Public Health Programme, in 2003, followed by the Second Health Programme from 2008. 2 These programmes have supported European collaborative projects annually across priority fields of health information, health threats and health determinants, and the actions for implementation have included grants for projects and organizations, health-related conferences and tenders for work on specific topics. Partners have included governmental and civil society organizations, health services and private and research organizations.
The EUPHA is a civil society organization coordinating public health research and practice in Europe. 3 For over 20 years, it has organized an annual conference, and publishes the European Journal of Public Health. EUPHA was a partner in two previous collaborations describing public health research in Europe. 4,5 PHIRE 6 was developed to continue this work and as well as preparing for the next phase of the European research programme-Horizon 2020. 7

Methods
The objectives and methods of PHIRE were developed according to the type of activities usually funded by the Health Programme. PHIRE also sought to promote European integration by actively engaging individuals and organizations through EUPHA structures. PHIRE was carried out in two phases. The first described the uptake within member states of selected European public health innovation projects, gained information on national programmes and calls and documented national public health research systems. The second phase brought this knowledge to national stakeholder workshops, created national reports and brought these together for comparative analysis.

Meanings
Public health research (see definition in Box 1) is concerned with how to improve the health of the population, and how to ensure the effective and efficient organization of health care. It operates at a complex level between scientists, individuals and society, and overlapping with clinical medicine (figure 1).
The World Health Organization has supported the development of health research systems, 9 in collaboration with the Council for Health Research and Development (COHRED), in low and middle income countries-although not in Europe. 10 Public health research systems form a subset of this field.
Innovation is a relatively new concept in its use at European level. In the health field, social innovation is seen as an important balance to technical and commercial innovation. Social innovations have been described as 'new ideas (products, services, models) that meet social needs more effectively than others and create new social relationships or collaborations'. 11 In PHIRE, demonstration projects of the Public Health Programme were public health innovations, which introduced new forms of practice across European countries.

Work packages
The management of PHIRE, as for other projects for the Health Programme, was divided into Work Packages, each led by one or Box 1 Definition of public health research used in SPHERE 8 'Public-health research refers to the organized quest for new knowledge to protect, promote and improve people's health. It is undertaken at population or health services level, in contrast to laboratory (cellular) or clinical (individual) health research; differs from public-health practice (which also uses scientific methods), as it is designed to obtain generalizable knowledge rather than to address specific programmes for service delivery; is usually goal-oriented, addressing questions of policy relevance, and may be published in either academic journals or reports; and uses a range of observational methods, including surveys, registers, data sets, case studies and statistical modelling, and draws on disciplines including epidemiology, sociology, psychology and economics, and interdisciplinary fields of environmental health, health promotion, disease prevention, health-care management, health-services research and health-systems research.' more partners. Work Packages 1-3 followed the standard requirements for the application. Work Package 1, for project coordination, was led by EUPHA itself from the office in Utrecht, the Netherlands. Work Package 2 concerned dissemination, drawing on the opportunities for publication and presentation available through EUPHA's conferences and journal, and its links with transnational and national public health organizations. Work Package 3, on evaluation, was led by the Netherlands National Institute for Health Services Research (NIVEL), and recorded progress and impacts of the project from participants and from external reviewers. Four Work Packages provided the technical elements of PHIRE. In Work Package 4, led by KI, EUPHA Sections mapped the impacts of eight innovative projects that had been co-funded through the first European public-health programme in 2003-05. Work Package 5, coordinated by EHESP, was focused on EUPHA members-the national public health associations-to review their health research structures, the national impacts of the innovative projects and provide a national report. Work Package 6, led by UKFPH, organized discussion and coordination between public health research stakeholders in Platform meetings at the European Public Health Conferences in Copenhagen, 2011, and Malta, 2012. Work Package 7, led by EHESP, created the database of PHIRE reports and publications. Full reports of PHIRE are at the web page www.eupha. org/phire.

Evaluation
Evaluation was a separate 'horizontal' work package in PHIRE. The evaluator was invited to all Management Committee meetings, and undertook reviews of both internal performance and external perceptions. The evaluator constructed a prospective framework, with timed reviews, and drew qualitative data on perceptions from workshops/platform meetings organized on PHIRE at the European Public Health conferences in Amsterdam, 12 Copenhagen 13 and Malta. 14

Questionnaire
In October 2010, the Chair of the Section Council (Professor Kristina Alexanderson) sent an invitation letter to Presidents of the EUPHA Sections to participate in PHIRE. At the European Public Health Conference in Amsterdam in November 2010, PHIRE was presented at the Section Council, further discussed individually with Section Presidents and presented at two annual meetings of Sections. Seven of the EUPHA Sections identified relevant innovative projects and chose to participate in PHIRE. One of the Sections took on two projects, so that eight innovations were finally included in PHIRE ( Table 1). The Section presidents were asked to contact their Section members, seeking one country informant (CI) from each of the 30 European countries in PHIRE. Where a Section did not have responses for a country, other EUPHA Section membership lists or other existing networks could be used.
PHIRE partners discussed what was meant by uptake and impact of public health interventions, how to measure these concepts at different structural levels and through various ways in society and how to record responses across different innovation projects retrospectively, with single informants in different countries, in as standardized way as possible. With an initial set of questions, PHIRE partners held a workshop in Stockholm, followed by telephone meetings and emails, to refine the questionnaire. The criteria agreed included the following: There should be a short core instrument with general questions for all projects. It should be possible to include project-specific items and country-specific items. Both close-and open-ended questions should be included-the latter because of the exploratory nature of PHIRE. Questions could include perceptions of impacts, and reasons why the innovation project succeeded in dissemination ('lessons learned'). The instrument was to be web based or could be completed with a telephone interview.
A small pilot study provided guidance for improvements of the questionnaire and web survey methods.

Data collection
The CIs for the survey were gained through members of EUPHA Sections supplemented by experts known to the Section president. 15 The invitations for CIs to participate in the PHIRE web survey were sent by email, with the link to the web-based questionnaire. With eight projects in 30 countries, the total possible reporting of innovation projects would have been 240. Experts were invited for three-quarters of these, and CIs were gained for 41%.
The type of information requested was often not well documented, and although the CIs were experts in their field, they sometimes had to seek out different sources, including having to contact other people. In a few cases more than one person was involved in answering the survey.
The EUPHA Section leads were not able to establish full coverage across European countries. About half of CIs were recruited from outside the EUPHA Sections. Contacting the CIs by telephone before getting the web survey was considered to lead to more positive Table 1 The eight projects chosen from the European Union Public Health Programme, with name of responsible EUPHA Section President CHOB-Children, Obesity and Associated Avoidable Chronic Diseases: Section of Food and Nutrition (Professor Christopher Birt, UK). To collect information on the extent and nature of food marketing to children and on existing measures at national level to counteract the effects. CSAP-Child Safety Action Plans: Section of Injury Prevention and Safety Promotion (Dr Mathilde Sengö lge, Austria). To apply comprehensive injury indicators, strategies of good practice and capacity building that enhance policy and decision-making to reduce child injuries. EAAD-European Alliance against Depression: Section of Public Mental Health (Professor Jutta Lindert, Germany). To spread information about suicide prevention at four levels (GPs, professionals, general public and at-risk individuals). ENHIS-Implementing Environmental and Health Information Systems in Europe: Section of Environment Related Diseases (Dr Peter van den Hazel, the Netherlands). To generate and analyse environmental health indicators within the EC Health Information System. EUCID-European Core Indicators in Diabetes Mellitus: Section of Chronic Diseases (Dr Iveta Nagyova-Rajnicova, Slovakia). To demonstrate and compare systems for recording and reporting on diabetes care. HA-Healthy Ageing: Section of Public Mental Health (Professor Jutta Lindert, Germany). To collect, review and analyse existing data and produce a report on practices and policies for older people's health. URHIS-European System of Urban Health Indicators: Section of Urban Public Health (Dr Arpana Verma, UK). To develop an urban health information and knowledge system using standardized methodology for data collection and comparison. VENICE-Vaccine European New Integrated Collaboration Effort: Section of Public Health Epidemiology (Professor Giuseppe La Torre, Italy). To facilitate and accelerate good practice within European countries on surveillance and monitoring of immunization programmes.

8
European Journal of Public Health responses, while sending the survey to colleagues with only general prior contact did not seem to have the same positive effect. No information was collected about those invitees who refused to respond or dropped out. Nevertheless, some invited CIs stated that this was not their area of expertise or interest. Others stated lack of time was the main reason for not participating. Weaknesses of the methods were recognized.
CIs had different levels of knowledge about the innovation projects. The limited experience of some CIs could mean that uptake and impacts were underestimated. CIs that had been directly involved in the chosen innovative project had greater information about impacts: however, this does not necessarily mean that they overestimated the impact. There were no CIs for half of the possible responses on the innovation projects. Lack of remuneration for CIs probably contributed to weak response rates. Single informants per country have more reporting bias than averages of multiple reports. PHIRE was a retrospective cross-sectional study. Although all eight innovation projects had continued beyond the initial project period, there were no baseline data from which to measure the impacts, nor the impacts of confounders. And while the CIs had general social survey expertise, they were not experienced in conducting this type of complex assessment.

Describing national public health research systems
National and comparative data on public health research systems were collected with the national public health associations (EUPHA members). Four PHIRE partners each worked with a regional grouping of seven to eight national associations. In the first phase, information was collected describing national public health research calls and programmes, and reports on national health research systems were reviewed. In the second phase, reports were prepared following national workshops, which based around information gathered in the first phase.
A structure for collecting information on public health research programmes and calls was developed by PHIRE partners, drawing on lessons learned with the preparation of country profiles in STEPS. 5 To focus the work, programmes and calls on public health research opened during the previous full year-2010-were chosen. The data collection form was piloted in each of the countries of the regional coordinators (France, Lithuania, Malta and Slovakia), and the final version was completed after a telephone conference.

Public health programmes and calls
Four areas were addressed: Identification of the funding thematic programme or call for research projects; Description of the programme/call; Funding and duration of the programme/call; Eligibility criteria and other conditions to obtain funding.
Emails were sent to country representatives of the EUPHA Governing Council asking each National Public Health Association to designate a responsible person to collaborate with PHIRE. A further letter was sent from EUPHA, acknowledging contributions received and again reminding National Public Health Associations about PHIRE.
A sequential strategy to gain collaboration from national public health associations was agreed between the regional coordinators: first an offer of contact by telephone; then offer assistance in organizing the information and filling the forms; then asking the first contact to suggest another person to be contacted. Where these approaches failed, information was sought from other outside contacts.
A typology of public health programmes and calls was developed by PHIRE partners, based on the general titles reported from countries. The typology was discussed in a project meeting, then by email, and the final classification was agreed.
Weaknesses of the methods were recognized.
The reports depended on the informant's knowledge of the country's public health research system-which differed to an unknown degree. Many country respondents stated that a specific and significant allocation of time was needed to collect the information on programmes and calls opened in 2010. The numbers of programmes and calls reported varied by country for different reasons. Larger countries, with larger total spending, tended to organize research calls separately, while smaller countries made a single overall call for research. In most countries, the opportunity (in principle) for public health research might exist within a broader call, but this was not necessarily identified by the informants. In many countries, research is supported through grants negotiated directly between institution and government department, rather than through the competitive programmes and calls.
Only one year was chosen: in some countries calls for research are not made every year-and moreover, 2010 was a particularly difficult budget year across European countries.

Internal evaluation
Information from web pages

Review of EPH conference platform meetings
PHIRE held workshops and platform meetings at EPH conferences. [12][13][14] At the second and third, with interest in the results from PHIRE, there were signatures of 89 attenders from 19 countries, including countries (Croatia, Turkey, Canada and the USA) from outside the EU27+3 group studied in PHIRE. The majority of attenders (72%) were researchers, 19% were policy makers (four were from the European Commission) and 9% from civil society organizations. The participants were sent follow-up emails, and these gained a range of comments, which were generally positive. 15 'I really enjoyed the workshop in Copenhagen' 'The individual presentations were quite clear' 'It's good to establish a European database for the funded public health projects' But they also gave encouragement for more dissemination.
'The results from PHIRE should be more widely published and used both on European and on national levels' 'The question about public health research as a priority should be reiterated constantly. Despite the increased EU funding to health research, public health may still be left with a minimal share' 'DG Research should give it higher funding priority within all of health-related research' 'From this experience emerged clearly the need for better coordination at EU as well as at Member State level. This requires more attention by EU level operators and by national operators, where the PHIRE: methods, structures and evaluation 9 Ministry of Health and the Ministry of Education and Research don't communicate enough'

Discussion
PHIRE is the first study to assess public health innovations and research together at national level across Europe. It complements the evaluation of the Public Health Programme undertaken for the European Commission, 16 which made electronic interviews and case-study reviews of selected projects at European level. PHIRE was directed towards public health researchers and policymakers through the structure of the EUPHA. The European Court of Auditors 17 identified weaknesses in the Public Health Programme, including that there was too large a range of project topics to choose from. The eight innovations assessed by PHIRE ranged broadly, and it was possible to track them retrospectively through CIs, but the findings cannot be generalized to all public health projects.
Evaluation and dissemination were undertaken by partners within PHIRE. Commentators noted the considerable variation across Europe in the field of public health, and reflected the known challenges of cross-national research. 18 The respondents were positive on the contribution of the PHIRE results, although reservations were also expressed.
Methods and quality systems for recording perceptions of public health experts about public health research need further development. PHIRE drew on previous experience in SPHERE 4 and STEPS 5 in assessing public health research systems. The European Union has a broader system for reporting on national research systems, but focusing down to public health research is challenging because this is not (yet) a single administrative category. More needs to be done in ensuring that information is complete across countries and recorded in comparable ways.
No country, except France, had well-organized information on competitive public health research programmes and calls. In France, a national system for collecting information about public health research calls and projects, GIS-IReSP, has been operating within the national research council Inserm (French Institute for Health and Medical Research) since 2004. 19 This database draws on 28 funding organizations, including national, regional and local, and across public, charity and commercial funders. Other larger countries with significant programmes, e.g. the UK and the Netherlands, do not bring the various programmes and calls together in one place.
Several possible classifications to describe thematic areas of competitive public health programmes and calls were considered. A Health Research Classification System has been adopted by the European Science Foundation. 20 A European mapping project, RICHE (Research into Child Health in Europe), 21 has created taxonomies for child health research. Classifications of public health activities include EUPHID indicators 22 and health promotion functions in Australia. 23 In SPHERE, priorities for public health research were mapped from perspectives of ministries of health, national public health associations and 80 civil society health organizations. 24 GIS-IReSP 19 has important listings of public health research in French. HR4E 25 mapped research projects in life sciences, genomics and biotechnology for health that were funded by the European Union's Fifth and Sixth Framework Research Programmes, and allocated them across 47 categories. National reviews of health research publications in Sweden 26 and Ireland 27 have used a classification of 30 categories. A further definition has been proposed in a report on public health research in preparation for Horizon 2020. 28 None of these matched the needs of PHIRE to describe health research programmes and calls. The objective was not to create a taxonomy on public health research, but to be able to classify the public health research calls at national level that were collected. All the subjects in the classification would cover at least one call: so that, even if a category would be logical, it would not be included if no call existed. Since a call might include many aspects, a main subject of public health research should be chosen and then classified accordingly: a programme or call was included in only one category.
Further development is also needed in describing public health research systems. The World Health Organization 9 suggested a conceptual framework covering five areas including Governance and Management (leadership, vision, advocacy, standards, ethics); Financing (allocation consistent with research priorities); Knowledge generation (i.e. research work); Use of Knowledge (with policymakers and practitioners); and Capacity Development (human, physical, intellectual). The COHRED created a questionnaire for reporting on national health research systems, 29 but not for Europe. PHIRE drew from the database of reports on public health research systems that were created for STEPS, and were revised with assistance of national public health associations. In our assessment of public health research systems, more attention must be given to the contributions of Ministries of Health and independent foundations-a diversity of funding is beneficial in increasing levels of public health research.

Conclusion
PHIRE's three objectives were to demonstrate uptake of public health innovations, develop understanding of public health research systems, and engage public health experts, associations and policymakers across Europe in these issues. The study's structure of work packages followed the project framework of the European Commission Public Health Programme.
Improved concepts and methodologies are needed for collecting systematic information on national uptake and impact of public health innovations, which could contribute to future evaluationboth of public health at national level and of the European Union's Public Health Programme.
Similarly, development is also needed on classifications of public health research programmes and calls, improving completeness of data on research systems at country level, and sensitizing national bodies to promote public health research for the coming European research framework programme 'Horizon 2020' and the European Research Area. 30 European Journal of Public Health