- Split View
-
Views
-
Cite
Cite
Sarah A Felknor, Paul A Schulte, Teresa M Schnorr, Regina Pana-Cryan, John Howard, Burden, Need and Impact: An Evidence-Based Method to Identify Worker Safety and Health Research Priorities, Annals of Work Exposures and Health, Volume 63, Issue 4, May 2019, Pages 375–385, https://doi.org/10.1093/annweh/wxz011
- Share Icon Share
Abstract
The importance of research and recommendations to address workforce safety and health derives from the continuing toll from worker fatalities, injuries, and illnesses. Estimates of the societal cost of work-related fatalities, injuries, and illnesses range up to $2.2 trillion in the USA from 2007 to 2015, which may be an underestimate of total societal costs. The ongoing changes in the nature of work, the workforce, and the workplace in the USA challenge old paradigms of worker safety and health research and require new decision criteria that are more solution oriented than observational and that result in interventions that can be readily applied to new occupational hazards and exposures. As public funding for science research programs becomes more constrained, and the demand for increased accountability of government spending grows, the need to demonstrate the impact or return on taxpayers’ investment becomes a necessity for research agencies. The National Institute for Occupational Safety and Health has developed an evidence-based method that uses the criteria of ‘burden’, ‘need’, and ‘impact’ to identify research priorities and aid in the evaluation of the taxpayers’ investment in research. This approach, named the BNI method, may be useful to other public and private sector research agencies or entities that need a systematic way to set research priorities and allocate increasingly scarce resources for research while ensuring the maximal return on investment.
Introduction
US federal agencies are required to establish priorities and measure performance against explicit criteria (US Congress, 1993, 2010). While these policies require agencies to establish strategic goals and performance measures, agencies have discretion over the methods they use to identify priorities. The National Institute for Occupational Safety and Health (NIOSH) was established in 1970 by the Occupational Safety and Health (OSH) Act as the primary federal research agency focused on worker safety and health with the mission to create new knowledge and transfer it into practice (US Congress, 1970). NIOSH is part of the US Centers for Disease Control and Prevention in the US Department of Health and Human Services and receives public funds to carry out its work.
A number of different approaches to priority setting in health research have been documented at the global, national, and local level. While the literature supports the notion that systematic and transparent methods are a useful tool to guide investments among research agencies and policy makers, no single approach works across the spectrum of health topics or focus areas (Rosenstock et al., 1998; Iavicoli et al., 2005; Viergever et al., 2010; Rehfuess et al., 2016; Mador et al., 2016; Yoshida, 2016). Supplementary Appendix 1, available at Annals of Occupational Hygiene online, summarizes these different approaches. The development of effective priority-setting approaches is dependent on the context in which research investments are made and the potential impact investments might have on the public good (Viergever et al., 2010).
The nature of work, the demographic composition of the workforce, and the places where work occurs in the USA continues to change and challenge traditional approaches to worker safety and health research. For example, work arrangements increasingly include temporary, contract, or ‘gig’ work arrangements alongside the traditional or standard one employer–one employee model, characterized by full-time employment protected by various labor laws including wage laws, workers’ compensation, and occupational safety and health protections. The workforce is increasingly older and more age, racial, and gender diverse, and work-life demands are increasing with the automation of some work processes and shortage of job security, creating new stresses on the workforce (Howard, 2017; Schulte et al., 2017a).
To respond to these challenges, NIOSH developed a systematic and transparent method to prioritize occupational safety and health research and allocate scarce research dollars that is based on the burden of occupational hazards, the need to conduct research to address the burden, and the potential impact or value that can be expected from the proposed research. The burden, need, and impact (BNI) method is structured such that the burden of workplace injury and illness drives investment and evaluation strategies that consider need and potential impact to do the most critical work on the most pressing issues.
Previous efforts at research prioritization
Evidence-based research priority setting strategies that include broad stakeholder input have been described as an effective way to build a prioritized research agenda (Rehfuess et al., 2016; Mador et al., 2016). National systems for identifying research priorities in occupational health have also been described. Many of these approaches rely on modifications of the Delphi technique that involve an iterative process of expert opinion that reaches ultimate consensus (Iavicoli et al., 2005).
NIOSH has a long history of using evidence to drive the identification of research priorities in worker safety and health (Perkins and Rose, 1979). The National Occupational Research Agenda (NORA) was launched by NIOSH in 1995 as a roadmap for occupational safety and health research for the nation that identified priority areas for research in partnership with stakeholder groups (Rosenstock et al., 1998; Howard, 2009). NORA has been continuously implemented in 10-year cycles since.
The first decade of NORA (1996–2006) provided a national agenda of occupational research priorities that reflected broad stakeholder input and expert opinion of relevant program area priorities (Rosenstock et al., 1998). NORA was organized into 21 focus areas that prioritized occupational safety and health research for the nation and NIOSH. NORA became a map by which the occupational safety and health community could identify, generate, design, and fund priority research efforts. By the time the first decade was launched, more than 500 individuals and organizations had contributed to the development of NORA. No previous occupational research agenda had captured such broad input.
The focus of the second decade of NORA (2006–2016) was to better move research into practice. NIOSH responded to that challenge with the identification of 10 industry-sector-based programs that would serve as the conduit to the working population in the USA. NIOSH also organized cross-sector programs to support sector program goals and priorities. Together, these programs contributed to the development of 90 strategic goals, 31 health outcome cross-sector goals, 80 additional cross-sector goals, and numerous sub-goals and objectives. This process resulted in a research program portfolio with more than 3000 goals that challenged previous prioritization methods.
The third decade of NORA (2016–2026) responded to the need to find an efficient and effective method to identify and integrate research priorities. The third decade of NORA (NORA 3) includes the 10 sector programs from the previous decade, organized by major areas of the US economy. Intersecting the sectors are cross-sector programs that are organized by the major health and safety issues affecting the US working population.
The BNI method was developed in NORA 3 to provide a strategic, structured, consistent, and transparent method to identify the highest research priorities and align funding decisions in a measurable, effective, and accountable manner.
The BNI method
The BNI method is an evidence-based approach to setting research priorities and aligning investment with research that has the greatest likelihood of significant impact to reduce the burden of worker injury and illness. NIOSH is using the BNI method within the framework of the sector and cross-sector program structure of NORA 3 to align research priorities and funding in a systematic and transparent manner.
Fig. 1 depicts the BNI method by which priority research goals are developed. Burden (actual or potential) identifies the most important health and safety issues to address by considering the evidence of exposure/hazard, injury/illness, disability/severity, and cost. Need provides evidence of the knowledge gap that needs to be addressed, consideration of the most appropriate methodological approach to address the need, the time fit for conducting that research at this point in time, the particular advantage NIOSH has to do the work, and the explicit stakeholder need. Impact identifies research with the greatest likelihood of reducing burden, potential for results to be used or disseminated by others, and the likelihood the research will generate knowledge that leads to follow-on research.
While the constructs of burden, need, and impact have always been considered by researchers, the BNI method provides a clear and systematic approach that is useful at both program and project level. At the project level, it formalizes thinking that investigators have long done. At the program level, it brings a new way of considering priorities and allocating resources.
Burden
Burden may be defined as risks from exposure to work-related hazards; occurrence of injuries, illnesses, and deaths due to work-related factors; and broad economic and social impacts including well-being (Schulte et al., 2017a). The assessment of burden is based on several main constructs: magnitude of the problem (such as the number or rate of cases); health impact severity; exposure to workers (such as number exposed, severity of exposure or both); societal costs; new or emerging issues; and relationship to work environment. For emerging issues, the burden will be anticipatory. Researchers should estimate the potential burden using the same parameters for existing burden (exposure/hazard, injury/illness, disability/severity, and cost), identify any assumptions, and provide a rationale for extrapolating potential burden to the population at risk.
Need
As with burden, need is a multifactorial concept. Need provides the rationale for NIOSH to conduct research to address high burden at a specific point in time. Need considers the evidence of the knowledge gap to be addressed and the appropriate methodological approach needed to address the burden (such as etiologic, intervention, or translational research). Need helps determine whether NIOSH is the most appropriate organization to do the work. Factors such as intellectual and financial capital, statutory authority, and mission relevance are a few considerations. Need also considers whether there is evidence of an explicit stakeholder need and why NIOSH should address the need.
Impact
Impact is an estimation of the potential for the research to positively affect worker health and safety on the basis of evident or anticipated results of the proposed research. Potential impact is expressed as potential reduction in burden that is likely to occur if the information from the proposed research is utilized in interventions or further research. The assessment of impact is based on these factors: the likelihood of the research to reduce burden or lead to plausible future actions to reduce burden; the use or dissemination of research results by others to set standards, policy, or guidance, or use by stakeholders to adopt results or use technology or methods developed to reduce burden; and the likelihood the research will generate knowledge that leads to follow-on research that builds on findings.
The criteria used to define burden, need, and impact at the program level and the individual project level are shown in Table 1. Additional review criteria are provided to reviewers to help assess the relative strength of proposals. The BNI criteria have been used to select intramural research projects since 2016 and will be used in the review of extramural research beginning in FY2019.
Burden criteria . | Need criteria . | Impact criteria . |
---|---|---|
Exposure/hazard. How many workers are exposed or at risk? Are there disparities among worker populations? Is there a trend in exposure or risk? Does a new or emerging burden show an increasing trend? Does the project address a high burden in a specific geographic region? Injury/illness. How many fatalities or illnesses have occurred? What is the incidence or prevalence of the injury or illness? Are there disparities among worker populations? Disability/severity. How serious is the health outcome under study? Is there evidence of disability, years of life lost or disabled, reduction in quality of life, or days away from work? Have the most relevant indicators been selected? Cost. What is the estimated cost such as medical expenses), productivity loss (such as absenteeism or presenteeism), lost wages, or disability payments? Have the most relevant indicators been selected? Additional review criteria for burden. Is there a general statement of the burden the proposed research will address? Is this an emerging issue? Does the proposed research address high overall burden in small population with large downstream impact? Does the proposed research address a burden in a particular sector or cross-sector that would have a significant economic consequence if that burden is not addressed? Have the most relevant indicators been selected? Are relevant comparison population estimates of burden provided to interpret the severity of burden to be addressed (e.g. overall numbers of injuries, illnesses, and deaths)? | Evidence of knowledge gap. Is there evidence that this activity will address a knowledge gap? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? Methodological approach. Is the proposed research method well defined (basic/ etiologic, intervention, translation, or surveillance) and appropriate to the proposed aims of the project? How does this methodological approach compare with other approaches that could be considered to fill the knowledge gap? Is the proposed research feasible and likely to address the stated need? Feasibility includes available capacity, resources, and technical expertise, as well as anticipated time to completion. If relevant, are letters of support from collaborators, stakeholders, or critical data sources provided? Is a summary provided of the research strategy strengths and weaknesses? Time fit. Is this the best time for undertaking this activity? NIOSH advantage. Is NIOSH ideally suited for this activity? Does the activity require NIOSH expertise or facilities, take advantage of a NIOSH partnership or relationships, or require neutrality or NIOSH convening authority? What strengths or unique advantages does NIOSH have in comparison with another organization that could undertake this activity? Stakeholder need. Does this study address an explicit stakeholder need? What is the evidence of that need and why should NIOSH address the need? What are the potential advantages of NIOSH undertaking this activity, in the broad context of research, policy, and practice? Additional review criteria for need. Is there a general statement of need? What is the rationale for the proposed work to be conducted by NIOSH at this point in time? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? | Likelihood of research to reduce burden. If the proposed work is successful as described, are the results likely to reduce the stated burden on the study population? For emerging issues or basic etiologic research, will the planned results likely lead to plausible future actions to reduce burden? Use or dissemination of research results by others. Is there potential for the proposed research to be used in setting standards, guidance, policy, or recommendations? Could the proposed research be adopted by employers, trade associations, professional organizations, or others? Is there potential for dissemination of research results by external organizations? Is there potential for others to adopt technology, training programs/materials, intervention strategies, or new surveillance methods used in or resulting from the proposed research? Is there a potential for technology to be transferred into the marketplace? Is there potential for partners to assist in tracking progress of research translation efforts? Have relevant partnerships needed to set the stage for research impact been described and documented? Follow-on research. Is the proposed research likely to generate information that leads to follow-on research that builds on the findings from this project? What type of follow-on research would be anticipated? Additional review criteria for impact. Is there a general statement of impact? Has the researcher expressed the potential reduction in burden that is likely to result if the proposed research is successful? |
Burden criteria . | Need criteria . | Impact criteria . |
---|---|---|
Exposure/hazard. How many workers are exposed or at risk? Are there disparities among worker populations? Is there a trend in exposure or risk? Does a new or emerging burden show an increasing trend? Does the project address a high burden in a specific geographic region? Injury/illness. How many fatalities or illnesses have occurred? What is the incidence or prevalence of the injury or illness? Are there disparities among worker populations? Disability/severity. How serious is the health outcome under study? Is there evidence of disability, years of life lost or disabled, reduction in quality of life, or days away from work? Have the most relevant indicators been selected? Cost. What is the estimated cost such as medical expenses), productivity loss (such as absenteeism or presenteeism), lost wages, or disability payments? Have the most relevant indicators been selected? Additional review criteria for burden. Is there a general statement of the burden the proposed research will address? Is this an emerging issue? Does the proposed research address high overall burden in small population with large downstream impact? Does the proposed research address a burden in a particular sector or cross-sector that would have a significant economic consequence if that burden is not addressed? Have the most relevant indicators been selected? Are relevant comparison population estimates of burden provided to interpret the severity of burden to be addressed (e.g. overall numbers of injuries, illnesses, and deaths)? | Evidence of knowledge gap. Is there evidence that this activity will address a knowledge gap? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? Methodological approach. Is the proposed research method well defined (basic/ etiologic, intervention, translation, or surveillance) and appropriate to the proposed aims of the project? How does this methodological approach compare with other approaches that could be considered to fill the knowledge gap? Is the proposed research feasible and likely to address the stated need? Feasibility includes available capacity, resources, and technical expertise, as well as anticipated time to completion. If relevant, are letters of support from collaborators, stakeholders, or critical data sources provided? Is a summary provided of the research strategy strengths and weaknesses? Time fit. Is this the best time for undertaking this activity? NIOSH advantage. Is NIOSH ideally suited for this activity? Does the activity require NIOSH expertise or facilities, take advantage of a NIOSH partnership or relationships, or require neutrality or NIOSH convening authority? What strengths or unique advantages does NIOSH have in comparison with another organization that could undertake this activity? Stakeholder need. Does this study address an explicit stakeholder need? What is the evidence of that need and why should NIOSH address the need? What are the potential advantages of NIOSH undertaking this activity, in the broad context of research, policy, and practice? Additional review criteria for need. Is there a general statement of need? What is the rationale for the proposed work to be conducted by NIOSH at this point in time? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? | Likelihood of research to reduce burden. If the proposed work is successful as described, are the results likely to reduce the stated burden on the study population? For emerging issues or basic etiologic research, will the planned results likely lead to plausible future actions to reduce burden? Use or dissemination of research results by others. Is there potential for the proposed research to be used in setting standards, guidance, policy, or recommendations? Could the proposed research be adopted by employers, trade associations, professional organizations, or others? Is there potential for dissemination of research results by external organizations? Is there potential for others to adopt technology, training programs/materials, intervention strategies, or new surveillance methods used in or resulting from the proposed research? Is there a potential for technology to be transferred into the marketplace? Is there potential for partners to assist in tracking progress of research translation efforts? Have relevant partnerships needed to set the stage for research impact been described and documented? Follow-on research. Is the proposed research likely to generate information that leads to follow-on research that builds on the findings from this project? What type of follow-on research would be anticipated? Additional review criteria for impact. Is there a general statement of impact? Has the researcher expressed the potential reduction in burden that is likely to result if the proposed research is successful? |
Burden criteria . | Need criteria . | Impact criteria . |
---|---|---|
Exposure/hazard. How many workers are exposed or at risk? Are there disparities among worker populations? Is there a trend in exposure or risk? Does a new or emerging burden show an increasing trend? Does the project address a high burden in a specific geographic region? Injury/illness. How many fatalities or illnesses have occurred? What is the incidence or prevalence of the injury or illness? Are there disparities among worker populations? Disability/severity. How serious is the health outcome under study? Is there evidence of disability, years of life lost or disabled, reduction in quality of life, or days away from work? Have the most relevant indicators been selected? Cost. What is the estimated cost such as medical expenses), productivity loss (such as absenteeism or presenteeism), lost wages, or disability payments? Have the most relevant indicators been selected? Additional review criteria for burden. Is there a general statement of the burden the proposed research will address? Is this an emerging issue? Does the proposed research address high overall burden in small population with large downstream impact? Does the proposed research address a burden in a particular sector or cross-sector that would have a significant economic consequence if that burden is not addressed? Have the most relevant indicators been selected? Are relevant comparison population estimates of burden provided to interpret the severity of burden to be addressed (e.g. overall numbers of injuries, illnesses, and deaths)? | Evidence of knowledge gap. Is there evidence that this activity will address a knowledge gap? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? Methodological approach. Is the proposed research method well defined (basic/ etiologic, intervention, translation, or surveillance) and appropriate to the proposed aims of the project? How does this methodological approach compare with other approaches that could be considered to fill the knowledge gap? Is the proposed research feasible and likely to address the stated need? Feasibility includes available capacity, resources, and technical expertise, as well as anticipated time to completion. If relevant, are letters of support from collaborators, stakeholders, or critical data sources provided? Is a summary provided of the research strategy strengths and weaknesses? Time fit. Is this the best time for undertaking this activity? NIOSH advantage. Is NIOSH ideally suited for this activity? Does the activity require NIOSH expertise or facilities, take advantage of a NIOSH partnership or relationships, or require neutrality or NIOSH convening authority? What strengths or unique advantages does NIOSH have in comparison with another organization that could undertake this activity? Stakeholder need. Does this study address an explicit stakeholder need? What is the evidence of that need and why should NIOSH address the need? What are the potential advantages of NIOSH undertaking this activity, in the broad context of research, policy, and practice? Additional review criteria for need. Is there a general statement of need? What is the rationale for the proposed work to be conducted by NIOSH at this point in time? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? | Likelihood of research to reduce burden. If the proposed work is successful as described, are the results likely to reduce the stated burden on the study population? For emerging issues or basic etiologic research, will the planned results likely lead to plausible future actions to reduce burden? Use or dissemination of research results by others. Is there potential for the proposed research to be used in setting standards, guidance, policy, or recommendations? Could the proposed research be adopted by employers, trade associations, professional organizations, or others? Is there potential for dissemination of research results by external organizations? Is there potential for others to adopt technology, training programs/materials, intervention strategies, or new surveillance methods used in or resulting from the proposed research? Is there a potential for technology to be transferred into the marketplace? Is there potential for partners to assist in tracking progress of research translation efforts? Have relevant partnerships needed to set the stage for research impact been described and documented? Follow-on research. Is the proposed research likely to generate information that leads to follow-on research that builds on the findings from this project? What type of follow-on research would be anticipated? Additional review criteria for impact. Is there a general statement of impact? Has the researcher expressed the potential reduction in burden that is likely to result if the proposed research is successful? |
Burden criteria . | Need criteria . | Impact criteria . |
---|---|---|
Exposure/hazard. How many workers are exposed or at risk? Are there disparities among worker populations? Is there a trend in exposure or risk? Does a new or emerging burden show an increasing trend? Does the project address a high burden in a specific geographic region? Injury/illness. How many fatalities or illnesses have occurred? What is the incidence or prevalence of the injury or illness? Are there disparities among worker populations? Disability/severity. How serious is the health outcome under study? Is there evidence of disability, years of life lost or disabled, reduction in quality of life, or days away from work? Have the most relevant indicators been selected? Cost. What is the estimated cost such as medical expenses), productivity loss (such as absenteeism or presenteeism), lost wages, or disability payments? Have the most relevant indicators been selected? Additional review criteria for burden. Is there a general statement of the burden the proposed research will address? Is this an emerging issue? Does the proposed research address high overall burden in small population with large downstream impact? Does the proposed research address a burden in a particular sector or cross-sector that would have a significant economic consequence if that burden is not addressed? Have the most relevant indicators been selected? Are relevant comparison population estimates of burden provided to interpret the severity of burden to be addressed (e.g. overall numbers of injuries, illnesses, and deaths)? | Evidence of knowledge gap. Is there evidence that this activity will address a knowledge gap? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? Methodological approach. Is the proposed research method well defined (basic/ etiologic, intervention, translation, or surveillance) and appropriate to the proposed aims of the project? How does this methodological approach compare with other approaches that could be considered to fill the knowledge gap? Is the proposed research feasible and likely to address the stated need? Feasibility includes available capacity, resources, and technical expertise, as well as anticipated time to completion. If relevant, are letters of support from collaborators, stakeholders, or critical data sources provided? Is a summary provided of the research strategy strengths and weaknesses? Time fit. Is this the best time for undertaking this activity? NIOSH advantage. Is NIOSH ideally suited for this activity? Does the activity require NIOSH expertise or facilities, take advantage of a NIOSH partnership or relationships, or require neutrality or NIOSH convening authority? What strengths or unique advantages does NIOSH have in comparison with another organization that could undertake this activity? Stakeholder need. Does this study address an explicit stakeholder need? What is the evidence of that need and why should NIOSH address the need? What are the potential advantages of NIOSH undertaking this activity, in the broad context of research, policy, and practice? Additional review criteria for need. Is there a general statement of need? What is the rationale for the proposed work to be conducted by NIOSH at this point in time? Does the proposed work build on previous or current NIOSH funded research (intramural or extramural)? Have the researchers addressed any overlap between previous or current research and this proposal? | Likelihood of research to reduce burden. If the proposed work is successful as described, are the results likely to reduce the stated burden on the study population? For emerging issues or basic etiologic research, will the planned results likely lead to plausible future actions to reduce burden? Use or dissemination of research results by others. Is there potential for the proposed research to be used in setting standards, guidance, policy, or recommendations? Could the proposed research be adopted by employers, trade associations, professional organizations, or others? Is there potential for dissemination of research results by external organizations? Is there potential for others to adopt technology, training programs/materials, intervention strategies, or new surveillance methods used in or resulting from the proposed research? Is there a potential for technology to be transferred into the marketplace? Is there potential for partners to assist in tracking progress of research translation efforts? Have relevant partnerships needed to set the stage for research impact been described and documented? Follow-on research. Is the proposed research likely to generate information that leads to follow-on research that builds on the findings from this project? What type of follow-on research would be anticipated? Additional review criteria for impact. Is there a general statement of impact? Has the researcher expressed the potential reduction in burden that is likely to result if the proposed research is successful? |
Implementing BNI
Since 2016, NIOSH has successfully implemented the BNI method at the program level to identify research priorities across programs and at the project level to select individual intramural research projects. In 2018, the BNI method was used to identify 74 priority research goals published in the NIOSH Strategic Plan: FY2019–2023 (NIOSH, 2018). The plan identifies strategic and intermediate goals for the NIOSH research portfolio. The strategic goals represent the major health and safety issues facing the US workforce and are the broad focus areas for research. They correspond to the cross-sector programs shown in Fig. 2.
Strategic goals
Reduce occupational cancer, cardiovascular disease, adverse reproductive outcomes, and other chronic diseases.
Reduce occupational hearing loss.
Reduce occupational immune, infectious, and dermal disease.
Reduce occupational musculoskeletal disorders.
Reduce occupational respiratory disease.
Improve workplace safety to reduce traumatic injuries.
Promote safe and healthy work design and well-being.
Intermediate goals
Intermediate goals describe the broad actions needed to achieve or help achieve the strategic goals and are represented in the ‘hot cells’ of the matrix in the Fig. 2 example. The intermediate goals further identify the health and safety outcome, the research focus area, the worker population, and the type of research needed to address these goals.
Also in 2018, NIOSH introduced the BNI criteria into the extramural space with the publication of new investigator-initiated research funding opportunity announcements that direct extramural researchers to address the priority goals published in the NIOSH Strategic Plan and address the BNI criteria in the Significance section of their research applications. The extramural funding announcements explicitly state that BNI criteria are to be addressed by researchers at the proposal stage and considered by reviewers at the review stage (National Institutes of Health (NIH), 2018).
From time to time, there may be other inputs into the research prioritization process that influence research priorities and funding decisions such as Congressional and Executive Branch mandates, formal program review recommendations, Federal Advisory Committee recommendations for new areas of research, and NIOSH Director’s discretion. These other inputs are considered as they arise and may be responded to with special funding opportunities.
Establishing program level priorities
In many agencies, multiple programs must compete for funding based on prioritized research goals. NIOSH has organized its research program into 10 industry sectors representing the major economic sectors in the USA, and seven health, safety and well-being cross-sectors representing the major health and safety issues among workers in the USA. These programs work together in an integrated approach to identify shared goals based on BNI criteria. At this stage in the research prioritization, programs consider primarily the burden to be addressed and the need to reduce the burden. Impact or potential impact is considered at the time projects are reviewed for funding.
Fig. 2 shows the matrix approach to OSH research prioritization between the NIOSH sector and cross-sector programs in NORA 3. This figure shows an example from the current NIOSH Strategic Plan of ‘hot cells’ where the musculoskeletal health program and several sector programs have identified the reduction of musculoskeletal disorders as a top priority based on a joint assessment of burden, need, and impact in each sector. This process is replicated in each cell of the matrix where all sector and cross-sector programs consider BNI and work collaboratively to identify top priority work.
Each cell of the matrix reflects interaction between competing programs for priority funding. Programs work together with subject matter experts to identify top burden areas and need for research. In this approach, when sectors, cross-sectors, and subject matter experts reach concurrence in their assessment of burden and need, a priority is established. Priorities are mapped into the cells of the matrix to reflect the integrated goals. The ‘hot cells’ in the matrix, where multiple program priorities intersect, become the foundation for the research strategic plan for the next cycle (5–10 years). Currently there are 47 ‘hot cells’ in the matrix with multiple program priorities with intermediate goals identified for every sector and cross-sector.
Selecting individual projects for funding
The BNI method has been successfully used at NIOSH to review intramural research proposals and select top-priority work for funding since 2016. Program subject matter experts review the rational for burden, need, and impact. Each program provides a merit score for burden, need, and impact using a 9-point scale for scientific merit (NIH, 2015). Projects are reviewed first by the relevant programs, providing individual scores for burden, need, and impact, which are averaged to determine a preliminary score [(B+N+I)/3]. NIOSH considered different weighting schemes for each factor and determined these three constructs are so equally fundamental that no weighting was used. The magnitude of each individual factor affects the final score. Intramural research projects are then reviewed and discussed by the NIOSH Secondary Review Committee (SRC), made up of senior leadership from diverse program areas. The SRC considers the program review and, after discussion, each SRC member provides an overall score, which is averaged to determine the final overall score. This protocol is modeled on the NIH Study Section peer review process (NIH, 2017).
Intramural projects that receive a strong overall score are recommended for funding. All projects must go through additional scientific peer-review to ensure that the research methodology is of appropriate rigor. Proposals selected for funding that do not meet acceptable standards by peer-review can be turned down or revised to assure that the best quality work is funded.
Intramural researchers are required to submit proposals that address the Institute priorities defined in the NIOSH Strategic Plan: FY2019–FY2023, which establishes the roadmap for solving the most pressing OSH problems in the major economic sectors in the USA. Beginning in 2018, extramural researchers are now directed to address the same research priorities, and funding priority will be given to those projects that address priority goals. Extramural researchers who choose to address research goals outside of the matrix of priorities must provide compelling evidence of burden, need, and potential for impact that would support consideration of their application.
Discussion
Early results
The BNI method was conceived of to develop a systematic and transparent method by which NIOSH could strategically drive OSH research for the nation with a clear set of attainable evidence-based prioritized research goals to ensure the best use of limited public funds. The second decade of NORA ended with 90 strategic goals, 31 health outcome goals, 80 additional goals for a total of over 3000 goals that challenged efforts to strategically align and prioritize OSH research and evaluation efforts. NIOSH needed a new approach that would result in a more strategic system of setting priorities that could be clearly articulated and integrated in the intramural and extramural communities of researchers.
The BNI method has provided a systematic approach to research prioritization in the third decade of NORA that has been used to develop the NIOSH Strategic Plan: 2019–2023, with 7 strategic goals and 64 priority goals that are more focused and clearly aligned to address the most pressing OSH issues (NIOSH, 2018). The effect of the BNI method on research prioritization in the intramural research competition is show in Table 2.
Research prioritization process . | Before BNI method . | After BNI method . |
---|---|---|
Research goals | Each NIOSH program established unique research goals. | All NIOSH programs collaborate on the development of shared priority research goals in a focused matrix approach of OSH health and safety outcomes and major economic sectors in the USA. |
Research priorities | Individual programs identified research priorities based on different inputs. Additional emphasis areas were identified by NIOSH and prioritized separately from program priorities. | All programs work collaboratively in a matrix approach to identify priorities based on a systematic and transparent process with clear criteria shared by all to identify burden, need, and impact. |
Research competition (intramural) | Two levels of review • Individual programs selected top projects based on program criteria and determined which projects could compete for funding. o Review and scoring criteria were not consistent across programs. • Secondary Review Committee assessed programmatic relevance without explicit criteria. o Emphasis areas were given priority. | Two levels of review • Programs provide a consensus review and score for all projects based on BNI criteria. o All projects reviewed and scored. • Secondary Review Committee reviews projects and program reviews to provide a final overall score based on BNI criteria. |
Research funding | Funding recommendations based on emphasis areas first, followed by best scoring projects, as funds allowed. | Funding recommendations based on BNI overall score, as funds allow. |
Research prioritization process . | Before BNI method . | After BNI method . |
---|---|---|
Research goals | Each NIOSH program established unique research goals. | All NIOSH programs collaborate on the development of shared priority research goals in a focused matrix approach of OSH health and safety outcomes and major economic sectors in the USA. |
Research priorities | Individual programs identified research priorities based on different inputs. Additional emphasis areas were identified by NIOSH and prioritized separately from program priorities. | All programs work collaboratively in a matrix approach to identify priorities based on a systematic and transparent process with clear criteria shared by all to identify burden, need, and impact. |
Research competition (intramural) | Two levels of review • Individual programs selected top projects based on program criteria and determined which projects could compete for funding. o Review and scoring criteria were not consistent across programs. • Secondary Review Committee assessed programmatic relevance without explicit criteria. o Emphasis areas were given priority. | Two levels of review • Programs provide a consensus review and score for all projects based on BNI criteria. o All projects reviewed and scored. • Secondary Review Committee reviews projects and program reviews to provide a final overall score based on BNI criteria. |
Research funding | Funding recommendations based on emphasis areas first, followed by best scoring projects, as funds allowed. | Funding recommendations based on BNI overall score, as funds allow. |
Research prioritization process . | Before BNI method . | After BNI method . |
---|---|---|
Research goals | Each NIOSH program established unique research goals. | All NIOSH programs collaborate on the development of shared priority research goals in a focused matrix approach of OSH health and safety outcomes and major economic sectors in the USA. |
Research priorities | Individual programs identified research priorities based on different inputs. Additional emphasis areas were identified by NIOSH and prioritized separately from program priorities. | All programs work collaboratively in a matrix approach to identify priorities based on a systematic and transparent process with clear criteria shared by all to identify burden, need, and impact. |
Research competition (intramural) | Two levels of review • Individual programs selected top projects based on program criteria and determined which projects could compete for funding. o Review and scoring criteria were not consistent across programs. • Secondary Review Committee assessed programmatic relevance without explicit criteria. o Emphasis areas were given priority. | Two levels of review • Programs provide a consensus review and score for all projects based on BNI criteria. o All projects reviewed and scored. • Secondary Review Committee reviews projects and program reviews to provide a final overall score based on BNI criteria. |
Research funding | Funding recommendations based on emphasis areas first, followed by best scoring projects, as funds allowed. | Funding recommendations based on BNI overall score, as funds allow. |
Research prioritization process . | Before BNI method . | After BNI method . |
---|---|---|
Research goals | Each NIOSH program established unique research goals. | All NIOSH programs collaborate on the development of shared priority research goals in a focused matrix approach of OSH health and safety outcomes and major economic sectors in the USA. |
Research priorities | Individual programs identified research priorities based on different inputs. Additional emphasis areas were identified by NIOSH and prioritized separately from program priorities. | All programs work collaboratively in a matrix approach to identify priorities based on a systematic and transparent process with clear criteria shared by all to identify burden, need, and impact. |
Research competition (intramural) | Two levels of review • Individual programs selected top projects based on program criteria and determined which projects could compete for funding. o Review and scoring criteria were not consistent across programs. • Secondary Review Committee assessed programmatic relevance without explicit criteria. o Emphasis areas were given priority. | Two levels of review • Programs provide a consensus review and score for all projects based on BNI criteria. o All projects reviewed and scored. • Secondary Review Committee reviews projects and program reviews to provide a final overall score based on BNI criteria. |
Research funding | Funding recommendations based on emphasis areas first, followed by best scoring projects, as funds allowed. | Funding recommendations based on BNI overall score, as funds allow. |
Evaluation of BNI
Evaluation of the BNI method to date has been largely process oriented and focused on the intramural research competition. The results of the first three cycles of implementation are shown in Table 3. These data show that under the BNI method, applications have become fewer and more focused while success rates increased, emerging issues are being addressed, the number of priority goals has been streamlined, a consistent number of these goals are being addressed by the annual intramural competition, and all sectors and cross-sectors have priority goals that are being addressed. Satisfaction surveys of NIOSH researchers and reviewers show an overall increase with their satisfaction with the BNI method.
. | 2016 . | 2017 . | 2018 . |
---|---|---|---|
Number of intramural applications | 55 | 36 | 35 |
Selected intramural projects | 23 (42%) | 25 (69%) | 20 (57%) |
Submitted applications addressing emerging issuesa | NA | 12 | 16 |
Selected projects addressing emerging issuesa | NA | 9 | 10 |
Number of priority research goals based on BNIb | 116 | 115 | 61 |
Number of priority goals addressed by selected projects | 35 (30%) | 37 (33%) | 22 (36%) |
Number of sector programs in selected projects | 10 | 10 | 10 |
Number of cross-sector programs in selected projects | 7 | 7 | 7 |
NIOSH researchers satisfied/very satisfied with BNIc | 49% | 38% | 64% |
NIOSH reviewers satisfied/very satisfied with BNIc | 66% | 75% | 74% |
. | 2016 . | 2017 . | 2018 . |
---|---|---|---|
Number of intramural applications | 55 | 36 | 35 |
Selected intramural projects | 23 (42%) | 25 (69%) | 20 (57%) |
Submitted applications addressing emerging issuesa | NA | 12 | 16 |
Selected projects addressing emerging issuesa | NA | 9 | 10 |
Number of priority research goals based on BNIb | 116 | 115 | 61 |
Number of priority goals addressed by selected projects | 35 (30%) | 37 (33%) | 22 (36%) |
Number of sector programs in selected projects | 10 | 10 | 10 |
Number of cross-sector programs in selected projects | 7 | 7 | 7 |
NIOSH researchers satisfied/very satisfied with BNIc | 49% | 38% | 64% |
NIOSH reviewers satisfied/very satisfied with BNIc | 66% | 75% | 74% |
aData available for 2017 and 2018 only. Emerging issues include emerging technologies, emerging burden, emerging hazards, emerging products, emerging industry and issues, and emerging workforce.
bBeginning in 2016, NIOSH programs worked in a matrix approach to integrated goals based on BNI.
cNIOSH researchers and reviewers were asked about their satisfaction with the review process based on the BNI method after each annual competition.
. | 2016 . | 2017 . | 2018 . |
---|---|---|---|
Number of intramural applications | 55 | 36 | 35 |
Selected intramural projects | 23 (42%) | 25 (69%) | 20 (57%) |
Submitted applications addressing emerging issuesa | NA | 12 | 16 |
Selected projects addressing emerging issuesa | NA | 9 | 10 |
Number of priority research goals based on BNIb | 116 | 115 | 61 |
Number of priority goals addressed by selected projects | 35 (30%) | 37 (33%) | 22 (36%) |
Number of sector programs in selected projects | 10 | 10 | 10 |
Number of cross-sector programs in selected projects | 7 | 7 | 7 |
NIOSH researchers satisfied/very satisfied with BNIc | 49% | 38% | 64% |
NIOSH reviewers satisfied/very satisfied with BNIc | 66% | 75% | 74% |
. | 2016 . | 2017 . | 2018 . |
---|---|---|---|
Number of intramural applications | 55 | 36 | 35 |
Selected intramural projects | 23 (42%) | 25 (69%) | 20 (57%) |
Submitted applications addressing emerging issuesa | NA | 12 | 16 |
Selected projects addressing emerging issuesa | NA | 9 | 10 |
Number of priority research goals based on BNIb | 116 | 115 | 61 |
Number of priority goals addressed by selected projects | 35 (30%) | 37 (33%) | 22 (36%) |
Number of sector programs in selected projects | 10 | 10 | 10 |
Number of cross-sector programs in selected projects | 7 | 7 | 7 |
NIOSH researchers satisfied/very satisfied with BNIc | 49% | 38% | 64% |
NIOSH reviewers satisfied/very satisfied with BNIc | 66% | 75% | 74% |
aData available for 2017 and 2018 only. Emerging issues include emerging technologies, emerging burden, emerging hazards, emerging products, emerging industry and issues, and emerging workforce.
bBeginning in 2016, NIOSH programs worked in a matrix approach to integrated goals based on BNI.
cNIOSH researchers and reviewers were asked about their satisfaction with the review process based on the BNI method after each annual competition.
At the end of each funding cycle, NIOSH reviews the projects selected to consider whether the equal weighting of burden, need, and impact has affected the priority order to the extent that more important projects were overlooked. To date, the projects selected for funding have been deemed appropriate.
NIOSH employs several formal evaluation methods to assess the impact of research and service activities during the decade cycles of NORA. The most recent second decade was reviewed and published in 2017 (NIOSH, 2017). Ongoing formal program evaluation efforts include a constellation of activities that assess project outputs and intermediate outcomes as well as the contribution NIOSH programs make to the achievement of end outcomes (Downes et al., 2018).
A logic model of the BNI method as it relates to the overarching goals of research prioritization and funding alignment are shown in Fig. 3. The BNI logic model guides the evaluation of BNI as an approach to aligning research priorities and funding. It is not a model for evaluating the impact of research projects. As the BNI method is implemented across the extramural research programs, additional evaluation efforts will be developed.
Limitations
There remain a number of issues to consider as the BNI method is implemented to test whether this is an efficient and effective research prioritization tool. Overall, the burden of occupational disease and injury is severely underestimated and understudied (Rosenman et al., 2006; Schulte et al., 2017a); however, what is known is a useful foundation upon which to assess the importance of proposed research.
There is concern that the BNI method might limit investigator creativity. The matrixed approach to goal development based on assessment of burden, need, and impact channels creativity and innovation to address the most pressing OSH issues faced by workers in the USA. It should be noted that these goals have several inputs other than NIOSH that include stakeholder input through NORA partnerships and other extramural partners. Consideration of this potential limitation must occur in the context of the absolute requirement that NIOSH serves as a good steward of limited public funds able to clearly justify the prioritization and allocation of research dollars, both intramurally and extramurally.
Work is needed to better compare burden across different subsectors of worker populations or health outcomes so that deliberate decisions can be made when there are high rates or prevalence of low-risk outcomes versus low rates of high-risk outcomes. For example, what weight should be placed on the relative importance of burden for high injury or illness rates in small worker populations or where there is evidence of significant disparities or high societal costs for these injuries and illness? Many burden estimates are based on old data that may not be relevant to contemporary scenarios. The success of using a burden determinant for research prioritization will depend on increasing surveillance and informatics capabilities, especially challenging the changing nature of work.
The issues of disparities in burden across worker populations needs further study. Disparities may exist because different industries have different exposures. However, disparities in burden among workers in the same work setting or industry that are disproportionately exposed to hazards are critical to any assessment of burden, need, and impact and needs more refinement in the BNI approach.
Assessment of need also has issues to address. Need must be considered in teleological terms; that is, to what extent does an individual project contribute to the ultimate reduction of burden? What other projects will be required and in what order? What is the best approach to address the burden? Should the agency invest more in etiologic research or is there a greater need to fil knowledge gaps in intervention or translation research?
The assessment of impact requires consideration of different time horizons. The impact of research is dependent on the extent to which it moves others (employers, other agencies, workers) to take action to protect worker safety, health, and well-being. For action to occur there is need for a research-to-practice (r2p) effort. There is also need to study r2p to identify best approaches, barriers, and tools for putting research into practice (Schulte et al., 2017b).
Overall, the BNI method is an explicit, evidence-based conceptualization that allows NIOSH to distinguish among many priorities the ones that it will focus on the most. Ultimately, it allows NIOSH to ensure that its use of public funds for research addresses important occupational safety and health problems and reflects a thoughtful investment of public funds in fulfilling a statutory mandate. The health and safety of workers and their productivity is a major determinant of the economic vitality of the Nation. BNI helps to enable NIOSH to contribute to improved worker health, safety, and well-being and, in doing so, contributes to securing the national interests.
Disclaimer
The findings and conclusions in this report are those of the author(s) and do not necessarily represent the official position of the National Institute for Occupational Safety and Health, Centers for Disease Control and Prevention.
Conflict of Interest
The authors are employed by the National Institute for Occupational Safety and Health (NIOSH). The authors designed and executed the study as part of their regular duties and have sole responsibility for the writing and content of the manuscript. The authors declare that they do not have a conflict of interest.
Acknowledgements
The authors thank the following members of the BNI Work Group at NIOSH who developed the core concepts of the BNI Method: Don Beezhold, Dawn Castillo, Maryann D’Alessandro, Kelley Durst, Lore Jackson Lee, Margaret Kitt, John Piacentino, and David Weissman. The authors thank Emily Novicki and the NIOSH Program, Policy, and Evaluation Office for their many contributions to the NIOSH strategic planning process. The authors thank the following for comments on earlier drafts of the manuscript: John Mendeloff, Kathy Rest, and Gregory Wagner.