Abstract
Decisions about how to organize and deliver health services are often more complex and seemingly less rational than decisions about what clinical care to provide. The concept of ‘Evidence-Based Management’, or what might more appropriately be termed ‘Evidence-Informed Improvement’, does not seem to have captured the hearts and minds of the people responsible for managing health-care provision. Organizational decision-making is more likely to be influenced by political, ideological and pragmatic factors, and by the personal experience of the decision-makers, than by science. Whilst some people would regard the messiness of management decision-making as inevitable, most would accept that decisions could be improved by making greater use of the established health service research evidence, and through a stronger commitment to developing new evidence. Over the last two or more decades the evidence base created by Health Service Researchers has grown in quantity and in quality and yet much of it remains invisible to the people who most need to use it. This paper explores how the disconnect between the traditional ‘producers’ of research evidence in academia, and the managerial and clinical ‘consumers’ of that evidence, has contributed to the challenge of embedding an evidence-informed approach to service improvement. The advantages of a closer working relationship between academia and health services are outlined and three approaches to evidence creation and utilization are described which attempt to maximize the influence of scientific evidence on managerial practice.
Introduction
Decisions about how to organize and deliver health services are often more complex and seemingly less rational than decisions about what clinical care to provide. The concept of ‘Evidence-Based Management’, or what might more appropriately be termed ‘Evidence-Informed Improvement’, does not seem to have captured the hearts and minds of the people responsible for managing health-care provision [1]. Organizational decision-making is more likely to be influenced by political, ideological and pragmatic factors, and by the personal experience of the decision-makers, than by science. Whilst some people would regard the messiness of management decision-making as inevitable, most would accept that decisions could be improved by making greater use of the established health service research evidence, and through a stronger commitment to developing new evidence [2].
Over the last two or more decades the evidence base created by Health Service Researchers has grown in quantity and in quality and yet much of it remains invisible to the people who most need to use it. This paper explores how the disconnect between the traditional ‘producers’ of research evidence in academia, and the managerial and clinical ‘consumers’ of that evidence, has contributed to the challenge of embedding an evidence-informed approach to service improvement. The advantages of a closer working relationship between academia and health services are outlined and three approaches to evidence creation and utilization are described which attempt to maximize the influence of scientific evidence on managerial practice.
The different worlds of the ivory towers and the swampy lowlands
The disconnect between academics and practitioners is not a new phenomenon. One hundred years ago a commentator reflected that ‘the scientific man has become too scientific and the practical man too practical and the result has been unfortunate for both’ [3]. More than 50 years ago calls were made for ‘no research without action, no action without research’ [4] and more recently the concept of ‘engaged scholarship’ has gained traction [5]. Research funding agencies in many countries, including the Agency for Health Research and Quality in the USA [6] and the National Institute for Health Research in the UK [7], are making a significant investment in applied health services research, the promotion of new partnerships between academia and health services and in knowledge mobilization. And yet research still has little influence on practice.
The challenge that this paper addresses is easily satirized. Academics are often criticized for occupying ivory towers, places described in Wikipedia as ‘metaphysical spaces of solitude and sanctity where idealists engage in intellectual pursuits which are disconnected from the realities of everyday life’ [8]. Viewed from this rarefied position the problems experienced by front-line staff can be explained in a straightforward way with theory and solved with technical solutions. But when viewed from what Schon has called the ‘swampy lowlands’ of front-line practice, the same problems become more difficult to describe and less easy to solve in rational ways [9]. An impasse is reached where the two world views remain far apart; practitioners—both clinicians and managers—are criticized for failing to use research evidence, and academics are criticized for pontificating from on high and for ignoring or misunderstanding the messy nature of practice.
A potential solution: co-creation of knowledge
The most commonly used approaches to creating and utilizing research evidence do not seem to have addressed the disconnect between the ivory towers and the swampy lowlands [10, 11]. The ‘push’ model, led by the producers of research evidence, is the traditional approach used in the health sector and can be effective when the evidence is largely uncontested, easily described and lands on fertile soil, but this is rarely the case. The ‘pull’ model, led by the users of the research, can be effective if users are knowledgeable, committed and resist the temptation to draw selectively on the available evidence but again this is often not the case.
As an alternative to push and pull approaches, the so-called the ‘co-creation’ model seems to many commentators to have the greatest potential to close the ‘know-do gap’ [12–14]. The concept of co-creation has been of great interest to the academic community for many years. It is based on an understanding that different ways of thinking about the nature of knowledge explain disagreements about how it might be created and used. Knowledge, of which evidence is one specific type, can be conceptualized in a technical way, as data, something that is explicit and represents the ‘truth’. But advocates of co-creation are more likely to conceptualize it as something that is tacit, value laden and constructed from social interaction and negotiation. Thinking about knowledge in this way determines that new knowledge is best created through dialogue and in partnership [15].
There are several reasons for the failure to date to get to a place where researchers produce more useful research and practitioners utilize health services research more effectively. First, it may just be a question of time. Interest in the utilization of research in managerial practice is relatively new and there was a gap of more than two decades between Cochrane's call for research to have greater influence over clinical practice [16] and the advent of the evidence-based medicine movement [17]. Secondly, most health service managers responsible for decisions about how care is organized and delivered have not been trained in the scientific tradition and are more likely to conceive the term ‘evidence’ in experiential than empirical terms [1]. Thirdly, the pace and nature of managerial decision-making is different from clinical decision-making and the nature of managerial evidence often means that it is less easy to apply in practice than clinical evidence [18]. Finally, and perhaps most importantly, neither the professional nor the organizational incentives for researchers to focus on impact, and practitioners to use research evidence, are strong.
The institutional and cultural barriers which prevent evidence informed improvement becoming the norm may be addressed in part by experimenting with practical approaches to breaking down the boundaries between research and practice. The following sections present three such approaches, Researchers in Residence, Science-Based Improvement and Rapid Response Evaluation Teams, and examples are described of their use in practice.
Researchers in residence
People with expert knowledge and skills tend to seek out, or find themselves in the presence of like-minded people and this process of socialization can diminish the ability of others to access this expertise [19]. Embedding researchers ‘in-residence’ with the people responsible for putting ideas into practice attempts to address this problem. The concept has a track record: Barnsley football club has a poet in residence, Heathrow airport has an artist in residence and the BBC recently appointed a writer in residence. A Researcher in Residence model was first introduced in the UK education sector in the late 1990s in an ambitious programme initiated by the Department for Education and made available to doctoral and post-doctoral academics funded by Research Councils UK and The Wellcome Trust [20, 21].***
The model has been slow to catch on in the health sector but there are some examples of academics working in close and sustained partnership with managers and clinicians in the health service. Paul Bate, a British anthropologist, spent some years first at Birmingham Heartlands and Solihull NHS Trust and then at University College London NHS Foundation Trust, working closely with the executive team helping to design and embed a clinically led management structure [22]. Martin Utley et al., operational researchers from the Clinical Operations Research Unit at University College London, have been working in partnership with clinicians and managers at Great Ormond Street Children's hospital to reduce the number of cancelled heart operations and increase the efficiency of the operating theatres. By bringing their mathematical modelling skills to bear they are having some considerable success in helping to address a longstanding problem at the hospital [23].
Researchers-in-Residence blur the traditional boundary between their expertise and that of the health service team by becoming an integral part of the team rather than external commentators. Front-line practitioners are often unsure at first, sceptical that the researcher will be nihilistic or slow down the decision-making process. But at the same time they understand the value of having someone in the team who is aware of the evidence in whatever field they are working, can help to interpret that evidence for the local context, who have a good theoretical understanding of models of achieving change in complex organizations, can help to use data more effectively and who can help evaluate whether something is working or not. The important feature of the model is that the Researcher-in-Residence brings a unique expertise to the table, respects the different body of expertise held by the managers and clinicians with whom they are working and is willing to actively negotiate a way of bringing these bodies of expertise together. Some academics are concerned that such an embedded model risks compromising scientific objectivity. This may be the case but is less of a problem if any potential trade-off between scientific objectivity and practical utility is made explicit in discussion between the partners.
Science-based service improvement
A second approach to co-creation, termed Science-based Service Improvement, attempts to address the criticism that traditional evaluation often ‘slows up’ service development. In this approach service improvement projects are able to progress at a pace driven by service need but all stages of the process, including design, implementation, embedment and spread as well as evaluation, are influenced by a research team. The evaluation in particular is designed in such a way as to achieve a balance between scientific rigour and a desire to reduce the burden on those delivering the work on the ground.
An example of this approach was carried out in an inner-city area in the east of London and Bristol, UK [24]. The health service team wanted to address a practical problem of how to improve the identification of victims of domestic violence and increase their referral to specialist community services for on-going support. Domestic violence is a major problem in parts of London, with studies suggesting that in some communities up to 70% of women are victims of significant abuse at some time in their lives. The health, social and economic implications are great and yet fewer than one in five victims are known to their primary care teams. A team of general practitioners, social workers, practice nurses and researchers developed a training programme with technical and social elements for staff, some parts of which had been proved to be effective in other studies [25].
The improvement programme was implemented across 48 general practices and evaluated using a cluster randomized controlled trial. The trial was carried out in a highly pragmatic way, minimizing the burden of data collection on the participants, many of whom were largely unaware that their large-scale improvement project was being evaluated. The results were impressive; the identification of new cases of domestic violence increased by 3-fold in comparison with controls and the referral rate to specialist services by 22-fold. The results were published in The Lancet and are being used to persuade a growing number of commissioning organizations across England to implement the model in their areas, a rare example of research rapidly influencing practice.
Rapid response evaluation teams
A third approach to co-creation, Rapid Response Evaluation Teams, attempts to address the problem of the shortage of evaluation expertise available to front-line staff. Realistically, it is only possible to evaluate a small number of improvement projects at scale in a conventional academic way, and there are therefore many improvement projects which are not subject to independent assessment [26]. The consequence is that there is little unbiased understanding of whether or how most improvement projects make a difference [2] and if they do make a difference then how that learning can be shared with others. It is not uncommon for practitioners to approach academic teams with requests to evaluate a project but too often the intervention has already started and there is no budget for the work; they are usually given short shrift.
There will always be limited capacity in the academic sector and it would not make sense to do large-scale independent evaluations of every service project. But there is growing interest in the concept of a Rapid Response Evaluation Team, a group of pragmatic researchers who could meet the practical needs of the service. They would go into an organization at relatively short notice, work with the service team using readily available data or data that is easily collected. The focus would be less on proving whether an intervention worked and more on providing new insights which help to improve the effectiveness of the project. Such work is less likely to be published in high impact peer-reviewed journals but more likely to be useful to front-line teams than much of the work that is currently of high scientific repute.
This approach is currently being used to evaluate a novel intervention in a large hospital in the South East of the UK, which has been the subject of considerable regulatory scrutiny. The idea was to place 24 doctors and midwives as Clinical Improvement Fellows into the hospital, most of whom had finished their specialist training but not yet been appointed to substantive senior posts. All of the fellows had been formally trained in improvement methods and they were paired with established medical consultants or midwives already working in the hospital. The fellows undertook half time clinical work, freeing up some of the time of their ‘paired’ colleagues, and spent the other half of their time doing systems improvement projects, again working and learning with their paired colleagues and other members of their teams. The model warranted formal evaluation but when the tender was advertised, the project had already started and only a small budget was available and there was little interest from the university sector.
A small evaluation team was therefore established with membership from one local university and a not-for-profit research organization. Using established theories of change, a pragmatic adaptation of realist evaluation methodology, and a variety of data sources, useful insights into the advantages and challenges associated with the model have emerged and been fed back to the front-line teams (unpublished work). In this example the evaluation team was established from scratch but it is possible that a large provider, or network of providers, might want to establish a small standing resource to undertake this kind of work, employing their own researchers or building a consultancy model with experienced university-based academics.
Common characteristics of ‘useful’ science
The three approaches described in this paper share a number of features which characterize an approach to the science of improvement, breaking down the barriers between the ivory towers of academia and the swampy lowlands of health services and attempting to achieve a balance between impact and scientific excellence. First, the approaches create new evidence in ways which can only be done in close partnership between decision-makers in the health service, researchers and those who use services. Secondly, they take place in or close to the front line of routine practice and aim explicitly to make a timely difference to patient care. And thirdly, they aim to produce generalizable or transferable knowledge by drawing on a wide range of disciplines, sectors, theories and methods, applied in rigorous but pragmatic ways.
In addition to these functional features they also reflect a more philosophical one. The scientific method is rightly regarded as the most rigorous and reproducible way of creating knowledge and in the medical world it is perceived as the dominant way of acquiring knowledge. But what is clear when we look at decision-making in the swampy lowlands of managerial practice is that people use other ways of knowing, including intuition, rationalism, personal experience and authority as much and sometimes more than they use science. This is a reality which is often criticized or ignored by scientists and perhaps lies at the heart of the difference between the worlds of the ivory towers and the swampy lowlands. It is possible that greater success in embedding Evidence-Informed Improvement would be achieved if this reality were recognized and the potential synergies between different ways of knowing were utilized rather than ignored.
Discussion
This paper is based on the hypotheses that managerial decisions would be improved by making greater use of research evidence and that good Health Services Research is having insufficient impact on managerial practice. Current approaches to ‘pushing’ or ‘pulling’ evidence are not adequately addressing the fundamental problem of a disconnect between two very different worlds. It postulates that approaches based on co-creation of knowledge, explicitly recognizing the tensions that sometimes exist between scientific rigour and utility, are more likely to result in research evidence being used to improve the quality of managerial decisions. The approaches described are being tested in practice and need to be further developed and evaluated.
Pragmatic approaches to co-creating knowledge based on embedding researchers within practice are clearly not a replacement for more objective established approaches to evaluating health service change. But the established models of health service research leave a number of important gaps which can be filled by the approaches described in this paper—smaller in scale, more affordable, more likely to engage front-line staff and decision-makers and more focused on utility. These characteristics are aligned to the working practices and incentives of people working in health services, but less so to those working in the academic sector. Whilst some academics, particularly those earlier and later in their careers, like the idea of being more immediately useful to front-line decision-makers, the reward systems in most academic institutions have not in the past encouraged the kinds of approaches described in this paper. However, there are signs that this is changing, for example with the advent of new organizational structures supporting academic service partnerships in the UK [27, 28] and the introduction of promotion criteria based on service improvements in several leading North American universities [29].
The long-established disconnect between the ivory towers of academia and the swampy lowland of the health services has deep historical and cultural roots. In an environment which is asking hard questions about the usefulness and impact of all institutions, the potential of addressing this disconnect is being recognized. A stronger commitment to embedding academic expertise in the front line of clinical care is likely to benefit both sectors and, most importantly, the people who use health services.
