The state of British policymaking: How can UK government become more effective?

Abstract

How can UK and devolved governments be more effective when addressing chronic problems like inequalities or crises like climate change? The dominant story is of pessimism: policymaking is bound to a Westminster tradition of short-termism, elitism, and centralization, and reform efforts are doomed to failure. We present a more cautiously optimistic account about the prospects for a more effective government, grounded in theory-informed lessons from two decades of UK and devolved government reform efforts. We describe a potentially more innovative and less blundering state and present a coherent Positive Public Policy agenda that can help to realize this potential.

1. Introduction

The dominant academic narrative of UK-wide government is of dysfunction and failure. It suggests that we should have profoundly low expectations for a new party entering UK government. The story goes as follows. In the last 25 years, UK and devolved governments have produced a repertoire of warm words to describe aspirational reforms, to make policymaking more strategic, place-based, evidence-based, participatory, or informed by behavioural insights. This rhetoric has not translated into substantive action. While UK and devolved governments recognize the promise of alternative ways of doing things in theory, they continue to prioritize in practice a narrow form of Westminster-style democratic accountability that skews policy attention and resources to short-term and centralized approaches. The result is that reform initiatives emerge to great fanfare, only to be forgotten or rejected when they struggle to fit into routine government business. While all four government experiences contribute to this narrative, the UK government receives the most frequent critical assessment, such as to describe systematic ‘governance failure’ or an ‘incoherent state’ (Gaskell et al. 2020; Richards et al. 2022).

This special issue relates this problem partly to the chronic features of UK government described by the asymmetric power model (APM) (Marsh et al. 2003, 2024). The APM states that inequalities in the economy and society are largely reflected in politics and government. It identifies UK governance problems in that context. Ineffectiveness relates to the mismatch between the ‘government knows best’ centralizing rhetoric of the ‘British Political Tradition’ and ‘Westminster Model’ on the one hand, and the fragmentation of decision-making and delivery on the other. In this sense, the APM helps to explain the protracted and blunder-prone effort to extricate Britain from the European Union and the reactive and inefficient response to the COVID-19 pandemic, as the machinery of government creaked and groaned in trying to deliver impossible promises and unworkable policies (Diamond 2023; Ward, Ward, and Kerr 2024).

This contrast between rhetoric and reality extends to government reforms. Scholars in this tradition show how ideas portrayed as rational and effective solutions to governing pathologies—such as transparency or participation—prove to be ad hoc and ineffective in practice, to the extent that they exacerbate the problems of governance that they profess to solve (Diamond 2013; compare with Bevir and Rhodes 2003).

We depart from this orthodoxy in two ways. First, we argue that key aspects of this contradictory policymaking dynamic are generalizable, and not specific to UK central government. The UK is a strong exemplar of the general limits to ‘effective government’ when there are fundamental contradictions between aspirations for reform. The search for effective government involves trade-offs between laudable aims, not the chance to find the holy grail of holistic reform. Second, we go beyond this UK political science focus on diagnosing repeatedly the chronic failures of UK politics and policymaking, which are already summarized in painful detail by Cairney & Kippin (2024). We explore the possibility that such insights can also inform a positive and practical research agenda. This practically focussed agenda is especially important for a new Labour UK government facing multiple crises.

To that end, we synthesize insights from policy studies to identify why many approaches to effective government seem so essential in theory but remain unfulfilled in practice. Then, we outline a Positive Public Policy (PoPP) approach to help shift the academic debate from explaining failure to identifying how to support success (inspired by Douglas et al. 2021). We use these negative and positive accounts to reconsider a series of approaches to effective government that have struggled to gain a foothold in the UK and devolved politics but could still represent coherent and feasible alternatives to the status quo. Of each approach, we ask: what exactly does it seek to do, how does it relate to current approaches to UK government, and what would it take to get it ‘over the line’ and renewed by a UK Labour government? Finally, we reflect on what these accounts of each aspiration tell us collectively about the potential for a coherent agenda for policymaking reform in the UK, in theory and practice.

2. Why do effective approaches in theory falter in practice?

UK political science describes UK government as uniquely or acutely dysfunctional. King and Crewe (2013) list a litany of historic ‘blunders’ and policy fiascos. More recently, we have witnessed scandals like the ‘Windrush betrayal’ (Gentleman 2019), the Post Office scandal, the ‘omnishambles’ of Brexit (Richardson and Rittberger 2020), and disastrous failures in responding to COVID (Cairney and Kippin 2024). Throughout, UK governments have made repeated statements of intent to make policy better (e.g. Cabinet Office 1999; HM Government 2012, 2022a), such as to enhance strategic capacity at the centre, devolve more responsibilities to allow place-based systems thinking, to be more preventive or participatory, and to draw better on evidence and new behavioural insights. However, the academic evidence suggests that little meaningful headway has been made (Cairney and Kippin 2024). Indeed, with the state of upheaval in Westminster—including the rapid churn of Prime Ministers and Ministerial teams, record-low public approval ratings, ongoing inflationary pressures, and crises in key public services—it often seems that things can only get worse (Richards et al. 2022).

Yet, this experience of frustration and failure is not a sign of British exceptionalism. A much wider and richer comparative literature points to policy fiascos across the world (Jennings, Lodge, and Ryan 2018). Aspirational initiatives to promote more effective government have been thwarted elsewhere. Indeed, a wealth of academic policy studies already help to explain the gap between abstract policymaker expectations and concrete reality. Cairney St. Denny, & Boswell (2022) synthesize insights from these studies to identify three broad reasons why ambitious reform programmes, to address problems such as profound inequalities, seem so promising in theory but make little headway in practice: they lack clarity, are not congruent with routine government business, and there is insufficient capacity to overcome obstacles to change. Here, we use these categories to explain a lack of progress with ‘effective government’ ambitions, to highlight the problems specific or not to the UK, and reflect broadly on UK and devolved government dilemmas. We use UK COVID-19 policy as an illustrative example for this initial discussion, since it exemplifies the portrayal of UK government blunders as symbolic of wide dysfunction (Gaskell et al. 2020).

2.1 A lack of clarity: Effective government can mean everything and therefore nothing

Governments can make rhetorical commitments to effective government because the ambition is akin to a valence issue: who would not want to make government more effective? The same can be said of many effective government principles: a lack of detail helps to generate political support and minimize opposition, but becomes unhelpful if there is insufficient consideration of what each principle means and how it contributes to a coherent overall story. For example, Cairney (2023) identifies many vaguely defined ‘principles’ that might be essential to a coherent narrative of effective government, including responsible and accountable; future-oriented; preventive; decentralized; co-productive; integrated; evidence-informed; and equitable. However, he finds no example of a coherent narrative that names each principle, defines it clearly, and shows how it is essential to effective government.

To some extent, UK and devolved governments have addressed the general problem of effective government by making some commitment to approaches such as ‘public value’ (Moore 1995) or design-led or systems-informed approaches that incorporate multiple principles (Cairney 2023). However, this language of reform, designed ostensibly to reconcile many effective government principles, is also vague and contested. Our overall impression is that governments are generally unable or unwilling to address the ambiguity of their ambitions. We may find sincere and committed reformers who have accepted the general normative case, seek practical ways to deliver their promises, but then face unanticipated trade-offs between aims and principles. We may also find less enthusiastic actors who exploit the ambiguous language of, and inevitable obstacles to, reforms because they are strategically convenient.

This general lack of clarity, regarding what principles are essential, what they mean, and how they fit together in an overall account of effective government, has played out time and time again in UK policymaking. For example, it characterized the UK government response to the COVID-19 pandemic (Gaskell et al. 2020; Boswell 2022; Diamond and Laffin 2022; Richards et al. 2022; Durose, Perry, and Richardson 2024). On the one hand, the Conservative UK government wanted a highly centralized and cocooned approach centred around a core executive, which informed the decision to strictly circumscribe sources of advisory input and centralize service delivery in the name of efficiency. On the other, it trumpeted the need for ‘maximum possible transparency’, devolved and local differentiation, and the activation of citizen and volunteer sector participation at a large scale. These aims made sense in isolation but did not fit together coherently, thus exacerbating confusion and duplication during the crisis.

2.2 A lack of congruence: ‘Effective government’ reforms do not fit well with current government commitments

Vague reform agendas struggle to compete with the more concrete and established language of government. Many organizations embrace the general language of long-termism and collaboration but still devote most attention and resources to their statutory commitments and more concrete performance standards that they are expected to meet. A lack of initial clarity exacerbates this problem since it downplays the major trade-offs or tensions between approaches that need to be resolved to make progress.

This trade-off between contradictory ambitions within government begins with the primary purpose of the state. On the one hand, the UK government has long sought to maintain a neoliberal approach that emphasizes modest state intervention, prioritizes support for the market to foster economic growth, and emphasizes individual responsibility for inequalities (an approach exhibiting the problems described by the APM). On the other, it often seems to embrace elements of approaches that seek a radical alternative, such as to boost state intervention to address the ‘social determinants’ of health inequalities and ‘out of school’ influences on education (an approach that would represent a meaningfully positive response to the structured inequalities explored by the APM).

Most importantly, no other ‘effective government’ principle competes well with the overarching commitment to accountability. A narrow form of Westminster-style democratic accountability commands policy attention and resources because the highest stakes outcomes relate to winning a UK general election. A fixation with short-term and tangible measures of success and competence overshadows longer-term and woollier commitments. A long tradition of a centralized, elitist, and exclusionary approach to government continues to prevail in Westminster practice (Marsh et al. 2003; Diamond 2013).

The UK government Covid-19 response revealed this tension acutely. It produced long-term pandemic planning on paper, but the strategy did not translate into sufficient capacity and the UK lacked effective coordination for the everyday work required. COVID-19 also surfaced debate between ‘doves’ who prioritized high state intervention to protect vulnerable populations and ‘hawks’ who prioritized a return to business-as-usual (Boswell et al. 2021; Cairney and Kippin 2024).

2.3 A lack of capacity: There is a big difference between rhetorical commitment and consistent application

The pursuit of ‘effective government’ begins as a valence issue that is hard to oppose, and ends as an agenda with insufficient capacity to endure. When decrying such cycles of enthusiasm and despair, it is common to describe, rather vaguely, a lack of ‘political will’ to see through commitments (Post, Raile, and Raile 2010). This criticism is insufficient without recognizing the routine obstacles faced even by the most since and energetic reformers. Policymakers operate in complex policymaking systems of which they have limited understanding and control, and it is not possible to simply pull levers to make reforms happen (Cairney 2020). Rather, reform requires the capacity to learn and respond continuously in relation to trial, error, and feedback. Without a proper appreciation of these routine constraints, reformers quickly become dispirited by the gap between their ambitions and political reality.

In that context, many potential reforms seem generally unattractive. First, some ‘preventive’ reforms or pushes for ‘early intervention’ in public services are akin to capital investments that seem prohibitively costly in the short-term with no guarantee of long-term rewards (Cairney and St. Denny 2020). Second, additional issues arise when dealing with future crises that seem relatively abstract, such as to invest heavily in energy system reforms in the name of climate change during a cost-of-living crisis (Cairney and Kippin 2024). Third, some ways to address these concerns have become less supported over time. For example, the idea of a social return to investment (spend £100 m now and save the equivalent of £1 bn later) competes badly with ‘cashable savings’ in the Treasury (Cairney and St Denny 2020). Furthermore, any new reform may test the patience of public sector staff who have endured many failed reforms in the past. Fourth, the need for collaboration across many autonomous or semi-autonomous organizations amplifies these constraints. There may be a commitment to the same agenda at multiple levels of government, but with some more invested, incentivized, or sure of their role than others.

The UK government Covid-19 response demonstrates many of these issues, such as when initially warm rhetoric about ‘building back better’ became cold in the face of urgent day-to-day service demands. We can also identify important gestures towards place-based coordination, ‘following the science’, ‘maximum possible transparency’, and unleashing citizen and civil society participation. However, these ideas proved hard to turn into concrete action, and became trumped by institutional risk-aversion and political calculations as the death toll mounted and the blame games began (Cairney 2021b; Flinders 2021).

However, this story is not simply one of Westminster exceptionalism. The UK’s central government was hardly alone in struggling to respond adequately to the crisis ( Boin et al. 2020). The contradictions and controversies that plagued the Conservative UK government bore a strong ‘family resemblance’ to those experienced in many other jurisdictions. Indeed, the pandemic laid bare the more general challenges for ‘effective government’, and the pathologies in democratic policymaking that tend to inhibit strategic capacity to deliver long-term planning and coordinative capacity. In other words, we need to understand ‘universal’ obstacles to effective government as well as the specific ways that governments address them.

3. How can we shift from explaining failure to supporting success?

This exercise in diagnosing failure has been perfected in political science research, but does not contribute much to policy analysis without supporting success. For that latter task, we offer the PoPP approach (based on the ‘positive public administration’, PPA, movement in comparative scholarship). PoPP is an orientation to thinking about, studying, and attempting to influence the practice of policymaking (policymaking is an umbrella term that includes public administration). At its heart is a challenge to the oft-stated assumption in political science that policymaking is characterized by failure. This failure assumption reflects and reinforces a low trust and high blame culture. Rather, PoPP seeks to focus attention on, and learn from, successful public policy (Douglas et al. 2021; Compton et al. 2022). It identifies opportunities for scaling up to facilitate effective policy interventions (Compton and ‘t Hart 2019; Luetjens, Mintrom and ‘t Hart 2019; De la Porte et al. 2022).

Three points help to develop this position. First, PoPP is not naïve. It does not represent the co-option of academe or a simplistic promotion of ‘instrumental knowledge production’ (van Ostaijen and Jhagroe 2022: 261). Rather, PoPP is capable of multi-level analyses that move beyond simplistic dichotomies of success/failure (Flinders 2023). Second, PoPP embraces inter-disciplinarity, emerging from insights of other ‘positive’ sub-fields (e.g. psychology and organizational studies) and close links to design-thinking (van Buuren et al. 2020). Third, PoPP challenges a historic academic fixation with ‘crises’, ‘disasters,’ and ‘fiascos’ in politics and democracy (Douglas et al. 2021). Not all policies fail, some are successful, and many are satisfactory. Learning from success is certainly preferable to an inability to learn from UK political failure.

To some extent, PoPP represents a conceptual balance to wider long-standing presumptions about governing dysfunction in UK political science (King and Crewe 2013; Gaskell et al. 2020). In this vein, we can follow the APM to state that Britain’s political institutions exhibit a historic tendency to be centralized and elitist, but also hope to learn from substantial transformations in, for instance, devolved or decentralized arrangements. Likewise, rather than treat standard operating procedures and incentives in the UK government as fixed drivers of pathologies like blame avoidance and institutional amnesia (e.g. Dunt 2023), we can also point to change and new configurations and practices that enable practitioners and scholars to learn and share lessons of experience.

To expand on and explain the benefits of this positive orientation, we turn our analytic attention to the unfulfilled ideas or aspirations that have dominated the reform agenda in the UK in the last few decades. Although we treat each approach separately in theory, they overlap strongly in practice as policy actors make sense of them (and the connections between them) in their policy routines. Each has roots in academic and practice-based research in policy studies, and has translated into aspirations in practice. Furthermore, there is a strong indication from pre-election debates that each approach will enjoy renewed prominence—in some guise—by the new UK Labour government.

The dominant political science narrative suggests that these ideas are doomed to inexorable failure against the backdrop of the UK’s elitist and dysfunctional governance, with a familiar cycle of naïve hope followed by cynical despair (Cairney and St Denny 2020). Still, it would not be naïve to expect that repeated reform efforts have an initially small but cumulative longer-term effect (Cianetti 2023). In that context, we re-examine the track record of each approach in the UK, with a view to reaching a nuanced understanding of the prospects of policymaking reform at this critical juncture. If the explanation for a lack of meaningful progress is that such ‘effective government’ approaches have been difficult to define, make congruent with business-as-usual policymaking, and overcome evergreen obstacles to policy change, why would the next reiteration be the gamechanger? What would it take to get it over the line, or make more progress, next time? Of each approach, we ask:

1. Clarity. What exactly does this approach propose or mean in practice? Revisiting its ideas allows us to identify then resolve previous confusion or contestation.

2. Congruity. How does it relate to dominant ways to define and deliver government? Revisiting allows us to identify previous pitfalls and anticipate future problems.

3. Capacity. What would it take to introduce and deliver this approach successfully? Revisiting allows us to gauge the impact of previous efforts to inform future activity.

4. Strategic state approaches

We begin with attempts to encourage more policy coherence and policymaking integration. The ‘strategic state’ is ‘a set of capabilities around the creation and delivery of an effective strategy at a country-wide level’ (Elliott 2020: 286). It is long-term and mission-based government that seeks to encourage decision-makers across government and the public sector to work together and consider the long-term impacts of their decisions on a shared strategic goal and underpinning objectives. In this sense, it shares much in common with similar ideas to promote ‘futures’, ‘foresight’, horizon-scanning, and long-term prevention in policymaking that have waxed and waned in Westminster in the last two decades (Cairney and St Denny 2020).

4.1 Clarity

The strategic state has caught on as an effective and cohesive framing for this constellation of ideas outside Westminster, especially in devolved governments. Unlike ‘futures’, ‘prevention’, or ‘foresight’ initiatives which have been piecemeal ‘add-ons’ that have struggled for meaningful impact in practice, a strategic state approach develops structures, systems, and processes of government that are aligned with a long-term vision and prioritized objectives, particularly in relation to well-being and sustainability. For example, the Welsh Government established a Future Generations Commission in 2015 to ‘act as a guardian for the interests of future generations in Wales, and to support the public bodies listed in the Act to work towards achieving the well-being goals’ (Welsh Government 2016). Furthermore, the Wellbeing and Sustainable Development (Scotland) Bill (Scottish Parliament 2022) is under consideration (the Well-being of Future Generations Bill—UK Parliament 2022—was not passed before the UK General Election in 2024).

4.2 Congruity

Though the Welsh example is promising, it is in Scotland that the Strategic State has been most advanced in the UK. Since 2007, steps have been taken to move away from a lack of strategic focus in the early years of devolution, and to embed the approach in routine government practice. Initially, coalition governments (1999–2003; 2003–07) designed programmes for government that had included over 400 separate commitments (Elliott 2023). Then, from 2007, the Scottish Government’s brought about several key reforms: (1) restructuring the civil service; (2) developing a National Performance Framework (NPF); and (3) leadership development. These reforms were intended to align the culture, organizational structure, and leadership of the civil service around a singular mission which was defined, at that time, as ‘creating a more successful country, with opportunities for all of Scotland to flourish, through increasing sustainable economic growth’ (Elliott 2020: 287). This approach informed key initiatives such as the Early Years Framework (Jung et al. 2010) and policing reforms to improve safety and well-being (Bland et al. 2021). Furthermore, significant investment was made in leadership development to enhance capacity and instil the ‘strategic state’, as well as principles of public value and adaptive leadership, in the culture of government and across the public sector (Scottish Leaders Forum 2023).

4.3 Capacity

There tends to be an ironic lack of sustainability in attempts to instil sustainability and long-term decision-making in government. Furthermore, it is difficult to draw clear lessons across the whole UK, since high-profile reforms get associated with a particular party of UK government and the desire of newly elected governments is to do something else as a way of distancing themselves from their predecessor (Macaulay 2021). This dynamic also has a devolved feature, in which key reforms—such as the strategic state—may be branded by devolved governments as a model to overcome the failures of ‘Westminster’ politics (e.g. Sturgeon 2023). That said, the Scottish Government demonstrates the undoubted capacity to develop a strategic state approach that is already ‘over the line’. Furthermore, the Scottish Government (2023) and former First Minister (Sturgeon 2023) have alleged that this more strategic approach informed a more purposeful and effective approach to COVID-19 compared to the UK government. The longer-term aim is to maximize its potential to defragment the public sector landscape and foster opportunities for meaningful coordination and collaboration (compare Elliott et al. 2022; Cairney 2024a,b).

5. Systems thinking and place-based approaches

We continue with attempts to foster more holistic and theory-informed approaches to problems and policymaking. Systems thinking is ‘concerned with the structure of a system, understanding, and defining its “boundaries” and making sense of the relationships between “agents” and the wider system’ (McGill et al. 2021: 2). Systemic outcomes result from a combination of many processes and decisions in different places and times, involving different actors. Each part affects other parts, and the process cannot be simply broken down into each action (Hawe, Shiell, and Riley 2009). This approach can be applied in different ways, to better understand (1) the interconnected nature of complex policy problems such as inequalities, (2) the complex policymaking processes that policy actors must navigate when addressing policy problems that transcend policy sectors or jurisdictions, and/or (3) the outcomes that emerge from a mix of policy instruments adopted or delivered by a multiplicity of policymaking organizations (Cairney 2024c). The holy grail is to connect these foci, to examine how to better connect multi-level and multi-sectoral action to ‘wicked’ problems and find meaningful ways to measure outcomes (see Cairney 2021a: 129–35 for a critical assessment).

5.1 Clarity

Systems thinking is not new to political science or policy and practice (Nel and Taeihagh 2024). However, there has been a renewed emphasis on translating a general idea into concrete approaches to complex social problems through horizontal coordination across government departments and non-governmental actors and vertical coordination between Whitehall and sub-national tiers (Bates et al. 2023). For example, tackling health inequalities requires action by non-health actors who make decisions that shape the wider determinants of health, while tackling climate change requires public and private action across energy, transport, and food sectors. If the problem is that many critical actors do not always see the interconnections between their activities and other parts of the system, systems thinking tools or approaches may represent a key route to effective governance.

5.2 Congruity

Systems thinking may seem at odds with a UK governance model shaped by a highly centralized and siloed approach to policy-making. Where can it be embedded in the Westminster model? Recent research argues that the best place to start is at the local level through strong place-based leadership (Ayres et al. 2023). Local leaders are well placed to understand the nature of complex problems and how to coordinate local assets to respond to the needs of affected communities. However, this requires cross-departmental government support to provide the right structures and incentives to enable local actors to join up their activities. A systems approach must fit with the business of government at different levels and scales.

Recent developments provide grounds for optimism. Both major political parties favour devolution to local areas and ‘hardly any voices can be found making the case for pausing or reversing the Government’s drive towards devolving power’ (Sandford 2023a: 1). Rather than a focus on sweeping constitutional change, a renewed enthusiasm for devolution can be found percolating in new ideas and practices across Whitehall that challenge assumptions about the centralizing tendencies of the UK state. The Conservative Government’s Levelling Up White Paper (HM Government 2022b) set out an ambitious plan to reduce spatial economic and social disparities between rich and poor parts of the UK. It acknowledged that past attempts at regional and local devolution have failed due to a top-down, siloed approach to target setting and implementation. Instead, it promoted a ‘whole system’ approach that acknowledges the links between different domains and parts of government. While the UK Labour government dropped the ‘levelling up’ phrase (renaming the Department for Levelling Up, Housing & Communities as Ministry of Housing, Communities, and Local Government), there are signs of a building consensus that devolved and integrated solutions can help to improve the effectiveness of government (Labour Party 2022).

5.3 Capacity

Two areas are critical to securing stronger capacity for place-based systems thinking in the UK government. First, the central government needs to develop and articulate a joined-up and long-term strategic vision for devolution in England, to match the more settled status of territorial devolution. This would provide the stability to develop robust central–local relations and the cross-government working to support local initiatives. The past few decades have been characterized by numerous reforms aimed at decentralizing powers and responsibilities to the sub-national tier (Sandford 2023b). The next phase of devolution will take place within the context of an unprecedented global economic, social, and health crisis. It will require funding models and accountability frameworks that grant far greater local autonomy (Centre for Cities 2023). Devolution needs to be championed by senior government ministers and underpinned by support from HM Treasury if progress is to be made.

Second, continued collaboration between Whitehall and the scientific research community is required to develop new evidence and data to support a systems approach at all levels of government (see below on evidence-informed government). Evidence and metrics are central to decisions about areas that would benefit from a systems approach, what can be devolved, and how local leaders can be held to account for performance. UK government departments are reaching out to the scientific community to request evidence that matches their areas of research interest (ARIs). Some signal a commitment to systems thinking at a local level (e.g. Department for Levelling Up, Housing and Communities 2022; Department of Health and Social Care 2023). This agenda presents an opportunity for scholars to ‘work across academic disciplines and fiefdoms to produce transdisciplinary evidence fit for purpose in a complex policy world’ (Ayres et al. 2023: 403).

6. Evidence-informed government

We now pivot to the oft-stated desire to use evidence for policy, as the aspirational evidence-based policymaking (EBPM) or more pragmatic ‘evidence informed’ approaches. Successive UK governments have argued for EBPM and the UK is widely seen as a leader in the field. Tony Blair famously stated that ‘what matters is what works’ (Sanderson 2006). The Coalition government (2010–15) created the UK’s ‘What Works network’ (Bristow Carter, and Martin 2016). The Conservative government leaned heavily on ‘the science’ in dealing with COVID-19 (compare Cairney 2021b; Kettell and Kerr 2022). However, recent years have witnessed a backlash against proponents of scientific or research evidence, with populist politicians proclaiming that experts are out of touch with public sentiment and senior advisers confronted with death threats during the pandemic (Smith 2023).

6.1 Clarity

There are important questions about what we mean by ‘evidence’ and ‘evidence informed’. Governments typically adopt a broad definition of evidence which includes administrative data, expert knowledge, evaluations, and consultations alongside primary research (e.g. Cabinet Office 1999). Many academics take a narrower view, but there are significant differences between sectors. In health, for example, randomized-control trials (RCTs) and systematic reviews remain the ‘gold standard’. In education, criminal justice, and social care there have been fierce debates about what counts as evidence and there is no prospect of unanimity about a single set of evidence standards (Breckon and Boaz 2023). Moreover, the vague and aspirational rhetoric of EBPM belies the complex, contested, and sometimes chaotic reality of policymaking. Simply ‘following the science’ is not feasible or desirable. Ministers also heed civil service advice, the views of ‘frontline’ professions, lived experience, and a multitude of other considerations in making what are, after all, political judgments (see public participation below). Consequently, many researchers espouse evidence-informed policy and practice, as a stance that recognizes that evidence will rarely be a decisive factor but can nevertheless be a valuable tool for defining problems, identifying potential solutions, appraising options, monitoring results, and fine-tuning implementation.

6.2 Congruity

Effective governments typically draw on research evidence to aid policy learning, guided by three fundamental questions: What has worked elsewhere? Why did it work? Could it work here? The UK is blessed with world-leading researchers who can be supported to address these issues. Most researchers and policymakers still inhabit the largely separate spheres described by Caplan (1979), such as when working to different timescales and incentives. Their research findings are frequently couched in academic jargon and hidden behind journal paywalls. However, recent years have seen significant efforts to bridge this research-policy divide. The weighting given to impact in Research Excellence Frameworks has prompted universities to take policy engagement much more seriously (Durrant and MacKillop 2022). UK government departments have published ARI (Boaz and Oliver 2023), and there has been growing recognition of the role of evidence intermediaries such as the What Works Centres, International Public Policy Observatory, Economics Observatory, and Local Policy Innovation Partnerships whose raison d’etre is to synthesize and mobilize research for policymakers (Cairney and Oliver 2020; Durrant et al. 2023).

6.3 Capacity

The challenge facing the new UK government is to nurture processes, culture, and capacity to ensure that research evidence is made available at the right time and in the right formats to be useful. The Labour government needs to build on existing infrastructure so it has the evidence it will need to help inform some of the difficult trade-offs it will inevitably face. Evidence can be particularly useful in the kind of resource-constrained environment facing a post-COVID, post-Brexit, low-growth UK. It can help to identify interventions that are likely to be most cost-effective, thereby reducing wasteful public spending, and it can highlight ways to safeguard vulnerable households and communities from the negative effects of tight spending rounds. It also opens the policy process to new ideas and innovative approaches. This search for higher capacity remains the big challenge. After more than a decade of austerity, governments at all levels lack staff with the time, training, and headspace needed to engage with evidence (and there remain significant barriers to more training or education—Elliott et al. 2024). But getting the best answers depends on asking the right questions, and this in turn requires dialogue between policymakers and evidence producers. Politicians and officials need to welcome external evidence into the heart of the policymaking process and work with researchers to co-create research agendas that address government priorities.

7. Public participation

Most of the approaches we describe make some reference to the benefits of citizen engagement to effective government, but participatory approaches should also be analysed as valuable in their own right. Calls for better participation imply a criticism of business-as-usual policymaking which is detached from and fails to value those affected by decisions. Participation represents a democratic good, advancing not only effectiveness but also legitimacy and justice (Fung 2003). It is a means of ensuring that policymaking is informed by more than science or evidence (Funtowicz and Ravetz 1993) and underpinned by enhanced consent and trust (Ostrom 1996). Participation reflects recognition that bringing the lived experience, practical insights, and local knowledge of citizens, residents, and service users into policymaking is crucial to collaborative government. Participation can support co-productive and future-oriented principles (Durose, Perry, and Richardson 2022) by providing the government with the expertise and foresight (Dinges, Biegelbauer, and Wilhelmer 2018; Pollio, Magee, and Salazar 2021) to meet and anticipate complex and wicked challenges, and open up new ways to frame and approach policy challenges.

7.1 Clarity

The case for greater participation in policymaking has been ‘cogently made, many times over’ (Richardson, Durose, and Perry 2019: 123), contributing to a huge number of government policies and suggested participatory reforms to government. However, there is a widespread acknowledgement of a significant gap between rhetorical commitment and consistent application. Initially, part of this issue is limited clarity. Public participation can be understood either as an ‘umbrella term’ to describe how ‘people’s concerns, needs, interests, and values’ (Nabatchi and Leighninger 2015: 6) are reflected in decision-making, or more broadly to encompass volunteering and mutual aid. Public participation may be categorized according to the techniques employed (Elstub and Escobar 2019), the role of the citizen (Durose et al. 2013), and the terms of engagement (Arnstein 1969; Cornwall 2008). Furthermore, whilst ‘public’ may refer to all people, it is also used to refer to groups specific to place, issue, or interest (Taylor 2011). Overall, participatory innovations vary widely according to context, and can often be temporary and local. Whilst a bewildering array of initiatives can be confusing, this variety and innovation can also foster learning. Indeed, growing experimentation with public participation, its expansion into new issues and questions, and the consolidation of a participatory ‘ecosystem’ for initiating and supporting such efforts, all help to clarify the value of different participatory approaches in supporting effective government (Bussu et al. 2022).

7.2 Congruity

Whilst public participation is now an established part of policymaking, and may be aligned with key principles of effective government, there are significant challenges regarding the congruence between participation and policymaking. First, meaningful participation may be perceived as a threat to established forms of policymaking: to exacerbate messiness, induce unhelpful politicization, and disrupt the established order and the perceived control of policymakers or primacy of a democratic mandate (Cairney 2023). Or, more tokenistic efforts prompt concern that government-led participation will only be employed to sustain and expand the power of government: preserving its ability to set the terms for mobilizing and harnessing community resources for self-management whilst divesting the state from responsibilities. Second, a meaningful commitment to participation makes upfront and sustained resource demands of government, in terms of supporting effective processes and responding to a re-framing of policy driven through participatory deliberation. Or, tokenistic public participation may involve commandeering the logic of ‘responsibilisation’ in which citizens are expected to take charge of their own problems (Clarke 2005; Durose, Perry, and Richardson 2024), This unclear role for participation contributes to a gradual and inconsistent trajectory in policymaking (Boyte 2005).

7.3 Capacity

We do not pretend that there is currently high and sufficient capacity for participation in UK public policy. Indeed, if we recognize the agency of participants, then it is not wholly within the hands of governments to induce their participation. Furthermore, partial, failed, and unsatisfactory attempts at participatory governance have exacerbated the fatigue, cynicism, and apathy of many participants. Participation may only become a fuller presence in government if demanded by those affected by its policies and if participation has clearer rewards. It often seems difficult to gauge what further traction public participation will gain without a shift in government’s capacity to harness the transformative potential of participation beyond a narrow (often economic) logic, and to challenge resistances to change that would better align institutional incentives for policymakers to engage in participation. However, there are promising examples of participatory policymaking on which to draw, and a clear need to make progress. Public responses to various crises and chronic situations—including the climate emergency, cost of living, COVID-19, and health inequalities—highlight the limits of policymaking detached from the lived realities of many in our society, and an impetus to challenge and reform through active citizenship.

8. Behavioural public policy

Compared to the other approaches we have surveyed, behavioural public policy (BPP) appears to be a clearer proposition that gained traction quickly in the UK government. Here, the issue is that BPP initiatives were quickly coined as ‘nudge’ (Thaler and Sunstein 2008) and became associated with a narrowly technocratic, over-individualistic, and manipulative approach to short-term problem-fixing (HoL 2011). The role of BPP in COVID-19 policy advice was also subjected to negative scrutiny when associated unfairly with the idea of behavioural fatigue (Oliver 2020) and alternatives to ‘lockdown’.

8.1 Clarity

BPP’s future potential derives from its maturation into better-defined new forms, using clearly articulated values to signal the essential ethical and structural dimensions that would inform any specific initiative. For example, the core innovation of BPP has been the applied use of cross-disciplinary insights about the inevitable biases in people’s decision-making, to make it easier to help people ‘do the right thing’ (John et al. 2019). By implication, BPP is already citizen-centred, but has struggled to translate this orientation into transparent policy processes and recognition of citizen agency. A different but equally essential core value has been a commitment to robust evaluation to build an evidence base, albeit while initially being criticized for the small-scale nature of its policy achievements, and its previous neglect of structural factors (Chater and Loewenstein 2022).

In that context, new ethics frameworks allow policy designers to make explicit trade-offs along continuums of transparency and citizen autonomy, considering the consequences for, and strength of preferences of, different stakeholders (Hallsworth and Kirkman 2020). Emerging work on more participatory and ‘relational’ BPP offers an alternative vision for more value-based, less technocratic, forms of BPP, with emerging examples in social housing (Absalom 2023). ‘Nudge plus’ approaches (Banerjee and John 2024) advocate for greater integration of human agency in BPP (Banerjee et al. forthcoming). A key implication is that behavioural insights interventions are no longer stuck in murky policy shadows, and could be co-designed with citizens (Richardson and John 2021).

8.2 Congruity

BPP has already made significant in-roads into UK government practices and public policy internationally (OECD 2017; John 2019; Hallsworth and Kirkman 2020). The Behavioural Insights Team (BIT), one of the first ‘nudge units’ set up in a central government body, was ‘spun out’ from UK government and is now an independent company with over 200 staff, operating internationally, with over 1,000 projects completed. The international spread of behavioural insights has started to answer questions about transferability beyond WEIRD contexts (western, educated, industrialized, rich, and democratic) (Henrich, Heine, and Norenzayan 2010). BPP has spread and seen some success. For example, according to Halpern (2019), there are now multiple BPP-designed policies generating more than £1 billion in public spending savings (‘unicorn policies’) as well as improved outcomes for citizens on health and wealth.

8.3 Capacity

BPP has established proof of concept in theory, and some practice, and now needs to move towards achieving longer-term change for citizens and greater citizen engagement (John 2018). For example, S-frame interventions (Chater and Loewenstein 2022) operate at a structural level, overcoming earlier biases in BPP towards individual-level interventions that weakened the potential scale and effectiveness of the approach. An emerging rapprochement with other approaches such as design thinking—which also has strong overlaps with systems thinking—helps reinforce BPPs ability to re-frame policy for more systemic change, rather than simply applying fixes to broken policies (e.g. Halpern 2019).

Given BPP’s history, we offer a more cautionary positive tale. These experiences clearly suggest that there is capacity to develop a BPP third wave, but possibilities include transformation, stagnation, and decline, or splintering into different assemblages (Ball 2022; Absalom 2023). There are frameworks and toolkits that could encourage decentralized applications (Michie, van Stralen and West 2011), but decentralized units—such as UK local and city-regional governments—need significant resources to design and deliver meaningful interventions. BPP will need to shake off its reputation for being a cheap and easy policy tool (Benartzi et al. 2017). Transformation will require more citizen engagement in BPP, and a focus on longer-term structural change, possibly even using behavioural insights to ‘nudge’ policymakers into thinking differently about the short and long term.

9. Conclusion: Will this time be any different?

How feasible are such approaches at this critical juncture in UK politics characterized by crisis and instability? The last few decades have involved a prolonged period of instability in UK government. Leaders and officials have operated in perma-crisis mode, bouncing reactively from financial crisis and austerity, through to Brexit and its protracted fallout, then the upheaval of COVID-19, and more recent energy and cost of living crises. The result has been a retreat to the cosseted and elitist approach to decision-making that has long been the norm in Westminster (Marsh et al. 2024). Does the immediate future represent a ‘window of opportunity’ for meaningful change? Or are these ideas for reform simply ‘old wine in new bottles’, with well-meaning enthusiasm soon to be dashed by inexorable realities?

While it is tempting to remain fatalistic, we stick to the tenets of PoPP to find grounds for cautious optimism. Our conclusion returns to our alternative framework—around clarity, congruity, and capacity—to identify how things might be different this time.

9.1 A clearer way forward

First, it is reassuring to see attractive but ambiguous slogans for reform transformed into more tractable sets of strategies, practices, and processes. Decades of research and practical experience behind public participation, BPP, evidence use, the strategic state and systems, and place-based systems thinking help to give clearer shape and contour to what each means in practice. Much fuzziness remains, but this fuzziness also remains politically valuable (Pollitt and Hupe 2011). What we need is just enough clarity, to give actors a strong enough sense of what they might be buying into, and to hold the most powerful to account for their lack of progress towards these reasonable and essential ends (Durose et al. 2017).

Second, there is also a heartening degree of synergy across the approaches we survey. Concerns about a lack of clarity are not just about the fuzziness of aspirational ideas in isolation, but about apparent conflicts or contradictions among them (Boswell 2022). At first pass, ‘nudging’ citizens, for example, would seem anathema to engaging them in public participation, while systems thinking would seem similarly at odds with the modularity of evidence-informed policymaking. However, our more nuanced reviews show alignment rather than disjuncture: we see that effective BPP can and should be done in a participatory manner, and that the holistic knowledge associated with systems thinking is in sync with contemporary ideas about systems for evidence use (and with holistic approaches to the ‘strategic state’). In other words, what emerges could be a complementary set of versions of these approaches that can underpin a coherent narrative for reform.

9.2 A more congruent agenda

We also see the potential for this unified agenda to be more congruent with routine government business. To flip the ‘old wine in new bottles’ concern on its head, the evergreen nature of these approaches becomes an advantage for achieving traction in the future (Cianetti 2023). Each represents a well-established tradition with a track record of institutionalization across the UK rather than rolled out from one powerful centre. Indeed, the devolved governments in Scotland and Wales and new combined authorities in England have often been the key sites of innovation. The qualified successes that exist in relation to each approach constitute incremental tweaks to existing structures and practices of democratic accountability, rather than a radical departure from the status quo. The promise lies in repetition and an accumulation of ‘good practices’ from within and elsewhere. They represent stories that policymakers prize as they grapple creatively with the ‘art of the possible’ to translate lofty ideals into practices that might work in context.

We acknowledge concern that this ability to be easily accommodated might make each a bolt-on to routine business. BPP or public participation, for instance, remain widely seen as cheap and easy additions to be commissioned for a specific output, without fundamentally altering how things work. This sticky mindset is a legacy of how disruptive innovations typically get a foothold in the policymaking process to begin with (Hjelmar 2021). Nevertheless, that foothold is clearly there, and the first step to mainstreaming is to become known and familiar. Proponents of reform across the UK are starting from a stronger institutional base than in earlier cycles of enthusiastic advocacy.

9.3 Enough capacity to realize change?

While the signs in terms of clarity and coherence are positive, years of instability in UK government have eroded underlying capacity. Chancellor Rachel Reeves (2024) claims to have inherited the ‘worst set of circumstances since the second world war’, and there is widespread fiscal stress among devolved and local governments. Our collection of approaches all promise to relieve these pressures in the long term. However, the perception may be that all require significant start-up costs in the context of scarce resources and high scrutiny. A more coherent reform narrative, better attuned to how government works, can go some way to lowering barriers and leveraging resources. Nevertheless, will policymakers be willing to take the ‘leap of faith’ required?

This final obstacle should prompt academic researchers and reform-minded advocates to work together to help make that leap easier and less risky. Now is the time to attract and devote resources towards trialling, tracking, and evaluating experimentation in more future-oriented, more holistic, and more participatory approaches to government. Although a cliché, more research really is needed. A concentrated programme of research, co-produced by policy scholars and policy participants, represents a way to help lower the barriers to initial action, refine and embed innovative approaches more deeply and systematically in routine government business, and drive forward a clear, coherent, and convincing narrative for reform.

The election in July 2024 of a Labour UK government redoubles the timeliness of this agenda for practice and research. Early signs point to a commitment to ideas such as radical pragmatism and a push towards stability before more ambitious change. For example, the first few days have seen the appointment of non-partisan experts in Ministerial teams and high-profile signals to reset relationships with civil servants in Whitehall and leaders across devolved regions. The promise is to put more impetus on being strategic, balanced, and relational to deal with profound governance challenges (see Flinders et al. 2024). This change represents an important window of opportunity for learning and applying the lessons of PoPP.

Conflict of interest statement. None declared.

Funding

No funding was received.