I have established a research programme at civilservicereformuk.com to investigate when, why and how civil service reform works. I aim to develop actionable insights that will help both politicians and officials to succeed in civil service reform.
Why is it so hard to describe the impact of civil service reform? Decades of research have produced little. Indices of government or civil service effectiveness are a fool’s paradise. The quest for causality in managerial reforms seems likely to be a road to nowhere.
This blog digs deeper into the challenge of capturing the impact of reforms and outlines how I plan to bypass them through my research programme. I plan to develop and test my proposition that a better approach would be to try to articulate the impact of reform in terms of the increased presence of some key organisational capabilities.
The starting point for civilservicereformuk.com is my view that both the public administration view and the prevailing narratives of civil service decline are partial, often misleading and fail to capture the cumulative and transformational impact of 65 years of reforms.
I need a way to articulate what successful impact looks like in order to declare a reform successful. I can then use that picture of impact to track back through what reformers did, and what it was about how they did it that led to this impact.
Plan A.
Surely it should be straightforward to judge the success of a reform? Evaluation Plan A is to take the stated impetus, objectives, milestones and targets from the reform plan, white paper or statement (making the brave assumption that these exist or are soundly based), and then see whether they were achieved? Or alternatively could we look at an overall index of measures of government or civil service effectiveness and see how the index changed following the implementation of a reform or package of reforms.
Sadly, real life isn’t like this. There are five complications that sink Plan A and point to the need for a Plan B.
Complication 1: Reform plans and statements are unreliable guides to what was intended let alone what actually happens.
There is often more political rhetoric and symbolism than analysis or evidence in major reform launches (Pollitt, 2013c). And the goals can be woolly and flexible. To understand reforms and their impact it is necessary to distinguish the rhetoric and symbolism of the reform narrative from the scope, style and intent of actions that follow. Even the most notable New Public Management (NPM) scholar wondered ‘whether NPM was mischaracterised as a set of public management policies designed to cut costs when in fact it may have been more about ideology and rhetoric?’ (Hood & Dixon, 2013).
This is why my analysis of the patterns and scope of 65 years of reforms focused on the actions that were put in place rather than taking the language of the white paper, review or plan as gospel.
Complication 2: Reforms evolve and change in implementation
The organic nature of reform creates several almost insurmountable barriers to efforts to evaluate their impact (Pollitt, 2013b) (Pollitt, 2013a):
Stated goals may not match the aims of a programme in practice, or they may be woolly at the outset only becoming clearer as design and implementation proceeds.
‘Many reforms are, in effect, redesigned during implementation because new aspects or difficulties are discovered ‘on the ground’.
The context in which the reform is taking shape may change dramatically.
The reality of institutional change is that intentions and goals will evolve as reforms are designed and implemented. This evolution is by no means a bad thing and instead is likely to be an important success factor. There is some compelling research in the field of institutional work which has found this process of discourse and adjustment to be a critical factor that creates engagement and allows adaption to fit the reform to local contexts and priorities (Cloutier et al., 2016) - hence increasing its impact and sustainability.
For some this adaptation and redesign tells a tale of centrally driven reforms being sabotaged by recalcitrant officials. But those of us who have worked on reforms in large public sector organisations might see it as instead trying to adapt the reform so that it works: taking the overall idea as a guide but dismissing or reinventing poorly designed central directives and implementation templates because they are unworkable. In reality both of those things are probably going on within a single reform effort - which in turn demands great skill in the conduct of the reform. Research in the fields of institutional work, strategy as practice and strategy process offers insights into the critical nature of this ‘work’ by reformers (who are distributed across the system rather than just being the central players in the cabinet office or treasury).
Complication 3: Developing purportedly objective indices of civil service effectiveness is a fool's mission.
Hopes for an objective view on whole of government and or civil service effectiveness that can provide a reliable anchor for reform impact are likely to be misplaced. Most academic efforts draw on administrative data (data collected and published by government organisations themselves and scores issued by government inspectors). Less often they draw on survey data which may also be set by political superiors, for example mandatory surveys of consumer satisfaction, but are often based on primary data collected by researchers’ surveys of public officials’ views of the performance of their organisations. A meta review of 92 studies of public service performance concluded:
Organisational performance, perhaps especially in the public sector, is open to a variety of interpretations and is politically contestable. The multiple stakeholders that judge performance include political principals, funders (usually higher levels of government) and service recipients (and non-recipients) outside the organisation, and professionals, managers and front-line staff inside public agencies. These stakeholders may disagree about the most important dimensions of performance, the indicators used to measure them and the scores on these indicators that constitute success or failure. Administrative indicators of performance are selected by governments and their agents, so their composition reflects the priorities of powerful groups within the State. …This means that their coverage is likely to vary across nations and over time. (Andrews et al., 2011)
A related but highly influential branch outside academic research is the thriving global industry assembling and promoting indices purporting to rank the effectiveness of governments and in some cases civil services. Some are long established, the World Bank’s Worldwide Governance Index and the Bertelsman Stiftung’s Transformation Index (Governance in international comparison). They are subject to criticism:
The WGI claim to measure governance; as yet no evidence has been offered that this is true. The WGI represent a complex atheoretical and as yet poorly articulated hypothesis for which no evidence has been advanced. (Thomas, 2010)
Indices tend to reflect the dominant paradigms of public management and democracy at the time of their creation. And they are inevitably skewed to what can be measured, and the existence of some roughly comparable datasets offering a time series. They tend to fall back on questionable surveys to fill data gaps or shore up the credibility of their index. Their claim to make meaningful comparisons is suspect given they cannot possibly account for culture, context and antecedents which are critical to the evolution and effectiveness of governments. It is a massive leap of faith to think these indices adequately capture effectiveness of civil services and government. If they are used as ways to identify countries with interesting practice and variations from which you might learn then they have some value.
A cynic might observe that two relatively recent indices each found that the country of their apparent commissioner/sponsor came out top of their respective league tables (Blavatnik’s InCiSe index of civil service effectiveness, and Oxford Insights Human Centred Public Services Index). At best the results reflect the influence of the sponsoring entity over what measures are chosen for inclusion or exclusion. Both rely heavily on data from the longer established indices supplemented by modest surveys.
And this is all before you come to the major challenge of ascribing causality to any changes that might be reflected in those indices and the supporting measures.
Complication 4: Is causality a road to nowhere?
It seems unpromising to try to attribute causality for improvements to civil service effectiveness to elements of specific reforms. Each link in any causal chain is contested: buffeted by context, culture, power, stakeholders and a complex array of variables.
It is exceptionally difficult to attribute outcomes or effectiveness to specific techniques, processes or structures in any organisation let alone an institution like the Civil service.
Complication 5: Reforms don’t take place in isolation.
Reforms do not come as neat interventions. They are a messy agglomeration over decades, sometimes taking 20 or 30 years from the simmering of the primordial soup of reform ideas to becoming embedded within the civil service. The path dependency of reforms means causality cannot be attributed to one phase of reform - later reforms only became possible because of previous reforms.
Reform is a long game. The patterns I picked out in my analysis of 65 years of reforms tell a story of iterative, evolutionary change. For example Next Steps is sometimes presented as a big bang reform but in fact it was the culmination of thirteen years of previous reforms. The path dependency of most reforms is a striking feature: they depend on and build on previous reforms; and the ideas they draw on had been developing, circulating and evolving over time until they found that opportunity to coalesce. As Kate Jenkins (a key figure in the development and delivery of Next Steps) observed : I do not say that Next Steps is a tremendous success because there are 103 agencies 10 or 15 years later. I say that it is a great success, as the FMI was a great success, because it has led on to the next thing, which is relevant to how the Civil Service is operating now. That is the real story of Civil Service reform (Kandiah, M., 2007). In all nine of the continuous themes revealed by my analysis there are clear path dependencies running from the 1960s to the 2010’s.
A large review of 519 studies of NPM impacts across Europe (Pollitt & Dan, 2013) found a mixed ‘hit or miss picture’. Whilst around half reported a positive impact, 47% of those looking at outputs found they did not improve, and 56% of those looking at outcomes reported no improvement. This European comparative review concluded that whilst NPM interventions could not be called a failure, the political, structural and cultural context was crucial to the success of NPM interventions. They compared NPM interventions to ‘a delicate plant [that] requires the right soil and care, more orchid than potato’. As well as being intrinsically hard to evaluate, the importance of the context to each intervention complicates the attribution of the causes of any outputs and impacts (Pollitt & Dan, 2013).
We need a plan B.
For fifty years the massed ranks of public administration researchers have struggled to crack these complications. I feel I am unlikely to succeed where they have failed by pursing Plan A.
The search for a soundly based articulation of reform impact and what it was that caused that impact is arguably never going to succeed.
Instead we might begin to plot an alternative route by accepting that success is inevitably a subjective judgement, reflecting power, stakeholders and context.
At the Institute for Government when we were trying to identify ‘successful reforms’ for our success case studies we took a methodologically dirty approach instead. We asked a bunch of reform minded officials to generate a list of what they personally saw as successful civil service reforms over their time in the service (they had served between 10-30 years). We then asked them to rate that list on the basis of which they thought were most successful ie: impactful. We then explored what people had in minds when they judged reforms more successful than not.
The explanations which emerged included: the change became embedded, it changed how I saw my job, I learned new better ways of doing my job, it created the platform for further reform, it changed how we saw our roles, I was held accountable for different things, people got promoted for their ability to manage and deliver as much as their policy making and advisory record, it created a much more conducive context for doing my job, it reduced unhelpful central controls and micro management.
Oddly we were never challenged subsequently on why we picked the cases we did. Given the group of officials we tested our cases with were representative of our primary audience perhaps there was something to be said for taking their perspective on what succeeded. This meant they were then interested in why and how those reforms had succeeded (rather than arguing about if they had succeeded). This provided a much stronger basis for the conversation that followed with practitioners about what they could learn from the practice of successful reformers. Actionable insights for reformers are the outputs I am aiming for from my research programme.
So as an alternative to the perhaps unattainable science of tracking causality I am inclined to adopt the approach advocated by a review by Joullie and Gould (2023) into the limitations of management research:
Rather than seek causality management researchers would better aim to ‘understand and explain deliberate actions, situational choices, ambiguities and constraints’ accepting that the actions of agents are not deterministically constrained. Such understanding needs to be rooted in the context of values, opportunities, and an imperfect pool of ideas and experience that are drawn on as choices are made and paths pursued.
Such an approach resonates very strongly with me and has influenced the development of my conceptual framework for understanding civil service reform. Unsurprisingly research fields with strong roots in sociology (strategy process, strategy as practice and institutional work) sit very comfortably within this view and will provide an enriching theoretical basis for my further research.
Exhibit 1. A conceptual framework for understanding civil service reform
Source: Peter Thomas
Plan C. Don’t give up completely on Plan A.
At the risk of contradicting myself at this stage in my research programme I am still not giving up on finding something more measurable and old school that can capture in part at least the impact of reforms. I am looking for suitable intermediate variables - which can in turn be plausibly expected to have a relationship with institutional effectiveness.
An increasing number of researchers have sought to address the theoretical limitations of the public administration tradition by drawing on theories from other fields. They have used theories from strategy process, strategy as practice, institutional work, sense-making and dynamic capability - often in combination - to understand change and its impact in public sector institutions. Within these research fields there are calls for greater efforts to bridge, connect and even combine theories – so as to benefit from their respective strengths and mitigate some of their limitations when applied in isolation.
One strand of my research programme is exploring whether the concept of dynamic capabilities - specifically knowledge based capabilities - may provide a tangible way to identify the impact of civil service reforms.
In the 1990’s dynamic capabilities (DC) were introduced as a framework to explain the ability of an organisation facing a rapidly changing environment to know their context and reconfigure their assets to maintain their competitiveness. These DCs were described as being organisational learning skills: generating new knowledge that would ‘reside in new patterns of activity or routines’ (Teece et al., 1997). DCs have become the dominant theoretical framework for understanding how organisations change (Piening, 2013).
Teece later broke DC’s down into three kinds of capacities: ‘(1) to sense and shape opportunities and threats, (2) to seize opportunities, and (3) to maintain competitiveness through enhancing, combining, protecting, and when necessary, reconfiguring the business enterprise’s intangible and tangible assets’ (Teece, 2009). DCs acted as strategic meta-routines through which organisations adapt, change or introduce new operational routines to improve their performance.
A seminal article (Zheng et al., 2011) argued that ‘the fundamental function of the firm is to integrate and use knowledge’ and drew out three Knowledge Building Capabilities that stand above others as special kinds of DCs: knowledge acquisition capabilities (KAC), knowledge generation capabilities (KGC) and knowledge combination activities (KCC). They found that this final knowledge combination capability contributed the most to innovation and performance, drawing on the raw material created by the other two processes. They noted the increasing role that alliances and networks play in organisations’ environment.
It could be possible to gauge this intermediate reform impact and connect it directly to the scope and practice of a series of reforms. To make the case for these intermediate impacts being relevant I would rely on a growing body research which is establishing a credible case for the impact of these intermediate outcomes on organisational effectiveness and longevity.
I will publish a long read in early Autumn that presents my exploration of the potential of this approach for making impact more tangible and measurable – with a good enough evidence base for being related to civil service effectiveness in terms of innovation and performance.
If the approach seems promising, then part of my subsequent research would focus on understanding the design of reforms and practice of reformers that built these key dynamic capabilities.
Peter Thomas 26-04-2024
For references see searching for success at civilservicereformuk.com.
Comments