Blog: Indicators for Public Sector Innovations: Is there a silver bullet?
As innovations in the public sector become more important both in public sector reform and academic discussions, also attempts to understand and measure the effects of these innovations are increasing. However, as with other public sector phenomena, the measurement attempts for public sector innovations are riddled with difficulties, both conceptual (what to measure) and technical (how to measure). For example, can changes in citizen’s trust towards public sector be linked to particular innovations? And if so, how? Or how should one use social media as a source of understanding the effects of innovations?
In this blog we take a look at existing practical attempts to develop indicators for public sector innovations. This is based on our research (preliminary results are found in a LIPSE working paper). This is, to our knowledge, the first attempt to comprehensively gather information about various international efforts. We only considered such exercises where actual indicators were developed and used at least once. We map five such exercises through extensive desk research and 13 interviews with surveyed project members. The projects were as following:
- MEPIN (Measuring Public Innovation in the Nordic Countries).
- European Public Sector Innovation Scoreboard (EPSIS).
- Australian Public Sector Innovation Indicators Project (APSII).
- NESTA, (United Kingdom) sponsored London School of Economics Public Policy Group (LSEPPG) public sector innovation index.
- Korean Government Innovation Index.
Based on our research and interviews with key members from all five projects, we can draw the following conclusions.
- Motivations behind creating indicators. The overarching idea behind creating new indicators for public sector innovations is strongly associated with justifying and legitimizing public sector activities. Within the wider context of political demand, the current attempts tend to also reflect individual preferences of sponsoring organizations. For example, EPSIS was designed in cooperation with various EC directorates and thus ended up measuring both innovation in public sector as well as innovation through public sector (i.e. how public sector influences innovation in private sector), whereas other attempts focused more on innovation within public sector. The nature of political demand also had other important consequences on how the measurement efforts evolved. At the same time, political demand has been rather inconsistent as in most cases it is not known if and how the measurement attempts will be prolonged.
- Theoretical discussions and ideas behind indicators. In all cases key people involved with developing indicators were economists by background and readily admitted that business innovations were an important theoretical vantage point. However, they also all admitted that the public sector has unique logics that are often difficult to separate from each other and, consequently, also to measure. Significantly, all projects realized that they would have needed substantial input from public administration and governance scholarship only after the empirical exercises. In addition, important difficulties were discovered in applying the existing private sector innovation measurement methodologies into the realm of public sector.
- Irrelevance of performance measurement. Performance measurement practices have played almost no role at all in developing public sector innovation indicators. While they have been one of the most studied issues in recent public management debates and could provide ample ideas for linking performance measurement with innovation. This seems to be strongly related to missing public administration and management competencies in all projects.
- Who uses indicators and how? Predicted use of indicators varies quite significantly from in-house management tools to country-to-country comparison and benchmarking. However, it is important to note here that all the interviewed stakeholders were very aware of the limitations of the measurement attempts, which is why we can also see a rather different mix of actual use of the measurement results.
- Future evaluations of indicators. No projects have been formally evaluated; it is also not certain that any of the projects will be repeated in the future (EPSIS, slightly altered, seems to be most certain to be repeated) or developed into a working tool for other organizations (OECD work on updating Oslo manual will utilize some of the work done in MEPIN and EPSIS). This seems to stem from difficulties both on the technical side (availability, comparability of data) but also practicality (in most cases the results do not tell much to managers, politicians or citizens and it is expensive to gather new forms of indicators).
Our research shows that while public sector innovations are difficult to measure, policy makers should encourage measurement attempts that utilize public administration scholarship on performance measurement and other similar approaches; equally important is to utilize new sources of data (social media, big data) and experiment with emerging evaluation practices and tools such as randomized control trials, living labs etc. However, as our research shows, key in successful use of such tools is to have rather complex evaluation frameworks that include political issues such as trust in public sector, and do not rely solely on few quantitative indicators.
There is no silver bullet in measuring public sector innovations.