Recommendations
Alternative download: A complete copy of the document
is also available in pdf format:
Trying It Out - The Role of 'Pilots' in Policy-Making Report of a Review of Government Pilots (PDF)
On this page:
The role of pilots
Pre-conditions of pilots
Key properties of pilots
Methods and practices of piloting
Using pilot results
The role of pilots
- The full-scale introduction of new policies and delivery mechanisms should,
wherever possible, be preceded by closely monitored pilots. Phased introductions
help not only to inform implementation but also to identify and prevent unintended
consequences. A pilot is an important first stage of regular, longer-term
policy monitoring and evaluation. (3.1; 3.2; 3.4; 3.7; 6.5; 6.6)
- Although pilots or policy trials may be costly in time and resources and
may carry political risks, they should be balanced against greater risk of
embedding preventable flaws into a new policy. Initial policy submissions
to Ministers should explicitly consider such factors and contain a section
on possible piloting strategies. (3.4; 3.5; 6.7)
- Advantage should be taken of the small scale and explicitly experimental
nature of pilots to encourage innovations in policy that might otherwise be
too risky or costly to embark on. (3.4; 3.6; 6.9)
- Pilots should vary in their nature and scope according to a range of factors
– not all of which are obstacles – such as tight timetables or low
budgets. Also important in shaping a piloting strategy should be the extent
of accumulated knowledge already available about that policy area. The scale
and complexity of any experimental treatment should be proportionate to its
likely utility. (3.7; 6.1; 6.2; 6.3; 6.6)
Pre-conditions of pilots
- A pilot should be undertaken in a spirit of experimentation. So, if it
is clear at the outset that a new policy and its delivery mechanisms are effectively
already cast in stone, a pilot is redundant and ought not to be undertaken.
(3.2; 6.7)
- Once embarked upon, a pilot must be allowed to run its course. Notwithstanding
the familiar pressures of government timetables, the full benefits of a policy
pilot will not be realised if the policy is rolled out before the results
of the pilot have been absorbed and acted upon. Early results may give a misleading
picture. (3.2; 3.4; 3.7; 6.2; 6.3; 6.8)
- Many policies take time to bed in; others are intended to achieve only modest
changes in outcomes. The timetable and scale of a pilot must take account
of such factors so as to avoid producing a false impression of policy failure.
(3.7; 6.3; 6.8)
- As with all policy development, pilots should be preceded by the systematic
gathering of evidence from the UK and abroad. (3.7; 6.6)
- The precise purpose(s) of a policy trial – whether it is to measure
a policy’s likely impact or to test its delivery mechanisms, or both
– must be made explicit in advance so that its methods and timetable
are framed accordingly. (3.1; 3.2; 6.1; 6.6)
Key properties of pilots
- Independence is critical. Pilots must be free from real or perceived pressure
to deliver ‘good news’ and be designed to bring out rather than
conceal a policy’s imperfections. To this end, the Ministers and civil
servants most closely involved with the policy should consider distancing
themselves from decisions about pilot methods and the dissemination of their
findings. (3.7; 6.4; 6.5)
- Methods matter. A poorly conceived or poorly specified pilot may be worse
than no pilot at all. To ensure that the methodology of a pilot is as bullet-proof
as possible, expert internal and external advice should be drawn on early,
and appropriate resources made available. (4.1; 4.2; 4.4; 6.6; 6.7; 6.8)
- Nomenclature matters too. The terms ‘pilot’ and ‘policy
trial’ should be reserved for rigorous early evaluations of a policy
or some of its elements rather than for other forms of research into a policy’s
early performance. (3.1; 3.2; 3.3)
- Tags such as ‘trailblazer’ or ‘pathfinder’ are best
avoided for genuine pilots or policy trials. By creating unrealistic expectations,
they tend to make neutral evaluation more difficult. (3.3)
- It must be recognised that the policy process is not always suited to rigorous
and necessarily lengthy pilots in advance of a policy roll-out. Time and resources
are limited and Ministers are often impatient to deliver. So provision for interim findings – always accompanied by appropriate health warnings
– must be anticipated. (3.7; 6.2; 6.3; 6.6; 6.7)
- To avoid systematic errors in the conduct of pilots, their budgets and
timetables should allow for adequate training of the staff who are to administer
processes such as allocating participants to ‘treatment’ and ‘control’
groups. Policy and research staff training should also include modules on
piloting and evaluation. (6.7)
Methods and practices of piloting
- There is no single best method of piloting a policy. Multiple methods of
measurement and assessment – including experimental, quasi-experimental
and qualitative techniques – should all be considered to get a complete
picture. (4.1; 4.2; 4.4)
- For policies designed to achieve change in individual behaviour
or outcomes, randomised controlled trials of individuals offer the most conclusive
test of their likely impact. Long under-used in the UK, they should more often
be considered as vehicles for rigorous trials. (4.2; 5.1; 5.2; 5.3)
- For policies designed to achieve change at an area, unit or service level (such as in schools, hospitals or job centres), randomised
area- or service-based trials offer the most conclusive test of impact and
should more often be used in preference to non-random (matched) trials. (3.6; 4.2; 4.4)
- However, since random allocation is sometimes impracticable and unsuited
to addressing certain questions (such as why a particular outcome may
have occurred), a battery of other techniques should also be considered, either
on their own or in tandem. (4.4)
- Rigour is by no means confined to the quantitative testing of new policy
initiatives. Well-founded qualitative research among both beneficiaries and
service providers should also feature in a comprehensive pilot. (4.1; 4.4)
- The ethical demands of pilots cannot all be met via informed consent from
participants. Inequities between beneficiaries and non-beneficiaries and the
risk of negative consequences for some participants both need attention. Such
problems should, however, be addressed and mitigated rather than treated as
insuperable obstacles to rigorous experimentation. (4.3; 6.3)
Using pilot results
- A pilot that reveals a policy to be flawed or ineffective should be viewed
as a success rather than a failure, having potentially helped to avert a potentially
larger political and/or financial embarrassment. (3.4)
- Pilots should be regarded less as ad hoc evaluations than as early
stages in a continuing process of accumulating policy-relevant evidence. (6.3; 6.6)
- Appropriate mechanisms should always be in place to adapt (or abandon)
a policy or its delivery mechanisms in the light of a pilot’s findings.
(3.2; 3.4; 3.7)
- To ensure the effective exploitation of policy-relevant evidence, departmental
dissemination strategies should ensure that both the results and methods of
pilots are made freely available within and outside government. (6.4; 6.7)
- Post-pilot reviews should be routinely undertaken and published as a means
of sharing experience and developing methods. (6.3; 6.7)
- An accessible central electronic repository of pilot reports should be
set up to facilitate easy reference to past successes and failures. (4.4; 6.7)
Contents | Previous | Next
Top
