Government Social Research
This document presents a comprehensive research design for evaluating the Employment Retention and Advancement (ERA) Demonstration Project. The demonstration will test a new strategy for improving job retention and advancement for low-wage workers in Britain. It represents a potentially important step in strengthening the country's evolving welfare-to-work and anti-poverty policies. Although a number of current measures help those on the margins of employment retain work and improve their earnings, policy to date has largely focused on reattaching people to the labour market and makes less provision for assisting people once in work.
The ERA demonstration also promises to make a significant contribution to the process of evidence-based policymaking in Britain. It differs from many of the pilot projects seen recently in Great Britain, in that there are no plans at the outset to roll out ERA on a national scale before results of the evaluation are known. Moreover, the effectiveness of the intervention will be tested as a large-scale, multi-site, random assignment social experiment. Testing a cutting-edge-but unproven-policy innovation through a demonstration project utilising random assignment will help determine whether projects of this type can make a useful contribution to devising effective social policy in Great Britain, whatever the substantive policy focus.
As part of its role in encouraging excellence in policy, research, and evaluation design, the Cabinet Office led the effort to design both the intervention policy and the evaluation for the ERA demonstration. The design team has worked in close collaboration with officials in stakeholder departments-Department for Work and Pensions (DWP), Inland Revenue (IR), Department for Education and Skills (DfES), and HM Treasury (HMT) (who were all represented on the Project's Steering and Advisory Groups). DWP will lead the operational phase of the project through Jobcentre Plus and oversee its evaluation.
ERA will be directed at individuals in three different low-income groups known to have difficulty retaining jobs or advancing to better positions:
ERA will offer both pre- and post-employment assistance. For the two New Deal groups, the programme will begin before they enter employment; for the WTC groups, it will begin after they have started working. Once in ERA, all participants will have access to a combination of work-related services and financial incentives for a substantial period after employment commences.
Work-Related Services
One-to-one support for each participant from a dedicated Advancement Support Adviser (ASA) is at the heart of the ERA service. These advisers will attempt to identify and address the problems that are keeping participants from staying employed and advancing to better positions. The ASAs will start to work with the New Deal participants when those individuals are either mandated to join or volunteer for New Deal 25-Plus, or volunteer for New Deal for Lone Parents. The ASAs will begin working with the WTC lone parents after they volunteer for ERA.
The ASAs will provide guidance on or direct assistance with such issues as: finding a job or a better job; gaining promotion; understanding how much work, or an increase in hours and wages, can 'pay' in terms of net income; dealing with workplace demands and pressures; finding appropriate education and training opportunities that can be combined with part-time or full-time employment; and arranging for support services such as childcare or assistance with personal problems or family circumstances that can impede steady employment. The ASAs will work with participants for a maximum of 33 months. For the New Deal target groups, this allows for up to 9 months of pre-employment services and a minimum of 24 months of post-employment assistance.
Financial Incentives
The ERA financial incentives are intended to encourage retention in full-time work (which the literature suggests is a better route than part-time work to an eventual escape from poverty - see Annex 1) and the accumulation of skills through advancement-focussed training. Thus, the incentives include bonus payments for participants who work full time (i.e., at least 30 hours per week) and for those who combine education or training with work. Full-time workers will be eligible to receive up to six bonus payments of £400 each (£2,400 maximum) over the course of the demonstration. To be eligible for each payment, a participant must work at least three months out of a set four-month period. Participants can also qualify for up to £1,000 in tuition assistance and a maximum £1,000 training bonus if they take part in approved education or training courses while employed. The training bonus, which will be available only upon successful completion of an approved course, will be paid at the rate of £8 per classroom hour.
Meaning of random assignment in the ERA demonstration
In the ERA demonstration, random assignment means that individuals eligible for the programme will be given an equal random chance of being assigned to a programme or control group. Those assigned to the control group will not have access to ERA services or incentives for the duration of the demonstration. However, they will continue to be eligible for all currently available non-ERA assistance (e.g., New Deal and Jobcentre Plus services and the WTC.) As shown in Figure 1 in Chapter 4 of this report, individuals in the New Deal target groups, who will not be employed when they enter the study, will be randomly assigned at the point that they would normally enter the New Deal. In contrast, as depicted in Figure 2, members of the WTC target group, who will already be working, will be randomly assigned after they volunteer for ERA.
Why use random assignment to test ERA?
Testing ERA through a random assignment research design will provide the most convincing evidence of its quantitative impact on important outcome measures. This is because, when properly executed, random assignment ensures that there are no measurable or unmeasurable pre-programme differences (e.g., higher motivation, greater skills, better access to opportunities), on average, between the programme and control groups; thus, any subsequent differences in outcomes between them can confidently be attributed to the programme. For example, the ERA evaluation will compare the average number of weeks worked after random assignment among members of the programme group to the average duration of employment among members of the control group. Because the only systematic difference between the two groups will be that the former received ERA services, any difference in the amount of employment between them can be traced directly to ERA.
The ERA demonstration offers the chance to use a large-scale random assignment social experiment as a tool for policy development. It is therefore an objective of this project to test the usefulness of this approach with an eye toward applying it in other carefully constructed pilot evaluations of social policy.
Among the key purposes of the evaluation of ERA are to learn:
1. An impact study (Chapter 4) will determine the effects of the ERA programme on outcomes related to participants' employment, such as job-entry rates, employment duration, earnings, wage growth, job quality, total income, and poverty rates, and on a variety of non-economic outcomes of interest, such as personal and family quality of life, material hardship, and outcomes for children. These outcomes will be measured over a five-year follow-up period for members of the randomly selected programme and control groups (see above). The difference between the groups on each outcome will indicate the programme's effect, or 'impact.' The study will estimate ERA's impacts separately for each target group within each demonstration site where ERA is implemented (see below), and for all of these sites combined. It will also examine impacts for key subgroups within the three target groups (for example, subgroups with different levels of educational attainment before random assignment). The analysis will rely on administrative data on employment, earnings, and WTC receipt maintained by Inland Revenue; DWP administrative records data on transfer benefit receipt; and a multi-wave client survey. The survey will be conducted at 12 and 24 months after entry into the study, and, if response rates are adequate, a five-year follow-up survey will also take place. If response rates were to be inadequate, an evaluation of outcomes after five years could be done using administrative data.
2. A process study (Chapter 3) will determine how the ERA model is adapted to local conditions in each of the selected Jobcentre Plus agencies, and how it differs in practice from what is offered to New Deal participants and users of Jobcentre Plus. It will also measure how much ERA participants actually take advantage of ERA-provided services and financial incentives as well as any other employment-related services they can access on their own. Importantly, the study will compare the extent of participation in all such activities between the programme and control groups in order to determine whether the ERA causes an increase in skill-building efforts that might contribute to advancement in the labour market. The process study will also examine the kinds of problems that impede participants' steady employment and progression in the labour market, and how staff attempt to respond to these challenges. Data for this study will come primarily from qualitative interviews with ERA and Jobcentre Plus staff, a quantitative staff survey, participation tracking data maintained by the DWP and Jobcentre Plus, a subset of questions in the client survey that focus on service receipt and use of financial incentives, and qualitative interviews with a sample of employers.
3. A cost study (Chapter 5), will determine how much it costs per person to operate ERA and its specific elements (e.g., caseworker services and the financial incentives). The study will rely on DWP expenditure data for Jobcentre Plus and the ERA programme, published and unpublished unit cost information on other education, training, and support services, and programme tracking and client survey data on clients' receipt of ERA, Jobcentre Plus, and other available services.
4. A cost-benefit study (Chapter 5), using data from each of the other studies, will determine the net economic gains and losses generated by ERA over a time horizon of five years of more. The study will focus particularly on how much participants (who stand to benefit from the programme) and the government budget and taxpayers (who will pay for it) gain or lose financially. Net gains or losses from the perspective of employers and for all these groups combined (i.e., a societal perspective) will also be computed.
Overall, the ERA evaluation will draw on the findings from each of these four studies to provide an integrated body of evidence on the operations, effectiveness, and economic consequences of the ERA programme. For example, the impact study will show whether the programme generates any substantial 'added value' in retention, advancement, and quality-of-life goals compared to what would have been achieved in its absence-that is, by the existing provision of New Deal, Working Tax Credit, and other services and supports for work. The cost-benefit analysis will show the potential economic 'return' on the government's investment in the ERA programme, and the net cost of achieving desirable effects in addition to retention and advancement (e.g., poverty reductions and improvement in family and child quality-of-life outcomes). If, based on these analyses, Ministers are inclined to replicate ERA nationally, the findings from the process, impact, and cost studies can offer guidance on what programme strategies should be emulated-or avoided-and whether the programme in general ought to be targeted toward certain groups in order to enhance the chances that the replicated version will be well-run, effective, and cost-efficient. Alternatively, if the impact and cost analyses show that the programme does not work as well as expected, or is too costly, the process study may help point to the reasons why and, possibly, how the programme might be improved. In sum, the different components of the evaluation, when taken together, will offer policymakers and administrators a firm basis of information for deciding whether to institutionalise the programme as a regular feature of Jobcentre Plus nationally, and if so, how best to do this.
The ERA programme will be set up as a special unit within selected Jobcentre Plus Districts and evaluated as part of a multi-site social experiment. The project design team recommend that six areas (Jobcentre Plus Districts) be included in the demonstration. This number of sites is considered to be a practical number from a cost and management standpoint, and one that would allow the evaluation to determine whether the ERA programme can be operated successfully and achieve positive impacts on participants when operated under a variety of local conditions and by different staff. This test of the programme model's 'robustness' is important for gauging the possible effect of a wider rollout of the new measures, which would be of interest to policymakers if the results of the evaluation were positive. Multiple sites will also provide some opportunities to learn about the 'best practices' for operating an effective ERA programme.
With these considerations in mind, the following criteria were used in identifying specific locations to include in the demonstration:
The proposed sites are listed in Table 1 below.
Table1: Proposed Demonstration Projects
| Area |
Region (1) |
|---|---|
| Renfrewshire, Inverclyde, Argyll and Bute | Scotland |
| South East Wales | North West |
| Manchester | North East |
| East London | LASER (2) |
| Gateshead and Tyneside | East Midlands |
| Derbyshire | |
Notes: (1)
Government Office Regions |
|
These districts were selected from a list of 25 candidate areas where fully integrated Jobcentre Plus offices are currently, or will soon be operating. The project design team sought to avoid areas that are likely to be implementing the fully integrated Jobcentre Plus model -a substantial administrative reform-at the same time that the ERA programme would need to be implemented. For that reason, consideration was given only to the 25 districts due to have been operating the Jobcentre Plus model for at least six months before the scheduled ERA programme start date of October 2003.
Recruitment and intake for the ERA programme would last for about one year. It is estimated that during this time, almost 27,000 individuals across these six locations and three target groups could be enrolled in the study sample and randomly assigned to the programme or control group. Half the sample (nearly 13,500) would be assigned to each research group. The total research sample in each site would range from about 2,200 individuals (1,150 in each research group) in South East Wales to about 6,600 individuals (approximately 3,300 in each research group) in Derbyshire.
Administrative records (e.g., earnings and benefit payment data) will be collected on all individuals who are randomly assigned. However, the client surveys will be administered to a representative subsample of these groups, with the total number of respondents projected to be approximately 4,800 across the six demonstration sites for the survey at 24 months.
The final decision on site selection will be made by the DWP and will be one of the crucial early decisions of the implementation phase. In addition to the research criteria specified above, DWP will need to consider each area's management capacity and administrative potential for operating a high quality ERA programme-one that is true to the programme design, can be implemented on the timetable required by the demonstration, and that merits a rigorous evaluation. A further consideration is the location of other pilot programmes at the sites.
Programme intake is envisaged to begin in October 2003 and continue through to September 2004. All ERA services will cease by the end of June 2007 (33 months after the end of intake in September 2004).
Results from the random assignment impact study could emerge as early as Spring 2005. These early findings will be based on administrative data and constitute limited and tentative comparisons of employment rates and levels of benefit claiming. Some partial information may also be available at this time on earnings. Earlier analysis than this of administrative data will be indicative of the general progress of the demonstration, for examples, whether the numbers coming forward and the numbers entering work are within expected ranges, but it will not provide any estimates of net impact.