Social research in practice
GSR as a catalyst for collaborative working
How to have an impact - the trials and tribulations of a GSR
Having an Impact
Developing Our Skills
The role of qualitative research
Parallel Session 1, Strand 1
Theme: Shaping Policy and Delivery
Social research in practice
Using research to deliver tax credits
- Victoria Hughes, HM Revenue and Customs
- Working with policy development (HMT) and delivery (HMRC) customers.
- HMRC formed as a department in 2005 after the O’Donnell review.
This brought together the former work of the Inland Revenue and Customs
and Excise and also forged closer links between both departments and HMT.
- Working with HMT colleagues provided more opportunities - colleagues
weren’t familiar with working with researchers.
- Work on tax credits - 6 million household claim tax credits, with the
household income in the previous year determining what level of tax credit
the household receives. If levels of income change the household needs
to tell the HMRC so that correct payments can be given.
- Part of the research role on this piece of work involved telling policy
colleagues about what could realistically be produced within a very short
timescale The research involved a quick turnaround with complex methodology
- cognitive interviews, qualitative research and two stages of quantitative
research using a triangulation of research methods.
- Findings - quantitative data is currently being analysed. The findings
will feed into the policy delivery decision making.
- Challenges -had to work to tight deadlines and the benefits of research
had not been seen by some of the more recent customers in HMRC and HMT.
- Opportunities - has seen the research process from start to end, ensures
providing research is fit for purpose, gained an understanding of policy
work and the time constraints they work to, and results have been fed
into a paper for the Pay.
Evaluating a complex funding programme
- the experience of the European Refugee Fund - Carolyne Ndofor-Tah, Home
Office
- Discussion of experience of the Refugee Fund (2005) - where refugees
encouraged to maximise their potential.
- Need to ensure evaluators understand what policy customers thought
they planned to do - feasibility study.
- Findings - project needed to clarify it’s exact objectives. It
was difficult to link intervention to the outcome and was hard to evaluate.
It was not possible to do an RCT and there was no control group.
- There was a need to develop clearer real aims.
- Next steps - process evaluation impact - helped to make changes as they
went along.
The value of pilots
- case study of the Dedicated Drugs Courts pilot - Greg Braun, Department
for Constitutional Affairs
- Examination of the value of pilots run in the DCA - recent increase
in high profile, high cost pilots. Increase most significant in crime
and criminal justice field.
- Dedicated Drugs Court pilot - 2005 process/outcome evaluation commissioned.
- Interim evaluation results are emerging
- Advantages of pilots - lots of interest created, can highlight additional
benefits not previously thought of, identify best/good practice, evidence
for a national roll out.
- Challenges - can you replicate the energy and enthusiasm in the national
roll out? Funding - the same for the roll out is not likely.
- The future - ministers aren’t waiting; and possibility of change
in leadership of government.
- Some answers; There are limited budgets for pilots. Need to ensure communications
for the expectations of ministers, make pilots more ‘roll out’
orientated from the start, be willing to accept that pilots might not
work but for good reasons.
Supporting Policy -
the role of statistics - Deborah Brown, Northern Ireland Statistics and
Research Agency
- Neighbourhood Renewal 0 list of areas to be targeted was created.
- Need to focus limited resources in more efficient ways.
- Co-ordinate action to reduce inequalities in urban communities and promote
social inclusion.
- 4 strategic objectives
- Community renewal - develop confidence
- Economic renewal - develop economic activity
- Social renewal - improve social conditions
- Physical renewal
- Had to work with two different deprivation measures
- Managed to help 1 in 6 people.
back to top
index
GSR as a catalyst for collaborative
working
Workplace employment
relations study - Barbara Kesley, Department of Trade and Industry
This longitudinal study mapping employment relations is a collaborative
multi-sponsored survey, which has been running since 1980.
- The collaboration includes a steering group of the DTI and academic
and research bodies (ESRC, PSI, ACAS), as well as the contractor, NatCen;
- A successful collaboration is enabled by a number of factors including
a MOA detailing membership roles, responsibilities and dissemination of
findings as well as good working relations and regular meetings are between
researchers;
- The main benefits of collaborative research are the ability of the research
team, with its range of different disciplines, to cover a wide range of
issues, with a degree of innovation. This also allows costs to be spread;
- However, a wide steering group, with even wider consultation, can lead
to raised expectations and delays around decisions about costs, both of
which place a burden on scope and timings;
- In the future having a tighter MOA definition of roles and dissemination
strategy could improve relations and better manage expectations;
- Additionally, a stable research team, with members having a commitment
for the life of one cycle (4 years) was considered important.
Collaboration as an important cross-cutting
issue - New directions for environment and health research in Scotland -
Julie Carr, Scottish Executive
- A case study of Knowledge Transfer (KT) and social research (SR) in
the context of environment and health was presented.
- KT is about bringing policy, practice and research interests together
to make best use of evidence for policy-making;
- Environment and health became a pilot for cross-boundary working and
tested the possibilities of working in the gap between the policy, practice
and academic communities;
- 2006 was recognised as an opportune time for getting the right people
(a wide range of key internal and external stakeholders) together at the
right moment;
- Committed SRs acted as deal breakers for KT (i.e. knowledge brokers)
to facilitate collaboration on environment and health;
- Collaboration was encouraged through an information day, which included
workshops, networking and facilitated ‘speed dating’, with
SRs acting as catalysts for KT;
- Successful outcomes to date include a new SR post in the environment
team, ongoing engagement with key stakeholders, as well as KT input to
other cross-cutting policy areas.
Building Capacity and Infrastructure for
research- CRC Cymru - Susan Denman, Welsh Assembly Government
Clinical research collaboration (CRC) - Cymru was established to provide
evidence for health and social care policy and practice in Wales, following
a review of the current arrangements in 2003.
- The CRC aims to provide high quality research and evidence, and improve
the translation of clinical research findings into policy and practice;
- An established academic network in Wales provides policy review, primary
and secondary research, and critiques research methodologies. The CRC
also have a responsive grants scheme and fund fellowships for new researchers;
- The set-up of the CRC is tailored to recognise and meet local evidence
needs, as well as maximising strengths and opportunities within Wales;
- In the future the CRC aims to increase the number of rigorous large
scale studies, some of which will utilise admin data and therefore minimise
the need for primary research;
- Successful outputs will allow a clear demonstration of the benefits
of investment in research, with policy automatically supported by evidence.
back to top
index
How to have an impact - the trials and tribulations
of a GSR.
Deliberative Polling:
A Case study: Researchers walk the tightrope: National Pensions Day - Michael
Kelly, Department for Work and Pensions
- National Pensions Day - looking for consensus on pension reform plus
ministerial appetite for wider public interaction
- Balancing rigorous research with Ministerial & policy requirements
and deadlines
- Putting information out there - challenges of working with information
officers, speech writers to ensure the correct use of information.
- Deliberative polling methodology developed by Prof James Fishkin: a
sample of the population vote on issues before and after a process of
deliberation - pensions ideal subject - complex, boring, inaccessible,
but needs to be grappled with.
- Whole day for people to immerse themselves in information - used a
framework approach - 4 options - pensioner poverty; higher taxes; work
longer; save more.
Tensions:
- Robust research Vs communications exercise - timescales & resources,
collaborative approach to working with ministers, policy communications,
contractors and analysts
- Representativeness Vs feasibility - quota sampling and recruitment
issues
- Deliberative approach Vs feeding into policy - biggest challenge was
developing appropriate, interesting and comprehensive stimulus material.
- Pilot events saw the loss of healthy lunches and the afternoon stretch
session!
What happened?
- 1000 people, 6 locations - linked by satellite. High profile event -
lead domestic news story on the day (18 March 2006)
- Findings confirmed proposed policy direction and provided evidence
for Pensions White Paper
Social research reflections
- Social research expertise added value to the day - need to ensure policy
and ministers buy-in to the expertise Social Research can provide from
the start
- Wide range of people involved so a good chance to raise profile of
social research within DWP
What works..? - Gita Anand, Scottish Executive
- Expectations of a new Government Social researcher - rhetoric v reality
- Pleasant surprises - more training opportunities, part of large research
community, working with other analysts
- Disillusionment - Analytical and Policy working relationship not exactly
straight forward!
One Example - Evaluation of Rural Community Gateway website
- Project Inception - what evaluating against?, agreeing aims and objectives,
what information could be collected, research design, evaluation criteria
to produce the Final Research Specification
- Project management - Policy and research different requirements time
Vs robustness, managing expectations
- Feedback - internal and external
What made it work with Policy colleagues
- Negotiation, confidence, trust, flexibility, translating big plans
into something manageable. In return Social Research was offered a seat
on the policy delivery steering group.
- Social Research as a body of knowledge that can contribute to policy
delivery as well as informing policy development
- Model of close working SR and policy - not always transferable as it
is dependent on relationships and personalities as well as structure
Influencing Policy Colleagues: Using Findings
from an Overseas Study Trip - Dr. Stella Mascarenhas-Keyes, Department of
Trade and Industry
Study trip in June and July - speaking at the International Small Business
Council conference in Melbourne and Lectures in India
- Getting out there - attending and speaking at events to raise your
profile
- Evidence base is not just research spent the study trip collecting
information and seeing how it is used in policy in other countries
- Ascertained policy interest before trip - networking and engaging interested
parties - looked at wide ranging policy issues and what was coming up
on the policy agenda
- 2/3 months before trip wrote paper and made contacts - writing in advance
to arrange meetings
Activities during trip
- Networking, exchanging information, semi-structured interviews, collecting
documents, collecting and annotating business cards and more networking!
- Typing up interviews
Feedback from trip
- Presentations and meetings on return to UK to analytical and policy
colleagues - disseminate to as wider range of people as possible
- Contributed to Pre- Budget review - putting forward new policy ideas
- proposed a pilot project based on an Indian model, to enable access
to finance for disadvantaged groups to start/grow businesses in UK
back to top
index
Having an Impact
Increasing GSR’s
impact - Teresa Williams, GSRU
- GSRU’s strategic direction, activities and what GSRU and GSR members
can do to contribute.
- Key challenges faced e.g. the scepticism by some of the use of social
research.
- The importance of professionalism e.g. the development of professional
communication skills, setting standards e.g. framework agreements and
having a stronger collective voice across analytical professions were
discussed.
- The challenge that PSG poses i.e. the increased demand for analysis
and that some senior policy colleagues think that they can do this alone.
We need to work with them to inform them what it is that we offer. A self-assessment
tool for policy makers is being developed and piloted.
- Appreciate that one size does not fit all, e.g. a training course is
not always appropriate or affordable.
- GSR members can contribute by making sure that their details are up
to date, emailing issues they want to see addressed and by participating
in the forthcoming GSRU questionnaire (which will be sent to all government
analysts), the results of which, will hopefully feed in to a planned conference
for the heads of the analytical professions.
Increasing the impact of our national
studies programme - Roger Sykes, Audit Commission
- Office for Public Management project on increasing impact.
- Local authorities are more likely to use national research if it is
currently relevant to them, the challenge is getting the right bit of
information to the right people at the right time.
- The AC has started to produce e-bulletins on their research projects;
this is more participatory and helps to plug the gap between the research
and the production of the final report.
- Research is only one of the evidence strands used.
- It was posed that the current model was 90% research and 10% dissemination,
should this be changed to 70%/30%? Are we skilled to do this? Do we want
to?
- We should be trying to develop a range of products from a national
report, as on its own it is not enough (especially to influence at the
local level).
- We should be committing resources to active dissemination.
- It can be difficult working with press and publicity teams. It was
suggested that for this to work most effectively they have to be involved
from the start.
- Finally this is something that the AC might consider running a workshop
on but is more than happy to speak further to individual departments.
back to top
index
Parallel Session 1, Strand 2
Theme: GSR and it's research at the leading edge
Developing Our Skills
The issue: Credibility of researchers measured by ability
to deliver timely research. However, often only become involved at a late
stage in policy development process when process has been agreed and difficult
to deliver on.
The process: Early involvement in policy development allows
policy and analysts to better understand strengths of each other. Frequent
communication improves confidence in each other and helps to manage expectations.
A well developed, systematic project management strategy is crucial.
Remedies: GSR competencies enable us to add value to the
work of policy. Build on professional knowledge and be prepared to find
the answer - networking very important. Commit to systematic project management
- visit the Programme and Project Support Office website for information!
back to top
index
The role of qualitative
research
Using qualitative research
to explore redesign options for the child support system - Simon Franklin,
Department for Work and Pensions.
Background
DWP commissioned a qualitative research project to understand parents’
preferences for a re-designed child support system. It was a small qualitative
project, made up of 5 focus groups with around 30 respondents.
The views of parents were closely aligned with the departmental approach
and consequently the results were widely reported. There was particular
interest from Special Advisers and press office who wanted to generalise
the results and claim that the government’s plans were in line with
wider parental views.
Main Issue
How do you raise the profile of qualitative research but also address the
danger of policy, press and special advisers, over emphasising the results
and generalising them to the wider population.
Questions from the audience
- Recruitment- how were people recruited for the groups? Did the recruitment
process adequately cover different parental views, reflecting both mothers
and fathers etc?
- Topic Guide - were the questions open enough?
- Role of the Moderator: did the results truly reflect the views of all
focus group participants, so that 1 member didn’t dominate the discussion.
Would in-depth interviews have been more appropriate to stop any contamination
effect?
The early stage integration for resettled
refugees brought to the UK - a qualitative evaluation - Mark McConaghy,
Home Office.
Background
The Home Office has commissioned a longitudinal study of resettled refugees’
experiences of settling in the UK. The purpose behind commissioning longitudinal
qualitative research was to understand emerging issues from the perspective
of the refugees and for them to be able to reflect on their changing views
and experiences at follow up interviews. Early research results (interviews
at 6 weeks, and at 6 months and at 18 months) were reported at the conference.
Main Issue
Issues faced when conducting a longitudinal qualitative study and how the
results fit with a quantitative study and are used and reported to policy
colleagues.
Questions from the audience
- Did the refugee route of entry to the UK influence their views and
experiences?
- Attrition could bias the results - how has this been managed?
- Our methodologies need to be more culturally sensitive so that the
type of questions we ask are interpreted as intended.
- Issues of respondent passivity and compliance - as the Home Office
was sponsoring the research but has an operational immigration role.
General Discussion:
What could GSRU do to promote a greater understanding of the role
of qualitative research?
The audience generally welcomed the idea of a GSRU Qualitative Research
Manifesto.
It would need to include:
- a clear theory of qualitative research which could be used to inform
sceptics, including other analyst groups such as economists and statisticians;
- information about how qualitative and quantitative research complement
each other;
- guidance which ensures people are confident about using qualitative
research methods;
- evidence that qualitative research can be used at different stages
in a project: at the beginning to help formulate hypotheses - tested using
quantitative research- and used again at the end of the project to gain
a greater why people have particular attitudes or behave in certain ways;
- guidance on analysing and interpreting qualitative evidence as distinct
from anecdotal evidence and opinion-based polls;
- the limitations of generalising from qualitative research without undermining
the approach;
- information about ‘private capture exercises’, where individuals
write down their views as well as expressing them in the focus group to
see whether the summary actually reflects the views of the group; and
provide
- examples of well-conducted qualitative research and how to distinguish
between good from bad qualitative evidence.
back to top
index
