The role of participants
Citizen Research
Inclusive Research?
The role of research in encouraging democratic participation
Maximising response and minimising burden
Action Research
Parallel session 3, Strand 1
Theme: Looking beyond our direct clients
The role of participants
Ethical research - the voice of research
participants - Jane Lewis, National Centre for Social Research
Background
This research was conducted to understand research participants’ concept
of ethics and how it related to their experience of taking part in interviews.
Main Issues
- Decision making process stage - information needed for people to decide
whether to take part: role of official letters; understanding who was
funding the research compared with who conducted it and what written consent
actually meant to the respondent.
- The interview interaction - being made to feel comfortable and not
being intimidated by the interviewer; no right and wrong answers;
- Style of questioning - clarifying why interviewers probe; why questions
are sometimes repeated which appear to challenge the validity of respondents
answers but is in fact a checking mechanism; and why seemingly personal
questions about income and family life are asked.
Questions from the audience
- How do we balance the need for interviewees to know that they don’t
have to answer questions, against the need to reduce non-response and
produce high-quality research?
- Is there potential for respondents to give feedback on draft reports
or more generally to see the final report so that they understand that
their time and views have been utilised?
- Do we want more active engagement by participants rather than seeing
the interviewer as the expert?
- People don’t trust government but seem to trust researchers working
on behalf of government. Why is this?
- Were NatCen objective enough to review research projects which they
had originally conducted?
Children’s participation - the researchers
of tomorrow - Jennifer Shepherd, Welsh Assembly Government
Background
This presentation looked at the evaluation of the work of the Children’s
Commissioner for Wales and also a project to support children in primary
school to design and carry out their own research. Both projects consider
how research can be developed to include the child’s perspective and
have provided useful insights for researchers and the effective involvement
of children in research.
Main Issue
The Children’s Commissioner project wanted to involve children in
the design of the evaluation. It has been able to develop a design that
children have agreed and want to be involved in. To facilitate this it was
necessary to provide skills and knowledge to a broad range of abilities
of children and young people, to help them make informed decisions. The
resources required with this approach raise questions about the value of
involving children when a similar research design may have been established
without their participation
The Young Researchers work also involved providing children with skills
and knowledge, this time to develop their own research projects. It has
enabled children to take responsibility for a project of their own, though
was reliant on supervision and guidance from teachers, which may have influenced
the work that the children have done.
Questions from the audience
Children’s involvement with research relied on their voluntary participation,
in addition to their statutory school work. This commitment is likely to
have affected the type of children volunteering for the project.
back to top
index
Citizen Research
‘Public perceptions in a devolved
Scotland’, Iain MacAllister, Scottish Executive
Iain MacAllister gave an overview of the Scottish Social Attitudes Survey
(SSAS), provided results of the 2004 survey, and some comments on the 2005
survey. The main points were:
- The SSAS surveys a representative sample of 1,500 people. The 2005 survey
included modules on the NHS, old age and national identity.
- This is important as it provides a time series of data on questions
of Scottish interest, to inform policymaking.
- In 2005 questions were added on attitudes towards public services, trust
and involvement in governance and perceptions of government.
- Results indicate that priorities are Health and Crime. Interestingly,
there are ‘local priorities’ as well national priorities.
According to the SSAS, these local priorities include: facilities for
young, affordable housing, crime, quality of jobs, good quality local
transport.
- The SSAS examines perceptions of public service performance. There are
differences in how people perceive the broader ‘service’ e.g.
National Health Service, compared to how they perceive individual services
that they are familiar with and have had experience of (e.g. GP services,
which 7 in 10 people were satisfied with ).
- Experience of services seems to be important in determining satisfaction.
For instance, those with experience of education (e.g. those with young
children) are the most positive about the quality of education.
- In the SSAS, people were asked to evaluate services, but there are issues
over whether they actually know who runs them. For instance, blame for
poor performance of the NHS is given to the UK Government, but it is a
Scottish Executive remit. Good performance generally is attributed to
the Scottish Executive.
- Trust in the Executive was high, although knowledge of its powers low.
As a result, the Executive is doing work to educate people about its powers.
- Comments from the floor included the fact that the results of this echo
those from GLA and Electoral commission research. There was also a question
about possible regional variations, however the response was that the
sample size is not large enough for any detailed breakdowns. There has
not been much work done so far to tie the SSAS in to BSAS.
Measuring the performance of public services:
capturing citizens’ experiences’, Leighton Phillips and Richard
Thurston, Welsh Assembly Government
Leighton Phillips and Richard Thurston asked the question “How good
are we at capturing citizens’ views of public services and using them
to drive improvements?”, and outlined research being undertaken by
the public services reform unit - ‘Making the Connections’ -
and the Office of the Chief Social Research Officer at the Welsh Assembly
Government. The main points were:
- Arguably, although social researchers are in an ideal position for capturing
citizens’ voices, we are currently in a state, of ‘conscious
incompetence’ - ie. we are aware that we need to improve our measurement
of citizens’ experiences of public services but we aren’t
yet sure how best to go about it. As social researchers we can contribute
much to a debate about measuring citizens' views - not least through critically
exploring the appropriate methods. For example, we might argue that there
is an over-reliance on ratings of satisfaction - which may not be adequate
alone to help identify what needs to be done to improve services.
- Devolution is playing a role in motivating and shaping the Assembly
Government’s interest in what citizens think about public services.
As this is relatively new, however, more thinking is needed in relation
to how to measure citizens’ experiences of public services. Another
issue related to devolution is the relative comparability of citizen satisfaction
data between countries and the need to coordinate at a UK level.
- Making the Connections sets out the Assembly Government’s approach
to public service reform. Citizens are seen as the key driver to the improvement
of public services, through having a stronger voice in service design,
management and delivery of service and better information on how services
are performing to challenge public bodies when services aren’t good
enough. The ‘Making the Connections’ research programme was
established to find out peoples’ views of a variety of public services
to help plan, and over the longer term monitor, reforms. There are a number
of strands to the research agenda to ensure that the views of a wide range
of people (e.g. people in different socio-economic circumstances) on a
number of aspects of service delivery are sought.
- In conducting citizen research, there are a number of things that need
to be considered. For instance, there are some groups for whom it is difficult
to capture voices e.g. how do you capture the voices of children in educational
inspection? Also, how do you ensure a balance between local and national
research, and what methods should be used in the context of European funding
pressures, the need to avoid narrow approaches, and the fact that local
authorities in Wales have little capacity to undertake their own work.
- For the first time in Wales, the Assembly Government has commissioned
a survey of citizens’ experiences of eighteen public service areas,
including schools, hospitals, recycling facilities, and dental practices.
Acknowledging the time it takes to establish a citizen survey and the
difficulty in seeking the views of ‘hard-to-reach’ groups
through household surveys, a programme of qualitative research has also
been undertaken. Over 350 people have been engaged through the qualitative
research and asked: what good customer service means to them?; what decisions
relating to public bodies they want to be involved in?; and how they want
to be involved?
- Care has been taken to maximise the dissemination opportunities for
the research results through the use of targeted presentations at, for
example, senior politicians and public service leaders in Wales. Innovative
techniques have also been used to capture the attention of the audience,
for example, a film has been produced to illustrate the research findings.
- A question from the floor was whether there are similarities in the
questions used with the English citizenship survey. A lot of the questions
were sourced from existing surveys, although there was no direct co-ordination
with this. For information, it was pointed out that in the Canadian government’s
work on customer satisfaction they control for expectations.
‘The Citizenship Survey’,
Robert Rutherfoord, DCLG
Robert Rutherfoord presented an overview and some results from the 2005
Citizenship Survey. The main points were:
- The Citizenship survey was established to meet the needs of the ‘communities’
policy agenda of 2000 and to monitor national PSAs. It is a household
survey of adults (10,000 adults, plus a boost of 5,000 respondents from
minority ethnic groups). It covers England and Wales, and is conducted
every 2 years.
- The survey covers a broad range of areas, from family and friends, to
volunteering, perceptions of community, to fear of crime.
- It also includes questions on community cohesion. Factor analysis shows
that there are relationships between feelings of cohesion and: sense of
community; similar life opportunities; respect for diversity; political
trust; and sense of belonging to the locality. There is also a relationship
between social cohesion and the deprivation of an area, with fewer people
living in deprived areas reporting feeling a sense of community cohesion.
- Alongside the Citizenship Survey, Local Area Surveys are also conducted
to monitor community cohesion. In 2005, (and planned for 2007), 10 areas
were surveyed.
- There are some interesting findings in relation to racial prejudice.
48% people feel that there is more racial prejudice today than 5 years
ago. This is an increase from 43% in 2001. However, non-white groups are
not as negative about this as white respondents - the increase in people
stating that this was the case was wholly in white respondents. People
who live in diverse areas are more positive.
- The results on discrimination, participation and satisfaction with local
services were described - see slides.
- There is currently exploration of how the survey can be linked to other
data sources. For instance, a JRF funded research project is using it
to look at small area level impacts to identify what determines place
attachment, neighbourhood instability and social mix, and the implications
of these for regeneration policy.
- The 2007 survey is moving to a continuous design, which can be used
for quarterly tracking and performance management at a national level.
- A question from the floor was whether, as the remit of the survey is
so broad, different departments own the different questions. This is not
the case, although there is informal ownership of sections of the agenda.
back to top
index
Inclusive Research?
Developing new survey methods for cross-cultural
research - Liz Pendry, Home Office
The Longitudinal Survey of Refugees is a new survey that will provide evidence
on the nature and extent of integration of different groups of refugees;
the process by which integration is achieved; and what influences, helps
and hinders the integration of refugees into UK society.
A questionnaire has been developed using National Statistics Harmonised
questions, questions from other national and international surveys; and
themes from research with the Gateway Protection Programme Refugees.
The survey needs to be inclusive for three reasons: to ensure representativeness,
minimise any bias and ensure validity of results. To ensure inclusiveness,
interviews will be carried out in the first language of the refugee. Where
this is not possible, interviews will be carried out using interpreters
with the key terminology being translated. It is proposed that cross-culturally
equivalent translated questionnaires be developed using TAT - Translation,
Adaptation and Tailoring. This approach overcomes some of the problems encountered
with traditional translation methods (such as translation/back-translation)
and realises more cross-culturally equivalent translations.
Before the translation, cognitive testing of the questionnaire was carried
out. This approach offers a way of focussing on how people answer questions,
explores the thought processes involved and identifies covert as well as
overt problems with the questions. The researchers concluded that conducting
cognitive interviews has value, even where the participants have limited
English and was also useful in helping to identify any cultural and population
specific issues with the questions.
People with disabilities in Northern Ireland
- how many are there? - Carly Niblock and Amanda Filmer, Northern Ireland
Statistics and Research Agency
The project followed a review of existing sources of information on disability
in Northern Ireland, which recommended a comprehensive survey on the prevalence
of disability in Northern Ireland and the experiences and socio-economic
circumstances of people with disabilities in Northern Ireland. In response,
the project aimed to provide an up-to-date, accurate picture of the type,
severity and prevalence of disability, the socio-economic circumstances
of people with disabilities and the experiences and barriers faced by people
with disabilities. The purpose of undertaking the project is to provide
information to policy-makers and providers delivering services to people
with disabilities.
To achieve this aim a survey was developed with the input of users and
potential users of such information. This included groups representing people
with disabilities.
Developing the survey included research into international current practice,
including an assessment of techniques and definitions. The survey was based
on the International Classification of Functioning, Disability and Health
(ICF) which considers disability in terms of the interaction between an
individual with a health condition and the barriers and facilitators in
the surrounding environment. The approach has been adopted by a number of
nations and has the advantages of being flexible; being broad in scope,
combining social and medical model of disability; the possibility of comparison
across nations; and the approach is endorsed by the World Health Organisation
and it’s member states. Questions were based on international surveys
and ONS harmonised questions to facilitate comparability.
The survey was designed to respond to diverse customer needs and to be
inclusive. Inclusiveness was viewed in terms of the covering essential topic
areas, ensuring the relevant people were involved from the start and ensuring
the language used was in the questionnaire was ‘plain English’.
The survey targets adults and children in both private households and communal
establishments to ensure inclusiveness. Communal establishments are more
likely to have a higher incidence of people with disability than are private
households. It was necessary to tailor three separate questionnaires for
adults, children and residents of communal establishments, as the needs
and experiences of these populations may be quite different.
There is no sampling frame containing our target sample. The target population
had to be screened from the whole population of private households and communal
establishments. The survey adopted a sampling approach used in Canada which
used two filtering questions to identify a sample group based on ICF principles.
The questions from the Canadian Participation and Activity Limitation Survey
were piloted to test their reliability on identifying our target sample
. In order to measure false negatives a more detailed follow-up questionnaire
was asked of all households who responded negatively to the two filter questions.
A false negative is someone who incorrectly screened out when they actually
have a difficulty and should be screened in. The survey was administered
using a mixed method approach. Face-to-face and telephone interviews were
used. The researchers found using a mixed method approach offered flexible
options for people to participate in the survey and provided the researchers
the opportunity of contacting more people.
The results of the pilot suggested that when the two screening questions
were applied to a NI population using the mixed mode approach they could
not be relied upon to identify the target sample.
In the absence of any other quick way of identifying our target sample
we started focussing on the structured follow up questionnaire used in the
pilot as a possible screening questionnaire, we also consulted widely with
the disability sector to identify 15 categories which the structured screening
questionnaire was developed around. After testing these questions we decided
to use the structured screening questionnaire as it was most likely to yield
a representative and inclusive target sample.
The Main Survey of Private Households is currently in the field with 85%
of fieldwork completed to date.
The Communal establishment strand of the survey is yet to start as we are
awaiting ethical approval first before carrying out research in NHS ran
establishments.
Headline results for the private household strand should be released early
2007.
back to top
index
The role of research in encouraging democratic
participation
Putting stakeholders on the map - Claire
Hinchcliffe, Scottish Executive
Claire’s presentation discussed what stakeholders are, how they might
be identified, and how to prioritise and map them by way of their interest
and influence. Key points included:
- The definition of stakeholders is very wide and yet narrower definitions
don’t really exist.
- Self-identification by stakeholders can be more useful than imposed
definitions, especially when the imposers represent the power broker.
- But people may self-exclude and so our role is to promote equality
of access and encourage participation .
- One useful way of prioritising stakeholders is to use Mendelow’s
approach and score each stakeholder out of ten for interest and influence
and then grid the results.
- But this approach does have problems because it suggests we should
focus on those with high interest and influence as key players, but the
group who have high interest and low influence should form the focus of
any engagement strategy.
- Conclusions then are that:
- who identifies and prioritises stakeholder is important;
- mapping does not equal engagement;
- it is important to devise strategies to encourage and mobilise
engagement;
- there is a role for social researchers to help policy colleagues
think about issues of engagement, equality of access and participation.
The role of social research in developing
the civil rights of disabled people - Grahame Whitfield, Office for Disability
Issues
Grahame’s presentation covered the role of the Office for Disability
Issues (ODI), how evidence underpins their work and the importance of the
forthcoming Equality Duty. Key points were:
- Disability affects a wide range of people, with 10 million disabled
people in Britain facing a large range of disadvantages in society.
- ODI’s mission is to put disabled people at the heart of policy
making in order to achieve the Government’s vision of substantive
equality for disabled people by 2025
- ODI organises its work around four key themes : childhood and family
support; transition to adulthood; employment; and independent living.
- Evidence underpins strategy in all areas because ODI need to know whether
changes being made have made a positive impact on outcomes for disabled
people.
- All public bodies (including central Departments) will need to have
a Disability Equality Scheme (DES) in place by December 2006, this will
affect all areas of government policy making and service delivery and
will need to involve disabled people.
- All Secretaries of State will have to report on their DES every three
years and the forthcoming Commission for Equality and Human Rights will
monitor compliance.
- ODI published their first annual report in July 2006
- Next steps for research and analysis include using and developing existing
sources of data, developing new data sources (including a new longitudinal
survey of disability), developing outcome based indicators and improving
the availability and accessibility of evidence.
- ODI will provide advice and support on involving disabled people in
and the coverage of issues relevant to disabled people in research
- Further information is available at www.officefordisability.gov.uk
back to top
index
Parallel session 3, strand 2
Theme: GSR and its research at the leading edge
Maximising response and minimising burden
Researching large complex organisations
- what works? - Ellen Springall, HM Revenue and Customs
- The focus of this presentation was on researching large corporates (i.e.
large multinational companies)
- HMRC carries out research on large businesses for a number of reasons,
including collecting detailed tax data and gaining a greater understanding
of their tax behaviour.
- HMRC are concerned about low response rates and an increasing research
burden on large businesses.
- Informal discussions with large businesses have shown that they are
generally ‘anti’ survey research; they wanted greater transparency
about the nature of the research; to know what the benefits are to themselves
and the company of taking part; have named correspondence; to have advanced
warning about forthcoming research requests; were not very concerned with
anonymity; thought findings should be more interactive; only wanted to
be interviewed by experienced and skilled researchers who knew the topic
area in detail.
- Some of these perceptions and recommendations will be easier to address
than others.
- As a consequence of this work, aiming to reduce burden by co-ordinating
and prioritising research needs, developing a research strategy; making
better use of existing evidence, review of sampling protocol (e.g. may
consider over-sampling in future).
- Trying to maximise response by communicating their research more widely,
reviewing the use of methodologies (e.g. alternatives to CATI surveys)
reviewing the type of research contractors used.
The ‘Taking Part’ Survey:
An experiment using incentives - Rebbecca Aust, Department for Culture,
Media and Sport and Joel Williams, BMRB
- This presentation gave the reasons for and the results from an experiment
using incentives.
- It was expected that there could be a low response rate to this survey
for several reasons - the general trend of decreasing response rates;
the lack of coherence due to its modular structure; sport and culture
may be off-putting to some; it lacks the aspect of civic responsibility.
- For these reasons an incentive was considered, however the literature
did not provide a definitive answer, and so an experiment was conducted.
- The experiment looked at unconditional incentives (sent out with advanced
letter) compared to conditional incentives (dependent on a completed survey),
cash compared to high street vouchers and the additional impact of re-issuing
initially unproductive addresses.
- The aim was not simply to increase response rate but to reduce non-response
bias.
- The conclusion was that - in this case - incentives proved a relatively
inexpensive way of reducing non-response bias and that the more expensive
re-issuing process did not add anything extra.
- Weighting for non-response bias was also discussed; however this is
regarded as a fairly blunt tool and not an absolute solution.
back to top
index
Action Research
Connecting policy, practice and community
through action research: Lessons from the Sustainable Health Action Research
Programme in Wales - Kaori Onoda, Welsh Assembly Government, Professor Steve
Cropper, Keele University and Professor Robert Moore, Liverpool university,
SHARP project representative.
Kaori Onoda (Welsh Assembly government), Prof. Steve Cropper (Keele University)
and Prof. Robert Moore (Liverpool University) offered different perspectives
on the Sustainable Health Action Research Programme (SHARP) in Wales, under
the title ‘connecting policy, practice and community through action
research’. Kaori highlighted some organisational and policy principles
that had proved successful for the central co-ordination team. The fact
that the programme was driven by research rather than policy or service
delivery and had a primary aim of learning rather than demonstrating impact,
lead to flexibility and openness and allowed for learning through interventions
not working. A supportive but non-intrusive model of central management
allowed a wide variety of projects to develop.
Robert identified the strengths of the programme as its ‘unprecedented
freedom’, generating an imaginative, innovative and broad range of
projects. He used a SHARP project based on a single housing estate to illustrate
his understanding of action research as ‘a very grounded and getting
up close’ to actions and processes behind social patterns that are
familiar at the macro level. He emphasised the need for action researchers
to see themselves as part of the researched population and to try to make
changes, rather than as a traditional detached observer. Total immersion
was essential to be able to identify the action needed to combat entrenched
processes of social exclusion. He advocated long term community level intervention
based on small steps, leading to growth in confidence, acquisition of skills
and increased human and social capital. He noted that an apparent lack of
change on the ground can disguise the fact that individuals have moved on.
Steve provided the evaluator’s view, discussing some of the tensions
involved in assessing complex community interventions within a flexible
programme of action research. Exploration of processes, identification of
underlying factors and assessment of the responsiveness of change, was as
valuable as data on inputs and outcomes. Specific settings and projects
meant findings were not necessarily generalisable, attribution of small-scale
change to the programme was problematic, and data was fragmentary, diverse
and unpredictable.
The final presentation, on ‘action research as an evaluative method’,
was given by Helen Bushell (Welsh Assembly Government) in relation to the
evaluation of the Communities First programme. An action research approach,
operating through close and frequent contact with all stakeholders, helped
to understand structures and relationships and to map changes in a constantly
evolving programme. Lessons were fed back during the lifetime of the programme,
and resulted in informed decisions being made about changes needed. Key
lessons from the evaluation were: the importance of trust in enabling stakeholders
to learn from mistakes, rather than hiding or defending these; and the need
for documentation and a narrative record of processes and changes that occur
on the ‘journey’.
back to top
index
