Executive Summary
| This is an
executive summary of the report and framework for appraising
the quality of qualitative evaluations. The report and framework
was produced by researchers at the National Centre for Social
Research on behalf of the Cabinet Office Strategy Unit. |
Background (Chapter 1)
This report presents the findings of a study carried out by a team
of researchers based at the National Centre for Social Research,
on behalf of the Strategy Unit in the Cabinet Office. The objective
of the study was to develop a framework which would guide assessments
of the quality of qualitative research evaluations. The study grew
out the Cabinet Office's responsibility for ensuring excellence
in government research and evaluation, and was a response to the
fact that, despite their growing use, there are no explicitly agreed
standards regarding what constitutes quality in qualitative research
evaluations.
The study involved a number of elements:
- a comprehensive review of the literature on qualitative
research methods relating to standards in qualitative research
- a review of qualitative research methods used in government
funded evaluation studies
- a review of existing frameworks for assessing quality
in qualitative research
- exploratory interviews with a range of people who have
an interest in quality assessment of qualitative research
and/or policy-related evaluations. These included academics
who have written about qualitative research from either
a theoretical or empirical perspective; authors of existing
frameworks; research practitioners; commissioners and funders;
and policy-makers who have used qualitative research evidence
in the development and evaluation of policies
- a workshop, involving the above groups, to refine the
framework initially developed
- a trial application of the framework to a small number
of studies
|
The framework (Chapters 2 and 7)
The framework is based around:-
Four guiding principles - that research should be
- contributory in advancing wider
knowledge or understanding
- defensible in design by providing
a research strategy which can address the evaluation questions
posed
- rigorous in conduct through
the systematic and transparent collection, analysis and
interpretation of qualitative data
- credible in claim through offering
well-founded and plausible arguments about the significance
of the data generated
eighteen appraisal questions
- How credible are the findings?
- How has knowledge or understanding been extended by the
research?
- How well does the evaluation address its original aims
and purpose?
- How well is the scope for drawing wider inference explained?
- How clear is the basis of evaluative appraisal?
- How defensible is the research design?
- How well defended are the sample design/target selection
of cases/documents?
- How well is the eventual sample composition and coverage
described?
- How well was the data collection carried out?
- How well has the approach to and formulation of analysis
been conveyed?
- How well are the contexts of data sources retained and
portrayed?
- How well has diversity of perspective and content been
explored?
- How well has detail, depth and complexity (i.e. richness)
of the data been conveyed?
- How clear are the links between data, interpretation
and conclusions - i.e how well can the route to any conclusions
be seen?
- How clear and coherent is the reporting?
- How clear are the assumptions/theoretical perspectives/values
that have shaped the form and output of the evaluation?
- What evidence is there of attention to ethical issues?
- How adequately has the research process been documented?
quality indicators
- for each appraisal question, a series of possible features
for consideration in the assessment of quality are proposed.
|
Scope and applicability of the framework
(Chapter 4)
Key aspects of the scope and applicability of the framework:
- The framework has been designed to be applied to appraisals
of the outputs of qualitative evaluations (reports, papers
and journal articles), although it would also have some
relevance to assessments of proposals or the conduct of
a study.
- Although the four guiding principles would be relevant
to any qualitative research method (and in many cases to
quantitative research too), the appraisal questions and
quality indicators are designed to focus on four methods:
in-depth interviews, focus groups, observation and documentary
analysis.
- The framework is designed to aid the informed judgement
of quality, but not to be prescriptive or to encourage the
mechanistic following of rules. The questions are phrased
as open-ended questions to reflect the fact that appraisals
of quality must allow judgement, and that standards are
inevitably shaped by the context and purpose of assessment.
- The framework is not intended to apply to the full range
of traditions or paradigms in qualitative research, but
equally it is not intended to be aligned with any individual
specific schools or approaches. Instead, some key philosophical
assumptions that are within or outside the scope of the
framework are described.
- The framework is designed to be applied to a wide range
of types of qualitative evaluation (including contextual
research, which examines the world in which a policy or
service operates; policy review and development; practice
evaluation and appraisals of specific schemes or interventions).
But the quality of the qualitative research on which an
evaluation is based is seen as lying at the heart of assessments
of quality.
- Most of the items included in the framework are heavily
recurrent in the wider literature, in existing frameworks
and among our interviews with research commissioners and
managers, policy-makers, funders, academics and practitioners.
They have also been selected to offer a series of readings
on the guiding principles and to cover all the stages and
processes involved in qualitative research.
|
Foundations of the framework (Chapters
3, 5 and 6)
The framework draws on and reflects a number of important and extensive
debates relating to quality in qualitative research.
- There are different views about whether evaluation is
an activity that is distinct from research and a number
of differentiating features are proposed. There are also
different definitions of qualitative research and different
views about how it contributes to policy evaluation.
- There is debate about whether it is feasible or desirable
to establish quality criteria for qualitative research and
how far these are different to criteria for quantitative
research. Positions range from a rejection of the notion
of criteria altogether, to the identification of criteria
or principles developed specifically to qualitative research,
to the retention of concepts drawn from quantitative research.
- There are debates about what is meant by criteria, and
about whether the particular philosophical assumptions and
methods of qualitative research make criteria more problematic.
- Conceptions of quality are influenced by the various
philosophical assumptions which underpin different approaches
to qualitative research. These epistemological and ontological
positions are diverse and span issues such as the nature
of reality, the relationship between the researcher and
the researched, the relationship between facts and values,
the nature of knowledge, and appropriate methods of research.
- Within varying conceptions of quality, the notions of
validity, reliability, objectivity and generalisation are
often given key importance. The meanings attributed to these
concepts, and how people view their applicability or otherwise
to qualitative research, varies extensively.
- The research involved a review of 29 existing frameworks
for assessing the quality of qualitative research. Many
were developed in the fields of medical or health services
research. They were developed for different purposes, including
the assessment of written outputs, reviewing proposals or
framing the conduct of research. Authors generally stress
the development and permissive nature of their frameworks.
There is much variety between the frameworks as to
- how far their philosophical orientation is specified
- how far their applicability to different qualitative
methods is specified
- the level of specification of criteria
- the length, format and coverage of the frameworks
- There are several common features, particularly around
the need for clarity in aims and objectives; appropriate
use of qualitative methods; appropriate sample design; clarity
about the analytical process and clarity about how the evidence
and conclusions are derived.
|
Concluding comments
about the framework (Chapter 8)
The framework draws heavily on existing frameworks, on the wider
literature and on the contribution of those who participated in
the study. It is, however, developmental, and it is recognised that
there will be alternative views about its structure, focus and content.
What is critical is that the framework is applied flexibly, and
not rigidly or prescriptively. Judgement and discretion lie at the
heart of quality appraisal, and assessments of quality will vary
between different contexts and purposes.
The study highlights two further points.
First, assessing quality requires a degree of expertise in the conduct
and use of research, and there is interest in more support, education
and guidance for non-research experts in their use of qualitative
research. Second, there is a need to consider a creative extension
of the range of qualitative research methods used in government
evaluations.
