Government Social Research
This document presents a framework for appraising the quality of qualitative evaluations. Use the hyperlinks below to navigate between the sections.
This document presents a framework for appraising the quality of qualitative evaluations. It was developed with particular reference to evaluations concerned with the development and implementation of social policy, programmes and practice. The framework was devised as part of a programme of research conducted on behalf of the Cabinet Office.
The research on which the framework was built involved:
All these strands of activity have heavily influenced the content of the framework and the premises that surround its operation.
A separate down-loadable report (624kb) on the study is available. The sections below describe the scope, application and coverage of the framework. Sections II and III provide important background information about the premises underpinning the framework and its intended usage. It is recommended that these sections are read by anyone using the framework for the first time. Those familiar with this background can move directly to Section IV where the content of the framework is presented.
The framework has been built on certain premises surrounding the nature of qualitative enquiry, how it can be used for evaluative purposes and how its quality can be assessed. These premises derive from extensive debate in the literature on qualitative research and evaluation. A brief summary is given here but full documentation can be found in the referenced sections of this report.
For the purposes of this framework, the quality of the qualitative research that generates the evidence for an evaluation is seen as lying at the heart of any assessment. This is because of a primary interest in evaluations based on empirical enquiry, which form the majority of government-based evaluative investigations. Qualitative research and qualitative evaluation are therefore seen as broadly synonymous in terms of the principles surrounding quality assessment. Because of this, the framework can also be used for qualitative research more generally, irrespective of whether it has an evaluative purpose.
Qualitative research
Qualitative research aims to provide an in depth understanding of people's experiences, perspectives and histories in the context of their personal circumstances or settings. Among many distinctive features, it is characterised by a concern with exploring phenomena from the perspective of those being studied; with the use of unstructured methods which are sensitive to the social context of the study; the capture of data which are detailed, rich and complex; a mainly inductive rather than deductive analytic process; developing explanations at the level of meaning or micro-social processes rather than context-free laws; and answering 'what is', 'how' and 'why' questions. It employs a variety of methods, including: exploratory interviews; focus groups; observation; conversation, discourse and narrative analysis; and documentary and video analysis.
Qualitative research is used in evaluation for a range of purposes. For evaluations of programmes services or interventions, these include identifying the factors that contribute to successful or unsuccessful delivery; identifying outcomes (intended or unintended) and how they occur; examining the nature of requirements of different groups within the target population; exploring the contexts in which policies operate; and exploring organisational aspects of delivery. Qualitative research can also be used in advance of policy development or implementation, for example, to examine an issue or problem that is poorly understood or to inform the kind of intervention required. A further use is to help develop evaluative criteria where these are unclear or where alternative criteria are sought.
Quality Standards in qualitative research
Despite many different approaches and traditions within qualitative and evaluative research, there are widespread concerns about quality. There is also shared interest in issues such as 'rigour'; the need for principles of practice to be made manifest; the importance of sound or 'robust' qualitative research evidence; and in the relevance and utility of research.
The nature of quality assessment in qualitative research
There is debate in the literature about whether the concepts of quality used to assess qualitative research should be roughly the same as, parallel to, or quite different from those used to assess quantitative research. This framework is based on the view that the concerns which lie behind customary conceptions of quality have relevance for qualitative enquiry but need to be reformulated - and assessed quite differently - within the domain of qualitative research. In other words, qualitative research should be assessed on its 'own terms' within premises that are central to its purpose, nature and conduct.
The formalisation of quality standards
There is some debate about the extent to which quality assessment of qualitative inquiry can be formalised. Alongside this there have been increasing calls for guidance about quality assessment so that criteria appropriate to qualitative research (rather than those imported from other research traditions) are used. This, in turn, has led to the generation of a number of checklists, guidelines and lists of appraisal questions for assessing qualitative research.
This framework draws heavily on previously developed quality criteria, both from the general methodological literature and from pre-existing guidelines. However, it also takes heed of the persistent concern that formalised criteria should avoid being rigidly procedural or over-prescriptive. It has therefore been devised to aid informed judgement, not mechanistic rule-following.
Because there has been an opportunity to build on other frameworks, it is comparatively comprehensive in its coverage. It is also distinctive in making explicit the philosophical assumptions within which it operates; and in the level of specificity of the quality 'indicators' on which to base an assessment.
Philosophical assumptions
There are numerous approaches, paradigms, schools and movements encompassed within what is broadly termed 'qualitative research'. They vary in terms of the ontological, epistemological and methodological assumptions on which they are based. The same issues arise in relation to different types of evaluation. A proliferation of approaches to evaluation exist, which, again, differ in terms of their philosophical, ideological and methodological premises. In both cases, the various traditions are often categorised under labelled movements or schools although their identities are not always consistently described.
It was not felt possible to design a single set of quality markers that would be appropriate for all the different qualitative and evaluative traditions that exist. The assumptions on which they are based are too diverse - in some cases contradictory - and the conception of what matters in terms of 'quality' can vary with these assumptions.
Instead, an 'elemental' approach has been used in which a range of premises about the nature of social reality and how the social world can be investigated have been identified and those which underpin the framework specified. It is thought that these assumptions are broadly those within which most government-funded qualitative inquiry operates. Taking this approach means that the framework will be relevant for a range of types of qualitative evaluations including practice evaluation, policy development and appraisal as well as evaluations of particular interventions, schemes or programmes. It will be the assumptions within which an evaluation operates, rather than the model of qualitative research or evaluation used, that will determine whether the framework will be of value to assess its quality.
Choice of appraisal terms
There are numerous appraisal items that could have been included in the framework. Those selected were chosen on the basis that:
The framework has been designed primarily to assess the outputs of qualitative inquiry- that is reports, papers and journal articles. This has inevitably affected both the items covered and, most crucially, how they can be demonstrated for assessment. There is, for example, heavy emphasis on the quality of the evidence and the analysis that has informed it and rather less on aspects of conduct that cannot be well judged from written output.
Some translation of the criteria included can be made to assess proposals. Guidance notes on this are given in Section IV. It is also hoped that the framework will have a wider educational function in the preparation of research protocols, the conduct and management of research and evaluation and in the training of social researchers, both within and outside government.
Relevance for different qualitative methods
The framework has been designed with a particular focus on the methods used most extensively in government-based evaluations, namely, interviews, focus groups, observation and documentary analysis. Nevertheless, the principles that have guided the design of the framework, and many of the questions suggested for appraisal, will have application to a wider range of qualitative methods (e.g. linguistic analysis, historical and archival analysis, multimedia methods etc.). However, quality indicators that are particularly relevant to other methods will need to be added.
Use of multiple methods
It is common to find that evaluations use a mix of methods to collect the required information. Sometimes these are a combination of qualitative methods but in other cases a mix of qualitative and quantitative approaches. Although some of the appraisal questions in the framework are method generic (e.g. Qs 1-3, 5, 6 etc.) others are specific to qualitative methods (e.g. Qs 11-13). It is therefore recommended that to assess any quantitative components of an evaluation, quality criteria that are specific to the method concerned will need to be added.
Application to different policy domains
The framework has been designed to assess qualitative evaluation across the spectrum of substantive fields covered by government departments. However, it could well be that assessment within a particular substantive area might need to address some additional questions. If so, these can be added to the framework as necessary.
The framework is built on a set of principles around which to frame and structure appraisal questions that might be asked of a piece of work in order to critically assess its quality. In each case, a set of quality indicators is listed - features that will help to form a judgement about how well the appraisal question has been addressed. Further questions might also be added depending on the purpose of the research and the approach it uses. It is then for the assessor to judge overall merit, based on the questions and indicators that are most relevant to the evaluation concerned.
Guiding principles
There are four1 central principles that underpin the content of the framework. All of these are based on themes that are highly recurrent in the literature and in the interviews conducted for the study. They advise that research should be:
These principles are presented at a sufficiently high level of abstraction that they would apply to a diversity of qualitative approaches. Indeed, most of them are simply emblems of sound and logical enquiry, whatever its form or purpose.
Appraisal questions
The guiding principles have been used to identify 18 appraisal questions to aid an assessment. The questions are listed in column a) of the framework. Between them, they cover all of the key features and processes involved in qualitative enquiry. They begin with assessment of the findings, move through different stages of the research process (design, sampling, data collection, analysis and reporting) and end with some general features of research conduct (reflexivity and neutrality, ethics and auditability).
It is suggested that the findings of the enquiry are given attention first, even though this is not a logical procedural order. This is because the nature of the evidence presented will help in assessing features of the research process (for example, the quality of the data collected, the visibility and logic of the analytic process). However, if readers prefer to look at research design and conduct before considering the evidence, they will need to return to Qs 6-18 before completing the assessment
Quality indicators
Beside each question, there is a series of quality indictors (column b)) which will help in answering the appraisal question. These provide pointers to the kinds of information needed to judge whether or not the quality feature concerned has been secured. They are not intended to be comprehensive and other indicators might well be added for specific studies. Equally, they are not intended to suggest essential requirements - it is highly unlikely that all these indicators will be present, or even relevant, in any one study.
Many of the quality indicators relate only to the methods specified in Section II (i.e. interviews, focus groups, observation and documentary analysis). For most appraisal questions, however, quality indicators that are relevant to other methods could be added.
The need for professional judgement
The assessment of a qualitative inquiry, using this framework, will require careful judgements on the part of the assessor. These, in turn, will require some knowledge of qualitative research and some expertise in using qualitative methods. Judgement will also be needed in deciding the weight to attach to particular indicators in order to assess its 'fitness for purpose' - that is, how well it addresses the objectives for which it was undertaken. For example, in a study carried out to evaluate the implementation of a new scheme, it may well be more important to have a detailed account of how practice has affected outcomes, or an accessibly written report, than to have a thorough literature review.
Use for assessing proposals
As was noted in Section III, the framework has been designed to assess outputs from qualitative inquiry. However, selected questions and indicators from the framework could be used to assess proposals for designing and conducting a qualitative evaluation (see particularly Qs 2, 5, 6, 7, 9, 10 and 16-18). This, of course, will require changing from past to present the tense within which questions are phrased. It is important to stress, however, that other questions not included in this framework, will also be relevant to proposals (for example, feasibility, timescale, resources, experience of research team). This framework is therefore not intended as a comprehensive aid for proposal assessment.