Government Social Research
Last Updated: 16/9/2008
There are two broad types of searching that can be done: comprehensive or purposive. The one that is most suitable for your policy questions will depend on the type of questions that you are asking and the conclusions that you want to be able to draw.
Comprehensive, or exhaustive searches aim to identify as much of the literature as possible that meets the inclusion criteria. The resulting set of studies should not be biased in any systematic way (any studies that were inadvertently missed should essentially be coming from the same pool of research as those that were found). Searches that are successful in identifying most of the literature are highly sensitive; those that identify mostly relevant studies and few not relevant ones are highly specific.
In order to identify as much relevant research as possible, systematic reviews emphasise sensitive searching. However, there is a trade-off between sensitivity and specificity with specificity decreasing as sensitivity increases. The more sensitive the search, the more time needs to be spent sifting out the irrelevant studies.
Given the time-sensitive nature of REAs, some sensitivity may have to be sacrificed in the knowledge that some potentially relevant research may be missed. However, the principle that the missing studies should not differ systematically from those included remains (i.e. it should remain representative of all possible research).
Comprehensive searches should be used for REAs that attempt to answer “What works?” questions. The model that underpins these REAs posits that it contains a random sample of all possible research in the area under investigation. Thus, while not all research may be identified, the sample is representative of the total population of potentially available research. The directions, and sizes, of observed effects are therefore taken to be a valid estimate of ‘real world’ phenomena.
In full systematic reviews a purposive search strategy aims to to collect examples of research across a variety of types - for example across several disciplines. It is thus different in its theoretical model to exhaustive searching. The aim is not, for example, to gain an overall picture of the balance of benefit and harm of a particular intervention, but rather to build up a ‘mosaic’ of different perspectives or points of view. No one perspective is taken to be more important than another, simply because it is reported in more than one study, and therefore ‘conceptual saturation’ is a more helpful principle to bear in mind, than simply aiming to retrieve every study – if additional studies do not add any more detail to the mosaic. This approach can also be used in REAs.
However, purposive REAs are also used as a pragmatic response to providing evidence to policy makers within a limited timeframe. Where an REA does not have time or resources to search exhaustively decisions may have to be taken about which sources to prioritise to identify as much of the available literature as possible. The risks of this approach are substantial because of the likely bias introduced and the potential for missing important studies. Purposive REAs require both good subject knowledge, so informed decisions can be made about which sources to priorities and experience of conducting REAs. All these decisions need to be recorded and stated explicitly in the final report; acknowledging any biases.
Search strategy - begin searching