Government Social Research
Last Updated: 16/9/2008
The time pressures of REAs mean that the searching process needs to be carried out quickly. You should consider the factors below to get the most out of the time available and to recognise the limitations caused by searching rapidly.
Searching is an activity that can go on indefinitely. One can never be 100% certain that every available reference has been retrieved. And, for currency, it is always necessary to keep searching for new studies. However, in an REA, it is especially important to be aware of time limitations. Searches should be designed so that they are achievable within the time available. They should only be allowed to overrun in the event of it becoming clear that the validity of the REA would be threatened if they were to be curtailed. Any weaknesses in the searching should be made explicit, acknowledging that the timescales have made the searching less than comprehensive (and hence possibly biased).
The skills of the REA team are a potential limitation – both in terms of being able to search quickly and in terms of their capacity to operationalise complex questions. If you are conducting the REA in-house it is worth using the skills of specialists in information management, the subject area and REA/systematic reviews at the searching stage. Searching is a skilled and resource intensive process and drawing in outside help will assist the quality and speed with which you can retrieve relevant studies. The type of advice that you can get from, for example librarians, is:
There are a number of ways in which the search can be limited (as stated in the inclusion criteria), shortening the time required for searching and for screening studies that were found.
Limiting the search may introduce bias into the REA. Certain studies are harder to find and will therefore be less likely to be included in the REA if the searching is limited. ‘Positive’ results (i.e. findings that report statistically significant impacts of an intervention) are more likely to be:
Essentially the less comprehensive the search the more risk there is of bias. REAs sacrifice the comprehensive nature of searching in return for more rapid findings. The likelihood of an REA being biased and the causes of this should be acknowledged and discussed alongside the findings.