Government Social Research
Before conclusions can be drawn from the studies that have been selected for inclusion in an REA, they need to be critically appraised to ensure that they are both relevant and that their findings are reliable.
The quality of the studies is likely to vary considerably and therefore the REA team must decide on clear explicit criteria for critically appraising the studies to separate those of higher quality from the weaker ones. There are three main dimensions considered in quality and relevance appraisal of studies (Gough 2007). These are:
For REA questions that are not looking for answers on “what works?” the first thing to consider is whether all three criteria have been considered.
Often methodological quality (A) is assessed but not whether the method is actually relevant to answering the REA question. It is important to remember that non-impact questions can be answered using quantitative and qualitative data (for example, implementation questions can use data from surveys)
The following are the key questions to ask under each dimension.
Those studies that remain in the REA after the critical appraisal stage form a critically appraised map of evidence – a map of all the studies relevant to your question whose quality has been assessed.