Government Social Research
Before conclusions can be drawn from the studies that have been selected for inclusion in an REA, they need to be critically appraised to ensure that they are both relevant and that their findings are reliable.
The quality of the studies is likely to vary considerably and therefore the REA team must decide on clear explicit criteria for critically appraising the studies to separate those of higher quality from the weaker ones. There are three main dimensions considered in quality and relevance appraisal of studies (Gough 2007). These are:
Most impact questions will have incorporated these three dimensions as part of the inclusion criteria and so only studies of the specified quality and relevance will have been included.
Where this is the case a decision still needs to be made about whether to exclude or weight studies that fail the quality appraisal. If you decide to exclude studies then all those below the quality standard will simply be thrown out of the REA. The alternative is that studies are ‘weighted’ according to their quality and relevance but still included in the REA.
It is often dimension (A) on which studies are weighted according to their methodological quality (for example whether they used a control group that was randomly assigned). It is important to consider how successfully the method was applied and reported in assessing methodological quality. For example, studies that have reported using Randomised Controlled Trials may have implemented them poorly or provided too little information on key parts of the method (such as the randomisations process) to be able to appraise the quality as high.
Those studies that remain in the REA after the critical appraisal stage form a critically appraised map of evidence – a map of all the studies relevant to your question whose quality has been assessed.
Assessing quality and relevance of studies: non-impact questions