Government Social Research
Weight of evidence
The EPPI-Centre organises the three dimensions of quality and relevance into a framework called ‘Weight of Evidence’ (WoE). Each study is weighted according to dimensions A, B and C in conjunction with each other. These judgements are combined into dimension D which signifies the overall WoE judgement. Then either the findings of lower quality studies are excluded, or given less weight in the synthesis. The WoE framework is questions N13-N16 in the data extraction form on the left.
The Maryland Scale of Scientific Methods (MSSM) was developed by Sherman and colleagues (1997) at the University of Maryland for their review of crime prevention interventions. It is a five-point scale for classifying the strength of methodologies used in “what works?” studies.
Although developed for the criminology field it has wider application; the five levels of methodological quality are generic and so can be applied to other areas of social science. Sherman and colleagues (1997) argue that only studies with a robust comparison group design can provide evidence that a programme has caused the reported impact. This equates to level three and above in the Maryland Scale. To see the scale itself click on the symbol on the left. For more background information on the scale see Sherman et al, 1997 (PDF, NCJRS website).
Maryland scale
| Level 1 | Observed correlation between an intervention and outcomes at a single point in time. A study that only measured the impact of the service using a questionnaire at the end of the intervention would fall into this level. |
| Level 2 | Temporal sequence between the intervention and the outcome clearly observed; or the presence of a comparison group that cannot be demonstrated to be comparable. A study that measured the outcomes of people who used a service before it was set up and after it finished would fit into this level. |
| Level 3 | A comparison between two or more comparable units of analysis, one with and one without the intervention. A matched-area design using two locations in the UK would fit into this category if the individuals in the research and the areas themselves were comparable. |
| Level 4 | Comparison between multiple units with and without the intervention, controlling for other factors or using comparison units that evidence only minor differences. A method such as propensity score matching, that used statistical techniques to ensure that the programme and comparison groups were similar would fall into this category. |
| Level 5 | Random assignment and analysis of comparable units to intervention and control groups. A well conducted Randomised Controlled Trial fits into this category. |
The Agency for Healthcare Research and Quality is the health services research arm of the U.S. Department of Health and Human Services (HHS). They have nearly 40 scales (each with 3 to 34 items) for appraising Randomised Controlled Trials.
Although there has been considerable development of economic evaluations in the health field, there is a lack of integration of economic with other social science evaluations and a lack of research based ‘best-practices’ for appraising such studies . The Weight of Evidence questions in the template below should help with appraising economic studies and it can be incorporated with the appraisal forms for qualitative and quantitative studies.
Weight of evidence
The EPPI-Centre organises the three dimensions into a framework called ‘Weight of Evidence’ (WoE). Each study is weighted according to dimensions A, B and C in conjunction with each other. There are different ways of combining these judgements. Whichever is used, dimension D is used to signify the overall WoE judgement. Then either the findings of lower quality studies are excluded, or they are given less weight in the synthesis. The WoE framework for economic evaluation is questions H3-6 in the data extraction form.
Appraising the quality of qualitative research is a contested field, not least because there is little agreement as to the difference the use of different methods make and whether the nature of qualitative enquiry is itself amenable to rigid quality framework.
Weight of evidence
The EPPI-Centre organises the three dimensions of quality and relevance into a framework called ‘Weight of Evidence’ (WoE). Each study is weighted according to dimensions A, B and C in conjunction with each other. These judgements are combined into dimension D which signifies the overall WoE judgement. Then either the findings of lower quality studies are excluded, or given less weight in the synthesis. The WoE framework is questions N13-N16 in the data extraction form.