For each intervention listed on Reducing Reoffending, one of two methods has been used to systematically identify, evaluate and summarise the published evidence which describes the intervention’s effect on reoffending.
Where there is a relevant systematic review of an intervention, a summary of the existing systematic review has been produced. Relevant systematic reviews are those which include evidence from UK and European studies, have been undertaken recently (ideally within the last 5 years) and where the included studies are similar to those used in the UK criminal justice system.
For interventions for which no relevant systematic review has been identified, a Rapid Evidence Assessment (REA) has been undertaken. Rapid Evidence Assessments are a form of systematic review, and are undertaken over a shorter period (approximately 3 months, rather than 12 months) than a traditional systematic review. REAs and systematic reviews systematically search for, evaluate and synthesise evidence about a specific intervention, and they are used to help policy makers to understand the impact of an intervention. Where possible an REA or systematic review will include a statistical meta-analysis of individual studies, in order to provide a clear indication of the likely impact (effect size) of the intervention.
A guide to the rating scales
The rating scales used on Reducing Reoffending allow users to filter interventions by different criteria: the category (type of intervention); the estimated impact of the intervention; the certainty of the estimated impact; the cost and ease of implementation of the intervention. Some of the rating scales have been adapted from existing well-validated scales (i.e. rating scales which have been found to be accurate and effective measures).
This scale indicates the impact of the intervention on reducing reoffending (the effect size). More detailed information about the intervention’s effect size can be found in the corresponding intervention summary (in the section ‘Does it work?’).
|Positive effect||Intervention reduces reoffending|
|No effect||Intervention has no effect on reoffending|
|Negative effect||Intervention increase reoffending|
|Effectiveness not measured||The effect of an intervention was not measured|
Certainty of impact
This scale tells us how confident we can be that the average impact (effect) identified by the reviewed studies estimates the ‘true’ effect. For summaries of systematic reviews it is derived from the authors’ quality ratings. For REAs it is derived from the transparent and systematic framework provided by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach (Balshem et al. 2011). The true effect is the actual impact of the intervention on reducing reoffending. Research attempts to measure the true effect of the intervention, but practical considerations mean that may not be possible for researchers to use research designs, data collection and analysis methods which most accurately measure the true effect. For example, a Randomised Controlled Trial (RCT) is a research design which can clearly help researchers to understand what is causing reductions (or increases) in reoffending. However it is often difficult (and ethically questionable) to randomly allocate individuals to receive an intervention (or not to). This means what researchers may need to use a non-randomised research design instead.
If the summarised evidence contains studies which are rated below ‘High’, we cannot be confident that the estimate of the effect (the average impact) is a good estimate of the true effect. The ratings for the summarised evidence are:
|High||We are very confident that the true effect lies close to that of the estimate of the effect|
|Moderate||We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different|
|Low||Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect|
|Very low||We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect|
This scale is intended to provide an indication of the financial cost associated with an intervention. It is anticipated that a number of studies will be based in the United States and/or will have been implemented several years ago. For this reason the costs presented here are indicative only.
|Not evaluated||The evidence reviewed did not include an economic evaluation|
This scale is an indicative estimated measure of the ease of implementation for particular interventions, and is loosely based on Dartington Social Research Unit’s own measure. The scale indicates whether the intervention is accompanied by the necessary support and information to enable its implementation in service systems. For example, interventions with a high degree of system readiness meet criteria such as the following (Dartington Social Research Unit, 2011):
- The intervention has been tested in ‘real world’ conditions.
- Technical support is available to help implement the intervention in new settings.
- There is a fidelity protocol or assessment checklist to accompany the intervention.
- There are explicit processes for ensuring that the intervention gets to the right people.
- There are training materials and implementation procedures.
- There is reported information on the financial and human resources required to deliver the intervention.
|5||High system readiness|
|4||High-moderate system readiness|
|3||Moderate system readiness|
|2||Low-moderate system readiness|
|1||Low system readiness|