Policy on ‘Empty reviews’ and non-randomised trial evidence

Background

The main rationale for Cochrane Reviews is to provide a summary of high quality evidence on the effectiveness of therapeutic interventions or diagnostic technologies. It is therefore expected that the great majority of reviews will be based on randomised trial (RCT) evidence. Non-RCT and observational evidence, although often more available, is difficult to interpret because of the well-recognised sources of bias inherent in such studies. Cochrane is developing methodology for carrying out reviews of non-RCT evidence of interventions but this is not yet in widespread use and is likely to be reserved for topics in which RCT evidence is for technical or ethical reasons unlikely ever to be available. However we are also aware that in the era of ‘personalised medicine’ new kinds of clinical trials may become commonplace. If so, we will need to seek technical advice on how best to address this development.

If a review is carried out and the search finds no RCT evidence, then it is possible to publish a so-called ‘empty review’ which makes clear the lack of high quality evidence of effectiveness and  draws no conclusions other than to recommend research is carried out. Such ‘empty’ reviews have a role to play in informing researchers and funders of gaps in the evidence. However it is important that the library is not populated with ‘empty’ reviews on minor topics or on the management of rare conditions for which RCT evidence is unlikely ever to be available.

Producing an ‘empty’ review still involves quite a lot of work not only for the reviewers but also for the LCG editorial team who have to arrange for peer review of the protocol and provide technical support if required, as well as editing the final review

Policy

In order to ensure that as many of our reviews address important topics and to minimise the number of ‘empty’ reviews on minor topics we propose the following policy when accepting titles:

• The application should provide evidence that there is at least one fully published RCT or evidence that at least one RCT is underway and likely to publish within 18 months of accepting the title and include a clinicaltrials.gov or equivalent trial registry number.
• There may be rare instances when only one RCT is known to have been published and, because of its significance, is generally regarded as being of high quality and ‘pivotal’ and further RCTs are very unlikely to have been carried out or be in progress. The Editors may then decide that a review of a single RCT would be appropriate.
• If no RCTs are known to have been published or to be in progress the applicant will need to provide clear justification as to why the topic is important and, if agreed by the editors, the review can proceed, even though it may in the end be ‘empty’.
• In exceptional circumstances the LCG may allow the development of a review of non-RCT evidence provided the topic is considered important but it is unlikely that there will ever be RCT evidence, and the authors are prepared to follow Cochrane methods for assessing bias of non-RCT evidence. Any conclusions about effectiveness will need to be very cautious, making clear the limitations in the evidence. The final title of the review should make clear that this is a review of non-RCT, observational evidence.