Systematic Reviews and Meta-Analysis (PDF 646 kB)

SYSTEMATIC REVIEWS AND
META-ANALYSIS

Objectives
• Define systematic review and meta-analysis
• Know how to
– access
– appraise
– interpret
the results of a systematic review

Reasons to use reviews
• Sheer volume of literature*
• Save time doing exhaustive literature
researches
*Wyatt: about 25,000 biomedical journals in print worldwide.
This number is rising by 4% pa - doubling time of 19 yrs

Review
• Any attempt to synthesise results and
conclusions of 2 or more publications on a
given topic
– e.g. editorials, working papers
• Problems
– Retrieval bias and publication bias
• Were all studies identified?
• Do not see how conclusions arrived at
• Reviewers may be biased / conflict of interest
• Languages excluded?

Were all studies that met the inclusion
criteria for the review identified successfully?
If not, were the results of the sample of studies
included in the review representative of the
results of all eligible studies?
Of controlled trials identified in five separate subject
areas within perinatal medicine, between 20 and 50
percent were identified using MEDLINE, compared
with between 85 and 100 percent using the Oxford
Database of Perinatal Trials.
Chalmers et al. 1989

Retrieval bias
What can be done about it?
• Systematic and comprehensive search for eligible
reports
How can it be prevented?
• Use of structured abstracts by investigators
• Improvement, extension and further development of
– Bibliographic retrieval systems
• Explicit search criteria

Publication bias
• A tendency among investigators, peer
reviewers and journal editors to allow
the direction and statistical
significance of research findings to
influence decisions regarding
submission and acceptance for
publication

Publication bias
Examples
Analysis of relative risks derived from studies examining the
relationship between passive smoking and lung cancer suggests that
studies in which no relationship has been detected may remain
unpublished. Vandenbrouke, Br Med J 1988;296:391-392
However, estimates of the effect of these suggests that
they would have little impact on the weight of evidence
against passive smoking, 2004
A survey of authors or published reports of randomized trials
revealed that between a quarter and a fifth of the trials that they
have ever conducted had never been published, and that trials in
which a new treatment had been found to be a superior to a
standard treatment were more likely than others to have been
published.
Dickersin et al.Contr Clin Trials1987;8:343-353

Pharmaceutical industry sponsorship and research
outcome and quality: systematic review.
BMJ 2003;326:1167-1170 (31 May),
Joel Lexchin, Lisa A Bero, Benjamin Djulbegovic, Otavio Clark
• Results
30 studies were included (1966 – 2002). Research funded
by drug companies was less likely to be published than
research funded by other sources. Studies sponsored by
pharmaceutical companies were more likely to have
outcomes favouring the sponsor than were studies with
other sponsors (odds ratio 4.05; 95% confidence
interval 2.98 to 5.51; 18 comparisons). None of the 13
studies that analysed methods reported that studies
funded by industry was of poorer quality.

Publication bias
• What can be done about it?
– Grey literature
– Contact authors

Pitfalls of reviews
• “Current medical reviews do not routinely
use scientific methods to identify, assess
and synthesise information.”
Mulrow, 1987
• 50 reviews in 4 major journals, 1985-86
– no statement of methods 49
– summary inappropriate 47

How can the situation
improve?
• Better reviews (high quality, more relevant)
– Cochrane collaboration
– Systematic reviews
• Improved access to reviews
– CCPC, Databases, Effectiveness Bulletins
• Readers more skilled in making sense of reviews
– CASP

What is a systematic review?
A review in which evidence on a topic has been
systematically identified and summarised according
to predetermined criteria.
• Specific clinical questions
• Predefined explicit methodology
• Reproducible
• Usually review of RCTs

Meta-analysis - statistical principles
The use of statistical methods to summarise the results of
independent studies into a single estimate giving more weight
to results from larger studies.
• No direct comparison of patients
• Summary statistics are calculated for each trial
• Individual estimates are pooled - overall pooled estimate (if
appropriate)
– Gives more precise estimate of effect size
• Interpretation / odds ratios

Types of review
Reviews
Systematic reviews
Meta-analysis

Why use systematic reviews?
• Volume of literature
• Provide a basis for rational decision making
• Are health care effects consistent?
• Limit bias and reduce random error
• Provides more reliable results
• Required for ethics committees, funding agencies

Aim: To help people make well-informed decisions about health
care by preparing, maintaining and promoting accessibility of
systematic reviews of the effects of health care interventions.
www.cochrane.org

6 principles of Cochrane
• Collaboration
– 5000 people in 50 countries
• Building on peoples’ existing enthusiasm and interests
– 50 Cochrane Review Groups - focus on particular areas of
health
• Minimising duplication effort
• Avoidance of bias
• Keeping up to date
• Ensuring access

Format for a (Cochrane) review
• Contact Cochrane
• Develop a protocol
• Formulate the problem
• Locate and select studies
• Critical appraisal of studies
– Score for selection, attrition, performance, detection, blinding
biases
• Collecting data
• Analysing and presenting results
• Interpreting results
• Improving and updating reviews

What is a Cochrane Review?
http://www.cochrane.org/reviews/revstruc.htm
• Cochrane Reviews investigate the effects of interventions
for prevention, treatment and rehabilitation in a
healthcare setting. They are designed to facilitate the
choices that doctors, patients, policy makers and others
face in health care. Most Cochrane Reviews are based on
randomized controlled trials, but other types of evidence
may also be taken into account, if appropriate. Cochrane
reviews have the following general features:

General features of a Cochrane review
• Structured format
• Detailed methods section
• The quality of clinical studies to be incorporated into a review is carefully
considered, using predefined criteria.
• A thorough and systematic search strategy, which includes searches for
unpublished and non-English records
• If the data collected in a review are of sufficient quality and similar
enough, they are summarised statistically in a meta-analysis,
– generally provides a better overall estimate of a clinical effect than the results
from individual studies.
• Reviews aim to be relatively easy to understand for non-experts
• Multinational editorial teams try to ensure that a review is applicable in
different parts of the world
• Reviews are updatable.
– Results from newly completed or identified clinical trials can be incorporated
into the review after publication. Additionally, readers can send in comments
and criticisms to any review, and reviews may be changed accordingly to
improve their quality.

Systematic review protocol
• State objectives and eligibility criteria
• Identify potentially eligible studies
• Apply eligibility criteria
• Refine protocol
• Publish protocol

Elements of a systematic review
1. Define the clinical question
2. Identify all completed studies, un/published
3. Select the studies that meet scientific validity criteria
4. Look for evidence of bias
5. Describe the scientific quality of the studies
6. Assess if quality systematically related to results
7. Describe studies with a forest plot
8. Assess if similar enough to justify combining results
– Calculate summary measure of effect & CI
Fletcher & Fletcher T12.1

How can we appraise a
review?
• How can we systematically appraise a
review?
• What questions should we ask?

10 questions to appraise a review
Critical Appraisal Skills Programme (CASP)
http://www.sph.nhs.uk/what-we-do/public-healthworkforce/resources/critical-appraisals-skills-programme
• Three basic types of question...
• Is it trustworthy?
– Screening questions
– Validity
– Detailed questions on methodology
• What does it say?
• Will it help?
– Results
– Relevance
Footnotes:
http://www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme
Last accessed April 2011

Is it trustworthy? – Validity
• Screening questions
– Did review address a clearly focused issue?
– Did authors review right type of study?
• Detailed questions on methodology
– Were all important relevant studies included?
– Did reviewers do enough to assess the quality
of the included studies?
• Randomisation system, scoring system, >1 assessor
– If meta-analysis performed, was it reasonable
to combine the results?

What does it say - Results
• Rates
• Relative and absolute risk, numbers needed to
treat + confidence intervals
• Clinical significance
• Statistical significance

Will it help? – Relevance
(What service providers should ask)
– Is my patient...
– Are the interventions available to my patient...
– Are the outcomes relevant to my patient...
.....sufficiently different from people in the review
to allow me to consider the findings inapplicable?

Design literature search for:
Quantitative systematic review of randomised
controlled trials comparing antibiotic with
placebo for acute cough in adults.
(BMJ 1998;316:906-10) Fahey T. et al.
• Aim:
– To establish whether antibiotics are effective in
the treatment of acute cough in the community

Literature search
• Medical subject headings (MeSH)
• Databases
• Language?
• Study type
• Patients

Literature search
• Medical subject headings (MeSH)
– Cough, bronchitis, sputum, respiratory tract
infections, chest infection
• Databases
– Medline, EMBASE, Science Citation Index,
Cochrane Controlled Trials Register
– Contacted authors - ?know of unpublished trials
– UK drug companies ?unpublished trials
• Language - not just English

• Study type
Literature search
Inclusion/exclusion criteria
– Prospective trials, formal or quasi-randomisation
– Placebo controlled
• Patients
– >12 years
– Family practice clinic, community based o/px
dept., hospital o/px dept.
– Acute cough, with/out sputum, no antibiotic in
preceding week
– COAD excluded

Footnote: Get class to interpret this table

Conclusions?
• Treatment with antibiotic
– Does not affect resolution of cough or alter course
of illness
• Benefits
– Marginal for most patients with acute cough
– May be outweighed by side effects (RR=1.51, CI
0.86 to 2.64)