Assessing Vaccine Efficacy and Effectiveness in ... - The INCLEN Trust

Assessing Vaccine Efficacy and
Effectiveness in Observational Studies
Steven Black, MD
Professor of Pediatrics
Center for Global Health
University of Cincinnati Children’s Hospital
Cincinnati, Ohio USA

Why do we care?
Why are observational assessments of effectiveness useful?
• Clinical trials may not give a “true” assessment
of the vaccine effect
– Population differences
– Clinical trial exclusion criteria
– Lack of measurement of some effects that might
not have been anticipated or that can not be
observed in a clinical trial
• Indirect effects

Efficacy Versus Effectiveness
• Efficacy: the specific reduction in attack rate
between the vaccinated and unvaccinated
group in a clinical trial:
ve= (1-Rv/Rp) x 100%
• Effectiveness: reduction in clinical events that
might be expected to be associated with the
disease but could also be caused by other
agents.

Observational Studies: The Good News
• Observational studies have the advantage of
being able to assess impact in the actual
population of interest.
• Results can be used to evaluate or confirm cost
effectiveness estimates.
• Changes in vaccine effect can be tracked over
time.
• Unanticipated outcomes can be assessed.
• Impact of partial or off schedule dosing can be
assessed.

Observational Studies: The Bad News
• Observational studies are observational !!
– There is no predefined control group.
– Non vaccinees tend to be different than vaccinees.
– Historical controls can introduce the risk of
confounding.
• Other non-vaccine secular changes can
dramatically impact assessment
– Cyclic changes in serotype distribution unrelated to
vaccine.
– Changes in clinical practice unrelated to vaccine.

QUESTION
Changes in disease incidence or serotype
distribution can be impacted by:
a. Vaccine
b. Antibiotic use
c. Secular trends
d. all of the above
e. Not a, b or c.

Observational Studies to Assess
Efficacy
Positive Examples

Cases per 100,000
Rates of invasive pneumococcal disease
among children

Cases/100,000 population
Rates of Penicillin-Nonsusceptible IPD
by Age Group
ABCs 1998/1999 and 2004
80
70
60
50
40
30
20
10
0
1998/9 2004
55% reduction

Rate per 100,000 population
Closing the Gap
Rates of IPD in Australian children

Cases/100,000 population
Example of an Indirect Population Effect
Invasive Pneumococcal Disease
40
35
30
25
20
15
10
5
0
1998
1999
Adults 65 years and older, 1998-2007
PCV-7
introduced
2000
2001
2002
2003
2004
2005
Nonvaccine
types
PCV-7
types
2006
2007
2007 vs.
baseline
+44%
-92%
baseline
Slide courtesy of Cynthia Whitney, CDC
Year
11

HPA Meningococcal C Surveillance

Resurgence of Hib Disease in the UK
BMJ VOLUME 329, 18 SEPTEMBER 2004

Why were these studies successful?
• Invasive disease assessment
– IPD is usually non-dsicretionary. That is, there is
relatively little variation over time in the tendency
to seek care.
• Surveillance network was in place and stable
for years prior to the assessment
– Laboratory and case ascertainment criteria did not
change over time

Observational Studies to Assess
Efficacy
Precautions

Red Flags for Observational
Effectiveness Studies
• Discretionary outcomes: Health care
utilization patterns may change the criteria for
seeking medical care
– New guidelines for otitis media treatment
– Stricter criteria for admission for pneumonia
• Lack of awareness or controlling for other
factors
– Antibiotic usage and serotype replacement.

Examples of Possibly Problematic
Observational Effectiveness Studies

Examples of Possibly Problematic
Observational Effectiveness Studies
Pneumococcal Serotype Replacement

Non-Vaccine Serotype Replacement

Secular trends in Spain
Spain: temporal trends of serotypes
that were (a) included and (b) the
most prevalent serotypes not included
in PCV7 among invasive isolates, 1979
to 2007
Some “replacement” strains were
increasing in prevalence years prior to
PCV introduction in 2001
Fenoll et al. J Clin Micro 2009; 47: 1012-1020

% of all IPD in children < 5 yrs
Single Clonal Expansion of S. pneumoniae Serotype 19A in Korean
Children Before PCV7 Introduction
25
20
23
• All 19A isolates were MDR
• All isolates from 2001 exhibited
Sequence Type (ST) 320
15
10
5
0
0
8
1991-1994 1995-1999 2000-2006
A dramatic increase in rates of IPD
caused by antibiotic-resistant
serotype 19A
can occur without vaccination
Year
Choi et al, 45 th IDSA abstract #202, 2007

Examples of Possibly Problematic
Observational Effectiveness Studies
Pneumonia

PCV7 Reduces Pneumonia Admissions
(2004)
65% 73%
39%
30%
17%
26%
41,000 pneumonia admissions
averted among < 2 yo ?????
Grijalva Lancet 2007;369:1179-86.

Hospital Utilization Management
Can Change Rates of Hospitalized Disease

Examples of Possibly Problematic
Observational Effectiveness Studies
Otitis Media

Impact of PCV7 on Otitis Media
Grijilva et al Pediatrics Sept 2006

Changes in Treatment Guidelines for Otitis Media:
Another Cause of Reduced Medical Visits

Observational Effectiveness
Studies
A MORE COMPLICATED EXAMPLE:
COMPARATIVE EFFECTIVENESS OF
TWO INFLUENZA VACCINES

Comparative Effectiveness of Adjuvanted and
Unadjuvanted Influenza Vaccine in the Elderly
• Elderly individuals are relatively unresponsive
to standard doses of influenza vaccine
• Elderly individuals are at higher risk of
complications of influenza
• Adjuvanted influenza vaccines are more
immunogenic in the elderly
• Study question: Are adjuvanted influenza
vaccines more effective in the prevention of
influenza?

Comparative Effectiveness of Adjuvanted and
Unadjuvanted Influenza Vaccine in the Elderly
• The LIVE study was conducted in Italy to compare
MF-59 adjuvanted influenza vaccine to nonadjuvanted
influenza vaccine
• Problem: As influenza vaccine was routinely
recommended AND adjuvanted vaccine was
preferentially recommended for higher risk
elderly, it was not considered ethical to conduct a
randomized study
• Solution: conduct an observational study and
adjust for confounding.

Comparative Effectiveness of Adjuvanted and
Unadjuvanted Influenza Vaccine in the Elderly

Observational Effectiveness Studies
• Work best for non-discretionary outcome such as
those requiring hospitalization (sepsis,
meningitis)
• Can assess population effects such as herd
immunity
• Are subject to confounding by secular trends
– Changing guidelines
– Other factors (antibiotics, cyclic patterns of disease)
• Overall are very useful, but require more
thoughtful interpretation than a RCT.

QUESTION
Which statement is NOT true about observational
studies
a. Provide a real life picture of a vaccine’s effect
b. Monitor changing effectiveness over time
c. Should only be done if clinical trials are not
possible
d. Are subject to confounding

Conclusions
• Observational studies offer the opportunity to assess
real life public health impact.
• Observations, both positive and negative, are subject
to confounding
– By changes in population composition
– By other changes in medical care practice
– By temporal trends.
• These studies are none the less indispensible
– Allow ability to assess unanticipated changes in vaccine
efficacy or epidemiology
– Can inform future vaccine development and prioritization