presentation systematic review - EMGO

OutlineResearch questions1. Search strategy2. Methods: article selection and data extraction3. Resultsa. Standardsb. Criteria

Research questions 11. Adequate search strategy?a. search strategy described?b. in which databases?

Research questions 22. Selection of articles and data extraction?a. article selection by at least twoindependent reviewers?b. data extraction by at least twoindependent reviewers?c. in- and exclusion criteria described?

Standards refer to: adequate design,statistical method, sample size, etc.• ‘Reliability should be calculated with an ICC’Criteria refer to: cut-off values for goodmeasurement properties• ‘ICC > 0.70’• ‘Cronbach’s alpha between 0.70 – 0.95’

Research questions 33. Were standards applied?…to determine if the measurement propertieswere evaluated properly?

Research questions3. Were standards applied?…to determine if the measurement propertieswere evaluated properly?4. Were criteria applied?…to determine which instruments have goodmeasurement properties?

Measurement properties:• Reliability: internal consistency, testretest,intra-rater, inter-rater reliability,measurement error• Validity: content, face, construct, criterionvalidity• Responsiveness• Interpretability

Methods• MEDLINE, EMBASE, and Psycinfo• Inclusion criteria:– ‘systematic review’– aim is to find all available health statusmeasurement instruments– evaluative application– measurement properties reported or evaluated• Article selection and data extraction by twoindependent reviewer

Results search and study selection• Search strategy described 84%• 1 database used 23%• 2 or 3 databases used 37%• 4 or more databases used 37%• In- and exclusion criteria described 65%• Article selection adequately performed 18%• Data extraction adequately performed 23%

Which properties were evaluated?6050403020reportedstandards appliedcriteria appliednumber of reviews100reliability (general)internal consistencyreproducibilitymeasurement errorvalidity (general)content validityconstruct validitycriterion validityresponsivenessinterpretability

Overview of the reviews100 reviewsResults describedN = 52Standards and/orcriteria appliedN = 48

Overview of the reviews100 reviewsResults describedN = 52Standards and/orcriteria appliedN = 48Standards appliedN = 23Criteria appliedN = 47

Overview of the reviews100 reviewsResults describedN = 52Standards and/orcriteria appliedN = 48Standards appliedN = 23Criteria appliedN = 47Ad hocappliedN = 13Detailedstandards appliedN = 10

Ad hoc application of standardsExamplesStandards not applied to all measurementproperties; e.g. only validity, not forreliabilityMethodological quality of validation studiesnot assessed for each instrumentGrading system not fully described

Detailed application of standards

Detailed application of standardsInternal consistency, Reliability, Measurement error, Constructvalidity, Responsiveness, InterpretabilityInternal consistency- Internal consistency of the questionnaire notcorrectly assessed+/- Cronbach’s alpha calculated for the whole scale or forpre-defined subscales+ Cronbach’s alpha calculated for either the whole scaleor for subscales depending on the outcome of the factoranalysis

Overview of the reviews100 reviewsResults describedN = 52Standards and/orcriteria appliedN = 48Standards appliedN = 23Criteria appliedN = 47Ad hocappliedN = 13Detailedstandards appliedN = 10Ad hocappliedN = 15Applied togeneralmeasurementpropertiesDetailed criteriaappliedN = 16N = 16

Ad hoc application of criteriaVague statement• “tools demonstrated acceptable reliability”Only applied to one property• “evidence of sufficient reliability(Cronbach’s alpha> 0.70)”

Detailed application of criteria• Cronbach’s alpha: poor if < 0.70, moderate to good if 0.71 -0.90, redundant > 0.90• ICC and kappa: poor if < 0.40, moderate if 0.41 - 0.60,substantial if 0.61 - 0.80, and good to almost perfect if > 0.81• Correlation coefficient: poor if < 0.7, moderate to good if >0.70

Combination of standard and criteriaInternal consistency+ Cronbach’s alpha calculated for each subscale, based on theoutcome of factor analysis; Cronbach’s alpha between 0.70 and0.90;+/- Doubtful methods; no factor analysis performed;- Cronbach’s alpha calculated for each subscale, based on theoutcome of factor analysis; Cronbach’s alpha below 0.70;? No information available

ConclusionMethodological quality of clinimetricreviews is low• Accepted method guidelines for systematic reviews often notapplied• More than half of the reviews only reported results withoutany evaluation of these results or the methods used• The reviews that did evaluate the results or the methodsused different standards and criteria• Quality assessment strongly hampered by inadequatereporting

COSMIN steering committeeLB Mokkink, CB Terwee, LM Bouter, HCW de VetEMGO Institute, VU University Medical Center (VUmc), Amsterdam, TheNetherlandsDL KnolEMGO Institute, and Department of Clinical Epidemiology and Biostatistics, VUmc,Amsterdam, The NetherlandsPW StratfordSchool of Rehabilitation Science and Department of Clinical Epidemiology andBiostatistics, McMaster University, Hamilton, CanadaJ AlonsoHealth Services Research Unit, Institute Municipal d'Investigacio Medica (IMIM-IMAS), Barcelona, SpainDL PatrickDepartment of Health Services, University of Washington, Seattle, USA