Critical Appraisal of Scientific Articles

MEDICINEREVIEW ARTICLECritical Appraisal of Scientific ArticlesPart 1 of a Series on Evaluation of Scientific PublicationsJean-Baptist du Prel, Bernd Röhrig, Maria BlettnerSUMMARYIntroduction: In the era of evidence-based medicine, one ofthe most important skills a physician needs is the ability toanalyze scientific literature critically. This is necessary tokeep medical knowledge up to date and to ensure optimalpatient care. The aim of this paper is to present anaccessible introduction into critical appraisal of scientificarticles.Methods: Using a selection of international literature, thereader is introduced to the principles of critical reading ofscientific articles in medicine. For the sake of conciseness,detailed description of statistical methods is omitted.Results: Widely accepted principles for critically appraisingscientific articles are outlined. Basic knowledge of studydesign, structuring of an article, the role of differentsections, of statistical presentations as well as sources oferror and limitation are presented. The reader does notrequire extensive methodological knowledge. As far asnecessary for critical appraisal of scientific articles,differences in research areas like epidemiology, clinical,and basic research are outlined. Further useful referencesare presented.Conclusion: Basic methodological knowledge is required toselect and interpret scientific articles correctly.Dtsch Arztebl Int 2009; 106(7): 100–5DOI: 10.3238/arztebl.2009.0100Key words: publication, critical appraisal, decision making,quality assurance, studyInstitut für Medizinische Biometrie, Epidemiologie und Informatik (IMBEI),Johannes Gutenberg-Universität, Mainz: Dr. med. du Prel, MPH, Dr. rer. nat.Röhrig, Prof. Dr. rer. nat. BlettnerDespite the increasing number of scientific publications,many physicians find themselves withless and less time to read what others have written. Selection,reading, and critical appraisal of publications is,however, necessary to stay up to date in one's field. Thisis also demanded by the precepts of evidence-basedmedicine (1, 2).Besides the medical content of a publication, its interpretationand evaluation also require understanding of thestatistical methodology. Sadly, not even in science are allterms always used correctly. The word "significance," forexample, has been overused because significant (or positive)results are easier to get published (3, 4).The aim of this article is to present the essential principlesof the evaluation of scientific publications. Withthe exception of a few specific features, these principlesapply equally to experimental, clinical, and epidemiologicalstudies. References to more detailed literatureare provided.Decision makingBefore starting a scientific article, the reader must beclear as to his/her intentions. For quick information on agiven subject, he/she is advised to read a recent reviewof some sort, whether a (simple) review article, a systematicreview, or a meta-analysis.The references in review articles point the readertowards more detailed information on the topic concerned.In the absence of any recent reviews on the desiredtheme, databases such as PubMed have to be consulted.Regular perusal of specialist journals is an obviousway of keeping up to date. The article title and abstracthelp the reader to decide whether the article merits closerattention. The title gives the potential reader a concise,accurate first impression of the article's content. Theabstract has the same basic structure as the article andrenders the essential points of the publication in greatlyshortened form. Reading the abstract is no substitute forcritically reading the whole article, but shows whetherthe authors have succeeded in summarizing aims,methods, results, and conclusions.The structure of scientific publicationsThe structure of scientific articles is essentially alwaysthe same. The title, summary and key words are followedby the main text. This is divided into Introduction,Methods, Results and Discussion (IMRAD), endingwhen appropriate with Conclusions and References.100 Deutsches Ärzteblatt International⏐Dtsch Arztebl Int 2009; 106(7): 100–5

MEDICINEThe content and purpose of the individual sections aredescribed in detail below.IntroductionThe Introduction sets out to familiarize the reader with thesubject matter of the investigation. The current state ofknowledge should be presented with reference to therecent literature and the necessity of the study should beclearly laid out. The findings of the studies cited should begiven in detail, quoting numerical results. Inexact phrasessuch as "inconsistent findings," "somewhat better" and soon are to be avoided. Overall, the text should give theimpression that the author has read the articles cited. Incase of doubt the reader is recommended to consult thesepublications him-/herself. A good publication backs up itscentral statements with references to the literature.Ideally, this section should progress from the generalto the specific. The introduction explains clearly whatquestion the study is intended to answer and why thechosen design is appropriate for this end.MethodsThis important section bears a certain resemblance to acookbook. The description of the procedures shouldgive the reader "recipes" that can be followed to repeatthe study. Here are found the essential data that permitappraisal of the study's validity (6). The methods sectioncan be divided into subsections with their own headings;for example, laboratory techniques can be describedseparately from statistical methods.The methods section should describe all stages ofplanning, the composition of the study sample (e.g.,patients, animals, cell lines), the execution of the study,and the statistical methods: Was a study protocol writtenbefore the study commenced? Was the investigationpreceded by a pilot study? Are location and study periodspecified? It should be stated in this section that the studywas carried out with the approval of the appropriateethics committee. The most important element of ascientific investigation is the study design. If for somereason the design is unacceptable, then so is the article,regardless of how the data were analyzed (7).The choice of study design should be explained anddepicted in clear terms. If important aspects of themethodology are left undescribed, the reader is advisedto be wary. If, for example, the method of randomizationis not specified, as is often the case (8), one ought not toassume that randomization took place at all (7). The statisticalmethods should be lucidly portrayed and complexstatistical parameters and procedures describedclearly, with references to the specialist literature. Box 1contains further questions that may be helpful in evaluationof the Methods section.Study design and implementation are described byAltman (7), Trampisch and Windeler (9), and Klug et al.(10). In experimental studies, precise depiction of thedesign and execution is vital. The accuracy of a method,i.e. its reliability (precision) and validity (correctness),must be stated. The explanatory power of the results of aclinical study is improved by the inclusion of a controlBOX 1Questions on methodology❃ Is the study design suited to fulfill the aims of the study?❃ Is it stated whether the study is confirmatory, exploratory or descriptive in nature?❃ What type of study was chosen, and does it permit the aims of the study to beaddressed?❃ Is the study's endpoint precisely defined?❃ What statistical measure is employed to characterize the endpoint?Do epidemiological studies, for instance, give the incidence (rate of newcases), prevalence (current number of cases), mortality (proportion of thepopulation that dies of the disease concerned), lethality (proportion of thosewith the disease who die of it) or the hospital admission rate (proportion ofthe population admitted to hospital because of the disease)?❃ Are the geographical area, the population, the study period (including durationof follow-up), and the intervals between investigations described in detail?group (active, historical, or placebo controls) and by therandomized assignment of patients to the different armsof the study. The quality can also be raised by blindingof the investigators, which guarantees identical treatmentand observation of all study participants. A clinicalstudy should as a rule include an estimation of the requirednumber of patients (case number planning) before thebeginning of the study. More detail on clinical studiescan be found, for instance, in the book by Schumacherand Schulgen (11). International recommendations speciallyformulated for the reporting of randomized, controlledclinical trials are presented in the most recentversion of the CONSORT Statement (ConsolidatedStandards of Reporting Trials) (12).Epidemiological investigations can be divided intointervention studies, cohort studies, case-control studies,cross-sectional studies, and ecological studies. Table 1outlines what type of study is best suited to what situation(13). One characteristic of a good publication is a preciseaccount of inclusion and exclusion criteria. How highwas the response rate (✞80% is good, ✜30% means noor only slight power), and how high was the rate of lossto follow-up, e.g. when participants move away or withdrawtheir cooperation? To determine whether participantsdiffer from nonparticipants, data on the lattershould be included. The selection criteria and the ratesof loss to follow-up permit conclusions as to whether thestudy sample is representative of the target population.A good study description includes information on missingvalues. Particularly in case-control studies, but also innonrandomized clinical studies and cohort studies, thechoice of the controls must be described precisely. Onlythen can one be sure that the control group is comparablewith the study group and shows no systematic discrepanciesthat can lead to misinterpretation (confounding)or other problems (13).Is it explained how measurements were conducted?Are the instruments and techniques, e.g. measuringdevices, scale of measured values, laboratory data, andDeutsches Ärzteblatt International⏐Dtsch Arztebl Int 2009; 106(7): 100–5 101

MEDICINETABLE 1The best type of study for epidemiological investigations (from 13)Purpose of investigationInvestigation of rare diseases,e.g. tumorsInvestigation of exposure to rare environmentalfactors, e.g. industrial chemicalsInvestigation of exposure to multiple agents,e.g. the joint effect of oral contraceptiveintake and smokingInvestigation of multiple endpoints,e.g. the risk of death from various causesEstimation of incidence in exposed populationsInvestigation of cofactors that vary over timeInvestigation of cause and effectStudy typeCase-control studiesCohort study in an exposedpopulationCase-control studiesCohort studiesExclusively cohort studiesPreferably cohort studiesIntervention studiestime point, described in sufficient detail? Were the measurementsmade under standardized—and thus comparable—conditionsin all patients? Details of measurementprocedures are important for assessment of accuracy(reliability, validity). The reader must see on whatkind of scale the variables are being measured (e.g. eyecolor, nominal; tumor stage, ordinal; bodyweight,metric), because the type of scale determines what kindof analysis is possible. Descriptive analysis employsdescriptive measured values and graphic and/or tabularpresentations, whereas in statistical analysis the choiceof test has to be taken into consideration. The interpretationand power of the results is also influenced by thescale type. For example, data on an ordinal scale shouldnot be expressed in terms of mean values.Was there a careful power calculation before the studystarted? If the number of cases is too low, a real difference,e.g. between the effects of two medications or inthe risk of disease in the presence vs. absence of a givenenvironmental factor, may not be detected. One thenspeaks of insufficient power.ResultsIn this section the findings should be presented clearlyand objectively, i.e. without interpretation. The interpretationof the results belongs in the ensuing discussion.The results section should address directly the aims ofthe study and be presented in a well-structured, readilyunderstandable and consistent manner. The findingsshould first be formulated descriptively, stating statisticalparameters such as case numbers, mean values, measuresof variation, and confidence intervals. This sectionshould include a comprehensive description of the studypopulation. A second, analytic subsection describes therelationship between characteristics, or estimates theeffect of a risk factor, say smoking behavior, on a dependentvariable, say lung cancer, and may include calculationof appropriate statistical models.Besides information on statistical significance in theform of p values, comprehensive description of the dataand details on confidence intervals and effect sizes arestrongly recommended (14, 15, 16). Tables and figuresmay improve the clarity, and the data therein should beself-explanatory.DiscussionIn this section the author should discuss his/her resultsfrankly and openly. Regardless of the study type, thereare essentially two goals:Comparison of the findings with the status quo—The Discussion should answer the following questions:How has the study added to the body of knowledge onthe given topic? What conclusions can be drawn fromthe results? Will the findings of the study lead the authorto reconsider or change his/her own professional behavior,e.g. to modify a treatment or take previously unconsideredfactors into account? Do the findings suggestfurther investigations? Does the study raise new, hithertounanswered questions? What are the implications of theresults for science, clinical routine, patient care, andmedical practice? Are the findings in accord with thoseof the majority of earlier studies? If not, why might thatbe? Do the results appear plausible from the biologicalor medical viewpoint?Critical analysis of the study's limitations—Mightsources of bias, whether random or systematic in nature,have affected the results? Even with painstaking planningand execution of the study, errors cannot be whollyexcluded. There may, for instance, be an unexpectedlyhigh rate of loss to follow-up (e.g. through patientsmoving away or refusing to participate further in thestudy). When comparing groups one should establishwhether there is any intergroup difference in the compositionof participants lost to follow-up. Such a discrepancycould potentially conceal a true difference betweenthe groups, e.g. in a case-control study with regard to arisk factor. A difference may also result from positiveselection of the study population. The Discussion mustdraw attention to any such differences and describe thepatients who do not complete the study. Possible distortionof the study results by missing values should also bediscussed.Systematic errors are particularly common in epidemiologicalstudies, because these are mostly observationalrather than experimental in nature. In case-control studies,a typical source of error is the retrospective determinationof the study participants' exposure. Their memories maynot be accurate (recall bias). A frequent source of errorin cohort studies is confounding. This occurs when twoclosely connected risk factors are both associated withthe dependent variable. Errors of this type can be correctedand revealed by adjustment for the confoundingfactor. For instance, the fact that smokers drink morecoffee than average could lead to the erroneous assumptionthat drinking coffee causes lung cancer. If potentialconfounders are not mentioned in the publication, thecritical reader should wonder whether the results mightnot be invalidated by this type of error. If possible confoundingfactors were not included in the analysis, thepotential sources of error should at least be critically102 Deutsches Ärzteblatt International⏐Dtsch Arztebl Int 2009; 106(7): 100–5

MEDICINEdebated. Detailed discussion of sources of error and meansof correction can be found in the books by Beagleholeand Webb (17, 18).Results that do not attain statistical significance mustalso be published. Unfortunately, greater importance isstill often attached to significant results, so that they aremore likely to be published than nonsignificant findings.This publication bias leads to systematic distortions inthe body of scientific knowledge. According to a recentreview this is particularly true for clinical studies (3).Only when all valid results of a well-planned and correctlyconducted study are published can useful conclusions bedrawn regarding the effect of a risk factor on the occurrenceof a disease, the value of a diagnostic procedure,the properties of a substance, or the success of an intervention,e.g. a treatment. The investigator and the journalpublishing the article are thus obliged to ensure thatdecisions on important issues can be taken in full knowledgeof all valid, scientifically substantiated findings.It should not be forgotten that statistical significance,i.e. the minimization of the likelihood of a chance result,is not the same as clinical relevance. With a largeenough sample, even minuscule differences can becomestatistically significant, but the findings are not automaticallyrelevant (13, 19). This is true both for epidemiologicalstudies, from the public health perspective,and for clinical studies, from the clinical perspective. Inboth cases, careful economic evaluation is required todecide whether to modify or retain existing practices. Atthe population level one must ask how often the investigatedrisk factor really occurs and whether a slightincrease in risk justifies wide-ranging public healthinterventions. From the clinical viewpoint, it must becarefully considered whether, for example, the slightlygreater efficacy of a new preparation justifies increasedcosts and possibly a higher incidence of side effects. Thereader has to appreciate the difference between statisticalsignificance and clinical relevance in order to evaluatethe results properly.ConclusionsThe authors should concentrate on the most importantfindings. Acrucial question is whether the interpretationsfollow logically from the results. One should avoid conclusionsthat are supported neither by one's own data norby the findings of others. It is wrong to refer to anexploratory data analysis as a proof. Even in confirmatorystudies, one's own results should, for the sake ofconsistency, always be considered in light of otherinvestigators' findings. When assessing the results andformulating the conclusions, the weaknesses of the studymust be given due consideration. The study can attainobjectivity only if the possibility of erroneous or chanceresults is admitted. The inclusion of nonsignificantresults contributes to the credibility of the study. "Notsignificant" should not be confused with "no association."Significant results should be considered from the viewpointof biological and medical plausibility.So-called levels of evidence scales, as used in someAmerican journals, can help the reader decide to whatBOX 2Critical questions❃ Does the study pose scientifically interesting questions?❃ Are statements and numerical data supported by literature citations?❃ Is the topic of the study medically relevant?❃ Is the study innovative?❃ Does the study investigate the predefined study goals?❃ Is the study design apt to address the aims and/or hypotheses?❃ Did practical difficulties (e.g. in recruitment or loss to follow-up) lead to majorcompromises in study implementation compared with the study protocol?❃ Was the number of missing values too large to permit meaningful analysis?❃ Was the number of cases too small and thus the statistical power of the studytoo low?❃ Was the course of the study poorly or inadequately monitored (missing values,confounding, time infringements)?❃ Do the data support the authors' conclusions?❃ Do the authors and/or the sponsor of the study have irreconcilable financial orideological conflicts of interest?extent his/her practice should be affected by the contentof a given publication (20). Until all journals offer recommendationsof this kind, the individual physician'sability to read scientific texts critically will continue toplay a decisive role in determining whether diagnosticand therapeutic practice are based on up-to-date medicalknowledge.ReferencesThe references are to be presented in the journal's standardstyle. The reference list must include all sourcescited in the text, tables and figures of the article. It is importantto ensure that the references are up to date, in orderto make it clear whether the publication incorporatesnew knowledge. The references cited should help thereader to explore the topic further.Acknowledgements and conflict of interest statementThis important section must provide information on anysponsors of the study. Any potential conflicts of interest,financial or otherwise, must be revealed in full (21).Table 2 and Box 2 summarize the essential questionswhich, when answered, will reveal the quality of anarticle. Not all of these questions apply to every publicationor every type of study. Further information on thewriting of scientific publications is supplied by Gardneret al. (19), Altman (7), and Altman et al. (22). Gardner etal. (23), Altman (7), and the CONSORT Statement (12)provide checklists to assist the evaluation of the statisticalcontent of medical studies.Conflict of interest statementThe authors declare no conflicts of interest as defined by the guidelines of theInternational Committee of Medical Journal Editors.Deutsches Ärzteblatt International⏐Dtsch Arztebl Int 2009; 106(7): 100–5 103

MEDICINETABLE 2Checklist to evaluate the quality of scientific publicationsYes Unclear NoDesignIs the aim of the study clearly described? ✮ ✮ ✮Are the study population(s) and the inclusion and exclusion criteria described in detail? ✮ ✮ ✮Were the patients allocated randomly to the different arms of the study? ✮ ✮ ✮If yes:Is the method of randomization described? ✮ ✮ ✮a) Is the number of cases discussed? ✮ ✮ ✮b) Were sufficient cases enrolled (e.g. Power ✞50%)? ✮ ✮ ✮Are the methods of measurement (e.g. laboratory examination, questionnaire, diagnostic test) suitable for determinationof the target variable (with regard to scale, time of investigation, standardization)? ✮ ✮ ✮Is there information regarding data loss (response rates, loss to follow-up, missing values)? ✮ ✮ ✮Study inception and implementationAre treatment and control groups matched with regard to major relevant characteristics(age, sex, smoking habits etc.)? ✮ ✮ ✮Are the drop-outs analyzed for differences between the treatment and control groups? ✮ ✮ ✮How many cases were observed over the whole study period? ✮ ✮ ✮Are side effects and adverse events during the study period described? ✮ ✮ ✮Analysis and evaluationHave the correct statistical parameters and methods been selected, and are they clearly described? ✮ ✮ ✮Are the statistical analyses clearly described? ✮ ✮ ✮Are the important parameters (prognostic factors) included in the analysis or at least discussed? ✮ ✮ ✮Is the presentation of the statistical parameters appropriate, comprehensive, and clear? ✮ ✮ ✮Are the effect sizes and confidence intervals stated for the principal findings? ✮ ✮ ✮Is it apparent why the given study design/statistical methods were chosen? ✮ ✮ ✮Are all conclusions supported by the study's findings? ✮ ✮ ✮By using a checklist such as this, the statistical and methodological soundness of a study can be assessed and improvements considered.Not all of the points in this checklist can be used to evaluate all study types; for example, randomization is particularly applicable to clinical studies.Manuscript submitted on 9. January 2007; revised version accepted on7. June 2007.Translated from the original German by David Roseveare.REFERENCES1. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, RW Scott: Evidencebased medicine: what it is and what it isn't. Editorial. BMJ1996; 312: 71–2.2. Albert DA: Deciding whether the conclusion of studies are justified: areview. Med Decision Making 1981; 1: 265–75.3. Gluud LL: Bias in clinical intervention research. Am J Epidemiol2006; 163: 493–501.4. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication biasin clinical research. Lancet 1991; 337: 867–72.5. Greenhalgh T: How to read a paper: getting your bearings (decidingwhat the paper is about). BMJ 1997; 315: 243–6.6. Kallet H: How to write the methods section of a research paper.Respir Care 2004; 49: 1229–32.7. Altman DG: Practical statistics for medical research. London:Chapman and Hall 1991.8. De Simonian R, Charette LJ, Mc Peek B, Mosteller F: Reporting onmethods in clinical trials. N Engl J Med 1982; 306: 1332–7.9. Trampisch HJ, Windeler J, Ehle B: Medizinische Statistik. Berlin, Heidelberg,New York: Springer 2000, 2nd Revised Edition.10. Klug SJ, Bender R, Blettner M, Lange S: Wichtige epidemiologischeStudientypen. Dtsch Med Wochenschr, 2004; 129: T7–T10.11. Schumacher M, Schulgen G: Methodik klinischer Studien. MethodischeGrundlagen der Planung, Durchführung und Auswertung. Berlin:Springer 2008, 3rd Revised Edition.12. Moher D, Schulz KF, Altman DG für die CONSORT Gruppe: Das CON-SORT Statement: Überarbeitete Empfehlungen zur Qualitätsverbesserungvon Reports randomisierter Studien im Parallel-Design.Dtsch Med Wochenschr 2004; 129: T16–T20.13. Blettner M, Heuer C, Razum O: Critical reading of epidemiologicalpapers. A guide. J Public Health 2001; 11: 97–101.14. Borenstein M: The case for confidence intervals in controlled clinicaltrials. Control Clin Trias 1994; 15: 411–28.15. Gardner MJ, Altman DG: Confidence intervals rather than P values:estimation rather than hypothesis testing. BMJ 1986; 292: 746–50.16. Bortz J, Lienert GA: Kurzgefaßte Statistik für die Klinische Forschung.Leitfaden für die verteilungsfreie Analyse kleiner Stichproben. Berlin:Springer 2003; 2nd Edition, 39–45.17. Beaglehole R, Bonita R, Kjellström T: Einführung in die Epidemiologie.Bern, Göttingen, Toronto, Seattle: Huber 1997.18. Webb P, Bain C, Pirozzo S: Essential epidemiology. An introductionfor students and health professionals. New York: Cambridge UniversityPress 2005.19. Gardner MJ, Machin D, Brynant TN, Altman DG: Statistics with confidence.Confidence intervals and statistical guidelines. London: BMJbooks 2002.104 Deutsches Ärzteblatt International⏐Dtsch Arztebl Int 2009; 106(7): 100–5

MEDICINE20. Ebell MH, Siwek J, Weiss BD et al.: Simplifying the language of evidenceto improve patient care. J Fam Pract 2004; 53: 111–20.21. Bero LA, Rennie D: Influences on the quality of published drug studies.Int J Technol Assess Health Care 1996; 12: 209–37.22. Altman DG, Gore SM, Gardner MJ, Pocock SJ: Statistical guidelinesfor contributers to medical journals. BMJ 1983; 286: 1489–93.23. Gardner MJ, Machin D, Campbell MJ: Use of check lists in assessingthe statistical content of medical studies. BMJ (Clin Res Ed) 1986;292: 810–2.Corresponding authorDr. med. Jean-Baptist du Prel, MPHInstitut für Medizinische Biometrie, Epidemiologie und Informatik (IMBEI)Universitätsklinikum MainzObere Zahlbacher Str. 6955131 Mainz, Germanyduprel@imbei.uni-mainz.deDeutsches Ärzteblatt International⏐Dtsch Arztebl Int 2009; 106(7): 100–5 105