an updated review of related biases - NIHR Journals Library

Health Technology Assessment 2010; Vol. 14: No. 8Dissemination and publicationof research findings: an updatedreview of related biasesF Song, S Parekh, L Hooper,YK Loke, J Ryder, AJ Sutton, C Hing,CS Kwok, C Pang, and I HarveyFebruary 2010DOI: 10.3310/hta14080Health Technology AssessmentNIHR HTA programmewww.hta.ac.uk

HTAHow to obtain copies of this and other HTA programme reportsAn electronic version of this title, in Adobe Acrobat format, is available for downloading free of charge forpersonal use from the HTA website (www.hta.ac.uk). A fully searchable DVD is also available (see below).Printed copies of HTA journal series issues cost £20 each (post and packing free in the UK) to bothpublic and private sector purchasers from our despatch agents.Non-UK purchasers will have to pay a small fee for post and packing. For European countries the cost is£2 per issue and for the rest of the world £3 per issue.How to order:– fax (with credit card details)– post (with credit card details or cheque)– phone during office hours (credit card only).Additionally the HTA website allows you to either print out your order or download a blank order form.Contact details are as follows:Synergie UK (HTA Department)Digital House, The Loddon CentreWade RoadBasingstokeHants RG24 8QWEmail: orders@hta.ac.ukTel: 0845 812 4000 – ask for ‘HTA Payment Services’(out-of-hours answer-phone service)Fax: 0845 812 4001 – put ‘HTA Order’ on the fax headerPayment methodsPaying by chequeIf you pay by cheque, the cheque must be in pounds sterling, made payable to University of Southamptonand drawn on a bank with a UK address.Paying by credit cardYou can order using your credit card by phone, fax or post.SubscriptionsNHS libraries can subscribe free of charge. Public libraries can subscribe at a reduced cost of £100 foreach volume (normally comprising 40–50 titles). The commercial subscription rate is £400 per volume(addresses within the UK) and £600 per volume (addresses outside the UK). Please see our website fordetails. Subscriptions can be purchased only for the current or forthcoming volume.How do I get a copy of HTA on DVD?Please use the form on the HTA website (www.hta.ac.uk/htacd/index.shtml). HTA on DVD is currently freeof charge worldwide.The website also provides information about the HTA programme and lists the membership of the variouscommittees.

Dissemination and publication ofresearch findings: an updated review ofrelated biasesF Song, 1,2 * S Parekh, 1,2 L Hooper, 1YK Loke, 1 J Ryder, 1 AJ Sutton, 3 C Hing, 4CS Kwok, 1 C Pang, 1 and I Harvey 11School of Medicine, Health Policy and Practice, University of East Anglia,Norwich, UK2School of Allied Health Professions, University of East Anglia, Norwich, UK3Department of Health Sciences, University of Leicester, UK4Watford General Hospital, Hertfordshire, UK*Corresponding authorDeclared competing interests of authors: nonePublished February 2010DOI: 10.3310/hta14080This report should be referenced as follows:Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publicationof research findings: an updated review of related biases. Health Technol Assess 2010;14(8).Health Technology Assessment is indexed and abstracted in Index Medicus/MEDLINE, ExcerptaMedica/EMBASE, Science Citation Index Expanded (SciSearch ® ) and Current Contents ® /ClinicalMedicine.

NIHR Health Technology Assessment programmeThe Health Technology Assessment (HTA) programme, part of the National Institute for HealthResearch (NIHR), was set up in 1993. It produces high-quality research information on theeffectiveness, costs and broader impact of health technologies for those who use, manage and provide carein the NHS. ‘Health technologies’ are broadly defined as all interventions used to promote health, preventand treat disease, and improve rehabilitation and long-term care.The research findings from the HTA programme directly influence decision-making bodies such as theNational Institute for Health and Clinical Excellence (NICE) and the National Screening Committee(NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that theyform a key component of the ‘National Knowledge Service’.The HTA programme is needs led in that it fills gaps in the evidence needed by the NHS. There are threeroutes to the start of projects.First is the commissioned route. Suggestions for research are actively sought from people working in theNHS, from the public and consumer groups and from professional bodies such as royal colleges and NHStrusts. These suggestions are carefully prioritised by panels of independent experts (including NHS serviceusers). The HTA programme then commissions the research by competitive tender.Second, the HTA programme provides grants for clinical trials for researchers who identify researchquestions. These are assessed for importance to patients and the NHS, and scientific rigour.Third, through its Technology Assessment Report (TAR) call-off contract, the HTA programmecommissions bespoke reports, principally for NICE, but also for other policy-makers. TARs bring togetherevidence on the value of specific technologies.Some HTA research projects, including TARs, may take only months, others need several years. Theycan cost from as little as £40,000 to over £1 million, and may involve synthesising existing evidence,undertaking a trial, or other research collecting new data to answer a research problem.The final reports from HTA projects are peer reviewed by a number of independent expert referees beforepublication in the widely read journal series Health Technology Assessment.Criteria for inclusion in the HTA journal seriesReports are published in the HTA journal series if (1) they have resulted from work for the HTAprogramme, and (2) they are of a sufficiently high scientific quality as assessed by the referees andeditors.Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search, appraisaland synthesis methods (to minimise biases and random errors) would, in theory, permit the replicationof the review by others.The research reported in this issue of the journal was commissioned by the National Coordinating Centrefor Research Methodology (NCCRM), and was formally transferred to the HTA programme in April 2007under the newly established NIHR Methodology Panel. The HTA programme project number is 06/92/02.The contractual start date was in October 2007. The draft report began editorial review in November2008 and was accepted for publication in June 2009. The commissioning brief was devised by the NCCRMwho specified the research question and study design. The authors have been wholly responsible for alldata collection, analysis and interpretation, and for writing up their work. The HTA editors and publisherhave tried to ensure the accuracy of the authors’ report and would like to thank the referees for theirconstructive comments on the draft document. However, they do not accept liability for damages or lossesarising from material published in this report.The views expressed in this publication are those of the authors and not necessarily those of the HTAprogramme or the Department of Health.Editor-in-Chief:Series Editors:Professor Tom Walley CBEDr Martin Ashton-Key, Dr Aileen Clarke, Professor Chris Hyde,Dr Tom Marshall, Dr John Powell, Dr Rob Riemsma and Professor Ken SteinISSN 1366-5278© 2010 Queen’s Printer and Controller of HMSOThis journal may be freely reproduced for the purposes of private research and study and may be included in professional journals provided thatsuitable acknowledgement is made and the reproduction is not associated with any form of advertising.Applications for commercial reproduction should be addressed to: NETSCC, Health Technology Assessment, Alpha House, University ofSouthampton Science Park, Southampton SO16 7NS, UK.Published by Prepress Projects Ltd, Perth, Scotland (www.prepress-projects.co.uk), on behalf of NETSCC, HTA.Printed on acid-free paper in the UK by Henry Ling Ltd, The Dorset Press, Dorchester.MR

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8AbstractDissemination and publication of research findings: anupdated review of related biasesF Song, 1,2 * S Parekh, 1,2 L Hooper, 1 YK Loke, 1 J Ryder, 1 AJ Sutton, 3C Hing, 4 CS Kwok, 1 C Pang, 1 and I Harvey 11School of Medicine, Health Policy and Practice, University of East Anglia, Norwich, UK2School of Allied Health Professions, University of East Anglia, Norwich, UK3Department of Health Sciences, University of Leicester, UK4Watford General Hospital, Hertfordshire, UK*Corresponding authorObjectives: To identify and appraise empirical studieson publication and related biases published since 1998;to assess methods to deal with publication and relatedbiases; and to examine, in a random sample of publishedsystematic reviews, measures taken to prevent, reduceand detect dissemination bias.Data sources: The main literature search, in August2008, covered the Cochrane Methodology RegisterDatabase, MEDLINE, EMBASE, AMED and CINAHL. InMay 2009, PubMed, PsycINFO and OpenSIGLE werealso searched. Reference lists of retrieved studies werealso examined.Review methods: In Part I, studies were classifiedas evidence or method studies and data were extractedaccording to types of dissemination bias or methodsfor dealing with it. Evidence from empirical studieswas summarised narratively. In Part II, 300 systematicreviews were randomly selected from MEDLINE andthe methods used to deal with publication and relatedbiases were assessed.Results: Studies with significant or positive resultswere more likely to be published than those withnon-significant or negative results, thereby confirmingfindings from a previous HTA report. There wasconvincing evidence that outcome reporting bias existsand has an impact on the pooled summary in systematicreviews. Studies with significant results tended to bepublished earlier than studies with non-significantresults, and empirical evidence suggests that publishedstudies tended to report a greater treatment effect thanthose from the grey literature. Exclusion of non-Englishlanguagestudies appeared to result in a high risk ofbias in some areas of research such as complementaryand alternative medicine. In a few cases, publicationand related biases had a potentially detrimental impacton patients or resource use. Publication bias can beprevented before a literature review (e.g. by prospectiveregistration of trials), or detected during a literaturereview (e.g. by locating unpublished studies, funnel plotand related tests, sensitivity analysis modelling), orits impact can be minimised after a literature review(e.g. by confirmatory large-scale trials, updating thesystematic review). The interpretation of funnel plot andrelated statistical tests, often used to assess publicationbias, was often too simplistic and likely misleading. Moresophisticated modelling methods have not been widelyused. Compared with systematic reviews published in1996, recent reviews of health-care interventions weremore likely to locate and include non-English-languagestudies and grey literature or unpublished studies, andto test for publication bias.Conclusions: Dissemination of research findingsis likely to be a biased process, although the actualimpact of such bias depends on specific circumstances.The prospective registration of clinical trials and theendorsement of reporting guidelines may reduceresearch dissemination bias in clinical research. Insystematic reviews, measures can be taken to minimisethe impact of dissemination bias by systematicallysearching for and including relevant studies that aredifficult to access. Statistical methods can be usefulfor sensitivity analyses. Further research is needed todevelop methods for qualitatively assessing the risk ofpublication bias in systematic reviews, and to evaluatethe effect of prospective registration of studies, openaccess policy and improved publication guidelines.iii© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8ContentsList of abbreviations ................................. viiExecutive summary .................................. ix1 Introduction .............................................. 1Definition of publication and relatedbiases ...................................................... 12 Review objectives and methods ............... 5Objectives ................................................... 5Review of empirical and methodologicalstudies .................................................... 5Assessment of a sample of publishedreviews ................................................... 73 Evidence from cohort studies ofpublication bias ......................................... 9Inception cohort studies ........................... 9Regulatory cohort studies ........................... 10Cohorts of meeting abstracts ...................... 11Manuscript cohort studies .......................... 11Pooled analyses of cohort studies .............. 12Factors associated with publication bias ..... 12Discussions of findings from cohortstudies .................................................... 14Conclusions ................................................ 194 Evidence of different types ofdissemination bias ..................................... 21Outcome reporting bias ............................ 21Time lag bias .............................................. 24Grey literature bias ..................................... 26Language bias ........................................... 29Citation bias ............................................... 31Duplicate (multiple) publication ................ 32Place of publication bias ............................. 34Country bias ............................................... 34Database indexing bias ............................... 35Media attention bias ................................... 35Limitations of the available evidence ......... 36Conclusions ............................................... 375 Consequences of dissemination bias ....... 39Basic research studies ................................. 39Observational studies ................................. 39Clinical trials .............................................. 39Summary .................................................... 406 Sources of publication bias ...................... 41Investigators and authors ........................... 41Editorial review process ............................. 42Readers and users of research findings ...... 46Research funding bodies and commercialinterests .................................................. 46Variation in study results ............................ 48Summary .................................................... 497 Prevention of publication bias ................. 51Changes in publication process ................. 51Prospective registration of trials ................. 53Open access policy ..................................... 54Right to publication .................................. 55Research sponsors’ guidelines .................... 55Confirmatory large-scale trials ................... 55Summary ................................................... 568 Reducing or detecting publication andrelated biases in systematic reviews ....... 57Literature searching .................................. 57Locating ongoing or unpublished trials .... 58Assessing the risk of publication bias ......... 59Funnel plot and related statisticalmethods ................................................. 60Other statistical and modelling methods ... 64Sophisticated modelling methods .............. 64Fixed or random-effects models ................ 65Updating systematic reviews ..................... 66Summary ................................................... 689 Survey of published systematicreviews ....................................................... 69Assessment of randomly selectedreviews .................................................... 69Assessors’ judgement ................................. 73Assessment of reviews that explicitlytested for publication bias ...................... 75Summary .................................................... 7810 Discussion .................................................. 81What is evidence on publication bias? ........ 81How to deal with publication bias? ............ 82Dealing with publication bias in publishedsystematic reviews .................................. 83Implications for researchers and decisionmakers.................................................... 85Recommendations for future research ....... 85v© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

ContentsAcknowledgements .................................. 87References ................................................. 89Appendix 1 Search strategies for electronicdatabases .................................................... 111Appendix 2 Data extraction sheet forempirical studies ........................................ 115Appendix 3 Data extraction sheet formethodological studies ............................... 117Appendix 4 Data extraction sheet –systematic reviews of treatment ................. 119Appendix 5 Main characteristics ofinception cohort studies of publicationbias ............................................................. 123Appendix 6 Main characteristics ofincluded regulatory cohort studies ofpublication bias: trials submitted toregulatory authorities ................................. 127Appendix 7 Main characteristics ofabstract cohort studies of publicationbias: abstracts presented at conferences ..... 129Appendix 8 Main characteristics ofmanuscript cohort studies of publicationbias: manuscripts submitted to journals ..... 135Appendix 9 Outcome reporting bias –characteristics of included studies .............. 137Appendix 10 Time lag bias – includedempirical studies ........................................ 141Appendix 11 Grey literature bias –included empirical studies ......................... 147Appendix 12 Language bias –included empirical studies ......................... 153Appendix 13 Citation bias –included empirical studies ......................... 155Appendix 14 Reasons given byinvestigators for studies not beingpublished .................................................... 157Appendix 15 Study findings and theacceptance of submitted manuscripts ........ 163Appendix 16 Case studies indicatingpharmaceutical companies or industry researchsponsorship as a source of publication andrelated biases .............................................. 165Appendix 17 List of 347 reviewsassessed ....................................................... 169Appendix 18 Original study proposal ...... 183Health Technology Assessment reportspublished to date ...................................... 195Health Technology Assessmentprogramme ............................................... 217vi

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8List of abbreviationsAIDSCDUSCICINAHLacquired immunodeficiencysyndromeClinical Data Update Systemconfidence intervalCumulative Index to Nursingand Allied Health LiteratureITTJIFLILACSNIHintention-to-treatjournal impact factorLatin American andCaribbean Health SciencesLiteratureNational Institutes of HealthCMRDCochrane MethodologyRegister DatabaseNSAIDnon-steroidal antiinflammatorydrugCONSORTCSRConsolidated Standards ofReporting TrialsCochrane Systematic ReviewORQUOROModds ratioQuality of Reporting of MetaanalysesCTSPDAREEQUATORFDAHRHRHRHRTICHICMJEIPDIQRIRBClinical Trials Search Portal(WHO clinical trial register)Database of Abstracts ofReviews of EffectivenessEnhancing the Quality andTransparency of HealthResearchUnited States Food and DrugAdministrationhazard ratiohazard ratio of hazard ratioshormone replacementtherapyInternational Conference onHarmonisationInternational Committee ofMedical Journal Editorsindividual patient datainterquartile rangeInstitutional Review BoardR&DRCTRECRORRRRTOGSIGLESSRISTARDSTROBETSAWHOresearch and developmentrandomised controlled trialResearch Ethics Committeeratio of odds ratiosrelative risk or rate ratioRadiation Therapy OncologyGroupSystem for Information onGrey Literature in Europeselective serotonin reuptakeinhibitorStatement for ReportingStudies of DiagnosticAccuracyStandards for the Reportingof Observational Studies inEpidemiologytrial sequential analysisWorld Health OrganizationISRCTNInternational StandardRandomised Controlled TrialNumberAll abbreviations that have been used in this report are listed here unless the abbreviation is wellknown (e.g. NHS), or it has been used only once, or it is a non-standard abbreviation used only infigures/tables/appendices, in which case the abbreviation is defined in the figure legend or in thenotes at the end of the table.vii© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Executive summaryBackgroundThe validity of research synthesis is threatenedif studies with significant or striking findings aremore likely to be published than those with nonsignificantresults. A previous Health TechnologyAssessment (HTA) monograph published in 2000by the present authors reviewed studies onpublication and related biases. Since then, manynew studies on publication and related biaseshave been published. This report aims to updatethe 2000 HTA monograph on publication bias bysynthesising findings from previous studies andnewly indentified ones.Objectives• To identify and appraise empirical studies onpublication and related biases published since1998.• To assess the usefulness and limitations ofavailable methods to deal with publication andrelated biases.• To examine in a random sample of publishedsystematic reviews, measures taken by theauthors to prevent, reduce and detect differenttypes of dissemination bias.MethodsPart I: Review of evidence andmethod studiesStudy selectionThe report included evidence studies that providedempirical evidence on the existence, consequences,causes and/or risk factors of dissemination bias;and method studies that developed or evaluatedmethods for preventing, reducing or detectingdissemination bias.Data sourcesThe following electronic databases were searched:Cochrane Methodology Register Database(CMRD), MEDLINE, EMBASE, AMED andCINAHL. The main literature search wasconducted in August 2008 and a final searchof PubMed, PsycINFO and OpenSIGLE was© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.conducted in May 2009 to identify more recentlypublished studies. We also examined reference listsof retrieved studies.Data extraction and synthesisThe identified studies were classified by onereviewer as evidence or method studies and checkedby a second reviewer. One reviewer extracted datadirectly into tables (specifically designed accordingto types of bias or methods), which were checked bya second reviewer. Evidence from empirical studieswas summarised narratively. Where appropriate,the results have been quantitatively pooled.Part II: Survey of publishedsystematic reviewsWe searched MEDLINE for systematic reviewspublished in 2006, and randomly selected 100reviews of effects of health-care interventions,50 reviews of diagnostic accuracy, 100 reviewsof association between risk factors and healthoutcomes, and 50 reviews of gene-diseaseassociations. We assessed the methods used todeal with publication and related biases in thesesystematic reviews.ResultsEmpirical evidence ondissemination biasUpdated analyses of data from cohort studiesconfirmed findings from the previous HTA reportthat studies with significant or positive resultsare more likely to be published than those withnon-significant or negative results. Publicationbias occurs mainly before the presentation offindings at conferences and before the submissionof manuscripts to journals. Recent high-qualitystudies have provided convincing evidence thatoutcome reporting bias exists and has an importantimpact on the pooled summary in systematicreviews. Studies with significant results tend onaverage to be published earlier than studies withnon-significant results, although the new evidenceis less clear than that from the previous review. Newempirical evidence suggests that published studiestend to report a greater treatment effect than thoseix

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 1IntroductionSynthesis of published research is becomingincreasingly important in providing relevantand valid research evidence to clinical and healthpolicy decision-making. However, the validity ofresearch synthesis based on published literaturewill be threatened if published studies comprisea biased selection of all studies that have beenconducted. 1A previous Health Technology Assessment (HTA)monograph published in 2000 comprehensivelyreviewed studies that provided empirical evidenceof publication and related biases, and studiesthat developed or tested methods for preventing,reducing or detecting publication and relatedbiases. 2 The review found evidence indicating thatstudies with significant or favourable results weremore likely to be published, or were likely to bepublished earlier than those with non-significantor unimportant results. There was limited andindirect evidence indicating the possibility of fullpublication bias, outcome reporting bias, duplicatepublication bias, and language bias. The reviewidentified little empirical evidence relating to theimpact of publication and related biases on healthpolicy, clinical decision-making and the outcomeof patient management. Considering that thespectrum of the accessibility of research results(dissemination profile) ranges from completelyinaccessible to easily accessible, it was suggestedthat a single term, ‘dissemination bias’, could beused to denote all types of publication and relatedbiases. 2In the previous HTA report published in 2000, theavailable methods for dealing with disseminationbiases were classified according to measures thatcould be taken before, during or after a literaturereview: to prevent publication bias before aliterature review (e.g. prospective registration oftrials), to reduce or detect publication and relatedbiases during a literature review (e.g. locating greyliterature or unpublished studies, and funnel plotrelated methods), and to minimise the impactof publication bias after a literature review (e.g.confirmatory large-scale trials, updating systematicreviews). 2 It was concluded that the ideal solutionto publication bias is the prospective, universalregistration of all studies at their inception. It was© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.also concluded, although debatable, that availablestatistical methods for detecting and adjustingpublication bias should be mainly used for thepurpose of sensitivity analysis. 2Since the publication of the 2000 HTA reporton publication bias, many new empirical studieson publication and related biases have beencompleted and published. For example, Eggeret al. (2003) provided further empirical evidenceon publication bias, language bias, grey literaturebias and MEDLINE index bias, 3 and Moher et al.(2003) evaluated language bias in meta-analysesof randomised controlled trials. 4 Recently, moreconvincing evidence on outcome reporting bias hasbeen published. 5–7 The new empirical evidence maycontradict or strengthen the empirical evidenceincluded in the previous HTA report. There arealso some new published studies that investigatedmethods for dealing with publication bias (e.g.references 8 and 9). More importantly, perhaps,new initiatives have been introduced to enhancethe prospective registration of clinical trials. 10This report aims to update the 2000 HTA reporton publication bias, by incorporating findingsfrom newly identified studies. We first discuss theconcepts and definitions about publication andrelated biases in this chapter. After a descriptionof review objectives and methods in Chapter2, evidence from empirical studies on theexistence and consequences of publication bias issummarised in Chapters 3 to 5. We discuss sourcesof publication bias in Chapter 6, while methodsfor dealing with publication bias are examinedin Chapters 7 and 8. The results of a survey ofsystematic reviews published in 2006 are presentedin Chapter 9. Finally, the major findings of thisupdated review are discussed in Chapter 10.Definition of publication andrelated biasesThe observation that many studies are neverpublished was termed ‘the file-drawer problem’by Rosenthal in 1979. 11 The importance of thisproblem depends on whether or not the publishedstudies are representative of all studies that have1

Introductionbeen conducted. If the published studies are thesame as, or a random sample of, all studies thathave been conducted, there will be no bias and theaverage estimate based on the published studieswill be similar to that based on all studies. If thepublished studies comprise a biased sample of allstudies that have been conducted, the results of aliterature review will be misleading. 12 The efficacyof a treatment will be exaggerated if studies withpositive results are more likely to be published thanthose with negative results.Publication bias is specifically defined as ‘thetendency on the parts of investigators, reviewers,and editors to submit or accept manuscripts forpublication based on the direction or strengthof the study findings’. 13 In the definition ofpublication bias, there are two basic concepts: studyfindings and publication status. Study findings arecommonly classified as being statistically significantor non-significant. In addition, study results may beclassified as being positive or negative, supportiveor unsupportive, favoured or disliked, strikingor unimportant. It should be noted that theclassification of study findings is often dependenton subjective judgement and may be unreliable.For example, people may have a differentunderstanding about what are positive or negativefindings.The formats of publication include full publicationin journals, presentation at scientific conferences,reports, book chapters, discussion papers,dissertations or theses. In fact, ‘publication is nota dichotomous event: rather it is a continuum’. 14Although a study that appears in a full report in ajournal is generally regarded as published, theremay be different opinions about whether it shouldbe classified as published or unpublished whenresults are presented in other formats.The accessibility of research results is dependentnot only on whether a study is published but alsoon when, where and in what format this occurs. Inthe 2000 HTA report on publication bias, we usedthe term ‘dissemination profile’ to describe theaccessibility of research results, or the possibilityof research findings being identified by potentialusers. The spectrum of the dissemination profileranges from completely inaccessible to easilyaccessible, according to whether, when, where andhow research is presented or stored. Disseminationbias occurs when the dissemination profile ofa study is determined by its results. The termdissemination bias could be used to embracepublication bias and other related biases caused bytime, type and language of publication, multiplepublication, selective citation, database index, andbiased media attention (see Box 1 for definitions).The advantages of the term ‘dissemination bias’are that it avoids the need to define publicationstatus and it is more directly related to accessibilitythan publication. For example, media attentioncan have a major impact on dissemination, but itis not normally included within the definition ofpublication bias. Along with the development ofinformation technology and changes in regulationsand policy, data from some ‘unpublished’ studiesmay be conveniently accessible to the public,and formal publication in journals is only oneof several ways to disseminate research findings.Therefore, dissemination bias is a better expressionto replace this broad use of the term publicationbias. However, the term ‘publication bias’ and‘publication and related biases’ are alreadyestablished in research literature and they will alsobe used for discussion in this report.2

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 2Review objectives and methodsThe current review is an update of a previousHealth Technology Assessment (HTA) report. 2This review is divided into two parts: a reviewof empirical and methodological studies onpublication and related biases, and a survey ofpublication bias in a sample of published systematicreviews.Objectives1. To identify relevant evidence studies publishedsince 1998. Evidence studies are defined asthose that provide empirical evidence on theexistence, consequences, causes and risk factorsof dissemination bias.2. To identify relevant method studies publishedsince 1998. Method studies are those thathave developed or investigated methodsfor preventing, reducing or detectingdissemination bias.3. To categorise evidence and method studiesidentified according to a conceptual frameworkof dissemination profile, and to criticallyappraise studies that provided direct empiricalevidence.4. To synthesise findings from newly identifiedand previously included studies to enable us toassess whether each type of dissemination biasdoes exist, and if so the extent of the effect thatit may have on results of systematic reviews andhence decision-making.5. To assess the usefulness and limitations ofavailable methods through synthesis of themethodological studies.6. To examine measures taken in a representativesample of published systematic reviews toprevent, reduce and detect different types ofdissemination bias. We included both narrativeand quantitative (meta-analytic) systematicreviews that evaluated effect of health-careinterventions, systematic reviews that evaluatedthe accuracy of diagnostic tests, systematicreviews that evaluated association betweengenes and disease, and systematic reviewsof epidemiological studies that evaluatedassociation of risk factors and health outcomes.7. To bring together current evidence onthe existence and scale of each type of© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.dissemination bias, effects of methods tocombat these biases, and current use ofthese methods to create recommendationsfor reviewers, policy-makers and healthprofessionals.Review of empirical andmethodological studiesCriteria for inclusionWe included studies that provide empiricalevidence on the existence, consequences, causesand/or risk factors of types of dissemination bias;and method studies that develop or evaluatemethods for preventing, reducing or detectingdissemination bias in biomedical or health-relatedresearch. Evidence studies are those that providedempirical evidence on the existence, consequences,causes and risk factors of types of disseminationbias; and method studies are those whose mainobjectives involved one of the following: to developor evaluate methods for preventing, reducing ordetecting dissemination bias. In some cases, a studymay be considered as both an evidence study and amethod study.Literature search strategyThe following health-related or biomedicalbibliographic databases were searched to identifyrelevant studies pertaining to empirical evidenceand methodological issues concerning publicationand related biases: MEDLINE, the CochraneMethodology Register Database (CMRD),EMBASE, AMED and the Cumulative Index toNursing and Allied Health Literature (CINAHL).The strategies used to search these electronicdatabases are presented in Appendix 1. Theperiod searched was from 1998 to August 2008.A further search of PubMed (from 2008 to 2009),PsycINFO (from 1998 to 2009), and OpenSIGLE(from 1998 to 2009) was carried out in May 2009by one reviewer to identify relevantly published orgrey literature studies. References (titles with orwithout abstracts) gathered by searching MEDLINEand CMRD were independently examined by tworeviewers. References from other databases were5

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8of dissemination profile of cohorts of studies.Indirect evidence referred to findings that couldpresumably have some relation with disseminationbias but where other alternative explanations couldnot be completely excluded. The availability ofempirical evidence is very different for differenttypes of research dissemination bias. This updatedreview focused on direct evidence, althoughindirect evidence was also considered when directevidence was limited or absent.The initial search of the electronic databasesyielded a total of 1353 records, with muchduplication, many studies being indexed inseveral different databases. These search resultswere assessed by one reviewer and 705 potentiallyrelevant articles were identified. These studieswere then independently assessed by two reviewersbased on their abstracts. Finally, 300 studies wereincluded, of which 109 were classified as evidencestudies, 52 as method studies and 9 as bothevidence and methods. The remaining studies wereclassified as background or other studies.Data extraction and synthesisWe planned to apply a checklist of qualityassessment critically to appraise studies thatprovided empirical evidence (Appendix 2).However, we found it was extremely difficultbecause of poor reliability of the checklist; differentreviewers often disagreed about the overall qualityof studies. The task of quality assessment was mademore difficult because the designs and objectives ofrelevant studies in this review were highly diverse.Considering the very limited time available, wedecided not to apply the checklist for qualityassessment. However, we did try to identify andsummarise the main limitations in studies thatprovided empirical evidence on publication biasin the review, although this assessment of studyvalidity was not as systematic as specified in theprotocol.Initially, data from the included studies wereindependently extracted by two reviewers usingseparate data extraction forms for empirical andmethodological studies (Appendix 2 and Appendix3) and any disagreements were resolved byconsensus. However, we found that data extractedusing Appendix 2 and Appendix 3 were ofteninsufficient, and the extraction of data from studiesdirectly into study tables was more flexible andefficient. To save time, one reviewer extracted datadirectly into tables, which were checked by a secondreviewer.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Findings from the newly identified studies and thepreviously identified studies 2 are summarised toassess each type of publication bias and the impactof these biases on the results of the systematicreview and consequently decision-making.Evidence and method studies were narrativelysynthesised. Where judged appropriate, the resultshave been quantitatively pooled (e.g. the odds ratioof full publication of studies according to results).Heterogeneity across studies within each subgroupwas measured using the I 2 statistic. 16 Meta-analyseswere carried out using Review Manager (RevManVersion 5.0. Copenhagen: The Nordic CochraneCentre, The Cochrane Collaboration, 2008).Assessment of a sample ofpublished reviewsIn the previous HTA report, 193 systematic reviewstaken from the Database of Abstracts of Reviews ofEffectiveness (DARE) were used to identify furtherevidence of dissemination bias and to illustratethe methods used in systematic reviews for dealingwith publication bias. However, there were severalshortcomings in our previous assessment. Firstly,systematic reviews included in the DARE databasemight on average have been of better quality thanthose from the general bibliographic databases(such as MEDLINE) so the representativenessof systematic reviews assessed in the previousHTA report was questionable. Secondly, 91% ofsystematic reviews evaluated effectiveness of healthcareinterventions and 9% evaluated the accuracyof diagnostic technologies, and these were notseparately assessed. The problem of disseminationbias might be different between the two types ofsystematic reviews. Thirdly, neither reviews ofepidemiological studies of association between riskfactors and health outcomes nor reviews of studiesof association between genes and diseases wereincluded in the previous HTA report.To overcome these shortcomings, in the currentupdated review we have obtained a representativesample of systematic reviews from the generalbibliographic database MEDLINE and haveseparately assessed them as (1) systematic reviewsof studies on effects of health-care interventionsor treatment reviews, (2) systematic reviews ofepidemiological studies on association between riskfactors and health outcomes or epidemiologicalreviews, (3) systematic reviews of genetic studies onassociation between genes and disease or geneticreviews, and (4) systematic reviews of studies onaccuracy of diagnostic tests or diagnostic reviews.7

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 3Evidence from cohort studiesof publication biasEvidence of publication bias can be classifiedas direct or indirect. 17 Indirect evidenceincludes observations of a disproportionately highpercentage of positive findings in the publishedliterature, and a larger effect size in small studiesas compared with large studies. This evidence isindirect because factors other than publicationbias may also lead to the observed disparities. Theexistence of publication bias was first suspectedby Sterling in 1959, after observing that 97%of studies published in four major psychologyjournals were statistically significant. 18 In 1995, thesame author concluded that the practices leadingto publication bias had not changed over a periodof 30 years. 19Direct evidence includes the admissions of biason the part of those involved in the publicationprocess (investigators, referees or editors),comparison of the results of published andunpublished studies, and the follow-up of cohortsof registered studies. 2 The 2000 HTA reporton publication bias included both direct andindirect evidence. Because of a large amount ofnew direct evidence, this updated review focuseson direct evidence from empirical studies, butindirect evidence will also be considered whendirect evidence is limited. Surveys of authors andinvestigators that provide evidence on publicationbias are included in Chapter 5 (sources ofpublication bias).This section includes any empirical studies thattracked a cohort of studies before their formalpublication and reported the rate of publication bystudy results. Cohort studies on time to publicationand selective outcome reporting will be reviewedlater. Included cohort studies of publication biaswere classified into four subgroups according tothe starting point of follow-up of cohorts: inceptioncohort studies, regulatory cohort studies, abstractcohort studies and manuscript cohort studies.A study that followed up a cohort of researchfrom their beginning (even if retrospectively) wastermed an inception cohort study. A regulatorycohort study refers to a study that examined formalpublication of research submitted to regulatory© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.authorities. An abstract cohort study refers toa study that investigated the subsequent fullpublication of abstracts presented at conferences.A manuscript cohort study refers to a study ofmanuscripts submitted to journals. Primary studiesincluded in cohort studies of publication bias maybe clinical trials, observational studies or basicresearch.Publication of a study was usually defined as thefull publication in journals. However, study resultsmay be categorised differently in the includedcohort studies. In this review, study results wereclassified as ‘statistically significant (p ≤ 0.05)’ versus‘non-significant (p > 0.05)’ or ‘positive’ versus‘non-positive’. Positive results included those thatwere considered being ‘positive’, ‘favourable’,‘significant’, ‘important’, ‘striking’, ‘showed effect’and ‘confirmatory’. Non-positive result refers toother results labelled as being ‘negative’, ‘nonsignificant’,‘less or not important’, ‘invalidating’,‘inconclusive’, ‘questionable’, ‘null’ and ‘neutral’.Inception cohort studiesFive cohort studies were included in the 2000HTA report. 20–24 Cohorts of research protocolsapproved by Research Ethics Committees (REC),Institutional Review Boards (IRB) or registered byresearch sponsors were followed up to investigatefactors associated with subsequent publication. Thestudy by Ioannidis included randomised controlledtrials (RCTs) conducted by two groups of trialists[sponsored by the National Institutes of Health(NIH) from 1986 to 1996] and was focused mainlyon time lag bias. 23 The rate of publication rangedfrom 60% to 98% for studies with statisticallysignificant results and from 20% to 85% for studieswith statistically non-significant results. Dickersin(1997) combined the results from four cohortstudies 20–22,24 and found that the pooled adjustedodds ratio (OR) for publication bias (publication ofstudies with significant or important results versusthose with unimportant results) was 2.54 (95% CI:1.44 to 4.47). 259

Evidence from cohort studies of publication bias10The updated review included seven additionalinception cohorts that provided data on the rate ofpublication according to study results (Appendix5). 26–32 Seven other inception cohort studieswere excluded because they did not examinethe association between publication and studyresults. 33–39The cohort study by Bardy (1998) included 188 ofthe 274 drug trials notified to the Finnish NationalAgency in 1987. 26 Study results were classified asbeing positive if the risk-benefit ratio was in favourof the drug under investigation, or if the objectiveof the study was supported. Results were consideredinconclusive if the risk-benefit assessment wasinconclusive or if the study was non-comparative,whereas studies were judged as negative if thebenefit-risk ratio was not in favour of the drug orno different from placebo. The rate of publicationwas 47% for positive results, 33% for inconclusiveresults, and 11% for negative results. 26Cronin and Sheldon (2004) sent a questionnaireto project leaders of 101 projects sponsored bythe UK NHS R&D (research and development)programme to obtain information on studyfindings and publication status. 27 The methodsuggested by Dickersin and Min 40 was used todefine study results. Studies were categorised as‘showed (an) effect’ or not, depending on whetherresults were statistically significant (p < 0.05) orconsidered to be of great importance. The rate ofpublication of studies with statistically significant orimportant results was not statistically significantlydifferent from those with non-significant or nonimportantresults (76% versus 64%). 27Two cohort studies by Decullier and colleagues(2005, 2006) followed up biomedical researchprotocols approved by French RECs in 1994 and1997. 28,29 In one of the two French studies, resultsof 501 completed studies were classified by originalinvestigators as being confirmatory, invalidatingor inconclusive (see Appendix 5 for details). 28 Therate of publication of studies with confirmatoryresults was higher than those with inconclusiveresults (OR = 4.59; 95% CI: 2.21 to 9.54). 28 In theother French study of 47 completed studies, theimportance of results was subjectively rated byinvestigators from 1 to 10, and important resultswere those graded as > 5. 29 The rate of publicationwas 70% for studies with important results and 60%for those with less important results (OR = 1.58;95% CI: 0.37 to 6.71). 29Misakian and Bero identified a cohort of 61 passivesmoking research projects that were sponsored by76 organisations between 1981 and 1995. 30 A semistructuredtelephone interview of investigators wascarried out to verify study results and publicationstatus. Study results were classified as statisticallysignificant or statistically non-significant. Themixed result refers to a situation in which at leastone of multiple primary outcomes was statisticallysignificant. The rate of publication was 85%for statistically significant results, 86% for nonsignificantresults, and only 14% for the mixedresults. 30The publication status of 68 RCTs processedthrough the pharmacy department of an eyehospital since 1963 was examined by Wormald etal. 31 This study was published only as a conferenceabstract and additional data were provided inDwan et al. 41 The rate of publication was 93% forstatistically significant results and 71% for nonsignificantresults.Zimpel and Windeler investigated thesubsequent publication of 140 medical theseson complementary medical subjects. 32 Resultswere classified as positive or non-positive (thisclassification is slightly unclear as the article waspublished in German). Publication status wastracked by searching MEDLINE and by personalcommunication with authors or supervisors. Therate of publication was 40% for positive results and28% for negative results. 32Regulatory cohort studiesNo regulatory cohort studies of publication biaswere included in the previous HTA report. In thisupdated review we identified four regulatory cohortstudies that examined formal publication of clinicaltrials submitted to regulatory authorities (Appendix6). 42–45 Of the four regulatory cohort studies, twodid not specify clinical fields 42,44 and two focusedon antidepressants. 43,45 One study was not includedbecause the association of journal publication andstudy results was not reported. 46Melander et al. conducted a study of 42randomised placebo-controlled trials of fiveantidepressants submitted by industry to theSwedish drug regulatory authority for marketingapproval. 43 Studies were classified according towhether they found the test drug was significantlymore effective than the placebo with the primaryoutcome. Publication status (including stand-alone,pooled or multiple publications) of the trials wasinvestigated by searching electronic bibliographic

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8databases and contacting the companies. All 21studies with significant results were published(stand-alone or pooled) while only 81% of studieswith non-significant results were published. 43Turner et al. examined 74 clinical trials of 12antidepressant agents submitted to the US Foodand Drug Administration (FDA) between 1987and 2004. 45 Trial results were classified as positive,questionable or negative according to the FDA’sregulatory decisions. Publication status of trials wasdetermined by searching literature databases andcontacting trial sponsors. The rate of publicationwas 97% for studies with positive results, 50% forstudies with questionable results, and 33% forstudies with negative results. 45In a study by Lee et al., formal publication of909 trials supporting 90 new drugs approved bythe FDA between 1998 and 2000 was verifiedby searches of PubMed and other databases. 42Statistical significance of the primary outcome wasdefined as being p < 0.05 or a CI excluding nodifference. For equivalency or non-inferiority trials,the statistically significant result refers to those with‘p > 0.05 or a CI including no difference or a CIexcluding the pre-specified difference described inthe trial’. 42 It was reported that the rate of formalpublication was higher for trials with significantresults than those with non-significant results (66%versus 36%).Similar to the above study, a more recentlypublished study included all efficacy trialssupporting new drug applications approved by theFDA from 2001 to 2002. 44 Favourable results werethose being significantly (p < 0.05) in favour of thenew drug or those confirming equivalence in noninferioritytrials. Trials with favourable results weremore likely to be published compared with trialswith not favourable or unknown results (82% versus64%). 44Cohorts of meetingabstractsIn the 2000 HTA report on publication bias, 2we identified eight reports that examinedthe association between study results and thesubsequent full publication of research initiallypresented as abstracts in meetings or journals. 47–54A Cochrane Methodology Review included 79studies of the subsequent full publication ofbiomedical research results initially presented asabstracts or in summary form. 55 Sixteen of the 79© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.studies reported data on the rate of publication bysignificance or importance of study results. Ourupdated search identified 22 additional cohortstudies of research abstracts that provided dataon publication bias (for details of all these studies,see Appendix 7). 56–77 Almost all of the 30 cohortstudies of conference abstracts were restricted to aspecific clinical field, such as emergency medicine,anaesthesiology, perinatal medicine, cystic fibrosisor oncology. The rate of full publication of meetingabstracts ranged from 37% to 81% for statisticallysignificant results, and from 22% to 70% for nonsignificantresults.Manuscript cohort studiesWe identified four studies of cohorts of manuscriptssubmitted to journals (Appendix 8). 78–81 Twostudies examined manuscripts submitted to generalmedical journals [Journal of the American MedicalAssociation (JAMA), British Medical Journal (BMJ),The Lancet and Annals of Internal Medicine] 78,81 andtwo used manuscripts submitted to the Journal ofBone and Joint Surgery (American Version). 79,80 Thestudy results of submitted papers were classifiedaccording to the significance of statistical tests(p < 0.05 or not) in the two studies of manuscriptssubmitted to general medical journals. 78,81 In thestudies of manuscripts submitted to the Journalof Bone and Joint Surgery, results were classified asbeing positive or negative or neutral, althoughthe definitions of these outcomes may be differentbetween the two studies (Appendix 8). 79,80 Resultsfrom these studies suggested that the acceptanceof submitted papers for publication by journals wasnot significantly associated with the direction orstrength of their findings.In the study of Olson et al., 81 133 acceptedmanuscripts were further examined and it wasfound that time to publication was not associatedwith statistical significance (median 7.8 monthsfor positive and 7.6 months for negative results,p = 0.44). 82 However, a subgroup analysis of 156manuscripts with a high level of evidence (level Ior II) in the study by Okike et al. found that theacceptance rate was significantly higher for studieswith positive or neutral results than for studies withnegative results (37%, 36% and 5% respectively;p = 0.02). 80These manuscript cohort studies are generally welldesigned and conducted. Although no conflict ofinterest was declared in these studies, this kind ofstudy will always need support or collaboration11

Evidence from cohort studies of publication bias12from editors of the journals. In prospective studies,editors’ decisions on the acceptance of manuscriptsmay be influenced by their awareness of theongoing study. 81Pooled analyses of cohortstudiesResults from different studies of publication biashave been quantitatively combined in previousreviews, 25,55,83 although it is still controversialbecause of heterogeneity across individual studies. 41Pooled estimates may improve statistical power andthe generalisability of results. In this review, theassociation between study results and the possibilityof subsequent publication was measured by usingodds ratios (OR). Heterogeneity across studieswithin each subgroup was measured using the I 2statistic. 16 A random-effects model was used inmeta-analyses.The formal publication of statistically significantresults (p < 0.05) could be compared with that ofnon-significant results in four inception cohortstudies, one regulatory cohort study, 12 abstractcohort studies and two manuscript cohort studies(Figure 2). The rate of publication of studies in thefour inception cohorts ranged from 60% to 93%for significant results and from 20% to 86% fornon-significant results. The rate of full publicationof meeting abstracts ranged from 37% to 81% forstatistically significant results, and from 22% to 70%for non-significant results. Heterogeneity across thefour cohort studies from the inception subgroupwas statistically significant (I 2 = 61%, p = 0.05).There was no statistically significant heterogeneityacross studies within the cohort studies of abstractsand cohort studies of manuscripts. The pooledodds ratio for publication bias by statisticalsignificance of results was 2.40 (95% CI: 1.18 to4.88) for the four inception cohort studies, 1.62(95% CI: 1.34 to 1.96) for the 12 abstract cohortstudies, and 1.15 (95% CI: 0.64 to 2.10) for the twomanuscript cohort studies (Figure 2).To include data from other cohort studies, apositive result was defined as being importantor confirmatory or significant, while a ‘nonpositive’result included negative, non-important,inconclusive or non-significant results. This moreinclusive definition of positive results allowedthe inclusion of all 12 inception cohort studies,four regulatory cohort studies, 29 abstract cohortstudies, and four manuscript cohort studies(Figure 3). There was statistically significantheterogeneity across cohort studies within theinception (p = 0.06), regulatory (p = 0.04) andabstract subgroups (p < 0.001). Pooled estimates ofodds ratios consistently indicated that studies withpositive results were more likely to be publishedthan studies with non-positive results, but this wasnot true after the submission to journals (Figure 3).Types of studies included in the cohort studiesvaried, and included basic experimental,observational and qualitative research, and clinicaltrials. When the analyses were restricted to clinicaltrials, the result was not significantly different fromthat based on all studies. Although the number ofcohort studies that could be included was small,clear evidence of publication bias can still beobserved when the analysis was restricted to clinicaltrials (Figure 4 and Figure 5).We constructed funnel plots separately for thefour subgroups of cohort studies (Figure 6). Theasymmetry of these funnel plots was tested usingthe method recommended by Peters et al. 9 (This is amethod of linear regression analysis, using log oddsratio as the dependent variable and the inverse ofthe total sample size as the independent variable.Please see Chapter 8 for more details about thismethod.) There was no statistically significantasymmetry for the funnel plots of inception cohortstudies (p = 0.178), regulatory cohort studies(p = 0.262), abstract cohort studies (p = 0.233) ormanuscript cohort studies (p = 0.942).Main results of the above meta-analyses of cohortstudies are summarised in Table 1.Factors associated withpublication biasSome cohort studies have examined the impactsof certain factors on the publication of research.The factors investigated included study design,type of study, sample size, funding source andinvestigators’ characteristics. Easterbrook etal. (1991) conducted subgroup analyses toexamine susceptibility to publication bias amongvarious subgroups of studies. They found thatobservational, laboratory-based experimentalstudies and non-randomised trials had greater riskof publication bias than RCTs. Factors associatedwith less bias included a concurrent comparisongroup, a high investigator rating of studyimportance and a sample size above 20. 22Dickersin et al. (1992) investigated the associationbetween the risk of publication bias and type ofstudy (observational, clinical trial), multi- or single

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8 τ =χ == = = = = = = τ =χ == = = = < τ =χ == = = = = Figure 2 Rate of publication of statistically significant versus non-significant results: all studies. After excluding Misakian and Bero(1998) 30 from inception cohort studies, pooled OR = 3.19 (2.21 to 4.61); heterogeneity: I 2 = 0%, p = 0.79.centre, sample size, funding source and principalinvestigator (PI) characteristics (such as gender,degree and rank). They found that none of thefactors examined was associated with publicationbias. 20Dickersin and Min (1993) reported that the OR forpublication bias was 0.84 (95% CI: 0.07 to 9.68)for multicentre studies compared to 21.14 (95%CI: 2.60 to 171.7) for single centre studies. 21 Inaddition, the risk of publication bias was differentbetween studies with a female PI (OR = 0.47;95% CI: 0.02 to 11.61) and studies with a malePI (OR = 20.70; 95% CI: 2.61 to 164.2). Oneinteresting explanation for the difference in studypublication between female and male PIs postedby Dickersin and Min was that ‘women have fewerstudies to manage’, related to their relatively lowerrank (35% of women PIs were professors comparedto 65% of male PIs), and are thus less selective instudy publication. They did not find an associationbetween publication bias and other study features13© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8 τ =χ == < = = < τ =χ == = = = = Figure 3 Rate of publication of positive versus non-positive results: all studies. After excluding Misakian and Bero (1998) 30 from inceptioncohort studies, pooled OR = 2.93 (2.31 to 3.71); heterogeneity: I 2 = 17%, p = 0.27 (continued).was examined in 16 of 79 abstract cohort studies. 55According to these 16 abstract cohort studies, thesubsequent full publication of conference abstractswas statistically significantly associated with positivestudy results (pooled OR = 1.28; 95% CI: 1.15 to1.42).Compared with the previous reviews on cohortstudies of publication bias, our review is moreinclusive in terms of types of studies and is thefirst to enable an explicit comparison of resultsfrom cohort studies of publication bias withfundamentally different sampling frames. Biased© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.selection for publication may affect researchdissemination over the whole process from beforestudy completion, to presentation of findings atconferences, manuscript submission to journals,and formal publication in journals. It seemsthat publication bias occurs mainly before thepresentation of findings at conferences and beforethe submission of manuscript to journals (see Figure2 and Figure 3). The subsequent publication ofconference abstracts was still biased but the extentof publication bias tended to be smaller comparedwith all studies conducted. After submission of15

Evidence from cohort studies of publication bias τ =χ == = = = < = = τ =χ == = = = < = = Figure 4 Rate of publication of statistically significant versus non-significant results: clinical trials only.16manuscript to journals, editorial decisions were notclearly associated with study results.Limitations of the availableevidence on publication biasThere are some caveats to the available evidenceon publication bias. Study findings have beendefined differently among the empirical studiesassessing publication bias. The most objectivemethod would be to classify quantitative results asstatistically significant (p < 0.05) or not. However,this was not always possible or appropriate. Whenother methods were used to classify study results asimportant or not, bias may be introduced due toinevitable subjectivity.The funnel plot asymmetry is not statisticallysignificant for inception, regulatory, abstractcohort studies and manuscript cohort studies (seeFigure 6). However, there are reasons to suspectthe existence of publication and reporting bias instudies of meeting abstracts. A large number ofreports of full publication of research abstracts wereassessed for inclusion into this review but did notmention the association between publication andstudy results and so were excluded. This associationmight not have been examined; or not reportedwhen the association was not significant. As anexample, Zaretsky and Imrie (2002) 77 reportedno significant difference (p = 0.53) in the rate ofsubsequent publication of 57 meeting abstractsbetween statistically significant and non-significantresults; but this study was not included in theanalysis as insufficient data were provided.Large cohort studies on publication bias usuallyincluded cases that were highly diverse in termsof research questions, designs and other studycharacteristics. Many factors (e.g. sample size,

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8 τ =χ == = = = < τ =χ == = = = = τ =χ == = = = < = = Figure 5 Rate of publication of positive versus non-positive results: clinical trials only.design, research question and investigators’characteristics) may be associated with both studyresults and the possibility of publication. Adjustedanalyses by some factors may be conducted but itwas generally impossible to exclude the impact ofconfounding factors on the observed associationbetween study results and formal publication.There is very limited and conflicting evidence onfactors (such as study design, sample size, etc.)that may be associated with publication bias. Toimprove the understanding of factors associatedwith publication bias, findings from qualitativeresearch on the process of research disseminationmay be helpful. 84,8517© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Evidence from cohort studies of publication bias108Inception cohortsRegulatory cohorts108661/SE41/SE4220–5 –4 –3 –2 –1 0 1 2 3 4 50–5 –4 –3 –2 –1 0 1 2 3 4 5In ORIn OR1086Abstract cohortsManuscript cohorts10861/SE41/SE4220–5 –4 –3 –2 –1 0 1 2 3 4 5In OR0–5 –4 –3 –2 –1 0 1 2 3 4 5In ORFigure 6 Funnel plots – publication of positive and non-positive results.Funnel plot asymmetry test using the method of Peters et al.: 9p = 0.178 for inception cohort studies; p = 0.262 for regulatory cohort studies; p = 0.233 for abstract cohort studies; and p = 0.942 formanuscript cohort studies.Table 1 Results of meta-analyses of cohort studies of publication biasCohort categoryNo. ofcohortstudies Pooled odds ratio (95% CI) Heterogeneity test: I 2 (p value)Statistically significant vs non-significant resultsInception cohorts 4 2.40 (1.18 to 4.88) 61% (0.05)Regulatory cohorts 1 11.06 (0.56 to 219.68)Abstract cohorts 12 1.62 (1.34 to 1.96) 22% (0.24)Manuscript cohorts 2 1.15 (0.64 to 2.10) 48% (0.17)Positive vs non-positive resultsInception cohorts 14 2.73 (2.06 to 3.62) 39% (0.06)Regulatory cohorts 4 5.00 (2.01 to 12.45) 64% (0.04)Title: 06-92-02 Abstract cohorts 29 1.62 (1.38 Proof to Stage: 1.93) 2Figure 62% (< Number: 0.001) 00.06.aiManuscript cohortsCactus Design and Illustration Ltd4 1.06 (0.80 to 1.39) 22% (0.28)18

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8There was statistically significant heterogeneitywithin subgroups of inception, regulatory andabstract cohort studies (see Table 1). The observedheterogeneity may be a result of differences instudy designs, research questions, how the cohortswere assembled, definitions of study results, andso on. For example, the statistically significantheterogeneity across inception cohort studies wasdue to one study by Misakian and Bero (see Figure2 and Figure 3). 30 After excluding this cohort study,there was no longer significant heterogeneityacross inception cohort studies. The cohort studyby Misakian and Bero included research on healtheffects of passive smoking, and the impact ofstatistical significance of results on publication maybe different from studies of other research topics. 30The four cohorts of trials submitted to regulatoryauthorities showed greater extent of publicationbias than other subgroups of cohort studies (seeFigure 3). 42–45 Only 855 primary studies wereincluded in the regulatory cohort studies, and twoof the four regulatory cohort studies focused ontrials of antidepressants. 43,45 Therefore, the fourregulatory cohort studies may be a biased selectionof all possible cases.ConclusionsDespite many caveats about the available empiricalevidence on publication bias, there is little doubtthat dissemination of research findings is likely tobe a biased process. There is consistent empiricalevidence that the publication of a study thatexhibits statistically significant or ‘important’results is more likely to occur than the publicationof a study that does not show such results. Indirectevidence indicates that publication bias occursmainly before the presentation of findings atconferences and the submission of manuscripts tojournals.19© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 4Evidence of different typesof dissemination biasThis chapter reviews available empiricalevidence on different types of researchdissemination bias, including outcome reportingbias, time lag publication bias, grey literature bias,language bias, citation bias, duplicate or multiplepublication bias, place of publication bias, databasebias, country bias and media attention bias.Outcome reporting biasOutcome reporting bias occurs when studieswith multiple outcomes report only some ofthe outcomes measured and the selection ofan outcome for reporting is associated with thestatistical significance or importance of the result.This bias is due to the incomplete reporting withinpublished studies, and is also called ‘within-studyreporting bias’ in order to distinguish it fromselective non-reporting of a whole study, 86 orpublication bias ‘in situ’. 87Number of outcomes measuredwithin trialsThe existence of a large number of measuredor calculated outcomes within a study is theprerequisite of selective reporting bias, whichis present in almost all research studies. Theselection of outcomes to report can be furtherclassified into three categories: 87 (1) the selection ofoutcomes investigated, (2) the selection of methodsto measure the selected outcome, and (3) theselection of results of multiple subgroup analyses.A large number of results can be generated by thecombination of all possible choices. Pocock et al.(1987) found that the median number of reportedend points was six per trial. 88 They also discussedselective reporting of results and related issues ofsubgroup analyses, repeated measurements overtime, multiple treatment groups, and multipletests of significance. 88 Tannock (1996) examined32 RCTs published in 1992 and found that themedian number of therapeutic end points per trialwas five (range 2–19) and 13 trials did not definetheir primary end point. 89 Each of the 32 trials,on average, reported six (range 1–31) statistical© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.comparisons of major outcome parameters;and more than half of the implied statisticalcomparisons had not been reported. 89The number of outcomes estimated from publishedarticles may underestimate the actual numberof outcomes measured in trial studies. Basedon information from trial protocols, Chan etal. 6,7 found that the median number of efficacyoutcomes was 20 per trial and the median numberof harm outcomes was six or five per trial.Although outcome reporting bias was highlysuspected, there was very limited empiricalresearch included in the 2000 HTA report. 90 Theupdated review has identified many recentlypublished empirical studies that provided directevidence with which to assess the existence andextent of outcome reporting bias (see Appendix 9for details of the included studies).Direct evidence on outcomereporting biasThe most direct evidence on the existence andextent of outcome reporting bias is from studiesthat compared outcomes specified in researchprotocols and those reported in subsequentarticles. A pilot study by Hahn et al. (2002) 36 wasthe first attempt to compare outcomes specifiedin trial protocols approved by a local REC andresults reported in subsequent publications. Theycompared outcomes in 15 pairs of protocols andjournal articles. Six of the 15 studies stated primaryoutcome variables in their protocols and four usedthe same outcomes as primary outcomes in thereports. An analysis plan was mentioned in eightstudies, but the plan was followed in only onepublished report.Chan and colleagues provided the mostdirect evidence on outcome reporting bias byinvestigating a cohort of 102 RCT protocolsapproved by the Danish REC from 1994 to 1995, 6and another cohort of 48 RCT protocols approvedby the Canadian Institutes of Health Research from1990 to 1998. 7 Data on unreported and reported21

Evidence of different types of dissemination bias22outcomes were collated based on trial protocols,subsequently published journal articles, and asurvey of trialists. If a published article providedinsufficient data for meta-analysis, the outcomewas defined as being incompletely reported. Theyfound that 50% of efficacy outcomes and 65% ofharm outcomes were incompletely reported in theDanish cohort; and 31% and 59% respectively inthe Canadian cohort. Primary outcomes specifiedin protocols were different from primary outcomesstated in the corresponding journal articles in 62%(Danish cohort) and 40% (Canadian cohort) ofcases. Statistically significant outcomes were morelikely to be fully reported than non-significantoutcomes. The odds ratio of an efficacy outcomebeing fully reported if it were statistically significantversus non-significant was 2.4 (95% CI: 1.4 to 4.0)in the Danish cohort and 2.7 (95% CI: 1.5 to 5.0)in the Canadian cohort. The biased reporting ofsignificant outcomes appears more severe for harmdata, the odds ratios were 4.7 (95% CI: 1.8 to 12.0)and 7.7 (95% CI: 0.5 to 111) respectively. 6,7Further work by Chan and Altman (2005) identified519 RCTs indexed in PubMed in December 2000,and they conducted a survey of trialists to obtaininformation on unreported outcomes. 5 The medianproportion of incompletely reported outcomesper trial was 42% for efficacy outcomes and 50%for harm outcomes. The pooled odds ratio foroutcome reporting bias was 2.0 (95% CI: 1.6 to 2.7)for efficacy outcomes and 1.9 (95% CI: 1.1 to 3.5)for harm outcomes. Reasons given by authors fornot reporting efficacy and harm outcomes includedspace constraints (47% and 25%), lack of clinicalimportance (37% and 75%), and being statisticallynon-significant (24% and 50%). 5Ghersi et al. (2006) compared 103 publishedRCTs and their protocols approved by CentralSydney Area Health Service REC from 1992 to1996. 91 They found that 17% of primary outcomesspecified in the protocols and 15% reportedin articles differed between the protocols andpublications. Trials for which all comparisons werestatistically significant were more likely to reportall outcomes fully (p = 0.06). As the study by Ghersiet al. was presented only as an abstract there was alack of information on its study design and otherresults. 91Other evidenceOne consequence of outcome reporting bias is thatmany trials cannot be included in meta-analysesbecause of incompletely reported outcomes inpublished papers. Although unreported data maybe available from trialists, such communication canbe time consuming and often does not result inadditional data becoming available.McCormack et al. 92 compared the results of a metaanalysisbased on published data with an updatedIPD (individual patient data, where the completeoriginal datasets of the included studies are used topool study data, rather than simply using summarymeasures from published reports) meta-analysis oftrials of hernia surgery. For the outcome of herniarecurrence, the number of contributing RCTswas similar and the results were not significantlydifferent between the two analyses. For the outcomeof persisting pain, IPD meta-analysis includedmany more RCTs (3 versus 20) and providedqualitatively divergent results, as compared withthe meta-analysis of published data. This case studyindicates that some outcomes (e.g. persisting pain)may be more vulnerable to selective reporting thanother outcomes (e.g. hernia recurrence). 92In a recent study, Bekkering et al. examined 767observational studies (with 3284 results) and foundthat only 61% of the reported results could beused in meta-analyses investigating dose–responseassociations between diet and prostate or bladdercancer. 93 Usable results were more likely to indicatethe existence of the association than those thatwere not usable. 93Furukawa et al. 94 investigated the associationbetween the proportion of contributing RCTs andthe pooled estimates of 156 Cochrane systematicreviews. A median of 46% [interquartile range(IQR) 20% to 75%] of identified RCTs in eachmeta-analysis contributed to the pooled estimates.The results of regression analysis revealed ageneral trend that the greater the proportion ofcontributing RCTs the smaller the treatment effect.It was concluded that outcome reporting may bebiased. 94A methodological study by Williamson andGamble 95 provided a motivating example and fourcases in which results of Cochrane reviews werecompared with results of sensitivity analysis (byimputation) when within-study outcome reportingbias was suspected. The example was a metaanalysisof beta-lactam versus a combination ofbeta-lactam and aminoglycoside in the treatmentof cancer patients with neutropenia, in which onlyfive of the nine eligible RCTs could be included.They found that the pooled treatment effect wasconsiderably decreased in sensitivity analysis where

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8the missing results were imputed. For the otherfour selected cases, within-study selection wassuspected in several trials but the impact on theconclusions of the meta-analyses was minimal. 95Scharf and Colevas compared adverseevents reported in 22 published articles andcorresponding protocols or data from ClinicalData Update System (CDUS, the National CancerInstitute’s electronic database of clinical trialinformation). 96 The study found considerablemismatch in high-grade adverse events betweenthe articles and the CDUS data, but it was notclear whether the mismatch was due to bias. It isimportant to note that published articles underreportedlow-grade adverse effects: only 58% oflow-grade adverse effects recorded in the CDUSdatabase were reported in articles. 96Selective reporting of multiplealternative analysesBias may be introduced by selective reporting ofmultiple results generated by different analyses fora given outcome. Melander et al. 43 compared 42trials of five selective serotonin reuptake inhibitor(SSRI) antidepressants submitted to the SwedishDrug Regulatory Authority for marketing approvaland published articles. The study considered onlyone outcome, response rate. Both intention-totreat(ITT) analysis and per protocol analysis werepresented in 41 of the 42 submitted reports, but inonly two stand-alone publications. The stand-alonepublications tended to report the more favourableresult by per protocol analysis. 43Compared with clinical trials, epidemiologicalstudies may be more susceptible to selectivereporting of results because of their exploratorynature. Kyzas et al. (2005) 97 conducted a metaanalysisto assess the association between aprognostic factor, the tumour suppressor protein53 (TP53), and mortality outcome of patientswith head and neck squamous cell cancer. Theycompared the results using (1) data from 18 studiesthat were indexed with ‘survival’ or ‘mortality’ inMEDLINE or EMBASE, (2) data from 13 publishedstudies that were not indexed with ‘survival’ or‘mortality’ in MEDLINE or EMBASE, and (3) dataretrieved from authors for 11 studies in which dataon mortality were collected but no usable data werereported. The pooled relative risk for mortality was1.27 (95% CI: 1.06 to 1.53) using 18 published andindexed studies, 1.13 (95% CI: 0.81 to 1.59) using13 published but not indexed studies, and 0.97(95% CI: 0.72 to 1.29) using retrieved data.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.According to the study by Kyzas et al., 97 TP53 statuscan be measured by different methods. Whenavailable, Kyzas et al. used immunohistochemistrydata, and a TP53-positive status was defined asnuclear staining in at least 10% of tumour cellsor at least moderate staining in qualitative scales.They also standardised all-cause mortality to24 months of follow-up. It was found that theassociation was stronger by using the definitionspreferred by each publication (RR 1.38; 1.13–1.67)than when definitions were standardised (RR 1.27;1.06–1.53).Kavvoura et al. investigated the discrepancybetween abstracts and full papers ofepidemiological studies using 389 abstracts and 50randomly selected full papers. 98 In the abstracts,88% reported one or more statistically significantrelative risks and only 43% reported one or morenon-significant relative risks. The prevalenceof significant results was less prominent in fulltexts of the articles. A median of nine (IQR 5–16)significant and six (IQR 3–16) non-significantrelative risks were presented in the full text of the50 articles. They also found that ‘investigatorsselectively present contrasts between more extremegroups when relative risks are inherently lower’. 98Summary of evidence onoutcome reporting biasThe most direct evidence is from the two studiesthat compared outcomes specified in trialprotocols and outcomes reported in subsequentpublications. 6,7 The results of unreported outcomeswere obtained by a survey of original investigators.Due to low response rates and insufficient data for2 × 2 tables, many included cases were excludedfrom the calculation of odds ratios. In the Danishcohort of 102 trials, 6 the odds ratio for reportingbias was based on 50 trials for efficacy outcomesand 18 trials for harm outcomes. Thirty trials forefficacy and only four trials for harm outcomeswere used in the Canadian cohort of 48 trials. 7This low response rate is likely to lead to anunderestimation of outcome reporting bias.Trials included in these two studies were mostlypublished before the CONSORT statementappeared in 2001, so it would be interesting toinvestigate whether outcome reporting bias hasbeen reduced in trials published since 2001.Findings from ongoing studies may provide furtherempirical evidence. For example, one study byGhersi et al. compared 103 published RCTs and23

Evidence of different types of dissemination bias24their protocols, and was presented in 2006 as aconference abstract. It is now available only inabstract form with insufficient study detail. 91 Wealso identified an ongoing MRC-funded study 99that aims to investigate the proportion and impactof within-study outcome reporting in an unselectedcohort of 300 Cochrane systematic reviews.Although case studies often yield evidence oflimited usefulness, they may provide evidenceindicating what further research is required. Forexample, the study by McCormack et al. of IPDmeta-analysis indicated that some subjectivelyassessed outcomes may be more vulnerable toreporting bias than objectively assessed outcomes. 92Time lag biasWhen the speed of publication depends on thedirection and strength of the study results, this isreferred to as time lag bias. 100 Empirical evidenceon time lag bias could be separated into twocategories: (1) the relationship between the studyresults and time to publication, and (2) changes inreported effect size over time.Time to publicationThe process of research is usually complex andinvolves several important milestones. Theseinclude development of the research proposal,approval by a research ethics committee, obtainingresearch funding, recruitment of participants,completion of follow-up, submission of manuscriptsto a journal, and final publication in peer-reviewedjournals. Measurement of elapsed ‘time topublication’ could be considered to start from anyof these milestones, for example, from the dateof REC approval, funding received, initiation orcompletion of enrolment, completion of follow-up,or manuscript submission.Four cohort studies were analysed in the 2000 HTAreport (see Appendix 10). Simes (1987) examinedthe time from trial closure to publication, using38 published or unpublished trials on advancedovarian cancer or multiple myeloma. 101 All sixtrials that showed a statistically significant survivaldifference were published within 5 years of studyclosure, while 5 of the 32 trials that showed nosignificant difference were published more than 5years after study closure, and 7 of the 32 trials withnon-significant results were not yet published. 101In a survey of 218 quantitative studies approvedby a hospital Ethics Committee in Australia, Sternand Simes observed that the median time fromgranting of ethical approval to the first publicationin a peer-reviewed journal was 4.8 years for studieswith significant results as compared with 8.0 yearsfor studies with null results (HR 2.32; 95% CI:1.47 to 3.66). 24 Adjustment for other factors thataffect publication (e.g. research design and fundingsource) did not change this result materially. Whenonly the large quantitative studies (sample size> 100) were analysed, the time lag bias remainedevident (HR 2.00; 95% CI: 1.09 to 3.66). 24 Studieswith non-significant trend (0.05 < p < 0.10) werepublished later compared with studies with nullresults (p > 0.10) (HR 0.43; 95% CI: 0.15 to 1.24).For qualitative studies (n = 103), there was no clearevidence of time lag bias involving studies withunimportant or negative results. 24Further empirical evidence on time lag bias camefrom a cohort of 66 completed phase 2 or phase3 trials, conducted between 1986 and 1996 by aclinical trials group on AIDS. 23 The results wereclassified as ‘positive’ if an experimental therapyfor AIDS was significantly (p < 0.05) better thanthe control therapy. ‘Negative results’ includedthose with no statistically significant difference andthose in favour of the control therapy. Definitionof publication was that the trial findings had to bepublished in a peer-reviewed journal. The mediantime from start of enrolment to publication was4.3 years for positive trials as compared to 6.4years for negative trials (p < 0.001). Positive trialswere submitted for publication more rapidly aftercompletion (median 1.0 year versus 1.6 years;p = 0.001) and were published more rapidly aftersubmission (median 0.8 years versus 1.1 years;p = 0.04), compared with negative trials. 23Misakian and Bero identified 61 completedstudies through a survey of 89 organisations thatsupported research on the health impact of passivesmoking. 30 Time to publication was assessedfrom the start date of funding because it wasdifficult to decide the time of study completion.‘Published studies’ were those that appearedin a peer-reviewed, non-peer-reviewed, or inpresspublication, but not if published only asabstracts. The median time from funding startto publication was 5 years (95% CI: 4 to 7) forstatistically non-significant studies, and 3 years(95% CI: 3 to 5) for statistically significant studies(p = 0.004). Multivariate analysis revealed that timeto publication was associated with the statisticalsignificance of the results (p = 0.004), experimentalstudy design (p = 0.01), study size (p = 0.01) andanimals as subjects (p = 0.03). 30

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8The updated review identified five new empiricalstudies of the interval before manuscriptsubmission to publication (Appendix 10), 27,37,102–104and one study of time from manuscript submissionto publication. 81Min and Dickersin found that statisticallysignificant or important results were associatedwith time from completion of enrolment tofull publication (HR = 1.75; 95% CI: 1.14 to2.93), according to a study of 242 observationalstudies initiated at Johns Hopkins University. 104However, Cronin and Sheldon examined a cohortof 70 studies sponsored by the UK NHS R&Dprogramme and did not observe a significantdifference in time from study completion topublication by whether a study result was significant(p < 0.05 or important) or not (HR = 0.53; 95%CI: 0.25 to 1.1). 27 Similarly, study results werenot found to be significantly associated with timeto publication, according to findings from theremaining three empirical studies (Appendix10); however, in all of the studies where data wereprovided the trend suggested a shorter time forstatistically significant or ‘positive’ studies than fornon-significant or negative ones, and the cohorts ofstudies were often small. 37,102,103Dickersin et al. tracked 133 manuscripts ofcomparative studies accepted for publication by theJournal of the American Medical Association (JAMA)and investigated time from manuscript submissionto publication. 82 Results were classified as positiveif a statistically significant (p < 0.05) difference wasreported for the primary outcomes. Seventy-eight(59%) manuscripts reported positive results, 51(38%) reported negative results and the results offour (3%) articles were unclear. The median timeinterval between submission and publication was7.8 months for positive studies versus 7.6 monthsfor negative studies (p = 0.44). 82 Findings of thisstudy indicated that time lag bias (if any) may likelyoccur before, not after, the manuscript submissionto journals.We also identified six studies that investigatedthe time from abstract presentation at meetingsto subsequent full publication (Appendix10). 49,60,61,69,70,105 The study by Krzyzanowska etal. 70 included 510 abstracts of large (n > 200)phase 3 trials presented at an oncology meetingbetween 1989 and 1998. They found that trialswith statistically significant results were publishedearlier than those with non-significant results(median time to publication 2.2 versus 3.0 years;HR = 1.4; 95% CI: 1.1 to 1.7). 70 Findings from© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.two other studies 60,69 also suggested that timefrom abstract presentation to full publication wasassociated with significant results. However, theobserved association between study results andtime from abstract presentation to publication wasnot statistically significant in the remaining threestudies (two of which were small in terms of thenumber of abstracts assessed). 49,61,105Change in reported effect sizeover timeThe 2000 HTA report on publication bias 2discussed only two brief reports on the temporaltrend of reported effect size. 106,107 Rothwell andRobertson found that the treatment effect wasoverestimated by early trials as compared withsubsequent trials in 20 of the 26 meta-analyses ofclinical trials. 106 In another report, a significantcorrelation (p < 0.10) between the year ofpublication and the treatment effect was observedin 4 of the 30 meta-analyses published in BMJ orJAMA during 1992–6. 107The updated review included two new case studiesof pharmaceutical interventions 108,109 and twostudies in research on ecology or evolution 110,111(Appendix 10). Gehr et al. found that reportedeffect size significantly decreased over time inthree of the four meta-analyses of studies onpharmaceutical interventions. 108 In a case studyof N-acetylcysteine in the prevention of contrastinducednephropathy, Vaitkus and Brar found thattrials published earlier reported more favourableresults than trials published later. 109In a study of 44 meta-analyses in ecology, Jennionsand Moller found a significant relationship betweenyear of publication and estimated effect size, andthe association remained significant even aftercontrolling for sample size (p < 0.01). 110 However,in another case study in the area of ecology, Leimuand Koricheva did not find a significant associationbetween the effect size and year of publication intwo meta-analyses of studies testing plant defencetheories. 111Ioannidis and Trikalinos hypothesised that highlycontradictory results are more likely to be rapidlypublished than other results. 112 They investigatedchanges in between-study variance over time in44 meta-analyses of epidemiological studies ongenetic associations and in 37 meta-analyses ofclinical trials of health-care interventions. It wasfound that early published studies tended to bemore heterogeneous than later published studies25

Evidence of different types of dissemination bias26in meta-analyses of genetic associations. There wasno significant change in the between-study varianceover time in meta-analyses of clinical trials. 112Summary of evidence on timelag biasEmpirical evidence on time lag bias came mainlyfrom studies that investigated time to publicationusing cohorts of studies. Because of the strictinclusion criteria, the Cochrane MethodologyReview on time lag bias 113 included only twocohort studies. 23,24 We have adopted a morecomprehensive approach and included morerelevant studies (Appendix 10).Four of the five newly identified cohort studies ontime lag bias (before submission) did not find asignificant association between time to publicationand study results. 27,37,101,103 These four studies wererelatively small, and the sources of the samplewere diverse. It is unclear whether poor qualitystudies may tend to overestimate or underestimatethe association between study results and time topublication. Of the six newly included studies oftime from abstract presentation at meetings to fullpublication, three large studies reported significanttime lag bias 60,69,70 while the other three studies(one large, two small) did not. 49,61,105Considering all the available evidence from cohortstudies on time to publication, we conclude thaton average studies with significant or importantresults still tend to be published earlier thanstudies with non-significant results. However, thisconclusion may not be generalisable to manyindividual cases. Studies included in the identifiedcohort studies were usually divergent in termsof research questions and design, so that theobserved association between study results andtime to publication may be influenced by otherconfounding factors. 109 Limited evidence suggeststhat study findings that were difficult to interpret,for example, when the p value was greater than0.05 but smaller than 0.10 or when results weremixed negative or positive, may take even longer tobe published than studies that had clear negativeresults.If it does exist, time lag bias is likely to occur beforemanuscript submission for journal publication. 81One consequence of time lag bias (earlierpublication of significant results) may be adiminishing effect size reported by studies overtime. Therefore, temporal trends of reported effectsize in meta-analysis may indicate the existence oftime lag bias (the ‘fading of reported effectiveness’coined by Gehr et al.). 108 Compared with studiesincluded in cohort studies of time to publication,studies in meta-analyses for the investigation oftemporal trends of reported effect size were muchmore similar in term of participants, interventionsand outcomes. However, time lag bias is onlyone of several possible explanations for changesin reported treatment effect over time. 108 It issurprising that only very limited research hasbeen conducted to investigate temporal trends ofreported effect size in meta-analysis.Grey literature biasThe distinction between grey literature andunpublished or published studies may sometimesbe ambiguous. Studies presented in the form ofgrey literature may be considered as published oras unpublished, according to different definitions. 2The Third International Conference on GreyLiterature defined grey literature as ‘that whichis produced on all levels of governmental,academic, business and industry in print andelectronic formats, but which is not controlled bycommercial publishers’. 114 Grey literature consistsof an immense range, which includes brochures,pamphlets, internal reports, memoranda, databaseson ongoing research, newsletters, conferenceproceedings and abstracts, technical reports,assignments and dissertations 115 as well as personalcorrespondence, web pages, data archives, policydocuments and book chapters.A survey published in 1993 found that 31% ofpublished meta-analyses included unpublisheddata. 116 The proportion of people who supportedthe inclusion of unpublished data in metaanalysisat that time was 78% for meta-analysts ormethodologists, while it was only 47% for journaleditors. 116 Taus et al. reported in 1999 that 11%of the 814 references of included studies in 75neurological reviews from the Cochrane Librarywere from books, theses or other unpublishedsources. 117 Tetzlaff et al. presented a survey in 2006that found that while both editors and reviewmethodologists had become more in favour ofincluding grey literature, editors were still lessinclined towards the inclusion of grey literature(69%) than systematic reviewers or methodologists(85%). 118Empirical evidence on grey literature bias may beseparated into two categories: (1) the subsequent

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8full publication of a cohort of grey literature (suchas meeting abstracts) according to study results,and (2) a comparison of results of fully publishedstudies and grey literature studies that aimedto answer the same research question. Evidencefrom the first category has been summarised inthe cohort studies section. This section focuseson empirical studies that compared results frompublished and corresponding grey literature.Unpublished and grey literature studies were notseparately considered, since it is usually impossibleto distinguish the two.Studies of multiple metaanalysesThe previous HTA report 2 included one studyusing multiple meta-analyses. 119 The updatedreview identified five new studies of multiple metaanalysesin which the results of published studieswere compared with those estimated by using greyliterature (see Appendix 11 for six included studiesof at least 10 meta-analyses, and 16 individual casestudies). 3,45,120–122McAuley et al. (2000) 119 investigated the impactof exclusion of grey literature from meta-analyseson the estimate of intervention effectiveness.In a sample of 135 meta-analyses, of which 41meta-analyses (30%) included some form of greyliterature (between 4.5% and 75% of includedstudies) the removal of grey literature resulted inan increase in the estimate of treatment effect ofat least 10% in nine meta-analyses and a reductionof treatment effect of at least 10% in five. Onaverage, published literature yielded significantlylarger estimates of treatment effect, by 15%compared with grey literature [ratio of odds ratio(ROR) = 1.15; 95% CI: 1.04 to 1.28]. This studyconcluded that the exclusion of grey literature canlead to overestimation of treatment effects. 119The empirical study by Egger et al. (2003) included60 meta-analyses in which results of publishedstudies (n = 630) could be compared with those ofgrey literature trials (n = 153). 3 Estimated treatmenteffect based on grey literature ranged from 97%more to 209% less beneficial than those basedon corresponding published trials. Pooled effectestimates from the grey literature were on average7% greater than those from published trials (ROR1.07; 95% CI: 0.98 to 1.15). 3 However, publishedtrials tended to have a larger sample size and be ofbetter quality than unpublished trials. Therefore,there is a possibility that bias could be introducedby including poor quality grey literature. 3© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.A recent study by Turner et al. (2008) comparedpublished and unpublished clinical trials of12 antidepressant agents. 45 From the FDAdatabase they identified 74 clinical trials of 12antidepressants approved by the FDA between1987 and 2004. Of the 74 trials, 23 wereunpublished. It was found that the standardisedeffect size (Hedges’ g) using data from the journalarticles (0.41; 95% CI: 0.36 to 0.45) was on average32% (ranged from 11% to 69%) greater thanthe effect size using data from the FDA reviews(0.31; 95% CI: 0.27 to 0.35) (sign test p < 0.001).In addition, negative or questionable findings(according to the FDA’s decision) were less likely tobe published, and if published, the result was oftenconveyed as a positive outcome. 45The remaining three studies of multiple metaanalysesin which the impact of inclusion of greyliterature could be investigated 120–122 consistentlyfound that published studies on average yieldeda greater estimate of treatment effect comparedwith grey literature studies, although the differencewas not statistically significant individually and thedirection of bias was unpredictable for individualreviews.Five of the six studies of multiple meta-analyseswere included in the Cochrane methodologyreview on grey literature bias. 123 Using data fromthree studies 3,119,122 suitable for combining in metaanalysis,Hopewell et al. estimated that publishedtrials on average suggested a 9% greater treatmenteffect than grey literature trials (pooled ROR forgrey literature versus published trials = 1.09; 95%CI: 1.03 to 1.16). 123Case studies of grey literaturebiasThe previous HTA report 2 reviewed several casestudies that investigated the impact of inclusionof grey literature. In the field of psychological andeducational research, several case studies reporteda tendency for the average effects reported injournal articles to be greater than the effectsreported in the corresponding dissertations. 124–126In the medical and health field, the previous HTAreport included four case studies. 127–130 However,the validity of empirical evidence from casestudies may be questionable because of selectivereporting. 123According to findings from the 16 case studiesincluded in this review (see Appendix 11 fordetails of the individual studies), published studies27

Evidence of different types of dissemination biastended to report a greater estimate of effect sizesthan grey literature, although the difference wasstatistically significant in only some studies. 109,131–134For example, in an IPD meta-analysis of paternalcell immunisation for recurrent miscarriage, Jenget al. (1995) found that the estimated relative riskwas 1.29 (95% CI: 1.03 to 1.60) using data fromfour published trials, and it was 1.01 (95% CI:0.74 to 1.28) by using data from four unpublishedstudies. 131 A meta-analysis of animal experimentalstudies of nicotinamide for stroke found thatabstracts reported a statistically significantly lowerestimate of effect size than fully published studies(p < 0.001). 132One case study by MacLean et al. was at first (in1999) presented as a meeting abstract 130 and thenfully published in 2003. 135 The study compareddata from published studies with data from FDANew Drug Application Reviews for assessingnon-steroidal anti-inflammatory drug (NSAID)-associated dyspepsia. The quality of unpublisheddata from FDA reviews was comparable with thatof published data. The pooled relative risk forNSAID-induced dyspepsia was 1.07 (95% CI: 0.70to 1.63) using the FDA data, and 1.21 (95% CI:0.81 to 1.81) using data from published trials.Meta-regression analyses found that estimatesvaried significantly by NSAID dose (p = 0.037)but were not related to whether the study waspublished or not (p = 0.73). 135 The reporteddifference between the published and greyliterature appeared greater in the abstract 130 thanthat in the full publication. 135It is interesting to compare two case studies onthe same topic. 136,137 Whittington et al. (2004)compared the results of published and unpublisheddata from clinical trials of SSRIs in childhooddepression. 136 They found that the results ofpublished trials indicated a favourable risk-benefitprofile for some SSRIs, while unpublished datatended to be unfavourable. They concluded that‘non-publication of trials, for whatever reason, orthe omission of important data from publishedtrials, can lead to erroneous recommendations fortreatment’. 136 In another case study by Wallace etal. (2006), a cumulative meta-analytic approachwas used to synthesise evidence from trials ofSSRIs in paediatric depression. 137 Although theunpublished data tended to suggest that the SSRIswere less efficacious and more harmful, the overallinterpretation of evidence on efficacy and safetywould not change on inclusion of unpublishedtrials. 137Batt et al. (2004) compared the quality, quantityand nature of grey and published evidence oncosts and cost-effectiveness of strategies to increasecoverage of routine immunisations in low andmiddle income countries. 138 Of 34 included studieson effectiveness from the grey literature, 63%metthe quality criteria set for inclusion, while 57% ofpublished literature met these criteria, suggestingthat in this area grey literature is of higher quality.Inclusion of grey literature almost doubled thenumber of included studies, covered differentgeographical areas, covered operational researchand finance (rather than the economics andpolicy-making covered in published literature) andwas more up to date. There were no statisticallysignificant differences between published and greyliterature in terms of effectiveness (final coverageor changes in coverage). 138Summary of evidence on greyliterature biasThere is good evidence that published literaturetends to be more positive about the effectiveness ofinterventions than corresponding grey literature,although this can vary in individual reviews. Thequality of grey literature studies can be higher,lower or the same as the corresponding publishedstudies.Studies of cohorts of meeting abstracts found that alarge number of abstracts presented at conferencemeetings will not be published in full, and thesubsequent publication of abstracts is associatedwith study results (see Cohorts of meeting abstractsand Pooled analyses of cohort studies). Suchstudies included abstracts of studies on diverseresearch questions, and it is difficult to excludethe influence of many confounding factors on theassociation between study results and subsequentfull publication. Therefore, findings from suchcohort studies provided only indirect evidence ongrey literature bias.More direct evidence on grey literature biascame from studies that compared the result ofpublished studies and grey literature within ameta-analysis. Many case studies were identified,but the interpretation of findings from these casestudies was complicated due to concern overpossible selective reporting. Therefore, empiricalstudies that used an unbiased sample of multiplemeta-analyses provided the most valid evidence ongrey literature bias. The updated review identifiedseveral recent studies of multiple meta-analyses28

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8in which the results of published studies could becompared with that of grey literature.Available evidence from good quality studiessuggested that published studies tend to reporta greater treatment effect compared with amore complete set of data from published andunpublished studies, but that for individual reviewsthe effects may not always be in this direction. Greyliterature studies may be relatively small and ofrelatively poor quality, although again this is notalways the case. The impact of grey literature inmeta-analysis is usually small, although occasionallydata from grey literature may have importantclinical implications. A case-by-case approach isrequired to decide whether grey literature shouldbe comprehensively searched and included insystematic reviews. The inclusion of grey literaturemay sometimes introduce bias, as will exclusion ofgrey literature in other cases. 3,139The most commonly included unpublished dataused in reviews are conference abstracts, butthere are difficulties in using data from abstractsas they provide limited information, may be onpartial datasets and may be misleading whencompared with later full publications. Evidenceon the importance and utility of other types ofunpublished material is less clear.Language biasMany prestigious international scientific journalsare published in English, and journals publishedin English are more likely to have greater journalimpact factors (JIF). 140 However, writing forjournals published in English can be more difficultfor researchers who are non-native Englishspeakers. 141–143Quality of studies publishedin English and non-EnglishlanguagesWhen fictitious manuscripts with identicalmethodological flaws were sent to referees, Nylennaet al. (1994) found that Scandinavian refereesawarded higher quality scores to English-languagemanuscripts than to the manuscripts in a referee’sown national language. 144Moher et al. (1996) compared completeness ofreporting of 133 trials published in English and96 trials published in French, German, Italian orSpanish. 145 They found no significant difference© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.between English and non-English trials in thecompleteness of reporting or overall quality score(51.0% versus 46.2%). It was therefore concludedthat all trial reports should be included insystematic reviews irrespective of the language inwhich they are published. 145Junker (1998) 146 identified deficiencies in thequality of reporting of 32 German and 89 Englishlanguagereports of placebo-controlled trialspublished by the same group of authors. The meanquality score was 8.4 (on a scale of 0 to 18), witha non-significant difference in the mean qualityscore between English and German-languagereports (0.27; 95% CI: – 0.97 to 1.52). However,Junker’s assessment is somewhat limited becausethe investigators looked only at published papersinvolving German-speaking authors from a singleresearch group. 146More recently, Moher et al. (2003) found thatthere were only minor differences in the quality ofreports between RCTs published in English andin non-English languages in a study of 42 metaanalyses.4 However, Egger et al. (2003) observedthat on average, 115 non-English-language trialstended to include fewer participants, were morelikely to show statistically significant results, andwere of lower methodological quality, than 485other trials published in English. 3Therefore, studies published in languages otherthan English cannot be generally excluded for thereason of study quality.The previous HTA report on publication bias 2included a study of multiple meta-analyses 147 anda study that compared 40 pairs of RCTs publishedin German and in English. 148 Since then, two majorHTA-supported studies have been completed andthese provide more evidence on the differences inestimated treatment effects between English andnon-English language trials in meta-analysis (seeAppendix 12 for details of the six studies includedin this section). 3,4Comparison of studies publishedin different languagesEgger et al. (1997) 148 identified 40 pairs of RCTs,each pair comprising an RCT published inGerman, and a matched RCT by the same authorpublished in English during the same period. Theinvestigators found that design characteristicsand quality features were similar between RCTspublished in German, and RCTs conducted in29

Evidence of different types of dissemination bias30German-speaking Europe that were published inEnglish. Statistically significant results (p < 0.05)were reported in 35% of German language articlesand 62% of English language articles (OR 3.75;95% CI: 1.25 to 11.3). Logistic regression analysisfound that a statistically significant finding wasthe only variable that was associated with a trial’spublication in English-language journals. It wastherefore concluded that ‘authors are more likely topublish RCTs in an English language journal if theresults were statistically significant’. 149A similar study by Heres et al. (2004) reportedsimilar findings in a comparison of 21 pairs oftrials in the field of neuroscience matched by thekey authors. 150 In this instance, significant resultswere reported in 33% of German-language articlesas compared with 57% of the English-languagearticles (Wilcoxon’s test p = 0.14). 150Studies of multiple metaanalysesDirect evidence on the impact of language biascomes from evaluations of multiple meta-analyseswhere the results of studies on the same researchquestion but published in different languages couldbe compared (Appendix 12).Gregoire et al. (1995) studied meta-analysespublished in eight medical journals betweenJanuary 1991 and April 1993. 147 They foundthat 28 of the 36 meta-analyses had languagerestrictions. By repeating the same searcheswithout language restrictions in these 28 metaanalyses,they identified 19 individual studies thathad not been included for language reasons. Theinclusion of eight of these 19 studies to the fivecorresponding meta-analyses did not change thefindings. However, inclusion of the other 11 studiesto the remaining seven corresponding metaanalyseshad the potential to modify the results.The most important difference was the changein the 95% CI of the overall OR estimated in ameta-analysis of selective decontamination of thedigestive tract in intensive care units. The pooledOR was 0.70 (95% CI: 0.45 to 1.09) in the originalmeta-analysis, and this became 0.67 (95% CI: 0.47to 0.95) after including a study published in a Swissjournal. 147Moher et al. (2000) examined a set of 19 metaanalysesto investigate whether different estimatesof treatment effect were obtained in meta-analysesrestricted to English-language studies comparedwith those without this restriction. Languagerestrictedmeta-analyses, compared with metaanalysesinvolving non-English language studies,did not differ with respect to the overall estimate ofeffectiveness (ROR 0.96; 95% CI: 0.78 to 1.18). 151Meta-analyses without language restrictions hadnarrower confidence intervals (average width 0.79;95% CI: 0.51 to 1.07) compared with languagerestrictedmeta-analyses (average width 0.92; 95%CI: 0.53 to 1.32), which represents a statisticallysignificant relative difference in precision of 16%.These findings were limited by small sample size,small sampling frame, limited clinical topics andlimited interventions. The meta-analyses that didinclude non-English-language trials had a very lownumber of such studies. Moreover, the majority(13/19) of the meta-analyses included only one trialpublished in languages other than English. 151A further study using 42 meta-analyses (including529 English- and 133 non-English-languagetrials) was conducted by Moher and his colleagues(2003). 4,152 The 42 meta-analyses included 34 metaanalysesof conventional interventions, and eightmeta-analyses of complementary and alternativemedicine. The exclusion of trials in languagesother than English, compared with their inclusion,did not yield a significantly different estimate oftreatment effect overall (ROR 1.11; 95% CI: 0.92to 1.34), or when the meta-analyses looked onlyat conventional interventions (ROR 1.02; 95%CI: 0.83 to 1.26). However, in meta-analyses ofcomplementary medicine, exclusion of non-Englishtrials resulted in a 63% smaller protective effect(ROR 1.63; 95% CI: 1.03 to 2.60). The authorsconcluded that language bias is unlikely to be aproblem for many meta-analyses in the field ofconventional medicine, but it may substantiallyalter the results of meta-analyses of complementarymedicine. 4Egger et al. (2003) provided further empiricalevidence by independently examining theinfluence of non-English-language trials in asample of meta-analyses. 3,153 They identified 50meta-analyses that included a total of 485 Englishlanguagetrials and 115 non-English-languagetrials. Within these meta-analyses, treatment effectestimates were on average 16% more beneficial innon-English-language trials (ROR 0.84; 95% CI:0.74 to 0.97) but with considerable heterogeneity.Excluding non-English-language studies led to avariety of changes, from a reduction in benefit of42% to an increase of 23%. The exclusion of non-English-language studies resulted in greater benefitof the intervention in five meta-analyses, reductionin benefit in 16 meta-analyses, with little or noeffect (

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8from 8.34 to 7.68 after exclusion of non-Englishlanguage trials. 3Summary of evidence onlanguage biasThe impact of excluding non-English-languagestudies in systematic reviews appears to be highlyheterogeneous. Different types of non-Englishlanguagestudies (involving different areas of healthcare, and from different countries) may either bemore or less likely to show statistically significanteffects than comparable English-language studies,and they may be of lower or similar methodologicalquality. However, a common finding was thatexclusion of non-English-language studies reducedthe precision of the estimate of effect.While there are specific areas where omittingnon-English-language studies appears to resultin a very high risk of bias (studies in the area ofcomplementary medicine, for example) theirexclusion in other areas may, or may not, result inbias. If exclusion does result in bias it is impossibleto assess beforehand which direction this bias maytake, as it may inflate or deflate the apparent effectsize. This will be difficult to assess unless non-English-language studies are first included andlater excluded. The best way to ensure that a reviewdoes not contain language bias is to search for andinclude relevant non-English language studies.The cost-effectiveness of this strategy (given theadditional searching and translation time andcosts) is unclear.Citation biasIn published articles, references to other studiesare cited for various reasons, for example, to showthe importance of a research question, to borrowmethods and techniques, or to give positive creditto the material referenced. 154 The chance of a studybeing cited by others may be associated with manyfactors like the journal impact factor, nationality ofauthors, working partnerships, etc. Citation biasoccurs when the probability that a study will becited is associated with the study result.The previous HTA report 2 in 2000 included severalstudies that provided empirical evidence oncitation bias. 155–160 Five recently published empiricalstudies on citation bias were identified in thisupdated review (see Appendix 13 for the includedstudies). 161–165 The previously cited studies will bediscussed first, followed by the newer studies.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Shadish et al. (1995) randomly selected one citationfrom each of 283 articles published in threepsychological journals and asked each author aboutthe most important reason for citing the selectedreferences. 155 It was found that citation was mostcommonly used to support the author’s argument,while study quality was not considered in mostcases. 155In one study examining the judgement anddecision literature, it was found that poorperformanceresults were significantly morelikely to be cited than positive good-performanceresults. 156 This could not be explained by thejournal’s popularity or the year of publication.This did suggest citation bias but the resultswere questionable since the poor-performanceand good-performance articles were publishedin different journals and reported differentevidence. 166Gotzsche (1987) examined the existence of citationbias by using 111 comparative trials on nonsteroidalanti-inflammatory drugs. 157 The trialresult was defined as positive if the benefit:harmratio was in favour of the experimental drug. Thepattern of citation was then classified as positive,neutral or negative selection of references bycomparing the proportion of references reportingpositive and negative results. For example,selection was classified as positive when theproportion of trials with a positive outcome in thereference list was higher than that in all availabletrials. Among the 76 trials in which citation biaswas probable, the selection of references wasclassified as neutral in 10, negative in 22, andpositive in 44. In conclusion, positive selection ofreferences is more likely to happen than neutraland negative selection, suggesting citation bias. 157Ravnskov (1995) examined citations in threeauthoritative reviews on diet–heart issues andfound that only one of six relevant RCTs with anegative outcome was cited and by only one ofthe three reviews. However, two, four and six nonrandomisedtrials with a positive outcome werecited in each review respectively, suggesting that‘fundamental parts of the diet-heart idea are basedon biased quotations’. 158Hutchison et al. (1995) assessed citation bias bycomparing the proportion of relevant supportiveand non-supportive trials used in 17 reviews onthe clinical effectiveness of pneumococcal vaccine.Supportive trials were defined as those thatreported significantly fewer failures in vaccinated31

Evidence of different types of dissemination bias32subjects than among the controls. It was found thatunsupportive trials were more likely to be citedthan supportive trials (11.9% versus 5.8%). Thetendency to cite recent trials may be one reasonfor this disproportionate citation of unsupportivestudies because six of the seven trials publishedafter 1980 were unsupportive and all seven trialspublished before 1980 were supportive. 159In an assessment by Song et al. (1997) of publishednarrative reviews on the prophylactic removal ofimpacted third molars, it was found that reviewswith similar aims included very different evidenceon which to draw conclusions. 160 Of 69 studies thatwere discussed in nine general reviews about theassociation between pathology and impacted thirdmolars, one was quoted in five reviews while 43were cited only once. This discrepancy in the useof relevant studies cannot be reasonably explainedby the year of publication or quality criteria.This selective citation of studies correspondedwith conflicting conclusions from these narrativereviews. 160Five more recent studies have been added tothis update. Chapman et al. (2009) examinedassociation between citation frequency andreported prevalence in studies of smokingamong schizophrenia patients. 162 They foundthat a 10% increase in reported prevalence ofsmoking was associated with a 61% (95% CI:30% to 98%) increase in citation rate. 162 Anotherstudy by Callaham et al. (2002) evaluated how204 emergency medicine studies presented toa meeting in 1991 were cited, and the factorsassociated with citation. 161 Predictors for citationfrequency were the impact factor of the journal, thepresence of a control group, newsworthiness scoreand sample size, while biased citation of positiveoutcomes was not observed. 161Kjaergard and Gluud (2002) reviewed 530 hepatobiliarydisease trials to assess whether trials withstatistically significant outcomes were cited moreoften than those with non-significant results. 163They found a significant positive associationbetween a statistically significant study outcomeand citation frequency. The citation frequency wasalso associated with disease area and adequategeneration of allocation sequence. 163In another study of 368 research papers publishedin four psychiatric journals, Nieminen et al. (2007)found that citation rate was related to p-value. 164Median number of citations for papers reporting‘significant’ and ‘non-significant’ results was 33versus 16 respectively. Compared with studies withnon-significant results, the ratio of citation rate forstudies with significant results was 1.63 (95% CI:1.32 to 2.02). 164Schmidt and Gotzsche (2005) investigatedreference bias in 42 narrative reviews of physicalinterventions on house dust mite antigens. 165Reference selection in each review was classifiedas positive, neutral or negative according towhether the proportion of trials with a statisticallysignificant outcome in the review was higherthan that among all trials available. For example,positive selection of references meant that theproportion of studies with positive results citedin a review was higher than the proportion ofpositive trials in all relevant trials available. Of the38 reviews in which physical interventions wererecommended, 10 reviews were neutral in termsof reference selection, 27 reviews had a positiveselection of references and one a negative selection.The four reviews that did not recommend physicalinterventions all had a negative selection ofreferences. 165Summary of evidence oncitation biasEmpirical evidence indicates that studies withpositive or significant results are on averageassociated with a higher frequency of citation,although this may not always be the case in specificareas of the literature. Non-systematic narrativereviews are a specific area where biased citationof research findings can result in misleadingconclusions.Duplicate (multiple)publicationDuplicate, redundant, repetitive or multiplepublications are defined as submission of similarmanuscripts to more than one journal or therepublication of the same data in two or morejournals. 167 It has been estimated that 10–25% ofthe published literature in biomedical sciencesrepresents duplicate or redundant publications. 168The publications may overlap partially orcompletely, representing a similar portion or majorcomponent of a study, and may share the samehypotheses, methods, results and/or discussion.Multiple publications of the same data in differentjournals has been condemned mainly for wastingjournal space and editors’ and referees’ time, aswell as readers’ time. 167,169–171 However, publicationof the same data in different ways may help to

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8disseminate important research results, providingany previous or parallel publications have beenexplicitly referenced. However, researchers andjournal editors may have different understandingabout duplicate publication and it is sometimesdifficult to distinguish the unacceptable redundantpublication from the acceptable ‘parallel’publication. 172 Recently, a database of duplicatepublication and potential plagiarism (Déjà vu) hasbeen developed by using a text similarity algorithmto identify extremely similar references fromMEDLINE. 173Duplicate publication can be classified as ‘overt’or ‘covert’. 174 Overt duplicate publication isdefined as reanalysis of data from a study withappropriate cross-referencing of original reports.Covert duplicate publication is when the same dataare published in different places or at differenttimes without adequate reference to a previous orparallel publication. Bias may be introduced insystematic reviews by including data from the samestudy more than once because of covert duplicatepublication.Empirical evidence on duplicatepublication biasThe previous HTA report on publication biasincluded several case studies that providedempirical evidence on duplicate publicationbias. 22,174–176 No new published studies and only oneconference abstract were identified in this updatedreview.Gotzsche (1989) examined 44 multiple publicationsof 31 controlled trials of NSAIDs in rheumatoidarthritis and found important reported differencesin design, exclusion of protocol violators, numberof effect variables, number of side effects, and thesignificance levels between duplicated publicationsof the same studies. 175 The conclusion becamemore positive for the new drugs in the latepublications of three trials. He also suggestedthat multiple publications were difficult to detectbecause the first author and the number of authorscited often differ. 175Tramer et al. (1997) assessed the impact in a metaanalysisof duplicate data on efficacy estimatesof ondansetron on postoperative emesis. It wasfound that, for three trials that were publishedin six reports, there was no cross-referencing. 174The estimated number-needed-to-treat (NNT) toprevent one vomit within 24 hours was 9.5 (95%CI: 6.9 to 15) in 16 non-duplicated reports and© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.3.9 (95% CI: 3.3 to 4.8) in the three reports thatwere duplicated. The efficacy was overestimatedby including duplicated data (NNT = 4.9; 95%CI: 4.4 to 5.6) compared with the report withoutduplicated data (NNT = 6.4; 95% CI: 5.3 to7.9). Tramer et al. also discussed difficulties inidentifying duplicated publications of the sametrial data. For example, the same trial might reporta different number of patients or different patientcharacteristics, or use completely different authorsin separate publications. 174Huston and Moher (1996) found that identifyingthe data from single centres of multicentre trialsof risperidone for schizophrenia was far fromsimple because of the chronology of publications,changing authorship, lack of transparency inreporting, and frequent citation of abstractsand unpublished reports. 176 For example, aNorth American trial had been reported inpart, transparently, and not so transparently,in six different publications by using differentauthor names. It had also been cited in severalunpublished forms. 176Easterbrook et al. (1991) conducted a surveyof studies approved by an REC and found thatstudies with significant results were more likely togenerate multiple publications and more likelyto be published in journals with a high citationimpact factor when compared with those with nonsignificantresults. 22 Vandekerckhove et al. (1993)identified a review of RCTs of infertility treatmentand found that ‘six studies with a significantresult (but none with a non-significant result)were reported in four publications from the sameinstitution’. 177The updating identified only an abstract by Martinet al. (2004) in which they examined the impactof including duplicate publications in a metaanalysisof off-pump versus on-pump coronaryartery bypass surgery. 178 Trials were classified ascovert duplicates when there was no citation ofthe original publication and non-covert duplicateswhen the publication declared the duplicationor cited the original publication. The authorsfound that a total of 15 (34%) of the 44 trialswere duplicate publications. Of the 15 duplicatepublished trials, 10 were covert and five werenon-covert publications. However, there was nosignificant difference in the estimate of mortalitywhen duplicates were included (OR 0.85; 95% CI:0.46 to 1.57) or not included (OR 0.86; 95% CI:0.48 to 1.54). 17833

Evidence of different types of dissemination bias34Summary of evidence onduplicate biasWe identified only very limited empirical evidencefrom case studies about the existence of duplicatepublication bias. However, it is clear that covertduplicate publication of data from the same studymay introduce bias in systematic reviews as theweights carried by particular studies are magnified.Place of publication biasBen-Shlomo and Davey-Smith (1994) foundthat the BMJ published more research articlessupporting the ‘early life hypothesis’ (about theimpact of early life development on the riskof adult disease) than the The Lancet. 179 Theysuggested that there may be ‘place of publication’bias because, for reasons of editorial policyor readers’ preference, one journal is moreenthusiastic towards publishing articles about agiven hypothesis than other journals. 179In a study that compared published and registeredtrials in advanced ovarian cancer, Simes (1986)found that trials with significant results (p < 0.05)in favour of the treatment tended to be publishedin prominent journals (such as the New EnglandJournal of Medicine and Cancer), while trials withnon-significant results tended to be publishedin less widely circulated journals. 127 Bero et al.(1994) compared 297 symposium articles injournal supplements and a sample of 100 journalarticles on environmental tobacco smokingpublished between 1995 and 1993, and found that‘symposium articles were more likely to agree withthe tobacco industry’s position (46% vs. 20%)’. 180This updated review includes a study by Penel andAdenis 181 that examined the association betweenthe results of 74 phase II trials investigatinganticancer targeted therapies and the impactfactors of journals publishing these trials. Positivetrials were defined as those with an objectiveresponse rate equal or superior to the prespecifiedefficacy threshold, and negative trials were thosewith a response rate lower than expected. It wasfound that positive results were more likely to bepublished in journals with high impact factors,compared with negative results (p = 0.004; median6.14 versus 2.71). 181We also identified a new study on location biasthat examined the results of clinical trials ofcomplementary and alternative medicine (CAM)therapies published in mainstream medicaljournals or in complementary medicine journals.Pittler et al. (2000) identified 19 systematicreviews that included 351 controlled trials ofcomplementary medicine. 182 Mainstream medicaljournals with a high impact factor tended topublish a relatively low proportion of trials withsignificant results compared with complementarymedicine journals (50% versus 63%). Theysuspected that ‘this may reflect the reluctance ofauthors to submit positive trial reports to these“flagship” orthodox journals, perceiving them tobe hostile to CAM’. 182Country biasThe causes of variable results from studies onthe same topic between different countries arecomplex, and selective publication is only onepossible explanation. The variable results betweendifferent countries were studied by Ottenbacherand DiFabio (1985). 183 They observed that theestimated efficacy of spinal manipulation therapywas greater in studies reported in English-languagejournals published outside the USA than for similarstudies in journals published in the USA (averageeffect size 0.45 versus 0.29). It was suggested thatthis finding might be explained by the existenceof publication bias and/or other interventioncharacteristics. 183A study by Vickers et al. (1998) examined 666abstracts from MEDLINE of clinical trialspublished up to 1995. 184 The proportion of positiveresults (when the test treatment was superiorto control) in trials comparing acupuncturewith controls was 100% for 50 trials originatingfrom China, Taiwan, Japan and Hong Kong. 184Conversely the results were 56.7% for 180 trialsoriginating from 14 western countries such asthe USA, UK, Sweden, Denmark, Germanyand Canada. The study also identified thatthe percentage of positive results in trials ofinterventions other than acupuncture was 99%for trials originating from China, 97% fromthe USSR/Russia, 95% from Taiwan, 89% fromJapan and 75% from England. It was concludedthat publication bias was a possible explanationfor the unusually high proportions of positiveresults reported from some countries. 184 Tang etal. (1999) confirmed the existence of publicationbias in Chinese journals of traditional medicine bypresenting an asymmetric funnel plot of 49 trials ofacupuncture in the treatment of stroke. 185Continuing the theme of more positive resultsappearing in published work from specificcountries, Pan et al. (2005) explored country

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8bias in the area of genetic epidemiology. 186 Theyworked with 13 gene-disease associations withexisting meta-analyses of at least 15 non-Chinesestudies and searched for relevant Chinese studies.Of the 161 studies found (augmenting the 301non-Chinese studies already included in the metaanalyses)only 20 were included in MEDLINE.Despite having smaller sample sizes than the non-Chinese studies, significantly more Chinese studiesshowed statistically significant associations (48%versus 18%) and the largest effects were seen inthe small sample of MEDLINE indexed Chinesestudies. This reinforces the finding that there arelarge bodies of literature that are commonly missedfrom meta-analyses using only MEDLINE, but thatsuch bodies may display high levels of publicationbias so that caution is needed in interpreting theresults of such groups.Lack of publication by authors from developingcountries may lead to ‘country bias’, both underrepresentingthe research questions of such areasand causing an important gap in our ability tolocate and synthesise the results of the whole bodyof conducted research. For example, King (2004)found that 31 countries accounted for 98% of theworld’s highly cited papers, the remaining 192countries accounting for less than 2%. 187 If theresults of such studies are different from the resultsof similar studies by researchers from developednations then we will observe publication bias.Database indexing biasDatabase indexing bias occurs when there isbiased indexing of published studies in literaturedatabases. 188 A literature database, such asMEDLINE or EMBASE, may not include andindex all published studies on a topic. 189–191 Theliterature search will be biased when it is based on adatabase in which the results of indexed studies aresystematically different from those of non-indexedstudies. This bias is likely because the result of astudy may determine whether and where the studyis published.This updated review identified no new studieson database indexing bias. The following twostudies were included in the previous HTAreport. A study by Zielinski in 1995 estimatedthat about 98% of journals indexed in the majorliterature databases were from western developedcountries. 192 Nieminen and Isohanni (1999)suggested that there was a bias against Europeanjournals in medical literature databases because27% of psychiatric research papers by Finnish© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.authors published in English were not indexed inMEDLINE. 193Media attention biasThe general population gets most of itsinformation about the latest developments inscience and medicine from the popular media.How the press presents the findings of thesedevelopments has a very powerful influence onpublic perception. Media attention bias occurswhen studies with striking results are more likelyto be covered by newspapers, radio and televisionnews. The overly optimistic portrayal of thescientific findings to the public affects the publicparticipation in policy discussions and createsunrealistic expectation of the potential benefits ofa new scientific development. 194 It was not clearwhether media coverage was influenced by people’sopinions about what is important, or whetherpeople’s judgements were influenced by the mediacoverage, although both directions of influence arepossible. 195The 2000 HTA report on publication bias 2 includedlimited evidence on media attention bias. Combsand Slovic in 1979 found that the coverage bytwo newspapers in the USA about causes of deathwas not related to the statistical frequency of theiroccurrence. 195 The newspaper overemphasisedhomicides, accidents and disasters, and underreporteddiseases as causes of death. Violentaccidents and homicides make more interestingand exciting stories than diseases. 195Houn et al. (1995) examined the popular presscoverage of research in the USA in 1985 and in1992 on the association between alcohol and breastcancer. 196 They identified 58 scientific articles and89 newspaper or magazine stories. Only 11 of these58 scientific articles were cited in the newspaperor magazine stories. Press stories cited all scientificarticles that were published in JAMA and the NEJMbut articles published in other journals were oftenignored by the newspaper and magazine reports.There was no significant difference between thescientific articles and press stories in the frequencyof reporting positive, negative or neutral results. Itwas concluded that ‘the vast majority of scientificstudies on alcohol and breast cancer were ignoredin press reports’. 196Koren and Klein (1991) 197 compared newspapercoverage in the USA of one positive study thatreported a significant association between radiationexposure and cancer risk 198 and one negative study35

Evidence of different types of dissemination biasthat did not, 199 published in the same issue of JAMAin 1991. Nine of the 19 newspaper reports coveredonly the positive study. In the other 10 reportsthat covered both the positive and the negativestudies, the average number of words was 354 forthe positive result and 192 for the negative result.It was suggested that the number, length andquality of newspaper reports on the positive studywere greater than news reports on the negativestudy, which suggests a bias against news reports ofstudies that show no effects or no adverse effects. 197This updated review identified two studiesthat examined the media coverage of abstractspresented at scientific meetings. Schwartz et al.(2002) examined 252 news stories about 147research articles presented at scientific meetingsin 1998, and found that the 43 abstracts thatreceived prominent news coverage were no morelikely to be formally published. 200 Woloshin andSchwartz (2006) found that the media coverage ofscientific meetings in major international outletsin 2003 often failed to report basic study facts, sothat the public would be likely to be misled aboutthe validity and relevance of the science presented,especially as there were no published findings torefer back to for confirmation. 201Whiteman et al. (2001) examined the scientificpublications that do and do not support anassociation between hormone replacementtherapy (HRT) and breast cancer to assess whetherthey were cited in the popular media in similarproportions. 202 A total of 32 scientific publicationswere identified, 20 (63%) of which had positiveconclusions in which the results supported theHRT–breast cancer association, and 12 (38%) didnot. Of the 203 citations in the media reports, 82%were of positive studies and 18% were of negativestudies, representing a significant excess of citationsof positive publications (p < 0.01). 202The reporting of clinical trials of herbal remediesby the popular media may be influenced by thedisclosure of funding information and competinginterest in the scientific and medical literature.Koper et al. (2006) 203 used a coding frame analysistechnique to systematically compare newspaperarticles with the reporting of the same trials in themedical literature. The analysis of 389 newspaperarticles from the UK, USA and Canada indicatedthat media coverage of conflicts of interest had aneffect on the overall tone of the article. 203Limitations of the availableevidenceEmpirical studies on publication and related biaseshave focused mainly on certain areas of researchsuch as clinical trials of health-care interventions.There is only very limited evidence on publicationbias in many other research fields including basicresearch and observational studies.Studies of publication and related biases themselvesmay be as vulnerable as other studies to theselective publication and reporting of significant orstriking findings. 1 Much of the empirical evidencecomes from case reports that may be selectivelyreported because of their striking findings.Studies that are less selective are able to providemore convincing evidence on publication andrelated biases, including cohorts of researchprotocols, submitted or registered studies.However, many empirical studies were basedon cohorts of studies that were diverse in termsof design and research questions. It is usuallyimpossible to exclude the impact of confoundingfactors on the observed association betweenstudy results and publication status. There is verylimited and conflicting evidence on factors thatmay be associated with the direction and extent ofpublication and related biases.Findings from individual empirical studies wereoften heterogeneous, and pooled estimates ofpublication and related biases can indicate someaverage trends but may not be generalisable tomany individual cases. A case-by-case approachis required to gauge the possible impact ofpublication and related biases and to decideappropriate measures to deal with these biases.ConclusionsThe 2000 HTA report included very limitedevidence on outcome reporting bias. Recentlypublished studies have provided convincingevidence that outcome reporting bias exists and islikely to have important effects on pooled summarydata within systematic reviews. Limited evidenceindicates that harm and subjectively assessedoutcomes may be more vulnerable to biasedselective reporting than efficacy and objectivelyassessed outcomes.36

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Studies with significant or positive results tend, onaverage, to be published earlier than studies withnon-significant or negative results. However, newevidence is less clear about time lag bias than wassuggested in the previous review. One consequenceof time lag bias would be a diminishing effectsize reported by studies over time, although verylimited research has been conducted to investigatetemporal trends of reported effect size in metaanalysis.The updated review identified substantially newevidence on grey literature bias. Evidence suggeststhat published studies tend to report a greatertreatment effect than those of grey literature orunpublished studies. However, for individual cases,the direction of bias is unpredictable, and greyliterature studies may be relatively small and ofpoor quality, although this is not always the case. Insome reviews inclusion of data from grey literatureor unpublished studies have important clinicalimplications.Substantially new evidence on language bias hasbeen identified. The impact of excluding non-English-language studies from systematic reviewswas highly heterogeneous. Exclusion of non-English-language studies from systematic reviewsmay be associated with greater, similar or smallerestimates of treatment effects. However, exclusionof non-English-language studies appears to resultin a particularly high risk of bias in some areas ofresearch such as complementary and alternativemedicine.Empirical evidence indicates that studies withsignificant or positive results are on averageassociated with a higher frequency of citation.Non-systematic narrative reviews are a specific areawhere biased citation of research findings can resultin misleading conclusions.The updated review identified very limited newevidence on duplicate publication bias, althoughit is clear that covert duplicate publication of datamay introduce bias in systematic reviews. Availableevidence on the existence of place of publicationbias, database or index bias, country bias andmedia attention bias is still very limited. Theimpact of these biases could be prevented in wellconductedsystematic reviews. 204There is limited evidence on place of publicationbias, database bias, country bias and mediaattention bias. It is helpful to be aware of thepotential existence of these biases.37© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 5Consequences of dissemination biasEvidence from empirical studies reviewed inChapter 3 suggests that the disseminationprofile of research findings may be associatedwith the strength or direction of study results. Asa direct consequence of publication and relatedbiases, published studies may provide misleadingestimates of treatment effects or associationsbetween variables. The previous 2000 HTA reportidentified very little direct evidence on the impactof publication and related biases on health policy,clinical decision-making and the outcome ofpatient management. 2 In this updated review,we considered consequences of publication biasaccording to types of studies, classifying them intothree categories: basic research, observationalstudies and clinical trials.Basic research studiesMany new treatments are initially investigated inbasic laboratory and animal research. Based onfindings from basic research, clinical trials maybe conducted to test an intervention in humans.However, subsequent clinical trials often fail toprovide confirmatory positive results, inconsistentwith findings from basic animal research. 205 Oneof several possible explanations for the observeddiscrepancies in results between basic researchand clinical trials is biased publication of positiveresults of basic studies. 206 If positive results frombasic research are more likely to be publishedthan negative results, results of published studiesof basic research will represent an overestimationof potential treatment effects. It is unlikely thatclinical trials that are designed based on falsepositivefindings from basic research will provide apositive result.Empirical evidence on the existence and impactof publication bias is very limited in the field ofbasic laboratory and animal research. This updatedreview included a case study of a neuroprotectivedrug, nicotinamide, for focal cerebral ischaemiain animal experimental studies. 132 The animalexperimental studies suggested potential efficacyof neuroprotective drugs, but clinical trials failedto confirm these drugs’ efficacy. Macleod et al.(2004) conducted a systematic review of animal© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.experimental studies of nicotinamide. They foundthat animal studies that were fully publishedshowed a greater effect (effect size 0.306; 95% CI:0.241 to 0.371) than studies that were presentedin abstract form (0.162; 95% CI: 0.066 to 0.258).It was suspected that some studies with negativeresults may not be available even in abstractform. 132Observational studiesA large number of epidemiological studies havebeen conducted to investigate various risk factorsassociated with diseases. 207 However, there arecontradictory findings from epidemiologicalstudies regarding many risk factors. 208 Forexample, the results of epidemiological studieswere contradictory regarding the risk of hair dyes,coffee, oat bran, oral contraceptives, environmentalexposure to residential radon, and the presence ofDDT metabolites in the bloodstream. 209Ioannidis and Trikalinos found that early publishedstudies of genetic associations tended to beextremely contradictory, and hypothesised that‘highly contradictory results are most tantalizingand attractive to investigators and editors’. 112 Twofurther studies (all by Ioannidis and his colleagues)found considerable outcome reporting bias instudies of cancer prognostic factors, 97 and in studiesof epidemiological risks. 98 Therefore, publicationand related biases may be an important reason formany of the controversies surrounding the resultsof epidemiological studies.Clinical trialsThe impact of publication bias in clinical trials willdepend on the extent of bias, and the underlyingeffects evaluated. The worst scenario would bewhere a harmful intervention is falsely reportedas effective because of publication bias, which mayresult in patients receiving a harmful treatment.If an ineffective intervention is falsely consideredas effective, patients may receive an ineffectivetreatment and be denied effective treatments.For an effective intervention, its effects may be39

Consequences of dissemination biasoverestimated because of publication bias. Newinterventions are generally more expensive thanconventional interventions, so overestimationof the efficacy of new interventions is likely toresult in increased cost without a correspondingimprovement in outcome.A perinatal trial observed that routinehospitalisation was associated with more unwantedoutcomes in women with uncomplicatedtwin pregnancies, but this finding remainedunpublished for 7 years. 210 Chalmers pointed outthat ‘at the very least, this delay led to continuedinappropriate deployment of limited resources; atworst, it may have resulted in the continued use ofa harmful policy’. 210The non-publication of research findings mayindirectly harm patients who are involved infuture research. For example, a clinical studymay find that an intervention is harmful but thisfinding is not published. Other investigators maysubsequently repeat the same research, testing theharmful treatment on different patients. In 1980,a trial tested lorcainide in patients with acute andrecovering myocardial infarction. More deathswere observed in the lorcainide group than in theplacebo group (9/48 versus 1/47). 211 The trial resultswere not published because the development oflorcainide was stopped for ‘commercial reasons’.About a decade later, an increased mortality wasobserved among patients treated with the relatedagents, encainide and flecainide, in two trials. 212,213Encainide, flecainide and lorcainide all belong toa class of I Cantiarrhythmic agents. If the results ofthe trial in 1980 had been published, the increasedmortality of patients included in the two later trialsmight have been avoided.Recently there were several high-profile cases ofalleged publication or reporting bias in drug trials.Rofecoxib was withdrawn from the market byMerck on 30 September 2004, because of increasedrisk of myocardial infarction and stroke accordingto unpublished data from a clinical trial. 214 Editorsof NEJM expressed their concern about a trial ofrofecoxib, published in the journal in 2000, inwhich three cases of myocardial infarction were notdisclosed in the article. 215 Although authors of thetrial denied any wrongdoing, 216 the NEJM editorsrestated their concern. 217 Further, in the yearbefore the withdrawal, the authors of a systematicreview had written directly to the primary authorof every published trial of rofecoxib asking aboutcardiovascular events and major bleeds; however,they received only a single reply and it did notprovide data on cardiovascular events. 218In a more recent case study, Psaty and Kronmal(2008) found biased reporting of findings fromclinical trials of rofecoxib for Alzheimer’s diseaseor cognitive impairment. 219 Before its withdrawal,rofecoxib had been used in more than 80 millionpatients. 214 Biased reporting of findings fromtrials may have encouraged more patients to userofecoxib and delayed the detection of harmfuleffects of rofecoxib.In 2003, medicine regulatory authorities in severalcountries advised that a new antidepressant,paroxetine, should not be used in children withdepression (with several other SSRIs added tothe list later on), based mainly on findings fromunpublished trials from industry. 220,221 This caseactually suggests that formal publication maynot be the most effective and timely approachto disseminating important research findings.Findings from clinical trials indicated thatthese SSRI antidepressants were ineffectiveand associated with increased suicidality andaggression in children with depression. Kondroand Sibbald (2004) revealed a drug company’sinternal document in which staff were advisedto withhold data about SSRI use in children. 222and GlaxoSmithKline was threatened with legalaction over concealment of trial results. 223 Ameta-analysis by Turner et al. examined 74 FDAregisteredclinical trials of antidepressants andfound that trials with positive results were morelikely to be published than those with negativeresults. 45 Ioannidis suspected that some relevanttrials conducted after market approval may not beincluded even in the FDA database. 224SummaryThe most important consequences of publicationbias include avoidable suffering of patients andwaste of limited resources. This updated reviewidentified only a couple of new cases that indicatethe detrimental impact of publication and relatedbiases. Consequences of publication and relatedbiases are different for different types of researchstudies. Because of the possibility of such bias, theintegrity of scientific research could have beenjeopardised.40

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 6Sources of publication biasBiased selection for publication may occurto a varying degree during all stages of thepublication process, from author submissionand peer review to editorial decision, due to avariety of reasons. 20,225,226 Since bias is a naturalhuman phenomenon, 227 publication bias maybe introduced intentionally or unintentionally,consciously or unconsciously. This chapterprovides an updated review of evidence aboutthe responsibility of investigators, journal editorsor peer-reviewers, and research sponsors forthe existence of publication bias. Other studylevelfactors (including sample size, underlyingtrue effect, study design and quality) that mayexacerbate the risk of publication of a biasedselection of studies are then discussed.Investigators and authorsThere are various reasons for not writing up anarticle or not submitting it, such as pressure fromresearch sponsors and instructions from journaleditors. The previous HTA report 2 included ninestudies of reasons given by investigators for notpublishing studies. 20–22,30,50,54,228–230 We identifiedan additional 12 studies in this updated review(Appendix 14). 28,35,67,70,72,231–237 It should be notedthat studies often used different ways to categorisereasons for non-publication and reasons given ina study may not be independent of each other. Forexample, citing ‘result not important enough’ asthe reason may be the cause of other given reasonssuch as ‘not worth the trouble’ and ‘not enoughtime’.Studies included in the previous HTA report andthose newly identified reported similar reasons fornot publishing (Appendix 14). Of the 21 studiesincluded, there are five studies of investigators ofprotocol cohorts, 11 studies of authors of meetingabstracts, and five studies of other or miscellaneousauthors. Percentages of specific reasons fromindividual studies were transformed to log oddsand pooled using random-effects model, althoughthere was significant heterogeneity across studies(Figure 7). The main reasons for non-publicationwere lack of time or low priority (34.5%; 95% CI:27.4% to 42.3%), results not important enough© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.(19.6%; 95% CI: 12.0% to 30.4%) and journalrejection (10.2%; 95% CI: 5.5% to 18.2%) (seeFigure 7). Pooled percentages of specific reasonswere similar across different types of empiricalstudies, except that the lack of time or low interestwere significantly higher in studies of meetingabstracts (43.1%; 95% CI: 35.9% to 50.6%) than instudies of protocol cohorts (23.8%; 95% CI: 15.9%to 34.0%) or studies of other authors (20.7%; 95%CI: 7.7% to 44.9%) (see Figure 7). In the five studiesof meeting abstracts, fear of journal rejectionwas given as a reason for 23.7% (95% CI: 8.9% to49.6%) of unpublished studies.It should be noted that ‘lack of time’ may bemore likely used as the excuse for not publishingunimportant results. The same researcher mayhave several different simultaneous studies thatneed attention, and may be reluctant to spendalready limited time on the preparation ofmanuscripts for studies with unimportant or nonsignificantresults that are less likely to be acceptedby high-profile journals. In a qualitative study ofcauses of publication bias in genetic epidemiology,an experienced researcher in genetic epidemiologyadmitted that because of time constraints and‘piles’ of results available, efforts will inevitablyfocus on the publication of ‘wonderful results’, not‘negative results’. 84 These findings indicated thatinvestigators may be the main source of publicationbias, for not writing up or submitting studies with‘unimportant’ results.Blumenthal et al. conducted a postal survey of 3394life sciences faculty members at 50 universities thatreceived the most funding from the NIH in 1993. 231Delay to publication by more than 6 months inthe last 3 years was reported at least once by 19%of the 2167 respondents. Principal reasons givenby respondents for delay to publication includedpatent application submission (46%), protectionof scientific lead (31%), patent negotiation(26%), time for resolution of intellectual propertyownership (17%) and slow dissemination ofundesired results (28%). 231According to a recent survey of 119 authors ofpapers published in six general medical journals,authors still considered that good study quality,41

Sources of publication biasEstimate (% and 95% CI)6050403020100TotalProtocolAbstractLack of time ornot interestedOtherTotalProtocolAbstractOtherTotalResults not important Rejected by journal Fear of beingrejectedProtocolAbstractOtherTotalAbstractFigure 7 Reasons given by investigators for not publishing: scatter plot.manuscript writing and statistical significance ofresults were important factors associated with thepossibility of a study being published. 238Findings from surveys of investigators aresupported by evidence from other studies. Sternand Simes found that quantitative studies withsignificant results were more likely to be submittedthan studies with null results (78% versus 54%,p < 0.001). 24 Ioannidis found that studies withpositive results were often submitted for publicationmore rapidly after completion than were negativestudies. 23factors at primary submission. For subsequentsubmission, manuscript acceptance (5.0) andwhether the manuscript usually publishes articleson the topic (4.7) were the most important factorsdetermining submission. 241McCambridge (2007) discussed a case ofpublication bias in reviews of drug education. 242A series of systematic reviews of drug educationin schools were conducted by Tobler et al., andformally published in 1986, 1997 and 2000. 243–245Findings from these reviews indicated thatinterventions delivered by mental health clinicianswere more effective than those by others; andCain and Detsky believed that even physicians interactive programmes were effective and noninteractivecan be biased. 227 A study investigating enthusiasmprogrammes were not. These findingsfor radiotherapy after radical mastectomy when have had considerable impact on research, policystage was not distinguished showed 21 out of and practice. Through personal communication,29 radiotherapists were enthusiastic compared McCambridge obtained some unpublished resultswith 5 out of 34 authors in other specialties. 239 of updated meta-analysis conducted by the sameA systematic review of risk of strokes and death team of the previous three systematic reviews. 242following endarterectomy for symptomatic carotid According to the unpublished results of updatedstenosis showed a higher risk in studies where meta-analysis, differences between differentneurologists assessed the patients and lowest where intervention programmes are no longer statisticallythe single author was affiliated to a department of significant, but these findings have not beensurgery (7.7% versus 2.3%). 240formally published in peer-reviewed journals at 4years after the end of the review project. The nonpositiveAuthors’ criteria for selecting journals wasfinding is likely to be one of the reasons forinvestigated in a study of all active clinical and non-publication. 242research faculty at Stanford University School ofMedicine. 241 A response rate of 63.7% with factorsranked from unimportant (1) to very important Editorial review process(6) showed journal prestige (5.2), makeup ofjournal’s readership (4.8), whether the journalEditorial policiesTitle: publishes 06-92-02 articles on the topic (4.8) and likelihood Proof Little Stage: is known 2 about Figure the actual Number: editorial 00.07.aiprocessof manuscript acceptance (4.4) to be the important itself. A semistructured interview of editors of42 Cactus Design and Illustration Ltd

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8three leading biomedical journals found a greatdiversity in editorial policies and proceduresbetween the journals. 246 A retrospective reviewof studies published in 2006 found that medicaljournals were more likely to publish reportsfrom their own editorial board than from otherjournals. 247 Although editorial rejection was not afrequent reason given by investigators for studiesremaining unpublished (see Figure 7), authorsmay not submit articles with ‘unimportant’ resultsbecause of anticipated rejection according tojournals’ instructions to authors and their own (orcolleagues’) experience.The 2000 HTA report on publication biasincluded several studies that surveyed authors orinvestigators about manuscript submission forpublication. 23,230,248 A study by Weber et al. reportedthat anticipated rejection by journals was citedas a reason for failure to submit a manuscript by20% of 179 authors. 230 In another study, 17 of45 submitted trials were rejected by at least onejournal, and at least four negative trials with over300 patients each were rejected two or three times,while no positive trial was multiply rejected. 23 Ina survey of 80 authors of articles published inpsychology or educational journals in 1988, 61%of the 68 respondents agreed that, if the researchresult is not statistically significant, there is littlechance of the manuscript being published. 248Several new studies identified in the updatedreview provide results similar to those reported inthe previous studies. 67,233,236,237 Anticipated rejectionby journals was the reason for not submitting astudy given by 10% of investigators in the articleby Vuckovic Dekic et al., 237 by 13% in Hashkes andUziel, 67 by 13% in Sprague et al., 236 and by up to26% in Hartling et al. 233The 2000 HTA report also included severalstudies that surveyed journal editors. A surveyof 429 editors or members of advisory boardsof 19 leading journals in management and therelated social sciences in 1974 found that nonsignificantresults, replications, lack of new data,similarity to recently published articles, or havingpreviously been presented at meetings were factorsassociated with reduced chance of acceptance. 249 Asurvey in 1996 of 36 editors of English-languagejournals found that editors primarily valuedthe significance and importance of the researchabove validity of the experimental and statisticalmethods. 250 Originality and clinical significance ofresults are also important criteria for manuscriptacceptance. Negative results may have less ofan effect on clinical practice, supporting their© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.publication in pay-to-publish journals or openaccess electronic journals unless they show a widelyused intervention is ineffective. 19,251 Qualitativecriteria for assessing study importance includeoriginality of results, predictability, triviality, narrowinterest, highly specialised and few/no clinicalimplications. 252 Unoriginality accounted for 14%of all reasons given for rejection of manuscriptsin 1989 by the American Journal of Surgery. 253Confirmatory studies, either positive or negative,have a low chance of being accepted. 254,255The updated review included only one newrelevant study that surveyed journal editors. Asurvey of the editors of 33 medical journals ownedby not-for-profit organisations showed that 70%reported having complete editorial freedomand the remainder reported having a high levelof freedom. 256 However, 42% reported beingpressurised by the association’s leadership, 30%by senior staff and 39% by rank-and-file members.Ultimately 48% of the journal’s board of directorshad authority to hire and 55% to fire the editorindicating that editorial independence fromjournal owners needs protection. 256Several cases of inappropriate instructions toauthors by journal editors that may lead topublication bias were reported in the 2000 HTAreport on publication bias. For example, a journalon diabetes clearly stated that ‘mere confirmationof known facts will be accepted only in exceptionalcases; the same applies to reports of experimentsand observations having no positive outcome’. 255More journal editors may have realised thedetrimental impact of selective publication ofpositive results, and we are not currently aware ofexplicit journal instructions to authors that may bea cause of publication bias. However, further effortswill be required to translate this change in journaleditorial policies to the submission behaviour ofauthors and investigators.Journal peer reviewJournal peer review has been defined as ‘theassessment by experts (peers) of materialsubmitted for publication in scientific and technicalperiodicals’. 257 Unacceptable biases in the peerreview process include biases related to certaintypes of author (prestige, gender, nationality), orcertain types of manuscript (language, innovation,positive/negative results). 225,258 The process ofjournal peer reviewing is a complex process andthere are many studies on different types of biasesin peer reviewing. This report considers only43

Sources of publication bias44biased peer reviewing process as a possible causeof selective publication according to study results.The 2000 HTA report on publication bias includedseveral studies that used sham papers to investigatepublication bias in the peer review process. No newrelevant studies have been identified in this updatereview, and studies included in the previous HTAreport are discussed below.Mahoney sent a sham paper with identicalexperimental procedures but different results to75 journal referees and found poor agreementbetween reviewers and bias against the manuscriptthat reported results conflicting with referees’ ownperspectives (confirmatory bias). 259 A further studygave a similar result, in which a sham paper abouttranscutaneous electrical nerve stimulation (TENS)was sent to 33 referees identified as pro- or contra-TENS. 260 Referees’ judgement was associated withpreconception and experience, and inter-raterreliability was again found to be poor. 260 Abbotand Ernst sent four versions of a sham study incomplementary medicine to 200 authors, andfound that the poor quality manuscript was rejectedsignificantly more often than the good qualitymanuscript (55% versus 16%; p < 0.05), and noevidence of peer-reviewer bias against a positive ornegative outcome. 261Abstract reviewing may be less predictable thanreviewing a full article. Ector et al. compared theagreement between reviewers in grading abstractssubmitted to a conference (the sixth EuropeanSymposium on Cardiac Pacing). 262 Each abstractwas graded on a scale of 1 to 10 by two peerreviewers.There was no statistically significantcorrelation between reviewers in 13 of the 28pairs. It was suggested that reviewing abstracts isless predictable and more likely to be biased thanreviewing a full article. 262 A study of 1983 posterssubmitted for three annual conferences for biaswas conducted by Blackburn et al. 263 Posters havingauthorship that included at least one reviewerreceived higher ratings than those having onlynon-reviewing authors. 263Wager et al. compared reviews from reviewersselected by authors and those selected by editors,and found that reviewer source had no impacton review quality or tone but that authornominatedreviewers were significantly morelikely to recommend acceptance and less likely torecommend rejection than editor-chosen reviewersafter initial review. 264 Another similar study alsofound that editor-selected reviewers were lesslikely to recommend acceptance than authorchosenreviewers, although there was no significantdifference in review quality or speed betweenthem. 265Geographical bias can also influence peer review. Ina Scandinavian study, two versions of a sham paperwith methodological flaws, one in Scandinavianand one in English, were sent to 180 Scandinavianreviewers. 144 The 156 referees who returned 312reviews considered the English-language versionsignificantly better than the Scandinavian version(p < 0.05). 144 A retrospective analysis of originalsubmissions received by the journal Gastroenterologyin 1995 and 1996 also showed geographical bias. 266There were 2355 US and 1297 non-US reviewers(p = 0.31), with US reviewers recommendingacceptance of papers submitted by US authorsmore often than non-US reviewers (p = 0.001).Non-US reviewers ranked US papers slightlymore favourably than non-US papers (p = 0.09),with US reviewers ranking US papers much morefavourably (p = 0.001). 266 However, a study of 3444papers submitted to the journal CardiovascularResearch between 1997 and 2002 showed that USreviewers assigned significantly higher priority tomanuscripts regardless of where the manuscriptwas from (p < 0.0005). 267 The same study alsofound that manuscripts received significantlyhigher priority ratings when reviewers and authorsoriginated from the same country (p < 0.05). 267Gender bias during peer review of manuscriptsand grant proposals has also been demonstratedin several studies. A study of manuscripts receivedby JAMA in 1991 comprising 1698 male and 462female authors with eight male editors, five femaleeditors, 2452 male and 930 female reviewersshowed significant gender differences. 268 Femaleeditors were assigned manuscripts from femalesmore often than males (p < 0.001). Female editorsalso used more reviewers per manuscript if sentfor other review and rejected more manuscripts(p < 0.001). 268 However, articles submitted were notaccepted at significantly different rates based ongender (p < 0.4). A Scandinavian study used a shampaper with either a female or male author to assessgender bias in 1637 randomly selected Swedishphysicians. 269 Female authors were ranked higherthan male authors, with female assessors upgradingfemale authors more than male authors and maleassessors reflecting no gender differences. 269 Astudy conducted by Caelleigh et al. of 50 femaleand 50 male reviewers showed no gender bias whenassessing an empirical study with two versions,one attributing lower forecast income of women to

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8intrinsic gender factors and the other attributingthe difference to extrinsic social learning factors. 270In a retrospective study, the effect of institutionalprestige on referees’ recommendation and editorialdecision was assessed. 271 Institutional prestigewas determined according to the monetary valueof research and training grants and contractsfunded by the National Institutes of Health.The association between the recommendationfor acceptance and institutional prestige wasobserved for the 147 brief reports (i.e. case reportsand similar short papers) but not for 258 majorpapers (such as case series, research reports andepidemiological studies). 271Study results and journaleditorial decisionsRejection by journals was given by investigatorsas a reason for 5% to 33% of non-publication ofstudies (see Figure 7). If the decision to accept orreject studies for publication is not based on studyfindings, the rejection of studies by journals will notresult in publication bias.The 2000 HTA report on publication bias includedseveral studies that provided limited evidence onthe acceptance of manuscript by study results.Epstein sent two versions of a fictitious paper witheither positive or negative results to 146 socialwork journals. 272 The positive manuscript wasaccepted in 35% and the negative manuscript wasaccepted in 25% (p > 0.05). 272 A study found that17 of 45 submitted trials were rejected by at leastone journal, four negative trials were rejected twoor three times and no positive trial was multiplyrejected. 23 However, another study found nodifference in the rate of publication of submittedmanuscripts between studies with significant resultsand studies with null results (87% versus 82%,p = 0.54). 24 In a case–control study of 100 acceptedand 100 rejected papers in two Spanish medicaljournals, it was found that publication status wasassociated with high study quality, not positivefindings. 273The updated review identified four studies thatfollowed cohorts of manuscripts submitted tojournals (Appendix 15). 78–81 Results of these fourstudies of manuscript cohorts have been discussedin Chapter 3. Two studies examined manuscriptssubmitted to general medical journals (JAMA, BMJ,The Lancet, and Annals of Internal Medicine) 78,81 andtwo used manuscripts submitted to the Journal ofBone and Joint Surgery (American Version). 79,80 The© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.study results of submitted papers were classifiedaccording to the significance of statistical testing(p < 0.05 or not) in the two studies of manuscriptssubmitted to general medical journals. 78,81 In thestudies of manuscripts submitted to the Journalof Bone and Joint Surgery, results were classified asbeing positive, negative or neutral, although thedefinitions of these outcomes may be differentbetween the two studies (Appendix 15). 79,80Figure 8 shows the results from the four studies andthe pooled odds ratio of acceptance rate (OR 1.06;95% CI: 0.80 to 1.39), which suggested that theacceptance of submitted papers for publication byjournals was not significantly associated with thedirection or strength of their findings. In addition,Olson et al. 81 further examined 133 acceptedmanuscripts and found that time to publicationwas not associated with statistical significance(median 7.8 months for positive and 7.6 monthsfor negative results, p = 0.44). 82Because the acceptance of manuscripts forpublication by journal editors was not determinedby the direction or strength of study results,the existence of publication bias may be largelydue to biased selection of studies to submit byinvestigators. This may also be supported by thefact that a large proportion of submitted papersshowed statistically significant results (51% to 87%)or positive results (71% to 72%) in the four cohortstudies. Since any author will inevitably considerthe possibility of their manuscripts being acceptedbefore submission, submitted studies with negativeresults may be a biased selection of all studies withnegative results.In Olson et al.’s cohort study of manuscriptssubmitted to JAMA, there was a tendency thatstudies with significant results had a higher rateof acceptance than studies with non-significant orunclear results (20.4% versus 15.2%, p = 0.07). 81 Inthe cohort study by Okike et al., a subgroup analysisof 156 manuscripts with a high level of evidence(level I or II) found that the acceptance rate wassignificantly higher for studies with positive orneutral results than for studies with negative results(37%, 36% and 5% respectively; p = 0.02). 80The studies included in Appendix 15 are generallywell designed and conducted. Although no conflictof interest was declared in the four cohort studiesof submitted manuscripts, this kind of study willalways need support or collaboration from editorsof the journal. In prospective studies, editors’decisions on the acceptance of manuscripts may45

Sources of publication bias τ =χ == = = = = Figure 8 Acceptance rate and results of studies submitted to journals for publication. Unadjusted odds ratio.46be influenced by their awareness of the ongoingstudy. 81 Therefore, biased selection for publicationby journals cannot be completely ruled out.Readers and users ofresearch findingsJournal editors’ policy may reflect readers’preferences, and it has been suggested that editorsshould find ways to incorporate the reader’sperspective into the peer review process andstudy the effects of their efforts. It is likely thatreaders’ preferences for certain findings may bean important reason for the biased publicationof studies in journals. We have identified no newrelevant studies in the updated review, althoughtwo studies were discussed in the previous HTAreport on publication bias. A survey of 452 readersshowed readers were generally satisfied with thequality of manuscripts but dissatisfied with the lackof manuscripts relevant to medical practice. 274 Thedifference of opinion between readers and peerreviewersmay be attributable to clinicians avoidingunestablished treatments but journals being morelikely to accept for publication manuscripts withnovel treatments. 274Research funding bodies andcommercial interestsResearch commissioning bias may contribute topublication bias since industry sponsors researchand often own the data, making them susceptibleto manipulation and suppression. Rosenberg notedthe conflict between dissemination of researchfindings with the protection of investors who havesupported the research that pervades modernscience. 275 An editorial in JAMA noted that 35%of signed agreements in a sample of universityindustryresearch centres allowed the sponsor todelete information from publication, 53% allowedpublication to be delayed and 30% allowed both. 276The 2000 HTA report on publication bias includedseveral studies that investigated associationbetween study results and industry sponsorshipin biomedical research. A study of clinical trialspublished in 1984 in five general medical journalsshowed 89% of drug company-funded trialssupported a new therapy compared with 61% ofgenerally funded trials (p = 0.002). 277 Another studyof 56 RCTs published between 1987 and 1990 andconcerning NSAIDs in the treatment of arthritisshowed that in all trials manufacturer-associateddrugs were reported to be comparable with (71%)or superior to (29%) the control drugs. 278 Of the22 trials that reported a drug with less toxicity, themanufacturer-associated drug’s safety was reportedto be superior in 86% of cases with justificationprovided in only 55%, suggesting selectivepublication or biased interpretation of results inmanufacturer-associated trials. 278Stelfox et al. examined the published safetyprofiles of calcium-channel antagonists andthe financial association of authors with thepharmaceutical industry. 279 They identified 77articles and a questionnaire was sent to 86 authorsof 70 articles, of whom 69 authors completedthe survey. Of the authors that supported thesafety of calcium-channel antagonists, 96% hadfinancial relationships with the manufacturerscompared to 60% of neutral authors and 37%of critical authors (p < 0.001). 279 Therefore, the

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8importance of full disclosure of relationships withpharmaceutical manufacturers in journal articleswas highlighted. 279–284The updated review identified several recentlypublished reviews or primary studies aboutindustry sponsorship in biomedical research. 285–287A systematic review published in 2003 by Bekelmanet al. found that industry research funding wasreceived by 23% to 28% of academic researchers,and was associated with restrictions on opencollaboration, data access or publication ofresults. 285 Pooling of results from eight studiesfound that industry sponsorship was statisticallysignificantly associated with pro-industryconclusions (pooled OR 3.60; 95% CI: 2.63 to4.91). 285 A similar systematic review by Lexchin etal., also published in 2003, found that ‘researchfunded by drug companies was less likely to bepublished than research funded by other sources’,and industry-sponsored studies were more likely toreport outcomes favouring the sponsor comparedwith studies supported by others (pooled OR 4.05;95% CI: 2.98 to 5.51). 286 Findings of the abovetwo systematic reviews in 2003 285,286 are confirmedby results of recently published studies. 288–298Jorgensen et al. compared Cochrane reviews withindustry-supported meta-analyses of the samedrugs. They found that industry-supported metaanalysesof drugs ‘were less transparent, had fewreservations about methodological limitationsof the included trials, and had more favourableconclusions than the corresponding Cochranereviews’. 299Sawka and Thabane pointed out significantheterogeneity across studies in the meta-analysisby Bekelman et al. of results of industry-sponsoredstudies, and suggested that the pooled odds ratiois ‘unconventional’. 300 In many studies included inthe two systematic reviews, 285,286 industry-sponsoredstudies may not be comparable with non-industrysponsoredstudies from many perspectives, 301although they had similar methodological quality.Studies that included homogeneous research interms of patients and interventions seemed lesslikely to find significant differences in resultsbetween industry-sponsored and non-industrysponsoredresearch. For example, in a metaanalysisof trials of antimuscarinic medicationsfor overactive bladder, Tulikangas et al. found ‘nodifference in outcomes when comparing studiesfunded by industry or not for tolterodine andoxybutynin’. 302 Barden et al. investigated industrybias using comparable trials on acute pain andmigraine and found no evidence indicating thatindustry-sponsored trials on acute pain andmigraine were biased. 301Therefore, publication bias (including outcomereporting bias) is only one of several possibleexplanations for observed association betweenfavourable results and the industry sponsorship.However, direct evidence showing industry’scommercial interests as a source of publicationbias does exist. 303 Identified case studies on biasedreporting of research due to commercial interestsare summarised in Appendix 16. 136,214,215,217,220,276,304–332Some pharmaceutical companies attempted tosuppress the publication of ‘negative’ resultsby taking legal action, and all cases occurredbefore 2000. 276,304–309 One company took legalaction against an investigator in order to stop thepublication of negative results from a study ondeferiprone in patients with thalassaemia. 305,306A study by Dong et al. showing bioequivalenceof generic and brand name levothyroxine wassuppressed by the pharmaceutical company dueto the deleterious effect of the results on the priceof the company’s product. 333 A pharmaceuticalcompany also tried to suppress a systematic reviewthat would have had a negative economic impacton statins. 334 Publication of a meta-analysis withunsupportive results of bovine somatotrophin wasblocked by a pharmaceutical company using itslegal rights over the raw data. 304It seems that industry is no longer able to suppressthe publication of results of entire sponsoredresearch. However, the updated review identifiedseveral new cases in which results of industrysponsoredresearch were selectively reported ormisrepresented in publication (Appendix 16).136,217,219,220,222,323,324,331,332,335Non-publication of‘negative’ results was common. 317–320,322,336 Forexample, Psaty and Kronmal compared publishedand unpublished mortality findings in two trialsof rofecoxib for Alzheimer’s disease. 219 The twopublished articles only mentioned on-treatmentmortality in the text without any statistical analyses,and concluded that rofecoxib is well tolerated. 330,337However, the company’s unpublished intention-totreatanalyses and the independent analyses basedon data provided by the sponsor in the New JerseyVioxx litigation found a statistically significantincrease in total mortality (HR 2.99; 95% CI1.55 to 5.56; and HR 2.13; 95% CI: 1.55 to 5.77,respectively). 21947© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Sources of publication biasThe direct evidence included in Appendix 16was mainly restricted to some high-profile caseswhere investigators determined to challengeindustry’s suppression of publication, or where theopen access policy facilitated the identification ofdiscrepancies between published and unpublishedresults. There may be many hidden cases whereresearch results were not disclosed becauseinvestigators gave in to the pressure from researchsponsors.A more recently published systematic reviewincluded seven studies that compared the reportingof adverse effects according to funding sources. 287There was no clear evidence that the reporting ofthe raw adverse effects data was biased. However,a drug was more likely to be interpreted as safe byindustry-funded authors compared with authorswithout pharmaceutical funding. 287Variation in study resultsIf the results from all possible studies were the sameor similar, selected publication of results wouldnot be biased. Greater variation in the results maybe associated with an increased risk of publicationbias. Factors that influence variation in study resultsinclude small sample size, small or moderate effectsize, subjective nature of outcome measurement,and complex interventions. However, we havenot been able to identify any studies that providedirect empirical evidence on results variation andpublication bias. In this section, the updated reviewSample size100200True OR = 1.0True OR = 0.8True OR = 0.640060080010001200–0.05 0.000.05 0.10 0.15 0.20 0.25 0.30Publication bias (In OR: true – estimated)Size n True OR = 1.0 True OR = 0.8 True OR = 0.650 –0.017 0.223 0.270100 0.019 0.206 0.164200 0.038 0.139 0.118300 0.016 0.125 0.042400 –0.009 0.083 0.022500 –0.026 0.084 0.014600 0.013 0.064 0.001Figure 9 Extent of publication bias and range of sample sizes: results of 1000 times simulation (selection based on p values; controlrate = 0.20).48

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8True OR (In OR)1.0 (0.000)0.9 (–0.105)n = 100n = 200n = 4000.8 (–0.223)0.7 (–0.357)0.6 (–0.511)0.5 (–0.693)0.4 (–0.916)0.3 (–1.204)–0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30Publication bias (In OR: true–estimated)True OR n = 100 n = 200 n = 4001.0 0.044 0.030 –0.0230.9 0.073 0.066 0.1340.8 0.148 0.177 0.1640.7 0.285 0.199 0.1460.6 0.269 0.177 0.0970.5 0.284 0.172 0.0530.4 0.203 0.060 0.0030.3 0.131 0.034 –0.013Figure 10 Extent of publication bias and the true effect size (ln OR): results of 1000 times simulation (selection based on p values;control rate = 0.20).identified no new studies, although computersimulations were further refined.Small sample sizeStudies with small sample sizes tend to producevariable results and present a range of resultsto select for publication. Simulations havedemonstrated that small sample size is associatedwith considerable publication bias when onlystudies with significant results are published. 338,339In practice, a small study with a non-significantresult may be readily abandoned without trying topublish because it is easy and cheap to carry out interms of time, staff and other resources invested.In addition, small trials may often be poorlydesigned and conducted. Therefore, the risk ofTitle: 01-72-02 Proof Stage: 2Cactus© 2010DesignQueen’sandPrinterIllustrationand ControllerLtdof HMSO. All rights reserved.publication bias will be great if many small trialshave been conducted. 340 However, small trials maystill be helpful in many aspects, and publicationbias should not be considered ‘as a good reason todiscourage trials with low power’. 341Figure 9 shows the results of a stochastic simulationinvestigating the relationship between publicationbias and the range of possible sample sizes. Givena true odds ratio of 0.73 and other conditionsassumed in the simulation, the estimated oddsratio is 0.23 when the sample sizes range from 20to 100, 0.44 when the sample sizes range from 20to 500, and 0.70 when the sample sizes range from20 to 5000. When the possible sample sizes rangefrom 20 to 10,000, the estimated odds ratio is 0.72,nearly identical to the true value of 0.73. Thus,the extent of bias due to selective publication ofFigure Number: 00.10.ai49

Sources of publication biassignificant results is reduced when there are manylarge-scale trials.Small effect sizeThe simulation results also indicated that theextent of bias, by selecting the significant resultsto publish, is greater when the true effect is smallor moderate than when the true effect is zero orlarge. 339 Figure 10 shows the results of a computersimulation about the relation between the trueeffect (log odds ratio) and the extent of bias. Thedifference between the true and the biased effectwas large when the true effect is small comparedwith that when the treatment effect is zero or larger.Therefore, a small or moderate effect (or weakassociation) can be considered as a risk factor forpublication bias. This risk factor may exist in mostcases because clinical trials are mainly designedto assess health-care interventions with small ormoderate (but clinically important) effects.Study design and other qualitycharacteristicsThe design quality of studies may be associatedwith the risk of publication bias. Non-randomisedstudies, single-centre studies, and phase I and IItrials might be more susceptible to publication biasthan randomised studies, multicentre studies andphase III trials. 12,342 Risk factors for publication biaswere assessed but not consistently identified acrossseveral cohort studies of publication bias. 20–22,24Irwig and colleagues 343 suggested that publicationbias is more of a problem for diagnostic tests thanfor randomised trials because ‘many studies oftest accuracy may use data collected primarily aspart of clinical care, there may be no clear recordof attempted evaluations’. It is therefore usefulto estimate how easy it would be for investigatorsto abandon a completed study with unimportantresults without publication, according to somestudy characteristics.SummaryInvestigators, peer-reviewers, editors and fundingbodies may all be responsible for the existence ofpublication bias. The dissemination profile of aresearch finding is determined by the interests ofresearch sponsors, investigators, peer-reviewersand editors. Evidence from newly identified studiesconfirmed findings from the previous HTA reportthat publication bias is often due to investigators’failure to write up and submit. However, itshould be recognised that the investigators’decision to write up an article and then submitit may be affected by pressure from researchsponsors, instruction from journal editors, andrequirements of the research award system. Newlyidentified as well as previous included evidenceindicates that the interests of research sponsors,particularly industry’s commercial interests, canrestrict the dissemination of the research findings.Large differences in likely study results acrosssimilar studies that can be easily conducted andabandoned will further exacerbate the biasedselection of findings for publication.50

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 7Prevention of publication biasMeasures that may prevent publication biasshould be logically designed according to thelikely sources of such bias. Although investigators,peer-reviewers, editors and funding bodies may allbe responsible for the existence of publication bias,the importance of these responsibilities in termsof preventing publication bias may be different.As discussed in Chapter 6, dissemination biasesare related to many complicated factors, andthese factors are inter-related. People’s tendencyto notice only a portion of relevant researchresults has complicated social, cultural, political,economic and psychological bases. In spite of thesedifficulties, it is possible that biased publication ofresearch and the impact of publication bias maybe prevented to a certain extent and its impactminimised. In this chapter, we review measuresthat may help to reduce the existence and impactof publication bias, including changes to thepublication of research, electronic publishing, anopen access policy, the prospective registration ofstudies at inception and confirmatory large-scalestudies.Changes in publicationprocessBecause of the huge number of published studiesand specialist information needs, health-carepractitioners, policy-makers and researchers mustselectively receive information that is perceivedrelevant. At the same time, curiosity about newand atypical events by general readers means thatto maintain a journal’s circulation, editors mayhave to accept studies for publication according toreaders’ preference and type of information thatreaders required.Investigators might not write up and submitstudies with negative results because of anticipatedrejection according to journals’ instructions toauthors and their own experience. The 2000 HTAreport on publication bias listed some measuresthat could reduce publication bias by journals,including accepting manuscripts for publicationmainly based on research protocols, 344 makingprospective registration of trials a precondition fortheir publication, 345 disclosing conflict of interest© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.or competing interests, 280 and electronicallypublishing and archiving research. 346 Earlyinitiatives included that by The Lancet, a generalmedical journal, which in 1997 began to assess andregister selected protocols of randomised trials andsystematic reviews, and to provide a commitmentto publish the main findings of the study. 347 In thesame year, over 100 medical journals around theworld invited readers to send in information onunpublished trials in a so called ‘trial amnesty’. 348Recently, biomedical journals have launchedseveral initiatives and made important progressin the prevention of publication bias. Theinternational guidelines for writing and editingpublications 349 may help to prevent incompleteand biased reporting by the endorsement ofsound reporting guidelines for specific studydesigns, including CONSORT (ConsolidatedStandards of Reporting Trials) for randomisedcontrolled trials, STARD (Statement for ReportingStudies of Diagnostic Accuracy) for studies ofdiagnostic accuracy, QUOROM (Quality ofReporting of Meta-analyses) for systematic reviews,and STROBE (Standards for the Reporting ofObservational Studies in Epidemiology) forobservational studies in epidemiology. 349 To helpeditors, peer-reviewers and authors to ensuretransparent and complete reporting of healthresearch, the EQUATOR (Enhancing the Qualityand Transparency of Health Research) network hasbeen developed. 350–352 Further empirical evidenceis required to indicate the impact of reportingguidelines on reporting bias.Below, we discuss the prevention of publication biasby improved peer review, disclosure of competinginterests, and electronic publication. The role ofmedical journals on the prospective registrationof trials will be discussed later under ‘Prospectiveregistration of trials’.Peer review processJournal peer review has been defined as ‘theassessment by experts (peers) of materialsubmitted for publication in scientific and technicalperiodicals’. 257 The aim of peer review is toimprove the general quality of published studies51

Prevention of publication biasand screen out articles with flawed methodologyor conclusions. 253,353 A large number of studies onthe peer reviewing process are available but few aredirectly relevant to the prevention of publicationbias. The previous HTA report on publication biasincluded some studies showing that the generalquality of peer review had not been improved byblinding peer-reviewers to authors’ identities. 354–357A recently published systematic review of studies oneditorial peer review by Jefferson et al. 358 identified19 comparative studies on editorial peer review.They included nine RCTs that investigated theeffect of blinding on peer review. Of the nine RCTs,five found no effect of blinding on review qualityand four found that blinding affected review qualityalthough these studies highlighted the difficultyof ensuring robust blinding procedures. Theeffect of a submission checklist was investigatedin two studies, with one showing no benefit andthe other showing some benefit (though thisstudy had a small sample size). The limitations ofthis systematic review (as noted by the authors)included atypical settings, involvement of fewmajor journals, small numbers of reviews andreviewers, and methodological weaknesses makingvalidity of the studies reviewed difficult to assess. 358Disclosure of commercialinterestIn order to reduce bias due to research fundingmany journals require authors to disclose their‘conflict of interests’ or ‘competing interests’. 349The updated review identified several studiesof disclosure of commercial interest of authors.Cooper et al. performed a study of thecharacteristics of conflict of interest policies ofbiomedical journals with regards to authors,peer-reviewers and editors using a survey. 359 Theresponse rate for the survey was 67%, with 93%of journals reporting having an author conflictof interest policy and 11% reporting that theyrestricted author submissions based on the conflictof interest policy. However, whilst 77% of journalsreported collecting conflict of interest information,only 57% published author disclosures. Of interest,only 3% of respondents published conflict ofinterest disclosures of peer-reviewers and 12%published editor conflict of interest disclosures. 359Conversely, a study by Cain et al. showed thatdisclosure may exacerbate bias rather thanprevent it with the possibility of a conflict ofinterest statement subconsciously absolving theauthor of responsibility. 360 Whilst a conflict ofinterest statement may appear transparent inacknowledging bias, it is insufficient to prevent it. 361Electronic publicationThe volume of electronic publishing hasgreatly increased due to its advantages of rapidpublication, no limit in length of articles, no limitin numbers of studies, interconnected articles,cost-effective dissemination, and cost-effectivearchiving. 362–365 Because of reduced or no spacelimitations, electronic publishing may reducepublication and related biases by publishingresearch protocols and by allowing studies to bejudged on their design and methodology ratherthan the immediate relevance of findings to currentpractice, novelty or exciting results. 366Publication of research protocols prior to studycompletion has been recommended as a measure toprevent poor medical research. 346 The developmentof electronic publishing has provided greatpotential for the publication of research protocols.Any discrepancies between the research protocolsand published studies will become transparent andoutcome reporting bias may be prevented by thepublication of protocols. 346Electronic journals with no space limitations mayencourage publication of studies with negativeor no significant results as well as those thatreplicate previous studies. 366 Peer review in thecontext of electronic publishing is still importantto ensure quality, but this could also be publishedin conjunction with the article. For example, onlineBioMed journals publish any accepted papers withtheir initially submitted versions and commentsfrom peer-reviewers plus responses from authors tothese comments.There are two forms of electronic publishing:printed journals with electronic supplementarymaterials, and electronic only journals. Sim andRennels have suggested using the Trial BankModel to publish traditional prose form studiesand a concurrent electronic database of additionaldata. 367 This model has been adopted by the BMJusing the ‘electronic long–paper short’ (ELPS)mode of publishing. 368 Medical journals have twobasic functions: medical recorder and medicalnewspaper. 369 The ELPS model, and the webbasedsupplementary material model, seem logicalchoices for these two different but related basicfunctions.52

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8As a specific remedy for publication bias, theJournal of Negative Results in Biomedicine, an openaccess online journal, was introduced in 2002 witha remit to publish studies with negative results. 370This journal is indexed by MEDLINE, EMBASE,Scopus and Google Scholar, making retrieval ofthe negative results it publishes more likely duringsystematic review. From inception to August 2008 ithad published 60 articles.The recent development of electronic publishinghas provided great opportunities for preventingpublication and related biases, but we have foundlittle direct evidence of how effective they arein practice. Electronic publishing itself won’t beable to resolve biased publication and reportingof research results. Many online open accessjournals [including BioMed and Public Library ofScience (PLoS)] charge authors a fee to cover thepublishing costs. It is still unclear what impactthis pay-to-publish model has on publication andrelated biases and the general quality of studiespublished in these open access electronic journals.Prospective registration oftrialsIn 1997, over 100 medical journals around theworld invited readers to send in information onunpublished trials in a so called ‘trial amnesty’. 348This was a request to retrospectively registerconducted trials, even where outcome datawere not provided, to enable other researchers,specifically systematic reviewers, to know of theexistence of the study and write to the trialists forfurther details. One year after its launching, only165 trials were registered 371 and it was considereda failure by 2004. 372 This failure of retrospectivelyregistering unpublished results led to furthersupport for the development of prospectiveregistration of studies.Accepting studies for publication based mainlyon their pre-submitted research protocol couldhelp to reduce publication bias by ensuring thatthe publication maintains its pre-stated primaryoutcomes, and is published regardless of whetherthat primary outcome shows a statisticallysignificant effect. In 1997, a general medicaljournal, The Lancet, began assessing and registeringselected protocols of randomised trials andsystematic reviews, and providing a commitmentto send for peer review the main clinical findingsof the study (there is currently no commitmentto publish a final paper) 347 (see also http://www.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.thelancet.com/journals/lancet/misc/protocol). Only75 protocols had been accepted and registered inThe Lancet by June 2007, 373 and up to the time ofwriting (August 2008) fewer than 100 protocolshad been registered in this way; other journals areyet to follow suit. Registration of study protocolsby paper journals, although a good idea, is clearlynot going to help to prevent publication bias in thebulk of research, and it is still feasible for a study tobe registered with The Lancet, but not published bythem if the results are not deemed appropriate forwhatever reason.Prepublication of protocols has been recommendedas an important measure to prevent poor medicalresearch. 346 Electronic publication of systematicreview protocols prior to study completion hasbecome part of the online-only Cochrane Database ofSystematic Reviews, where all accepted reviews havetheir protocols peer reviewed and published beforethe completed review is published. This allowsfor feedback by any reader on the methodologyor question addressed and prevents duplicationof effort in defining research that is about tobe undertaken. The development of electronicpublishing has provided great potential for thepublication of research protocols. Theoreticallythis ensures that any discrepancies between theresearch protocols and published studies becometransparent and outcome reporting bias may beprevented. 346Boissel et al. defined a clinical trial registry ‘as adatabase of planned, ongoing or completed clinicaltrials, published as well as unpublished, in whichdetails concerning the trial’s objectives, maindesign features, sample size, and tested treatmentare stored’. 374 It has been generally accepted thatprospective registration of trials at their inceptionmay prevent publication bias. 103 Even if not alltrials are registered, a prospective registration ofsome trials may provide an unbiased sample of allstudies that have been conducted. 1 For example,the International Cancer Research Data Bank wasused to assess alkylating agent therapy in advancedovarian cancer. 127The Clinical Trials Registry of the InternationalCommittee on Thrombosis and Haemostasis,established in 1974, may be the first registry ofclinical trials. 375 In 1988, Easterbrook identified24 registries of clinical trials. 376 Clinical trialsincluded in these registries were prospectively orretrospectively identified by surveying selectedindividuals, organisations, pharmaceuticalcompanies or other industries; from conferences53

Prevention of publication bias54and selected journals; searching other relatedregistries of trials; and by funding bodies orresearch ethic committees. 376 Currently, thetwo most important registries of clinical trialsare the International Standard RandomisedControlled Trial Number (ISRCTN) register andClinicalTrials.gov. The ISRCTN (available athttp://www.Controlled-trials.com) was launchedin 2000 by the publisher Current Science Groupand its ownership was transferred to a not-forprofitentity in 2005. 377 ClinicalTrials.gov (at www.ClinicalTrials.gov) was developed in 2004 by theNational Library of Medicine. As of August 2008ISRCTN included over 7000 trial registrations,while ClinicalTrials.gov had registered almost60,000. Other important trial registries include theAustralian Clinical Trials Registry, the NetherlandsTrial Registry, and UMIN (University HospitalMedical Information Network) Clinical TrialsRegistry. 378Voluntary registration of clinical trials isoften incomplete and a mandatory system bygovernment regulation may be necessary. 379,380Spain’s Royal Decree of 1978 and a MinisterialOrder of 1982 established a register of clinicaltrials. 381 The FDA Modernisation Act of 1997 in theUSA also requests the establishment of a federallyfunded database containing information on bothgovernment-funded and privately funded trials ofdrugs designed to treat serious or life-threateningconditions. 382The World Health Organization (WHO) hasrecognised the importance of trial registrationand initiated a project in 2005 to set internationalstandards for clinical trial registration. 383 TheWHO Registry Network provides prospective trialregistries with a forum to exchange informationand work together to establish best practicefor clinical trial registration. By establishinginternational standards, the WHO Trial RegistryPortal will help to prevent selective disseminationof trial information by specific trial registers. TheClinical Trials Search Portal (CTSP) could besearched online by users to identify registered trialsincluded in WHO’s trial registration database. 384The most influential initiative is the introductionof a trials registration policy by the InternationalCommittee of Medical Journal Editors (ICMJE) in2004. 10 Since 2005, as a condition of considerationfor publication in ICMJE member journals, trialsmust register at or before the onset of patientenrolment in a registry that is accessible to thepublic at no charge, open to all prospectiveregistrations, and managed by a not-for-profitorganisation. 10 A similar compulsory registrationof clinical trials is also required by BMJ, withmodified criteria for suitable registries. 372 Thepolicy of compulsory trial registration required byjournals has greatly increased the number of trialsregistered. 385 The number of trials registered inthe ClinicalTrials.gov database increased by morethan 70% between May and October 2005, after theimplementation of ICMJE’s policy. 386Even prospective registration of trials may not besufficient to prevent all biases in the publicationof trial results, including outcome reporting bias.The registration of all trial results (published andunpublished) has been advocated. 387,388 Althoughthere is still disagreement about the registrationof trial results, 378 publicly available full summaryresults from trials will help to reduce publicationbias and outcome reporting bias. In addition,the prospective registration of research hascurrently focused mainly on phase III and phaseIV trials. Biased selection for publication of earlystage trials and non-trial studies (such as basicresearch, observational studies and studies ofdiagnostic accuracy) is still far from being resolved.For example, biased publication of early stagephase I studies may increase the failure rate ofphase II studies. 389 Choi et al. suggested a globalregistry of anticipated public health studies.However, the establishment and maintenance of acomprehensive registration of non-trial studies willneed to overcome more difficulties. 390Trials registries will only be helpful in reducingpublication bias if systematic reviewers includethe registries in their search strategies and resultsof trials are accessible. Ramsey and Scogginsidentified 2028 completed or terminated cancertrials from NIH’s ClinicalTrials.gov registryin September 2007. 391 They then searchedClinicalTrials.gov and PubMed for peer-reviewedpublications of these trials. It was found thatonly 19.5% of the 1791 completed cancer trialsand 3.4% of the 237 terminated trials had beenpublished in peer-reviewed journals. 391Open access policySince 1997, the practice of incomplete release ofinformation about licensed drugs in Europe forreasons of commercial interests and intellectualproperty has been challenged. 392–394 Abrahamand Lewis 392 suggested that ‘the secrecy andconfidentiality of EU medicines regulation is not

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8essential for a viable pharmaceutical industry’,considering that European pharmaceuticalcompanies often obtain data on competitors’products by using the US Freedom of InformationAct. There were already ‘encouraging signs’ in 1998within the pharmaceutical industry to improvepublic access to the findings of clinical studies thatthe industry sponsored. 394 However, there weresetbacks in transparent reporting of clinical trialssponsored by the pharmaceutical industry untilseveral recent high-profile cases of incompletereporting of industry sponsored trials. 380Recently, policies of mandatory open access tothe results of studies they sponsored have beenadopted by many important research fundingbodies, including the UK Medical Research Council(MRC), the Wellcome Foundation, the EuropeanResearch Council, the Canadian Institutes ofHealth Research, and the National Institutesof Health (NIH) in the USA. 395,396 However, thecurrent policies of open access focus on theopenness of results of non-industry-sponsoredstudies, and it cannot prevent biased publicationand reporting of results from industry-sponsoredresearch.There is no convincing evidence indicating thatonline open access to published studies increasesthe number of their citations. 397 A study found thatthe possibility of an article being found on a nonpublisherwebsite was higher for articles publishedin journals with higher impact factors. 398 Therefore,self-archiving for open access is likely to be selectiveand may be biased. 399Right to publicationSome high-profile cases of publication biasreviewed in Chapter 6 were due to suppressionby the industry of publication of negative resultsfrom industry-sponsored research. In 1997, apharmaceutical company changed its policy aboutdissemination of research it sponsored, allowinginvestigators ‘to publish studies conductedunder generally accepted standards of scientificrigour without company prior approval’, subjectto the ‘right to review prepublication drafts toaddress intellectual property issues’. 400 To preventpublication bias due to industry suppression,Rennie recommended that investigators ‘shouldnever allow sponsors veto power’, 276 and Rosenbergsuggested that scientists should refuse ‘to keepinformation confidential and refuse to signany agreements for the transfer of information© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.or reagent that included a requirement ofconfidentiality. 275 A recently published study foundthat standards for clinical trial agreements withindustry varied considerably among 107 academicmedical centres, and subsequent disputes onintellectual property (30%) and control of or accessto data (17%) were common. 401Research sponsors’guidelinesFunders of clinical research often requireinvestigators of sponsored studies to followresearch guidelines. Influential researchguidelines include International Conference onHarmonisation (ICH Topic E6), the EU ClinicalTrial Directive, the Declaration of Helsinki, andthe CONSORT Statement. 41 Dwan et al. surveyedresearch guidelines issued by 17 organisationsand 56 charities that funded clinical trials. 41 Theyfound that 11 of these organisations or charitiesemphasise the publication of both positive andnegative results, and three request the adherenceto trial protocols in data analysis and that anychanges need to be explicitly reported. It wasconcluded that research funders’ guidelines shouldbe improved to prevent selective reporting ofoutcomes. 41Confirmatory large-scaletrialsFor the purpose of avoiding moderate biases andmoderate random errors in assessing or refutingmoderate benefits, a large number of patientsin randomised controlled trials are required. 402Large-scale trials are generally believed to beless vulnerable to publication bias; this is thefundamental assumption of many methods fordetecting publication bias. When the existence ofpublication bias is likely and the consequences ofsuch bias are clinically important, a confirmatory,multicentre, large-scale trial may be conducted toprovide more convincing evidence. The updatedreview included no new studies for this section.Disagreements in results between large-scale trialsand corresponding meta-analyses of small trialswere observed in empirical studies. 403–405 Althoughsmall trials tend to lack statistical power and bemore vulnerable to publication bias, the systematicreview of small studies may provide usefulinformation about whether a confirmatory largestudy is required and how to design such a study. 40655

Prevention of publication biasLarge-scale confirmatory trials become necessaryafter a systematic review has reported a clinicallysignificant finding but publication bias cannotbe safely excluded as an alternative explanation.Confirmatory large trials are still important evenwhen prospective registries of trials are available.This is because publication bias is only one ofmany potential threats to trial validity. Comparedwith a universal register of all trials, confirmatorylarge trials are more selective about the researchareas and objectives, but more flexible at the sametime to minimise the impact of other biases, forexample, biases related to study design, selection ofcontrol, participants and setting.SummaryThe first step for the prevention of publicationbias is a wide public awareness of detrimentalconsequences of publication bias, and the needfor the results of all studies to be made accessible.Important actions by government, journals andresearch sponsors have been taken after severalhigh-profile cases of incomplete disclosure of trialresults by pharmaceutical companies were recentlybrought to light.Changes in editorial policy, the peer reviewprocess, disclosure of commercial interest,electronic publication, trial registration, and openaccess policy may all help to prevent publicationand related biases, although there is as yet littledirect evidence as to how well they work in practice.The recent development of electronic publicationprovides great technical potential to overcomesome limitations of conventional printed journals.Publication and related biases may be reduced byelectronic publication because of unlimited space,linkage between references, timely publication, andcost-effective dissemination and archiving.One important solution to publication bias maybe the prospective registration of all studies atinception. Voluntary registration of clinical trialsis usually incomplete. The policy of compulsorytrial registration adopted by the InternationalCommittee of Medical Journals in 2004 may be themost influential initiative to promote prospectiveregistration of clinical trials. Further mandatorygovernment regulations may still be required.The development of prospective trial registrationitself is not sufficient for the prevention ofpublication bias. It is important to make sure thatresults of registered trials are publically accessible.The usefulness of trial registrations relies onsystematic reviewers searching them, using the datathey provide and spending time contacting trialistswhere studies have not yet been published.Successful efforts so far have focused on biasedreporting of phase III and phase IV trials,because of their immediate health consequences.Prospective registration of basic laboratoryresearch, early stage clinical studies andobservational studies is still underdeveloped. Openaccess policy is often mandatory only to public- orcharity-supported research. Therefore, althoughpublication bias might be reduced it could still bea problem in many fields of biomedical and healthresearch.56

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 8Reducing or detecting publication andrelated biases in systematic reviewsMethods that could be useful to reduce ordetect publication bias in systematic reviews arediscussed in this chapter. Literature searching,locating unpublished trials and assessment ofthe risk of publication bias will be discussed first.Then methods designed to be used in metaanalysisare discussed, including funnel plotand related statistical methods, fail-safe N, andmore sophisticated modelling methods. Finally,the importance of updating systematic reviews isdiscussed.Literature searchingIf time and resources were unlimited, it is possiblethat a literature search could identify all publishedstudies relevant to a particular review question.In the real world, a balance must instead bestruck between sensitivity (the number of relevantstudies identified as a proportion of the totalnumber in existence) and precision (the number ofrelevant studies identified as a proportion of thetotal number retrieved). 2 These two parameterstend to be inversely correlated, such that effortexpended on increasing sensitivity is costly interms of retrieving non-eligible studies, whichmust subsequently be excluded on an individualbasis. Bennett has developed methodology toassess the number of studies potentially missed in asystematic review. 407Despite the need to work within a realisticframework, two main issues at this stage of areview can create significant bias in any subsequentanalysis: limited exploitation of searchingmodalities, and low sensitivity within electronicsearch strategies.Limited exploitation of searchingmodalitiesResearch literature exists in a number of differentformats, such as peer-reviewed material publishedin academic journals, conference abstracts(which may or may not be peer reviewed), otherforms of grey literature and personal records.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.To carry out an effective systematic review, it isnecessary to devise a literature search that can takeaccount of this diversity. 204 Systematic reviewershave a range of searching modalities at theirdisposal (e.g. electronic bibliographic databases,citation tracking, hand-searching of journalsand bibliographies, contacting experts, etc.),but sometimes only one or two are exploited. Inparticular, search strategies sometimes concentratealmost exclusively on electronic bibliographicdatabases. This approach may lead to biasedsearching. For example, studies reporting nonsignificantresults may be overlooked if they tendto be consigned to lower profile journals or othersources that are poorly indexed (or not indexedat all). 408–410 Hand-searching initiatives are oneof the few means of addressing the issue of poorindexing. 408,411 Such work may also tend to appearmore often as grey literature or other unpublishedmaterial (due to publication bias), in which case itmay again be overlooked if the search is restrictedto standard bibliographic databases.In addition, reliance on a standard protocol-drivenstrategy (e.g. prespecified bibliographic databasesearches supplemented by limited hand-searching)may be associated with the false impression thata search is necessarily exhaustive. Greenhalghand Peacock recently showed that only 30% of thestudies eligible for their review could be obtainedfrom a purely protocol-driven search, comparedwith the additional inclusions identified through‘snow-balling’ (reference and citation tracking) anddrawing on personal knowledge (both within andbeyond the research team). 412Low sensitivity within electronicsearch strategiesSome degree of bias is clearly possible whena search is based exclusively on bibliographicdatabases, but in some cases the effects may betoo small to influence the results of a systematicreview; the power of contemporary searchplatforms to scan the vast numbers of recordsheld in such databases also underlines the valueof electronic searches. However, further bias may57

Prevention of publication biasarise if searches are designed without adequatesensitivity. At a simple level, searches are sometimesonly applied to a single database (generallyMEDLINE), or a limited set of databases, eventhough individual databases may only havepartial coverage of the journals and other sourcesholding relevant studies. 413 Search results will thennecessarily be constrained by the same criteria usedto identify sources for inclusion on the databasesthemselves. The incorporation of excessivelyspecific search terms (e.g. methodological terms)may also tend to reduce (rather than increase)sensitivity if the terms are used to limit (ratherthan expand) search results. Sensitivity may insteadbe maximised by favouring relatively genericterms, 414,415 albeit potentially at the cost of somereduction in precision. Similarly, the difficultiesnoted above in terms of the quality of databaseindexing 408 necessitate designing electronicstrategies that combine text and index termseffectively. Language restrictions (most commonlyto English) may also greatly reduce sensitivitywhere they are applied, depending on the amountof research published in the excluded languages.This may be a further source of bias wherestudy outcome is linked to publication language(particularly biased publication of non-significantresults in non-English-language journals): althoughsome evidence indicates the effects on metaanalysesmay be small, comprehensive searchingwithout language restriction remains an importantsafeguard. 153,416,417In addressing a particular research question, itis important that systematic reviewers attemptto reduce the potential impact of disseminationbias on subsequent analyses. Accepting thatthere is a trade-off between search sensitivity andprecision, 2 literature searches should thereforedraw on as many different searching modalitiesas are necessary to identify relevant studies. Inparticular, searches may need to be extendedbeyond standard bibliographic databases toidentify material from, for example, conferenceabstracts, other forms of grey literature (such asreports by companies, governments or regulatorybodies), and personal or research group data(which may only be disseminated locally). 204,409Electronic search strategies should also be designedto ensure that the desired level of sensitivity isachieved (thus reducing biases associated withparticular databases, search terms, languagerestrictions and so on), for example by developingsensitive search filters. 418 However, it is clear thatreviewers face particular obstacles in attempting tosearch comprehensively, in relation to both the greyliterature, and ongoing/unpublished trials.Grey literature and non-EnglishlanguagestudiesThe ease with which conference abstracts can beidentified is not always clear, because researchorganisations and societies may have differentpolicies on publishing this material as officialproceedings (i.e. in society journals). 419 In manycases, conference proceedings are well indexedon databases such as MEDLINE, the CochraneLibrary, and possibly SIGLE (System forInformation on Grey Literature in Europe), butit may be necessary to determine the best meansof identifying abstracts in each individual casebefore a full search is performed. For example,reviewers could check with relevant societiesand organisations to determine how conferenceabstracts are processed and design search strategiesaccordingly. Company and regulatory authorityreports can be an additional source of usefulunpublished data. For example, FDA reviewswere used in one study to identify 10 unpublishedFDA trial reports. 135 Although assessment ofmethodological quality was problematic due topoorer reporting, including this unpublished workdid not appear to confound effect estimates byintroducing ‘small study bias’. 420Even well-designed and non-language-restrictedsearches run on MEDLINE and EMBASE maymiss a large number of non-English-languagestudies. One potential solution to this is tosearch the Cochrane Library. The CochraneCollaboration has an extensive programme ofhand-searching that covers a wide range of journalsto ensure that controlled trials from a wide rangeof sources (including non-English-languagejournals and conference abstracts) are identifiedand correctly indexed. Additionally searching oflanguage-specific databases (such as LILACS –Latin American and Caribbean Health SciencesLiterature) may be appropriate, but relies on thereviewer having some knowledge of the relevantlanguage(s) to ensure that the correct terms areentered.Locating ongoing orunpublished trialsOngoing trials can be defined as any trials thathave started but where the results are not yet58

Prevention of publication bias60Funnel plot and relatedstatistical methodsBecause of a larger random error, the results fromsmaller studies will be more widely spread aroundthe average effect as compared with the resultsfrom larger studies. A plot of sample size versustreatment effect from individual studies in a metaanalysisshould be shaped like a funnel if there isno publication bias or small-study effects. 441 To helpthe interpretation of funnel plot asymmetry, Peterset al. have recently proposed contour-enhancedfunnel plots by adding contour lines indicatingconventional milestones in levels of statisticalsignificance (e.g. for p < 0.01, p < 0.05). 442 Acontour-enhanced funnel plot makes it possible toascertain whether areas where studies are perceivedto be missing are in areas of statistical significanceand hence whether publication bias is the likelycause of the asymmetry.If the chance of publication is greater for trialswith statistically significant results, the shape ofthe funnel plot may become skewed. In a funnelplot, the treatment effects from individual studiesare often plotted against their standard errors (orthe inverse of the standard errors) instead of thecorresponding sample sizes (Figure 11). The use ofstandard errors has some advantages because thestatistical significance is determined not only by thesample size but also by the level of variation in theoutcome measured, or the number of events in thecase of categorical data. 443Light and Pillemer described two ways in whichthe shape of the funnel plot can be modified whenstudies with statistically significant results are morelikely to be published. 441 Firstly, assume that thetrue treatment effect is zero. Then the results ofsmall studies can be statistically significant onlywhen they are far away from zero, either positiveor negative. If studies with significant results arepublished and studies with results around zeroare not published, the funnel plot may not beobviously skewed but there will be an empty areaaround zero (see Figure 11, 1-B). These polarisedresults (significant negative or positive results) maycause many debates; however, the overall estimateobtained by combining all studies is unlikely to bebiased.Secondly, when the true treatment effect is smallor moderate but not zero, small studies reportinga small effect size will not be statistically significantand therefore are less likely to be published, whilesmall studies reporting a large effect size maybe statistically significant and more likely to bepublished. Consequently, there will be a lack ofsmall studies with small effect in the funnel plot,and the funnel plot will be skewed with a largereffect among smaller studies and a smaller effectamong larger studies (see Figure 11, 2-B). This willresult in an overestimation of the treatment effectin a meta-analysis.The selection of a study for publication may be afunction of many variables, such as sample size,level of statistical significance, extent or directionof difference between comparison groups, anddesign quality of a study. If the publication of astudy is associated with the direction of the results,the extent of publication bias may be much greaterthan when publication is associated with only thelevel of statistical significance. Figure 11, 1-C and1-D are the funnel plots in which the selection isa function of statistical significance and samplesize when the true treatment effect is zero. Figure11, 2-A to 2-D show the funnel plots of the resultsof computer simulation under different selectionassumptions when there is a small treatment effect(true odds ratio 0.8).Limitations of funnel plot fordetecting publication biasFor a funnel plot to be useful there needs to be arange of studies with varying sizes. The funnel plotis an informal and subjective method for assessingpotential small-study effect; different people mayinterpret the same plot differently. The visualimpression of a funnel plot may change dependingon which measure of trial magnitude (SE, variance,sample size, etc.) or which outcome scale (e.g. riskdifference, relative risk, odds ratio) is used. 444It should be stressed that a skewed funnel plot maybe caused by factors other than publication bias.Possible sources of asymmetry include differentintensity of intervention, differences in underlyingrisk, poor methodological design of small studies,inadequate analysis, fraud, choice of effectmeasure, and chance. 149 Clinical heterogeneity as asource of funnel plot asymmetry can be illustratedusing data adopted from the meta-analysis byHofmeyr et al. of calcium supplementation inpregnancy for the prevention of pre-eclampsia. 445The funnel plot (Figure 12) is visually asymmetric,showing a tendency for large trials to be associatedwith a smaller treatment effect. However,it has been noted that the effect of calciumsupplementation was greater for women with ahigh baseline risk of hypertension (Figure 13). 445

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 81-A: True OR = 1, no bias651-B: True OR = 1, selection p < 0.05651/SE431/SE4321–2.0 –1.0 0.0 1.0 2.0In OR21–2.0 –1.0 0.0 1.0 2.0In OR1-C: True OR = 1, selection p < 0.05 and n651-D: True OR = 1, selection p < 1.96 and n651/SE431/SE4321–2.0 –1.0 0.0 1.0 2.0In OR21–2.0 –1.0 0.0 1.0 2.0In OR2-A: True OR = 0.8, no bias652-B: True OR = 0.8, selection p < 0.05651/SE431/SE4321–2.0 –1.0 0.0 1.0 2.0In OR21–2.0 –1.0 0.0 1.0 2.0In OR2-C: True OR = 0.8, selection p < 0.05 and n652-D: True OR = 0.8, selection z < –1.96 and n651/SE431/SE4321–2.0 –1.0 0.0 1.0 2.0In OR21–2.0 –1.0 0.0 1.0 2.0In ORFigure 11 Funnel plots of the results of computer simulations, under different assumptions about biased selection.61© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Title: 06-92-02 Proof Stage: 3Figure Number: 00.11.aiCactus Design and Illustration Ltd

Prevention of publication bias00.5SubgroupsLow hypertension-risk womenHigh hypertension-risk womenSE [log(OR)]11.520.005 0.1 1 10 200Odds ratioFigure 12 Funnel plot of odds ratio (on log scale) against inverse of standard error. Meta-analysis of calcium supplementation for theprevention of pre-eclampsia. Results of funnel plot asymmetry: Begg’s test p = 0.537; Egger’s test p < 0.0001; Peters’ test: p = 0.002.Study orsubgroupCalciumPiaceboEvents Total Events TotalWeightOdds ratioM-H, Random,95% CIOdds ratioM-H, Random, 95% CI1.2.1 Low hypertension-risk womenBelizan 1991 446 15 579 23 588 13.5% 0.65 (0.34 to 1.27)CPEP 1997 447 158 2163 168 2173 19.2% 0.94 (0.75 to 1.18)Crowther 1999 448 10 227 23 229 12.1% 0.41 (0.19 to 0.89)Lopez-Jaramillo 1989 449 2 55 12 51 5.4% 0.12 (0.03 to 0.58)Purwar 1996 450 2 97 11 93 5.5% 0.16 (0.03 to 0.73)Villar 1987 451 1 25 3 27 2.8% 0.33 (0.03 to 3.44)WHO 2006 452 171 4151 186 4161 19.3% 0.92 (0.74 to 1.14)Subtotal (95% CI) 7297 7322 77.7% 0.65 (0.45 to 0.93)Total events 359 426Heterogeneity: τ 2 = 0.10; χ 2 = 16.61, df = 6 (p = 0.01); I 2 = 64%Test for overall effect: z = 2.38 (p = 0.02)1.2.2 High hypertension-risk womenLopez-Jaramillo 1990 453 0 22 8 34 1.9% 0.07 (0.00 to 1.27)Lopez-Jaramillo 1997 454 4 125 21 135 8.5% 0.18 (0.06 to 0.54)Niromanesh 2001 455 1 15 7 15 2.9% 0.08 (0.01 to 0.79)Sanchez-Ramos 1994 456 4 29 15 34 7.2% 0.20 (0.06 to 0.71)Villar 1990 457 0 90 3 88 1.8% 0.13 (0.01 to 2.65)Total (95% CI) 281 306 22.3% 0.16 (0.08 to 0.33)Total events 9 54Heterogeneity: τ 2 = 0.00; χ 2 = 0.86, df = 4 (p = 0.93); I 2 = 0%Test for overall effect: z = 4.94 (p < 0.00001)Total (95% CI) 7578 7628 100.0% 0.42 (0.28 to 0.65)Total events 368 480Heterogeneity: τ 2 = 0.23; χ 2 = 37.36, df = 11 (p = 0.0001); I 2 = 71%Test for overall effect: z = 4.01 (p < 0.0001)62Title: 06-92-02 Proof Stage: 2Cactus Design and Illustration LtdFigure0.005Number:0.100.12ai1 10 200Favours treatment Favours controlFigure 13 Calcium supplementation in pregnant women for preventing pre-eclampsia: meta-analysis of 12 randomised controlled trials.Data from Hofmeyr et al. 444 Copyright Cochrane Collaboration, reproduced with permission.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Pregnant women included in smaller trials tendedto have a higher baseline risk of hypertension, ascompared with those in larger trials. Therefore, thediscrepancy in results between smaller and largertrials in this case may be partially explained by thedifferent baseline risk of hypertension and otherpatient characteristics (dietary calcium intake). 445Terrin et al. asked 41 medical researchers visuallyto assess funnel plots of simulated meta-analyses(each included 10 trials). 458 They found that 44%of the funnel plots showed moderate to veryhigh asymmetry when publication bias did notexist, whereas 34% of the funnel plots showed noclear asymmetry even when publication bias didexist. That is, the shape of a funnel plot may beasymmetric purely by chance without selection bias,and a symmetrical funnel plot cannot exclude theexistence of publication bias. It was concluded that‘researchers who assess for publication bias usingthe funnel plot may be misled by its shape’. 458Statistical tests for funnel plotasymmetryIt is often difficult for people to decide visuallywhether a funnel plot is asymmetrical or not.Therefore, some statistical methods have beendeveloped to examine formally the skewness of afunnel plot. The 2000 HTA report on publicationbias 2 included two methods for testing funnelplot asymmetry: the rank correlation test by Beggand Mazumdar (1994) 340 and a linear regressionmethod by Egger et al. (1997). 149 Since then,some new or modified tests have been proposed,all designed to test whether studies with smallersample size or greater variation in results tend toreport greater treatment effects in meta-analysis(Table 2). 8,9,149,340,444,459–462 Almost simultaneously,many recent studies have been conducted tocompare the performance of these tests. 8,9,459–468Different tests often lead to different conclusionsin terms of the funnel plot asymmetry. All theproposed tests have some important limitations,including low statistical power to identify funnelplot asymmetry when it exists, and inflated ratesof type I error when funnel plot asymmetry doesnot exist. The performance of tests for funnel plotasymmetry is particularly poor when heterogeneityin meta-analysis is large. 469Ioannidis and Trikalinos (2007) 470 suggested thatstatistical tests for funnel plot asymmetry may beappropriate only if the following four criteria aremet:• no significant heterogeneity• I 2 < 50%• 10 or more studies (with statistically significantresults in at least one study)• ratio of the maximal to minimal variance acrossstudies > 4.A survey of 846 independent meta-analyses fromthe Cochrane Database of Systematic Reviews foundthat only 12% of the meta-analyses met all theabove four criteria. 470 The number of studies wasfewer than 10 in 74% of the meta-analyses andnone of the studies was statistically significantin 34% of the meta-analyses. About 30% ofthe meta-analyses had statistically significantheterogeneity. 470 However, it should be noted thatthe above criteria were not based on convincingempirical evidence and further simulation studiesmay help to investigate how valid these criteria are.In the recently updated Cochrane Handbook forSystematic Reviews of Interventions, 469 Sterne et al.have provided some recommendations about theuse of statistical tests for funnel plot asymmetry.The main points of their recommendations aresummarised below.• The tests for funnel plot asymmetry should notbe used in meta-analyses that include less than10 studies.• To test funnel plot asymmetry, studies includedin a meta-analysis should have different sizes,although it is not clear when the difference instudy sizes are sufficient.• Egger’s test can be used for continuousoutcomes measured by mean differences.• For dichotomous outcomes measured by oddsratio, tests proposed by Harbord et al., 459 Peterset al. 9 or Rucker et al. 461 can be used in theabsence of significant heterogeneity (ι 2 < 0.1).• Arcsine random-effect regression test 461 shouldbe used when both treatment effect andheterogeneity are large (e.g. ι 2 > 0.1).• Funnel plot testing strategy should be specifiedin advance, and only one test should be used.Other tests for funnel plot asymmetry includedin Table 2 are not recommended in the updatedCochrane Handbook for Systematic Reviews 469 mainlybecause of low power 8,340,460,462 or lack of statisticalevaluation. 444 There is very limited empiricalevidence on the performance of the available testsfor dichotomous outcomes measured as risk ratiosor risk differences, and for continuous outcomesmeasured by standardised mean differences. 46963© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Prevention of publication bias64Trim and fill methodThe trim and fill method is a rank-based dataaugmentation technique, designed to estimatethe number of missing studies and to provide anestimate of the treatment effect by adjusting forfunnel plot asymmetry in a meta-analysis. 471,472Briefly, the asymmetrical outlying part of thefunnel is firstly ‘trimmed off ’ after estimating howmany studies are in the asymmetrical part. Thenthe symmetrical remainder is used to estimate the‘true’ centre of the funnel. Finally, the ‘true’ meanand its variance are estimated based on the ‘filled’funnel plot in which the trimmed studies and theirmissing ‘counterparts’ symmetrical about the centreare replaced. 471An early simulation study reported that the trimand fill method estimates the point estimate ofthe overall effect size approximately correctlyand the coverage of the confidence interval issubstantially improved, compared with ignoringpublication bias. 471 However, further simulationstudies found that the trim and fill method mayperform poorly with high false-positive findingswhen heterogeneity in meta-analysis is large. 473,474Because of the existence of clinical heterogeneity,as with all other funnel plot-based tests, the trimand fill method may provide a misleading estimateby spuriously adjusting for bias that actually doesnot exist.Other statistical andmodelling methodsFail-safe N methodsSeveral methods have been proposed to estimatethe number of unpublished studies in a metaanalysis.11,475–479 The first and most commonly usedis Rosenthal’s fail-safe N, designed to estimate thenumber of unpublished studies required, with zeroeffect on average (z = 0), to overturn a significantresult (p < 0.05) in a meta-analysis. 11 If the numberof unpublished studies with null results requiredto overturn the statistically significant result islarge, and therefore unlikely to exist, the impact ofpublication bias is considered to be ignorable andthus the results obtained from published studies areheld to be robust.The plausible number of unpublished studiesmay be hundreds in some areas or only a few inothers. Therefore, the estimated fail-safe N shouldbe considered in proportion to the number ofpublished studies (K). Rosenthal suggested that thefail-safe N may be considered as being unlikely if itis greater than a tolerance level of ‘5K + 10’. 11There are problems with the fail-safe N method. 480Firstly, the method overemphasises the importanceof statistical significance. Secondly, it may bemisleading when the unpublished studies havean average effect that is in the opposite directionto the observed meta-analysis. If the unpublishedstudies reported contrary results comparedwith those in the published studies, the numberof unpublished studies required to overturn asignificant result would be smaller than thatestimated, assuming an average effect of zero inunpublished studies. In addition, the interpretationof estimated fail-safe N may be misleading becauseit is often difficult to decide the plausible numberof unpublished studies. Becker has suggested that‘the failsafe N should be abandoned in favour ofother more informative analyses.’ 481Recently, a weight function method of sensitivityanalysis proposed by Copas and Jackson 482 hasbeen used to estimate a range of possible numbersof unpublished studies in a meta-analysis. Thismethod is discussed below.Sophisticated modellingmethodsThe impact of missing studies may also be assessedby using more sophisticated modelling methods.Many of the sophisticated modelling methods werediscussed in depth in the 2000 HTA report 2 and ina review article by Sutton et al. 483 These methodsare usually based on weighted distribution theoryderived from both classical 338,484–492 or Bayesian 493–497perspectives. There are two aspects to the selectionmodels that use weighted distribution theory:an effect size model, which specifies what thedistribution of the effect size estimate would be ifthere were no selection, and the selection model,which specifies how this effect size distributionis modified by the selection process. 498 In somemethods the nature of the selection process ispredefined by the researcher, while in others it isdictated by the available data.The appropriate application of modelling methodsto test publication bias usually requires a largenumber of studies (e.g. 100 or more) in a metaanalysis.499 However, the number of studies wasfewer than 10 in most published meta-analyses. 470In addition, it is difficult if not impossibleempirically to verify the validity of assumedselection processes, since the true mechanismsand extent of biased publication or reportingare usually unknown. 482 Therefore, it has been

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8recognised that weight function models should beused to conduct sensitivity analyses, rather than toprovide a single ‘correct’ estimate by adjusting forthe assumed selection bias. 482,499A sensitivity analysis method using weightfunction for assessing the impact of publicationbias has been proposed by Copas et al. 482,489,500The probability of study selection is assumedto be associated with estimated effect sizes andcorresponding standard errors. Then a range ofplausible values for inestimable parameters canbe tested using the model to provide a range ofcorresponding estimates on the size of bias or thenumber of unpublished studies, which can be usedto indicate the possible impact of selection biasunder different assumptions. 482Vevea and Woods recently proposed a new weightfunction model for the purpose of sensitivityanalysis. 499 Similar to the sensitivity analysisapproach proposed by Copas, 482 Vevea’s new modelcan be used to conduct sensitivity analyses using arange of assumed weight function parameters (suchas moderate or severe, one- or two-tailed selection),rather than to provide ‘a best guess’ at the trueeffect size. 499Many proposed modelling methods require alarge number of studies and therefore may notbe appropriate for use in typical meta-analyses.Sensitivity analysis approaches proposed by Copaset al. 482,489,500 and by Vevea and Woods 499 could beused even when the number of studies is not large.Unfortunately, the complexity of these methodsmeans that they have mostly been used only bystatistical experts in method studies. Furtherdevelopment of user-friendly software is requiredto bring the methods into more mainstream use.Methods for detecting outcomereporting biasThe existence of outcome reporting bias issuspected when some eligible studies could notbe included in a meta-analysis due to a lack ofdata on an outcome. Outcome reporting bias isnot a problem for studies that had not measuredthe outcome of interest. Therefore, it is crucial,although difficult, to investigate whether theoutcome was measured or not in studies in whichthe outcome has not been reported. Publishedstudies can compared with their protocols whenavailable, or authors of published studies may becontacted to request clarifications about or data onthe unreported outcomes. 439© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Williamson and Gamble proposed a methodto investigate the possible impact of outcomereporting bias by imputing data for unreportedoutcomes. 95 More recently, they compared theirimputation method and the sensitivity analysismethod proposed by Copas and Jackson 482 in theassessment of outcome reporting bias. 501 Resultsof simulation indicate that outcome reportingbias may be overadjusted by using the imputationmethod as compared to the Copas method. 501Other statistical methodsBennett et al. proposed that a capture-recapturemethod may be used to assess the risk ofpublication bias. 407 The performance of thecapture-recapture method has not been properlyinvestigated by simulations.Ioannidis and Trikalinos proposed a test for anexcess of significant findings. 502 The exploratorytest can be used to estimate the number of theexpected studies with statistically significantresults according to certain assumptions. Thenthe number of expected significant studies iscompared with the number of observed studieswith statistically significant results. 502 Publicationand related bias may be one of the reasons for theexcess of significant studies.Fixed or random-effectsmodelsIn meta-analysis, larger studies will give greaterweight than smaller studies when results arequantitatively combined. 503 This procedure mayhave an advantage in reducing the impact ofpublication bias because less weight is givento smaller studies that are more vulnerable topublication bias.There are two statistical models that can be usedto combine results of individual studies in a metaanalysis:the fixed-effects model or the randomeffectsmodel. 504 In the fixed-effect model it isassumed that all individual studies are measuring asingle value of the true effect and that the observeddifference between the studies is due to samplingerror. The precision (e.g. the inverse of withinstudyvariance) of individual results is employed asthe weight for each study to estimate the pooledresult in meta-analysis using the fixed-effect model.By contrast, the random-effects model assumes thatindividual studies are measuring a distribution ofeffects. In addition to the variation within studies,65

Prevention of publication bias66the variation between studies is also incorporatedinto a meta-analysis using the random-effectsmodel. The differences between the fixed-effectsmodel and the random-effects model are oftenignorable when heterogeneity is small. 505 Whenthere is large heterogeneity between individualstudies, the confidence interval estimated by usingthe random-effects model will be wider than that byusing the fixed-effect model.Jackson investigated the impact of publication biason estimates of between-study variance (ι 2 -statistic)in meta-analysis. 506 The results of mathematicalanalysis demonstrate that publication bias mayincrease or decrease the between-study variance inmeta-analysis.The weights used to combine individual studiesare the inverse of within-study variances in thefixed-effect model, and are the inverse of totalvariance (i.e. within-study variance plus betweenstudyvariance) in the random-effects model.Therefore, by giving relatively larger weights tosmaller studies, the random-effects model maybe more vulnerable to publication bias than thefixed-effect model. 507 For this reason, it has beenrecommended that the result of the random-effectsmodel should be compared with the result of thefixed-effect model when there is heterogeneity inmeta-analysis. 469 If the pooled effect size by therandom-effects model is greater than that by thefixed-effect model, underlying causes will needto be investigated, to exclude the possibility ofpublication bias.Updating systematicreviewsUpdating of systematic reviews may reduce theimpact of publication bias because of the followingreasons. First, publication of studies with negativeor less favourable results may have a longer delaythan studies with positive or favourable results. 113Secondly, large-scale confirmatory trials are usuallyconducted and published after the publicationof early small trials. Small trials may be morevulnerable to biased selection for publication,and the conclusions based on limited evidencefrom early small trials may be overturned bymore convincing evidence from later large-scaletrials. 403–405Jadad et al. 508 compared 36 Cochrane reviewswith 39 meta-analyses published in paper-basedjournals (randomly selected sample) publishedin 1995. They found that, within 2 years afterpublication, 18 of the 36 Cochrane reviews hadbeen updated versus only one of the 39 reviewspublished in paper-based journals. Possible reasonsgiven for a very low update rate among paperbasedreviews included editors of such journalsnot being sufficiently interested in publishingupdated versions of previously published systematicreviews, authors not being aware of such interest, orauthors lacking the interest or resources to updatepreviously published reviews. 508French et al. examined the effect of updatingCochrane systematic reviews from 1998 to 2002. 509They found that 137 (38%) of the 362 completedreviews published in 1998 were updated and hadincluded new studies by 2002. Among the 119reviews that included new studies with comparableresults, statistical significance of the primaryoutcome was changed from significant (p < 0.05)to non-significant (p > 0.05) in five and from nonsignificantto significant in six reviews. There wasno mention of any changes in the direction ofthe estimates of treatment effects. French et al.concluded that ‘a priority-setting approach to theupdating of Cochrane systematic reviews may bemore appropriate than a time-based approach’. 509A recent study by Shojania et al. used 100 randomlyselected systematic reviews of conventional therapypublished from 1995 to 2005 to investigateimportant changes in evidence by updating. 510They defined quantitative signals for updating asany ‘changes in statistical significance or relativechanges in effect magnitude of at least 50%’, andqualitative signals for updating as ‘new informationabout harm and caveats about the previouslyreported findings that would affect clinical decisionmaking’. Important changes in evidence wereobserved in 57% of the reviews. Updating wasrequired for important changes in evidence by15% of reviews within 1 year after publication,and by 23% of reviews within 2 years after theirpublication. Multivariate analysis suggested thatsystematic reviews with cardiovascular topicsand heterogeneity may need to be frequentlyupdated. 510Shojania et al. 510 also reported details aboutimportant changes in evidence for seven selectedsystematic reviews. Compared with findings fromthe original seven reviews, important changes inevidence suggested that treatments of interest wereless beneficial in four cases, and more beneficial inanother two cases, and less harmful in one. 510 Thislimited evidence indicates that estimated treatment

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Table 2 Alternative statistical tests for funnel plot asymmetryStudy (year) Tests CommentsBegg and Mazumdar(1994) 340 Rank correlation test of the associationbetween standardised effect size and itsvarianceEgger et al. (1997) 149Tang and Liu (2000) 444Macaskill et al. 2001 460The method is based on a regression analysis ofGalbraith’s radial plot. 515 The standard normaldeviate (SND) is defined as the ln OR dividedby its standard error (SE). The SND is thenregressed against the estimate’s precision (theinverse of the SE), weighted by the inverse ofthe variance. The intercept of the regressionline provides a measure of funnel plotasymmetryA sample size-based linear regression in whichthe estimate of the effect size is regressedagainst the square root of the average numberof participants in the two trial groups, weightedby the sample sizeLinear regression of the estimated treatmenteffects (dependent variable) and correspondingsample size (Nt), weighted by the inverse of thevariance of the logit of the pooled proportion(using the marginal total)Deeks et al. (2005) 8 Linear regression of ln OR against 1/(ESS) 1/2 ,weighting by ESS, where effective sample sizeESS = 4⋅N0⋅N1/NtHarbord et al. (2006) 459Peters et al. (2006) 9Schwarzer et al. (2007) 462Rucker et al. (2008) 461Modified Egger’s test: a regression of Z/√Vagainst √V, where Z is the efficient score, and Vthe score varianceLinear regression of estimated treatment effectagainst 1/√Nt; weight used as in Macaskill’s testRank correlation test, based on observed andexpected cell frequencies, and the variance ofthe observed cell frequencies, in 2 × 2 tablesThe test is based on the arcsine transformationto stabilise the variance of binomial randomvariables. Then arcsine transformed statisticscan be used to replace variables used inBegg’s test, Egger’s test, or a random-effectsregression analysisIt suffers from a lack of power and thepossibility of funnel plot asymmetry cannotbe ruled out when the test is non-significant,particularly when the number of studies is smallThe test is unbiased for continuous outcomes.For binary data, Egger’s test is biased due tothe intrinsic association of the SND and itsprecision. 516 Egger’s test is more powerful thanBegg’s rank correlation test, but has high falsepositiverates, particularly when the treatmenteffect and/or the number of studies is large 464Tang and Liu (2000) mentioned that theinterpretation of the intercept alpha and its pvalue is the same as that of Egger’s method. Thepower and type I error of the method have notbeen properly investigatedThe correlation between the weight andtreatment effect is reduced. The statisticalpower of the test is lowThe test is developed for the evaluation ofdiagnostic odds ratio (DOR). The power islikely to be low, particularly when heterogeneityacross a study is largeWith large heterogeneity (e.g. ι 2 > 0.1), the testhas the problems of high false-positive rate andlow power similar to Egger’s and Macaskill’smethodsThe test is superior to Egger’s test in terms ofmore appropriate type I error rates. As withother statistical tests for funnel plot asymmetry,the statistical power is low when heterogeneityis large (e.g. ι 2 > 0.1)The test is developed for meta-analysis withsparse binary data. The power of the test is lowCompared with other tests, arcsinetransformed random-effects regression hasimproved power when both effect size andheterogeneity are large. The test is relativelyconservative with small sample size and in theabsence of heterogeneityeffects may be reduced or increased by updatingsystematic reviews.Updating a meta-analysis involves repetitivestatistical testing and the risk of type I errors willincrease. 511 Under certain circumstances, typeI errors due to repetitive tests for meta-analysis© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.updating may be greater than publication bias. 512To adjust for random error risk, Brok et al.recommended the use of trial sequential analysis(TSA) in meta-analysis. 511 However, possible type IIerrors and other biases are also important and therisk of type I error should not be a reason for notupdating systematic reviews.67

Prevention of publication biasSummaryMany methods have been suggested forpreventing, testing or adjusting for publicationbias. The available methods could be classified asmethods for preventing publication bias (discussedin Chapter 6) and methods for dealing withpublication bias in systematic reviews (discussed inthis chapter). In addition, it is possible to classifythe available methods according to the stage of aliterature review: to prevent publication bias beforea literature review (e.g. prospective registrationof trials), to detect publication bias during aliterature review (e.g. locating unpublishedstudies, funnel plot and related tests, sensitivityanalysis modelling), or to minimise the impactof publication bias after a literature review (e.g.confirmatory large-scale trials, updating thesystematic review).The recent development of clinical trial registrationand electronic publication of results from clinicaltrials will facilitate the identification and locationof ongoing or unpublished clinical trials. However,for non-trial studies, including basic laboratoryresearch, epidemiological studies and even earlystage clinical studies, publication and resultreporting bias seems still at large as before.Funnel plot and related statistical tests have beenwidely used in systematic reviews to assess thepossibility of publication bias. Unfortunately, theinterpretation of results of funnel plot-related testswas often too simplistic and likely misleading. 513,514Therefore, detailed recommendations have beenrecently proposed about when and how to usethe funnel plot-related statistical tests in metaanalysis,469,470 which may facilitate more cautiousinterpretation of funnel plot asymmetry. However,the current recommendations about tests for funnelplot asymmetry are based on very limited and fastchanging empirical evidence, and they may have tobe revised when new evidence emerges.Many sophisticated modelling methods have notbeen widely used in systematic reviews, possiblybecause of their complexity and lack of userfriendlysoftware. The main development ofsophisticated modelling methods perhaps is themore general recognition that these methodsshould be used to conduct sensitivity analyses,rather than to provide an estimate of the ‘true’effect size by adjusting for assumed selection bias.However, it is unclear how useful such sensitivityanalyses are when the results of meta-analyses areused for decision-making in practice.We concluded previously in the 2000 HTA reportthat all statistical methods ‘are by nature indirectand exploratory, and often based on certain strictassumptions that can be difficult to justify in thereal world’; and ‘the attempt at identifying oradjusting for publication bias in a systematic reviewshould be mainly used for the purpose of sensitivityanalyses’. 2 The updated review indicates that theabove conclusions are still held.68

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 9Survey of published systematic reviewsIn our previous Health Technology Assessment(HTA) report, 2 193 systematic reviews from theDatabase of Abstracts of Reviews of Effectiveness(DARE) were used to identify any evidenceof dissemination bias and to illustrate themethods used in systematic reviews for dealingwith publication bias. However, the systematicreviews included in DARE were probably, onaverage, of higher quality than those fromgeneral bibliographic databases and hence therepresentativeness of the reviews assessed wasquestionable. In addition, reviews on effectivenessof health-care interventions and accuracy ofdiagnostic technologies were not assessedseparately. The problem of dissemination biasmight be different between the two types ofsystematic reviews. In the current updated reviewwe have obtained a sample of systematic reviewsfrom a general bibliographic database (Medline)and classified these reviews into the followingcategories: (1) systematic reviews of studies oneffects of health-care interventions, (2) systematicreviews of studies on accuracy of diagnostic tests,(3) systematic reviews of epidemiological studieson association between risk factors and healthoutcomes, and (4) systematic reviews of geneticstudies on association between genes and disease.We also assessed a sample of systematic reviews thatexplicitly discussed publication bias.Assessment of randomlyselected reviewsWe searched MEDLINE for systematic reviewspublished in 2006 (see Chapter 2 for methods)and randomly selected 100 systematic reviews ofstudies of effectiveness of interventions, 50 reviewsof studies of diagnostic accuracy, 100 reviews ofepidemiological studies on risk factors and healthoutcomes, and 50 reviews of studies of gene-diseaseassociations (Appendix 17). The reviews wereassessed independently by two reviewers using adata extraction form, tailored to the type of reviewbeing assessed (Appendix 4).Main characteristics of includedreviewsThe 100 treatment effectiveness reviews comprised54 reviews of pharmaceutical interventions, 10reviews of psycho-educational interventions, 11reviews of surgical interventions, and 25 reviews ofmixed or other interventions. The median numberof individual studies included in each review was 14(range 2 to 198).The tests or techniques investigated in the 50reviews of diagnostic accuracy included laboratorytests (n = 21), imaging techniques (n = 28),electrical tests (n = 5), clinical examinations (n = 10)and other tests (n = 7) (several reviews assessedmore than one test or technique). The mediannumber of studies included in the 50 reviews was20 (range 4 to 213).Risk factors investigated in 100 reviews ofepidemiological studies included lifestyle (n = 31),environmental (n = 17), biomedical (n = 45), mental(n = 6) and other factors (n = 18). Cancer (n = 24)and cardiovascular diseases (n = 20) were commonhealth outcomes considered in these reviews. Themedian number of studies included in each reviewwas 20 (range 3 to 200).In 50 reviews of gene-disease association,diseases investigated included cancer (n = 13),cardiovascular disease (n = 4), diabetes (n = 3) andmental diseases (n = 15). The median number ofstudies per review was 13 (range 3 to 86).Among the 300 systematic reviews 83 werenarrative systematic reviews in which the resultsof the primary studies were summarised but notstatistically combined, and 217 were meta-analyses,in which statistical methods were used to combinethe results of two or more primary studies (Table 3).There were 16 (16%) Cochrane reviews amongstthe 100 treatment reviews.Systematic literature searchSimilar to the findings reported in the 2000 HTAreport, MEDLINE (74% to 95%) and checking69© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Survey of published systematic reviewsTable 3 Types of reviews – narrative or meta-analysesNarratives (%) Meta-analyses (%) Total (%)Treatment 40 (40%) 60 (60%) 100Diagnostic 9 (18%) 41 (82%) 50Risk factor 32 (32%) 68 (68%) 100Genetic 2 (4%) 48 (96%) 50Total 83 (28%) 217 (72%) 300reference lists of retrieved studies (42% to 73%)were most commonly used to search literature(Figure 14). EMBASE was now searched in abouthalf of the treatment and diagnostic reviews (50%and 54%, respectively), compared with only 17%in the previous reviews. There was a considerableincrease in the utilisation of the Cochrane Library,from only 5% in reviews published in 1996 to 58%in treatment reviews, 46% in diagnostic reviews,24% in risk-factor reviews and 6% in geneticreviews. The search of the CINAHL (CumulativeIndex to Nursing and Allied Health Literature)database increased from 8% in 1996 to 24% intreatment reviews, 20% in diagnostic reviews, and18% in risk factor reviews in 2006.Prospective registers of clinical trials and otherstudies were searched for unpublished or ongoingstudies in 18 treatment reviews, three diagnosticreviews and two risk-factor reviews (Table 4). TheUK National Research Register and ClinicalTrials.gov are the two commonly searched prospectiveregisters.Language restrictionThe current review examined whether languagerestrictions (e.g. included studies were limited toEnglish-language ones only) were applied by thereview authors. It was found that 35% of the reviewsshowed language restriction, with the majority ofthem being restricted to English language. Thirtyfivepercent of the reviews did not explicitly reportwhether any language restrictions were applied ornot. The proportion of reviews that explicitly statedno language restriction was 39% in treatmentreviews, 42% in diagnostic reviews, and 20% in riskfactorreviews and genetic reviews (Table 5).Non-English-language studies were explicitlysearched for in 45% treatment reviews, 52%diagnostic reviews, 34% risk-factor reviews and22% genetic reviews. Overall, 39% of the reviewsexplicitly searched for non-English-languagestudies, compared with 19% in the reviewspublished in 1996. However, only 15% of thereviews actually included non-English-languagestudies (Table 5). Authors did not always explicitlymention that they had searched for non-Englishlanguagestudies, even though non-English studieswere indeed listed in the review.Treatment reviews and diagnostic reviews weremore likely to have no language restrictions,and more frequently searched for or includednon-English-language literature compared withTable 4 Searching of prospective registers of trials or other studies in reviews70Study registerTreatmentreviewDiagnosticreviewRisk-factorreviewPhysician Data Query clinical trial 2registerUK National Research Register 8 3 2ClinicalTrials.gov 8 1Current Controlled Trials 3 1 1Other/unclear 5Number of reviews searched studyregisters18/100 3/50 2/100 0/50Genetic review

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8MEDLINEEMBASEPsycINFOCochraneTreatmentDiagnosticRisk factorGeneticReferencesCINAHLHandsearchExpertsCompanyPreceedingsSIGLEOther0 20406080 100Sources searchedTreatment Diagnostic Risk factors Geneticn = 100 (%) n = 50 (%) n = 100 (%) n = 50 (%)MEDLINE 95 (95) 47 (94) 95 (95) 37 (74)EMBASE 50 (50) 27 (54) 37 (37) 7 (14)PsycINFO 11 (11) 2 (4) 23 (23) 2 (4)Cochrane 58 (58) 23 (46) 24 (24) 3 (6)References 71 (71) 42 (42) 73 (73) 23 (46)CINAHL 24 (24) 10 (20) 18 (18) 0 (0)Handsearch 30 (30) 17 (34) 5 (5) 1 (2)Experts/Authors 33 (33) 18 (36) 20 (20) 2 (4)Company 19 (19) 0 (0) 1 (1) 0 (0)Proceedings 35 (35) 8 (16) 11 (11) 3 (6)SIGLE 0 (0) 4 (8) 1 (1) 0 (0)Other 60 (60) 18 (36) 50 (50) 9 (18)Figure 14 Databases searched for literature in 300 reviews.risk-factor and genetic reviews. The proportionof reviews in which non-English-language studieswere explicitly searched for or included was 47%in treatment reviews, 56% in diagnostic reviews,39% in risk-factor reviews and 28% in geneticreviews (Table 5). The current findings indicate animprovement compared with previous findings,where only about 30% of the reviews searched foror included non-English-language studies. 2Grey literature and unpublishedstudiesThe Third International Conference on GreyLiterature has defined grey literature as ‘thatwhich is produced on all levels of governmental,academic, business and industry in print andelectronic formats, but which is not controlledby commercial publishers’. 114 In this review, wehave attempted to separate grey literature andother unpublished studies. The commonly usedTitle: 06-92-02 Proof Stage: 3© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Cactus Design and Illustration LtdFigure Number: 00.14.ai71

Survey of published systematic reviewsTable 5 Language restriction in 300 reviewsLanguage restrictionTreatmentn = 100 (%)Diagnosticn = 50 (%)Risk factorn = 100 (%)Geneticn = 50 (%)No restriction 39 (39%) 21 (42%) 20 (20%) 10 (20%)Restricted to English 23 (23%) 15 (30%) 31 (31%) 11 (22%)Restricted to two or more 6 (6%) 4 (8%) 14 (14%) 1 (2%)languagesUnclear 32 (32%) 10 (20%) 35 (35%) 28 (56%)Non-English-language studies 45 (45%) 26 (52%) 34 (34%) 11 (22%)searched forNon-English-language studies 14 (14%) 14 (28%) 10 (10%) 6 (12%)includedNon-English-language studiessearched for or included47 (47%) 28 (56%) 39 (39%) 14 (28%)methods to identify grey literature were searchingconference abstracts, meeting proceedings andgrey literature-specific databases like SIGLE andLILACS. Checking the reference list of the reviewsindicates that conference abstracts were frequentlyincluded. Grey literature was explicitly soughtin 50% of treatment reviews, 30% of diagnosticreviews, 32% of risk-factor reviews, and only 8%of genetic reviews (Table 6). Overall, 34% of the300 reviews explicitly searched for grey literature,although only 13% included them.To identify other unpublished studies, thecommonly used method was through contactingauthors or experts, and pharmaceutical companies.Of the 300 reviews, 27% explicitly searched forother unpublished studies and only 8% actuallyincluded them (Table 6). When grey literature andother unpublished studies were combined, theproportion of reviews that explicitly searched forgrey or unpublished studies was 58% for treatmentreviews, 36% for diagnostic reviews, 35% for riskfactorreviews and 10% for genetic reviews (Table 6).In addition, Table 6 also shows that grey literatureand unpublished studies were more likely to beincluded in treatment reviews than diagnostic orrisk-factor reviews.The previous HTA report showed that only about35% of reviews explicitly searched for or includedstudies that were unpublished or presented asabstracts. 2 In reviews published in 2006, this was61% for treatment reviews, 36% for diagnosticreviews, 41% for risk-factor reviews and 20% forgenetic reviews (Table 6).Table 6 Grey literature and unpublished studies in reviewsTreatmentn = 100 (%)Diagnosticn = 50 (%)Risk factorn = 100 (%)Geneticn = 50 (%)Grey literatureSearched for 50 (50%) 15 (30%) 32 (32%) 4 (8%)Included 17 (17%) 5 (10%) 12 (12%) 5 (10%)Other unpublished studiesSearched for 49 (49%) 7 (14%) 20 (20%) 5 (10%)Included 14 (14%) 2 (4%) 2 (2%) 6 (12%)72Grey literature or unpublished studiesSearched for 58 (58%) 18 (36%) 35 (35%) 5 (10%)Included 24 (24%) 6 (12%) 12 (12%) 8 (16%)Searched for or included 61 (61%) 18 (36%) 41 (41%) 10 (20%)

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Consideration of outcomereporting biasOutcome reporting bias is related to theincomplete reporting within published studiesand occurs when studies with multiple outcomesselectively report only some of the measuredoutcomes. We examined whether outcomereporting bias was considered and/or reported inour sample of 300 reviews. We found that outcomereporting bias was explicitly mentioned in 18% oftreatment reviews, 14% of diagnostic reviews, 3% ofrisk-factor reviews and 16% of genetic reviews.Methods used to test forpublication biasAvailable tests for publication bias were not used inthe majority of treatment reviews (79%), diagnosticreviews (76%) and risk-factor reviews (69%) (Table7). Compared with other reviews, publication biaswas more likely to be tested in genetic reviews,possibly due to perceived high risk of bias in suchreviews. The most commonly used methods fortesting the association between sample sizes andtreatment effects were funnel plots complementedby other related methods (Egger’s and Begg’s test).Egger’s test was explicitly used in 45 reviews andBegg’s test in 24 reviews. Funnel plot and otherrelated methods were used in 26% of the 300reviews, compared with less than 6% in the 193reviews published in 1996. In contrast to reviewspublished in 1996, the fail-safe N method wasused in far fewer reviews (7% versus 1%). All otherstatistical methods to test publication bias were onlyrarely used. In the reviews that explicitly tested forpublication bias, 23% of the 21 treatment reviews,42% of the 12 diagnostic reviews, 52% in the 31risk-factor reviews, and 48% in the 27 geneticreviews showed some evidence of the existence orabsence of publication bias.Consideration of publication biasIn accordance with the findings of the previousreport, 2 publication bias was discussed ormentioned more often in the meta-analyses thanin the narrative reviews (Table 8). The possibility ofpotential publication bias was discussed more oftenin the genetic reviews (70%) than in treatmentreviews (32%), diagnostic reviews (48%) and riskfactorreviews (42%) (Table 8).When conclusions of authors of reviews wereclassified as positive, not positive and unclear, wefound positive conclusions in 61% of treatmentreviews, 35% of diagnostic reviews, 91% of riskfactorreviews and 32% of genetic reviews. Becauseof the small number of available reviews, it is notclear whether authors’ conclusions were associatedwith whether or not publication bias was explicitlytested or considered in a review (Table 8). Similarly,there was no clear trend to show that publicationbias was more or less likely to be explicitly testedor discussed in meta-analyses that reportedstatistically significant results (Table 8).Assessors’ judgementEfforts taken to minimisepublication biasAs part of our study, we used two assessors to assessindependently the efforts taken by review authorsto minimise publication bias within the selectedsample of reviews. The assessors’ judgementswere based on the following measures: literaturesearching approach used, consideration of outcomereporting bias, reporting of any missing outcomesand methods used to deal with the missingoutcomes, and discussion of publication bias andany methods used to deal with publication bias. Foreach review, two assessors independently scoredTable 7 Methods used to deal with publication bias in reviewsTreatment Diagnostic Risk factor GeneticNot used 79 (79%) 38 (76%) 69 (69%) 23 (46%)Funnel plot and 15 (15%) 9 (18%) 27 (27%) 26 (52%)related methodsEgger’s test 9 (9%) 2 (4%) 16 (16%) 18 (36%)Begg’s test 5 (5%) 0 8 (8%) 11 (22%)Trim-fill method 0 0 2 (2%) 0Fail-safe N 2 (2%) 0 1 (1%) 0Modelling 0 0 0 1 (2%)Other 2 (2%) 1 (2%) 5 (5%) 5 (10%)73© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Survey of published systematic reviews74Table 8 Consideration of publication bias and results or conclusions of reviewsTreatment reviews Diagnostic reviews Risk factor reviews Genetic reviewsTotal Tested % Discussed % Total Tested % Discussed % Total Tested % Discussed % Total Tested % Discussed %Authors’ conclusionsPositive 61 26.2% 32.8% 35 28.6% 48.6% 91 34.1% 45.1% 32 56.3% 71.9%Not positive 12 25.0% 33.3% 9 1/9 4/9 5 0/5 1/5 17 47.1% 64.7%Uncertain 27 7.4% 29.6% 6 1/6 3/6 4 0/4 0/4 1 1/1 1/1Meta-analysisresultsNot conducted 40 2.5% 15.0% 32 6.3% 21.9% 2 0/2 1/2Significant 49 36.7% 44.9% 66 43.9% 53.0% 33 57.6% 72.7%Non-significant 11 18.2% 36.4% 2 0/2 0/2 15 53.3% 66.7%Total 100 21.0% 32.0% 50 24.0% 48.0% 100 31.0% 42.0% 50 54.0% 70.0%Actual numbers are given when denominators are < 10.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8the level of efforts taken by the review authors toreduce publication bias (based on the assumptionthat all other assessors differed in a constant wayfrom the first assessor). Efforts taken to minimisebias were ‘sufficient’ if the review attempted tosearch and probably include non-English-languagestudies, grey literature and unpublished studies,considered outcome reporting bias, the issue ofpublication bias and reported any missing outcomedata. ‘Partial’ efforts to minimise bias were whenthe review searched at least two of the three, i.e.non-English-language studies and/or grey literatureand/or unpublished studies, may or may not haveconsidered outcome reporting bias and publicationbias, and may or may not have reported missingoutcome data. Efforts taken to minimise bias were‘insufficient’ when no attempts were made by thereview authors to search for non-English-languagestudies, grey literature or unpublished studies,they did not consider outcome reporting biasor publication bias, and did not report missingoutcome data. These judgements were thenconverted to scores as follows: insufficient = 0;partial = 1 and sufficient = 2, and pooled togetherfor each assessor.Table 9 shows the results of assessors’ judgementabout whether the review authors’ efforts tominimise publication bias were sufficient or not.There was a fair agreement between assessorsfor the treatment reviews (κ = 0.30) and riskfactor reviews (κ = 0.35). Moderate agreementwas seen for genetic (κ = 0.43) and diagnosticreviews (κ = 0.40). The rate of agreement betweenassessors was 55% for treatment reviews, 68% fordiagnostic and risk factor reviews, and 74% forgenetic reviews. Based on the agreed judgementby two independent assessors, efforts to minimisepublication bias were less likely to be insufficient intreatment reviews (18%) compared with diagnosticreviews (30%), risk factor reviews (55%) and geneticreviews (56%) (Table 9).Risk of publication biasTwo assessors also independently assessed thepossibility that review authors’ conclusions mightbe invalid because of possible publication andrelated biases. The judgement for assessing thepotential risk of publication bias was based onthe efforts taken to minimise publication bias,discussion of publication bias, methods used todeal with publication bias and finally the authors’conclusion. The assessment was subjectivewithout proper validation of the criteria. Twoassessors independently scored the perceived© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.risk of publication bias in systematic reviews.Risk of bias was marked as ‘high’ if the effortstaken to minimise publication bias were partialor insufficient, publication bias was not discussedand the authors’ conclusions were positive. Riskof bias was ‘moderate’ if partial efforts were takento minimise bias, publication bias was probablyconsidered, and the author’s conclusions mighthave been positive with cautious interpretation.Risk of bias was ‘low’ if partial or sufficient effortswere taken to minimise bias, publication bias wasconsidered with some methods used to deal with it,and the author’s conclusions were negative. Thesejudgements were converted to the following scores:low = 0, moderate = 1 and high = 2.Table 10 shows the results of assessors’ assessment ofthe risk of publication bias in reviews. Agreementbetween assessors was poor for treatment reviews(κ = 0.17), fair for genetic and diagnostic reviews(κ = 0.29 and κ = 0.25 respectively), and moderatefor risk factor reviews (κ = 0.44). The rate ofagreement between the two assessors was 53% fortreatment reviews, 68% for diagnostic reviews,73% for risk factor reviews and 60% for geneticreviews. According to the agreed judgement bytwo independent assessors, the rate of moderateto high risk of publication bias was relatively lowerin treatment reviews (48%) in comparison withdiagnostic reviews (64%), risk factors reviews (71%)and genetic reviews (58%).Assessment of reviewsthat explicitly tested forpublication biasA random sample was obtained of 50 reviews,published from 2000 to 2008 in Medline, thatexplicitly tested for or considered publicationbias. Data extraction of these 50 reviews wasindependently conducted by two reviewers andany disagreements were resolved by discussionbetween the two reviewers or by a third reviewer.Three reviews were further excluded as the fullpublication could not be obtained; hence a total of47 reviews were analysed.Of the 47 reviews included, 18 (38%) evaluatedeffects of treatment intervention, 16 (34%)studied the association between various riskfactors and disease, seven (15%) studied theassociation between a specific gene and disease,three (6%) evaluated the accuracy of diagnostictests, and the remaining three (6%) had otherobjectives. We analysed if non-English-language75

Survey of published systematic reviewsTable 9 Assessors’ judgement about efforts taken in reviews to minimise the risk of publication biasTreatment reviewsAssessor-2Assessor-1 Insufficient Partial Sufficient TotalInsufficient 18 4 1 23Partial 24 34 8 66Sufficient 0 8 3 11Total 42 46 12 100Diagnostic reviewsAssessor-2Assessor-1 Insufficient Partial Sufficient TotalInsufficient 15 3 0 18Partial 9 19 1 29Sufficient 0 3 0 3Total 24 25 1 50Risk factor reviewsAssessor-2Assessor-1 Insufficient Partial Sufficient TotalInsufficient 55 5 0 60Partial 19 12 4 35Sufficient 3 1 1 5Total 77 18 5 100Genetic reviewsAssessor-2Assessor-1 Insufficient Partial Sufficient TotalInsufficient 28 12 0 40Partial 1 8 0 9Sufficient 0 0 1 1Total 29 20 1 5076studies, grey literature and unpublished studieswere searched for and included in these reviews.Non-English-language studies were explicitlysearched for in 38% of reviews including fournarrative reviews and 14 meta-analyses. None ofthe four narrative reviews included non-Englishlanguagestudies. The remaining reviews (40%)were unclear regarding the search for or inclusionof non-English-language studies. Grey literaturewas searched for in 14% of the reviews and 13%included them. A majority of the reviews (85%) didnot clearly mention searching for or including greyliterature. The analysis showed that 72% of thereviews did not mention searching for or includingunpublished studies. Only nine (19%) exclusivelysearched for unpublished studies and only two (4%)included them.The assessment of the sources searched in thesereviews is consistent with the findings of the 300reviews assessed in the first part of this chapter.Medline (96%) was the most commonly searcheddatabase. Other sources used to identify literaturewere checking of reference lists of identifiedstudies (81%), Embase (64%), specialisedand other databases (53%), Cochrane Library(47%), contacting authors or experts (30%),hand searching journals (17%), Cinahl (17%),PsycINFO (15%), conference proceedings (13%),SIGLE (6%), and contacting pharmaceuticalcompanies (6%). Four percent of the reviews didnot state the sources searched. Language restrictionwas not clearly stated in 36% of the reviews, and36% of the reviews were restricted to one or more

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Table 10 Assessors’ judgement about risk of publication bias in reviewsTreatment reviewsAssessor-2Assessor-1 Low Moderate High TotalLow 5 7 1 13Moderate 12 35 16 63High 0 11 13 24Total 17 53 30 100Diagnostic reviewsAssessor-2Assessor-1 Low Moderate High TotalLow 2 1 0 3Moderate 3 29 9 41High 0 3 3 6Total 5 33 12 50Risk factor reviewsAssessor-2Assessor-1 Low Moderate High TotalLow 2 0 4 6Moderate 3 14 20 37High 0 0 57 57Total 5 14 81 100Genetic reviewsAssessor-2Assessor-1 Low Moderate High TotalLow 1 0 0 1Moderate 3 13 10 26High 2 5 16 23Total 6 18 26 50languages (more commonly the English language23%). Outcome reporting bias was not consideredin 87% of the reviews, and only 11% of the reviewsreported missing outcome data.The analysis of reviews for discussion of publicationbias and the methods used to test publication biasshowed that publication bias was discussed in 44(94%) of the reviews, and that funnel plot was themost commonly used method to detect publicationbias (75%). This method was then followed byEgger’s test (49%) and Begg’s test (32%). Methodslike identifying unpublished trials, the fail-safe Nmethod and trim and fill method were only rarelyused (2% each). Of the 44 reviews in which authorsexplicitly considered risk of publication bias, 19© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.(43%) reviews had a high risk of bias, eight (18%)had a moderate risk of bias and 17 (39%) had a lowrisk of bias.We further assessed the association of discussionof publication bias with the review authors’conclusions and found that 39 reviews thatdiscussed publication bias had significant resultswith considerable uncertainty. Another four reviewsthat discussed publication bias had non-significantresults and only one review had significant results.This indicates that in most of the reviews theauthors have interpreted the results with cautionwhen there is any consideration or existence ofpublication bias.77

Survey of published systematic reviews78Two assessors independently assessed the effortstaken to minimise publication bias within theselected sample, and the risk of publicationbias. In assessing whether the efforts taken tominimise publication bias were sufficient, partialor insufficient, there was a fair agreement betweenassessors (κ = 0.32). The judgement for assessingthe potential risk of publication bias was basedon the efforts taken to minimise publicationbias, discussion of publication bias, methodsused to deal with publication bias and finally theauthors’ conclusion. The 47 reviews showed apoor agreement between assessors for the risk ofpublication bias (κ = 0.09).SummaryA survey of systematic reviews of studies oftreatment efficacy and diagnostic accuracypublished in 1996 concluded that the issue ofpublication bias was largely being ignored insystematic reviews, and very few of them actuallyused any methods to deal with publicationbias. However, in the current survey of reviewspublished in 2006, there was some improvementin the methods used to deal with publication bias.Reviews of health-care interventions (therapeuticor diagnostic) are making greater efforts to locateand/or include non-English-language studies (47%versus 30%), and grey literature or unpublishedstudies (53% versus 35%). A thorough literaturesearch while conducting a systematic review mayreduce the possibility of excluding unpublishedstudies, those published in non-English languagesor as grey literature. It is always advisable tosearch more than one electronic database as manyjournals are indexed in only one of the commonlyused databases. 517Compared with the previous sample of reviews,there was an increase in the use of availablemethods to test for publication bias in recentreviews (22% versus 17%). However, the proportionof reviews in which publication bias was explicitlydiscussed remained the same between recenttreatment and diagnostic reviews and the previoussample (37% versus 36%).The previous assessment recognised that theproblems of publication and related biases weremore often dealt with in meta-analysis than innarrative reviews. This finding is unchangedin the updated review and which could merelybe a reflection of marked heterogeneity withinthe sample. Assessment of the narrative reviewsshowed an overall lack of efforts taken to reduce orminimise publication bias in all four categories ofreviews.Funnel plot and related statistical tests (includingEgger’s test and Begg’s test) are common methodsused to detect publication bias in systematicreviews, particularly in risk factor reviews. The failsafeN method was used in some previous reviewsbut it was much less likely to be used in recentreviews (7% versus 1%). All other methods are not,or very rarely, used in the 300 general reviews andin the 44 reviews in which publication bias wasexplicitly tested.Non-English-language studies and grey literatureor unpublished studies were more likely to beexplicitly searched for in treatment and diagnosticreviews, compared with reviews of epidemiologicalstudies (50% versus 35%, and 53% versus 34%,respectively). Conversely, publication bias wasless likely to be tested and discussed in treatmentand diagnostic reviews than in epidemiologicalreviews (22% versus 39%, and 37% versus 51%,respectively). These differences between reviews ofintervention studies and reviews of observationalstudies are possibly due to different approachestaken by authors in different fields to deal withperceived problems of publication bias. In arecent study that examined the frequency anddeterminants of full publication of studies ofdiagnostic accuracy submitted as abstracts atinternational stroke meetings, it was found that76% of 160 abstracts were subsequently publishedin full and that clinical utility of results orother study characteristics did not predict theirpublication. However, this study was unable toassess the extent of a possible bias in the selectionof abstracts for presentation. 57When assessors were asked to assess independentlythe level of efforts taken to minimise publicationbias and the risk of publication bias in reviews,the rate of agreement was on an average 64% and63% respectively. Based on the agreed judgement,reviews of treatment effect were more likely tohave insufficient efforts to minimise publicationbias, but less likely to have moderate or highrisk of publication bias, compared with reviewsof diagnostic accuracy or risk factors (includinggene-disease association). According to data from44 reviews in which risk of publication bias wasexplicitly considered by authors, 43% of reviewshad a high risk, 18% had a moderate risk and 39%had a low risk of publication bias.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8The assessment of reviews was challenging in manyways. Most of the variables in the data extractionform were assessed subjectively as ‘yes’, ‘no’ or‘unclear’ and hence information may have beenlost. For example, many studies reported that non-English-language studies were included, but towhat extent they were searched for was unclear. Theextent of searching for studies in languages otherthan English may vary, from having no languagerestriction in a PubMed search to running searchesin specific non-English language databases. Thesame applies for grey literature and unpublishedstudies. The risk of publication bias was assessedfrom several perspectives: completeness ofliterature search, findings of any efforts to detectpublication bias, and results of meta-analysis. Thisassessment was qualitative and the criteria have notbeen properly validated. However, we reported theresults of the assessment of risk of bias to illustratedifficulties in any such attempts.79© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Chapter 10DiscussionAvailable evidence onpublication biasThe updated review confirmed findings from theprevious HTA report that studies with significant orimportant results were more likely to be publishedthan those with non-significant or ‘unimportant’results. It appears that publication bias occursmainly before the presentation of findings atconferences and the submission of manuscriptsto journals. However, factors associated withpublication bias remain unclear. The existence ofoutcome reporting bias has been demonstratedby recently published empirical studies. There islimited evidence indicating that harm outcomesand subjectively assessed outcomes may be morevulnerable to reporting bias than efficacy outcomesand objectively assessed outcomes.Studies with significant or important results were,on average, published earlier than studies withnon-significant results, although the new evidencewas less clear than was suggested in the previousreport. Any time lag bias is likely to occur beforemanuscript submission for journal publication. 81Substantial new evidence on grey literature andlanguage bias was identified in this updated review.Grey literature or non-English-language studieson average reported smaller treatment effects thanstudies that were formally published or studies thatwere published in English. However, the directionand extent of bias was usually unpredictable.There is limited evidence indicating that the riskof language bias may be particularly high in someareas of research such as complementary andalternative medicine. The updated review alsoidentified limited new evidence on citation bias,duplicate publication bias, place of publicationbias, database bias and media attention bias.As a direct consequence of publication and relatedbiases, estimates based on published studies maybe misleading. For example, publication andrelated bias may result in an overestimation oftreatment effects or an underestimation of adverseeffects. In this updated review, the consequencesof publication and related biases were separatelydiscussed for basic (animal and laboratory)research, observational studies and clinical trials.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Biased publication of results of basic research mayexplain negative results from subsequent clinicaltrials. Contradictory findings from epidemiologicalstudies may be partly due to publication andrelated biases. The consequences of publicationbias in clinical trials may be more serious, resultingin the use of less cost-effective, or ineffective, oreven harmful interventions in clinical practice.This updated review identified a few new cases thatindicated the detrimental impact of publicationand related biases.This updated review confirmed findings fromthe previous HTA report that the most commonreason for publication bias was that investigatorsfailed to write up or submit studies with nonsignificantresults (see Figure 7). However, it shouldbe recognised that investigators’ decision to writeup an article and then submit it may be affectedby pressure from research sponsors, instructionfrom journal editors, and requirements of theresearch award system. Clearly, commercial andother competing interests of research sponsorsand investigators may influence the profile ofdissemination of research findings.Limitations of evidence studieson publication biasThe most important evidence on publicationbias comes from cohort studies showing thatthe publication of studies is associated with thestrength or direction of the results. However, thedefinition of publication status and classificationof study results are often different in empiricalstudies of publication bias. For time lag bias,time to publication could be measured startingfrom different time points (e.g. approval by REC,recruitment of participants, completion of followup)during the process of research. Therefore, biasmay be introduced in studies of publication biasbecause of inevitable subjectivity in the choice ofdefinitions and methods.Large cohort studies on publication and relatedbiases usually included cases that were highlydiverse in terms of research questions, designsand other study characteristics. Many factors(e.g. sample size, design, research question and81

Discussion82investigators’ characteristics) may be associatedwith both study results and the possibility ofpublication. Adjusted analyses by some factorsmay be conducted but it was generally impossibleto exclude the impact of confounding factors onthe observed association between study results andformal publication. However, confounding factorsmay not be a problem in many single case studiesthat provided empirical evidence on publicationand related biases. But, evidence from case studiesis susceptible to bias due to selective reporting. 123There are several high-quality empirical studiesthat were less selective and in which the impactof confounding factors could be controlled. Forexample, Egger et al. (2003) 3 and Moher et al.(2003) 4 used multiple meta-analyses to investigategrey literature and language bias (see Chapter 3).The results of trials published in English and thosepublished in non-English languages was comparedwithin each meta-analysis that aimed to answerthe same clinical question. In empirical studiesby Chan et al., 6,7 outcomes reported in publishedpapers were compared with outcomes specified inprotocols within each trial, so that the observedoutcome reporting bias is unlikely to be due toconfounding factors. However, generalisability isstill an issue even for findings from these goodquality empirical studies.Studies of publication bias themselves may beas vulnerable as other studies to the selectivepublication of significant or striking findings. 1,518Dubben and Beck-Bornholdt (2005) used thefunnel plot approach, and found no evidenceof publication bias in studies of publicationbias. 519 They acknowledged that the analysis washandicapped by insufficient power (with only 26included studies) and also by the diverse definitionsof publication bias in the primary studies. Songet al. pointed out that the study had other moreimportant limitations so that dissemination biasof studies on publication bias could not be safelyexcluded. 520 Funnel plot analysis was used inChapter 3 to detect small study effects in cohortstudies of publication bias (see Figure 6), and theplot was not statistically significantly asymmetric.However, there is still reason to suspect theexistence of publication or reporting bias in studiesof publication bias. We identified a large numberof reports of full publication of meeting abstracts,and the association between study results andfull publication had not been reported in mostof these reports. It is often unclear whether thisassociation had not been examined, or was notreported because the association proved to be nonsignificant.In addition, we have mentioned earlierthat single case studies that provided empiricalevidence on publication bias may be biased becauseof selective reporting of striking findings.The existence of publication and related biaseswas usually confirmed by comparing results ofpublished studies with those of unpublishedstudies. However, the actual impact of such bias isbest investigated by a comparison of the result ofpublished studies with the result of a combinationof published and unpublished studies. In mostcases, the actual impact of publication and relatedbiases was non-significant in a systematic reviewthat combined evidence from all relevant studies. 3How to deal withpublication bias?The consequences of publication bias werepreviously overlooked by many leading experts. 2According to Beveridge, research with nonsignificantresults ‘clutters up the journals and doesmore harm than good to the author’s reputationin the minds of the discerning.’ 521 A book aboutethics in the dissemination of new knowledgeeven recommended that ‘it is preferable to publishpositive research findings, because they advanceknowledge’. 522The importance of ‘negative’ findings fromresearch has now been generally recognised. Awide public awareness of detrimental consequencesof publication bias has promoted recent efforts toprevent and reduce publication bias. For example,regulatory authorities, journals and researchsponsors have taken action to improve the currentsituation because of several high-profile cases ofincomplete disclosure of negative results of trialssponsored by pharmaceutical companies (seeChapter 7).According to the stage of a literature review,measures to combat publication and related biascan be classified as those before, during or aftera literature review (see Figure 1). 2 Table 11 showsvarious methods that can be used to deal withdifferent types of publication and related biased.For example, methods that can be used to combatthe non-publication of ‘negative’ findings includeprospective registration of studies, disclosure ofdata from unpublished studies, searching for andinclusion of unpublished studies, and assessment ofrisk of publication bias in systematic reviews.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8The recent development of information technologyand electronic publication provides great technicalpotential to overcome some limitations ofconventional printed journals. Publication biasmay be reduced by publication in electronic mediawith unlimited space, direct electronic linkagebetween references, timely publication, and costeffectivedissemination and archiving. In addition,electronic open-access databases maintained byregulatory bodies, research societies or researchsponsors are increasingly important sources ofpublished and unpublished studies.It seems still reasonable to claim that ‘the idealsolution to publication bias is the prospective,universal registration of all studies at theirinception’. 2 Voluntary registration of trials is usuallyincomplete. The most important developmentwas initiated by the compulsory policy of trialregistration adopted by the InternationalCommittee of Medical Journals in 2004. 10 Effortsso far have focused on the registration, publicationand disclosure of confirmatory phase III/IV trialsdue to the perceived immediate consequences. Inspite of the greater risk of publication bias, therehave been considerable difficulties facing theprospective registration of and disclosure of datafrom unpublished basic research, observationalstudies and early stage exploratory clinical trials.Trials registers will only be helpful in reducingpublication bias if systematic reviewers includethe registries in their search strategies and resultsof trials are accessible. According to findingspresented in Chapter 9 (see Table 4), only 18% ofthe treatment reviews and few reviews of diagnosticand epidemiological studies searched prospectiveresearch registers.Certain types of dissemination bias, such asdatabase bias, duplicate publication bias,citation bias and media attention bias, could bedealt with by following approaches adopted instandard systematic reviews (Table 11). The risk ofpublication bias may be assessed in a systematicreview according to certain risk factors associatedwith publication bias, although the method hasnot been adequately investigated in empiricalstudies (Chapter 8). Funnel plot and relatedstatistical methods have been widely used to assesspublication bias in systematic reviews. Becausethe interpretation of a funnel plot can often bemisleading, some recommendations have beenrecently proposed about when, and how to usethe funnel plot and related statistical tests. 469,470However, these recommendations were based on© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.limited and fast-changing evidence and have notbeen empirically validated.Many complex statistical methods have beendeveloped to detect or even adjust for assumedpublication bias in meta-analysis. But they havenever or very rarely been used in practice, possiblybecause of their complexity and the lack of userfriendlysoftware. More importantly, the usefulnessof any statistical methods, simple or complex,may be very limited in typical systematic reviewsor meta-analyses (Chapter 9). It is now generallyrecognised that sophisticated modelling methodsmay be used to conduct sensitivity analyses, ratherthan to provide an adjusted estimate, althoughthe usefulness of such sensitivity analyses is stillunclear.Dealing with publicationbias in published systematicreviewsThe 2000 HTA report on publication bias surveyeda sample of 193 systematic reviews published in1996, and concluded that the issue of publicationbias was largely being ignored, and methods todeal with publication bias were rarely used inthese reviews. 2 This updated review found that in300 systematic reviews published in 2006 therehave been some improvements in dealing withpublication bias (Chapter 9). Compared withreviews published in 1996, recently publishedreviews made greater efforts to locate and includegrey literature or unpublished studies and studiespublished in non-English languages. In addition,more recently published reviews used methods toassess publication bias in systematic reviews.The previous report found that for publications in1996 the problems of publication bias were moreoften dealt with in meta-analyses than in narrativereviews. This phenomenon is also observed insystematic reviews published in 2006, which maybe due to lack of methods that can be used innarrative reviews.We observed some differences between differentcategories of systematic reviews published in2006. Grey literature, unpublished studiesor non-English-language studies were morelikely to be searched for in reviews of treatmentefficacy or diagnostic accuracy than in reviewsof epidemiological studies. However, the riskof publication bias was less likely to be tested inreviews of treatment and diagnosis compared83

DiscussionTable 11 Publication-related biases and methods to deal with these biasesDissemination biasMethods/approachesNon-publication of‘negative’ resultsTime lag biasOutcome reportingbiasProspective registration of studies, publication of research protocolsRight to publication Open access policy/regulation, improved research funders’guidelinesEndorsement of sound reportingguidelines for journal publicationDisclosure of unpublished studies or dataSystematic literature review – Searching for and including greyliterature, unpublished studies/data, andnon-English-language studiesAssessing risk of publication bias in systematic reviews– Considering risk factors, funnel plot andrelated testsContacting authors for missing data or clarificationIndividual patient data meta-analysis Updating systematic reviewsConfirmatory studiesGrey literature biasLanguage biasDatabase biasDuplicate biasCitation biasMedia biaswith reviews of epidemiological studies (Chapter8). These differences between different types ofreviews may be caused by the different availabilityof sources of grey literature or unpublishedstudies, and perceived risk of publication bias indifferent types of primary studies. For example,many initiatives have been taken to prevent biasedpublication of clinical trials and there are somegood sources of grey literature and unpublishedtrials. At the same time, the limitations of availablemethods to test publication bias in systematicreviews have been more widely recognised.Therefore, the authors of reviews of treatmentefficacy may focus their efforts on the completenessof literature search, rather than on the assessmentof publication bias. However, there have beenno great efforts to prevent publication bias inepidemiological studies. No good databases ofunpublished epidemiological studies could beused by the authors of reviews of epidemiologicalstudies. In view of the great risk of publicationbias, authors of reviews of epidemiological studiesmay have to rely on the available methods to testpublication bias, even if the results of such tests areoften difficult to interpret.84

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Implications for researchersand decision-makers• There is little doubt that dissemination ofresearch findings is likely to be a biasedprocess, although the actual impact of suchbias is often uncertain, depending on specificcircumstances. Therefore, the potentialproblem of publication and related bias shouldbe taken into consideration by all who areinvolved in evidence-based decision-making.• Decision-makers, research funders and RECsat the national and international level shouldcontinue to support the development ofprospective research registration, and theimplementation of research open-access policy.• Practical and sound reporting guidelinesshould be endorsed by journals, and authorsshould report all measured outcomes in theirstudies.• Whenever possible, a thorough literaturesearch should be conducted in systematicreviews to identify all relevant studies. Registersof clinical trials and available databases ofunpublished studies should be routinelysearched for relevant clinical trials.• The impact of grey literature or studiespublished in languages other than Englishmay be non-significant in many cases.However, the exclusion of grey literature ornon-English-language studies may introducebias in a systematic review, particularly in thefield of complementary medicine. Therefore,systematic reviews should not routinely excludeunpublished studies or conference abstracts.The quality of unpublished studies or abstractsshould be assessed using the same criteria asfor formally published studies.• Outcome reporting bias has been confirmedby new evidence and should be seriouslyconsidered in systematic reviews. Whenrelevant studies cannot be included owing toa lack of data on relevant outcomes, originalauthors should be contacted to clarify whetherthe outcome was actually measured, and toobtain data on missing outcomes.• Funnel plot and related statistical tests can beused to detect ‘small study effect’. However, itis usually impossible to separate the influenceof factors other than publication bias on theobserved association between the estimatedeffects and sample sizes across studies in metaanalysis.The inappropriate interpretation ofthe funnel plot and its related tests may bereduced by following recent recommendationsin the updated Cochrane Handbook for SystematicReviews. 469• The risk of publication bias should bequalitatively assessed according to suspectedfactors associated with publication bias,including small sample size, small effectsize, the shape of a funnel plot, the potentialnumber of studies that may have beenconducted, conflicting interests of investigatorsor research sponsors, and any other director indirect evidence. The estimated risk ofpublication bias should be incorporated intothe review’s conclusions.• Large-scale confirmatory studies becomenecessary after a systematic review has reporteda clinically significant finding, but publicationbias cannot be safely excluded.Recommendations forfuture research• Further empirical research is needed toevaluate the effect of prospective registrationof studies, open-access policy and improvedpublication guidelines in the prevention ofresearch dissemination bias.• The role of the developments in computerscience and information technology forthe prevention and reduction of researchdissemination bias needs to be investigated byfurther research.• There is still a lack of evidence aboutthe impact of publication bias on healthdecision-making and the outcomes of patientmanagement. Further research is required inthis area.• Many systematic reviews still have to dependupon studies identified retrospectively from thepublished literature, particularly in systematicreviews of basic research and observationalstudies. Further research is required to developmethods that can be used qualitatively ornarratively to assess the risk of publication biasin systematic reviews.• Many available statistical methods to testpublication bias have never, or very rarely, beenused in systematic reviews. Further researchshould focus on the practical application ofthese statistical methods.85© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8AcknowledgementsWe would like to thank Julie Reynolds forproviding secretarial support to this review.Contribution of authorsFujian Song (Reader in Research Synthesis)developed the review protocol, and was involvedin the literature search, data extraction anddrafting of the report. Sheetal Parekh (ResearchAssociate) searched the literature, extracted datafrom included studies, and was involved in thepreparation of draft chapters. Lee Hooper (SeniorLecturer in Research Synthesis and Nutrition)and Yoon Loke (Senior Lecturer in ClinicalPharmacology) were involved in the developmentof the protocol, checked the data extraction andcommented on the draft. Jon Ryder (InformationOfficer) designed the literature search strategy,searched the literature, drafted sections, extracteddata and commented on the draft. Alex Sutton(Reader in Medical Statistics) was involved inthe development of the protocol, providedmethodological support, and commented onthe draft. Caroline Hing (Consultant in Traumaand Orthopaedics) extracted and checked data,drafted a chapter, and commented on the draft.Shing Kwok (Research Assistant) and ChunPang (Research Assistant) extracted and checkeddata, were involved in drafting a chapter, andcommented on the draft. Ian Harvey (Professorof Epidemiology and Public Health) was involvedin the development of the protocol, providedmethodological support, and commented on thedraft.87© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8References1. Begg CB, Berlin JA. Publication bias; a problemin interpreting medical data. J Roy Stat Soc A1988;151:445–63.2. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ.Publication and related biases. Health Technol Assess2000;4:1–115.3. Egger M, Juni P, Bartlett C, Holenstein F, SterneJ. How important are comprehensive literaturesearches and the assessment of trial quality insystematic reviews? Empirical study. Health TechnolAssess 2003;7:1–76.4. Moher D, Pham B, Lawson ML, Klassen TP. Theinclusion of reports of randomised trials publishedin languages other than English in systematicreviews. Health Technol Assess 2003;7:1–90.5. Chan A-W, Altman DG. Identifying outcomereporting bias in randomised trials on PubMed:review of publications and survey of authors. BMJ2005;330:753.6. Chan A-W, Hrobjartsson A, Haahr MT, GotzschePC, Altman DG. Empirical evidence for selectivereporting of outcomes in randomized trials:comparison of protocols to published articles. JAMA2004;291:2457–65.7. Chan A-W, Krleza-Jeri K, Schmid I, AltmanDG. Outcome reporting bias in randomizedtrials funded by the Canadian Institutes ofHealth Research [see comment]. Can Med Assoc J2004;171:735–40.8. Deeks JJ, Macaskill P, Irwig L. The performanceof tests of publication bias and other samplesize effects in systematic reviews of diagnostictest accuracy was assessed [see comment]. J ClinEpidemiol 2005;58:882–93.9. Peters JL, Sutton AJ, Jones DR, Abrams KR,Rushton L. Comparison of two methods todetect publication bias in meta-analysis. JAMA2006;295:676–80.10. DeAngelis CD, Drazen JM, Frizelle FA, Haug C,Hoey J, Horton R, et al. Clinical trial registration:a statement from the International Committee ofMedical Journal Editors. Jama 2004;292:1363–4.11. Rosenthal R. The ‘file drawer problem’ andtolerance for null results. Psychol Bull 1979;86:638–41.12. Begg CB, Berlin JA. Publication bias anddissemination of clinical research. J Natl Cancer Inst1989;81:107–15.13. Dickersin K. The existence of publicationbias and risk factors for its occurrence. JAMA1990;263:1385–9.14. Smith R. Editorial: What is publication? BMJ1999;318:142.15. Edwards S, Liford RJ, Kiauka S. Different types ofsystematic review in health services research. InBlack N, Brazier H, Fitzpatrick R, Reeves B, editors.Health services research methods: a guide to best practice.London: BMJ Books; 1998.16. Higgins JP, Thompson SG, Deeks JJ, Altman DG.Measuring inconsistency in meta-analyses. BMJ2003;327:557–60.17. Sohn D. Publications bias and the evaluation ofpsychotherapy efficacy in reviews of the researchliterature. Clin Psychol Rev 1996;16:147–56.18. Sterling T. Publication decisions and their possibleeffects on inferences drawn from tests of significance– or vice versa. Am Stat Assoc J 1959;54:30–4.19. Sterling TD, Rosenbaum WL, Weinkam JJ.Publication decisions revisited – the effect of theoutcome of statistical tests on the decision topublish and vice-versa. Am Stat 1995;49:108–12.20. Dickersin K, Min YI, Meinert CL. Factorsinfluencing publication of research results. Followup of applications submitted to two institutionalreview boards. JAMA 1992;267:374–8.21. Dickersin K, Min. YI. NIH clinical trials andpublication bias. Online J Curr Clin Trials 1993. Doc.No. 50(3).22. Easterbrook PJ, Berlin JA, Gopalan R, MatthewsDR. Publication bias in clinical research. Lancet1991;337:867–72.89© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

References9023. Ioannidis J. Effect of the statistical significance ofresults on the time to completion and publication ofrandomized efficacy trials. JAMA 1998;279:281–6.24. Stern JM, Simes RJ. Publication bias: evidence ofdelayed publication in a cohort study of clinicalresearch projects. BMJ 1997;315:640–5.25. Dickersin K. How important is publication bias? Asynthesis of available data. Aids Educ Prev 1997;9(1SA):15–21.26. Bardy AH. Bias in reporting clinical trials. Brit J ClinPharmaco 1998;46:147–50.27. Cronin E, Sheldon T. Factors influencing thepublication of health research. Int J Technol Assess2004;20:351–5.28. Decullier E, Lheritier V, Chapuis F. Fate ofbiomedical research protocols and publication biasin France: retrospective cohort study [see comment].BMJ 2005;331:19.29. Decullier E, Chapuis F. Impact of funding onbiomedical research: a retrospective cohort study.BMC Public Health 2006;6:165.30. Misakian AL, Bero LA. Publication bias andresearch on passive smoking: comparison ofpublished and unpublished studies. Jama1998;280:250–3.31. Wormald R, Bloom J, Evans J, Oldfled K.Publication bias in eye trials. In Second InternationalConference Scientific Basis of Health Science and 5thCochrane Colloquium. Amsterdam, The Netherlands:Cochrane Collaboration; 1997.32. Zimpel T, Windeler J. [Publications of dissertationson unconventional medical therapy and diagnosisprocedures – a contribution to ‘publication bias’].Forsch Komp Klas Nat 2000;7:71–4.33. Blumle A, Antes G, Schumacher M, Just H, vonElm E. Clinical research projects at a Germanmedical faculty: follow-up from ethical approvalto publication and citation by others. J Med Ethics2008;34:e20.34. Druss BG, Marcus SC. Tracking publicationoutcomes of National Institutes of Health grants.Am J Med 2005;118:658–63.35. Cooper H, Charlton K. Finding the missing science:the fate of studies submitted for review by a humansubjects committee. Psychol Methods 1997;2:447–52.36. Hahn S, Williamson PR, Hutton JL. Investigationof within-study selective reporting in clinicalresearch: follow-up of applications submitted to alocal research ethics committee. J Eval Clin Pract2002;8:353–9.37. Hall R, de Antueno C, Webber A. Publication biasin the medical literature: a review by a CanadianResearch Ethics Board. Can J Anaesth 2007;54:380–8.38. Pich J, Carne X, Arnaiz J-A, Gomez B, Trilla A,Rodes J. Role of a research ethics committee infollow-up and publication of results [see comment].Lancet 2003;361:1015–6.39. von Elm E, Rollin A, Blumle A, Huwiler K, WitschiM, Egger M. Publication and non-publication ofclinical trials: longitudinal study of applicationssubmitted to a research ethics committee. Swiss MedWkly 2008;138:197–203.40. Dickersin K, Min YI. Publication bias: the problemthat won’t go away. Ann N Y Acad Sci 1993;703:135–46.41. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW,Cronin E, et al. Systematic review of the empiricalevidence of study publication bias and outcomereporting bias. PLoS ONE 2008;3(8):e3081.42. Lee K, Bacchetti P, Sim I. Publication of clinicaltrials supporting successful new drug applications: aliterature analysis. PLoS Med 2008;5(9):e191.43. Melander H, Ahlqvist-Rastad J, Meijer G,Beermann B. Evidence b(i)ased medicine – selectivereporting from studies sponsored by pharmaceuticalindustry: review of studies in new drug applications.BMJ 2003;326:1171–3.44. Rising K, Bacchetti P, Bero L. Reporting biasin drug trials submitted to the Food and DrugAdministration: review of publication andpresentation. PLoS Med 2008;5(11):e217; discussione217.45. Turner EH, Matthews AM, Linardatos E, Tell RA,Rosenthal R. Selective publication of antidepressanttrials and its influence on apparent efficacy. N Engl JMed 2008;358:252–60.46. van Luijn JCF, Stolk P, Gribnau FWJ, LeufkensHGM. Gap in publication of comparativeinformation on new medicines. Brit J Clin Pharmacol2008;65:716–22.47. Callaham ML, Wears RL, Weber EJ, BartonC, Young G. Positive-outcome bias and otherlimitations in the outcome of research abstractssubmitted to a scientific meeting. JAMA1998;280:254–7. Erratum appears in JAMA1998;280:1232.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 848. Chalmers I, Adams M, Dickersin K, HetheringtonJ, Tamow-Mordi W, Meinert C, et al. A cohort studyof summary reports of controlled trials. JAMA1990;263:1401–5.49. Cheng K, Preston C, Ashby D, O’Hea U, Smyth RL.Time to publication as full reports of abstracts ofrandomized controlled trials in cystic fibrosis. PediatrPulm 1998;26:101–5.50. DeBellefeuille C, Morrison CA, Tannock IF. Thefate of abstracts submitted to a cancer meeting:factors which influence presentation and subsequentpublication. Ann Oncol 1992;3:187–91.51. Landry VL. The publication outcome for the paperspresented at the 1990 ABA conference. J Burn CareRehabil 1996;17:23A–6A.52. Loep M, Kleijnen J. Full publication of abstractsinitially published in the Netherlands Journalof Medicine, 1999. Data obtained by personalcommunication with the author, Jo Kleijnen, Centrefor Reviews and Dissemination, University of York,UK, June 1999.53. Petticrew M, Gilbody S, Song F. Lost information?The fate of papers presented at the 40th Society forSocial Medicine Conference. J Epidemiol Commun H1999;53:442–3.54. Scherer RW, Dickersin K, Langenberg P. Fullpublication of results initially presented in abstracts.A meta-analysis JAMA 1994;272:158–62. Publishederratum appears in JAMA 1994;272:1410.55. Scherer RW, Langenberg P, von Elm E. Fullpublication of results initially presented in abstracts.Cochrane Database Syst Rev 2007;2:MR000005.DOI:10.1002/14651858.MR000005.pub3.56. Akbari-Kamrani M, Shakiba B, Parsian S. Transitionfrom congress abstract to full publication for clinicaltrials presented at laser meetings. Laser Med Sci2008;23:295–9.57. Brazzelli M, Lewis SC, Deeks JJ, Sandercock PAG.No evidence of bias in the process of publication ofdiagnostic accuracy studies in stroke submitted asabstracts. J Clin Epidemiol 2009;62:425–30.58. Castillo J, Garcia Guasch R, Cifuentes I. Fate ofabstracts from the Paris 1995 European Societyof Anaesthesiologists meeting. Eur J Anaesthesiol2002;19:888–93.59. Delamere F, Williams H. To what extent areconference abstracts reporting randomisedcontrolled trials of skin diseases publishedsubsequently? [abstract]. XIII Cochrane Colloquium,Oct 22–26 2005; Melbourne, Australia.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.60. Eloubeidi MA, Wade SB, Provenzale D. Factorsassociated with acceptance and full publicationof GI endoscopic research originally published inabstract form [see comment]. Gastrointest Endosc2001;53:275–82.61. Evers JL. Publication bias in reproductive research.Hum Reprod 2000;15:2063–6.62. Glick N, MacDonald I, Knoll G, Brabant A,Gourishankar S. Factors associated with publicationfollowing presentation at a transplantation meeting.Am J Transplant 2006;6:552–6.63. Ha TH, Yoon DY, Goo DH, Chang SK, Seo YL, YunEJ, et al. Publication rates for abstracts presented byKorean investigators at major radiology meetings.Korean J Radiol 2008;9:303–11.64. Halpern SH, Palmer C, Angle P, Tarshis J. Publishedabstracts in obstetrical anesthesia: full publicationrates and data reliability. Anesthesiology 2001;94:A69.65. Harris IA, Mourad M, Kadir A, Solomon MJ,Young JM. Publication bias in abstracts presentedto the annual meeting of the American Academyof Orthopaedic Surgeons. J Orthopaed Surg (HongKong) 2007;15:62–6.66. Harris IA, Mourad MS, Kadir A, Solomon MJ,Young JM. Publication bias in papers presentedto the Australian Orthopaedic Association AnnualScientific Meeting. Aust NZ J Surg 2006;76:427–31.67. Hashkes PJ, Uziel Y. The publication rateof abstracts from the 4th Park City PediatricRheumatology meeting in peer-reviewed journals:what factors influenced publication? J Rheumatol2003;30:597–602.68. Kiroff GK. Publication bias in presentations tothe Annual Scientific Congress. Aust NZ J Surg2001;71:167–71.69. Klassen TP, Wiebe N, Russell K, Stevens K, HartlingL, Craig WR, et al. Abstracts of randomizedcontrolled trials presented at the society forpediatric research meeting: an example ofpublication bias [comment]. Arch Pediat Adol Med2002;156:474–9.70. Krzyzanowska MK, Pintilie M, Tannock IF. Factorsassociated with failure to publish large randomizedtrials presented at an oncology meeting. Jama2003;290:495–501.71. Peng PH, Wasserman JM, Rosenfeld RM. Factorsinfluencing publication of abstracts presented at theAAO-HNS Annual Meeting. Otolaryng Head Neck2006;135:197–203.91

References9272. Sanossian N, Ohanian AG, Saver JL, Kim LI,Ovbiagele B. Frequency and determinants ofnonpublication of research in the stroke literature.Stroke 2006;37:2588–92.73. Smith WA, Cancel QV, Tseng TY, Sultan S, Vieweg J,Dahm P. Factors associated with the full publicationof studies presented in abstract form at the annualmeeting of the American Urological Association. JUrol 2007;177:1084–8; discussion 1088–9.74. Timmer A, Hilsden RJ, Cole J, Hailey D, SutherlandLR. Publication bias in gastroenterological research– a retrospective cohort study based on abstractssubmitted to a scientific meeting. BMC MedicalResearch Methodology 2002;2(1):7.75. Vecchi S, Belleudi V, Amato L, Davoli M, PerucciCA. Does direction of results of abstracts submittedto scientific conferences on drug addiction predictfull publication? BMC Med Res Methodol 2009;9:23.76. Zamakhshary M, Abuznadah W, Zacny J,Giacomantonio M. Research publication in pediatricsurgery: a cross-sectional study of papers presentedat the Canadian Association of Pediatric Surgeonsand the American Pediatric Surgery Association. JPediatr Surg 2006;41:1298–301.77. Zaretsky Y, Imrie K. The fate of phase III trialabstracts presented at the American Society ofHematology [abstract]. Blood 2002;100:185a.78. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P,Bero LA. Predictors of publication: characteristicsof submitted manuscripts associated withacceptance at major biomedical journals. Med J Aust2006;184:621–6.79. Lynch JR, Cunningham MRA, Warme WJ,Schaad DC, Wolf FM, Leopold SS. Commerciallyfunded and United States-based research is morelikely to be published; good-quality studies withnegative outcomes are not. J Bone Joint Surg Am2007;89:1010–8.80. Okike K, Kocher MS, Mehlman CT, HeckmanJD, Bhandari M. Publication bias in orthopaedicresearch: an analysis of scientific factors associatedwith publication in the Journal of Bone and JointSurgery (American Volume). J Bone Joint Surg Am2008;90:595–601.81. Olson CM, Rennie D, Cook D, Dickersin K,Flanagin A, Hogan JW, et al. Publication bias ineditorial decision making. Jama 2002;287:2825–8.82. Dickersin K, Olson CM, Rennie D, Cook D,Flanagin A, Zhu Q, et al. Association between timeinterval to publication and statistical significance.Jama 2002;287:2829–31.83. Hopewell S, Loudon K, Clarke MJ, Oxman AD,Dickersin K. Publication bias in clinical trials dueto statistical significance or direction of trial results.Cochrane Database Syst Rev 2009;1:MR000006.DOI:10.1002/14651858.MR000006.pub3.84. Dickersin K, Ssemanda E, Mansell C, RennieD. What do the JAMA editors say when theydiscuss manuscripts that they are considering forpublication? Developing a schema for classifyingthe content of editorial discussion. BMC Med ResMethodol 2007;7:44.85. Calnan M, Smith GD, Sterne JAC. The publicationprocess itself was the major cause of publicationbias in genetic epidemiology. J Clin Epidemiol2006;59:1312–8.86. Hahn S, Garner P, Williamson P. Are systematicreviews taking heterogeneity into account? Ananalysis from the Infectious Diseases Module of theCochrane Library. J Eval Clin Pract 2000;6:231–3.87. Phillips CV. Publication bias in situ. BMC Med ResMethodol 2004;4:20.88. Pocock SJ, Hughes MD, Lee RJ. Statistical problemsin the reporting of clinical trials. A survey of threemedical journals. N Engl J Med 1987;317:426–32.89. Tannock IF. False-positive results in clinical trials:multiple significance tests and the problemof unreported comparisons. J Natl Cancer Inst1996;88:206–7.90. West RRJ, D.A. Publication bias in statisticaloverview of trials: example of psychologicalrehabilitation following myocardial infarction[abstract]. The 2nd International Conference on theScientific Basis of Health Services and 5th CochraneColloquium, 8–12 October 1997, Amsterdam.91. Ghersi D, Clarke M, Simes J. Selective reporting ofthe primary outcomes of clinical trials: a follow-upstudy [abstract]. 14th Cochrane Colloquium, 23–26October 2006, Dublin, Ireland.92. McCormack K, Grant A, Scott N. Value of updatinga systematic review in surgery using individualpatient data. Br J Surg 2004;91:495–9.93. Bekkering GE, Harris RJ, Thomas S, MayerAM, Beynon R, Ness AR, et al. How much of thedata published in observational studies of theassociation between diet and prostate or bladdercancer is usable for meta-analysis? Am J Epidemiol2008;167:1017–26.94. Furukawa TA, Watanabe N, Omori IM, MontoriVM, Guyatt GH. Association between unreportedoutcomes and effect size estimates in Cochranemeta-analyses. JAMA 2007;297:468–70.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 895. Williamson PR, Gamble C. Identification andimpact of outcome selection bias in meta-analysis.Stat Med 2005;24:1547–61.96. Scharf O, Colevas AD. Adverse event reporting inpublications compared with sponsor database forcancer clinical trials. J Clin Oncol 2006;24:3933–8.97. Kyzas PA, Loizou KT, Ioannidis JPA. Selectivereporting biases in cancer prognostic factor studies[see comment]. J Natl Cancer I 2005;97:1043–55.98. Kavvoura FK, Liberopoulos G, Ioannidis JPA.Selection in reported epidemiological risks: anempirical assessment. PLoS Medicine/Public Library ofScience 2007;4(3):e79.99. Williamson P, Gamble C, Jacoby A, Altman D.Understanding the process and impact of withinstudyselective reporting bias [abstract]. 14thCochrane Colloquium, 23–26 October 2006, Dublin,Ireland.100. Jadad AR, Rennie D. The randomized controlledtrial gets a middle-aged checkup. JAMA1998;279:319–20.101. Simes RJ. Confronting publication bias: a cohortdesign for meta analysis. Stat Med 1987;6:11–29.102. Liebeskind DS, Kidwell CS, Sayre JW, Saver JL.Evidence of publication bias in reporting acutestroke clinical trials. Neurology 2006;67:973–9.103. Soares H, Kumar A, Clarke M, Djulbegovic B. Howlong does it take to publish a high quality trial inoncology? [abstract]. 13th Cochrane Colloquium, 22–26 October 2005, Melbourne, Australia.104. Min YI, Dickersin K. Rate of full publication andtime to full publication of observational studies[abstract]. Am J Epidemiol 2005;161(Suppl.):S76.105. Callaham ML, Weber E, Young G, Wears R, BartonC. Time to publication of studies was not affectedby whether results were positive [comment]. BMJ1998;316:1536.106. Rothwell PM, Robertson G. Meta-analyses ofrandomised controlled trials [letter]. Lancet1997;350:1181–2.107. Song F, Gilbody S. Bias in meta-analysis detectedby a simple, graphical test. Increase in studies ofpublication bias coincided with increasing use ofmeta-analysis [comment]. BMJ 1998;316:471.108. Gehr BT, Weiss C, Porzsolt F. The fading of reportedeffectiveness. A meta-analysis of randomisedcontrolled trials. BMC Med Res Methodol 2006;6:25.109. Vaitkus PT, Brar C. N-acetylcysteine in theprevention of contrast-induced nephropathy:publication bias perpetuated by meta-analyses [seecomment]. Am Heart J 2007;153:275–80.110. Jennions MD, Moller AP. Relationships fadewith time: a meta-analysis of temporal trends inpublication in ecology and evolution. Proceedings ofthe Royal Society of London Series B: Biological Sciences2002;269(1486):43–48.111. Leimu R, Koricheva J. Cumulative metaanalysis:a new tool for detection of temporaltrends and publication bias in ecology. P R Soc B2004;271:1961–6.112. Ioannidis JP, Trikalinos TA. Early extremecontradictory estimates may appear in publishedresearch: the Proteus phenomenon in moleculargenetics research and randomized trials. J ClinEpidemiol 2005;58:543–9.113. Hopewell S, Clarke M, Stewart L, Tierney J.Time to publication for results of clinical trials.Cochrane Database Syst Rev 2007;2:MR000011.DOI:10.1002/14651858.MR000011.pub3.114. Auger CP. Information sources in grey literature. 4thedn. London/Melbourne/Munich/New Providence,NJ: Bowker-Saur; 1998.115. Helmer D. Grey literature. In Etext on HealthTechnology Assessment (HTA) information resources.URL: www.nlm.nih.gov/archive//2060905/nichsr/ehta/chapter10.html. 2004.116. Cook DJ, Guyatt GH, Ryan G, Clifton J,Buckingham L, Willan A, et al. Shouldunpublished data be included in meta analyses?Current convictions and controversies. JAMA1993;269:2749–53.117. Taus C, Pucci E, Giuliani G, Telaro E, Pistotti V. Theuse of ‘grey literature’ in a sub-set of neurologicalCochrane reviews. 7th Cochrane Colloquium, 5–9October 1999, Rome, Italy.118. Tetzlaff J, Moher D, Pham B, Altman D. Surveyof views on including grey literature in systematicreviews [abstract]. 14th Cochrane Colloquium, 23–26October 2006, Dublin, Ireland.119. McAuley L, Pham B, Tugwell P, Moher D. Doesthe inclusion of grey literature influence estimatesof intervention effectiveness reported in metaanalyses?Lancet 2000;356:1228–31.120. Burdett S, Stewart LA, Tierney JF. Publication biasand meta-analyses: a practical example. Int J TechnolAssess 2003;19:129–34.93© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

References94121. Ferguson D, Laupacis A, Salmi LR, McAlister FA,Huet C. What should be included in meta-analyses?An exploration of methodological issues using theISPOT meta-analyses [review]. Int J Technol Assess2000;16:1109–19.122. Hopewell S. Impact of grey literature on systematicreviews of randomised trials. D Phil thesis. Universityof Oxford, 2004.123. Hopewell S, McDonald S, Clarke M, Egger M. Greyliterature in meta-analyses of randomized trials ofhealth care interventions. Cochrane Database SystRev 2007;2:MR000010. DOI:10.1002/14651858.MR000010.pub3.124. Glass GV, McGaw B, Smith ML. Meta-analysis insocial research. London: Sage Publications; 1981.125. Smith M. Publication bias and meta-analysis. EvalEduc 1980;4:22–4.126. White KR. The relation between socioeconomicstatus and academic achievement. Psychol Bull1982;91:461–81.127. Simes RJ. Publication bias: the case for aninternational registry of clinical trials. J Clin Oncol1986;4:1529–41.128. Detsky AS, Baker JP, O’Rourke K, Goel V.Perioperative parenteral nutrition: a meta-analysis.Ann Intern Med 1987;107:195–203.129. Devine EC. Empirical assessment of publicationbias: lessons from two meta-analyses. 7th CochraneColloquium, 5–9 October 1999, Rome, Italy.130. MacLean C, Morton S, Straus W, Ofman J, RothE, Shekelle P. Unpublished data from UnitedStates Food and Drug Administration New DrugApplication Reviews: How do they compare topublished data when assessing NonsteroidalAntiinflammatory Drug (NSAm) associateddyspepsia? 7th Cochrane Colloquium, 5–9 October1999, Rome, Italy.131. Jeng GT, Scott JR, Burmeister LF. A comparison ofmeta-analytic results using literature vs individualpatient data. Paternal cell immunization forrecurrent miscarriage. JAMA 1995;274:830–6.132. Macleod MR, O’Collins T, Howells DW, DonnanGA. Pooling of animal experimental data revealsinfluence of study design and publication bias.Stroke 2004;35:1203–8.133. Man-Son-Hing M, Wells G, Lau A. Quinine fornocturnal leg cramps: a meta-analysis includingunpublished data. J Gen Intern Med 1998;13:600–6.134. McLeod BD, Weisz JR. Using dissertations toexamine potential bias in child and adolescentclinical trials. J Consult Clin Psych 2004;72:235–51.135. MacLean CH, Morton SC, Ofman JJ, Roth EA,Shekelle PG, Southern California Evidence BasedPractice Center. How useful are unpublished datafrom the Food and Drug Administration in metaanalysis?J Clin Epidemiol 2003;56:44–51.136. Whittington CJ, Kendall T, Fonagy P, Cottrell D,Cotgrove A, Boddington E. Selective serotoninreuptake inhibitors in childhood depression:systematic review of published versus unpublisheddata. Lancet 2004;363:1341–5.137. Wallace AE, Neily J, Weeks WB, Friedman MJ. Acumulative meta-analysis of selective serotoninreuptake inhibitors in pediatric depression: didunpublished studies influence the efficacy/safetydebate? J Child Adult Psychop 2006;16:37–58.138. Batt K, Fox Rushby JA, Castillo Riquelme M. Thecosts, effects and cost-effectiveness of strategies toincrease coverage of routine immunizations in lowandmiddle-income countries: systematic review ofthe grey literature. Bull WHO 2004;82:689–96.139. Davey-Smith G, Egger M. Meta-analysis:unresolved issues and future developments. BMJ1998;316:221–5.140. Winkmann G, Schlutius S, Schweim HG. Publicationlanguages of impact factor journals and of medicalbibliographic databanks. Klinische Monatsblatter furAugenheilkunde 2002;219:65–71.141. Vandenbrouche J. On not being born a nativespeaker of English. BMJ 1989;298:1461–2.142. Herrera AJ. Language bias discredits the peerreviewsystem. Nature 1999;397:467.143. Coates R, Sturgeon B, Bohannan J, Pasini E.Language and publication in ‘CardiovascularResearch’ articles. Cardiovasc Res 2002;53:279–85.144. Nylenna M, Riis P, Karlsson Y. Multiple blindedreviews of the same two manuscripts. Effects ofreferee characteristics and publication language.JAMA 1994;272:149–51.145. Moher D, Fortin P, Jadad AR, Juni P, Klassen T,Le Lorier J, et al. Completeness of reporting oftrials published in languages other than English:implications for conduct and reporting of systematicreviews. Lancet 1996;347:363–6.146. Junker CA. Adherence to published standards ofreporting: a comparison of placebo-controlledtrials published in English or German. Jama1998;280:247–9.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8147. Gregoire G, Derderian F, Le Lorier J. Selectingthe language of the publications included in ameta-analysis: is there a Tower of Babel bias? J ClinEpidemiol 1995;48:159–63.148. Egger M, Zellweger-Zahner T, Schneider M,Junker C, Lengeler C, Antes G. Language bias inrandomised controlled trials published in Englishand German. Lancet 1997;350:326–9.149. Egger M, Davey Smith G, Schneider M, MinderC. Bias in meta-analysis detected by a simple,graphical test. BMJ 1997;315:629–34.150. Heres S, Wagenpfeil S, Hamann J, Kissling W,Leucht S. Language bias in neuroscience – is theTower of Babel located in Germany? Eur Psychiat2004;19:230–2.151. Moher D, Pham B, Klassen TP, Schulz KF, Berlin JA,Jadad AR, et al. What contributions do languagesother than English make on the results of metaanalyses?J Clin Epidemiol 2000;53:964–72.152. Pham B, Klassen TP, Lawson ML, Moher D.Language of publication restrictions in systematicreviews gave different results depending onwhether the intervention was conventional orcomplementary. J Clin Epidemiol 2005;58:769–76.Erratum appears in J Clin Epidemiol 2006;59:216.153. Juni P, Holenstein F, Sterne J, Bartlett C, EggerM. Direction and impact of language bias in metaanalysesof controlled trials: empirical study [seecomment]. Int J Epidemiol 2002;31:115–23.154. Brooks TA. Private acts and public objects: aninvestigation of citer motivations. J Am Soc Inform Sci1985;36:223–9.155. Shadish WR, Tolliver D, Gray M, SenGupta SK.Author judgements about works they cite: threestudies from psychology journals. Soc Stud Sci1995;25:477–98.156. Christensenszalanski JJJ, Beach LR. The citationbias – fad and fashion in the judgment and decisionliterature. Am Psychol 1984;39:75–8.157. Gotzsche PC. Reference bias in reports of drugtrials. Br Med J (Clin Res Ed) 1987;295:654–6.158. Ravnskov U. Quotation bias in reviews of the dietheartidea. J Clin Epidemiol 1995;48:713–9.159. Hutchison BG, Oxman AD, Lloyd S.Comprehensiveness and bias in reporting clinicaltrials. Study of reviews of pneumococcal vaccineeffectiveness. Can Fam Physician 1995;41:1356–60.160. Song F, Landes DP, Glenny AM, Sheldon TA.Prophylactic removal of impacted third molars:© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.an assessment of published reviews. Br Dent J1997;182:339–46.161. Callaham M, Wears RL, Weber E. Journal prestige,publication bias, and other characteristics associatedwith citation of published studies in peer-reviewedjournals. Jama 2002;287:2847–50.162. Chapman S, Ragg M, McGeechan K. Citation biasin reported smoking prevalence in people withschizophrenia. Aust NZ J Psychiat 2009;43:277–82.163. Kjaergard LL, Gluud C. Citation bias of hepatobiliaryrandomized clinical trials. J Clin Epidemiol2002;55:407–10.164. Nieminen P, Rucker G, Miettunen J, Carpenter J,Schumacher M. Statistically significant papers inpsychiatry were cited more often than others. J ClinEpidemiol 2007;60:939–46.165. Schmidt LM, Gotzsche PC. Of mites and men:reference bias in narrative review articles: asystematic review. J Fam Practice 2005;54:334–8.166. Robins RW, Craik KH. Is there a citation bias in thejudgment and decision literature. Organ Behav HumDec 1993;54:225–44.167. Johnson C. Repetitive, duplicate, and redundantpublications: a review for authors and readers. JManip Physiol Ther 2006;29:505–9.168. Jefferson T. Redundant publication in biomedicalsciences: scientific misconduct or necessity? Sci Eng1998;4:135–40.169. Angell M, Relman AS. Redundant publication. NewEngl J Med 1989;320:1212–3.170. Fulginiti VA. Unfortunately, more on duplicatepublication. Am J Dis Child 1985;139:865–6.171. Lock S. Repetitive publication: a waste that muststop. Br Med J 1984;288:661–2.172. Yank V, Barnes D. Consensus and contentionregarding redundant publications in clinicalresearch: cross-sectional survey of editors andauthors. J Med Ethics 2003;29:109–14.173. Errami M, Hicks JM, Fisher W, Trusty D, WrenJD, Long TC, et al. Deja vu: A study of duplicatecitations in Medline. Bioinformatics 2007:btm574.174. Tramer MR, Reynolds DJ, Moore RA, McQuay HJ.Impact of covert duplicate publication on metaanalysis:a case study. BMJ 1997;315:635–40.175. Gotzsche PC. Multiple publication of reports ofdrug trials. Eur J Clin Pharmacol 1989;36:429–32.95

References96176. Huston P, Moher D. Redundancy, disaggregation,and the integrity of medical research. Lancet1996;347:1024–6.177. Vandekerckhove P, O’Donovan PA, LilfordRJ, Harada TW. Infertility treatment: fromcookery to science. The epidemiology ofrandomised controlled trials. Br J Obstet Gynaecol1993;100:1005–36.178. Martin J, Bainbridge D, Cheng D. Impact ofduplicate publication in a meta-analysis of off-pumpversus on-pump coronary artery bypass surgery[abstract]. 12th Cochrane Colloquium, 2–6 October2004, Ottawa, Canada.179. Ben-Shlomo Y, Davey-Smith G. ‘Place of publication’bias? [letter]. Bmj 1994;309:274.180. Bero LA, Galbraith A, Rennie D. Sponsoredsymposia on environmental tobacco-smoke. JAMA1994;271:612–7.181. Penel N, Adenis A. Publication biases and phase IItrials investigating anticancer targeted therapies.Invest New Drug 2009;27:287–8.182. Pittler MH, Abbot NC, Harkness EF, Ernst E.Location bias in controlled clinical trials ofcomplementary/alternative therapies. J ClinEpidemiol 2000;53:485–9.183. Ottenbacher K, DiFabio RP. Efficacy of spinalmanipulation/mobilization therapy. A meta-analysis.Spine 1985;10:833–7.184. Vickers A, Goyal N, Harland R, Rees R. Do certaincountries produce only positive results? A systematicreview of controlled trials. Control Clin Trials1998;19:159–66.185. Tang JL, Zhan SY, Ernst E. Review of randomisedcontrolled trials of traditional Chinese medicine.BMJ 1999;319:160–1.186. Pan Z, Trikalinos TA, Kavvoura FK, Lau J, IoannidisJPA. Local literature bias in genetic epidemiology:an empirical evaluation of the Chinese literature[see comment]. PLoS Med 2005;2:e334.187. King DA. The scientific impact of nations. Nature2004;430:311–6. Erratum appears in Nature2004;432:8.188. Felson DT. Bias in meta analytic research. J ClinEpidemiol 1992;45:885–92.189. Dickersin K, Hewitt P, Mutch L, Chalmers I,Chalmers TC. Perusing the literature: comparisonof MEDLINE searching with a perinatal trialsdatabase. Control Clin Trials 1985;6:306–17.190. Gotzsche PC, Lange B. Comparison of searchstrategies for recalling double-blind trials fromMedline. Dan Med Bull 1991;38:476–8.191. Adams CE, Power A, Frederick K, Lefebvre C. Aninvestigation of the adequacy of Medline searchesfor randomized controlled trials (Rcts) of the effectsof mental health care. Psychol Med 1994;24:741–8.192. Zielinski C. New equities of information in anelectronic age. BMJ 1995;310:1480–1.193. Nieminen P, Isohanni M. Bias against Europeanjournals in medical publication databases. Lancet1999;353:1592.194. Caulfield T. Biotechnology and the popular press:hype and the selling of science. Trends Biotech2004;22:337–9.195. Combs B, Slovic P. Causes of death: biasednewspaper coverage and biased judgements.Journalism Q 1979;56:837–43.196. Houn F, Bober MA, Huerta EE, Hursting SD,Lemon S, Weed DL. The association betweenalcohol and breast cancer: popular press coverageof research. Am J Public Health 1995;85:1082–6.197. Koren G, Klein N. Bias against negative studiesin newspaper reports of medical research. JAMA1991;266:1824–6.198. Wing S, Shy C, Wood J, Wolf S, Cragle D, Frome E.Mortality among workers at Oak Ridge NationalLaboratory. JAMA 1991;265:1397–402.199. Jablon S, Hrubec Z, Boice JJ. Cancer in populationsliving near nuclear facilities: a survey of mortalitynationwide and incidence in two states. JAMA1991;265:1403–8.200. Schwartz LM, Woloshin S, Baczek L. Mediacoverage of scientific meetings. Too much, toosoon? Jama 2002;287:2859–63.201. Woloshin S, Schwartz LM. Media reporting onresearch presented at scientific meetings: morecaution needed. Med J Aust 2006;184:576–80.202. Whiteman MK, Cui Y, Flaws JA, Langenberg P,Bush TL. Media coverage of women’s health issues:is there a bias in the reporting of an associationbetween hormone replacement therapy and breastcancer? J Women Health Gen-B 2001;10:571–7.203. Koper M, Bubela T, Caulfield T, Boon H. Mediaportrayal of conflicts of interest in herbal remedyclinical trials. Health Law Rev 2006;15:9–11.204. Lefebvre C, Manheimer E, Glanville J. Searchingfor studies. In Higgins J, Green S, editors. Cochrane

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Handbook for Systematic Reviews of InterventionsVersion 5.00 (Updated Feb 2008). The CochraneCollaboration, 2008.205. Perel P, Roberts I, Sena E, Wheble P, Briscoe C,Sandercock P, et al. Comparison of treatment effectsbetween animal experiments and clinical trials:systematic review. BMJ 2007;334:197.206. Hackam DG. Translating animal research intoclinical benefit. BMJ 2007;334:163–4.207. Skolbekken JA. The risk epidemic in medicaljournals. Soc Sci Med 1995;40:291–305.208. Angell M, Kassirer JP. Clinical research – whatshould the public believe? New Engl J Med1994;331:189–90.209. Taubes G. Epidemiology faces its limits. Science1995;269:164–9.210. Chalmers I. Under-reporting research is scientificmisconduct. JAMA 1990;263:1405–8.211. Cowley AJ, Skene A, Stainer K, Hampton JR. Theeffect of lorcainide on arrhythmias and survivalin patients with acute myocardial infarction –an example of publication bias. Int J Cardiol1993;40:161–6.212. The Cardiac Arrhythmia Suppression Trial (CAST)Investigators. Preliminary report: effect of encainideand flecainide on mortality in a randomised trial ofarrhythmia suppression after myocardial infarction.N Engl J Med 1989;321:406–12.213. The Cardiac Arrhythmia Suppression Trial IIInvestigators. Effect of the antiarrhythmic agentmoricisine on survival after myocardial infarction. NEngl J Med 1992;327:227–33.214. Topol EJ. Failing the public health – rofecoxib,Merck, and the FDA. N Engl J Med 2004;351:1707–9.215. Curfman GD, Morrissey S, Drazen JM. Expressionof concern: Bombardier et al., ‘Comparison ofupper gastrointestinal toxicity of rofecoxib andnaproxen in patients with rheumatoid arthritis.’N Engl J Med 2000;343:1520–8.[See editorial in NEngl J Med 2005;353:2813–4.]216. Bombardier C, Laine L, Burgos-Vargas R, Davis B,Day R, Ferraz MB, et al. Response to expressionof concern regarding VIGOR study. N Engl J Med2006;354:1196–9.217. Curfman GD, Morrissey S, Drazen JM. Expressionof concern reaffirmed. N Engl J Med 2006;354:1193.218. Brown TJ, Hooper L, Elliott RA, Payne K, WebbR, Roberts C, et al. A comparison of the costeffectivenessof five strategies for the preventionof non-steroidal anti-inflammatory drug-inducedgastrointestinal toxicity: a systematic reviewwith economic modelling. Health Technol Assess2006;10:iii–iv, xi–xiii, 1–183.219. Psaty BM, Kronmal RA. Reporting mortalityfindings in trials of rofecoxib for Alzheimer diseaseor cognitive impairment: a case study basedon documents from rofecoxib litigation. JAMA2008;299:1813–7.220. Garland EJ. Facing the evidence: antidepressanttreatment in children and adolescents. Can MedAssoc J 2004;170:489–91.221. Jureidini JN, Doecke CJ, Mansfield PR, Haby MM,Menkes DB, Tonkin AL. Efficacy and safety ofantidepressants for children and adolescents. BMJ2004;328:879–83.222. Kondro W, Sibbald B. Drug company expertsadvised staff to withhold data about SSRI use inchildren. Can Med Assoc J 2004;170:783.223. Dyer O. GlaxoSmithKline faces US lawsuit overconcealment of trial results. BMJ 2004;328:1395.224. Ioannidis JP. Effectiveness of antidepressants:an evidence myth constructed from a thousandrandomized trials? Philos Ethics Humanit Med2008;3:14.225. Godlee F, Dickersin K. Bias, subjectivity, chance, andconflict of interest in editorial decisions. In GodleeF, Jefferson T, editors. Peer review in health sciences.London: BMJ Books; 1999. pp. 57–78.226. MacCoun R. Biases in the interpretation and use ofresearch results. Annu Rev Psychol 1998;49:259–87.227. Cain DM, Detsky AS. Everyone’s a little bit biased(even physicians). JAMA 2008;299:2893–5.228. Dickersin K, Chan S, Chalmers TC, Sacks HS, SmithH, Jr. Publication bias and clinical trials. Control ClinTrials 1987;8:343–53.229. Rotton J, Foos PW, Vanmeek L, Levitt M.Publication practices and the file drawer problem– a survey of published authors. J Soc Behav Pers1995;10:1–13.230. Weber EJ, Callaham ML, Wears RL, Barton C,Young G. Unpublished research from a medicalspecialty meeting: why investigators fail to publish.Jama 1998;280:257–9.97© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

References98231. Blumenthal D, Campbell EG, Anderson MS,Causino N, Louis KS. Withholding research resultsin academic life science. JAMA 1997;277:1224–8.232. Camacho LH, Bacik J, Cheung A, SpriggsDR. Presentation and subsequent publicationrates of phase I oncology clinical trials. Cancer2005;104:1497–504.233. Hartling L, Craig WR, Russell K, Stevens K,Klassen TP. Factors influencing the publicationof randomized controlled trials in child healthresearch [see comment]. Arch Pediat Adol Med2004;158:983–7.234. Hopewell S, Clarke M. Methodologists and theirmethods. Do methodologists write up theirconference presentations or is it just 15 minutes offame? Int J Technol Assess 2001;17:601–3.235. Machan C, Ammenwerth E, Bodner T. Publicationbias in medical informatics evaluation research: is itan issue or not? St Heal T 2006;124:957–62.236. Sprague S, Bhandari M, Devereaux PJ,Swiontkowski MF, Tornetta P III, Cook DJ, etal. Barriers to full-text publication followingpresentation of abstracts at annual orthopaedicmeetings. J Bone Joint Surg Am 2003;85-a(1):158–63.237. Vuckovic Dekic L, Gajic Veljanoski O, JovicevicBekic A, Jelic S. Research results presented atscientific meetings: to publish or not? Arch Oncol2001;9:161–3.238. Shakiba B, Salmasian H, Yousefi-NooraieR, Rohanizadegan M. Factors influencingeditors’ decision on acceptance or rejection ofmanuscripts: the authors’ perspective. Arch Iran Med2008;11:257–62.239. Chalmers TC, Frank CS, Reitman D. Minimizingthe three stages of publication bias. JAMA1990;263:1392–5.240. Rothwell PM, Slattery J, Warlow CP. A systematicreview of the risks of stroke and death due toendarterectomy for symptomatic carotid stenosis.Stroke 1996;27:260–5.241. Frank E. Authors’ criteria for selecting journals.JAMA 1994;272:163–4.242. McCambridge J. A case study of publication bias inan influential series of reviews of drug education.Drug Alcohol Rev 2007;26:463–8.243. Tobler N. Meta-analysis of 143 adolescent drugprevention programs: quantitative outcome resultsof program participants compare to a control orcomparison group. J Drug Issues 1986;16:537–67.244. Tobler N, Stratton H. Effectiveness of school-baseddrug prevention programs: a meta-analysis of theresearch. J Prim Prev 1997;18:71–128.245. Tobler N, Roona MR, Ochshorn P, Marshall DG,Streke AV, Stackpole KM. School-based adolescentdrug prevention programs: 1998 meta-analysis. JPrim Prev 2000;20:275–336.246. Lee K, Boyd E, Bero L. A look inside the blackbox: a description of the editorial process at threeleading biomedical journals [abstract]. 12th CochraneColloquium, 2–6 October 2004; Ottawa, Canada.247. Luty J, Arokiadass SMR, Easow JM, AnapreddyJR. Preferential publication of editorial boardmembers in medical specialty journals. J Med Ethics2009;35:200–2.248. Kupfersmid J, Fiala M. A survey of attitudes andbehaviors of authors who publish in psychologyand education journals [comments]. Am Psychol1991;46:249–50.249. Kerr S, Tolliver J, Petree D. Manuscriptcharacteristics which influence acceptance formanagement and social science journals. AcadManage J 1977;20:132–41.250. Radford DR, Smillie L, Wilson RF, Grace AM. Thecriteria used by editors of scientific dental journalsin the assessment of manuscripts submitted forpublication. Br Dent J 1999;187:376–9.251. Angell M. Negative studies. N Engl J Med1989;321:464–6.252. Kassirer JP, Campion EW. Peer review: crudeand understudied, but indispensable. JAMA1994;272:96–7.253. Abby M, Massey MD, Galandiuk S, Polk HC. Peerreview is an effective screening process to evaluatemedical manuscripts. JAMA 1994;272:105–7.254. Zelen M. Guidelines for publishing papers oncancer clinical trials: responsibilities of editors andauthors. J Clin Oncol 1983;1:164–9.255. Editor. Manuscript guideline. Diabetologia1984;25:4A.256. Davis RM, Mullner M. Editorial independence atmedical journals owned by professional associations:a survey of editors. Sci Eng Ethics 2002;8:513–28.257. Bailar JC, Patterson K. Journal peer review:the need for a research agenda. N Engl J Med1985;312:654–7.258. Hojat M, Gonnella JS, Caelleigh AS. Impartialjudgment by the ‘gatekeepers’ of science: fallibility

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8and accountability in the peer review process. AdvHealth Sci Educ 2003;8:75–96.259. Mahoney MJ. Publication prejudices: anexperimental study of confirmatory bias in the peerreview system. Cognitive Ther Res 1977;1:161–75.260. Ernst E, Resch KL. Reviewer bias – a blindedexperimental study. J Lab Clin Med 1994;124:178–82.261. Abbot NE, Ernst E. Publication bias: directionof outcome less important than scientific quality.Perfusion 1998;11:182–4.262. Ector H, Aubert A, Stroobandt R. Review ofthe reviewer [editorial]. Pacing Clin Electrophys1995;18:1215–7.263. Blackburn JL, Hakel MD. An examination ofsources of peer-review bias. Psychol Sci 2006;17:378–82.264. Wager E, Parkin EC, Tamber PS. Are reviewerssuggested by authors as good as those chosen byeditors? Results of a rater-blinded, retrospectivestudy. BMC Med 2006;4:13.265. Rivara FP, Cummings P, Ringold S, Bergman AB,Joffe A, Christakis DA. A comparison of reviewersselected by editors and reviewers suggested byauthors [see comment]. J Pediatr 2007;151:202–5.266. Link AM. US and non-US submissions: an analysisof reviewer bias. Jama 1998;280:246–7.267. Opthof T, Coronel R, Janse MJ. The significance ofthe peer review process against the background ofbias: priority ratings of reviewers and editors andthe prediction of citation, the role of geographicalbias. Cardiovasc Res 2002;56:339–46.268. Gilbert JR, Williams ES, Lundberg GD. Is theregender bias in JAMA’s peer review process? JAMA1994;272:139–42.269. Johansson EE, Risberg G, Hamberg K, WestmanG. Gender bias in female physician assessments.Women considered better suited for qualitativeresearch. Scand J Prim Health Care 2002;20:79–84.270. Caelleigh AS, Hojat M, Steinecke A, GonnellaJS. Effects of reviewers’ gender on assessmentsof a gender-related standardized manuscript. 4thInternational Congress on Peer Review in BiomedicalPublication, 14–16 September 2001, Barcelona, Spain.271. Garfunkel JM, Ulshen MH, Hamrick HJ, LawsonEE. Effect of institutional prestige on reviewers’recommendations and editorial decisions. JAMA1994;272:137–8.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.272. Epstein WM. Confirmation response bias amongsocial work journals. Sci Techol Hum Values1990;15:9–38.273. Campillo C. Publication bias in two Spanish medicaljournals. The International Congress on Biomedical PeerReview and Global Communications, September 1997,Prague, Czech Republic.274. Justice AC, Berlin JA, Fletcher SW, Fletcher RH,Goodman SN. Do readers and peer reviewers agreeon manuscript quality? JAMA 1994;272:117–9.275. Rosenberg SA. Secrecy in medical research. NewEngl J Med 1996;334:392–4.276. Rennie D. Thyroid storm. JAMA 1997;277:1238–43.277. Davidson RA. Source of funding and outcome ofclinical trials. J Gen Intern Med 1986;1:155–8.278. Rochon PA, Gurwitz JH, Simms RW, FortinPR, Felson DT, Minaker KL, et al. A study ofmanufacturer-supported trials of nonsteroidal antiinflammatorydrugs in the treatment of arthritis.Arch Intern Med 1994;154:157–63.279. Stelfox HT, Chua G, O’Rourke K, Detsky AS.Conflict of interest in the debate over calciumchannelantagonists. New Engl J Med 1998;338:101–6.280. Smith R. Beyond conflict of interest: transparency isthe key. BMJ 1998;317:291–2.281. Opie L. Conflict of interest in the debate overcalcium-channel antagonists. N Engl J Med1998;338:1696–7; author reply 1697–8.282. Detsky AS, Stelfox HT. Correspondence: conflictof interest in the debate over calcium-channelantagonists. N Engl J Med 1998;338:1698.283. Meltzer JI. Correspondence: conflict of interest inthe debate over calcium-channel antagonists. N EnglJ Med 1998;338:1696.284. Strandgaard S. Correspondence: conflict of interestin the debate over calcium-channel antagonists. NEngl J Med 1998;338:1697.285. Bekelman JE, Li Y, Gross CP. Scope and impact offinancial conflicts of interest in biomedical research:a systematic review. Jama 2003;289:454–65.286. Lexchin J, Bero LA, Djulbegovic B, Clark O.Pharmaceutical industry sponsorship and researchoutcome and quality: systematic review. BMJ2003;326:1167–70.287. Golder S, Loke YK. Is there evidence for biasedreporting of published adverse effects data in99

References100pharmaceutical industry-funded studies? Brit J ClinPharmaco 2008;66:767–73.288. Als Nielsen B, Chen W, Gluud C, KjaergardLL. Association of funding and conclusions inrandomized drug trials: a reflection of treatmenteffect or adverse events? Jama 2003;290:921–8.289. Baker CB, Johnsrud MT, Crismon ML, RosenheckRA, Woods SW. Quantitative analysis of sponsorshipbias in economic studies of antidepressants [review].Brit J Psychiat 2003;183:498–506.290. Bhandari M, Busse JW, Jackowski D, MontoriVM, Schunemann H, Sprague S, et al. Associationbetween industry funding and statistically significantpro-industry findings in medical and surgicalrandomized trials [see comment]. Can Med Assoc J2004;170:477–80.291. Buchkowsky SS, Jewesson PJ. Industry sponsorshipand authorship of clinical trials over 20 years [seecomment]. Ann Pharmacother 2004;38:579–85.292. Montgomery JH, Byerly M, Carmody T, Li B, MillerDR, Varghese F, et al. An analysis of the effect offunding source in randomized clinical trials ofsecond generation antipsychotics for the treatmentof schizophrenia. Control Clin Trials 2004;25:598–612.293. Perlis CS, Harwood M, Perlis RH. Extent andimpact of industry sponsorship conflicts of interestin dermatology research. J Am Acad Dermatol2005;52:967–71.294. Liss H. Publication bias in the pulmonary/allergyliterature: effect of pharmaceutical companysponsorship [see comment]. Isr Med Assoc J2006;8:451–4.295. Ridker PM, Torres J. Reported outcomes in majorcardiovascular clinical trials funded by for-profitand not-for-profit organizations: 2000–2005. Jama2006;295:2270–4.296. Etter J-F, Burri M, Stapleton J. The impact ofpharmaceutical company funding on results ofrandomized trials of nicotine replacement therapyfor smoking cessation: a meta-analysis. Addiction2007;102:815–22.297. Huss A, Egger M, Hug K, Huwiler Muntener K,Roosli M. Source of funding and results of studiesof health effects of mobile phone use: systematicreview of experimental studies. Environ Health Persp2007;115:1–4.298. Lesser LI, Ebbeling CB, Goozner M, Wypij D,Ludwig DS. Relationship between funding sourceand conclusion among nutrition-related scientificarticles. PLoS Med 2007;4(1):e5.299. Jorgensen AW, Hilden J, Gotzsche PC. Cochranereviews compared with industry supported metaanalysesand other meta-analyses of the same drugs:systematic review [review]. Bmj 2006;333:782–5.300. Sawka AM, Thabane L. Effect of industrysponsorship on the results of biomedical research[comment]. Jama 2003;289:2502–3.301. Barden J, Derry S, McQuay HJ, Moore RA. Biasfrom industry trial funding? A framework, asuggested approach, and a negative result. Pain2006;121:207–18.302. Tulikangas PK, Ayers A, O’Sullivan DM. A metaanalysiscomparing trials of antimuscarinicmedications funded by industry or not. BJU Int2006;98:377–80.303. Ahmer S, Haider II, Anderson D, Arya P. Dopharmaceutical companies selectively report clinicaltrial data? Pak J Med Sci 2006;22:338–46.304. Millstone E, Brunner E, White I. Plagiarism orprotecting public health. Nature 1994;371:647–8.305. Nathan DG, Weatherall DJ. Academia and industry:lessons from the unfortunate events in Toronto.Lancet 1999;353:771–2.306. Nathan DG, Weatherall DJ. Academic freedom inclinical research. N Engl J Med 2002;347:1368–71.307. Shuchman M. Drug company threatens legal actionover Canadian guidelines. BMJ 1999;319:1388.308. Skolnick AA. Drug firm suit fails to halt publicationof Canadian Health Technology Report. JAMA1998;280:683–4.309. Wise J. Research suppressed for seven years by drugcompany. BMJ 1997;314:1145.310. McCarthy M. Company sought to block paper’spublication (news). Lancet 2000;356:1659.311. Dat NV. Letters: Outcomes of a trial of HIV-1immunogen in patients with HIV infection. Lancet2001;285:2191.312. Goldberg BS, Stricker RB. Letters: Outcomes ofa trial of HIV-1 immunogen in patients with HIVinfection. Lancet 2001;285:2192.313. Gotch FM. Letters: Outcomes of a trial of HIV-1immunogen in patients with HIV infection. Lancet2001;285:2192.314. Khahn JO, Lagakos S. Letters: Outcomes of atrial of HIV-1 immunogen in patients with HIVinfection. Lancet 2001;285:2193–4.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8315. Lurie P, Wolfe SM. Letters: Outcomes of a trial ofHIV-1 immunogen in patients with HIV infection.Lancet 2001;285:2193.316. Poretz DM. Letters: Outcomes of a trial of HIV-1immunogen in patients with HIV infection. Lancet2001;285:2192–3.317. Lauritsen K, Kavelund T, Larsen LS, Rask-Madsen J. Withholding unfavourable results indrug company sponsored clinical trials. Lancet1987;i:1091.318. Symmonds M, Matheson NJ, Harnden A.Guidelines on neuraminidase inhibitors in childrenare not supported by evidence. BMJ 2004;328:227.319. van Heteren G. Wyeth suppresses research onpill, programme claims [see comment]. BMJ2001;322:571.320. van Veldhuisen DJ, Poole Wilson PA. Theunderreporting of results and possible mechanismsof ‘negative’ drug trials in patients with chronicheart failure. Int J Cardiol 2001;80:19–27.321. Wilmshurst P. Policing the drug industry. Lancet1986;ii:1280–1.322. Wilmshurst P. An investigation by the ABPI(Association of the British Pharmaceutical Industry).Lancet 1987;i:104.323. Dieppe PA, Ebrahim S, Martin RM, Juni P.Lessons from the withdrawal of rofecoxib. BMJ2004;329:867–8.324. Gottlieb S. Researchers deny any attempt to misleadthe public over JAMA article on arthritis drug. BMJ2001;323:301.325. Hrachovec J. Publication bias with cetirizine inatopic dermatitis: safe but ineffective? [comment]. JAllergy Clin Immun 2002;110:818; author reply 818.326. Silverstein FE, Faich G, Goldstein GL. Letter:Reporting of 6-month vs 12-month data in a clinicaltrial of celecoxib. JAMA 2001;286:2399.327. Wright JM, Perry JL. Letter: Reporting of 6-monthvs 12-month data in a clinical trial of celecoxib.JAMA 2001;286:2398–9.328. Applegate WB, Furberg CD, Byington RP,Grimm R, Jr. The Multicenter IsradipineDiuretic Atherosclerosis Study (MIDAS). JAMA1997;277:297; author reply 297–8.329. Lenzer J. Alteplase for stroke: money and optimisticclaims buttress the ‘brain attack’ campaign. BMJ2002;324:723–9.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.330. Reines SA, Block GA, Morris JC, Liu G, Nessly ML,Lines CR, et al. Rofecoxib: no effect on Alzheimer’sdisease in a 1-year, randomized, blinded, controlledstudy. Neurology 2004;62:66–71.331. Steinman MA, Bero LA, Chren MM, Landefeld CS.Narrative review: the promotion of gabapentin: ananalysis of internal industry documents. Ann InternMed 2006;145:284–93.332. Alasbali T, Smith M, Geffen N, Trope GE, FlanaganJG, Jin Y, et al. Discrepancy between results andabstract conclusions in industry- vs nonindustryfundedstudies comparing topical prostaglandins[see comment]. Am J Ophthalmol 2009;147:33–38.e2.333. Dong BJ, Hauck WW, Gambertoglio JG, Gee L,White JR, Bubp JL, et al. Bioequivalence of genericand brand-name levothyroxine products in thetreatment of hypothyroidism. JAMA 1997;277:1205–13.334. CCOHTA. Canadian Coordinating Office For HealthTechnology Assessment: Annual Report 1997–1998.Ottawa: CCOHTA; 1998.335. Metcalfe S, Burgess C, Laking G, Evans J, Wells S,Crausaz S. Trastuzumab: possible publication bias.Lancet 2008;371:1646–8.336. Panahloo Z. Data on neuraminidase inhibitors weremade available. BMJ 2004;328:523.337. Thal LJ, Ferris SH, Kirby L, Block GA, LinesCR, Yuen E, et al. A randomized, double-blindstudy of rofecoxib in patients with mild cognitiveimpairment. Neuropsychopharmacology 2005;30:1204–15.338. Hedges LV. Estimation of effect size undernonrandom sampling: the effects of censoringstudies yielding statistically insignificant meandifferences. J Educ Stat 1984;9:61–85.339. Lane DM, Dunlap WP. Estimating effect size: biasresulting from the significance criterion in editorialdecisions. Br J Math Statist Psychol 1978;31:107–12.340. Begg CB, Mazumdar M. Operating characteristicsof a rank correlation test for publication bias.Biometrics 1994;50:1088–101.341. Borm GF, den Heijer M, Zielhuis GA. Publicationbias was not a good reason to discourage trials withlow power. J Clin Epidemiol 2009;62:47.e1–10.342. Berlin JA, Begg CB, Louis TA. An assessment ofpublication bias using a sample of published clinicaltrials. J Am Stat Assoc 1989;84:381–92.101

References102343. Irwig L, Macaskill P, Glasziou P, Fahey M. Metaanalyticmethods for diagnostic test accuracy. J ClinEpidemiol 1995;48:119–30.344. Newcombe RG. Discussion of the paper byBegg and Berlin: Publication bias: a problemin interpreting medical data. J R Stat Soc A1988;151:448–9.345. Julian D. Meta-analysis and the meta-epidemiologyof clinical research. Registration of trials should berequired by editors and registering agencies [letter].BMJ 1998;316:311.346. Chalmers I, Altman D. How can medical journalshelp prevent poor medical research? Someopportunities presented by electronic publishing.Lancet 1999;353:490–3.347. The Lancet. URL: http://www.thelancet.com.348. Horton R. Medical editors trial amnesty. Lancet1997;350:756.349. ICMJE. Uniform requirements for manuscriptssubmitted to biomedical journals: writing andediting for biomedical publication. InternationalCouncil of Medical Journal Editors, 2007. URL:http://www.icmje.org/.350. Altman DG, Simera I, Hoey J, Moher D, SchulzK. EQUATOR: reporting guidelines for healthresearch. Lancet 2008;371:1149–50.351. Moher D, Simera I, Schulz KF, Hoey J, AltmanDG. Helping editors, peer reviewers and authorsimprove the clarity, completeness and transparencyof reporting health research. BMC Med 2008;6:13.352. Simera I, Altman DG, Moher D, Schulz KF, HoeyJ. Guidelines for reporting health research: theEQUATOR network’s survey of guideline authors.PLoS Med 2008;5(6):e139.353. Fletcher RH, Fletcher SW. The effectiveness ofeditorial peer review. Peer Review in Health Sciences1999:45–55.354. Godlee F, Gale CR, Martyn CN. Effect on thequality of peer review of blinding reviewers andasking them to sign their reports: a randomizedcontrolled trial. JAMA 1998;280:237–40.355. Laband DN, Piette MJ. A citation analysis of theimpact of blinded peer review. JAMA 1994;272:147–9.356. van-Rooyen S, Godlee F, Evans S, Smith R, BlackN. Effect of blinding and unmasking on thequality of peer review. A randomized trial. JAMA1998;280:234–7.357. Yankauer A. How blind is blind review? Am J PublicHealth 1991;81:843–5.358. Jefferson T, Alderson P, Wager E, Davidoff F. Effectsof editorial peer review. A systematic review. Jama2002;287:2784–6.359. Cooper RJ, Gupta M, Wilkes MS, Hoffman JR.Conflict of interest disclosure policies and practicesin peer-reviewed biomedical journals. J Gen InternMed 2006;21:1248–52.360. Cain DM, Loewenstein G, Moore DA. The dirt oncoming clean: perverse effects of disclosing conflictsof interest. J Legal Stud 2005;34:1–25.361. Miller FG, Brody H. Viewpoint: professionalintegrity in industry-sponsored clinical trials. AcadMed 2005;80:899–904.362. Bero L. The electronic future: what might an onlinescientific paper look like in five years’ time? BMJ1997;315:1692.363. Fletcher RH, Fletcher SW. The future ofmedical journals in the western world. Lancet1998;352(SII):30–3.364. Huth EJ. Is the medical world ready for electronicjournals? [editorial]. Online J Curr Clin Trials1992(7).365. Huth EJ. Electronic publishing in the healthsciences. Bull PAHO 1995;29:81–7.366. Song F, Eastwood A, Gilbody S, Duley L. The role ofelectronic journals in reducing publication bias. MedInform Internet 1999;24:223–9.367. Sim I, Rennels G. A trial bank model for thepublication of clinical trials. Proceedings of the AnnualSymposium on Computer Applications in Medical Care;1995. American Medical Informatics Association.368. Delamothe T, Mullner M, Smith R. Pleasing bothauthors and readers. A combination of short printarticles and longer electronic ones may help us dothis. BMJ 1999;318:888–9.369. Fox T. Crisis in communication: the functions and futureof medical journals. London: The Athlone Press;1965.370. Pfeffer C, Olsen BR. Editorial: Journal of negativeresults in biomedicine. J Negative Results Biomed2002;1:2.371. Roberts I. An amnesty for unpublished trials. BMJ1998;317:763–4.372. Abbasi K. Compulsory registration of clinical trials.Bmj 2004;329:637–8.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8373. Al-Marzouki S, Roberts I, Evans S, Marshall T.Selective reporting in clinical trials: analysis of trialprotocols accepted by The Lancet [see comment].Lancet 2008;372:201.374. Boissel JP, Ad Hoc Working Party of theInternational Collaborative Group on ClinicalTrial Registries. Position paper and consensusrecommendation on clinical trial registries. ClinTrial Meta-Anal 1993;28:255–66.375. Verstraete M. International Committee onThrombosis and Haemostasis, Basil, 8–11September 1974. Subcommittee on ClinicalInvestigations: Registry of prospective clinical trials.Thromb Diath Haemorrh 1975;33:655–63.376. Easterbrook PJ. Directory of registries of clinicaltrials. Stat Med 1992;11:345–423.377. Haug C, Gotzsche PC, Schroeder TV. Registriesand registration of clinical trials. N Engl J Med2005;353:2811–2.378. Zarin DA, Ide NC, Tse T, Harlan WR, West JC,Lindberg DA. Issues in the registration of clinicaltrials. JAMA 2007;297:2112–20.379. Chalmers I. Government regulation is needed toprevent biased under-reporting of clinical trials[comment]. BMJ 2004;329:462.380. Chalmers I. From optimism to disillusion aboutcommitment to transparency in the medicoindustrialcomplex [see comment]. J R Soc Med2006;99:337–41.381. Dickersin K, Garcia Lopez F. Regulatory processeffects clinical trial registration in Spain. Control ClinTrial 1992;13:507–12.382. Chollar S. A registry for clinical trials [news]. AnnIntern Med 1998;128:701–2.383. Gulmezoglu AM, Pang T, Horton R, DickersinK. WHO facilitates international collaboration insetting standards for clinical trial registration. Lancet2005;365:1829–31.384. Ghersi D, Pang T. En route to international clinicaltrial transparency. Lancet 2008;372:1531–2.385. Laine C, Horton R, DeAngelis CD, DrazenJM, Frizelle FA, Godlee F, et al. Clinical trialregistration: looking back and moving ahead. Lancet2007;369:1909–11.386. Zarin DA, Tse T, Ide NC. Trial registration atClinicalTrials.gov between May and October 2005.N Engl J Med 2005;353:2779–87.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.387. Krleza Jeric K, Chan AW, Dickersin K, Sim I,Grimshaw J, Gluud C. Principles for internationalregistration of protocol information and resultsfrom human trials of health related interventions:Ottawa statement (part 1) Bmj 2005;330:956–8.Erratum appears in BMJ 2005;330:1258.388. Sim I, Detmer DE. Beyond trial registration: aglobal trial bank for clinical trial reporting. PLoSMed 2005;2(11):e365.389. Chalmers I. TGN1412 and The Lancet’s solicitationof reports of phase I trials [comment]. Lancet2006;368:2206–7.390. Choi BCK, Frank J, Mindell JS, Orlova A, Lin V,Vaillancourt ADMG, et al. Vision for a global registryof anticipated public health studies. Am J PublicHealth 2007;97(Suppl. 1):S82–7.391. Ramsey S, Scoggins J. Commentary: practicingon the tip of an information iceberg? Evidenceof underpublication of registered clinical trials inoncology [see comment]. Oncologist 2008;13:925–9.392. Abraham J, Lewis G. Secrecy and transparencyof medicines licensing in the EU. Lancet1998;352:480–2.393. Bardy AH. Freedom of information. Lancet1998;352:1229.394. Roberts I, Li-Wan-Po A, Chalmers I. Intellectualproperty, drug licensing, freedom of information,and public health. Lancet 1998;352:726–9.395. Mayor S. Opening the lid on open access. BMJ2008;336:688–9.396. Lenzer J. US Congress and European ResearchCouncil insist on open access to research results.BMJ 2008;336:176–7.397. Craig ID, Plume AM, McVeigh ME, Pringle J,Amin M. Do open access articles have greatercitation impact? A critical review of the literature. JInformetrics 2007;1:239–48.398. Wren JD. Open access and openly accessible:a study of scientific publications shared via theinternet [see comment]. BMJ 2005;330:1128.399. Suber P. Open access, impact, and demand. BMJ2005;330:1097–8.400. Eckert CH. Bioequivalence of levothyroxinepreparations: industry sponsorship and academicfreedom. JAMA 1997;277:1200.401. Mello MM, Clarridge BR, Studdert DM.Academic medical centers’ standards for clinical-103

References104trial agreements with industry. N Engl J Med2005;352:2202–10.402. Peto R, Collins R, Gray R. Large-scale randomizedevidence: large, simple trials and overviews of trials.J Clin Epidemiol 1995;48:23–40.403. Cappelleri JC, Ioannidis JPA, Schmid CH, deFerranti SD, Aubert M, Chalmers TC, et al. Largetrials vs meta-analysis of smaller trials. How do theirresults compare? JAMA 1996;276:1332–8.404. LeLorier J, Gregoire G, Benhaddad A, Lapierre J,Derderian F. Discrepancies between meta-analysesand subsequent large randomized, controlled trials.N Engl J Med 1997;337:536–42.405. Villar J, Carroli G, Belizan JM. Predictive abilityof meta-analyses of randomised controlled. Lancet1995;345:772–6.406. Flournoy N, Olkin I. Do small trials square withlarge ones? Lancet 1995;345:741–2.407. Bennett DA, Latham NK, Stretton C, Anderson CS.Capture-recapture is a potentially useful methodfor assessing publication bias. J Clin Epidemiol2004;57:349–57.408. Fedorowicz Z, Amin F, Eisinga A, Al Sayyad J.Handsearching for ‘buried’ randomized trials inBahrain medical journals [abstract]. 13th CochraneColloquium, 22–26 October 2005, Melbourne, Australia.409. Peinemann F, Sauerland S, Lange S. Identificationof unpublished studies contribute to a systematicreview on negative pressure wound therapy[abstract]. 14th Cochrane Colloquium, 23–26 October2006, Dublin, Ireland.410. Prentice VJ, Sayers MK, Milan S. Accessibility oftrial data to EBM reviews – lessons for systematicreviewers and the pharmaceutical industry. FirstSymposium on Systematic Reviews: beyond thebasics; 1998 Jan 8–9, Oxford, UK.411. Lusher ALC. European collaboration to identifyreports of controlled trials in specialized health carejournals published in Western Europe – helping toovercome some of the barriers associated with trialidentification The BIOMED Specialized HealthCare Journals Project. 9th Cochrane Colloquium, 9–13October 2001, Lyon, France.412. Greenhalgh T, Peacock R. Effectiveness andefficiency of search methods in systematic reviewsof complex evidence: audit of primary sources. BMJ2005;331:1064–5.413. McDonald S. Improving access to the internationalcoverage of reports of controlled trials in electronicdatabases: a search of the Australasian MedicalIndex. Health Info Libr J 2002;19:14–20.414. Bagett R, Chiquette E, Anagnostelis B, Mulrow C.Locating reports of serious adverse drug reactions.7th Cochrane Colloquium, 5–9 October 1999, Rome,Italy.415. Kassai B, Sonie S, Shah NR, Boissel J-P. Literaturesearch parameters marginally improved the pooledestimate accuracy for ultrasound in detecting deepvenous thrombosis. J Clin Epidemiol 2006;59:710–4.416. Clarke M. Commentary: searching for trials forsystematic reviews: what difference does it make?[comment]. Int J Epidemiol 2002;31:123–4.417. Sterne JAC, Bartlett C, Jüni P, Egger M. Do weneed comprehensive literature searches? A study ofpublication and language bias in meta-analyses ofcontrolled trials. 3rd Symposium on Systematic Reviews:Beyond the Basics, July 2000 in Oxford.418. Robinson KA, Dickersin K. Development of a highlysensitive search strategy for the retrieval of reportsof controlled trials using PubMed. Int J Epidemiol2002;31:150–3.419. Kelly JA. Scientific meeting abstracts: significance,access, and trends. B Med Libr Assoc 1998;86:68–76.420. Bennett DAJA. FDA: untapped source ofunpublished trials. Lancet 2003;361:1402–3.421. Song FJ, Fry Smith A, Davenport C, Bayliss S, AdiY, Wilson JS, et al. Identification and assessmentof ongoing trials in health technology assessmentreviews. Health Technol Assess 2004;8:1–87.422. Scott R. Answering the unanswered questions:ongoing trials of statins and antihypertensivesin type 2 diabetes. Acta Diabetol 2002;39(Suppl.2):S46–51.423. Chalmers I. Using systematic reviews and registersof ongoing trials for scientific and ethical trialdesign, monitoring, and reporting. In Egger M,Davey Smith G, Altman DG, editors. Systematicreviews in healthcare: meta-analysis in context. London:BMJ Books; 2000. pp. 429–43.424. Evsenbach G. Use of the world-wide-web to identifyunpublished evidence for systematic reviews – thefuture role of the internet to improve informationidentification. 7th Cochrane Colloquium, October 1999,Rome, Italy.425. Eysenbach G, Tuische J, Diepgen TL. Evaluationof the usefulness of internet searches to identifyunpublished clinical trials for systematic reviews.Chinese J Evidence Based Med 2002;2:196–200.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8426. Reveiz L, Andres Felipe C, Egdar Guillermo O.Using e-mail for identifying unpublished andongoing clinical trials and those published innon-indexed journals [abstract]. 12th CochraneColloquium: Bridging the Gaps; 2004 Oct 2–6, Ottawa,Ontario, Canada 2004:177–178.427. Reveiz L, Cardona AF, Ospina EG, de Agular S. Ane-mail survey identified unpublished studies forsystematic reviews. Journal of Clinical Epidemiology2006;59(7):755–758.428. Kober T. Obtaining information on clinical trials:a challenging dilemma for Cochrane reviewers[abstract]. 11th Cochrane Colloquium, 26–31 October2003, Barcelona, Spain.429. Milton JLSGR. Well-known signatory does not affectresponse to a request for information from authorsof clinical trials: a randomised controlled trial. 9thCochrane Colloquium, 9–13 October 2001 Lyon, France.430. Higgins J, Soornro M, Roberts I, Clarke M.Collecting unpublished data for systematic reviews:a proposal for a randomised trial. 7th CochraneColloquium, 5–9 October 1999, Rome, Italy.431. Shukla V. The challenge of obtaining unpublishedinformation from the drug industry [abstract]. 11thCochrane Colloquium, 26–31 October 2003, Barcelona,Spain.432. Hadhazy V, Ezzo J, Berman B. How valuable iseffort to contact authors to obtain missing data insystematic reviews. 7th Cochrane Colloquium, 5–9October 1999, Rome, Italy.433. Coursol A, Wagner EE. Effect of positive findings onsubmission and acceptance rates: a note on metaanalysisbias. Prof Psychol 1986;17:136–7.434. Greenwald AG. Consequences of prejudice againstthe null hypothesis. Psychol Bull 1975;82:1–20.435. Hetherington J, Dickersin K, Chalmers I, MeinertCL. Retrospective and prospective identificationof unpublished controlled trials: lessons from asurvey of obstetricians and pediatricians. Pediatrics1989;84:374–80.436. Shadish WR, Doherty M, Montgomery LM. Howmany studies are in the file drawer – an estimatefrom the family marital psychotherapy literature.Clin Psychol Rev 1989;9:589–603.437. Sommer B. The file drawer effect and publicationrates in menstrual cycle research. Psychol WomenQuart 1987;11:233–42.438. McManus R, Wilson S, Delaney B, Fitzmaurice D,Hyde C, Tobias R, et al. Review of the usefulnessof contacting other experts when conducting© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.a literature search for systematic reviews. BMJ1998;317:1562–3.439. Higgins J, Altman DG. Assessing risk of bias inincluded studies. In Higgins J, Green S, editors.Cochrane handbook for systematic reviews of interventionsversion 5.0.0 (updated February 2008). The CochraneCollaboration, 2008.440. Ioannidis JP. Why most published researchfindings are false: author’s reply to Goodman andGreenland. PLoS Med 2007;4(6):e215.441. Light RJ, Pillemer DB. Summing up: the scienceof reviewing research. Cambridge, MA; London:Harvard University Press; 1984.442. Peters JL, Sutton AJ, Jones DR, Abrams KR,Rushton L. Contour-enhanced meta-analysisfunnel plots help distinguish publication biasfrom other causes of asymmetry. J Clin Epidemiol2008;61(10):991–6.443. Sterne JA, Egger M. Funnel plots for detecting biasin meta-analysis: guidelines on choice of axis. J ClinEpidemiol 2001;54:1046–55.444. Tang JL, Liu JL. Misleading funnel plot fordetection of bias in meta-analysis. J Clin Epidemiol2000;53:477–84.445. Hofmeyr GJ, Atallah AN, Duley L. Calciumsupplementation during pregnancy for preventinghypertensive disorders and related problems.Cochrane Database Syst Rev 2006;3:CD001059.446. Belizan JM, Villar J, Gonzalez L, CampodonicoL, Bergel E. Calcium supplementation to preventhypertensive disorders of pregnancy. N Engl J Med1991;325:1399–405.447. Levine RJ, Hauth JC, Curet LB, Sibai BM, CatalanoPM, Morris CD, et al. Trial of calcium to preventpreeclampsia. N Engl J Med 1997;337:69–76.448. Crowther CA, Hiller JE, Pridmore B, BryceR, Duggan P, Hague WM, et al. Calciumsupplementation in nulliparous women for theprevention of pregnancy-induced hypertension,preeclampsia and preterm birth: an Australianrandomized trial. Aust NZ J Obstet Gynaecol1999;39:12–18.449. Lopez-Jaramillo P, Narvaez M, Weigel RM, YepezR. Calcium supplementation reduces the risk ofpregnancy-induced hypertension in an Andespopulation. Br J Obstet Gynaecol 1989;96:648–55.450. Purwar M, Kulkarni H, Motghare V, Dhole S.Calcium supplementation and prevention ofpregnancy induced hypertension. J Obstet GynaecolRes 1996;22:425–30.105

References106451. Villar J, Repke J, Belizan JM, Pareja G. Calciumsupplementation reduces blood pressure duringpregnancy: results of a randomized controlledclinical trial. Obstet Gynecol 1987;70:317–22.452. Villar J, Abdel-Aleem H, Merialdi M, Mathai M,Ali M, Zavaleta N, et al. World Health Organizationrandomized trial of calcium supplementationamong low calcium intake pregnant women. Am JObstet Gynecol 2006;194:639–49.453. Lopez-Jaramillo P, Narvaez M, Felix C, Lopez A.Dietary calcium supplementation and prevention ofpregnancy hypertension. Lancet 1990;335:293.454. Lopez-Jaramillo P, Delgado F, Jacome P, Teran E,Ruano C, Rivera J. Calcium supplementation andthe risk of preeclampsia in Ecuadorian pregnantteenagers. Obstet Gynecol 1997;90:162–7.455. Niromanesh S, Laghaii S, Mosavi-Jarrahi A.Supplementary calcium in prevention of preeclampsia.Int J Gynecol Obstet 2001;74:17–21.456. Sanchez-Ramos L, Briones DK, Kaunitz AM,Delvalle GO, Gaudier FL, Walker KD. Preventionof pregnancy-induced hypertension by calciumsupplementation in angiotensin II-sensitivepatients. Obstet Gynecol 1994;84:349–53.457. Villar J, Repke JT. Calcium supplementation duringpregnancy may reduce preterm delivery in high-riskpopulations. Am J Obstet Gynecol 1990;163:1124–31.458. Terrin N, Schmid CH, Lau J. In an empiricalevaluation of the funnel plot, researchers could notvisually identify publication bias. J Clin Epidemiol2005;58:894–901.459. Harbord RM, Egger M, Sterne JAC. A modifiedtest for small-study effects in meta-analyses ofcontrolled trials with binary endpoints. Stat Med2006;25:3443–57.460. Macaskill P, Walter SD, Irwig L. A comparison ofmethods to detect publication bias in meta-analysis[see comment]. Stat Med 2001;20:641–54.461. Rucker G, Schwarzer G, Carpenter J. Arcsine testfor publication bias in meta-analyses with binaryoutcomes. Stat Med 2008;27:746–63.462. Schwarzer G, Antes G, Schumacher M. A test forpublication bias in meta-analysis with sparse binarydata. Stat Med 2007;26:721–33.463. Pham B, Platt R, McAuley L, Klassen TP, MoherD. Is there a ‘best’ way to detect and minimizepublication bias? An empirical evaluation. EvalHealth Prof 2001;24:109–25.464. Schwarzer G, Antes G, Schumacher M. Inflationof type I error rate in two statistical tests for thedetection of publication bias in meta-analyses withbinary outcomes. Stat Med 2002;21:2465–77.465. Kromrey JD, Rendina Gobioff G. On knowingwhat we do not know: an empirical comparison ofmethods to detect publication bias in meta-analysis.Educ Psychol Meas 2006;66:357–73.466. Hayashino Y, Noguchi Y, Fukui T. Systematicevaluation and comparison of statistical tests forpublication bias. J Epidemiol 2005;15:235–43.467. Saveleva E, Selinski S. Meta-analyses with binaryoutcomes: how many studies need to be omittedto detect a publication bias? J Toxicol Env Health A2008;71:845–50.468. Moreno SG, Sutton AJ, Ades AE, Stanley TD,Abrams KR, Peters JL, et al. Assessment ofregression-based methods to adjust for publicationbias through a comprehensive simulation study.BMC Med Res Methodol 2009;9:2.469. Sterne J, Egger M, Moher D. Addressing reportingbiases. In Higgins J, Green S, editors. Cochranehandbook for systematic reviews of intervention. version5.0.0 (updated February 2008). The CochraneCollaboration; 2008. Chapter 10. Available fromwww.cochrane-handbook.org.470. Ioannidis JPA, Trikalinos TA. The appropriatenessof asymmetry tests for publication bias inmeta-analyses: a large survey. Can Med Assoc J2007;176:1091–6.471. Duval S, Tweedie R. Trim and fill: A simplefunnel-plot-based method of testing and adjustingfor publication bias in meta-analysis. Biometrics2000;56:455–63.472. Taylor S, Tweedie R. Trim and fill: a simple funnelplot based method of adjusting for publication biasin meta-analysis. Unpublished manuscript. 1998.473. Terrin N, Schmid CH, Lau J, Olkin I. Adjusting forpublication bias in the presence of heterogeneity.Stat Med 2003;22:2113–26. Erratum appears in StatMed 2005;24:825–6.474. Peters JL, Sutton AJ, Jones DR, Abrams KR,Rushton L. Performance of the trim and fill methodin the presence of publication bias and betweenstudyheterogeneity. Stat Med 2007;26:4544–62.475. Orwin RG. A fail-safe N for effect size in metaanalysis.J Educ Stat 1983;8:157–9.476. Klein S, Simes J, Blackburn GL. Total parenteralnutrition and cancer clinical trials. Cancer1986;58:1378–86.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8477. Ashworth SD, Osburn HG, Callender JC, BoyleKA. The effects of unrepresented studies on therobustness of validity generalization results. PersPsychol 1992;45:341–60.478. Gleser LJ, Olkin I. Models for estimatingthe number of unpublished studies. Stat Med1996;15:2493–507.479. Rosenberg MS. The file-drawer problem revisited:a general weighted method for calculating fail-safenumbers in meta-analysis. Evolution 2005;59:464–8.480. Evans S. Statistician’s comments on: ‘Fail safe N’ isa useful mathematical measure of the stability ofresults [letter; comment]. BMJ 1996;312:125.481. Becker BJ. Failsafe N or file-drawer number. InRothstein HR, Sutton AJ, Borenstein M, editors.Publication bias in meta-analysis: prevention, assessmentand adjustments. Chichester, UK: John Wiley & SonsLtd; 2005. Chapter 7.482. Copas J, Jackson D. A bound for publication biasbased on the fraction of unpublished studies.Biometrics 2004;60:146–53.483. Sutton AJ, Song F, Gilbody SM, Abrams KR.Modelling publication bias in meta-analysis: areview. Stat Methods Med Res 2000;9:421–445.484. Dear HBG, Begg CB. An approach for assessingpublication bias prior to performing a metaanalysis.Stat Sci 1992;7:237–45.485. Hedges LV. Modelling publication selection effectsin meta-analysis. Stat Sci 1992;7:246–55.486. Iyengar S, Greenhouse JB. Selection models andthe file drawer problem. Stat Sci 1988;3:109–35.487. Vevea JL, Hedges LV. A general linear model forestimating effect size In the presence of publicationbias. Psychometrika 1995;60:419–35.488. Rust RT, Lehmann DR, Farley JU. Estimatingpublication bias in metaanalysis. J Marketing Res1990;27:220–6.489. Copas J. What works? selectivity models and metaanalysis.J R Stat Soc 1999;162:95–109.490. Baker R, Jackson D. Using journal impact factors tocorrect for the publication bias of medical studies.Biometrics 2006;62:785–92.491. Bowden J, Thompson JR, Burton P. Using pseudodatato correct for publication bias in meta-analysis.Stat Med 2006;25:3798–813.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.492. Formann AK. Estimating the proportion of studiesmissing for meta-analysis due to publication bias.Contemp Clin Trials 2008;29:732–9.493. Cleary RJ, Casella G. An application of Gibbssampling to estimation in meta-analysis: accountingfor publication bias. J Educ Behav Stat 1997;22:141–54.494. Givens GH, Smith DD, Tweedie RL. Publicationbias in meta-analysis: a Bayesian data-augmentationapproach to account for issues exemplified in thepassive smoking debate. Stat Sci 1997;12:221–50.495. Larose DT, Dey KK. Modeling publicationbias using weighted distributions in a Bayesianframework. Comput Stat Data An 1998;26:279–302.496. Silliman NP. Nonparametric classes of weightfunctions to model publication bias. Biometrika1997;84:909–18.497. Silliman NP. Hierarchical selection models withapplications in meta-analysis. J Am Stat Assoc1997;92:926–36.498. Hedges LV, Vevea JL. Estimating effect size underpublication bias: Small sample properties androbustness of a random effects selection model. JEduc Behav Stat 1996;21:299–332.499. Vevea JL, Woods CM. Publication bias in researchsynthesis: sensitivity analysis using a priori weightfunctions. Psychol Methods 2005;10:428–43.500. Copas JB, Shi JQ. A sensitivity analysis forpublication bias in systematic reviews. Stat MethodsMed Res 2001;10:251–65.501. Williamson PR, Gamble C. Application andinvestigation of a bound for outcome reporting bias.Trials 2007;8:9.502. Ioannidis JP, Trikalinos TA. An exploratory testfor an excess of significant findings. Clin Trials2007;4:245–53.503. Whitehead A, Whitehead J. A general parametricapproach to the meta-analysis of randomizedclinical trials. Stat Med 1991;10:1665.504. DerSimonian R, Laird N. Meta-analysis in clinicaltrials. Contro Clin Trials 1986;7:177–88.505. Berlin JA, Laird NM, Sacks HS, Chalmers TC. Acomparison of statistical methods for combiningevent rates from clinical trials. Stat Med 1989;8:141–51.506. Jackson D. The implications of publication biasfor meta-analysis’ other parameter. Stat Med2006;25:2911–21.107

References108507. Greenland S. Invited commentary: a critical look atsome popular meta-analytic methods. Am J Epidemiol1994;140:290–6.508. Jadad AR, Cook DJ, Jones A, Klassen TP, TugwellP, Moher M, et al. Methodology and reports ofsystematic reviews and meta-analyses. A comparisonof Cochrane reviews with articles published inpaper-based journals. JAMA 1998;280:278–80.509. French SD, McDonald S, McKenzie JE, Green SE.Investing in updating: how do conclusions changewhen Cochrane systematic reviews are updated?BMC Med Res Methodol 2005;5:33.510. Shojania KG, Sampson M, Ansari MT, Ji J, DoucetteS, Moher D. How quickly do systematic reviews goout of date? A survival analysis [see comment]. AnnIntern Med 2007;147:224–33.511. Brok J, Thorlund K, Gluud C, Wetterslev J. Trialsequential analysis reveals insufficient informationsize and potentially false positive results in manymeta-analyses. J Clin Epidemiol 2008;61:763–9.512. Borm GF, Donders AR. Updating meta-analysesleads to larger type I errors than publication bias. JClin Epidemiol 2009;62(8):825–30.513. Ioannidis J, Trikalinos T. Appropriateness ofasymmetry tests for publication bias in metaanalysis:large-scale survey [abstract]. 14th CochraneColloquium; 23–26 October 2006, Dublin, Ireland.514. Lau J, Ioannidis JP, Terrin N, Schmid CH, OlkinI. The case of the misleading funnel plot. Bmj2006;333:597–600.515. Galbraith RF. A note on graphical presentation ofestimated odds ratios from several clinical trials. StatMed 1988;7:889–94.516. Irwig L, Macaskill P, Berry G, Glasziou P. Bias inmeta-analysis detected by a simple, graphical test.Graphical test is itself biased [comment]. BMJ1998;316:470; author reply 470–1.517. McDonald S, Taylor L, Adams C. Searching theright database. A comparison of four databases forpsychiatry journals. Health Libr Rev 1999;16:151–6.518. Laupacis A. Methodological studies of systematicreviews: is there publication bias? Arch Intern Med1997;157:357.519. Dubben H-H, Beck-Bornholdt H-P. Systematicreview of publication bias in studies on publicationbias [see comment]. BMJ 2005;331:433–4.520. Song F. Review of publication bias in studieson publication bias: studies on publication biasare probably susceptible to the bias they study[comment]. BMJ 2005;331:637–8.521. Beveridge WIB. The art of scientific investigation.London: Mercury Books; 1961.522. Meslin E. Toward an ethic in dissemination ofnew knowledge in primary care research. In DunnE, editor. Disseminating research/changing practice.Thousand Oaks: Sage; 1994.523. Anderson SJ. Some thoughts on the reporting ofadverse events in phase II cancer clinical trials. JClin Oncol 2006;24:3821–2.524. Sutton AJ, Duval SJ, Tweedie RL, Abrams KR,Jones DR. Empirical assessment of the effectof publication bias on meta-analyses. BMJ2000;320:1574–7.525. Jennions MD, Moller AP. Publication bias in ecologyand evolution: an empirical assessment usingthe ‘trim and fill’ method. Biol Rev Camb Philos2002;77:211–22.526. Mieog S, Ghersi D. Is timing important insystematic reviews of interventions for breastcancer? [abstract]. 13th Cochrane Colloquium; 22–26October 2005, Melbourne, Australia.527. Bhandari M, Guyatt GH, Tong D, Adili A,Shaughnessy SG. Reamed versus nonreamedintramedullary nailing of lower extremity long bonefractures: a systematic overview and meta-analysis. JOrthop Trauma 2000;14:2–9.528. Horn J, Limburg M. Calcium antagonists forischemic stroke: a systematic review. Stroke2001;32:570–6.529. Marinovich L, Ghersi D, Lord S. Data maturityand systematic reviews of new health technologies[abstract]. 12th Cochrane Colloquium, 2–6 October2004, Ottawa, Canada.530. Martin JLR, Perez V, Sacristan M, Alvarez E. Isgrey literature essential for a better control ofpublication bias in psychiatry? An example fromthree meta-analyses of schizophrenia. Eur Psychiatr2005;20:550–3.531. Maguire M, Hutton J, Marson A. The impact ofincluding grey and non-English literature on thecombined effect estimates from non-randomizedadd-on anti-epileptic studies [abstract]. 14thCochrane Colloquium, 23–26 October 2006, Dublin,Ireland.532. Moller AP, Thornhill R, Gangestad SW. Direct andindirect tests for publication bias: asymmetry andsexual selection. Anim Behav 2005;70:497–506.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8533. Bartlett C, Sterne JA, Juni P, Egger M. Can weignore the non-English literature? A study oflanguage bias in meta-analyses of controlled trials[abstract]. 16th Annual Meeting of the InternationalSociety of Technology Assessment in Health Care, 18–21June 2000, The Hague, The Netherlands.534. Frame SM. Rapid responses to: Article aboutCanadian guidelines on proton pump inhibitors wasmisleading. BMJ 2000;320:1143.535. Hrachovec JB, Mora M. Letter: Reporting of6-month vs 12-month data in a clinical trial ofcelecoxib. JAMA 2001;286:2398.536. Reicin A, Shapiro D. Response to expression ofconcern regarding VIGOR study. N Engl J Med2006;354:1196–9.537. Perez EA, Suman VJ. Lack of publication biasrelated to results from trastuzumab study. Lancet2008;372:626–7.109© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 1Search strategies for electronic databasesMedlineSearch terms1 *publications/2 exp publication bias/3 (bias$adj3 (publication$or disseminat$or language$or reporting or grey or gray or citation$or time delay or time lag ornational or country or location or conference or abstract or duplicat$or multiple publication$)).tw,ot.4 ((reference$or database$or index$) adj2 bias$).tw,ot.5 (file adj drawer$).tw,ot.6 (time adj2 (completion or publication)).tw,ot.7 unpublished research.tw,ot.8 (fail$adj2 publish$).tw,ot.9 Or/1–810 Limit 9 to yr=’1998–2007’Cochrane Methodology RegisterSearch terms1 ‘Study identification’ or2 ‘Information retrieval’ or3 ‘Unpublished data’ or4 ‘Missing data’ or5 ‘Updating and cumulative meta-analysis’ or6 ‘Prospective meta-analysis’ or7 ‘Small study effects’ or8 ‘Small trial bias’ or9 ‘Funding’ or10 ‘Outcome reporting bias’ or11 ‘Bias in review’ or12 (bias* NEAR/3 (publication* or disseminat* or language* or reporting or grey or gray or citation* or time delay or timelag or national or country or location or conference or abstract or reference* or index* or database* or duplicat* ormultiple publication*)) in Title, Abstract or Keywords13 from 1998 to 2007 in Cochrane Methodology Register111© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 1EmbaseSearch terms1 (bias$adj3 (publication$or disseminat$or language$or reporting or grey or gray or citation$or time delay or time lagor national or country or location or conference or abstract or duplicat$or multiple publication$)).tw,ot.2 ((reference$or database$or index$) adj2 bias$).tw,ot.3 (file adj drawer$).tw,ot.4 (time adj2 (completion or publication)).tw,ot.5 unpublished research.tw,ot.6 (fail$adj2 publish$).tw,ot.7 Or/1–68 Limit 9 to yr=’1998–2007’AMEDSearch terms1 exp publications/2 publication bias.tw,ti.3 (bias$adj3 (publication$or disseminat$or language$or reporting or grey or gray or citation$or time delay or time lagor national or country or location or conference or abstract or duplicat$or multiple publication$)).tw,ti.4 ((reference$or database$or index$) adj2 bias$).tw,ti.5 (time adj2 (completion or publication)).tw,ti.6 unpublished research.tw,ti.7 (fail$adj2 publish$).tw,ti.8 or/1–79 limit 8 to yr=’1998–2007’CINAHLSearch terms1 exp publications/2 publication bias.tw,ti.3 (bias$adj3 (publication$or disseminat$or language$or reporting or grey or gray or citation$or time delay or time lagor national or country or location or conference or abstract or duplicat$or multiple publication$)).tw,ti.4 ((reference$or database$or index$) adj2 bias$).tw,ti.5 (time adj2 (completion or publication)).tw,ti.6 unpublished research.tw,ti.7 (fail$adj2 publish$).tw,ti.8 or/1–79 limit 8 to yr=’1998–2007’112

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8MEDLINE search strategy – part IISearch terms1 publication bias2 reporting bias3 OR/1–24 systematic review5 meta-analysis6 OR/4–57 loattrfull text [sb] AND loattrfree full text [sb] AND has abstract[text]8 2000 [PDAT]: 2008 [PDAT]9 English [lang]10 AND/7–9113© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 2Data extraction sheet for empirical studiesAppendix 2. Data extraction sheet for empirical studiesData extraction sheet for empirical studiesAuthor (year): ________________________________Title: ___________________________________________________________Source: __________________________________________________________Study design & objectives: ____________________________________Issues:o Existence/identifying o Causes/risk factorso Consequence o Other: ___________Categories:o Non-publication o Incomplete publicationo Limited accessibility o Other: _____________Specific bias:o Publication bias o Grey literature biaso Language bias o Reporting biaso Abstract bias o Time delay biaso Database index bias o Citation biaso Duplicate bias o Media attention biaso Other: ________________Areas:o general health ospecific health (e.g., obesity) o other: ____________Study results: o Significant/important bias o Non-significant o Can’t tellDetails:Original authors’ conclusions:__________________________________________________________Evidence: o Direct o IndirectFor indirect evidence, stop. For studies with direct evidence, continue:Scientific rigorousness (hints: prospective or retrospective, sample selection bias)_________________________________________________________________Sample representativeness (hints: research field, participants, outcomes, interventions; study designs)__________________________________________________________________Appropriateness of data analysis (hints: consider objectives, available data and methods of dataanalysis)__________________________________________________________________Appropriateness of interpretations (hints: limitations of the study should be taken into consideration)_________________________________________________________________Overall study quality:o High (hint: without considerable concern on study validity)o Moderate (hint: with some concern on study validity)o Low (hint: with considerable concern on study validity)o Can’t tellAny other comments:115© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 3Data extraction sheet formethodological studiesAppendix 3. Data extraction sheet for methodological studiesData extraction sheet for methodological studiesAuthor (year): ___________________________________________________Title: ________________________________________________________Source: ______________________________________________________Study design:________________________________________________Study objectives:o New method o Established method o Evidence of usefulness/limitationsMethods:o Study registration o Literature searcho Funnel plot o Statistical/modellingo Updating reviews o Publication processo Research ethics/policy o Confirmatory studieso Other: _____________Purpose:o Preventing bias o Reducing bias o Detecting bias o Adjusting biaso Other:_____Stage of literature review:o Before literature review o In literature review o After literature reviewWhat dissemination bias the method is relevant:o Publication bias o Grey literature biaso Language bias o Reporting biaso Abstract bias o Time delay biaso Database index bias o Citation biaso Duplicate bias o Media attention biaso Other: ________________Main findings and conclusions:______________________________________________________________Resources required to use the method:________________________________________________________________Reviewer’s commentary (study’s validity, scientific rigorousness, method’s usefulness andlimitations, any empirical evidence provided):117© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 4Data extraction sheet – systematicreviews of treatment119© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 4Appendix 4.Data Extraction Sheet – Systematic Reviews of TreatmentData Extraction Sheet – Systematic Reviews of TreatmentAuthor (year): __________________ Reviewer: ______________Date: ______Title: ______________________________________________________________________Source: ____________________________________________________________________Review Characteristics:Participants: _______________________________________________________________Interventions (and control): ___________________________________________________Drug Surgical Educational/behavioural Alternative OtherOutcomes: ________________________________________________________________Total no of outcomes evaluated: ________Primary outcome(s) defined? Yes NoType of reviews: Narrative Meta-analysis; How many? ______Designs of included studies:RCTs/CCTs (Study = ; patients = )Other_______________ (Study = ; patients = )How were differences between studies investigated?Narrative Meta-regression / SubgroupStatistical: I 2 = ____ P = ____Other: _________NASources searched to identify studies: Not statedMEDLINE EMBASE Psychlit Cochrane ReferencesCINAHL Handsearch Experts/authors CompanyProceedings Other ______________________________________Language restriction: Unclear No Yes, what language(s) included:_____________Non-English language studies:Searched: Unclear No Yes If yes, search methods: ___________Included: Unclear No Yes, how many _____For Main analysis; sensitivity analysis?Grey literature/conference abstracts:Searched: Unclear No Yes If yes, search methods: __________Included: Unclear No Yes, how many _____For main analysis; sensitivity analysis?Other unpublished studies:Searched: Unclear No Yes If yes, search methods: __________Included: Unclear No Yes, how many _____For main analysis; sensitivity analysis?Are all the relevant trials included in meta-analyses? Unclear YesNo, how many _________Outcome reporting bias considered? No YesWere there missing outcome data? Unclear No Yes120If yes, methods used to deal with missing data on outcomes: _________________________Methods used for dealing with publication bias:Not usedIdentify unpublished studiesProspective registerFail-safe N191

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Funnel plotEgger’s methodModellingRank correlation (Begg method)Large scale trialsOther: ___________Details:Issue of publication bias discussed? No YesEvidence on publication and related bias: Not available AvailableIf available, details (such as results of published trials versus unpublished trials; or shape of Funnel plotor related methods)Meta-analysis results:Not applicable (no meta-analysis)Statistically significant (at least one primary outcomes)Non significant (primary outcomes)Authors’ conclusion:Significant/positive (At least one intervention recommended; or significant difference found)Non-significant/not important (No intervention recommended, or no significant differences)Unclear (Not able to judge; neither positive nor negative; lack of evidence)___________________________________________________________________________Assessor’s Judgement:Efforts to minimise publication bias Sufficient Partial InsufficientRisk of Publication Bias (Considering the possibility that authors’ conclusion might be wrong becauseof possible publication and related biases):High Moderate LowReasons, if any:___________________________________________________________________________Any other comments:192121© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 5Main characteristics of inceptioncohort studies of publication bias123© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 5124Cohort typesSpecialityStudy designFollow-upVerification ofpublication statusand results ofunpublished studies Publication rateDefinition of study results and othernotesStudyClinical trialsFollow-up: 5–6 yearsMEDLINE searched forpublicationsQuestionnaires sent totrial sponsors for trialresultsPositive 47% (52/111)Inconclusive 33% (11/33)Negative 11% (5/44)Positive: the drug better (or equivalent toin equivalent trials) than comparators, orthe objective of the study supported orconfirmedInconclusive: exploratory studies ornon-comparative or the risk-benefit wasinconclusivePublication: published in journals includedin MedlineBardy 1998 26 Clinical trials onmedicinal productsnotified to the NationalAgency for Medicines in1987, FinlandMixed specialityMixed. Includingquantitative (47%) andqualitative (53%) researchFollow-up: > 2 yearsQuestionnaires sent toinvestigators17% failed to respondQuantitative or qualitativeeffect (n = 70) published in peerreviewedjournals:Showed an effect 76% (26/34)No effect 64% (23/36)Methods used in Dickersin et al. 20 wereadopted to classify findingsCronin and Sheldon 2004 27 Studies sponsoredby the NHS R&Dprogramme (the NorthThames RegionalOffice) from 07/1995 to12/1998Mixed specialityMixed:Ob = 51%CT = 25%Follow-up: 8 yearsQuestionnaires sent toinvestigators, up to 3times20% failed to respondCompleted studies:Important 70% (26/37)Less important 60% (6/10)Investigators rated the importance ofresults from 1 to 10. Important resultswere those > 5Decullier and Chapuis 2006 29 Protocols submittedfor funding to the G.Lyon regional scientificcommittee in 1997Mixed specialityMixed:Ob = 13%Exp (CT) = 87%.Follow-up: 5–7 yearsQuestionnaires sent toinvestigators, or fromREC databases31% failed to respondConfirmatory 69% (129/188)Invalidating 19% (3/16)Inconclusive 32% (14/44)Confirmatory: results confirming studyhypothesisInvalidating: results invalidating studyhypothesisInconclusive: not confirming orinvalidatingDecullier et al. 2005 28 Biomedical researchprotocol approved byFrench RECs in 1994Mixed: biomedicalresearchNIH 1979 fundedclinical trials that werecompleted by 1988Mixed specialityClinical trials:CT = 100%Follow-up: 9 yearsContacting andtelephone interview ofinvestigators26% failed to respondSig/important 98% (121/124)Non-significant 85% (63/74)Significant results: p < 0.05 or deemed tobe of ‘great importance’Non-significant results: all other resultsDickersin and Min1993 21

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Cohort typesSpecialityStudyStudy designFollow-upVerification ofpublication statusand results ofunpublished studies Publication rateDefinition of study results and othernotesDickersin et al. 1992 20 Studies approved byIRBs at Johns HopkinsHealth Institutions upto the end of 1980Mixed specialityMixed:Ob = 37%/85%Exp = 17%/9%CT = 46%/6%Follow-up: > 7 yearsTelephone interview ofinvestigators30% failed to provideadequate dataMedicine & hospital:Sig/important 89% (184/208)Non-significant 69% (93/134)Public health:Sig/important 71% (75/106)Non-significant 58% (38/66)Clinical trials (both centres):Sig/important 87% (84/96)Non-significant 72% (52/72)Significant results: p < 0.05 or resultsconsidered to be of great importanceNon-significant results: all other resultsRisk of publication bias may beunderestimated by excluding studies dueto lack of informationUnpublished data for clinical trialsobtained from Hopewell et al. 2009 83Easterbrook et al. 1991 22 Studies approved by theCentral Oxford RECbetween 1984 and 1987Mixed specialityMixed:Ob = 30%Exp = 18%CT = 52%Follow-up: 3–6 yearsQuestionnaires sentto investigators,followed by a telephoneinterview8% failed to respond orprovide adequate dataFully published:Significant 60% (93/154)Non-significant trend 35%(12/34)No difference 34% (33/97)Published or presented:Significant 85% (131/154)Non-significant trend 65%(22/34)No difference 56% (54/97)Significant results: p < 0.05Non-significant trend: difference with a pvalue of ≥ 0.05Null: no differenceExamined factors associated withpublication (but not necessarilypublication bias)Ioannidis 1998 23 RCTs conducted bytwo trialist groups(sponsored by the NIH)from 1986 to 1996AIDS/HIVClinical trial:CT = 100%Follow-up: 1–10 yearsInformation obtainedfrom a database of HIVtrials sponsored byNIH. Supplemental datafrom investigators andstaff responsible for theprotocolsPositive 74% (20/27)Non-positive 41% (16/39)Positive: statistically significant (p < 0.05)in favour of an experimental armNon-positive: significantly in favour ofthe control arm or non-significant. Thefocus of the study was time lag bias. Dataobtained from Hopewell et al. 113Misakian and Bero 1998 30 Research on passivesmoking funded by 76organisations between1981 and 1995Health effects of passivesmokingMixed:Exp = 23%Obs = 77%Follow-up: median 5 yearsSemistructuredtelephone interview ofinvestigators17% failed to respondSignificant 85% (28/33)Non-significant 86% (18/21)Mixed 14% (1/7)Statistically significant: p ≤ 0.05 Mixedresults: multiple primary outcomes atleast one was statistically significantCox regression analysis was used toestimate hazard ratio125© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 5126Verification ofpublication statusand results ofunpublished studies Publication rateStudy designFollow-upDefinition of study results and othernotesCohort typesSpecialityStudyQuantitative studiesSignificant 68% (99/146)Non-significant trend 20% (4/20)No difference 44% (23/52)Questionnaires sent toinvestigators30% failed to respondMixed:Ob = 22%Exp = 22%CT = 56% (details available)Follow-up: 3–12 yearsStern and Simes 1997 24 Studies submitted toRoyal Prince AlfredHospital REC between1979 and 1988Mixed specialityQuantitative studiesSignificant: p < 0.05Non-significant trend: p ≥ 0.05 ≤ 0.10No difference: p ≥ 0.10Classification of qualitative studies basedon principal investigators’ judgementWith data on time delayed publication.Qualitative studies similarly vulnerable topublication biasQualitative studiesStriking 70% (19/27)Important/definite 59% (35/59)Negative/unimportant 53% (9/17)Clinical trials (n = 167)Quantitative trials:Significant 72% (55/76)Non-significant trend 20% (3/15)Null 38% (15/39)Qualitative trials:Striking 50% (3/6)Important/definite 61% (11/18)Negative/unimportant 69% (9/13)Retrospective review Significant 93% (14/15)Non-significant 71% (15/21)Significant: p < 0.05Non-significant: p ≥ 0.05Published as a brief abstract, additionaldata from Dwan et al. 41RCTsFollow-up: > 2 yearsWormald et al. 1997 31 Randomised trialsprocessed through thePharmacy of MoorfieldsEye Hospital since 1963Eye healthPositive 40% (43/107)Negative 28% (15/53)Literature search andcontacting investigatorsResponse rate unclearFull publication in German, informationobtained mainly from the abstractMixedFollow-up: > 5 yearsZimpel and Windeler 2000 32 140 medical theses oncomplementary medicalsubjectsComplementarymedicineCT, clinical trial; RCT, randomised controlled trial; Ob, observational study; Exp, experimental study.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 6Main characteristics of included regulatorycohort studies of publication bias: trialssubmitted to regulatory authorities127© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 6128Cohort typesSpecialityStudy designFollow-upVerification of publicationstatus and results ofunpublished studies Publication rateDefinition of study results andother notesStudyClinical trialsFollow-up: > 5 yearsSearches of PubMed and otherdatabasesTrial results obtained fromFDA filesStatistically significant 66%(285/432)Not statistically significant 36%(52/144)Statistical significance (primaryoutcome): p < 0.05 or if an equivalencytrial p > 0.05 or a CI excluding theprespecified differenceLee et al. 2008 42 Trials supporting newdrugs approved by FDAbetween 1998 and 2000Mixed specialityClinical trialsFollow-up: unclearLiterature search forpublicationsStudy results based onsubmitted materialStand-alone or pooled publications:Significant 100% (21/21)Non-significant 81% (17/21)Stand-alone publications:Significant 90% (19/21)Non-significant 29% (6/21)Also provided data on duplicate, andselective outcome reporting biasMelander et al. 2003 43 RCTs of five SSRIssubmitted to theSwedish drug regulatoryauthority for marketingapprovalMental: depressionClinical trialsFollow-up: > 4 yearsSearch of PubMed andCochrane Library andcontacting sponsors andauthorsTrial results obtained fromFDA filesPrimary outcomesFavourable 82% (102/124)Not favourable 66% (19/29)Unknown 60% (6/10)ConclusionFavourable 79% (90/114)Not favourable 64% (7/11)Unknown 57% (4/7)Favourable: statistically significantly infavour of the new drug or equivalencewas foundNote: pooled publication not reportedseparately was considered ‘notpublished’Rising et al. 2008 44 Efficacy trials supportingnew drug applicationsapproved by FDA from2001 to 2002Mixed specialityPhase 2 and 3 clinicaltrialsFollow-up: > 2 yearsLiterature search andcontacting sponsors forpublicationsTrial results obtained fromFDA filesPositive 97% (37/38)Questionable 50% (6/12)Negative 33% (8/24)Trial results were classified according tothe FDA’s regulatory decisionsTurner et al. 2008 45 Trials of 12antidepressants,submitted to FDAbetween 1987 and 2004Mental: depression

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 7Main characteristics of abstract cohortstudies of publication bias: abstractspresented at conferences129© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 7130Cohort typesSpecialityStudy designFollow-upVerification ofpublication status andresults of unpublishedstudies Publication rateDefinition of study results and othernotesStudyPresented meetingabstractsLaser medicine andsurgeryClinical trialsFollow-up: > 3 yearsLiterature search for fullpublicationsSignificant (p < 0.05): 51% (23/45)Non-significant 50% (22/44)Positive 43% (59/137)Not-positive 38% (3/8)Positive: if stated the intervention hassome beneficial effect. Negative: against theintervention. Equivocal: no statement aboutthe effectiveness or the comparison groupsconsidered the sameAkbari-Kamraniet al. 2008 56Studies of diagnostictestsFollow-up: > 2 yearsLiterature search andcontacting authors for fullpublicationsClinical utilityAccurate 76% (107/141)Possibly or non-informative 68%(13/19)Sensitivity (median 0.91)Above median 77% (38/49)Below median 74% (34/46)Not given 75% (49/65)Specificity (median 0.91)Above median 71% (30/42)Below median 73% (27/37)Not given 79% (64/81)Accurate: diagnostic accuracy of the testwas high enough to recommend its use inclinical practice. Possibly useful: having agood sensitivity but not necessarily a goodspecificity (and vice versa). Non-informative:the accuracy of the test was not goodenough to recommend its use in clinicalpractice or equivalent to or not better thanthat of an existing alternative testBrazzelli et al. Presented meeting2009 57 abstractsStrokeMixed:CT = 26%Follow-up: 5 yearsLiterature search andcontacting authors for fullpublicationsPositive 50% (77/153)Not positive 49% (36/74)Positive results: beneficial results or p < 0.05.Publication rates from CMRD by Scherer etal. 55Callaham et al. Submitted meeting1998 47 abstractsEmergency medicineMixed:Ob = 69%RCT = 31%.Follow-up: 4–5 yearsLiterature search for fullpublicationsSignificant 44% (160/361)Non-significant 41% (23/56)Significant: p < 0.05Castillo et al. Presented meeting2002 58 abstractsAnaesthesiologyClinical trials:CT = 100%.Follow-up: > 4 yearsThe Oxford Databaseof Perinatal Trials wassearched to identify fullpublicationsPositive 33% (32/98)Neutral/negative 41% (32/78)Positive: the test treatment superior to thecontrolNegative: the test treatment potentiallyharmfulNeutral: no real differenceChalmers et al. 1990 48 Summary trial reportspublished from 1940 to1984, included in theOxford Database ofPerinatal TrialsPerinatal

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Cohort typesSpecialityStudyStudy designFollow-upVerification ofpublication status andresults of unpublishedstudies Publication rateDefinition of study results and othernotesCheng et al. 1998 49 Abstracts fromthree internationalconferences over a 30-year periodCystic fibrosisClinical trials:CT = 100%.Follow-up: notreportedLiterature search for fullpublicationsPositive 38% (43/113)Negative 33% (14/42)Positive: authors concluded the testtreatment was superior to control or equallyeffective (in equivalence trials)Publication rates from CMRD by Schereret al. log-rank tests showed no significantdifference in time to publication between‘positive’ or ‘negative’ results (p = 0.54)De Bellefeuille Submitted meetinget al. 1992 50 abstractsClinical oncologyMixed:CT = 48%Follow-up: 5 yearsLiterature search andcontacting authors for fullpublicationsPositive 74% (48/65)Negative 32% (10/31)Neutral/descriptive 56% (57/101)Positive results: p < 0.05 or beneficial tointerventionsDelamere and Conference abstractsWilliams 2005 59 DermatologyClinical trials:CT = 100%Follow-up: 3–5 yearsLiterature search for fullpublicationsPositive 68% (15/22)Negative 0% (0/2)Neutral 17% (1/6)Unclear definition of positive or negativeresults. Only abstract availableEloubeidi et al. 2001 60 Submitted meetingabstractsGastrointestinalendoscopyMixed:RCT = 9%Follow-up: 4 yearsLiterature search for fullpublicationsSignificant 37% (36/98)Non-significant: 22% (77/353)Positive: statistically significant p < 0.05Multivariate adjusted OR: 0.97 (0.58–1.60);HR = 1.92 (1.28–2.87). Also with data onpresentation acceptanceEvers 2000 61 Presented meetingabstractsReproductiveRCTs = 100%Follow-up: 4–8 yearsLiterature search for fullpublicationsSignificant 59% (41/69)Non-significant 46% (38/82)Significant: p < 0.05Glick et al. Presented meeting2006 62 abstractsOrgan transplantationMixed:Ob = 81%Other = 13%RCT = 6%Follow-up 4–5 yearsLiterature search for fullpublicationsSignificant 52% (208/397)Not specified 59% (304/516)Non-significant 41% (95/234)Significant: p < 0.05Statistical significance was excluded frommultivariate analysis due to > 5% of datamissingHa et al. 2008 63 Presented meetingabstractsRadiologyMixedFollow-up: > 4 yearsLiterature search for fullpublicationsPositive 29% (288/982)Negative 11% (13/115)Positive outcomes: benefical or statisticallysignificant resultsNegative: non-positive results131© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 7132Cohort typesSpecialityStudy designFollow-upVerification ofpublication status andresults of unpublishedstudies Publication rateDefinition of study results and othernotesStudyMixedFollow-up: > 5 yearsLiterature search for fullpublicationsPositive 35% (29/83)Not positive 19% (9/47)Positive results: significant results. Lack ofdetails. Unpublished data reported in Schereret al. 55Halpern et al. Presented meeting2001 64 abstractsAnaesthesiologyMixed:Ob = 72%Other = 26%RCT = 2%Follow-up: 5 yearsLiterature search andcontacting authors for fullpublicationsPositive 34% (45/132)Negative 50% (5/10)Neutral 21% (12/58)Significant (p < 0.05) 50% (12/24)Non-significant 28% (50/176)Positive results: beneficial regardless of pvaluesNegative results: against the interventionNeutral results: no opinionAfter adjusting for study setting (clinical orlaboratory), neither statistical significance(p = 0.2) nor direction of outcome (p = 0.3)were significantly associated with publicationHarris et al. Presented meeting2006 66 abstractsOrthopaedicsMixed:Ob = 82%Other = 12%RCT = 6%Follow-up: 5 yearsLiterature search andcontacting authors for fullpublicationsPositive 61% (123/203)Negative 53% (18/34)Neutral 43% (35/81)Significant (p < 0.05) 68% (69/101)Non-significant 49% (107/217)Positive results: beneficial regardless of pvaluesNegative results: against the interventionNeutral results: no opinionHarris et al. Presented meeting2007 65 abstractsOrthopaedicsMixed:Basic = 5%Ob = 69%Analytic/CT = 26%Follow-up: 3 yearsLiterature search andcontacting authors for fullpublicationsPositive 48% (54/112)Negative 14% (2/14)Neutral 27% (36/131)Definition of positive, negative or neutralresults not providedHashkes and Presented meetingUziel 2003 67 abstractsPaediatric rheumatologyMixed:RCT = 4%CT = 31%(subgroup – clinicaltrials)Follow-up: 3–5 yearsContacted authors for fullpublicationsAll studies:Positive 71% (98/139)Negative/inclusive 48% (76/159)Clinical trials:Positive 92% (11/12)Negative/inclusive 50% (4/8)Significance or importance of the resultsbased on information from authors, but lackof detailsA single investigator assessed meetingabstractsKiroff 2001 68 Presented meetingabstractsSurgery

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Cohort typesSpecialityStudyStudy designFollow-upVerification ofpublication status andresults of unpublishedstudies Publication rateDefinition of study results and othernotesKlassen et al. Presented meeting2002 69 abstractsPaediatricRCT = 100%Follow-up: 5–8 yearsLiterature search for fullpublicationsFavouring treatment 69% (162/235)Non-favourable 50% (93/187)Favouring treatment: overall conclusionsfavoured the interventionWith data on time to publication, andabstract biasKrzyzanowska Presented meetinget al. 2003 70 abstractsOncologyLarge clinical trials(n > 200):CT = 100%Follow-up: < 5 yearsLiterature search andcontacting authors for fullpublicationsSignificant 81% (181/223)Non-significant 68% (195/287)Positive 81% (148/183)Negative 70% (229/327)Significant results: p ≤ 0.05Positive results: p ≤ 0.05 in favour of theexperimental treatmentWith data on time to publicationLandry 1996 51 Presented meetingabstractsBurn researchMixed:CT = 27%.Follow-up: 4 yearsLiterature search for fullpublicationsPositive 41% (24/58)Non-positive 18% (20/110)Positive: p < 0.05 or stated to be positiveData not clearly presented. Publication ratefrom CMRD by Scherer et al.Loep and Abstracts initiallyKleijnen 1999 52 published in a journalUnclearClinical trialsFollow-up: > 1 yearLiterature search andcontacting authors for fullpublicationsPositive 81% (72/89)Negative 81% (34/42)Data from the 2000 HTA report onpublication bias, based on unpublishedmanuscriptPeng et al. 2006 71 Presented meetingabstractsOtolaryngology: headand neck surgeryMixedFollow-up: > 5 yearsLiterature search for fullpublicationsPositive 61% (189/337)Negative 50% (13/26)Unclear definition of positive or negativeresultsPetticrew et al. Presented meeting1999 53 abstractsSocial medicineMixed:CT = 5%Follow-up: 2 yearsLiterature search andcontacting authors for fullpublicationsPositive 50% (22/36)Uncertain 56% (19/34)Negative 57% (4/7)Classification of results based on subjectiveassessment of the study results and theauthors’ conclusionsSanossian et al. Presented meeting2006 72 abstractsStrokeMixed:CT = 2%Follow-up: 5 yearsLiterature search andcontacting authors for fullpublicationsPositive 62% (136/220)Non-positive 62% (83/133)Positive: beneficial or supported hypothesisor objective, and either p < 0.05 or nostatistical test reportedAdjusted publication rate: 64% for positiveand 59% for non-positive results. Clinicaltrials 100% publishedScherer et al. Presented meeting1994 54 abstractsOphthalmologyClinical trials:CT = 100%Follow-up: 3 yearsLiterature search andcontacting authors for fullpublicationsSignificant 72% (33/46)Non-significant 59% (28/47)Statistically significant p < 0.05133© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 7134Cohort typesSpecialityStudy designFollow-upVerification ofpublication status andresults of unpublishedstudies Publication rateDefinition of study results and othernotesStudyMixed clinicalresearchFollow-up: > 2 yearsLiterature search for fullpublicationsSignificant 47% (521/1120)Non-significant 43% (86/202)Positive results were those showingstatistically significant results (p < 0.05)regardless of the directionSmith et al. Presented meeting2007 73 abstractsUrologyControlled clinicaltrials (39%), otherclinical research (40%)and basic studies(21%)Follow-up: 3–6 yearsLiterature search for fullpublicationsAll abstractsSignificant (p < 0.05): 50% (177/354)Non-significant 47% (69/147)Equivocal 43% (144/335)Controlled clinical trialsSignificant (p < 0.05): 60% (84/140)Non-significant 48% (47/99)Equivocal 45% (39/87)Significant results: p < 0.05. Equivocal results:no statements concerning the statisticalsignificance of the main or the majority ofotucomesTimmer et al. 2002 74 A random sample ofabstracts submitted to aconferenceGastroenterologyClinical trials:CT = 100%Follow-up: > 5 yearsLiterature search for fullpublicationsPositive 75% (120/161)Negative/null 47% (24/51)No statistical results 61% (198/325)No results: 39% (17/44)Positive results: statistically significant results(p < 0.05) in favour of experimental armNegative or null: significant results (p < 0.05)in the control arm or not significant(p ≥ 0.05)Vecchi et al. Presented meeting2006 75 abstractsDrug addictionMixed:Basic = 25%Ob = 36%RCT = 1%Follow-up: < 2 yearsLiterature search for fullpublicationsSignificant (p < 0.05) 70% (105/151)Non-significant 41% (13/32)Significant results: p < 0.05Zamakhshary et Presented meetingal. 2006 76 abstractsPaediatricPhase III trials (n = 57)Follow-up: 7 yearsLiterature search for fullpublicationsThe rates of publication of positiveand negative results not significantlydifferent (p = 0.53)Only a short abstract availableZaretsky and Presented meetingImrie 2002 77 abstractsHaemotology

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 8Main characteristics of manuscriptcohort studies of publication bias:manuscripts submitted to journalsStudy Methods Main findingsOlson et al.2002 81 Cohort study of 745 manuscripts of controlledtrials submitted to JAMA from 02/1996 to 08/1999Lee et al. 2006 78Outcome classification:1. Statistically significant (p < 0.05) for the primaryoutcome2. Statistically non-significant3. UnclearCohort study of 1107 manuscripts of originalresearch (including qualitative research, excludingsingle case reports) submitted to BMJ, Lancet andAnnals of Internal Medicine during 01–03/2003 andduring 11/2003–02/2004Outcome classification:1. Statistically significant (p < 0.05) for the primaryoutcome2. Non-significantLynch et al.2007 79 Cohort study of 209 manuscripts of originalresearch on hip or knee arthroplasty submittedto the Journal of Bone and Joint Surgery (AmericanVolume) between 01/2004 and 06/2005Outcome classification:1. Positive or favourable or significant2. Negative or non-supportive or no difference3. Not analysable – unclearOkike et al. Cohort study of 855 manuscripts as scientific2008 80 articles submitted to the Journal of Bone and JointSurgery (American Version) between 01/2004 and06/2005Outcome classification:1. Positive – favoured experimental item2. Negative – favoured existing standard of careover the experimental item3. Neutral – no difference4. Not applicableProportion of studies with different results:51.4% (n = 383) with significant results45.7% (n = 341) with non-significant results2.8% (n = 21) with unclear resultsAcceptance rate:20.4% (78/383) for significant results15.0% (51/341) for non-significant results19.0% (4/21) for unclear results.Logistic regression analysis: significant vs nonsignificantresults OR = 1.30 (95% CI: 0.87 to 1.96)Proportion of different statistical results:86.8% (n = 718) with significant results13.2% (n = 109) with non-significant resultsAcceptance rate:4.9% (35/718) for significant results6.4% (7/109) for non-significant resultsMultivariate analysis: OR = 0.83 (95% CI: 0.34 to1.96)Proportion of studies with different results:70.8% (n = 148) with positive results23.4% (n = 49) with negative results5.7% (n = 12) with unclear resultsAcceptance rate:30.4% (45/148) for positive results36.7% (18/49) for negative results8.3% (1/12) for unclear resultsDifference in publication rate between positive andnegative outcomes was not statistically significant(p = 0.41)Proportion of studies with different results:72.5% (n = 620) with positive results12.3% (n = 105) with negative results15.2% (n = 130) with neutral resultsAcceptance rate:21.3% (132/620) for positive results21.0% (22/105) for negative findings24.6% (32/130) for neutral resultsMultivariate analysis: positive vs nonpositiveOR = 0.92 (95% CI: 0.62 to 1.35)135© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 9Outcome reporting bias –characteristics of included studies137© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 9138Study Cohort or sample, methods Findings Notes61% of results were usable in dose–response meta-analyses. Themost important reason for results not being usable was the absenceof sufficient information on exposure levels in the different groups.Results that showed evidence of an association were more likely tobe usable than results that found no such evidenceBekkering et al. 2008 93 Based on two systematic reviews of 767observational studies (with 3284 results) of theassociation between diet and prostate or bladdercancer. The paper examined the proportion ofresults that had detail sufficient for use in metaanalysesestimating dose–response associationsOf the 48 published RCTs, only 30for efficacy and 4 for harm outcomeswere included for analyses. Only 22trialists provided information about thestatistical significance of unreportedoutcomes. Trials were also excludedif they had fully reported outcomes,or had only incompletely reportedoutcomes, or all outcomes beingsignificant or non-significant. It is unclearwhether the exclusion of trials mightsystematically affect the estimatedassociations between outcome reportingand study resultsThe median number of participants per trial was 299. A medianof 31% of efficacy outcomes and 59% of harm outcomes wereincompletely reported. Statistically significant outcomes had higherodds than non-significant outcomes of being fully reported: OR = 2.7(95% CI: 1.5 to 5.0) for efficacy outcomes and OR = 7.7 (95% CI: 0.5to 111) for harm outcomesPrimary outcomes differed between protocols and publications for40% of the trialsRCT protocols approved by the CanadianInstitute of Health Research from 1990 to 1998and subsequent journal publications. Reportedand unreported outcomes were recorded fromprotocols and journal articles. If a publishedarticle provided insufficient data for meta-analysis,the outcome was defined as being incompletelyreported (i.e. partial + qualitative + unreported).Used odds ratios to measure association betweenthe completeness of outcome reporting andstatistical significanceOf 105 RCTs approved for funding, 48 published;with a total of 1402 outcomes measured: 1233efficacy outcomes in 48 trials and 169 harmoutcomes in 26 trialsChan et al.2004 (CMAJ) 7Journal articles published mainly in1998–9, before the 2001 CONSORTstatement. Survey response rate was low,which may lead to an underestimation ofbias. 49/99 trials measuring efficacy and54/72 trials measuring harm outcomeswere excluded from the analysis ofreporting bias due to entire rowsor columns being empty in the 2 × 2table. 22% of efficacy and 35% of harmoutcomes were ineligible for analysis dueto unknown statistical significanceOverall 50% of efficacy and 65% of harm outcomes per trial wereincompletely reported. Statistically significant outcomes had a higherodds of being fully reported compared with non-significant outcomesfor both efficacy (OR = 2.4; 95% CI: 1.4 to 4.0) and harm (OR = 4.7;95% CI: 1.8 to 12.0) data. In comparing published articles withprotocols, 62% of trials had at least one primary outcome that waschanged, introduced, or omitted. 86% of survey responders (42/49)denied the existence of unreported outcomes despite clear evidenceto the contraryExploratory meta-regression analysis found no significant associationbetween the extent of bias and the source of funding, sample size, ornumber of study centresRCT protocols approved by the REC in Denmarkin 1994–5. Reported and unreported outcomeswere recorded from protocols, journal articles anda survey of trialists. If a published article providedinsufficient data for meta-analysis, the outcomewas defined as being incompletely reported. Oddsratios were used to measure association betweenthe completeness of outcome reporting andstatistical significanceIdentified 102 RCT protocols, 122 correspondingjournal articles and 3736 outcomesChan et al.2004 (JAMA) 6Low response rate may lead tounderestimation of bias. Trial protocolswere not reviewed so that fewerunreported outcomes were identifiedcompared with previous two studies bythe same investigatorsMedian percentage of incompletely reported outcomes per trial: 42%for efficacy outcomes and 50% for harm outcomes. 33% (169/505) oftrials had at least one unreported efficacy outcome and 28% (85/308)of trials had unreported harms data. Pooled ORs for outcomereporting bias: 2.0 (95% CI: 1.6 to 2.7) for efficacy outcomes, and 1.9(95% CI: 1.1 to 3.5) for harm outcomesChan et al. RCTs indexed in PubMed appeared in Dec 2000.2005 5 Trialists were surveyed to obtain information onunreported outcomes

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Study Cohort or sample, methods Findings Notes(44% of trials found to have one ormore unreported outcomes in thisstudy, compared with 76% and 98% oftrials in the earlier studies based on trialprotocols)Reasons given by authors for not reporting efficacy and harmoutcomes, respectively: space constraints (47%, 25%); not clinicallyimportant (37%, 75%); not statistically significant (24%, 50%); not yetsubmitted (22%, 6%); not yet analysed (17%, 6%)519 trials identified with 10,557 outcomes. Surveyresponders (69%) provided information onunreported outcomes (but unreliable)Concluded that the under-reportingappears to be biased. Discussed possibleexplanations for incomplete inclusionof outcomes. Did not assess potentialconfounders or explanatory variablesA median of 46% (IQR 20–75%) of identified RCTs in each metaanalysiscontributed to the pooled estimates. Regression analysis ofpercentage contributing RCTs and effect size: β = – 0.16 (95% CI:– 0.29 to – 0.01) for OR and β = – 0.18 (95% CI: – 0.35 to – 0.01) forSMD. When outcomes favoured the control, regression coefficientwas β = 0.16 (95% CI: – 0.16 to 0.45) for OR and β = – 0.08 (95% CI:– 0.37 to 0.22) for SMDFurukawa et al. 2007 94 A random sample of 156 Cochrane systematicreviews with 10 or more RCTs. Examinedpercentage of identified RCTs that contributedin meta-analysis; and the association betweenthe percentage of RCTs contributing and pooledestimates (odds ratio and standardised meandifference)Only abstract availableNot clear about the association betweenstatistical significant and the selectivereporting of primary outcomesAuthors contacted for full publication17% of primary outcomes in the protocol were not reported asprimary outcomes in the publication. 15% of primary outcomes inthe publication not declared as primary outcomes in the protocol.Trials where all of the comparisons were statistically significant weremore likely to report fully all of their comparisons (p = 0.06)A comparison of 103 published RCTs and theirprotocols considered by Central Sydney AreaHealth service ERC from Jan 1992 to Dec 1996Ghersi etal. 2006 91(abstract)A pilot study: before this study, ‘theoriginal protocol for a study and itssubsequent report have never beencompared in order to summarize theconsistency between them using astructured framework for an assessmentof within-study selection’Of the 15 published studies, six stated which outcome variables wereof primary interest and four showed consistencies in the report.Eight mentioned an analysis plan. However, seven of these eightstudies did not follow their prescribed analysis plan: the analysis ofoutcome variables or associations between certain variables wasfound to be missing from the reportHahn et al. 2002 36 A pilot study comparing local REC approvedprotocols and results presented in subsequentpublications41 (73%) responses received from trialists of 56LREC approved projects (5 years ago). 18 published(but publications available only for 15. For theremaining three, two researchers did not agree toprovide their article nor any references, and thecopy of one presented as a conference poster wasno longer availableThe preponderance of significantfindings was less prominent in the fulltexts of the articlesSelection of results by multiple analysesmethods to measure the same outcome.The use of extreme contrasts mayindicate zero or very small effect sizeIn the abstracts: 88% reported ≥ 1 significant RR and only 43%reported ≥ 1 non-significant RRFull text of the 50 articles: a median of 9 (IQR 5–16) significant and 6(IQR 3–16) non-significant RRs were presentedParadoxically, the smallest presented RRs were based on thecontrasts of extreme quintilesKavvoura et al. 2007 98 389 abstracts and 50 randomly selected full papersof epidemiological studies. Examined percentage ofreporting statistically significant and non-significantresults in the abstract and the association betweenthe RRs and the type of contrast usedThe definitions of outcomes were alsoselected to exaggerate the associationDiscussed implications and approachesto dealing with the selective reportingbiasn = number of trials (number of patients)Published/indexed: n = 18 (1364); RR = 1.27 (95% CI: 1.06 to 1.53)Published/not indexed: n = 13 (1028); RR = 1.13 (95% CI: 0.81 to 1.59)Retrieved: n = 11 (996); RR = 0.97 (95% CI: 0.72 to 1.29)All studies: n = 42 (3388); RR = 1.16 (95% CI: 0.99 to 1.35)The association was stronger by using the definitions preferred byeach publication (RR = 1.38, 1.13–1.67) than when definitions werestandardised (RR = 1.27, 1.06 to 1.53)Kyzas et al. 2005 97 Meta-analysis of studies on the associationbetween the tumour suppressor protein (TP53)and mortality outcome of patients with head andneck squamous cell cancer. Study categories: (1)published and indexed with ‘mortality’ or ‘survival’;(2) published but without ‘mortality’ or ‘survival’index in Medline and Embase; (3) retrieved: dataretrieved from authors for those that suggestedmortality collected but reported no usable data139© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 9140Study Cohort or sample, methods Findings NotesThis case study indicates that someoutcomes (e.g. persisting pain, anoutcome rarely reported) may be morevulnerable than others (e.g. herniarecurrence) to the selective reporting.IPD meta-analysis may be an approachto dealing with selective reportingHernia recurrence outcomes: numbers of contributing RCTs weresimilar and the results had no significant differencePersisting pain outcome: many more RCTs were included in IPDupdate (e.g. 20 vs 3), and results were qualitatively divergentA meta-analysis based on aggregate published datawas compared with an updated IPD meta-analysisof trials of hernia surgeryMcCormacket al. 2004 92Different from other studies: onlyone outcome (response rate) wasconsidered. The selection based ondifferent analysis methods (ITT vsper protocol). Also provided data onduplicate biasAll 21 trials with significant results but only 17 of the 21 trials withnon-significant results were published. Many publications ignoredthe results of ITT analyses and reported the more favourable perprotocol analyses onlyMelander et al. 2003 43 42 placebo controlled trials of 5 SSRIs submittedto the Swedish drug regulatory authority formarketing approval. Published versions wereidentified by searching literature databases andinquire to the sponsoring companiesMismatch in AE reporting but notclear whether it was biased in termsof results. Anderson (2006) 523 pointedout some caveats about this study. Lowgradeand recurrent AEs were underreported27% of high-grade AEs in articles could not be matched to agentattributableAEs in the CDUS. 28% of CDUS high-grade AEs couldnot be matched to AEs in the corresponding article. In 14 of 22articles, the number of high-grade AEs in CDUS differed from thenumber in the articles by ≥ 20%. 58% of low-grade AEs in CDUSwere reported in articlesScharf and Colevas 2006 96 A comparison of adverse events (AEs) reported in22 published articles and corresponding protocolsand data from Clinical Data Update System(CDUS). CDUS monitored phase II trials that wereactive between 03/1998 and 10/2003A good discussion of relevant issues andthe imputation methodIs funnel plot appropriate to detectreporting bias?‘Assumed that the results were notselectively reported as continuousrather than binary results’ – disagree orlack of evidence?The motivating example: five of the nine eligible trials included formortality outcome. Imputed pooled estimate decreased treatmenteffect considerablyFor the four selected CSRs, within-study selection was evident orsuspected in several trials, but the impact on the conclusions of themeta-analyses was minimalA comparison of results of Cochrane systematicreviews (CSRs) and the results of sensitivityanalyses (by imputation) when within-studyoutcome selection bias was suspected. Includedone motivating example and four CSRs fromSutton et al. (2000) 524Williamsonand Gamble2005 95MRC-funded ongoing projectWilliamson et To estimate the percentage and impact of withinstudyal. 2006 99 selective reporting in an unselected cohortof 300 Cochrane systematic reviewsERC,Ethics Review Committee; LREC,Local Research Ethics Committee; SMD,standardised mean deviation.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 10Time lag bias –included empirical studies141© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 10142Study Methods Results NotesCohorts of studies collated from various sourcesTime to publication from study closure Three trials (unpublished) didPositive Non-positivenot have survival information, andassumed to be not statisticallysignificant1–2 2 11Simes 1987 101 38 trials located by a search of literature and trial registry,on advanced ovarian cancer and multiple myelomaPositive studies – those that showed significant survivaldifference3–5 4 96–10 0 5Not yet published: 0 7Analyses were based on 321completed studies from 520studies for which questionnaireswere completedResults for clinical trialswere similar to those for allquantitative studiesQuantitative studies (n = 218)Median time from approval by REC to publication in journals 4.82(3.87–5.72) years for studies with significant results (p < 0.05), 7.99(6.91 to ∝) for studies with null results (p > 0.10)Adjusted hazard ratio (vs null results p > 0.10): 2.34 (95% CI:1.47to 3.73) for significant results (p < 0.05), and 0.43 (95% CI: 0.15 to1.24) for intermediate results (p = 0.05–0.10)Retrospective cohort of 748 studies submitted to theRoyal Prince Alfred Hospital REC from 1979 to 1988Results of quantitative studies (n = 218) were classified as:significant (p < 0.05), non-significant trend (p = 0.05–0.10)and non-significant or null (p ≥ 0.10)Results of qualitative studies (n = 103) classified as: striking,important/definite, or unimportant/negativeStern and Simes1997 24Qualitative studies (n = 103)No clear evidence on publication biasThe total number of eligible trialswas 109Median time from start of enrolment to publication was 4.3years for positive trials and 6.4 years for negative trials (p < 0.001;HR = 3.7 (95% CI: 1.8 to 7.7)Median time from completion of follow-up to publication was 1.7years for positive trials and 3.0 for negative trials (p < 0.001)Positive trials were submitted for publication more rapidly aftercompletion than were negative trials (median 1.0 vs 1.6 years;p = 0.001) and were published more rapidly after submission(median 0.8 vs 1.1 years; p = 0.04)Ioannidis 1998 23 Retrospective cohort of 66 completed phase 2/3 trialsconducted between 1986 and 1996 by the AIDS clinicaltrials group and by Terry Beirn Community Programs forClinical Research on AIDSPositive results defined as those with p < 0.05 favouringthe experimental therapy; and negative results defined asthose with p > 0.05 or favouring the controlUnclear whether the methodfor results classification wasprespecified or not. If significantand mixed results werecombined, the publication ratewas lower than that for nonsignificantstudies (73% vs 86%)Median time from funding start date to publication was 3 years forsignificant studies, 6 years for mixed results, and 5 years for nonsignificantresultsHazard ratio: 1.0 for non-significant results; 0.12 (95% CI: 0.02to 0.97) for mixed; and 1.19 (95% CI: 0.95 to 3.84) for significantstudies61 completed studies identified by a survey of 89organisations funding for research on passive smoking andinvestigatorsResults classified as significant (p < 0.01), mixed (multipleprimary outcomes with at least one significant outcome),or non-significant (p > 0.05)Misakian andBero 1998 30

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Study Methods Results NotesOnly available as an abstract.Likely to be a subsample fromthe previous study by Min andDickersin (1992) 20Cox regression analysis found that time to full publication wasassociated with statistically significant results for the primaryoutcome specified (HR = 1.75, 95% CI: 1.14 to 2.93)242 observational studies that completed enrolment,initiated at Johns Hopkins University. Study results andfull publication were verified by interview with theinvestigatorsResults were classified as statistically significant or notMin andDickersin2005 104Only accepted studies wereincludedMedian time between submission and publication was 7.8 monthsfor studies with positive results vs 7.6 months for reports withnegative results (p = 0.44)Dickersin 2002 82 A cohort of 133 comparative studies submitted to andaccepted for publication in JAMAResults classified as positive when p < 0.05, or negativewhen p > 0.05, for the main outcomeContacted the author for detailsTime between completion of study and publication in a journal,univariate survival analysis hazard ratio 0.53 (95% CI: 0.25 to 1.1),p = 0.10Cronin and Sheldon 2004 27 A cohort of 70 studies sponsored by NHS R&Dprogramme from 1995 to 1998; with a survey of projectleadersResults were classified as showed effect (p < 0.05 orimportant) or notPublished phase III trials only. Anabstract without full publicationTrial results were not associated with the average time topublication (from the date of trial initiation, or closure to accrual)56 published phase III trials conducted by RadiationTherapy Oncology Group (RTOG)Results were classified according to the original trialists’preferences between experimental and standardtreatmentSoares et al.2005 103Of the 84 published studies, 71reported statistically significantresults. The early phase studieswere less likely to be publishedthan late stage studiesMedian time from the end of recruitment or follow-up asdescribed in the methods section until publication was 2.71 (range0.38 to 8.42) years. No difference in the time to publication fortrials reporting significant results vs non-significant results (2.67 vs3.00 years, p = 0.87)Hall et al. 2007 37 53 published studies with sufficient data on time topublication, from 84 published studies from 190 researchprotocols submitted to the Capital District HealthAuthority REB in Halifax, Canada, for the period 1995–6Results classified as statistically significant (p < 0.05) or notStudies identified by searchingelectronic databases andCochrane library. Subgroupanalysis available forpharmaceutical sponsoredtrials. Also identified 19 trialsin abstract form only and fourunpublished trialsMean time (beneficial vs no beneficial studies)From enrolment initiation to publication (n = 65): 4.1 vs 4.2 years,p = 0.70From enrolment completion to publication (n = 55): 2.0 vs 2.3 years,p = 0.21From enrolment completion to submission (n = 27): 1.8 vs 2.2,p = 0.19Between submission and acceptance (n = 51): 0.37 vs 0.37 years,p = 0.92From acceptance to publication (n = 61): 0.38 vs 0.40 years, p = 0.93Cases with available data from 159 acute stroke clinicaltrials fully published from 1955 to 1999Results were classified as beneficial, or non-beneficial(harmful or neutral), according to authors’ final judgementLiebeskind et al.2006 102143© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 10144Study Methods Results NotesCohorts of abstracts presented at meetingsCallaham et al. compared theirresult with that of Stern andSimes (1997) 24The mean time from the meeting to publication was no differentbetween studies with positive results and those with negativeresults (1.6 vs 1.3 years, p = 0.20)Callaham et al. 1998 105 493 research abstracts submitted for presentation atthe 1991 meeting of the Society of Academic EmergencyMedicine. Subsequent full publication verified by searchingMEDLINE or contacting authorsResults were classified as positive (beneficial orstatistically significant) or negativeNo consistent factors werefoundLog-rank tests did not show a significant difference in time topublication between results classified as positive or negative(p = 0.54). Cox regression model also did not demonstrate anyevidence of an association between time to publication and resultsand sample sizeCheng et al. 1998 49 178 abstracts of RCTs in cystic fibrosis (CF), presentedat three CF conferences over a 30-year period. SearchedCochrane trial registers for full publicationsResults were classified as positive or negative, accordingto authors’ conclusion about whether the test treatmentwas superior to the controlMultivariate Cox proportional hazards analysis: time to fullpublication for significant vs non-significant results HR = 1.92 (95%CI: 1.28 to 2.87) (p = 0.0015)Eloubeidi et al. 2001 60 461 research abstracts submitted to a GI endoscopicresearch meeting in 1994. Full publication was tracked bya literature searchResults were classified as statistically significant (p < 0.05)or notLog-rank testing showed no significant differences in publicationrate between studies with significant results and those with nonsignificantresults (χ 2 = 3.06, p = 0.08)Evers 2000 61 151 abstracts of RCTs presented at the European Societyof Human Reproduction and Embryology. Electronicdatabases and major journals were searched forsubsequent full publicationResults were classified as significant (p < 0.05) or notTime to publication frompresentation was shownseparately for positive andnegative studies (but withoutstatistical testing details)After 5 years, the rate of publication was 66% for positive resultsand 44% for non-positive resultsKlassen et al. 2002 69 447 abstracts of phase 3 RCTs presented at the Societyfor Paediatric Research Meeting (1992–5). Subsequentpublication was ascertained by a literature searchResults were classified according to authors’ conclusionwhether the result was in favour of the test treatmentFrom abstract presentation topublicationMedian time from abstract presentation to full publication 2.2 yearsfor significant results and 3.0 years for non-significant resultsHazard ratio for significant vs non-significant results 1.4 (95% CI:1.1 to 1.7)Krzyzanowska et al. 2003 70 510 abstracts from large phase 3 RCTs presented at anoncology conference. Subsequent full publication wastracked by a literature search and survey of originalinvestigatorsResults were classified as significant (p < 0.05) or not forthe primary end point

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Study Methods Results NotesTemporal trends of reported effect sizeThe temporal trends wereindependent of trial sizeEarly trials (year 1 and 2) overestimated the treatment effectcompared with a meta-analysis of the subsequent trials in 20 of the26 meta-analyses. The average difference in relative odds was 35%(95% CI: 15% to 55%)Rothwell and 26 meta-analyses (with 241 trials in total) from a1997 106 treatment effect might be related to year of publicationRobertson literature search were used to test the hypothesis thatMore cases (n = 10) showed asignificant correlation betweensample size and treatment effectFour of the 38 meta-analyses showed a significant correlation(p < 0.10) between the year of publication and the treatment effect38 meta-analyses published in BMJ or JAMA during 1992–6.Rank correlation analysis of year of publication and thetreatment effectSong andGilbody 1998 107Effect sizes in RCTs decreasesover time in three of the fourcases, caused mainly by baselinedifferences. The phenomenonwas termed ‘fading of reportedeffectiveness’ by authorsPravastatin on LDL-C: effect size significantly reduced (– 3.22% inevery 5 years, p < 0.0001)Atorvastatin on LDL-C: no significant change in effect size (+ 0.31%,p = 0.86)Timolol (– 0.56 mmHg, p < 0.0001) and latanoprost (– 1.78 mmHg,p = 0.007) decreased over timeMeta-analyses of lipid-lowering drugs pravastatin(RCTs = 64) and atorvastatin (RCTs = 35), and antiglaucomadrugs timolol (RCTs = 75) and latanoprost(RCTs = 32)Regression analysis of reported effect size against year ofpublicationGehr et al.2006 108The study also reported biasrelated to journal impactfactor and a comparison of fullpublication and abstractsTrials published earlier reported more favourable results than thetrials published laterVaitkus and Brar 2007 109 A case study: meta-analysis of N-acetylcysteine in theprevention of contrast-induced nephropathy, included27 RCTs published only as manuscripts (n = 12) or asabstracts (n = 2) or both (n = 13)There was also a negativerelationship between the samplesize and effect size. Severalpossible explanations werementionedAt the original meta-analyses level, there was a significant negativerelationship between year of publication and effect size (r = – 0.133,p < 0.01; n = 44), the association remained after controlling forsample size (r = – 0.105, p < 0.01; n = 39)Jennions and Moller 2002 525 44 published meta-analyses in ecology and evolution thatprovided sufficient data on primary studies includedExamined the relationship between estimated effect sizesand years of publicationStudies of ecology andrecommended the use ofcumulative meta-analysisCorrelation analyses revealed no significant association betweenthe magnitude of effect size and publication year in either of thetwo meta-analysesTwo meta-analyses of studies testing two plant defencetheories in ecologyCorrelation analysis and cumulative meta-analysisLeimu andKoricheva2004 111Focused on the use of abstractsin meta-analysis. Results obtainedfrom a poster on internet.Note: different conclusion fromMarinovich et al. (2005) 529 ‘datamaturity and systematic reviewsof new health technologies’Inclusion of abstracts does not alter conclusions over timefor overall survival and rate of mastectomy. For loco-regionaltreatment, inclusion of a recent abstract substantially amplifies thebeneficial effect of preoperative chemotherapyMieog and To explore the impact over time of the inclusion of dataGhersi 2005 526 from abstracts on a Cochrane review of preoperativechemotherapy for early breast cancerLDL-C, low-density lipoprotein C; REB, Research Ethics Board.145© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 11Grey literature bias –included empirical studies147© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 11148Study Methods Main findings NotesStudies of multiple (10 or more) meta-analysesOnly five of the identified grey literature itemswere still not fully published at the time of analysisReported treatment effect was (on average)statistically non-significantly greater in publishedtrials compared with grey literature trialsFergusson et al. 2000 121 10 meta-analyses of perioperative transfusion,identified from a project by the International Studyof Perio-operative TransfusionGrey literature included 15 abstracts, one letter,three conference proceedings and one unpublishedreportA few places were unclear about the removal ofabstracts from the meta-analysesIn 14 of the 41 analyses removal of the greyliterature changed the estimate of treatment effectby ≥ 10% (treatment effect was greater in ninecases but smaller in five cases)On average published trials vs grey literatureyielded larger treatment effect (ROR 1.15; 95% CI:1.04 to 1.28)McAuley et al. 2000 119 41 meta-analyses that used binary outcomes,included 365 published and 102 trials available onlyas grey literatureGrey literature included: abstracts (61%),unpublished trials (17%) or in press (3%), bookchapters (6%), theses (2%), company reports (3%)Included only meta-analyses that conductedcomprehensive literature search, used binaryoutcome and had five or more trials. Greyliterature bias may be greater in meta-analyses thatincluded fewer trials or without a comprehensiveliterature searchReported treatment effect based on grey literatureranged from 97% more to 209% less beneficialthan those from the corresponding published trials.Pooled effect estimates from the grey literaturewere on average greater than those from publishedtrials (ROR: 1.07; 95% CI: 0.98 to 1.15)Egger et al. 2003 3 60 meta-analyses of health-care interventions thatincluded 630 published and 153 unpublished trials153 unpublished studies including abstracts (45%),book sections (14%), theses (3%) and other forms(37%)The authors considered that the observed biaswas ‘less pronounced than reported by McAuley’,possibly because of clinical trials’ long history inthe cancer field11 cases were IPD meta-analyses, and 53% of greyliterature was unpublished data. Grey literaturein this study also included trials published in non-English-language journals (2% only)Estimated treatment effect using data from fullypublished trials tended to be greater than thatwith grey literature trials, although the direction ofbias was not always predictableHazard ratio of hazard ratios (HRHR) for 11 metaanalysesof fully published data was 0.93 (95% CI:0.90 to 0.97) compared with HRHR for 11 metaanalysesincluding all data of 0.96 (95% CI: 0.93 to0.99)Burdett et al. 2003 120 11 of the 13 individual patient meta-analysesof cancer, conducted at MRC Clinical Trial Unit(London), that included both published trials(n = 75) and grey literature trials (n = 45)Grey literature trials included unpublished data(53%), abstracts (38%), book chapters (7%) andnon-English-language publications (2%)Data obtained from Hopewell (2007) 113 (CochraneMethodology Review)Reported treatment effect was (on average)statistically non-significantly smaller in greyliterature trials than that in published trials. ROR =1.05 (95% CI: 0.83 to 1.33)Hopewell 2004 122 17 meta-analyses in cancer, included 264 RCTs,identified from CDSRGrey literature included conference abstracts,letters, books, government and pharmaceuticalreports, theses, file drawer data and personalcorrespondenceAlso included in the review of cohort studies.Negative findings (according to FDA’s decision)were less likely to be published, and if published,the negative result was often conveyed as apositive outcomeThe effect size based on the journal articles (0.41;95% CI: 0.36 to 0.45) was on average 32% (rangedfrom 11% to 69%) greater than the effect sizebased on the FDA reviews (0.31; 95% CI: 0.27 to0.35) (sign test p < 0.001)Turner et al. 2008 45 74 clinical trials of 12 antidepressants submittedand approved by FDA between 1984 and 2004.Of the 74 trials from FDA reviews, 23 wereunpublishedUnpublished trials from FDA databases

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Study Methods Main findings NotesCase studiesThe review included five published trials.Combined evidence from published andunpublished data suggested that paroxetine,sertraline, venlafaxine and citalopram are notefficacious and pose a possible increased risk ofsuicidal ideation, serious adverse events or bothTwo published trials and unpublished data suggestthat fluoxetine has a favourable risk-benefit profile.Published data from one trial of paroxetine andtwo trials of sertraline suggest equivocal or weakpositive risk-benefit profiles; but in both cases,addition of unpublished data indicates that risksoutweigh benefits. Data from unpublished trials ofcitalopram and venlafaxine show unfavourable riskbenefitprofilesWhittington et al. 2004 136 A review of five SSRIs (fluoxetine, paroxetine,sertraline, citalopram, venlafaxine) in childhooddepression, included five published and sixunpublished trials (plus some unpublished data forpublished studies)Unpublished studies obtained from a review by theCommittee on Safety of Medicines (UK)The review concluded that ‘unpublished studiesdid not substantially alter the risk-to-benefitdetermination’, which is different from Whitingtonet al.’s 136 conclusionsThe single unpublished trial (with a negative result)did not substantially influence interpretationof the overall efficacy rates. Similarly, omissionof unpublished results did not influenceinterpretation of safety outcomesWallace et al. 2006 137 A meta-analysis of SSRIs in paediatric depression,included six published trials and one unpublishedtrial for efficacy meta-analysis; and seven publishedand four unpublished trials for safety analysisUnpublished studies obtained from the website ofthe Committee on Safety of Medicines (UK)Available only in the abstract formIn the two meta-analyses, published studiesyielded larger average estimates of effect thandissertationsDevine 1999 129 Two meta-analyses of psychoeducational care.One meta-analysis of surgical patients included 80published and 102 unpublished studies. Anothermeta-analysis of cancer patients included 43published and 35 unpublished studiesGrey literature: theses or dissertationsTreatment effect was greater by pooling data frompublished trials (RR 1.29; 95% CI: 1.03 to 1.60)than that from unpublished trials (RR 1.01; 95% CI:0.74 to 1.28)Jeng et al. 1995 131 An IPD meta-analysis of paternal cell immunisationfor recurrent miscarriage, included four publishedand four unpublished RCTsUnpublished trials identified from submissionsto the American Society for ReproductiveImmunologyNote the direction of biasResults of studies presented as abstracts reportedgreater treatment effect than those of publishedtrials. Pooled mean difference in fatality: 0.046(p = 0.21) using data from published trials and0.079 (p = 0.03) from abstractsDetsky et al. 1987 128 A meta-analysis of perioperative parenteralnutrition for reducing complications and fatalitiesfrom major surgeryGrey literature: abstractsPooled reduction in number of cramps per 4-weekperiod: 8.83 (95% CI: 4.16 to 13.49) from poolingpublished trials and 3.60 (95% CI: 2.15 to 5.05)from all trialsMan-Son-Hing et al. 1998 133 A meta-analysis of quinine for nocturnal legcramps, included four published and fourunpublished RCTsGrey literature: unpublished data from FDA,German or British drug regulatory bodies, anddrug companies149© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 11150Study Methods Main findings NotesWhether a study was published or unpublished didnot significantly alter the relative risk of non-unionbetween the two interventionsBhandari et al. 2000 527 A meta-analysis of reamed vs non-reamedintramedullary nailing of lower extremity longbone fractures, included four published and fiveunpublished trialsPublished trials showed no difference betweenthe treatment and placebo (RR 1.02; 95% CI: 0.96to 1.08), whereas unpublished trials showed asignificant unfavourable effect of treatment (RR1.14; 95% CI: 1.00 to 1.30)Horn and Limburg 2001 528 A meta-analysis of calcium antagonists forischaemic stroke, used 18 published and fourunpublished trialsUnpublished studies identified by contactinginvestigators and companiesAn example of possible confounding factors forthe association between publication status and thetreatment effectThis study was presented by MacLean et al. 130 in1999, but with seemingly different conclusionsPooled RR of dyspepsia: 1.07 (95% CI: 0.70 to 1.63)using FDA data and 1.21 (95% CI: 0.81 to 1.81)using published trials. Meta-regression analysesfound that estimates varied significantly by NSAIDdose (p = 0.037) but were not related to whetherthe study was published or not (p = 0.73)MacLean et al. 2003 135 A review of 15 published trials and 11 unpublisheddata from FDA, that reported relative risk ofdyspepsia from NSAIDsUnpublished FDA dataAnimal experimental researchAbstracts reported a significantly lower estimateof effect size than fully published studies (p < 0.001)Macleod et al. 2004 132 A meta-analysis of animal experimental studies ofnicotinamide for stroke, included 11 fully publishedstudies and three in abstract onlyGrey literature: abstract onlyAvailable in the abstract form onlyEarliest abstracts for the three RCTs were foundto have presented incomplete patient numbers(3/3), mis-stated primary and/or secondary endpoints (2/3), mis-stated baseline data (1/3) andmissed presenting follow-up results (3/3)Marinovich et al. 2004 529 A case study of four RCTs (three fully published)and 46 corresponding abstracts on sirolimuselutingstentsGrey literature: abstract onlyThe comparison of the methodological qualitycomposite score favoured dissertations overpublished studies. Dissertations included smallersample sizesIn weighted least squares analysis the pooled effectsize for published studies (mean 0.50, SD = 1.64)was twice that for dissertations (mean 0.23, SD =1.26), and the difference was statistically significant(p < 0.01)A case study of randomised trials of psychotherapyin children and adolescents included 134 publishedtrials and 121 dissertationsGrey literature: dissertationMcLeod and Weisz2004 134

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Study Methods Main findings NotesIt was unclear about why the three meta-analyseswere selectedThe authors of this article suggested that cautionwas required when including grey literature inmeta-analysisThe pooled effect size using all trials (includingunpublished data) was similar to that obtainedwhen using only published trials in all three casesMartin et al. 2005 530 Three meta-analyses of antipsychotics forschizophrenia: olanzapine vs typical antipsychotics(one published and four unpublished trials);quetiapine vs classical antipsychotics (threepublished and three unpublished trials); andrisperidone vs typical antipsychotics (15 publishedand eight unpublished trials)Type of grey literature unclearAvailable as an abstract; considered nonrandomisedstudiesNo mention of findings for levetiracetam andgabapentinMedian reported responder and seizure freedomrates were higher in topiramate studies publishedin full vs those from abstracts (62% vs 55% and16% vs 8%, respectively)Maguire et al. 2006 531 A review of non-randomised studies in refractoryepilepsy: 42 published vs 54 abstracts for add-ontopiramate; 36 published vs 44 abstracts for addonlevetiracetam; 26 published vs 21 abstracts foradd-on gabapentin therapyGrey literature: abstract onlySignificant publication bias identified among trialsstudying the ability of N-acetylcysteine to preventcontrast-induced nephropathyPublished manuscripts consistently showed amore beneficial treatment-effect estimate thanunpublished abstracts over time. 17 publishedmeta-analyses in this area included on average 76%of contemporaneously available manuscripts and13% of available abstractsVaitkus and Brar 2007 109 A meta-analysis of clinical trials of N-acetylcysteinein the prevention of contrast-induced nephropathy;included 12 fully published manuscripts only, 13 asabstracts followed by full publication, and two asabstracts onlyGrey literature: abstractBasic researchUnpublished and published studies did not differsignificantly in effect size adjusted for sample size,using random-effects meta-analysisMoller et al. 2005 532 A review of basic animal research on therelationship between asymmetry and sexualselection, included 105 published and 20unpublished data setsUnpublished data were obtained by a survey ofinvestigatorsCDSR, Cochrane Database of Systematic Reviews.151© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 12Language bias –included empirical studies153© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 12154Study Methods Main findings NotesRCTs within the same pair were likely toconsider different research questions as thematching was only by author and time ofpublicationOnly 35% of German-language articles, compared with 65%of English language articles, reported significant differences(p < 0.05) in the main end point (McNemar’s test p = 0.002)Egger et al. 1997 148 40 pairs of RCTs (in the field of internal medicine),matched for first author and time of publication,with one report published in German and the otherin EnglishKey authors defined as the first, second orlast authorSimilar methods and findings to Egger et al.1997 14833.3% of German articles and 57.1% of English-languagearticles reported significant findings (Wilcoxon’s test p = 0.14)Heres et al. 21 pairs of RCTs on neuroscience, matched for the2004 150 key authors and time of publication, with one inGerman and the other in EnglishStudies of multiple meta-analyses in which results of English trials were compared with results of non-English-language trialsHowever, the difference between theoriginal meta-analyses and updated metaanalyseswas non-significant, even whenconsidering the meta-analysis of selectivegut decontaminationThe statistical significance was changed in one meta-analysisof selective gut decontamination from an OR of 0.70 (95% CI:0.45 to 1.09) in the original analysis to an OR of 0.67 (95% CI:0.47 to 0.95) after including studies published in German andSwissGregoire et al. 1995 147 28 meta-analyses with language restrictions,identified from eight medical journals. For sevenof the meta-analyses, 11 studies published in theexcluded languages were identified by repeatingthe original search strategies without languagerestrictionsThe analysis is based on a small numberof meta-analyses. Limitations like samplingframe, clinical topics and interventionscould have affected the results. More caseswere included in Moher et al. 2003 4There was no significant difference in the estimated treatmenteffect for language-restricted analyses compared with languageinclusive analyses (ROR = 0.96; 95% CI: 0.78 to 1.18)Language inclusive meta-analyses had narrower confidenceintervals (average width = 0.79; 95% CI: 0.51 to 1.07)compared with language-restricted meta-analyses (averagewidth = 0.92; 95% CI: 0.53 to 1.32)Moher et al. 2000 151 19 meta-analyses that included 206 English and 33non-English-language trialsNon-English languages: one in Dutch, one in Danish,12 in French, 12 in German, four in Italian, two inSpanish and one in ChineseThere were only minor differences in thequality of reports of RCTs published inEnglish vs in other languagesLanguage-restricted meta-analyses vs language inclusive metaanalyses(random-effects): ROR = 1.11 (95% CI: 0.92 to 1.34)for all cases; ROR = 1.02 (0.83 to 1.26) for 34 meta-analysesof conventional interventions; and ROR = 1.63 (1.03 to 2.60)for eight meta-analyses of complementary and alternativemedicine42 meta-analyses that included 529 English and 133non-English-language trialsNon-English languages: 57 in German, 52 in French,15 in Italian, six in Spanish, three in Danish, two inDutch, one each in Japanese and PortugueseMoher et al.2003 4,152Compared with English-language trials,non-English-language trials included fewerparticipants but were more likely to showstatistically significant results, and tended tobe of lower methodological qualityTreatment effect estimates were on average 16% morebeneficial in non-English-language trials (ROR = 0.84; 95%CI: 0.74 to 0.97). The change in estimated treatment effectafter including or excluding non-English-language trials wasless than 5% in 29 (58%) meta-analyses. In the remaining 21meta-analyses, five (10%) showed more benefit and 16 (32%)showed less benefit after exclusion of non-English-anguagetrialsNone of the meta-analyses changed statistical significance atthe 5% levelEgger et al. 2003 3,153,533 50 meta-analyses (each had five or more trials andcombined binary outcomes) that included 485English and 115 non-English-language trialsNon-English languages: 36.5% in German, 25.2%in French, 10.4% in Italian, 7.0% in Japanese, 6.1%in Spanish, 5.2% in Portuguese, 7.0% in four otherEuropean languages, and 2.6% in Chinese

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 13Citation bias –included empirical studies155© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 13156Study Methods Main findings NotesPrevalence of smoking in a special population withschizophreniaA 10% increase in reported prevalence of smokingwas associated with a 61% (95% CI: 30% to 98%)increase in citation rate. After adjusting for journalimpact factor, a 10% increase in prevalence ofsmoking was associated with a 28% (1% to 62%)increase in citation rateChapman et al. An examination of citations of 42 studies on tobacco2009 162 smoking among schizophrenia subjectsExamined studies from one meeting onlyMean citations per year = 2.04 (95% CI: 1.6 to 2.4)in 440 different journals. Factors associated withcitation: JIF, newsworthiness score, quality scorePositive outcome bias not evidentCallaham et al. 2002 161 Citations of 204 published articles originallysubmitted to a 1991 emergency medicine meetingwere identified by searching Science Citation IndexCalculated number of times an article was cited peryear and mean impact factorSeveral confounders like accessibility and popularityof a journal, which might be associated with citationrates, were not includedPositive association between a statistically significantresult and the citation frequency (β = 1.21; 95%CI: 1.10 to 1.33). Disease area and allocationconcealment were also significantly predictors of thecitation frequencyKjaergard and Gluud 2002 163 530 RCTs of hepato-biliary diseases. Trial resultsand quality were assessed. Number of citations foreach trial was obtained from Science Citation Index.Linear regression analysis conducted‘Self-citations’ were excludedMedian number of citations: 33 for significant results(n = 287) and 16 for non-significant results (n = 81)Citation rate ratio for papers reporting ‘p < 0.05’ onprimary outcome was 1.63 (95% CI: 1.32 to 2.02)Nieminen et al. 2007 164 368 research papers published in 1996 in fourpsychiatric journals. Regression analysis to relatecitation frequency to statistical significance (p < 0.05),adjusting for confoundersThe study identified severe bias in selection ofreferences in narrative reviewsOf the 38 positive reviews and in terms of selectionof references, 10 reviews were neutral, 27 reviewshad a positive selection and one a negative selection.The four reviews that did not recommend physicalinterventions all had a negative selection ofreferencesSchmidt et al. 2005 165 A review of 42 reviews that included trials on clinicaleffects of physical interventions on house-dustmite antigens. Compared the proportion of trialreferences with significant results between citedreferences and all trials available

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 14Reasons given by investigators forstudies not being publishedStudyCooper 1997 35Cohort of studiessubmitted for reviewby a human subjectscommitteeMixed designDecullier 2005 28Cohort of researchprotocolsMixed designsDickersin 1992 20Cohort of researchprotocolsMixed designDickersin 1993 21Cohort of researchprotocolsClinical trialsReasons for non-publicationWhy the study was not prepared for a journal publication (n = 159)Publication not an aim: 48%Class project only: 30%Assistant lost interest: 26%No significant results: 22%Results were not interesting: 20%Design or operational problems: 12%Researchers did not recall: 6%Others lost interest: 2%Reasons given by investigators for not publishing (n = 102)Negative results: 27 (26%)Writing or submission in progress: 23 (23%)Published in other forms: 23 (23%)Paper rejected: 5 (5%)Other reasons: 17 (17%)Not available: 7 (7%)Main reasons Total School ofMedicineTotal unpublished studies 124 (100%) 65 59Manuscript rejected by journal 6 (5%) 2 4Total not submitted 118 (95%) 63 55Results not interesting 37 (30%) 26 11Design or operational problems 40 (32%) 17 23Publication not an aim 16 (13%) 8 8Other reasons 25 (20%) 12 13Total unpublished trials: 100% (n = 14)Not interesting or no time: 42.8%Co-investigator/operational problems: 37.5%Data analysis not completed: 14.3%Rejected by journal: 0%No reason given: 7.1%School ofPublic Health157© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 14StudyEasterbrook1991 22Cohort of researchprotocolsMixed designCamacho 2005 232Survey of authors ofabstracts presentedat annual meeting ofthe American Societyof Clinical Oncologyin 1997Clinical trialsDe Bellefeuille1992 50Cohort of meetingabstractsMixed designReasons for non-publicationTotal Significant Non-significant Null(n = 78) (n = 23) (n = 12) (n = 43)Submitted or published 35 (45%) 20 4 11elsewhereNot submitted/published at all 43 (55%) 3 8 32Null results 26 (33%) 26Methodology or logistic 21 (27%) 3 5 13problemSponsor has control of data 19 (24%) 11 2 6Analysis incomplete 19 (24%) 10 2 7Manuscript rejected 16 (21%) 7 1 8Publication not aim of study 13 (17%) 6 4 3Too busy or lost interest 11 (14%) 3 5 3Unimportant results 10 (13%) 2 1 7Co-investigator left 5 (6%) 0 1 4Factors affecting the publication of phase I clinical trialsReason Novel agent Non-novel Total(n = 36) (n = 29) (n = 65)Lack of time 12 11 23 (35%)Manuscript in preparation 10 5 15 (23%)Relocation of authors 11 3 14 (22%)Incomplete study 6 7 13 (20%)Results considered not interesting 7 4 11 (17%)Rejection from peer-reviewed journal 3 2 5 (8%)Manuscript submitted 1 5 6 (9%)Not in the sponsor’s interest 2 1 3 (5%)Conflict of interest 1 0 1 (2%)Other 1 1 2 (3%)Novel – agents not approved by the Food and Drug Administration at the time of submissionNon-novel – at least one agent approvedReasons for non-publication (based on n = 41 respondents)Lack of time/other resources: 13 (32%)Insufficient priority: 9 (22%)Incomplete study with intent to publish eventually: 5 (12%)Article not accepted for publication: 4 (10%)Modification of data after submission of abstract: 1 (2%)Other: 12 (29%)158

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8StudyHartling 2004 233Survey of authors ofabstracts presentedat the Society forPaediatric Researchmeetings from 1992to 1995Clinical trialsHashkes 2003 67Cohort of meetingabstractsMixed designHopewell 2001 234Cohort of abstractsat meetings onsystematic reviewsMethodologicalresearchKrzyzanowska2003 70Survey of authors ofabstracts presentedat the annual meetingof the AmericanSociety of ClinicalOncology 1989–98Clinical trialsReasons for non-publicationTotal number of unpublished studies n = 47Total number of unsubmitted studies n = 39Important reasons given by authors for non-publication:Not enough time (n = 39): 56.4%Too much trouble with co-authors (n = 38): 28.9%Thought that journal was unlikely to accept (n = 38): 26.3%Results were not statistically significant (n = 38): 23.7%Results were not important enough (n = 38): 18.4%Others published with similar findings (n = 38): 15.8%Study quality poor (n = 37): 13.5%Not worth the trouble (n = 37): 10.8%Results did not support the hypothesis (n = 38): 5.3%Reasons for non-submission of abstract for publication (n = 97)Case report: 8 (8%)Previously reported: 5 (5%)Non-positive results: 2 (2%)Methodological problems: 2 (2%)Desire to expand paper: 42 (43%)Low priority or lack of time: 47 (48%)Fear of rejection: 13 (13%)Author moved or passed away: 4 (4%)No decision on journal: 1 (1%)Reasons for non-publication of abstracts (n = 22)Low priority or too busy: 9 (24%)Not deemed appropriate: 7 (19%)Findings became rapidly outdated: 2 (5%)Rejected by journal as not deemed relevant to the general readership: 1Subject area was too specific with limited interest to a wider audience: 1Internal Cochrane issue: 1Concerns over unity of approach: 1Note: Authors of 15 non-published abstracts did not given a reason. These 15 unpublished abstractswere not includedReasons for lack of publication (based on 40 responses)Lack of time, funds, or other resources: 14 (35%)Study incomplete, with eventual intent to publish: 6 (15%)Article submitted, but not accepted for publication: 5 (13%)Manuscript in preparation: 5 (13%)Manuscript under review: 4 (10%)Insufficient priority to warrant publication: 4 (10%)Other: 5 (13%)Not provided: 6 (15%)159© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 14StudySanossian 2006 72Survey of authors ofresearch abstractspresented at theannual InternationalStroke Conferencein 2000Mixed designReasons for non-publicationReasons for non-publication (n = 74)No time: 28 (38%)Low priority: 11 (15%)Co-author responsibility or lack of participation: 10 (14%)Study ongoing: 8 (11%)Methodological limitations: 6 (8%)Different version published: 3 (4%)Other similar articles published: 2 (3%)Does not recall: 1 (1%)No reason given: 5 (7%)Scherer et al.1994 54 Number of unpublished abstracts of RCTs (n = 32)Incomplete studies: 16%Manuscript rejected: 19%No time to prepare: 28%Problem of study design: 9%Sprague 2003 236Cohort of meetingabstracts and surveyof authorsMixed designVuckovic-Dekic2001 237Survey of Serbianauthors of abstractspresented atCongress of theBalkan Union ofOncology 1996–8Mixed designReasons for failure to submit a manuscript to a journalReason (n = 71) aNo. of responsesNo time to prepare for publication 33 (46%)Study is still ongoing 22 (31%)Responsibility for manuscript belongs to a co-author 14 (20%)Difficulty with co-authors (lack of participation) 12 (17%)Pursuit of publication given a low priority 9 (13%)Low likelihood of acceptance for publication because 9 (13%)of methodological limitations of study (e.g. weak studydesign or small sample size)Other papers with similar findings already published 3 (4%)Plan to submit paper for publication 3 (3 (4%)Results not important enough 1 (1%)Statistical analysis was not positive 1 (1%)Low likelihood of acceptance by journal because of 1 (1%)insufficient interest to readersDifferent version of data published 1 (1%)Reasons for not submitting studies (n = 21)Not enough time: 10 (48%)Thought journals unlikely to accept: 2 (10%)Results not important enough: 1 (5%)Other papers with similar findings: 1 (5%)Too much trouble with co-authors: 1 (5%)Other reasons: 6 (29%)160

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8StudyWeber 1998 230Cohort of meetingabstractsMixed designsBlumenthal et al.1997 231Survey of life sciencesfaculty members at50 universities in theUSADickersin 1987 228Survey of authors ofpublished trialsClinical trialsMachan et al.2006 234 (conferenceabstract)An email surveyof membersof EuropeanFederation ofMedical Informaticsand InternationalMedical InformaticsAssociationEvaluation studiesMisakian 1998 30Reasons for non-publicationReasons for failure to submit to a journal (n = 179)Not enough time: 74 (41%)Thought journals unlikely to accept: 35 (20%)Results not important enough: 21 (12%)Trouble with co-authors: 16 (9%)Not worth the trouble: 13 (7%)Other papers with similar findings: 11 (6%)Statistical analysis not positive: 7 (4%)Other reasons: 40 (22%)Reasons given for delay to publication (n = 412)Patent application submission: 46%Protection of scientific lead: 31%Patent negotiation: 26%Resolution of intellectual property ownership: 17%Slow dissemination of undesired results: 28%Total unsubmitted trials: 100% (n = 102)Analysis in progress: 14.7%Results negative: 34.3%Lack of interest: 15.7%Sample size or poor methodology: 4.9%Controversy: 2.9%Other or unknown: 27.5%Unpublished evaluation studies (n = 104)Generalisability limited: 26%Study not yet finished: 18%No time for writing: 11%Results seemed not of interest to others: 10%Methods inadequate/sampling insufficient: 9%Organisations prohibited publication: 9%Rejected by journal: 6%Results too negative: 5%No interest in academic output: 5%Evaluation of first prototype only: 4%Reasons for unpublished results (n = 59) of passive smokingOngoing data collection or analysis: 56%Lack of time: 44%Competing priorities: 19%Statistically non-significant results: 3%Manuscript rejected: 7%Rotton et al.1995 229 Proportion of reasons given by 468 authors for not publishingFailure to replicate: 5%Manuscript rejected: 33%Non-hypothesised results: 5%Inexplicable results: 22%Non-significance: 60%a The number of responses is greater than the number of respondents because respondents were allowed to choosemore than one responseThere may be two or more reasons for each unpublished study.161© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 15Study findings and the acceptanceof submitted manuscriptsStudy Methods Main findingsOlson et al.2002 81 Cohort study of 745 manuscripts of controlledtrials submitted to JAMA from 02/1996 to 08/1999Outcome classification:1. Statistically significant (p < 0.05) for the primaryoutcome2. Statistically non-significant3. UnclearProportion of studies with different results:51.4% (n = 383) with significant results45.7% (n = 341) with non-significant results2.8% (n = 21) with unclear resultsAcceptance rate:20.4% (78/383) for significant results15.0% (51/341) for non-significant results19.0% (4/21) for unclear resultsLogistic regression analysis: significant vs nonsignificantresults OR = 1.30 (95% CI: 0.87 to 1.96)Lee et al. 2006 78Cohort study of 1107 manuscripts of originalresearch (including qualitative research, excludingsingle case reports) submitted to BMJ, Lancet andAnnals of Internal Medicine between 01/2003 and03/2003 and between 11/2003 and 02/2004Outcome classification:1. Statistically significant (p < 0.05) for the primaryoutcome2. Non-significantProportion of different statistical results:86.8% (n = 718) with significant results13.2% (n = 109) with non-significant resultsAcceptance rate:4.9% (35/718) for significant results6.4% (7/109) for non-significant resultsMultivariate analysis: OR = 0.83 (95% CI: 0.34 to1.96)Lynch et al.2007 79 Cohort study of 209 manuscripts of originalresearch on hip or knee arthroplasty submittedto the Journal of Bone and Joint Surgery (AmericanVolume) between 01/2004 and 06/2005Outcome classification:1. Positive or favourable or significant2. Negative or non-supportive or no difference3. Not analysable – unclearProportion of studies with different results:70.8% (n = 148) with positive results23.4% (n = 49) with negative results5.7% (n = 12) with unclear resultsAcceptance rate:30.4% (45/148) for positive results36.7% (18/49) for negative results8.3% (1/12) for unclear resultsDifference in publication rate between positive andnegative outcomes was not statistically significant(p = 0.41)Okike et al.2008 80Cohort study of 855 manuscripts as scientificarticles submitted to the Journal of Bone and JointSurgery (American Version) between 01/2004 and06/2005Outcome classification:1. Positive – favoured experimental item2. Negative – favoured existing standard of careover the experimental item3. Neutral – no difference4. Not applicableProportion of studies with different results:72.5% (n = 620) with positive results12.3% (n = 105) with negative results15.2% (n = 130) with neutral resultsAcceptance rate:21.3% (132/620) for positive results21.0% (22/105) for negative findings24.6% (32/130) for neutral resultsMultivariate analysis: positive vs non-positiveOR = 0.92 (95% CI: 0.62 to 1.35)163© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 16Case studies indicating pharmaceuticalcompanies or industry research sponsorshipas a source of publication and related biasesCasesNathan and Weatherall 1999 305 and 2002 306Publication suppression:Deferiprone for the prevention of irontoxicity in patients with thalassaemiaRennie 1997 276,309Publication suppression:Bioequivalence of brand name and genericforms of thyroxine sodiumSkolnick 1998 308Publication suppression:HTA report on cholesterol-lowering statindrugsMillstone et al. 1994 304Publication suppression:Increased somatic cells in cow’s milk andbovine somatotrophin (BST)Shuchman 1999 307Publication suppression:Ontario Ministry of Health: omeprazole anddraft prescribing guidelinesMcCarthy 2000 310–316Publication suppression:Remune (HIV-1 immunogen) for HIVinfectionLauritsen 1987 317Non-publication:Prostaglandin for gastric ulcerSymmonds et al. 2004 318 and Panahloo 2004 336Non-publication:Neuraminidase inhibitors (oseltamivir) forasthmatic children suffering from influenzaWilmshurst 1986 321 and 1987 322Non-publication:AmrinoneBrief descriptionsA company-sponsored trial in 1989 found that the drug might be harmful. Thecompany took legal action against the investigator, Dr Nancy Olivieri, in orderto stop the disclosure of the negative findingA company-sponsored study in 1987 by Dong et al. showed bioequivalenceof generic and brand name levothyroxine. The publication of the trial wassuppressed for 7 years by the pharmaceutical company due to deleteriouseffect of results on price of company’s productA pharmaceutical company tried unsuccessfully to suppress the publication offindings from a health technology assessment on cholesterol-lowering statindrugs by the Canadian Coordinating Office of Health Technology Assessmentin 1997Millstone et al. reported that their meta-analysis with unsupportive resultsof BST was blocked by a pharmaceutical company using legal rights over rawdataShuchman reported that a company threatened legal action over draftprescribing guidelines that concluded that all proton pump inhibitors hadequivalent effect on peptic ulcers and gastro-oesophageal reflux disease.However, the company responded by saying that the company ‘is not pursuingany legal action against any physician’ 534A company-sponsored trial found no difference in efficacy between thevaccine and placebo. The manufacturer of Remune attempted to block thepaper’s publication because authors refused to include a post-hoc subgroupanalysisA company-sponsored trial compared prostaglandin analogue withranitidine for gastric ulcer, and stopped in 1985. Ranitidine was better thanprostaglandin in all centres except one. One of the trial centres in Denmarkhad asked for a copy of the report in March 1986 but had still not receivedthe full report by April 1987Two trials showed no significant difference in time to freedom from illnessbetween children taking the drug and placebo. Data were submitted toEuropean Marketing Authorisation, but not publishedA company discontinued trials that showed negative results and failed toreport adverse events of amrinone165© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 16Casesvan Heteren 2001 319Non-publication:Deep venous thrombosis after using thirdgeneration oral contraceptive pillsvan Veldhuisen and Poole-Wilson 2001 320Non-publication:‘Negative’ drug trials in patients with chronicheart failureGottlieb 2001 324,326,327,535Selective publication:Celecoxib for arthritisTopol 2004 214,215,217,323Selective publication:Rofecoxib for arthritisSteinman 2006 331Selective publication:Internal industry documents and thepromotion of gabapentin for off-label usesGarland 2004 220Selective publication:Paroxetine and venlafaxine for depression inchildren and adolescentsReines 2004 330,337Selective publication:Rofecoxib for Alzheimer’s disease orcognitive impairmentWhittington et al. 2004 136Selective publication:Selective serotonin reuptake inhibitors(SSRIs) in children and adolescentsApplegate et al. 1997 328Selective publication:IsradipineBrief descriptionsResults of a study on the risk of deep venous thrombosis after using thirdgeneration contraceptive pills were submitted to the European MedicinesEvaluation Agency in 1999, but remained unpublished. The company statedthat ‘the study was not submitted for publication because it was felt that thestudy did not offer any new scientific information’van Veldhuisen and Poole-Wilson (2001) discussed three unpublished trialsthat were terminated prematurely because of increased mortality or adverseeffects. These trials were presented at conferences but not fully publishedA trial published in JAMA in 2000 concluded that celecoxib was associatedwith a lower incidence of symptomatic ulcers and ulcer complicationscompared with ibuprofen and diclofenac. However, the publication was basedon the 6-month data. Unpublished 12-month data (submitted to FDA) wasmuch less favourable for celecoxibA trial of rofecoxib vs naproxen in patients with rheumatoid arthritis did notinclude three cases of myocardial infarctions (MIs) in the rofecoxib arm. Theauthors of the trial explained that the three MIs were observed after the cutoffdate for reporting cardiovascular events 216,536Steinman et al. reviewed internal industry documents about the promotionof gabapentin for off-label uses. They found that the company’s ‘managementexpressed concern that negative results could harm promotional efforts,andseveral documents indicate the intention to publish and publicis results only ifthey reflected favourably on gabapentin’Garland reported that none of the large negative trials of paroxetine andvenlafaxine in children and adolescents were published. The GlaxoSmithKlineinternal document revealed that the company experts advised staff towithhold data about SSRI use in children. 222 GSK faces US lawsuit overconcealment of trial results in 2004 223The two published trials of rofecoxib for Alzheimer’s disease only mentionedon-treatment mortality in the text without any statistical analyses, andconcluded that rofecoxib is well tolerated. However, the company’sunpublished intention-to-treat analyses and the independent analyses basedon data provided by the sponsor in the New Jersey Vioxx litigation found astatistically significant increase in total mortality (HR 2.99; 95% CI: 1.55 to5.56; and HR 2.13; 95% CI: 1.55 to 5.77 respectively) 219Whittington et al. compared results of published trials and unpublished data.They concluded that published data presented a favourable risk-benefitprofile, whereas unpublished data indicated that risks could outweigh benefitsof these drugs (except fluoxetine) in children and adolescentsSeveral investigators of a multicentre trial of isradipine dropped out when thepaper was in preparation, because ‘the sponsor of the study was attemptingto wield undue influence on the nature of the final paper’166

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8CasesMetcalfe et al. 2008 335Selective (or delayed) publication:Trastuzumab (Herceptin ® ) for early breastcancerLenzer 2002 329Delayed publication:Alteplase (a thrombolytic agent) for acuteischaemic strokeLenzer 2002 329Delayed publication:Release of results from a trial on ezetimibe –a cholesterol-lowering drugAlasbali et al. 2009 332Discrepancy between results and abstractconclusionsTopical prostaglandinsBrief descriptionsAn industry-sponsored three-arm trial (NCCTG-N9831) directly comparedsequential, concurrent Herceptin, and usual care control. According to aconference abstract in 2005, interim results indicated concurrent Herceptinwas more effective than sequential therapy (HR 0.64; 95% CI: 0.46 to 0.91).However, a journal paper in 2005 reported only data on concurrent therapyand the control, without including sequential-group data. Because of thesemissing data, ‘sequential trastuzumab seems more effective than it probablyis’. 335 The principal investigator for the trial responded that the publication ofconcurrent therapy data was according to an analysis plan prespecified whiledata on sequential therapy were not sufficiently mature 537A trial found that alteplase did not improve stroke recovery and increasedmortality. The negative result was not published for 6 years after the trial’scompletionNegative results of a trial on ezetimibe were released by the company onlyafter a US Congressional inquiry was set up to look into why the results hadnot been published 2 years after the study was completedAlasbali et al. examined the discrepancy between the statistical significanceof the publication’s main outcome measure and its abstract conclusions. Thepublished abstract conclusion was not consistent with the results of the mainoutcome measure in 18 of 29 industry-funded studies compared with zeroof 10 non-industry-funded studies on the efficacy of topical prostaglandinanalogues167© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 17List of 347 reviews assessedTreatment reviews (n = 100)Abdulla J, Kober L, Christensen E, Torp-Pedersen C.Effect of beta-blocker therapy on functional status inpatients with heart failure – a meta-analysis. Eur J HeartFail 2006;8:522–31.Agosti R, Duke RK, Chrubasik JE, Chrubasik S.Effectiveness of Petasites hybridus preparations inthe prophylaxis of migraine: a systematic review.Phytomedicine 2006;13:743–6.Auperin A, Le Pechoux C, Pignon JP, Koning C, JeremicB, Clamon G, et al. Concomitant radio-chemotherapybased on platin compounds in patients with locallyadvanced non-small cell lung cancer (NSCLC): a metaanalysisof individual data from 1764 patients. Ann Oncol2006;17:473–83.Ayalon L, Gum AM, Feliciano L, Arean PA. Effectivenessof nonpharmacological interventions for themanagement of neuropsychiatric symptoms in patientswith dementia: a systematic review. Arch Intern Med2006;166:2182–8.Bainbridge D, Cheng DC, Martin JE, Novick R. NSAIDanalgesia,pain control and morbidity in cardiothoracicsurgery. Can J Anaesth 2006;53:46–59.Beelmann A, Losel F. Child social skills training indevelopmental crime prevention: effects on antisocialbehavior and social competence. Psicothema 2006;18:603–10.Ben Amar M. Cannabinoids in medicine: A review oftheir therapeutic potential. J Ethnopharmacol 2006;105:1–25.Bergman R, Parkes M. Systematic review: the use ofmesalazine in inflammatory bowel disease. AlimentPharmacol Ther 2006;23:841–55.Bouza C, Lopez T, Magro A, Navalpotro L, AmateJM. Efficacy and safety of balloon kyphoplasty inthe treatment of vertebral compression fractures: asystematic review. Eur Spine J 2006;15:1050–67.Bria E, Ciccarese M, Giannarelli D, Cuppone F, NisticoC, Nuzzo C, et al. Early switch with aromatase inhibitorsas adjuvant hormonal therapy for postmenopausal breastcancer: pooled-analysis of 8794 patients. Cancer Treat Rev2006;32:325–32.Brok J, Gluud LL, Gluud C. Ribavirin monotherapyfor chronic hepatitis C infection: a Cochrane Hepato-© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Biliary Group systematic review and meta-analysis ofrandomized trials. Am J Gastroenterol 2006;101:842–7.Bujko K, Kepka L, Michalski W, Nowacki MP. Does rectalcancer shrinkage induced by preoperative radio(chemo)therapy increase the likelihood of anterior resection? Asystematic review of randomised trials. Radiother Oncol2006;80:4–12.Busquets JM, Hwang PH. Endoscopic resection ofsinonasal inverted papilloma: a meta-analysis. OtolaryngolHead Neck Surg 2006;134:476–82.Chavez-Tapia NC, Barrientos-Gutierrez T, Tellez-AvilaFI, Sanchez-Avila F, Montano-Reyes MA, Uribe M.Insulin sensitizers in treatment of nonalcoholic fattyliver disease: Systematic review. World J Gastroenterol2006;12:7826–31.Clarke J, van Tulder M, Blomberg S, de Vet H, van derHeijden G, Bronfort G. Traction for low back pain withor without sciatica: an updated systematic review withinthe framework of the Cochrane collaboration. Spine2006;31:1591–9.Collins CE, Warren J, Neve M, McCoy P, Stokes BJ.Measuring effectiveness of dietetic interventions in childobesity: a systematic review of randomized trials. ArchPediatr Adolesc Med 2006;160:906–22.Cook SA, Rosser R, Salmon P. Is cosmetic surgery aneffective psychotherapeutic intervention? A systematicreview of the evidence. J Plast Reconstr Aesthet Surg2006;59:1133–51.Dahlof CG, Pascual J, Dodick DW, Dowson AJ. Efficacy,speed of action and tolerability of almotriptan in theacute treatment of migraine: pooled individual patientdata from four randomized, double-blind, placebocontrolledclinical trials. Cephalalgia 2006;26:400–8.Davey P, Brown E, Fenelon L, Finch R, Gould I,Holmes A, et al. Systematic review of antimicrobial drugprescribing in hospitals. Emerg Infect Dis 2006;12:211–6.De Schryver EL, Algra A, van Gijn J. Dipyridamolefor preventing stroke and other vascular events inpatients with vascular disease. Cochrane Database Syst Rev2006(2):CD001820.Demyttenaere S, Feldman LS, Fried GM. Effect ofpneumoperitoneum on renal perfusion and function: asystematic review. Surg Endosc 2007;21:152–60.169

Appendix 17170Dibra A, Kastrati A, Alfonso F, Seyfarth M, Perez-Vizcayno MJ, Mehilli J, et al. Effectiveness of drug-elutingstents in patients with bare-metal in-stent restenosis:meta-analysis of randomized trials. J Am Coll Cardiol2007;49:616–23.Dos Santos-Neto LL, de Vilhena Toledo MA, Medeiros-Souza P, de Souza GA. The use of herbal medicine inAlzheimer’s disease-a systematic review. Evid BasedComplement Alternat Med 2006;3:441–5.Eisenberg E, McNicol ED, Carr DB. Efficacy of muopioidagonists in the treatment of evoked neuropathicpain: Systematic review of randomized controlled trials.Eur J Pain 2006;10:667–76.Elahi MM, Khan JS. Living with off-pump coronaryartery surgery: evolution, development, and clinicalpotential for coronary heart disease patients. Heart SurgForum 2006;9:E630–7.Elder JS, Diaz M, Caldamone AA, Cendron M,Greenfield S, Hurwitz R, et al. Endoscopic therapy forvesicoureteral reflux: a meta-analysis. I. Reflux resolutionand urinary tract infection. J Urol 2006;175:716–22.Elkins MR, Jones A, van der Schans C. Positiveexpiratory pressure physiotherapy for airway clearancein people with cystic fibrosis. Cochrane Database Syst Rev2006(2):CD003147.Faber E, Kuiper JI, Burdorf A, Miedema HS, VerhaarJA. Treatment of impingement syndrome: a systematicreview of the effects on functional limitations and returnto work. J Occup Rehabil 2006;16:7–25.Fabrizi F, Dixit V, Martin P. Meta-analysis: antiviraltherapy of hepatitis B virus-associatedglomerulonephritis. Aliment Pharmacol Ther 2006;24:781–8.Falagas ME, Manta KG, Ntziora F, Vardakas KZ.Linezolid for the treatment of patients with endocarditis:a systematic review of the published evidence. J AntimicrobChemother 2006;58:273–80.Finer NN, Barrington KJ. Nitric oxide for respiratoryfailure in infants born at or near term. Cochrane DatabaseSyst Rev 2006(4):CD000399.Francis J, Johnson B, Niehaus M. Quality of life inpatients with implantable cardioverter defibrillators.Indian Pacing Electrophysiol J 2006;6:173–81.Fung AT, Reid SE, Jones MP, Healey PR, McCluskeyPJ, Craig JC. Meta-analysis of randomised controlledtrials comparing latanoprost with brimonidine in thetreatment of open-angle glaucoma, ocular hypertensionor normal-tension glaucoma. Br J Ophthalmol2007;91:62–8.Gafter-Gvili A, Paul M, Fraser A, Leibovici L. Effect ofquinolone prophylaxis in afebrile neutropenic patientson microbial resistance: systematic review and metaanalysis.J Antimicrob Chemother 2007;59:5–22.Gagner M, Boza C. Laparoscopic duodenal switch formorbid obesity. Expert Rev Med Devices 2006;3:105–12.Gajdos P, Chevret S, Toyka K. Intravenousimmunoglobulin for myasthenia gravis. Cochrane DatabaseSyst Rev 2006(2):CD002277.Getz M, Hutzler Y, Vermeer A. Effects of aquaticinterventions in children with neuromotor impairments:a systematic review of the literature. Clin Rehabil2006;20:927–36.Glasmacher A, Hahn C, Hoffmann F, Naumann R,Goldschmidt H, von Lilienfeld-Toal M, et al. A systematicreview of phase-II trials of thalidomide monotherapy inpatients with relapsed or refractory multiple myeloma.Br J Haematol 2006;132:584–93.Goonetilleke KS, Siriwardena AK. Systematic review ofperi-operative nutritional supplementation in patientsundergoing pancreaticoduodenectomy. JPancreas2006;7:5–13.Gupta VK. Botulinum toxin – a treatment for migraine?A systematic review. Pain Med 2006;7:386–94.Hadley G, Derry S, Moore RA. Imiquimod for actinickeratosis: systematic review and meta-analysis. J InvestDermatol 2006;126:1251–5.Helin RD, Angeles ST, Bhat R. Octreotide therapy forchylothorax in infants and children: A brief review.Pediatr Crit Care Med 2006;7:576–9.Hickey BE, Francis D, Lehman MH. Sequencing ofchemotherapy and radiation therapy for early breastcancer. Cochrane Database Syst Rev 2006(4):CD005212.Ho KM, Sheridan DJ. Meta-analysis of frusemide toprevent or treat acute renal failure. BMJ 2006;333:420.Huang HY, Caballero B, Chang S, Alberg AJ, SembaRD, Schneyer CR, et al. The efficacy and safety ofmultivitamin and mineral supplement use to preventcancer and chronic disease in adults: a systematic reviewfor a National Institutes of Health state-of-the-scienceconference. Ann Intern Med 2006;145:372–85.Ingram C, Courneya KS, Kingston D. The effects ofexercise on body weight and composition in breastcancer survivors: an integrative systematic review. OncolNurs Forum 2006;33:937–47; quiz 948–50.Issa AM, Mojica WA, Morton SC, Traina S, Newberry SJ,Hilton LG, et al. The efficacy of omega-3 fatty acids oncognitive function in aging and dementia: a systematicreview. Dement Geriatr Cogn Disord 2006;21:88–96.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Jimbo M, Nease DE Jr, Ruffin MTt, Rana GK.Information technology and cancer prevention. CACancer J Clin 2006;56:26–36; quiz 48–9.Johnson CE, Danhauer JL, Reith AC, Latiolais LN. Asystematic review of the nonacoustic benefits of boneanchoredhearing AIDS. Ear Hear 2006;27:703–13.Kalanda GC, Hill J, Verhoeff FH, Brabin BJ.Comparative efficacy of chloroquine and sulphadoxine–pyrimethamine in pregnant women and children: ameta-analysis. Trop Med Int Health 2006;11:569–77.Keeley EC, Boura JA, Grines CL. Comparison of primaryand facilitated percutaneous coronary interventions forST-elevation myocardial infarction: quantitative review ofrandomised trials. Lancet 2006;367:579–88.Kelley GA, Kelley KS. Aerobic exercise and HDL2-C:a meta-analysis of randomized controlled trials.Atherosclerosis 2006;184:207–15.Khanna A, Walker GR, Livingstone AS, ArheartKL, Rocha-Lima C, Koniaris LG. Is adjuvant 5-FUbasedchemoradiotherapy for resectable pancreaticadenocarcinoma beneficial? A meta-analysis of anunanswered question. J Gastrointest Surg 2006;10:689–97.Kingma JJ, de Knikker R, Wittink HM, Takken T.Eccentric overload training in patients with chronicAchilles tendinopathy: a systematic review. Br J SportsMed 2007;41:e3.Kirby D, Obasi A, Laris BA. The effectiveness of sexeducation and HIV education interventions in schoolsin developing countries. World Health Organ Tech Rep Ser2006;938:103–50; discussion 317–41.Kleiner-Fisman G, Herzog J, Fisman DN, Tamma F,Lyons KE, Pahwa R, et al. Subthalamic nucleus deepbrain stimulation: summary and meta-analysis ofoutcomes. Mov Disord 2006;21(Suppl. 14):S290–304.Kyrgiou M, Salanti G, Pavlidis N, Paraskevaidis E,Ioannidis JP. Survival benefits with diverse chemotherapyregimens for ovarian cancer: meta-analysis of multipletreatments. J Natl Cancer Inst 2006;98:1655–63.Lander JA, Weltman BJ, So SS. EMLA and amethocainefor reduction of children’s pain associated with needleinsertion. Cochrane Database Syst Rev 2006;3:CD004236.Law M, Rudnicka AR. Statin safety: a systematic review.Am J Cardiol 2006;97(8A):52C–60C.Le Corvoisier P, Hittinger L, Chanson P, Montagne O,Macquin-Mavier I, Maison P. Cardiac effects of growthhormone treatment in chronic heart failure: A metaanalysis.J Clin Endocrinol Metab 2007;92:180–5.Lee SJ, Schover LR, Partridge AH, Patrizio P, WallaceWH, Hagerty K, et al. American Society of Clinical© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Oncology recommendations on fertility preservation incancer patients. J Clin Oncol 2006;24:2917–31.Liberatore Rdel R Jr, Damiani D. Insulin pump therapyin type 1 diabetes mellitus. J Pediatr (Rio J) 2006;82:249–54.Lodi G, Sardella A, Bez C, Demarosi F, Carrassi A.Interventions for treating oral leukoplakia. CochraneDatabase Syst Rev 2006(4):CD001829.Lyman GH, Glaspy J. Are there clinical benefits withearly erythropoietic intervention for chemotherapyinducedanemia? A systematic review. Cancer2006;106:223–33.McCart MR, Priester PE, Davies WH, Azen R.Differential effectiveness of behavioral parent-trainingand cognitive-behavioral therapy for antisocial youth: ameta-analysis. J Abnorm Child Psychol 2006;34:527–43.McPhail MJ, Abu-Hilal M, Johnson CD. A meta-analysiscomparing suprapubic and transurethral catheterizationfor bladder drainage after abdominal surgery. Br J Surg2006;93:1038–44.Meijering S, Corstjens AM, Tulleken JE, Meertens JH,Zijlstra JG, Ligtenberg JJ. Towards a feasible algorithmfor tight glycaemic control in critically ill patients: asystematic review of the literature. Crit Care 2006;10:R19.Milne AC, Avenell A, Potter J. Meta-analysis: protein andenergy supplementation in older people. Ann Intern Med2006;144:37–48.Murphy MH, Nevill AM, Murtagh EM, Holder RL. Theeffect of walking on fitness, fatness and resting bloodpressure: a meta-analysis of randomised, controlledtrials. Prev Med 2007;44:377–85.Niebauer K, Dewilde S, Fox-Rushby J, Revicki DA.Impact of omalizumab on quality-of-life outcomes inpatients with moderate-to-severe allergic asthma. AnnAllergy Asthma Immunol 2006;96:316–26.Noble J, Ellis PM, Mackay JA, Evans WK. Second-line orsubsequent systemic therapy for recurrent or progressivenon-small cell lung cancer: a systematic review andpractice guideline. J Thorac Oncol 2006;1:1042–58.Oktay K, Cil AP, Bang H. Efficiency of oocytecryopreservation: a meta-analysis. Fertil Steril2006;86:70–80.Oliver D, Connelly JB, Victor CR, Shaw FE, WhiteheadA, Genc Y, et al. Strategies to prevent falls and fracturesin hospitals and care homes and effect of cognitiveimpairment: systematic review and meta-analyses. BMJ2007;334:82.Papakostas GI, Fava M. A meta-analysis of clinical trialscomparing milnacipran, a serotonin–norepinephrine171

Appendix 17172reuptake inhibitor, with a selective serotonin reuptakeinhibitor for the treatment of major depressive disorder.Eur Neuropsychopharmacol 2007;17:32–6.Paul M, Yahav D, Fraser A, Leibovici L. Empiricalantibiotic monotherapy for febrile neutropenia:systematic review and meta-analysis of randomizedcontrolled trials. J Antimicrob Chemother 2006;57:176–89.Petignat P, du Bois A, Bruchim I, Fink D, ProvencherDM. Should intraperitoneal chemotherapy be consideredas standard first-line treatment in advanced stage ovariancancer? Crit Rev Oncol Hematol 2007;62:137–47.Petrie J, Bunn F, Byrne G. Parenting programmes forpreventing tobacco, alcohol or drugs misuse in children

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8events: meta-analysis of randomized trials. JAMA2006;296:1619–32.Risk factor reviews (n = 100)Aalto TJ, Malmivaara A, Kovacs F, Herno A, Alen M,Salmi L, et al. Preoperative predictors for postoperativeclinical outcome in lumbar spinal stenosis: systematicreview. Spine 2006;31:E648–63.Abhinav K, Al-Chalabi A, Hortobagyi T, Leigh PN.Electrical injury and amyotrophic lateral sclerosis: asystematic review of the literature. J Neurol NeurosurgPsychiatry 2007;78:450–3.Akobeng AK, Ramanan AV, Buchan I, Heller RF. Effectof breast feeding on risk of coeliac disease: a systematicreview and meta-analysis of observational studies. ArchDis Child 2006;91:39–43.Alder N, Fenty J, Warren F, Sutton AJ, Rushton L, JonesDR, et al. Meta-analysis of mortality and cancer incidenceamong workers in the synthetic rubber-producingindustry. Am J Epidemiol 2006;164:405–20.Alexander DD, Mink PJ, Mandel JH, Kelsh MA. A metaanalysisof occupational trichloroethylene exposureand multiple myeloma or leukaemia. Occup Med (Lond)2006;56:485–93.Altman MR, Lydon-Rochelle MT. Prolonged secondstage of labor and risk of adverse maternal and perinataloutcomes: a systematic review. Birth 2006;33:315–22.Ambroise D, Wild P, Moulin JJ. Update of a meta-analysison lung cancer and welding. Scand J Work Environ Health2006;32:22–31.Anderson MA, Levsen J, Dusio ME, Bryant PJ,Brown SM, Burr CM, et al. Evidence-based factors inreadmission of patients with heart failure. J Nurs CareQual 2006;21:160–7.Asia PCSC. Coronary risk prediction for those withand without diabetes. Eur J Cardiovasc Prev Rehabil2006;13:30–6.Atsma F, Bartelink ML, Grobbee DE, van der SchouwYT. Postmenopausal status and early menopause asindependent risk factors for cardiovascular disease: ameta-analysis. Menopause 2006;13:265–79.Azarpazhooh A, Leake JL. Systematic review of theassociation between respiratory diseases and oral health.J Periodontol 2006;77:1465–82.Batty GD, Deary IJ, Gottfredson LS. Premorbid (earlylife) IQ and later mortality risk: systematic review. AnnEpidemiol 2007;17:278–88.Beam JW, Buckley B. Community-acquired methicillinresistantStaphylococcus aureus: prevalence and risk factors.J Athl Train 2006;41:337–40.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Brimble KS, Walker M, Margetts PJ, Kundhal KK, RabbatCG. Meta-analysis: peritoneal membrane transport,mortality, and technique failure in peritoneal dialysis. JAm Soc Nephrol 2006;17:2591–8.Butterworth AS, Higgins JP, Pharoah P. Relativeand absolute risk of colorectal cancer for individualswith a family history: a meta-analysis. Eur J Cancer2006;42:216–27.Carter OB. The weighty issue of Australian televisionfood advertising and childhood obesity. Health Promot JAustr 2006;17:5–11.Cheng JY, Ng EM, Ko JS, Chen RY. Physical activity anderectile dysfunction: meta-analysis of population-basedstudies. Int J Impot Res 2007;19:245–52.Cornish J, Tan E, Teare J, Teoh TG, Rai R, Clark SK,et al. A meta-analysis on the influence of inflammatorybowel disease on pregnancy. Gut 2007;56:830–7.Curtis KM, Mohllajee AP, Martins SL, Peterson HB.Combined oral contraceptive use among womenwith hypertension: a systematic review. Contraception2006;73:179–88.da Silva Dal Pizzol T, Knop FP, Mengue SS. Prenatalexposure to misoprostol and congenital anomalies:systematic review and meta-analysis. Reprod Toxicol2006;22:666–71.Dauchet L, Amouyel P, Hercberg S, Dallongeville J.Fruit and vegetable consumption and risk of coronaryheart disease: a meta-analysis of cohort studies. J Nutr2006;136:2588–93.den Boer JJ, Oostendorp RA, Beems T, Munneke M,Oerlemans M, Evers AW. A systematic review of biopsychosocialrisk factors for an unfavourable outcomeafter lumbar disc surgery. Eur Spine J 2006;15:527–36.Ding EL, Song Y, Malik VS, Liu S. Sex differencesof endogenous sex hormones and risk of type 2diabetes: a systematic review and meta-analysis. JAMA2006;295:1288–99.Dionne CE, Dunn KM, Croft PR. Does back painprevalence really decrease with increasing age? Asystematic review. Age Ageing 2006;35:229–34.Fahmy NM, Mahmud S, Aprikian AG. Delay inthe surgical treatment of bladder cancer andsurvival: systematic review of the literature. Eur Urol2006;50:1176–82.Falagas ME, Kopterides P. Risk factors for the isolationof multi-drug-resistant Acinetobacter baumannii andPseudomonas aeruginosa: a systematic review of theliterature. J Hosp Infect 2006;64:7–15.173

Appendix 17174Falleti MG, Maruff P, Burman P, Harris A. The effects ofgrowth hormone (GH) deficiency and GH replacementon cognitive performance in adults: a meta-analysisof the current literature. Psychoneuroendocrinology2006;31:681–91.Flores-Mateo G, Navas-Acien A, Pastor-Barriuso R,Guallar E. Selenium and coronary heart disease: a metaanalysis.Am J Clin Nutr 2006;84:762–73.Franks HM, Roesch SC. Appraisals and coping inpeople living with cancer: a meta-analysis. Psychooncology2006;15:1027–37.Furber AS, Maheswaran R, Newell JN, Carroll C. Issmoking tobacco an independent risk factor for HIVinfection and progression to AIDS? A systemic review. SexTransm Infect 2007;83:41–6.Galassi A, Reynolds K, He J. Metabolic syndrome andrisk of cardiovascular disease: a meta-analysis. Am J Med2006;119:812–9.Gheeraert PJ, De Buyzere ML, Taeymans YM, GillebertTC, Henriques JP, De Backer G, et al. Risk factors forprimary ventricular fibrillation during acute myocardialinfarction: a systematic review and meta-analysis. EurHeart J 2006;27:2499–510.Gomes B, Higginson IJ. Factors influencing death athome in terminally ill patients with cancer: systematicreview. BMJ 2006;332:515–21.Hay JL, McCaul KD, Magnan RE. Does worryabout breast cancer predict screening behaviors? Ameta-analysis of the prospective evidence. Prev Med2006;42:401–8.He FJ, Nowson CA, MacGregor GA. Fruit and vegetableconsumption and stroke: meta-analysis of cohort studies.Lancet 2006;367:320–6.Hernandez-Diaz S, Varas-Lorenzo C, Garcia RodriguezLA. Non-steroidal antiinflammatory drugs and the riskof acute myocardial infarction. Basic Clin PharmacolToxicol 2006;98:266–74.Hobbs CG, Sterne JA, Bailey M, Heyderman RS, BirchallMA, Thomas SJ. Human papillomavirus and head andneck cancer: a systematic review and meta-analysis. ClinOtolaryngol 2006;31:259–66.Holscher T, Bentzen SM, Baumann M. Influenceof connective tissue diseases on the expression ofradiation side effects: a systematic review. Radiother Oncol2006;78:123–30.Huang JS, Lee TA, Lu MC. Prenatal programming ofchildhood overweight and obesity. Matern Child Health J2007;11:461–73.Hughes JR, Carpenter MJ. Does smoking reductionincrease future cessation and decrease disease risk? Aqualitative review. Nicotine Tob Res 2006;8:739–49.Jackson CA, Sudlow CL. Is hypertension a more frequentrisk factor for deep than for lobar supratentorialintracerebral haemorrhage? J Neurol Neurosurg Psychiatry2006;77:1244–52.Jewett M, Rendon R, Dranitsaris G, Drachenberg D,Tanguay S, Donnelly B, et al. Does prolonging thetime to renal cancer surgery affect long-term cancercontrol: a systematic review of the literature. Can J Urol2006;13(Suppl. 3):54–61.Kamphuis CB, Giskes K, de Bruijn GJ, Wendel-Vos W,Brug J, van Lenthe FJ. Environmental determinantsof fruit and vegetable consumption among adults: asystematic review. Br J Nutr 2006;96:620–35.Kantovitz KR, Pascon FM, Rontani RM, Gaviao MB.Obesity and dental caries – A systematic review. OralHealth Prev Dent 2006;4:137–44.Kasper JS, Giovannucci E. A meta-analysis of diabetesmellitus and the risk of prostate cancer. Cancer EpidemiolBiomarkers Prev 2006;15:2056–62.Khambalia A, Joshi P, Brussoni M, Raina P, MorrongielloB, Macarthur C. Risk factors for unintentional injuriesdue to falls in children aged 0–6 years: a systematicreview. Inj Prev 2006;12:378–81.Knol MJ, Twisk JW, Beekman AT, Heine RJ, Snoek FJ,Pouwer F. Depression as a risk factor for the onset oftype 2 diabetes mellitus. A meta-analysis. Diabetologia2006;49:837–45.Koppes LL, Dekker JM, Hendriks HF, Bouter LM, HeineRJ. Meta-analysis of the relationship between alcoholconsumption and coronary heart disease and mortality intype 2 diabetic patients. Diabetologia 2006;49:648–52.Kubo A, Corley DA. Body mass index andadenocarcinomas of the esophagus or gastric cardia: asystematic review and meta-analysis. Cancer EpidemiolBiomarkers Prev 2006;15:872–8.Langan SM, Williams HC. What causes worseningof eczema? A systematic review. Br J Dermatol2006;155:504–14.Larsson SC, Orsini N, Wolk A. Milk, milk products andlactose intake and ovarian cancer risk: a meta-analysis ofepidemiological studies. Int J Cancer 2006;118:431–41.Leandro G, Mangia A, Hui J, Fabris P, Rubbia-BrandtL, Colloredo G, et al. Relationship between steatosis,inflammation, and fibrosis in chronic hepatitis C: ameta-analysis of individual patient data. Gastroenterology2006;130:1636–42.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Lee PN, Forey BA. Environmental tobacco smokeexposure and risk of stroke in nonsmokers: A review withmeta-analysis. J Stroke Cerebrovasc Dis 2006;15:190–201.Leeners B, Richter-Appelt H, Imthurn B, Rath W.Influence of childhood sexual abuse on pregnancy,delivery, and the early postpartum period in adultwomen. J Psychosom Res 2006;61:139–51.Littlejohn C. Does socio-economic status influencethe acceptability of, attendance for, and outcome of,screening and brief interventions for alcohol misuse: areview. Alcohol Alcohol 2006;41:540–5.Ma H, Bernstein L, Pike MC, Ursin G. Reproductivefactors and breast cancer risk according to joint estrogenand progesterone receptor status: a meta-analysis ofepidemiological studies. Breast Cancer Res 2006;8:R43.Maguire S, Mann M, John N, Ellaway B, Sibert JR,Kemp AM. Does cardiopulmonary resuscitation causerib fractures in children? A systematic review. Child AbuseNegl 2006;30:739–51.Mahon NE, Yarcheski A, Yarcheski TJ, Cannella BL,Hanks MM. A meta-analytic study of predictors forloneliness during adolescence. Nurs Res 2006;55:308–15.Maki DG, Kluger DM, Crnich CJ. The risk ofbloodstream infection in adults with differentintravascular devices: a systematic review of 200published prospective studies. Mayo Clin Proc2006;81:1159–71.Mehra R, Moore BA, Crothers K, Tetrault J, FiellinDA. The association between marijuana smokingand lung cancer: a systematic review. Arch Intern Med2006;166:1359–67.Mills EJ, Nachega JB, Bangsberg DR, Singh S, RachlisB, Wu P, et al. Adherence to HAART: a systematic reviewof developed and developing nation patient-reportedbarriers and facilitators. PLoS Med 2006;3:e438.Mills EJ, Seely D, Rachlis B, Griffith L, Wu P, Wilson K, etal. Barriers to participation in clinical trials of cancer: ameta-analysis and systematic review of patient-reportedfactors. Lancet Oncol 2006;7:141–8.Mizoue T, Tanaka K, Tsuji I, Wakai K, Nagata C, OtaniT, et al. Alcohol drinking and colorectal cancer risk: anevaluation based on a systematic review of epidemiologicevidence among the Japanese population. Jpn J ClinOncol 2006;36:582–97.Murphy VE, Clifton VL, Gibson PG. Asthmaexacerbations during pregnancy: incidence andassociation with adverse pregnancy outcomes. Thorax2006;61:169–76.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Naito M, Yuasa H, Nomura Y, Nakayama T, HamajimaN, Hanada N. Oral health status and health-relatedquality of life: a systematic review. J Oral Sci 2006;48:1–7.Nakagami T, Qiao Q, Tuomilehto J, Balkau B,Tajima N, Hu G, et al. Screen-detected diabetes,hypertension and hypercholesterolemia as predictorsof cardiovascular mortality in five populations of Asianorigin: the DECODA study. Eur J Cardiovasc Prev Rehabil2006;13:555–61.Neri M, Ugolini D, Bonassi S, Fucic A, Holland N,Knudsen LE, et al. Children’s exposure to environmentalpollutants and biomarkers of genetic damage. II. Resultsof a comprehensive literature search and meta-analysis.Mutat Res 2006;612:14–39.Newman LA, Griffith KA, Jatoi I, Simon MS, CroweJP, Colditz GA. Meta-analysis of survival in AfricanAmerican and white American patients with breastcancer: ethnicity compared with socioeconomic status. JClin Oncol 2006;24:1342–9.Omarova A, Phillips CJ. A meta-analysis of literaturedata relating to the relationships between cadmiumintake and toxicity indicators in humans. Environ Res2007;103:432–40.Onega T, Baron J, MacKenzie T. Cancer after total jointarthroplasty: a meta-analysis. Cancer Epidemiol BiomarkersPrev 2006;15:1532–7.Ong KK, Loos RJ. Rapid infancy weight gain andsubsequent obesity: systematic reviews and hopefulsuggestions. Acta Paediatr 2006;95:904–8.Papatheodoridis GV, Sougioultzis S, ArchimandritisAJ. Effects of Helicobacter pylori and nonsteroidal antiinflammatorydrugs on peptic ulcer disease: a systematicreview. Clin Gastroenterol Hepatol 2006;4:130–42.Paradies Y. A systematic review of empirical researchon self-reported racism and health. Int J Epidemiol2006;35:888–901.Paydarfar JA, Birkmeyer NJ. Complications in headand neck surgery: a meta-analysis of postlaryngectomypharyngocutaneous fistula. Arch Otolaryngol Head NeckSurg 2006;132:67–72.Reis FJ, Sousa TA, Oliveira MS, Dantas N, SilveiraM, Braghiroly MI, et al. Is hepatitis C virus a cause ofidiopathic dilated cardiomyopathy?: A systematic reviewof literature. Braz J Infect Dis 2006;10:199–202.Rhodes RE, Smith NE. Personality correlates of physicalactivity: a review and meta-analysis. Br J Sports Med2006;40:958–65.Robertson L, Wu O, Langhorne P, Twaddle S, ClarkP, Lowe GD, et al. Thrombophilia in pregnancy: asystematic review. Br J Haematol 2006;132:171–96.175

Appendix 17176Rocha AT, de Vasconcellos AG, da Luz Neto ER,Araujo DM, Alves ES, Lopes AA. Risk of venousthromboembolism and efficacy of thromboprophylaxis inhospitalized obese medical patients and in obese patientsundergoing bariatric surgery. Obes Surg 2006;16:1645–55.Rodrigues MC, Mello RR, Fonseca SC. Learningdifficulties in schoolchildren born with very low birthweight. J Pediatr (Rio J) 2006;82:6–14.Simpson SH, Eurich DT, Majumdar SR, Padwal RS,Tsuyuki RT, Varney J, et al. A meta-analysis of theassociation between adherence to drug therapy andmortality. BMJ 2006;333:15.Singh G, Wu O, Langhorne P, Madhok R. Risk of acutemyocardial infarction with nonselective non-steroidalanti-inflammatory drugs: a meta-analysis. Arthritis ResTher 2006;8:R153.Smeets RJ, Wade D, Hidding A, Van Leeuwen PJ,Vlaeyen JW, Knottnerus JA. The association of physicaldeconditioning and chronic low back pain: a hypothesisorientedsystematic review. Disabil Rehabil 2006;28:673–93.Szilagyi A, Nathwani U, Vinokuroff C, Correa JA, ShrierI. The effect of lactose maldigestion on the relationshipbetween dairy food intake and colorectal cancer: asystematic review. Nutr Cancer 2006;55:141–50.Thornton J, Kelly SP, Harrison RA, Edwards R. Cigarettesmoking and thyroid eye disease: a systematic review. Eye2007;21:1135–45.Tinazzi E, Ficarra V, Simeoni S, Artibani W, LunardiC. Reactive arthritis following BCG immunotherapyfor urinary bladder carcinoma: a systematic review.Rheumatol Int 2006;26:481–8.Tokumaru O, Haruki K, Bacal K, Katagiri T, YamamotoT, Sakurai Y. Incidence of cancer among female flightattendants: a meta-analysis. J Travel Med 2006;13:127–32.Truong KD, Ma S. A systematic review of relationsbetween neighborhoods and mental health. J Ment HealthPolicy Econ 2006;9:137–54.Vamvakas EC. Pneumonia as a complication of bloodproduct transfusion in the critically ill: transfusionrelatedimmunomodulation (TRIM). Crit Care Med2006;34(5 Suppl.):S151–9.van der Horst K, Oenema A, Ferreira I, Wendel-VosW, Giskes K, van Lenthe F, et al. A systematic reviewof environmental correlates of obesity-related dietarybehaviors in youth. Health Educ Res 2007;22:203–26.Van Maele-Fabry G, Libotte V, Willems J, Lison D. Reviewand meta-analysis of risk estimates for prostate cancer inpesticide manufacturing workers. Cancer Causes Control2006;17:353–73.van Velzen JM, van Bennekom CA, Polomski W,Slootman JR, van der Woude LH, Houdijk H. Physicalcapacity and walking ability after lower limb amputation:a systematic review. Clin Rehabil 2006;20:999–1016.Wakai K, Inoue M, Mizoue T, Tanaka K, Tsuji I,Nagata C, et al. Tobacco smoking and lung cancerrisk: an evaluation based on a systematic reviewof epidemiological evidence among the Japanesepopulation. Jpn J Clin Oncol 2006;36:309–24.Warner L, Stone KM, Macaluso M, Buehler JW, AustinHD. Condom use and risk of gonorrhea and Chlamydia:a systematic review of design and measurement factorsassessed in epidemiologic studies. Sex Transm Dis2006;33(1):36–51.Whalley GA, Gamble GD, Doughty RN. The prognosticsignificance of restrictive diastolic filling associated withheart failure: a meta-analysis. Int J Cardiol 2007;116:70–7.Wind J, Lagarde SM, Ten Kate FJ, Ubbink DT,Bemelman WA, van Lanschot JJ. A systematic reviewon the significance of extracapsular lymph nodeinvolvement in gastrointestinal malignancies. Eur J SurgOncol 2007;33:401–8.Wohl M, Gorwood P. Paternal ages below or above35 years old are associated with a different riskof schizophrenia in the offspring. Eur Psychiatry2007;22:22–6.Zhang B, Wing YK. Sex differences in insomnia: a metaanalysis.Sleep 2006;29:85–93.Zhang XF, Attia J, D’Este C, Ma XY. The relationshipbetween higher blood pressure and ischaemic,haemorrhagic stroke among Chinese and Caucasians:meta-analysis. Eur J Cardiovasc Prev Rehabil 2006;13:429–37.Zhao Y, Wang S, Aunan K, Seip HM, Hao J. Air pollutionand lung cancer risks in China – a meta-analysis. Sci TotalEnviron 2006;366:500–13.Zumkeller N, Brenner H, Zwahlen M, Rothenbacher D.Helicobacter pylori infection and colorectal cancer risk: ameta-analysis. Helicobacter 2006;11:75–80.Diagnostic reviews (n = 50)Alongi F, Ragusa P, Montemaggi P, Bona CM. Combiningindependent studies of diagnostic fluorodeoxyglucosepositron-emission tomography and computedtomography in mediastinal lymph node staging for nonsmallcell lung cancer. Tumori 2006;92:327–33.Bandera E, Botteri M, Minelli C, Sutton A, Abrams KR,Latronico N. Cerebral blood flow threshold of ischemic

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8penumbra and infarct core in acute ischemic stroke: asystematic review. Stroke 2006;37:1334–9.Beattie WS, Abdelnaem E, Wijeysundera DN, BuckleyDN. A meta-analytic comparison of preoperative stressechocardiography and nuclear scintigraphy imaging.Anesth Analg 2006;102:8–16.Benatar M. A systematic review of diagnostic studies inmyasthenia gravis. Neuromuscul Disord 2006;16:459–67.Benjaminse A, Gokeler A, van der Schans CP. Clinicaldiagnosis of an anterior cruciate ligament rupture: ameta-analysis. J Orthop Sports Phys Ther 2006;36:267–88.Biagini E, Shaw LJ, Poldermans D, Schinkel AF, RizzelloV, Elhendy A, et al. Accuracy of non-invasive techniquesfor diagnosis of coronary artery disease and predictionof cardiac events in patients with left bundle branchblock: a meta-analysis. Eur J Nucl Med Mol Imaging2006;33:1442–51.Brealey S, Scally A, Hahn S, Thomas N, Godfrey C,Crane S. Accuracy of radiographers red dot or triage ofaccident and emergency radiographs in clinical practice:a systematic review. Clin Radiol 2006;61:604–15.Carnes D, Ashby D, Underwood M. A systematic reviewof pain drawing literature: should pain drawings be usedfor psychologic screening? Clin J Pain 2006;22:449–57.Chappuis F, Rijal S, Soto A, Menten J, Boelaert M. Ameta-analysis of the diagnostic performance of thedirect agglutination test and rK39 dipstick for visceralleishmaniasis. BMJ 2006;333:723.Christou MA, Siontis GC, Katritsis DG, IoannidisJP. Meta-analysis of fractional flow reserve versusquantitative coronary angiography and noninvasiveimaging for evaluation of myocardial ischemia. Am JCardiol 2007;99:450–6.Davenport C, Cheng EY, Kwok YT, Lai AH, WakabayashiT, Hyde C, et al. Assessing the diagnostic test accuracyof natriuretic peptides and ECG in the diagnosis of leftventricular systolic dysfunction: a systematic review andmeta-analysis. Br J Gen Pract 2006;56:48–56.de Graaf I, Prak A, Bierma-Zeinstra S, Thomas S,Peul W, Koes B. Diagnosis of lumbar spinal stenosis: asystematic review of the accuracy of diagnostic tests. Spine2006;31:1168–76.Detsky ME, McDonald DR, Baerlocher MO, TomlinsonGA, McCrory DC, Booth CM. Does this patient withheadache have a migraine or need neuroimaging? JAMA2006;296:1274–83.Doria AS, Moineddin R, Kellenberger CJ, EpelmanM, Beyene J, Schuh S, et al. US or CT for diagnosis ofappendicitis in children and adults? A meta-analysis.Radiology 2006;241:83–94.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Gisbert JP, Abraira V. Accuracy of Helicobacter pyloridiagnostic tests in patients with bleeding peptic ulcer:a systematic review and meta-analysis. Am J Gastroenterol2006;101:848–63.Gisbert JP, de la Morena F, Abraira V. Accuracy ofmonoclonal stool antigen test for the diagnosis of H.pylori infection: a systematic review and meta-analysis. AmJ Gastroenterol 2006;101:1921–30.Gupta SG, Wang LC, Penas PF, Gellenthin M, Lee SJ,Nghiem P. Sentinel lymph node biopsy for evaluationand treatment of patients with Merkel cell carcinoma:The Dana-Farber experience and meta-analysis of theliterature. Arch Dermatol 2006;142:685–90.Hogg K, Brown G, Dunning J, Wright J, Carley S, FoexB, et al. Diagnosis of pulmonary embolism with CTpulmonary angiography: a systematic review. Emerg Med J2006;23:172–8.Hollingworth W, Medina LS, Lenkinski RE, Shibata DK,Bernal B, Zurakowski D, et al. A systematic literaturereview of magnetic resonance spectroscopy for thecharacterization of brain tumors. AJNR Am J Neuroradiol2006;27:1404–11.Jones AE, Fiechtl JF, Brown MD, Ballew JJ, Kline JA.Procalcitonin test in the diagnosis of bacteremia: a metaanalysis.Ann Emerg Med 2007;50:34–41.Karassa FB, Afeltra A, Ambrozic A, Chang DM, DeKeyser F, Doria A, et al. Accuracy of anti-ribosomalP protein antibody testing for the diagnosis ofneuropsychiatric systemic lupus erythematosus: aninternational meta-analysis. Arthritis Rheum 2006;54:312–24.King J, Thatcher N, Pickering C, Hasleton P. Sensitivityand specificity of immunohistochemical antibodiesused to distinguish between benign and malignantpleural disease: a systematic review of published reports.Histopathology 2006;49:561–8.Koliopoulos G, Arbyn M, Martin-Hirsch P, KyrgiouM, Prendiville W, Paraskevaidis E. Diagnostic accuracyof human papillomavirus testing in primary cervicalscreening: a systematic review and meta-analysis of nonrandomizedstudies. Gynecol Oncol 2007;104:232–46.Locker T, Goodacre S, Sampson F, Webster A, SuttonAJ. Meta-analysis of plethysmography and rheographyin the diagnosis of deep vein thrombosis. Emerg Med J2006;23:630–5.Lou L, Lagravere MO, Compton S, Major PW, Flores-Mir C. Accuracy of measurements and reliability oflandmark identification with computed tomography(CT) techniques in the maxillofacial area: a systematicreview. Oral Surg Oral Med Oral Pathol Oral Radiol Endod2007;104:402–11.177

Appendix 17178Major MP, Flores-Mir C, Major PW. Assessment of lateralcephalometric diagnosis of adenoid hypertrophy andposterior upper airway obstruction: a systematic review.Am J Orthod Dentofacial Orthop 2006;130:700–8.Martin A, Portaels F, Palomino JC. Colorimetric redoxindicatormethods for the rapid detection of multidrugresistance in Mycobacterium tuberculosis: a systematic reviewand meta-analysis. J Antimicrob Chemother 2007;59:175–83.Martin JL, Williams KS, Sutton AJ, Abrams KR, AssassaRP. Systematic review and meta-analysis of methods ofdiagnostic assessment for urinary incontinence. NeurourolUrodyn 2006;25:674–83; discussion 684.Myers ER, Bastian LA, Havrilesky LJ, Kulasingam SL,Terplan MS, Cline KE, et al. Management of adnexalmass. Evid Rep Technol Assess (Full Rep) 2006(130):1–145.Nijkeuter M, Ginsberg JS, Huisman MV. Diagnosisof deep vein thrombosis and pulmonary embolismin pregnancy: a systematic review. J Thromb Haemost2006;4:496–500.Peacock F, Morris DL, Anwaruddin S, Christenson RH,Collinson PO, Goodacre SW, et al. Meta-analysis ofischemia-modified albumin to rule out acute coronarysyndromes in the emergency department. Am Heart J2006;152:253–62.Purkayastha S, Chow A, Athanasiou T, Cambaroudis A,Panesar S, Kinross J, et al. Does serum procalcitonin havea role in evaluating the severity of acute pancreatitis? Aquestion revisited. World J Surg 2006;30:1713–21.Riquelme A, Calvo M, Salech F, Valderrama S, Pattillo A,Arellano M, et al. Value of adenosine deaminase (ADA) inascitic fluid for the diagnosis of tuberculous peritonitis: ameta-analysis. J Clin Gastroenterol 2006;40:705–10.Rodgers M, Nixon J, Hempel S, Aho T, Kelly J, NealD, et al. Diagnostic tests and algorithms used in theinvestigation of haematuria: systematic reviews andeconomic evaluation. Health Technol Assess 2006;10:iii-iv,xi-259.Rubinstein SM, Pool JJ, van Tulder MW, Riphagen, II,de Vet HC. A systematic review of the diagnostic accuracyof provocative tests of the neck for diagnosing cervicalradiculopathy. Eur Spine J 2007;16:307–19.Sgouros SN, Pereira SP. Systematic review: sphincter ofOddi dysfunction – non-invasive diagnostic methods andlong-term outcome after endoscopic sphincterotomy.Aliment Pharmacol Ther 2006;24:237–46.Stein PD, Beemath A, Kayali F, Skaf E, Sanchez J,Olson RE. Multidetector computed tomography for thediagnosis of coronary artery disease: a systematic review.Am J Med 2006;119:203–16.Steingart KR, Henry M, Ng V, Hopewell PC, Ramsay A,Cunningham J, et al. Fluorescence versus conventionalsputum smear microscopy for tuberculosis: a systematicreview. Lancet Infect Dis 2006;6:570–81.Steingart KR, Ng V, Henry M, Hopewell PC, RamsayA, Cunningham J, et al. Sputum processing methodsto improve the sensitivity of smear microscopy fortuberculosis: a systematic review. Lancet Infect Dis2006;6:664–74.Sun Z, Jiang W. Diagnostic value of multislice computedtomography angiography in coronary artery disease: ameta-analysis. Eur J Radiol 2006;60:279–86.Tomlinson A, Khanal S, Ramaesh K, Diaper C, McFadyenA. Tear film osmolarity: determination of a referent fordry eye diagnosis. Invest Ophthalmol Vis Sci 2006;47:4309–15.Tuon FF, Litvoc MN, Lopes MI. Adenosine deaminaseand tuberculous pericarditis – a systematic review withmeta-analysis. Acta Trop 2006;99:67–74.Vakil N, Moayyedi P, Fennerty MB, Talley NJ. Limitedvalue of alarm features in the diagnosis of uppergastrointestinal malignancy: systematic review and metaanalysis.Gastroenterology 2006;131:390–401; quiz 659–60.van der Zaag-Loonen HJ, Dikkers R, de Bock GH,Oudkerk M. The clinical value of a negative multidetectorcomputed tomographic angiography in patientssuspected of coronary artery disease: A meta-analysis.Eur Radiol 2006;16:2748–56.Verma D, Kapadia A, Eisen GM, Adler DG. EUS vsMRCP for detection of choledocholithiasis. GastrointestEndosc 2006;64:248–54.Wardlaw JM, Chappell FM, Best JJ, Wartolowska K, BerryE. Non-invasive imaging compared with intra-arterialangiography in the diagnosis of symptomatic carotidstenosis: a meta-analysis. Lancet 2006;367:1503–12.Whiting P, Harbord R, Main C, Deeks JJ, Filippini G,Egger M, et al. Accuracy of magnetic resonance imagingfor the diagnosis of multiple sclerosis: systematic review.BMJ 2006;332:875–84.Whiting P, Westwood M, Bojke L, Palmer S, RichardsonG, Cooper J, et al. Clinical effectiveness and costeffectivenessof tests for the diagnosis and investigationof urinary tract infection in children: a systematic reviewand economic model. Health Technol Assess 2006;10:iii–iv,xi–xiii, 1–154.Will O, Purkayastha S, Chan C, Athanasiou T, Darzi AW,Gedroyc W, et al. Diagnostic precision of nanoparticleenhancedMRI for lymph-node metastases: a metaanalysis.Lancet Oncol 2006;7:52–60.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Xing Y, Foy M, Cox DD, Kuerer HM, Hunt KK, CormierJN. Meta-analysis of sentinel lymph node biopsy afterpreoperative chemotherapy in patients with breastcancer. Br J Surg 2006;93:539–46.Genetic reviews (n = 50)Akomolafe A, Lunetta KL, Erlich PM, Cupples LA,Baldwin CT, Huyck M, et al. Genetic association betweenendothelial nitric oxide synthase and Alzheimer disease.Clin Genet 2006;70:49–56.Annese V, Valvano MR, Palmieri O, Latiano A, Bossa F,Andriulli A. Multidrug resistance 1 gene in inflammatorybowel disease: a meta-analysis. World J Gastroenterol2006;12:3636–44.Aoki T, Hirota T, Tamari M, Ichikawa K, Takeda K,Arinami T, et al. An association between asthma andTNF-308G/A polymorphism: meta-analysis. J Hum Genet2006;51:677–85.Arias A, Feinn R, Kranzler HR. Association of anAsn40Asp (A118G) polymorphism in the mu-opioidreceptor gene with substance dependence: a metaanalysis.Drug Alcohol Depend 2006;83:262–8.Baglietto L, Jenkins MA, Severi G, Giles GG, BishopDT, Boyle P, et al. Measures of familial aggregationdepend on definition of family history: meta-analysis forcolorectal cancer. J Clin Epidemiol 2006;59:114–24.Barroso I, Luan J, Sandhu MS, Franks PW, Crowley V,Schafer AJ, et al.. Meta-analysis of the Gly482Ser variantin PPARGC1A in type 2 diabetes and related phenotypes.Diabetologia 2006;49:501–5.Blomqvist ME, Reynolds C, Katzov H, Feuk L, AndreasenN, Bogdanovic N, et al. Towards compendia of negativegenetic association studies: an example for Alzheimerdisease. Hum Genet 2006;119:29–37.Boccia S, La Torre G, Gianfagna F, Mannocci A, RicciardiG. Glutathione S-transferase T1 status and gastriccancer risk: a meta-analysis of the literature. Mutagenesis2006;21:115–23.Borlak J, Reamon-Buettner SM. N-acetyltransferase 2(NAT2) gene polymorphisms in colon and lung cancerpatients. BMC Med Genet 2006;7:58.Burwick RM, Ramsay PP, Haines JL, Hauser SL,Oksenberg JR, Pericak-Vance MA, et al. APOE epsilonvariation in multiple sclerosis susceptibility and diseaseseverity: some answers. Neurology 2006;66:1373–83.Camargo MC, Mera R, Correa P, Peek RM Jr, FonthamET, Goodman KJ, et al. Interleukin-1beta andinterleukin-1 receptor antagonist gene polymorphismsand gastric cancer: a meta-analysis. Cancer EpidemiolBiomarkers Prev 2006;15:1674–87.© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.Casas JP, Cavalleri GL, Bautista LE, Smeeth L,Humphries SE, Hingorani AD. Endothelial nitric oxidesynthase gene polymorphisms and cardiovasculardisease: a HuGE review. Am J Epidemiol 2006;164:921–35.Daher S, Sass N, Oliveira LG, Mattar R. Cytokinegenotyping in preeclampsia. Am J Reprod Immunol2006;55:130–5.Delgado-Vega AM, Anaya JM. Meta-analysis of HLA-DRB1 polymorphism in Latin American patients withrheumatoid arthritis. Autoimmun Rev 2007;6:402–8.Diep CB, Kleivi K, Ribeiro FR, Teixeira MR, LindgjaerdeOC, Lothe RA. The order of genetic events associatedwith colorectal cancer progression inferred from metaanalysisof copy number changes. Genes ChromosomesCancer 2006;45:31–41.Fang Y, Rivadeneira F, van Meurs JB, Pols HA, IoannidisJP, Uitterlinden AG. Vitamin D receptor gene BsmI andTaqI polymorphisms and fracture risk: a meta-analysis.Bone 2006;39:938–45.Fletcher O, Johnson N, Palles C, dos Santos Silva I,McCormack V, Whittaker J, et al. Inconsistent associationbetween the STK15 F31I genetic polymorphism andbreast cancer risk. J Natl Cancer Inst 2006;98:1014–8.Gao J, Shan G, Sun B, Thompson PJ, Gao X. Associationbetween polymorphism of tumour necrosis factoralpha-308 gene promoter and asthma: a meta-analysis.Thorax 2006;61:466–71.Healy DG, Abou-Sleiman PM, Casas JP, Ahmadi KR,Lynch T, Gandhi S, et al. UCHL-1 is not a Parkinson’sdisease susceptibility gene. Ann Neurol 2006;59:627–33.Huang X, Chen P, Kaufer DI, Troster AI, Poole C.Apolipoprotein E and dementia in Parkinson disease: ameta-analysis. Arch Neurol 2006;63:189–93.Huang Y, Han S, Li Y, Mao Y, Xie Y. Different rolesof MTHFR C677T and A1298C polymorphisms incolorectal adenoma and colorectal cancer: a metaanalysis.J Hum Genet 2007;52:73–85.Jeong SH, Joo EJ, Ahn YM, Lee KY, Kim YS.Investigation of genetic association between humanFrizzled homolog 3 gene (FZD3) and schizophrenia:results in a Korean population and evidence from metaanalysis.Psychiatry Res 2006;143:1–11.Kendler KS, Baker JH. Genetic influences on measuresof the environment: a systematic review. Psychol Med2007;37:615–26.Koschny R, Holland H, Koschny T, Vitzthum HE.Comparative genomic hybridization pattern of nonanaplasticand anaplastic oligodendrogliomas – a metaanalysis.Pathol Res Pract 2006;202:23–30.179

Appendix 17180Lee SA, Lee KM, Park SK, Choi JY, Kim B, Nam J, etal. Genetic polymorphism of XRCC3 Thr241Met andbreast cancer risk: case-control study in Korean womenand meta-analysis of 12 studies. Breast Cancer Res Treat2007;103:71–6.Lee YH, Rho YH, Choi SJ, Ji JD, Song GG, Nath SK, etal. The PTPN22 C1858T functional polymorphism andautoimmune diseases – a meta-analysis. Rheumatology(Oxford) 2007;46:49–56.Lewis SJ, Lawlor DA, Davey Smith G, Araya R, TimpsonN, Day IN, et al. The thermolabile variant of MTHFR isassociated with depression in the British Women’s Heartand Health Study and a meta-analysis. Mol Psychiatry2006;11:352–60.Li D, He L. Association study of the G-protein signaling4 (RGS4) and proline dehydrogenase (PRODH) geneswith schizophrenia: a meta-analysis. Eur J Hum Genet2006;14:1130–5.Li H, Tai BC. RNASEL gene polymorphisms and therisk of prostate cancer: a meta-analysis. Clin Cancer Res2006;12:5713–9.Marti A, Ochoa MC, Sanchez-Villegas A, MartinezJA, Martinez-Gonzalez MA, Hebebrand J, et al. Metaanalysison the effect of the N363S polymorphism of theglucocorticoid receptor gene (GRL) on human obesity.BMC Med Genet 2006;7:50.Medica I, Kastrin A, Peterlin B. Genetic polymorphismsin vasoactive genes and preeclampsia: a meta-analysis.Eur J Obstet Gynecol Reprod Biol 2007;131:115–26.Nishimura F, Shibasaki M, Ichikawa K, Arinami T,Noguchi E. Failure to find an association betweenCD14–159C/T polymorphism and asthma: a familybasedassociation test and meta-analysis. Allergol Int2006;55:55–8.Noso S, Ikegami H, Fujisawa T, Kawabata Y, Asano K,Hiromine Y, et al. Association of SUMO4, as a candidategene for IDDM5, with susceptibility to type 1 diabetes inAsian populations. Ann N Y Acad Sci 2006;1079:41–6.Philibert RA. A meta-analysis of the association of theHOPA12bp polymorphism and schizophrenia. PsychiatrGenet 2006;16:73–6.Qian L, Zhao J, Shi Y, Zhao X, Feng G, Xu F, et al. Brainderivedneurotrophic factor and risk of schizophrenia: anassociation study and meta-analysis. Biochem Biophys ResCommun 2007;353:738–43.Serretti A, Kato M, De Ronchi D, Kinoshita T. Metaanalysisof serotonin transporter gene promoterpolymorphism (5-HTTLPR) association with selectiveserotonin reuptake inhibitor efficacy in depressedpatients. Mol Psychiatry 2007;12:247–57.Shirts BH, Wood J, Yolken RH, Nimgaonkar VL.Association study of IL10, IL1beta, and IL1RN andschizophrenia using tag SNPs from a comprehensivedatabase: suggestive association with rs16944 at IL1beta.Schizophr Res 2006;88:235–44.Talkowski ME, Seltman H, Bassett AS, Brzustowicz LM,Chen X, Chowdari KV, et al. Evaluation of a susceptibilitygene for schizophrenia: genotype based meta-analysisof RGS4 polymorphisms from thirteen independentsamples. Biol Psychiatry 2006;60:152–62.Tello-Ruiz MK, Curley C, DelMonte T, Giallourakis C,Kirby A, Miller K, et al. Haplotype-based associationanalysis of 56 functional candidate genes in the IBD6locus on chromosome 19. Eur J Hum Genet 2006;14:780–90.Tenesa A, Campbell H, Barnetson R, Porteous M,Dunlop M, Farrington SM. Association of MUTYH andcolorectal cancer. Br J Cancer 2006;95:239–42.Thakkinstian A, Bowe S, McEvoy M, Smith W, Attia J.Association between apolipoprotein E polymorphismsand age-related macular degeneration: A HuGE reviewand meta-analysis. Am J Epidemiol 2006;164:813–22.Tonjes A, Scholz M, Loeffler M, Stumvoll M. Associationof Pro12Ala polymorphism in peroxisome proliferatoractivatedreceptor gamma with Pre-diabetic phenotypes:meta-analysis of 57 studies on nondiabetic individuals.Diabetes Care 2006;29:2489–97.Tripathy CB, Roy N. Meta-analysis of glutathioneS-transferase M1 genotype and risk toward head andneck cancer. Head Neck 2006;28:217–24.Tsantes AE, Nikolopoulos GK, Bagos PG, VaiopoulosG, Travlou A. Lack of association between the plateletglycoprotein Ia C807T gene polymorphism andcoronary artery disease: a meta-analysis. Int J Cardiol2007;118:189–96.Vollmert C, Hahn S, Lamina C, Huth C, Kolz M,Schopfer-Wendels A, et al. Calpain-10 variants andhaplotypes are associated with polycystic ovarysyndrome in Caucasians. Am J Physiol Endocrinol Metab2007;292:E836–44.Wang Z, Wei J, Zhang X, Guo Y, Xu Q, Liu S, et al. Areview and re-evaluation of an association between theNOTCH4 locus and schizophrenia. Am J Med Genet BNeuropsychiatr Genet 2006;141:902–6.Webb EL, Rudd MF, Houlston RS. Case-control, kincohortand meta-analyses provide no support for STK15F31I as a low penetrance colorectal cancer allele. Br JCancer 2006;95:1047–9.Wells PS, Anderson JL, Scarvelis DK, Doucette SP,Gagnon F. Factor XIII Val34Leu variant is protective

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8against venous thromboembolism: a HuGE review andmeta-analysis. Am J Epidemiol 2006;164:101–9.Yang YC, Chang TY, Lee YJ, Su TH, Dang CW, Wu CC,et al. HLA-DRB1 alleles and cervical squamous cellcarcinoma: experimental study and meta-analysis. HumImmunol 2006;67:331–40.Zintzaras E, Kitsios G, Stefanidis I. Endothelial NOsynthase gene polymorphisms and hypertension: a metaanalysis.Hypertension 2006;48:700–10.Reviews that explicitly testedpublication bias (n = 47)Bell CM, Urbach DR, Ray JG, Bayoumi A, Rosen AB,Greenberg D, et al. Bias in published cost effectivenessstudies: systematic review. Bmj 2006;332:699–703.Boodhwani M, Rubens FD, Sellke FW, Mesana TG, RuelM. Mortality and myocardial infarction following surgicalversus percutaneous revascularization of isolated leftanterior descending artery disease: a meta-analysis. Eur JCardiothorac Surg 2006;29:65–70.Chen XC, Xu MT, Zhou W, Han CL, Chen WQ. Ameta-analysis of relationship between beta-fibrinogengene -148C/T polymorphism and susceptibility tocerebral infarction in Han Chinese. Chin Med J (Engl)2007;120:1198–202.Chodosh J, Morton SC, Mojica W, Maglione M, SuttorpMJ, Hilton L, et al. Meta-analysis: chronic disease selfmanagementprograms for older adults. Ann Intern Med2005;143:427–38.Clark EM, Tobias JH, Ness AR. Association between bonedensity and fractures in children: a systematic review andmeta-analysis. Pediatrics 2006;117:e291–7.Cruciani M, Lipsky BA, Mengoli C, de Lalla F. Aregranulocyte colony-stimulating factors beneficial intreating diabetic foot infections?: A meta-analysis.Diabetes Care 2005;28:454–60.Cruciani M, Mengoli C, Malena M, Bosco O, SerpelloniG, Grossi P. Antifungal prophylaxis in liver transplantpatients: a systematic review and meta-analysis. LiverTranspl 2006;12:850–8.Cruz DN, Perazella MA, Bellomo R, de Cal M,Polanco N, Corradi V, et al. Effectiveness of polymyxinB-immobilized fiber column in sepsis: a systematicreview. Crit Care 2007;11:R47.Dherani M, Pope D, Mascarenhas M, Smith KR, WeberM, Bruce N. Indoor air pollution from unprocessed solidfuel use and pneumonia risk in children aged under fiveyears: a systematic review and meta-analysis. Bull WorldHealth Organ 2008;86:390–8C.Doulton TW, He FJ, MacGregor GA. Systematic reviewof combined angiotensin-converting enzyme inhibition© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.and angiotensin receptor blockade in hypertension.Hypertension 2005;45:880–6.Doust JA, Pietrzak E, Dobson A, Glasziou P. How welldoes B-type natriuretic peptide predict death and cardiacevents in patients with heart failure: systematic review.BMJ 2005;330:625.Garcia-Closas M, Malats N, Silverman D, Dosemeci M,Kogevinas M, Hein DW, et al. NAT2 slow acetylation,GSTM1 null genotype, and risk of bladder cancer: resultsfrom the Spanish Bladder Cancer Study and metaanalyses.Lancet 2005;366:649–59.Gautam M, Cheruvattath R, Balan V. Recurrence ofautoimmune liver disease after liver transplantation: asystematic review. Liver Transpl 2006;12:1813–24.Griffin S, Ellis S, Fitzgerald-Barron A, Rose J, Egger M.Nebulised steroid in the treatment of croup: a systematicreview of randomised controlled trials. Br J Gen Pract2000;50:135–41.Hayden JA, van Tulder MW, Malmivaara AV, Koes BW.Meta-analysis: exercise therapy for nonspecific low backpain. Ann Intern Med 2005;142:765–75.Hayden JA, van Tulder MW, Tomlinson G. Systematicreview: strategies for using exercise therapy to improveoutcomes in chronic low back pain. Ann Intern Med2005;142:776–85.Hulten E, Jackson JL, Douglas K, George S, VillinesTC. The effect of early, intensive statin therapy on acutecoronary syndrome: a meta-analysis of randomizedcontrolled trials. Arch Intern Med 2006;166:1814–21.Kasper JS, Giovannucci E. A meta-analysis of diabetesmellitus and the risk of prostate cancer. Cancer EpidemiolBiomarkers Prev 2006;15:2056–62.Li D, Sham PC, Owen MJ, He L. Meta-analysis showssignificant association between dopamine system genesand attention deficit hyperactivity disorder (ADHD).Hum Mol Genet 2006;15:2276–84.Liu T, Zeng D, Zeng C, He X. Association betweenMYOC.mt1 promoter polymorphism and risk of primaryopen-angle glaucoma: a systematic review and metaanalysis.Med Sci Monit 2008;14:RA87–93.Maggard MA, Shugarman LR, Suttorp M, MaglioneM, Sugerman HJ, Livingston EH, et al. Metaanalysis:surgical treatment of obesity. Ann Intern Med2005;142:547–59.Meert AP, Martin B, Delmotte P, Berghmans T, LafitteJJ, Mascaux C, et al. The role of EGF-R expression onpatient survival in lung cancer: a systematic review withmeta-analysis. Eur Respir J 2002;20:975–81.181

Appendix 17Minelli C, Thompson JR, Tobin MD, Abrams KR. Anintegrated approach to the meta-analysis of geneticassociation studies using Mendelian randomization. Am JEpidemiol 2004;160:445–52.Morris RK, Khan KS, Coomarasamy A, Robson SC,Kleijnen J. The value of predicting restriction of fetalgrowth and compromise of its wellbeing: Systematicquantitative overviews (meta-analysis) of test accuracyliterature. BMC Pregnancy Childbirth 2007;7:3.Muradin GS, Bosch JL, Stijnen T, Hunink MG. Balloondilation and stent implantation for treatment offemoropopliteal arterial disease: meta-analysis. Radiology2001;221:137–45.Newman DJ, Mattock MB, Dawnay AB, Kerry S, McGuireA, Yaqoob M, et al. Systematic review on urine albumintesting for early detection of diabetic complications.Health Technol Assess 2005;9:iii–vi, xiii–163.Nowak AK, Stockler MR, Chow PK, Findlay M. Use oftamoxifen in advanced-stage hepatocellular carcinoma. Asystematic review. Cancer 2005;103:1408–14.Ntais C, Polycarpou A, Ioannidis JP. Association of theCYP17 gene polymorphism with the risk of prostatecancer: a meta-analysis. Cancer Epidemiol Biomarkers Prev2003;12:120–6.Orlando LA, Kulasingam SL, Matchar DB. Metaanalysis:the detection of pancreatic malignancy withpositron emission tomography. Aliment Pharmacol Ther2004;20:1063–70.Owen CG, Shah A, Henshaw K, Smeeth L, Sheikh A.Topical treatments for seasonal allergic conjunctivitis:systematic review and meta-analysis of efficacy andeffectiveness. Br J Gen Pract 2004;54:451–6.Owen CG, Whincup PH, Gilg JA, Cook DG. Effectof breast feeding in infancy on blood pressure inlater life: systematic review and meta-analysis. BMJ2003;327:1189–95.Ownby RL, Crocco E, Acevedo A, John V, LoewensteinD. Depression and risk for Alzheimer disease: systematicreview, meta-analysis, and metaregression analysis. ArchGen Psychiatry 2006;63:530–8.Petticrew M, Bell R, Hunter D. Influence of psychologicalcoping on survival and recurrence in people with cancer:systematic review. BMJ 2002;325:1066.Qin LQ, Xu JY, Wang PY, Hoshi K. Soyfood intake inthe prevention of breast cancer risk in women: a metaanalysisof observational epidemiological studies. J NutrSci Vitaminol (Tokyo) 2006;52:428–36.Selvin E, Marinopoulos S, Berkenblit G, Rami T,Brancati FL, Powe NR, et al. Meta-analysis: glycosylatedhemoglobin and cardiovascular disease in diabetesmellitus. Ann Intern Med 2004;141:421–31.Stone J, Sharpe M, Carson A, Lewis SC, Thomas B,Goldbeck R, et al. Are functional motor and sensorysymptoms really more frequent on the left? A systematicreview. J Neurol Neurosurg Psychiatry 2002;73:578–81.Strippoli GF, Navaneethan SD, Johnson DW, PerkovicV, Pellegrini F, Nicolucci A, et al.. Effects of statins inpatients with chronic kidney disease: meta-analysis andmeta-regression of randomised controlled trials. BMJ2008;336:645–51.Tsai AC, Morton SC, Mangione CM, Keeler EB. A metaanalysisof interventions to improve care for chronicillnesses. Am J Manag Care 2005;11:478–88.van Kempen EE, Kruize H, Boshuizen HC, AmelingCB, Staatsen BA, de Hollander AE. The associationbetween noise exposure and blood pressure and ischemicheart disease: a meta-analysis. Environ Health Perspect2002;110:307–17.Van Maele-Fabry G, Willems JL. Occupation relatedpesticide exposure and cancer of the prostate: a metaanalysis.Occup Environ Med 2003;60:634–42.Veglia F, Matullo G, Vineis P. Bulky DNA adductsand risk of cancer: a meta-analysis. Cancer EpidemiolBiomarkers Prev 2003;12:157–60.Wasnich RD, Miller PD. Antifracture efficacy ofantiresorptive agents are related to changes in bonedensity. J Clin Endocrinol Metab 2000;85:231–6.Wellman RJ, Sugarman DB, DiFranza JR, Winickoff JP.The extent to which tobacco marketing and tobacco usein films contribute to children’s use of tobacco: a metaanalysis.Arch Pediatr Adolesc Med 2006;160:1285–96.Winkley K, Ismail K, Landau S, Eisler I. Psychologicalinterventions to improve glycaemic control in patientswith type 1 diabetes: systematic review and meta-analysisof randomised controlled trials. BMJ 2006;333:65.Zafarmand MH, van der Schouw YT, Grobbee DE, deLeeuw PW, Bots ML. The M235T polymorphism in theAGT gene and CHD risk: evidence of a Hardy-Weinbergequilibrium violation and publication bias in a metaanalysis.PLoS ONE 2008;3:e2533.Zheng M, Bai J, Yuan B, Lin F, You J, Lu M, et al. Metaanalysisof prophylactic corticosteroid use in post-ERCPpancreatitis. BMC Gastroenterology 2008;8:6.182Schernhammer ES, Colditz GA. Suicide rates amongphysicians: a quantitative and gender assessment (metaanalysis).Am J Psychiatry 2004;161:2295–302.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 18Original study proposal183© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 18II. 0BDETAILS OF PROPOSED RESEARCHDetailed outline of proposed research (see attached notes for guidance).Title:RM05/JH29: Dissemination bias in medical and health related research - an updated synthesisof empirical evidence and a critical assessment of available methodsBackgroundSynthesis of published research is becoming increasingly important in providing relevant and valid researchevidence to clinical and health policy decision making. However, the validity of research synthesis based onpublished literature will be threatened if published studies comprise a biased selection of all studies that havebeen conducted.A previous HTA monograph published in 2000 systematically reviewed studies that provided empiricalevidence on publication and related biases, and studies that developed or tested methods for preventing,reducing, or detecting publication and related biases. 1 The review found evidence indicating that studies withsignificant or favourable results were more likely to be published, or were likely to be published earlier thanthose with non-significant results. There was limited and indirect evidence indicating the possibility of fullpublication bias, outcome reporting bias, duplicate publication bias, and language bias. The review identifiedlittle empirical evidence relating to the impact of publication and related biases on health policy, clinicaldecision making and the outcome of patient management. Considering that the spectrum of the accessibilityof research results (dissemination profile) ranges from completely inaccessible to easily accessible, it wassuggested that a single term “dissemination bias” could be used to denote all types of publication and relatedbiases. 1In the previous HTA report, the available methods for dealing with dissemination biases were classifiedaccording to measures that could be taken before, during or after a literature review: to prevent publicationbias before a literature review (eg, prospective registration of trials), to reduce or detect publication andrelated biases during a literature review (eg, locating grey literature or unpublished studies, and funnel plotrelated methods), and to minimise the impact of publication bias after a literature review (eg, confirmatorylarge scale trials, updating systematic reviews). 1 It was concluded that the ideal solution to publication bias isthe prospective, universal registration of all studies at their inception. It was concluded, although debatable,that available statistical methods for detecting and adjusting publication bias should be mainly used for thepurpose of sensitivity analysis.Since the publication of the HTA review of publication and related biases, many new studies on publicationand related biases have been published. For example, Egger et al (2003) provided further empirical evidenceon publication bias, language bias, grey literature bias, and MEDLINE index bias, 2 and Moher et al (2003)evaluated language bias in meta-analyses of randomised controlled trials. 3 Recently, more convincingevidence on outcome reporting bias has been published. 4,5 The new empirical evidence may contradict orstrengthen the empirical evidence included in the previous HTA report. Funnel plot and related statisticalmethods have been applied in new studies to collections of meta-analyses to estimate possible publicationbias in systematic reviews. 6-8 There are also new published studies that investigated methods for dealingwith publication bias (for example 9-11 ).Dubben and Beck-Bornholdt (2005) used the funnel plot approach, and found no evidence of publication biasin studies of publication bias. 12 They acknowledged that the analysis was handicapped by insufficient power(with only 26 included studies) and also by the diverse definitions of publication bias in the primary studies.However, Song et al pointed out that the study had other, more important, limitations so that disseminationbias of studies on publication bias could not be safely excluded. 13Purpose (aims, objectives)1. To identify all relevant empirical studies published since 1998. Empirical studies are defined as those thatprovide empirical evidence on the existence, consequences, causes and/or risk factors of dissemination bias.2. To identify all relevant methodological studies published since 1998. Methodological studies are those thathave developed or investigated methods for preventing, reducing or detecting dissemination bias.3. To categorise empirical and methodological studies identified according to a conceptual framework ofdissemination profile, and to critically appraise studies that provided direct empirical evidence.- 1 –184

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 84. To synthesise findings from newly identified and previously included studies to enable us to assesswhether each type of dissemination bias does exist, and if so the extent of the effect that it may have onresults of systematic reviews and hence decision making.5. To assess the possibility of dissemination bias of studies that provide empirical evidences on disseminationbias.6. To assess the usefulness and limitations of available methods, and resources required to use thesemethods to combat each type of dissemination bias, through synthesis of the methodological studies.7. To examine measures taken in a representative sample of published systematic reviews to prevent, reduceand detect different types of dissemination bias. We will include both narrative and quantitative (metaanalytic)systematic reviews that evaluated effect of healthcare interventions, systematic reviews thatevaluated the accuracy of diagnostic tests, and systematic reviews of epidemiological studies that evaluatedassociation of risk factors and health outcomes.8. To bring together current evidence on the existence and scale of each type of dissemination bias, effectsand costs of methods to combat these biases, and current use of these methods to create recommendationsfor reviewers, policy makers, health professionals and service users, and to disseminate theserecommendations.Investigation methodsThe review contains three parts: (1) review of empirical and methodological studies; (2) an assessment ofpublished systematic reviews; (3) synthesising all findings, providing and disseminating recommendations.Part 1. Review of empirical and methodological studiesMethods used in the previous HTA report will be modified to identify and categorise relevant studies. We willadopt a new framework to categorise relevant studies, and important studies will be assessed using a morecritical and structured approach. Details of the review methods are described below and in the Figure.UCriteria for inclusion and literature search strategiesA preliminary literature search indicated that there are a large number of potentially relevant studies in fieldsof medical and health related research, and searches in the area of social sciences produced few studies inthe initial review, so we plan to focus on dissemination bias in health and related research. We will includestudies that provide empirical evidence on the existence, consequences, causes, and/or risk factors of typesof dissemination bias; and studies that develop or evaluate methods for preventing, reducing or detectingdissemination bias.Literature searches for methodological studies are often difficult because of ill-defined boundaries andinappropriate indexing in commonly used bibliographic databases. 14 Our previous experience and initialsearching suggests that the most productive and efficient methods include searching the CochraneMethodology Register, references of retrieved articles and citation search of key studies.We will search the Cochrane Methodology Register (CRM) and MEDLINE for relevant empirical andmethodological studies published since 1998. We will compare studies identified from the MEDLINE andthose identified from the Cochrane Methodology Register, to check the completeness and usefulness of thetwo bibliographic databases for methodological reviews. Key words used in the search of electronicdatabases will include: publication bias, dissemination bias, language bias, national bias, country bias,reporting bias, grey literature bias, conference/abstract bias, full publication bias, citation bias, time lag/delaybias, reference bias, selection bias, location bias, duplication or multiple publication bias, database bias, indexbias, and file drawer. References (titles with or without abstracts) gathered by searching the CMR andMEDLINE will be assessed independently by two reviewers for inclusion or exclusion. Any disagreement willbe discussed.We will also search EMBASE (from 2005 to 2007, as EMBASE is searched for the Methodology Register, butwe will ensure that we have included the most recent references), Ahmed (1998-2007), Cinahl (1998-2007),PsychInfo (1998-2007), SIGLE (1998-2007) and Dissertation Abstracts (1998-2007) for any additionalrelevant studies. Searching of EMBASE, Ahmed, Cinahl, PsychInfor, SIGLE, and Dissertation Abstracts willbe conducted by one reviewer. References of retrieved reviews and studies will be examined by onereviewer to identify additional relevant studies, including any relevant studies published before 1998 but- 2 –185© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 18missed in the previous HTA report. Citations of the key studies will also be searched. The literature searchwill not be restricted by publication language.We have conducted a preliminary search of the Cochrane Methodology Register for relevant studies thatpublished since 1998. According to titles (with or without abstracts) of identified references, there are a largenumber of possibly relevant studies (300-400, after excluding obvious duplicates). More than 200 referenceswere empirical studies, including publication bias (n=26), publication of conference abstracts (n=66), outcomereporting bias (n=26), country or language bias (n=39), grey literature (n=13), time lag bias (n=15), causes ofpublication bias (n=16), citation bias (n=16). We found 58 studies of methods for dealing with publicationbias, including 30 studies of statistical methods, 17 studies of literature search methods, and 11 studies ofother methods (eg, trial registration or large scale confirmation trials).UClassification of identified studiesAccording to findings from our preliminary literature search, relevant studies are numerous in quantity andsubstantially diverse in quality. It is crucial to classify identified studies using a pre-specified structure tofacilitate subsequent assessment and synthesis (Figure).First , one reviewer will classify identified studies as (1) evidence studies or (2) methodological studies.Evidence studies are defined as studies that provide empirical evidence on the existence, extent,consequences, causes or risk factors of dissemination bias. Methodological studies are defined as those thatdevelop or investigate methods for preventing, reducing or detecting dissemination bias. Some studies maybe classified as both an evidence and a methods study.UReview of evidence studiesEvidence studies will be categorised into various types of dissemination bias, according to a framework ofdissemination profile (that is, accessibility of research results): non-publication (never, or delayed); incompletepublication (e.g. biased outcome reporting, data dredged subgroup effects, biased full publication ofconference abstracts); published but difficult to access (e.g. grey literature, language bias, database bias);other biased dissemination activities (e.g. citation bias, duplicate bias). It is possible that some studies maybe included in more than one category.Then evidence studies will be further separated into two groups: studies that provided direct evidence, andstudies that provided indirect evidence. Direct evidence refers to data or observations that could be used todirectly indicate dissemination bias, including admissions of bias on the part of those involved in thepublication process, comparison of the results of published and unpublished studies, and the prospective orretrospective follow-up of dissemination profile of cohorts of studies. Indirect evidence refers to observationsthat could be explained indirectly by dissemination bias but other alternative explanations could not beexcluded. For instance, a disproportionately high proportion of positive findings in the published literaturemight provide indirect evidence, as might larger effect sizes in smaller studies compared with larger studies.We will apply a checklist of quality assessment to critically appraise studies that provided direct empiricalevidence and studies that assessed association between sample size and effects in multiple meta-analyses,with regard to scientific rigorousness, the sample’s representativeness, and appropriateness of data analysesand interpretation. Topic specific items will be considered if judged appropriate for different types of bias.More details about the proposed quality assessment are described below.• Scientific rigorousness: We aim to detect potential threats to the validity of study results. Forexample, prospective studies are more valid than retrospective studies. Selection and inclusion ofsamples may be more or less biased, and whether assessments and judgements were independentlyduplicated.• Generalisability: It is important to consider whether results of studies could be generalisable todifferent fields of research, settings, and designs. For example, dissemination bias in randomisedtrials may not be similar to that in epidemiological studies.• Appropriateness of data analysis: We will assess whether the data analysis method is appropriateto address the objectives of the study.• Appropriateness of interpretation: Limitations of the study should be considered when results of astudy were interpreted.Relevant studies included in the previous HTA report 1 will also be critically appraised using the samechecklist.186- 3 –

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Using a standardised appraisal form (Appendix 1), categorisation of all evidence studies and critical appraisalof selected studies will be independently carried out by two reviewers. Disagreements will be resolved bydiscussion.Figure. Classification of identified relevant studiesStudies identifiedDisseminationprofileUNon-publicationU:Evidence studiesCauses,Risk factorsConsequences- publication bias- time delay bias …UIncomplete publicationU:- outcome reporting- abstract bias …ULimited accessibilityU:- grey literature bias- language bias- database bias …ExistenceUOther biased disseminationU:- citation bias- duplicate bias- media attention bias …Asymmetrical funnel plots in multiplemeta-analysesMethod studiesStage of literature review---------------------------------------------------------- Otherpre- during- post-- Prospective study registration- Research publication policy and process: freedom ofinformation, research policy, journal editorial and peerreviewing process, electronic publications- …- Comprehensive literature search: identifying unpublished,grey literature, no language restriction, …- Funnel plot and related methods- Other statistical and modelling methods- Detecting publication bias in narrative reviews- Updating systematic reviews- Confirmatory studies- Updating systematic reviews- Scepticism about research findings- …Any other methods/approachesDirect evidence & studies ofasymmetrical funnel plot inmultiple meta-analysesOther indirectevidenceUTo be critically appraised in terms of :- scientific rigorousness- sample representativeness- appropriateness of data analysis- appropriateness of interpretationA brief, andmostlynarrativesummary offindingsUMethods to be critically appraised interms of :- underlying assumptions- conditions in which to be used- usefulness- limitations- resource required- 4 –187© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 18UAssessment of dissemination bias in empirical evidence studiesUIn this review, we will assess whether dissemination bias is also a problem for studies that provide empiricalevidence on dissemination bias. Results of included empirical evidence studies will be independentlycategorised by two reviewers as positive (significant dissemination bias), non-significant (no cleardissemination bias), or can’t tell (see Appendix 1). Then we will examine the association of the results ofempirical studies and studies’ quality and dissemination profile (including time of publication, journal impactfactor, number of citations after a given period of publication, and study’s impact onguidelines/recommendations for systematic reviews).UReview of methodological studiesThere may be multiple studies investigating the same method. Method studies will be categorised accordingto methods they investigated, to generate a list of available methods and corresponding studies identified.Then each method will be cross-classified from two aspects: (1) type of dissemination bias and (2) stage ofliterature review (see Figure). It is possible that the same method may be relevant to different types ofdissemination bias or applicable to the different stage of a literature review. Using a standardised methodclassification sheet (Appendix 2), the review of method studies will be conducted by one reviewer andchecked by a second reviewer. Disagreements will be resolved by discussion.Based on findings of included studies, available methods will be critically appraised in terms of underlyingassumptions, conditions under which the method could be used, usefulness, limitations, and resourcerequired.UPresentation and summary of literature review findingsData extracted from the included studies and results of critically appraisal will be presented in tables anddescribed narratively. If appropriate, results from individual studies will be quantitatively pooled (for example,pooled odds ratio of full publication of conference abstracts with statistically significant results versus thosewith statistically non-significant results). Results of critical appraisal will be taken into consideration tointerpret and explain findings from empirical and methodological studies. We will highlight whether findingsfrom studies newly identified contradict or strengthen findings from studies included in the previous HTAreport.Part 2. Assessment of a sample of published systematic reviewsIn the previous HTA report, 193 systematic reviews taken from the Database of Abstract of Reviews ofEffectiveness (Centre for Reviews and Dissemination, University of York) were used to identify furtherevidence of dissemination bias and to illustrate the methods used in systematic reviews for dealing withpublication bias. However, there are several shortcomings in the previous assessment. First, systematicreviews included in the DARE database might on average have better quality than those from the generalbibliographic databases (such as MEDLINE) so that the representativeness of systematic reviews assessed inthe previous HTA report may be questionable. Secondly, 91% of systematic reviews that evaluated theeffectiveness of healthcare interventions and 9% that evaluated the accuracy of diagnostic technologies werenot separately assessed. The problem of dissemination bias might be different between the two types ofsystematic reviews. Thirdly, systematic reviews of epidemiological studies of association between risk factorsand health outcomes were not included in the previous HTA report.To overcome these shortcomings, we plan to obtain a representative sample of systematic reviews from thegeneral bibliographic database MEDLINE, including (1) systematic reviews of studies on effects of healthcareinterventions, (2) systematic reviews of studies on accuracy of diagnostic tests, and (3) systematic reviews ofepidemiological studies on association between risk factors and health outcomes.A preliminary search of MEDLINE using “systematic review” or “meta-analysis” (in titles or inabstracts) identified 2779 English-language references published in 2005. We examined the first 300of the 2779 references and identified 109 systematic reviews that evaluated effects of healthcareinterventions (including preventive interventions), 13 systematic reviews of studies of diagnostic tests,and 53 systematic reviews of epidemiological studies (including 18 systematic reviews of geneticstudies). This preliminary exercise indicates that there are about 1009 systematic review of effects ofhealth interventions, about 120 systematic reviews of diagnostic tests, and about 490 systematicreviews of epidemiological studies. The following approach is based on findings from this preliminarywork.188 - 5 –

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8UIdentifying and sampling systematic reviewsFirst, we will search MEDLINE for systematic reviews published in 2005. References identified fromMEDLINE will be examined by one reviewer and categorised as systematic reviews of effects of healthcareinterventions, systematic reviews of accuracy of diagnostic tests, or systematic reviews of epidemiologicalstudies (genetic epidemiology or not). Then we will use computer-generated random numbers to obtain arandom sample of 100 systematic reviews of effects of healthcare interventions, 50 systematic reviews ofaccuracy of diagnostic tests, and 100 systematic reviews of epidemiological studies (of which 50 will bereviews of genetic epidemiology studies) from all identified systematic reviews.UExtracting data from included systematic reviewsThe data extraction from included systematic reviews will be independently conducted by two reviewers tocollect the following information (see preliminary data extraction sheet in Appendix 3): type of review (effect,diagnostic, epidemiological), method of data synthesis (narrative or quantitative), whether the issue ofpublication bias was considered, whether unpublished studies or those published in non-English languageswere searched for and included; methods used for dealing with publication bias; any evidence on theexistence, extent and consequence of publication bias.A checklist will also be applied independently by two reviewers to assess the overall quality of includedsystematic reviews (see Appendix 3). Any disagreements between the two reviewers will be resolved bydiscussion.UAnalysing data from included systematic reviewsData extracted from systematic reviews of effects of healthcare interventions, systematic reviews of accuracyof diagnostic test, and systematic reviews of epidemiological studies will be separately presented andcompared. We will also examine the subgroup of genetic epidemiology reviews separately. We will comparefindings from narrative systematic reviews and quantitative systematic reviews (meta-analyses). Systematicreviews of effects of healthcare interventions and systematic reviews of diagnostic accuracy published in 2005will be compared with those included in the previous HTA report to examine whether the reporting andtreatment of dissemination bias has improved over time.Part 3. Synthesising findings from Part 1 and Part 2Findings from Part 1 will illuminate the existence or otherwise, extent and potential impact on policy ofdifferent types of dissemination bias, and suggest a range of methods for dealing with such biases. Part 2 willprovide findings about what actually happens in the practice of systematic reviews. Part 3 aims to comparefindings from Part 1 and Part 2, and to identify gaps between empirical and methodological research ondissemination bias, and actual practice of systematic reviews (see the proposed summary table below). Forexample, considerable resource might be wasted in systematic reviews identifying, translating, and assessingstudies published in languages other than English if evidence suggests that language bias is not a problem.Some statistical methods developed may have rarely been used in practice for various reasons (eg, toocomplicated or no additional advantages as compared with simple methods).Bias categoryEvidence ofexistenceMethods tocombat this biasUsefulness andlimitations of themethodResourcesrequired to usethe methodType of disseminationbiasBased onliterature reviewBased onliterature reviewEffect of themethods, as wellas limitations,based onliterature reviewand a sample ofsystematicreviewsIncluding time,staff, andcosts. Basedon literaturereview andexperience- 6 –189© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 18Main references1. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess 2000;4:1-115.2. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and theassessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7:1-76.3. Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages otherthan English in systematic reviews. Health Technology Assessment 2003;7:1-90.4. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomesin randomized trials: comparison of protocols to published articles. Jama 2004;291:2457-65.5. Chan AW,.Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications andsurvey of authors. BMJ 2005;330:753.6. Song F, Khan KS, Dinnes J, Sutton AJ. Asymmetric funnel plots and publication bias in meta-analyses of diagnosticaccuracy. Int J Epidemiol 2002;31:88-95.7. Sterne J, Jüni P, Schulz K, Altman D, Bartlett C, Egger M. Statistical methods for assessing the influence of studycharacteristics on treatment effects in 'meta-epidemiological' research. Statistics in Medicine 2002;21:1513-24.8. Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR. Empirical assessment of effect of publication bias on metaanalyses.BMJ 2000;320:1574-7.9. Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in the presence of heterogeneity. Stat.Med2003;22:2113-26.10. Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematicreviews of diagnostic test accuracy was assessed. J Clin Epidemiol 2005;58:882-93.11. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of Two Methods to Detect Publication Bias in Metaanalysis.JAMA: The Journal of the American Medical Association 2006;295:676-80.12. Dubben HH,.Beck-Bornholdt HP. Systematic review of publication bias in studies on publication bias. BMJ 2005.13. Song F. Review of publication bias in studies on publication bias: studies on publication bias are probably susceptible tothe bias they study. BMJ 2005;331:637-8.14. Edwards S, Lilford RJ, Kiauka S. Different types of systematic review in health services research. In Black N, Brazier J,Fitzpatrick R, Reeves B, eds. Health services research methods: a guide to best practice, pp 255-9. London: BMJ Books,1998.- 7 –190

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 1. Data Extraction Sheet - Empirical Evidence StudiesPreliminary sheet, will be modified by pilot testingAuthor (year): _______________ Source: ___________________________Title: __________________________________________________________Reviewer: _______-----------------------------------------------------------------------------------------------Study design & objectives:-----------------------------------------------------------------------------------------------Issues: Existence/identifying Causes/risk factors Consequence Other: ___________Categories: Non-publication Incomplete publication Limited accessibility Other: _____________Specific bias: Publication bias Grey literature bias Language bias Reporting bia Abstract bias Time delay bias Database index bias Citation bias Duplicate bias Media attention bias Other: ________________Areas: general health specific health (eg, obesity): other: _____________________-----------------------------------------------------------------------------------------------Study results: Significant/important bias Non-significant Can’t tellDetails:-----------------------------------------------------------------------------------------------Original authors’ conclusions:-----------------------------------------------------------------------------------------------Evidence: Direct IndirectFor indirect evidence, stop. For studies with direct evidence, continue:-----------------------------------------------------------------------------------------------Scientific rigorousness (hints: prospective or retrospective, sample selection bias)---------------------------------------------------------------------------------------------------------------------Sample representativeness (hints: research field, participants, outcomes, interventions; study designs)----------------------------------------------------------------------------------------------------------------------Appropriateness of data analysis (hints: consider objectives, available data and methods of data analysis)----------------------------------------------------------------------------------------------------------------------Appropriateness of interpretations (hints: limitations of the study should be taken into consideration)-----------------------------------------------------------------------------------------------------------------------Overall study quality: High (hint: without considerable concern on study validity) Moderate (hint: with some concern on study validity) Low (hint: with considerable concern on study validity) Can’t tell------------------------------------------------------------------------------------------------------------------------Any other comments:- 8 –191© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Appendix 18Appendix 2. Data Extraction Sheet – Methodological StudiesPreliminary sheet, will be modified by pilot testingAuthor (year): _______________ Source: ___________________________ Reviewer: _________Title: __________________________________________________________Study design:---------------------------------------------------------------------------------------------------------------------Study objectives: New method, Established method, Evidence of usefulness/limitations---------------------------------------------------------------------------------------------------------------------Methods: Study registration Literature search Funnel plot Statistical/modelling Updating reviews Publication process Research ethics/policy Confirmatory studies other: _____________---------------------------------------------------------------------------------------------------------------------Purpose: Preventing bias Reducing bias, Detecting bias, Adjusting bias, other: _____________--------------------------------------------------------------------------------------------------------------------Stage of literature review: Before literature review, In literature review, After literature review---------------------------------------------------------------------------------------------------------------------What dissemination bias the method is relevant: Publication bias Grey literature bias Language bias Reporting bias Abstract bias Time delay bias Database index bias Citation bias Duplicate bias Media attention bias Other: ________________----------------------------------------------------------------------------------------------------------------------Main findings and conclusions:-----------------------------------------------------------------------------------------------------------------------Resources required to use the method:------------------------------------------------------------------------------------------------------------------------Reviewer’s commentary (study’s validity, scientific rigorousness, method’s usefulness and limitations,any empirical evidence provided):192- 9 –

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Appendix 3. Data Extraction Sheet – Systematic reviewsPreliminary sheet, will be modified by pilot testingAuthor (year): _____________ Source: __________________________________________________ReviewerU: _______Objectives: □ Effectiveness /adverse effects □. Diagnostic □. Genetic epidemiology□. Other epidemiologyType of reviews: □. Narrative □. Meta-analysisDesigns of included studies: □ RCTs/CCTs (Study= ; patients= )□ Diagnostic accuracy studies (Study= ; patients= )□ Epidemiological studies (Study= ; patients= )□ Other__________ (Study= ; patients= )How were differences between studies investigated?□ NA □ Narrative □ Statistical □ Meta-regression □ Sensitivity/subgrouup □ OtherAuthors’ conclusion:□ Significant/positive: At least one intervention recommended; or sig. difference found between interventions.□ Non-sig./not important: No intervention is recommended, or no sig. differences found among interventions.□ Unclear:No able to judge; neither positive nor negative; lack of evidence.Sources searched to identify studies:□ Not stated□ MEDLINE □ EMBASE □ Psychlit □ Cochrane □ Bibliographies□ Handsearch □ Experts/authors □ Company □ Proceedings□ Other: ________________________________Non-English language studies:□ Unclear□ Searched Yes No If yes, search methods:□ Identified Yes No How many?□ Included Yes No If included, a). for main analysis b). for sensitivity analysis?Unpublished studies:□ Unclear□ Searched Yes No If yes, search methods:□ Identified Yes No How many?□ Included Yes No If included, a). for main analysis b). for sensitivity analysis?Issue of publication bias discussed? □ No □ YesMethods used for dealing with publication bias:□ Not used□ Identify unpublished studies□ Prospective register □ Fail-safe N□ Funnel plot□ Rank correlation□ Egger’s method□ Large scale trials□ Modelling□ Other: ___________Details:Evidence on publication bias □ Not available □ Available, If available, details(such as, results of published trials versus unpublished trials; or shape of Funnel plot or related methods)---------------------------------------------------------------------------------------------------------------------------------------------------------Systematic review’s overall quality:1. Well defined review question □ Yes □ Partially □ No □ Can’t tell2. Identification of all relevant studies □ Yes □ Partially □ No □ Can’t tell3. Appropriate assessment of study quality □ Yes □ Partially □ No □ Can’t tell4. Reliable and accurate data extraction □ Yes □ Partially □ No □ Can’t tell5. Appropriate investigation of heterogeneity □ Yes □ Partially □ No □ Can’t tell6. Appropriate data synthesis □ Yes □ Partially □ No □ Can’t tell7. Appropriate interpretation of results □ Yes □ Partially □ No □ Can’t tell- 10 –193© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Volume 1, 1997Health Technology Assessment reportspublished to dateNo. 1Home parenteral nutrition: a systematicreview.By Richards DM, Deeks JJ, SheldonTA, Shaffer JL.No. 2Diagnosis, management and screeningof early localised prostate cancer.A review by Selley S, Donovan J,Faulkner A, Coast J, Gillatt D.No. 3The diagnosis, management, treatmentand costs of prostate cancer in Englandand Wales.A review by Chamberlain J, Melia J,Moss S, Brown J.No. 4Screening for fragile X syndrome.A review by Murray J, Cuckle H,Taylor G, Hewison J.No. 5A review of near patient testing inprimary care.By Hobbs FDR, Delaney BC,Fitzmaurice DA, Wilson S, Hyde CJ,Thorpe GH, et al.No. 6Systematic review of outpatient servicesfor chronic pain control.By McQuay HJ, Moore RA, EcclestonC, Morley S, de C Williams AC.No. 7Neonatal screening for inborn errors ofmetabolism: cost, yield and outcome.A review by Pollitt RJ, Green A,McCabe CJ, Booth A, Cooper NJ,Leonard JV, et al.No. 8Preschool vision screening.A review by Snowdon SK,Stewart-Brown SL.No. 9Implications of socio-cultural contextsfor the ethics of clinical trials.A review by Ashcroft RE, ChadwickDW, Clark SRL, Edwards RHT, Frith L,Hutton JL.No. 10A critical review of the role of neonatalhearing screening in the detection ofcongenital hearing impairment.By Davis A, Bamford J, Wilson I,Ramkalawan T, Forshaw M, Wright S.No. 11Newborn screening for inborn errors ofmetabolism: a systematic review.By Seymour CA, Thomason MJ,Chalmers RA, Addison GM, Bain MD,Cockburn F, et al.No. 12Routine preoperative testing: asystematic review of the evidence.By Munro J, Booth A, Nicholl J.No. 13Systematic review of the effectiveness oflaxatives in the elderly.By Petticrew M, Watt I, Sheldon T.No. 14When and how to assess fast-changingtechnologies: a comparative study ofmedical applications of four generictechnologies.A review by Mowatt G, Bower DJ,Brebner JA, Cairns JA, Grant AM,McKee L.Volume 2, 1998No. 1Antenatal screening for Down’ssyndrome.A review by Wald NJ, Kennard A,Hackshaw A, McGuire A.No. 2Screening for ovarian cancer: asystematic review.By Bell R, Petticrew M, Luengo S,Sheldon TA.No. 3Consensus development methods,and their use in clinical guidelinedevelopment.A review by Murphy MK, Black NA,Lamping DL, McKee CM, SandersonCFB, Askham J, et al.No. 4A cost–utility analysis of interferon betafor multiple sclerosis.By Parkin D, McNamee P, Jacoby A,Miller P, Thomas S, Bates D.No. 5Effectiveness and efficiency of methodsof dialysis therapy for end-stage renaldisease: systematic reviews.By MacLeod A, Grant A, DonaldsonC, Khan I, Campbell M, Daly C, et al.No. 6Effectiveness of hip prostheses inprimary total hip replacement: a criticalreview of evidence and an economicmodel.By Faulkner A, Kennedy LG, BaxterK, Donovan J, Wilkinson M, Bevan G.No. 7Antimicrobial prophylaxis in colorectalsurgery: a systematic review ofrandomised controlled trials.By Song F, Glenny AM.No. 8Bone marrow and peripheralblood stem cell transplantation formalignancy.A review by Johnson PWM,Simnett SJ, Sweetenham JW, Morgan GJ,Stewart LA.No. 9Screening for speech and languagedelay: a systematic review of theliterature.By Law J, Boyle J, Harris F,Harkness A, Nye C.No. 10Resource allocation for chronicstable angina: a systematicreview of effectiveness, costs andcost-effectiveness of alternativeinterventions.By Sculpher MJ, Petticrew M,Kelland JL, Elliott RA, Holdright DR,Buxton MJ.No. 11Detection, adherence and control ofhypertension for the prevention ofstroke: a systematic review.By Ebrahim S.No. 12Postoperative analgesia and vomiting,with special reference to day-casesurgery: a systematic review.By McQuay HJ, Moore RA.No. 13Choosing between randomised andnonrandomised studies: a systematicreview.By Britton A, McKee M, Black N,McPherson K, Sanderson C, Bain C.No. 14Evaluating patient-based outcomemeasures for use in clinical trials.A review by Fitzpatrick R, Davey C,Buxton MJ, Jones DR.195© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to date196No. 15Ethical issues in the design and conductof randomised controlled trials.A review by Edwards SJL, Lilford RJ,Braunholtz DA, Jackson JC, Hewison J,Thornton J.No. 16Qualitative research methods in healthtechnology assessment: a review of theliterature.By Murphy E, Dingwall R,Greatbatch D, Parker S, Watson P.No. 17The costs and benefits of paramedicskills in pre-hospital trauma care.By Nicholl J, Hughes S, Dixon S,Turner J, Yates D.No. 18Systematic review of endoscopicultrasound in gastro-oesophagealcancer.By Harris KM, Kelly S, Berry E,Hutton J, Roderick P, Cullingworth J,et al.No. 19Systematic reviews of trials and otherstudies.By Sutton AJ, Abrams KR, Jones DR,Sheldon TA, Song F.No. 20Primary total hip replacement surgery:a systematic review of outcomesand modelling of cost-effectivenessassociated with different prostheses.A review by Fitzpatrick R, ShortallE, Sculpher M, Murray D, Morris R,Lodge M, et al.Volume 3, 1999No. 1Informed decision making: anannotated bibliography and systematicreview.By Bekker H, Thornton JG,Airey CM, Connelly JB, Hewison J,Robinson MB, et al.No. 2Handling uncertainty when performingeconomic evaluation of healthcareinterventions.A review by Briggs AH, Gray AM.No. 3The role of expectancies in the placeboeffect and their use in the delivery ofhealth care: a systematic review.By Crow R, Gage H, Hampson S,Hart J, Kimber A, Thomas H.No. 4A randomised controlled trial ofdifferent approaches to universalantenatal HIV testing: uptake andacceptability. Annex: Antenatal HIVtesting – assessment of a routinevoluntary approach.By Simpson WM, Johnstone FD,Boyd FM, Goldberg DJ, Hart GJ,Gormley SM, et al.No. 5Methods for evaluating area-wide andorganisation-based interventions inhealth and health care: a systematicreview.By Ukoumunne OC, Gulliford MC,Chinn S, Sterne JAC, Burney PGJ.No. 6Assessing the costs of healthcaretechnologies in clinical trials.A review by Johnston K, Buxton MJ,Jones DR, Fitzpatrick R.No. 7Cooperatives and their primary careemergency centres: organisation andimpact.By Hallam L, Henthorne K.No. 8Screening for cystic fibrosis.A review by Murray J, Cuckle H,Taylor G, Littlewood J, Hewison J.No. 9A review of the use of health statusmeasures in economic evaluation.By Brazier J, Deverill M, Green C,Harper R, Booth A.No. 10Methods for the analysis of qualityof-lifeand survival data in healthtechnology assessment.A review by Billingham LJ,Abrams KR, Jones DR.No. 11Antenatal and neonatalhaemoglobinopathy screening in theUK: review and economic analysis.By Zeuner D, Ades AE, Karnon J,Brown J, Dezateux C, Anionwu EN.No. 12Assessing the quality of reports ofrandomised trials: implications for theconduct of meta-analyses.A review by Moher D, Cook DJ,Jadad AR, Tugwell P, Moher M,Jones A, et al.No. 13‘Early warning systems’ for identifyingnew healthcare technologies.By Robert G, Stevens A, Gabbay J.No. 14A systematic review of the role ofhuman papillomavirus testing within acervical screening programme.By Cuzick J, Sasieni P, Davies P,Adams J, Normand C, Frater A, et al.No. 15Near patient testing in diabetes clinics:appraising the costs and outcomes.By Grieve R, Beech R, Vincent J,Mazurkiewicz J.No. 16Positron emission tomography:establishing priorities for healthtechnology assessment.A review by Robert G, Milne R.No. 17 (Pt 1)The debridement of chronic wounds: asystematic review.By Bradley M, Cullum N, Sheldon T.No. 17 (Pt 2)Systematic reviews of wound caremanagement: (2) Dressings and topicalagents used in the healing of chronicwounds.By Bradley M, Cullum N, Nelson EA,Petticrew M, Sheldon T, Torgerson D.No. 18A systematic literature review ofspiral and electron beam computedtomography: with particular referenceto clinical applications in hepaticlesions, pulmonary embolus andcoronary artery disease.By Berry E, Kelly S, Hutton J,Harris KM, Roderick P, Boyce JC, et al.No. 19What role for statins? A review andeconomic model.By Ebrahim S, Davey SmithG, McCabe C, Payne N, Pickin M,Sheldon TA, et al.No. 20Factors that limit the quality, numberand progress of randomised controlledtrials.A review by Prescott RJ, Counsell CE,Gillespie WJ, Grant AM, Russell IT,Kiauka S, et al.No. 21Antimicrobial prophylaxis in total hipreplacement: a systematic review.By Glenny AM, Song F.No. 22Health promoting schools and healthpromotion in schools: two systematicreviews.By Lister-Sharp D, Chapman S,Stewart-Brown S, Sowden A.No. 23Economic evaluation of a primarycare-based education programme forpatients with osteoarthritis of the knee.A review by Lord J, Victor C,Littlejohns P, Ross FM, Axford JS.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Volume 4, 2000No. 1The estimation of marginal timepreference in a UK-wide sample(TEMPUS) project.A review by Cairns JA,van der Pol MM.No. 2Geriatric rehabilitation followingfractures in older people: a systematicreview.By Cameron I, Crotty M, Currie C,Finnegan T, Gillespie L, Gillespie W,et al.No. 3Screening for sickle cell disease andthalassaemia: a systematic review withsupplementary research.By Davies SC, Cronin E, Gill M,Greengross P, Hickman M, Normand C.No. 4Community provision of hearing aidsand related audiology services.A review by Reeves DJ, Alborz A,Hickson FS, Bamford JM.No. 5False-negative results in screeningprogrammes: systematic review ofimpact and implications.By Petticrew MP, Sowden AJ,Lister-Sharp D, Wright K.No. 6Costs and benefits of communitypostnatal support workers: arandomised controlled trial.By Morrell CJ, Spiby H, Stewart P,Walters S, Morgan A.No. 7Implantable contraceptives (subdermalimplants and hormonally impregnatedintrauterine systems) versus otherforms of reversible contraceptives: twosystematic reviews to assess relativeeffectiveness, acceptability, tolerabilityand cost-effectiveness.By French RS, Cowan FM,Mansour DJA, Morris S, Procter T,Hughes D, et al.No. 8An introduction to statistical methodsfor health technology assessment.A review by White SJ, Ashby D,Brown PJ.No. 9Disease-modifying drugs for multiplesclerosis: a rapid and systematic review.By Clegg A, Bryant J, Milne R.No. 10Publication and related biases.A review by Song F, Eastwood AJ,Gilbody S, Duley L, Sutton AJ.No. 11Cost and outcome implications of theorganisation of vascular services.By Michaels J, Brazier J,Palfreyman S, Shackley P, Slack R.No. 12Monitoring blood glucose control indiabetes mellitus: a systematic review.By Coster S, Gulliford MC, Seed PT,Powrie JK, Swaminathan R.No. 13The effectiveness of domiciliaryhealth visiting: a systematic review ofinternational studies and a selectivereview of the British literature.By Elkan R, Kendrick D, Hewitt M,Robinson JJA, Tolley K, Blair M, et al.No. 14The determinants of screening uptakeand interventions for increasinguptake: a systematic review.By Jepson R, Clegg A, Forbes C,Lewis R, Sowden A, Kleijnen J.No. 15The effectiveness and cost-effectivenessof prophylactic removal of wisdomteeth.A rapid review by Song F, O’Meara S,Wilson P, Golder S, Kleijnen J.No. 16Ultrasound screening in pregnancy:a systematic review of the clinicaleffectiveness, cost-effectiveness andwomen’s views.By Bricker L, Garcia J, Henderson J,Mugford M, Neilson J, Roberts T, et al.No. 17A rapid and systematic review of theeffectiveness and cost-effectiveness ofthe taxanes used in the treatment ofadvanced breast and ovarian cancer.By Lister-Sharp D, McDonagh MS,Khan KS, Kleijnen J.No. 18Liquid-based cytology in cervicalscreening: a rapid and systematicreview.By Payne N, Chilcott J, McGoogan E.No. 19Randomised controlled trial of nondirectivecounselling, cognitive–behaviour therapy and usual generalpractitioner care in the management ofdepression as well as mixed anxiety anddepression in primary care.By King M, Sibbald B, Ward E,Bower P, Lloyd M, Gabbay M, et al.No. 20Routine referral for radiography ofpatients presenting with low back pain:is patients’ outcome influenced by GPs’referral for plain radiography?By Kerry S, Hilton S, Patel S,Dundas D, Rink E, Lord J.No. 21Systematic reviews of wound caremanagement: (3) antimicrobial agentsfor chronic wounds; (4) diabetic footulceration.By O’Meara S, Cullum N, Majid M,Sheldon T.No. 22Using routine data to complementand enhance the results of randomisedcontrolled trials.By Lewsey JD, Leyland AH, MurrayGD, Boddy FA.No. 23Coronary artery stents in the treatmentof ischaemic heart disease: a rapid andsystematic review.By Meads C, Cummins C, Jolly K,Stevens A, Burls A, Hyde C.No. 24Outcome measures for adult criticalcare: a systematic review.By Hayes JA, Black NA, Jenkinson C,Young JD, Rowan KM, Daly K, et al.No. 25A systematic review to evaluate theeffectiveness of interventions topromote the initiation of breastfeeding.By Fairbank L, O’Meara S,Renfrew MJ, Woolridge M, Sowden AJ,Lister-Sharp D.No. 26Implantable cardioverter defibrillators:arrhythmias. A rapid and systematicreview.By Parkes J, Bryant J, Milne R.No. 27Treatments for fatigue in multiplesclerosis: a rapid and systematic review.By Brañas P, Jordan R, Fry-Smith A,Burls A, Hyde C.No. 28Early asthma prophylaxis, naturalhistory, skeletal development andeconomy (EASE): a pilot randomisedcontrolled trial.By Baxter-Jones ADG, Helms PJ,Russell G, Grant A, Ross S, Cairns JA,et al.No. 29Screening for hypercholesterolaemiaversus case finding for familialhypercholesterolaemia: a systematicreview and cost-effectiveness analysis.By Marks D, WonderlingD, Thorogood M, Lambert H,Humphries SE, Neil HAW.No. 30A rapid and systematic review ofthe clinical effectiveness and costeffectivenessof glycoprotein IIb/IIIaantagonists in the medical managementof unstable angina.By McDonagh MS, Bachmann LM,Golder S, Kleijnen J, ter Riet G.197© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to date198No. 31A randomised controlled trialof prehospital intravenous fluidreplacement therapy in serious trauma.By Turner J, Nicholl J, Webber L,Cox H, Dixon S, Yates D.No. 32Intrathecal pumps for giving opioids inchronic pain: a systematic review.By Williams JE, Louw G,Towlerton G.No. 33Combination therapy (interferonalfa and ribavirin) in the treatmentof chronic hepatitis C: a rapid andsystematic review.By Shepherd J, Waugh N,Hewitson P.No. 34A systematic review of comparisons ofeffect sizes derived from randomisedand non-randomised studies.By MacLehose RR, Reeves BC,Harvey IM, Sheldon TA, Russell IT,Black AMS.No. 35Intravascular ultrasound-guidedinterventions in coronary arterydisease: a systematic literature review,with decision-analytic modelling, ofoutcomes and cost-effectiveness.By Berry E, Kelly S, Hutton J,Lindsay HSJ, Blaxill JM, Evans JA, et al.No. 36A randomised controlled trial toevaluate the effectiveness and costeffectivenessof counselling patientswith chronic depression.By Simpson S, Corney R,Fitzgerald P, Beecham J.No. 37Systematic review of treatments foratopic eczema.By Hoare C, Li Wan Po A,Williams H.No. 38Bayesian methods in health technologyassessment: a review.By Spiegelhalter DJ, Myles JP,Jones DR, Abrams KR.No. 39The management of dyspepsia: asystematic review.By Delaney B, Moayyedi P, Deeks J,Innes M, Soo S, Barton P, et al.No. 40A systematic review of treatments forsevere psoriasis.By Griffiths CEM, Clark CM,Chalmers RJG, Li Wan Po A,Williams HC.Volume 5, 2001No. 1Clinical and cost-effectivenessof donepezil, rivastigmine andgalantamine for Alzheimer’s disease: arapid and systematic review.By Clegg A, Bryant J, Nicholson T,McIntyre L, De Broe S, Gerard K, et al.No. 2The clinical effectiveness and costeffectivenessof riluzole for motorneurone disease: a rapid and systematicreview.By Stewart A, Sandercock J, Bryan S,Hyde C, Barton PM, Fry-Smith A, et al.No. 3Equity and the economic evaluation ofhealthcare.By Sassi F, Archard L, Le Grand J.No. 4Quality-of-life measures in chronicdiseases of childhood.By Eiser C, Morse R.No. 5Eliciting public preferences forhealthcare: a systematic review oftechniques.By Ryan M, Scott DA, Reeves C, BateA, van Teijlingen ER, Russell EM, et al.No. 6General health status measures forpeople with cognitive impairment:learning disability and acquired braininjury.By Riemsma RP, Forbes CA,Glanville JM, Eastwood AJ, Kleijnen J.No. 7An assessment of screening strategiesfor fragile X syndrome in the UK.By Pembrey ME, Barnicoat AJ,Carmichael B, Bobrow M, Turner G.No. 8Issues in methodological research:perspectives from researchers andcommissioners.By Lilford RJ, Richardson A, StevensA, Fitzpatrick R, Edwards S, Rock F, et al.No. 9Systematic reviews of woundcare management: (5) beds;(6) compression; (7) laser therapy,therapeutic ultrasound, electrotherapyand electromagnetic therapy.By Cullum N, Nelson EA,Flemming K, Sheldon T.No. 10Effects of educational and psychosocialinterventions for adolescents withdiabetes mellitus: a systematic review.By Hampson SE, Skinner TC, Hart J,Storey L, Gage H, Foxcroft D, et al.No. 11Effectiveness of autologous chondrocytetransplantation for hyaline cartilagedefects in knees: a rapid and systematicreview.By Jobanputra P, Parry D, Fry-SmithA, Burls A.No. 12Statistical assessment of the learningcurves of health technologies.By Ramsay CR, Grant AM, WallaceSA, Garthwaite PH, Monk AF, Russell IT.No. 13The effectiveness and cost-effectivenessof temozolomide for the treatment ofrecurrent malignant glioma: a rapidand systematic review.By Dinnes J, Cave C, Huang S,Major K, Milne R.No. 14A rapid and systematic review ofthe clinical effectiveness and costeffectivenessof debriding agents intreating surgical wounds healing bysecondary intention.By Lewis R, Whiting P, ter Riet G,O’Meara S, Glanville J.No. 15Home treatment for mental healthproblems: a systematic review.By Burns T, Knapp M, Catty J,Healey A, Henderson J, Watt H, et al.No. 16How to develop cost-consciousguidelines.By Eccles M, Mason J.No. 17The role of specialist nurses in multiplesclerosis: a rapid and systematic review.By De Broe S, Christopher F,Waugh N.No. 18A rapid and systematic reviewof the clinical effectiveness andcost-effectiveness of orlistat in themanagement of obesity.By O’Meara S, Riemsma R,Shirran L, Mather L, ter Riet G.No. 19The clinical effectiveness and costeffectivenessof pioglitazone fortype 2 diabetes mellitus: a rapid andsystematic review.By Chilcott J, Wight J, Lloyd JonesM, Tappenden P.No. 20Extended scope of nursing practice:a multicentre randomised controlledtrial of appropriately trained nursesand preregistration house officers inpreoperative assessment in electivegeneral surgery.By Kinley H, Czoski-Murray C,George S, McCabe C, Primrose J,Reilly C, et al.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 21Systematic reviews of the effectivenessof day care for people with severemental disorders: (1) Acute day hospitalversus admission; (2) Vocationalrehabilitation; (3) Day hospital versusoutpatient care.By Marshall M, Crowther R,Almaraz- Serrano A, Creed F, Sledge W,Kluiter H, et al.No. 22The measurement and monitoring ofsurgical adverse events.By Bruce J, Russell EM, Mollison J,Krukowski ZH.No. 31Design and use of questionnaires: areview of best practice applicable tosurveys of health service staff andpatients.By McColl E, Jacoby A, Thomas L,Soutter J, Bamford C, Steen N, et al.No. 32A rapid and systematic review ofthe clinical effectiveness and costeffectivenessof paclitaxel, docetaxel,gemcitabine and vinorelbine in nonsmall-celllung cancer.By Clegg A, Scott DA, Sidhu M,Hewitson P, Waugh N.No. 5The clinical effectiveness and costeffectivenessof inhaler devices usedin the routine management of chronicasthma in older children: a systematicreview and economic evaluation.By Peters J, Stevenson M, Beverley C,Lim J, Smith S.No. 6The clinical effectiveness and costeffectivenessof sibutramine in themanagement of obesity: a technologyassessment.By O’Meara S, Riemsma R, ShirranL, Mather L, ter Riet G.No. 23Action research: a systematic review andguidance for assessment.By Waterman H, Tillen D, Dickson R,de Koning K.No. 24A rapid and systematic review ofthe clinical effectiveness and costeffectivenessof gemcitabine for thetreatment of pancreatic cancer.By Ward S, Morris E, Bansback N,Calvert N, Crellin A, Forman D, et al.No. 25A rapid and systematic review of theevidence for the clinical effectivenessand cost-effectiveness of irinotecan,oxaliplatin and raltitrexed for thetreatment of advanced colorectalcancer.By Lloyd Jones M, Hummel S,Bansback N, Orr B, Seymour M.No. 26Comparison of the effectiveness ofinhaler devices in asthma and chronicobstructive airways disease: a systematicreview of the literature.By Brocklebank D, Ram F, Wright J,Barry P, Cates C, Davies L, et al.No. 27The cost-effectiveness of magneticresonance imaging for investigation ofthe knee joint.By Bryan S, Weatherburn G, BungayH, Hatrick C, Salas C, Parry D, et al.No. 28A rapid and systematic review ofthe clinical effectiveness and costeffectivenessof topotecan for ovariancancer.By Forbes C, Shirran L, Bagnall A-M,Duffy S, ter Riet G.No. 29Superseded by a report published in alater volume.No. 30The role of radiography in primarycare patients with low back pain of atleast 6 weeks duration: a randomised(unblinded) controlled trial.By Kendrick D, Fielding K, BentleyE, Miller P, Kerslake R, Pringle M.No. 33Subgroup analyses in randomisedcontrolled trials: quantifying the risksof false-positives and false-negatives.By Brookes ST, Whitley E, Peters TJ,Mulheran PA, Egger M, Davey Smith G.No. 34Depot antipsychotic medicationin the treatment of patients withschizophrenia: (1) Meta-review; (2)Patient and nurse attitudes.By David AS, Adams C.No. 35A systematic review of controlledtrials of the effectiveness and costeffectivenessof brief psychologicaltreatments for depression.By Churchill R, Hunot V, Corney R,Knapp M, McGuire H, Tylee A, et al.No. 36Cost analysis of child healthsurveillance.By Sanderson D, Wright D, Acton C,Duree D.Volume 6, 2002No. 1A study of the methods used to selectreview criteria for clinical audit.By Hearnshaw H, Harker R,Cheater F, Baker R, Grimshaw G.No. 2Fludarabine as second-line therapy forB cell chronic lymphocytic leukaemia: atechnology assessment.By Hyde C, Wake B, Bryan S, BartonP, Fry-Smith A, Davenport C, et al.No. 3Rituximab as third-line treatment forrefractory or recurrent Stage III or IVfollicular non-Hodgkin’s lymphoma:a systematic review and economicevaluation.By Wake B, Hyde C, Bryan S, BartonP, Song F, Fry-Smith A, et al.No. 4A systematic review of dischargearrangements for older people.By Parker SG, Peet SM, McPhersonA, Cannaby AM, Baker R, Wilson A, et al.No. 7The cost-effectiveness of magneticresonance angiography for carotidartery stenosis and peripheral vasculardisease: a systematic review.By Berry E, Kelly S, Westwood ME,Davies LM, Gough MJ, Bamford JM,et al.No. 8Promoting physical activity in SouthAsian Muslim women through ‘exerciseon prescription’.By Carroll B, Ali N, Azam N.No. 9Zanamivir for the treatment ofinfluenza in adults: a systematic reviewand economic evaluation.By Burls A, Clark W, Stewart T,Preston C, Bryan S, Jefferson T, et al.No. 10A review of the natural history andepidemiology of multiple sclerosis:implications for resource allocation andhealth economic models.By Richards RG, Sampson FC,Beard SM, Tappenden P.No. 11Screening for gestational diabetes:a systematic review and economicevaluation.By Scott DA, Loveman E, McIntyreL, Waugh N.No. 12The clinical effectiveness and costeffectivenessof surgery for people withmorbid obesity: a systematic review andeconomic evaluation.By Clegg AJ, Colquitt J, Sidhu MK,Royle P, Loveman E, Walker A.No. 13The clinical effectiveness oftrastuzumab for breast cancer: asystematic review.By Lewis R, Bagnall A-M, Forbes C,Shirran E, Duffy S, Kleijnen J, et al.No. 14The clinical effectiveness and costeffectivenessof vinorelbine for breastcancer: a systematic review andeconomic evaluation.By Lewis R, Bagnall A-M, King S,Woolacott N, Forbes C, Shirran L, et al.199© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to dateNo. 15A systematic review of the effectivenessand cost-effectiveness of metal-onmetalhip resurfacing arthroplasty fortreatment of hip disease.By Vale L, Wyness L, McCormack K,McKenzie L, Brazzelli M, Stearns SC.No. 16The clinical effectiveness and costeffectivenessof bupropion and nicotinereplacement therapy for smokingcessation: a systematic review andeconomic evaluation.By Woolacott NF, Jones L, Forbes CA,Mather LC, Sowden AJ, Song FJ, et al.No. 17A systematic review of effectivenessand economic evaluation of new drugtreatments for juvenile idiopathicarthritis: etanercept.By Cummins C, Connock M,Fry-Smith A, Burls A.No. 24A systematic review of the effectivenessof interventions based on a stages-ofchangeapproach to promote individualbehaviour change.By Riemsma RP, Pattenden J, BridleC, Sowden AJ, Mather L, Watt IS, et al.No. 25A systematic review update of theclinical effectiveness and costeffectivenessof glycoprotein IIb/IIIaantagonists.By Robinson M, Ginnelly L, SculpherM, Jones L, Riemsma R, Palmer S, et al.No. 26A systematic review of the effectiveness,cost-effectiveness and barriers toimplementation of thrombolytic andneuroprotective therapy for acuteischaemic stroke in the NHS.By Sandercock P, Berge E, Dennis M,Forbes J, Hand P, Kwan J, et al.No. 33The effectiveness and cost-effectivenessof imatinib in chronic myeloidleukaemia: a systematic review.By Garside R, Round A, Dalziel K,Stein K, Royle R.No. 34A comparative study of hypertonicsaline, daily and alternate-day rhDNasein children with cystic fibrosis.By Suri R, Wallis C, Bush A,Thompson S, Normand C, Flather M,et al.No. 35A systematic review of the costs andeffectiveness of different models ofpaediatric home care.By Parker G, Bhakta P, Lovett CA,Paisley S, Olsen R, Turner D, et al.Volume 7, 2003200No. 18Clinical effectiveness and costeffectivenessof growth hormone inchildren: a systematic review andeconomic evaluation.By Bryant J, Cave C, Mihaylova B,Chase D, McIntyre L, Gerard K, et al.No. 19Clinical effectiveness and costeffectivenessof growth hormonein adults in relation to impact onquality of life: a systematic review andeconomic evaluation.By Bryant J, Loveman E, Chase D,Mihaylova B, Cave C, Gerard K, et al.No. 20Clinical medication review by apharmacist of patients on repeatprescriptions in general practice: arandomised controlled trial.By Zermansky AG, Petty DR, RaynorDK, Lowe CJ, Freementle N, Vail A.No. 21The effectiveness of infliximab andetanercept for the treatment ofrheumatoid arthritis: a systematicreview and economic evaluation.By Jobanputra P, Barton P, Bryan S,Burls A.No. 22A systematic review and economicevaluation of computerised cognitivebehaviour therapy for depression andanxiety.By Kaltenthaler E, Shackley P,Stevens K, Beverley C, Parry G,Chilcott J.No. 23A systematic review and economicevaluation of pegylated liposomaldoxorubicin hydrochloride for ovariancancer.By Forbes C, Wilby J, Richardson G,Sculpher M, Mather L, Reimsma R.No. 27A randomised controlled crossover trialof nurse practitioner versus doctorledoutpatient care in a bronchiectasisclinic.By Caine N, Sharples LD,Hollingworth W, French J, Keogan M,Exley A, et al.No. 28Clinical effectiveness and cost –consequences of selective serotoninreuptake inhibitors in the treatment ofsex offenders.By Adi Y, Ashcroft D, Browne K,Beech A, Fry-Smith A, Hyde C.No. 29Treatment of established osteoporosis:a systematic review and cost–utilityanalysis.By Kanis JA, Brazier JE, StevensonM, Calvert NW, Lloyd Jones M.No. 30Which anaesthetic agents are costeffectivein day surgery? Literaturereview, national survey of practice andrandomised controlled trial.By Elliott RA Payne K, Moore JK,Davies LM, Harper NJN, St Leger AS,et al.No. 31Screening for hepatitis C amonginjecting drug users and ingenitourinary medicine clinics:systematic reviews of effectiveness,modelling study and national survey ofcurrent practice.By Stein K, Dalziel K, Walker A,McIntyre L, Jenkins B, Horne J, et al.No. 32The measurement of satisfaction withhealthcare: implications for practicefrom a systematic review of theliterature.By Crow R, Gage H, Hampson S,Hart J, Kimber A, Storey L, et al.No. 1How important are comprehensiveliterature searches and the assessmentof trial quality in systematic reviews?Empirical study.By Egger M, Jüni P, Bartlett C,Holenstein F, Sterne J.No. 2Systematic review of the effectivenessand cost-effectiveness, and economicevaluation, of home versus hospital orsatellite unit haemodialysis for peoplewith end-stage renal failure.By Mowatt G, Vale L, Perez J, WynessL, Fraser C, MacLeod A, et al.No. 3Systematic review and economicevaluation of the effectiveness ofinfliximab for the treatment of Crohn’sdisease.By Clark W, Raftery J, Barton P,Song F, Fry-Smith A, Burls A.No. 4A review of the clinical effectivenessand cost-effectiveness of routine anti-Dprophylaxis for pregnant women whoare rhesus negative.By Chilcott J, Lloyd Jones M, WightJ, Forman K, Wray J, Beverley C, et al.No. 5Systematic review and evaluation of theuse of tumour markers in paediatriconcology: Ewing’s sarcoma andneuroblastoma.By Riley RD, Burchill SA,Abrams KR, Heney D, Lambert PC,Jones DR, et al.No. 6The cost-effectiveness of screening forHelicobacter pylori to reduce mortalityand morbidity from gastric cancer andpeptic ulcer disease: a discrete-eventsimulation model.By Roderick P, Davies R, Raftery J,Crabbe D, Pearce R, Bhandari P, et al.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 7The clinical effectiveness and costeffectivenessof routine dental checks:a systematic review and economicevaluation.By Davenport C, Elley K, SalasC, Taylor-Weetman CL, Fry-Smith A,Bryan S, et al.No. 8A multicentre randomised controlledtrial assessing the costs and benefitsof using structured information andanalysis of women’s preferences in themanagement of menorrhagia.By Kennedy ADM, Sculpher MJ,Coulter A, Dwyer N, Rees M, Horsley S,et al.No. 9Clinical effectiveness and cost–utilityof photodynamic therapy for wetage-related macular degeneration:a systematic review and economicevaluation.By Meads C, Salas C, Roberts T,Moore D, Fry-Smith A, Hyde C.No. 10Evaluation of molecular tests forprenatal diagnosis of chromosomeabnormalities.By Grimshaw GM, Szczepura A,Hultén M, MacDonald F, Nevin NC,Sutton F, et al.No. 11First and second trimester antenatalscreening for Down’s syndrome:the results of the Serum, Urine andUltrasound Screening Study (SURUSS).By Wald NJ, Rodeck C, HackshawAK, Walters J, Chitty L, Mackinson AM.No. 12The effectiveness and cost-effectivenessof ultrasound locating devices forcentral venous access: a systematicreview and economic evaluation.By Calvert N, Hind D, McWilliamsRG, Thomas SM, Beverley C,Davidson A.No. 13A systematic review of atypicalantipsychotics in schizophrenia.By Bagnall A-M, Jones L, Lewis R,Ginnelly L, Glanville J, Torgerson D,et al.No. 14Prostate Testing for Cancer andTreatment (ProtecT) feasibility study.By Donovan J, Hamdy F, Neal D,Peters T, Oliver S, Brindle L, et al.No. 15Early thrombolysis for the treatmentof acute myocardial infarction: asystematic review and economicevaluation.By Boland A, Dundar Y, Bagust A,Haycox A, Hill R, Mujica Mota R, et al.No. 16Screening for fragile X syndrome: aliterature review and modelling.By Song FJ, Barton P, SleightholmeV, Yao GL, Fry-Smith A.No. 17Systematic review of endoscopic sinussurgery for nasal polyps.By Dalziel K, Stein K, Round A,Garside R, Royle P.No. 18Towards efficient guidelines: how tomonitor guideline use in primary care.By Hutchinson A, McIntosh A,Cox S, Gilbert C.No. 19Effectiveness and cost-effectivenessof acute hospital-based spinal cordinjuries services: systematic review.By Bagnall A-M, Jones L, RichardsonG, Duffy S, Riemsma R.No. 20Prioritisation of health technologyassessment. The PATHS model:methods and case studies.By Townsend J, Buxton M,Harper G.No. 21Systematic review of the clinicaleffectiveness and cost-effectiveness oftension-free vaginal tape for treatmentof urinary stress incontinence.By Cody J, Wyness L, Wallace S,Glazener C, Kilonzo M, Stearns S, et al.No. 22The clinical and cost-effectiveness ofpatient education models for diabetes:a systematic review and economicevaluation.By Loveman E, Cave C, Green C,Royle P, Dunn N, Waugh N.No. 23The role of modelling in prioritisingand planning clinical trials.By Chilcott J, Brennan A, Booth A,Karnon J, Tappenden P.No. 24Cost–benefit evaluation of routineinfluenza immunisation in people65–74 years of age.By Allsup S, Gosney M, Haycox A,Regan M.No. 25The clinical and cost-effectiveness ofpulsatile machine perfusion versus coldstorage of kidneys for transplantationretrieved from heart-beating and nonheart-beatingdonors.By Wight J, Chilcott J, Holmes M,Brewer N.No. 26Can randomised trials rely on existingelectronic data? A feasibility study toexplore the value of routine data inhealth technology assessment.By Williams JG, Cheung WY,Cohen DR, Hutchings HA, Longo MF,Russell IT.No. 27Evaluating non-randomisedintervention studies.By Deeks JJ, Dinnes J, D’Amico R,Sowden AJ, Sakarovitch C, Song F, et al.No. 28A randomised controlled trial to assessthe impact of a package comprising apatient-orientated, evidence-based selfhelpguidebook and patient-centredconsultations on disease managementand satisfaction in inflammatory boweldisease.By Kennedy A, Nelson E, Reeves D,Richardson G, Roberts C, Robinson A,et al.No. 29The effectiveness of diagnostic tests forthe assessment of shoulder pain dueto soft tissue disorders: a systematicreview.By Dinnes J, Loveman E, McIntyre L,Waugh N.No. 30The value of digital imaging in diabeticretinopathy.By Sharp PF, Olson J, Strachan F,Hipwell J, Ludbrook A, O’Donnell M,et al.No. 31Lowering blood pressure to preventmyocardial infarction and stroke: a newpreventive strategy.By Law M, Wald N, Morris J.No. 32Clinical and cost-effectiveness ofcapecitabine and tegafur with uracil forthe treatment of metastatic colorectalcancer: systematic review and economicevaluation.By Ward S, Kaltenthaler E, Cowan J,Brewer N.No. 33Clinical and cost-effectiveness of newand emerging technologies for earlylocalised prostate cancer: a systematicreview.By Hummel S, Paisley S, Morgan A,Currie E, Brewer N.No. 34Literature searching for clinical andcost-effectiveness studies used in healthtechnology assessment reports carriedout for the National Institute forClinical Excellence appraisal system.By Royle P, Waugh N.201© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to dateNo. 35Systematic review and economicdecision modelling for the preventionand treatment of influenza A and B.By Turner D, Wailoo A, Nicholson K,Cooper N, Sutton A, Abrams K.No. 36A randomised controlled trialto evaluate the clinical and costeffectivenessof Hickman line insertionsin adult cancer patients by nurses.By Boland A, Haycox A, Bagust A,Fitzsimmons L.No. 37Redesigning postnatal care: arandomised controlled trial of protocolbasedmidwifery-led care focusedon individual women’s physical andpsychological health needs.By MacArthur C, Winter HR,Bick DE, Lilford RJ, Lancashire RJ,Knowles H, et al.No. 38Estimating implied rates of discount inhealthcare decision-making.By West RR, McNabb R, ThompsonAGH, Sheldon TA, Grimley Evans J.No. 39Systematic review of isolation policiesin the hospital management ofmethicillin-resistant Staphylococcusaureus: a review of the literaturewith epidemiological and economicmodelling.By Cooper BS, Stone SP, Kibbler CC,Cookson BD, Roberts JA, Medley GF,et al.No. 40Treatments for spasticity and pain inmultiple sclerosis: a systematic review.By Beard S, Hunn A, Wight J.No. 41The inclusion of reports of randomisedtrials published in languages other thanEnglish in systematic reviews.By Moher D, Pham B, Lawson ML,Klassen TP.No. 2Systematic review and modelling of theinvestigation of acute and chronic chestpain presenting in primary care.By Mant J, McManus RJ, Oakes RAL,Delaney BC, Barton PM, Deeks JJ, et al.No. 3The effectiveness and cost-effectivenessof microwave and thermal balloonendometrial ablation for heavymenstrual bleeding: a systematic reviewand economic modelling.By Garside R, Stein K, Wyatt K,Round A, Price A.No. 4A systematic review of the role ofbisphosphonates in metastatic disease.By Ross JR, Saunders Y,Edmonds PM, Patel S, Wonderling D,Normand C, et al.No. 5Systematic review of the clinicaleffectiveness and cost-effectivenessof capecitabine (Xeloda ® ) for locallyadvanced and/or metastatic breastcancer.By Jones L, Hawkins N, Westwood M,Wright K, Richardson G, Riemsma R.No. 6Effectiveness and efficiency of guidelinedissemination and implementationstrategies.By Grimshaw JM, Thomas RE,MacLennan G, Fraser C, Ramsay CR,Vale L, et al.No. 7Clinical effectiveness and costs of theSugarbaker procedure for the treatmentof pseudomyxoma peritonei.By Bryant J, Clegg AJ, Sidhu MK,Brodin H, Royle P, Davidson P.No. 8Psychological treatment for insomniain the regulation of long-term hypnoticdrug use.By Morgan K, Dixon S, Mathers N,Thompson J, Tomeny M.No. 11The use of modelling to evaluatenew drugs for patients with a chroniccondition: the case of antibodiesagainst tumour necrosis factor inrheumatoid arthritis.By Barton P, Jobanputra P, Wilson J,Bryan S, Burls A.No. 12Clinical effectiveness and costeffectivenessof neonatal screeningfor inborn errors of metabolism usingtandem mass spectrometry: a systematicreview.By Pandor A, Eastham J, Beverley C,Chilcott J, Paisley S.No. 13Clinical effectiveness and costeffectivenessof pioglitazone androsiglitazone in the treatment of type2 diabetes: a systematic review andeconomic evaluation.By Czoski-Murray C, Warren E,Chilcott J, Beverley C, Psyllaki MA,Cowan J.No. 14Routine examination of the newborn:the EMREN study. Evaluation of anextension of the midwife role includinga randomised controlled trial ofappropriately trained midwives andpaediatric senior house officers.By Townsend J, Wolke D, Hayes J,Davé S, Rogers C, Bloomfield L, et al.No. 15Involving consumers in research anddevelopment agenda setting for theNHS: developing an evidence-basedapproach.By Oliver S, Clarke-Jones L, Rees R,Milne R, Buchanan P, Gabbay J, et al.No. 16A multi-centre randomised controlledtrial of minimally invasive directcoronary bypass grafting versuspercutaneous transluminal coronaryangioplasty with stenting for proximalstenosis of the left anterior descendingcoronary artery.By Reeves BC, Angelini GD, BryanAJ, Taylor FC, Cripps T, Spyt TJ, et al.202No. 42The impact of screening on futurehealth-promoting behaviours andhealth beliefs: a systematic review.By Bankhead CR, Brett J, Bukach C,Webster P, Stewart-Brown S, Munafo M,et al.Volume 8, 2004No. 1What is the best imaging strategy foracute stroke?By Wardlaw JM, Keir SL, Seymour J,Lewis S, Sandercock PAG, Dennis MS,et al.No. 9Improving the evaluation oftherapeutic interventions in multiplesclerosis: development of a patientbasedmeasure of outcome.By Hobart JC, Riazi A, Lamping DL,Fitzpatrick R, Thompson AJ.No. 10A systematic review and economicevaluation of magnetic resonancecholangiopancreatography comparedwith diagnostic endoscopic retrogradecholangiopancreatography.By Kaltenthaler E, Bravo Vergel Y,Chilcott J, Thomas S, Blakeborough T,Walters SJ, et al.No. 17Does early magnetic resonance imaginginfluence management or improveoutcome in patients referred tosecondary care with low back pain? Apragmatic randomised controlled trial.By Gilbert FJ, Grant AM, GillanMGC, Vale L, Scott NW, Campbell MK,et al.No. 18The clinical and cost-effectivenessof anakinra for the treatment ofrheumatoid arthritis in adults: asystematic review and economicanalysis.By Clark W, Jobanputra P, Barton P,Burls A.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 19A rapid and systematic review andeconomic evaluation of the clinicaland cost-effectiveness of newer drugsfor treatment of mania associated withbipolar affective disorder.By Bridle C, Palmer S, Bagnall A-M,Darba J, Duffy S, Sculpher M, et al.No. 20Liquid-based cytology in cervicalscreening: an updated rapid andsystematic review and economicanalysis.By Karnon J, Peters J, Platt J,Chilcott J, McGoogan E, Brewer N.No. 21Systematic review of the long-termeffects and economic consequences oftreatments for obesity and implicationsfor health improvement.By Avenell A, Broom J, Brown TJ,Poobalan A, Aucott L, Stearns SC, et al.No. 22Autoantibody testing in childrenwith newly diagnosed type 1 diabetesmellitus.By Dretzke J, Cummins C,Sandercock J, Fry-Smith A, Barrett T,Burls A.No. 23Clinical effectiveness and costeffectivenessof prehospital intravenousfluids in trauma patients.By Dretzke J, Sandercock J, BaylissS, Burls A.No. 24Newer hypnotic drugs for the shorttermmanagement of insomnia: asystematic review and economicevaluation.By Dündar Y, Boland A, Strobl J,Dodd S, Haycox A, Bagust A, et al.No. 25Development and validation ofmethods for assessing the quality ofdiagnostic accuracy studies.By Whiting P, Rutjes AWS, Dinnes J,Reitsma JB, Bossuyt PMM, Kleijnen J.No. 26EVALUATE hysterectomy trial:a multicentre randomised trialcomparing abdominal, vaginal andlaparoscopic methods of hysterectomy.By Garry R, Fountain J, Brown J,Manca A, Mason S, Sculpher M, et al.No. 27Methods for expected value ofinformation analysis in complex healtheconomic models: developments onthe health economics of interferon-βand glatiramer acetate for multiplesclerosis.By Tappenden P, Chilcott JB,Eggington S, Oakley J, McCabe C.No. 28Effectiveness and cost-effectivenessof imatinib for first-line treatmentof chronic myeloid leukaemia inchronic phase: a systematic review andeconomic analysis.By Dalziel K, Round A, Stein K,Garside R, Price A.No. 29VenUS I: a randomised controlled trialof two types of bandage for treatingvenous leg ulcers.By Iglesias C, Nelson EA, CullumNA, Torgerson DJ, on behalf of theVenUS Team.No. 30Systematic review of the effectivenessand cost-effectiveness, and economicevaluation, of myocardial perfusionscintigraphy for the diagnosis andmanagement of angina and myocardialinfarction.By Mowatt G, Vale L, Brazzelli M,Hernandez R, Murray A, Scott N, et al.No. 31A pilot study on the use of decisiontheory and value of informationanalysis as part of the NHS HealthTechnology Assessment programme.By Claxton K, Ginnelly L, SculpherM, Philips Z, Palmer S.No. 32The Social Support and Family HealthStudy: a randomised controlled trialand economic evaluation of twoalternative forms of postnatal supportfor mothers living in disadvantagedinner-city areas.By Wiggins M, Oakley A, Roberts I,Turner H, Rajan L, Austerberry H, et al.No. 33Psychosocial aspects of geneticscreening of pregnant women andnewborns: a systematic review.By Green JM, Hewison J, Bekker HL,Bryant, Cuckle HS.No. 34Evaluation of abnormal uterinebleeding: comparison of threeoutpatient procedures within cohortsdefined by age and menopausal status.By Critchley HOD, Warner P, Lee AJ,Brechin S, Guise J, Graham B.No. 35Coronary artery stents: a rapidsystematic review and economicevaluation.By Hill R, Bagust A, Bakhai A,Dickson R, Dündar Y, Haycox A, et al.No. 36Review of guidelines for good practicein decision-analytic modelling in healthtechnology assessment.By Philips Z, Ginnelly L, Sculpher M,Claxton K, Golder S, Riemsma R, et al.No. 37Rituximab (MabThera ® ) for aggressivenon-Hodgkin’s lymphoma: systematicreview and economic evaluation.By Knight C, Hind D, Brewer N,Abbott V.No. 38Clinical effectiveness and costeffectivenessof clopidogrel andmodified-release dipyridamole in thesecondary prevention of occlusivevascular events: a systematic review andeconomic evaluation.By Jones L, Griffin S, Palmer S, MainC, Orton V, Sculpher M, et al.No. 39Pegylated interferon α-2a and -2bin combination with ribavirin in thetreatment of chronic hepatitis C:a systematic review and economicevaluation.By Shepherd J, Brodin H, Cave C,Waugh N, Price A, Gabbay J.No. 40Clopidogrel used in combination withaspirin compared with aspirin alonein the treatment of non-ST-segmentelevationacute coronary syndromes:a systematic review and economicevaluation.By Main C, Palmer S, Griffin S, JonesL, Orton V, Sculpher M, et al.No. 41Provision, uptake and cost of cardiacrehabilitation programmes: improvingservices to under-represented groups.By Beswick AD, Rees K, Griebsch I,Taylor FC, Burke M, West RR, et al.No. 42Involving South Asian patients inclinical trials.By Hussain-Gambles M, Leese B,Atkin K, Brown J, Mason S, Tovey P.No. 43Clinical and cost-effectiveness ofcontinuous subcutaneous insulininfusion for diabetes.By Colquitt JL, Green C, Sidhu MK,Hartwell D, Waugh N.No. 44Identification and assessment ofongoing trials in health technologyassessment reviews.By Song FJ, Fry-Smith A, DavenportC, Bayliss S, Adi Y, Wilson JS, et al.No. 45Systematic review and economicevaluation of a long-acting insulinanalogue, insulin glargineBy Warren E, Weatherley-Jones E,Chilcott J, Beverley C.203© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to date204No. 46Supplementation of a home-basedexercise programme with a classbasedprogramme for peoplewith osteoarthritis of the knees: arandomised controlled trial and healtheconomic analysis.By McCarthy CJ, Mills PM, Pullen R,Richardson G, Hawkins N, Roberts CR,et al.No. 47Clinical and cost-effectiveness of oncedailyversus more frequent use of samepotency topical corticosteroids foratopic eczema: a systematic review andeconomic evaluation.By Green C, Colquitt JL, Kirby J,Davidson P, Payne E.No. 48Acupuncture of chronic headachedisorders in primary care: randomisedcontrolled trial and economic analysis.By Vickers AJ, Rees RW, Zollman CE,McCarney R, Smith CM, Ellis N, et al.No. 49Generalisability in economic evaluationstudies in healthcare: a review and casestudies.By Sculpher MJ, Pang FS, Manca A,Drummond MF, Golder S, Urdahl H,et al.No. 50Virtual outreach: a randomisedcontrolled trial and economicevaluation of joint teleconferencedmedical consultations.By Wallace P, Barber J, Clayton W,Currell R, Fleming K, Garner P, et al.Volume 9, 2005No. 1Randomised controlled multipletreatment comparison to provide a costeffectivenessrationale for the selectionof antimicrobial therapy in acne.By Ozolins M, Eady EA, Avery A,Cunliffe WJ, O’Neill C, Simpson NB,et al.No. 2Do the findings of case series studiesvary significantly according tomethodological characteristics?By Dalziel K, Round A, Stein K,Garside R, Castelnuovo E, Payne L.No. 3Improving the referral processfor familial breast cancer geneticcounselling: findings of threerandomised controlled trials of twointerventions.By Wilson BJ, Torrance N,Mollison J, Wordsworth S, Gray JR,Haites NE, et al.No. 4Randomised evaluation of alternativeelectrosurgical modalities to treatbladder outflow obstruction in menwith benign prostatic hyperplasia.By Fowler C, McAllister W, Plail R,Karim O, Yang Q.No. 5A pragmatic randomised controlledtrial of the cost-effectiveness ofpalliative therapies for patients withinoperable oesophageal cancer.By Shenfine J, McNamee P, Steen N,Bond J, Griffin SM.No. 6Impact of computer-aided detectionprompts on the sensitivity andspecificity of screening mammography.By Taylor P, Champness J, Given-Wilson R, Johnston K, Potts H.No. 7Issues in data monitoring and interimanalysis of trials.By Grant AM, Altman DG, BabikerAB, Campbell MK, Clemens FJ,Darbyshire JH, et al.No. 8Lay public’s understanding of equipoiseand randomisation in randomisedcontrolled trials.By Robinson EJ, Kerr CEP,Stevens AJ, Lilford RJ, Braunholtz DA,Edwards SJ, et al.No. 9Clinical and cost-effectiveness ofelectroconvulsive therapy for depressiveillness, schizophrenia, catatoniaand mania: systematic reviews andeconomic modelling studies.By Greenhalgh J, Knight C, Hind D,Beverley C, Walters S.No. 10Measurement of health-related qualityof life for people with dementia:development of a new instrument(DEMQOL) and an evaluation ofcurrent methodology.By Smith SC, Lamping DL, BanerjeeS, Harwood R, Foley B, Smith P, et al.No. 11Clinical effectiveness and costeffectivenessof drotrecogin alfa(activated) (Xigris ® ) for the treatmentof severe sepsis in adults: a systematicreview and economic evaluation.By Green C, Dinnes J, Takeda A,Shepherd J, Hartwell D, Cave C, et al.No. 12A methodological review of howheterogeneity has been examined insystematic reviews of diagnostic testaccuracy.By Dinnes J, Deeks J, Kirby J,Roderick P.No. 13Cervical screening programmes: canautomation help? Evidence fromsystematic reviews, an economicanalysis and a simulation modellingexercise applied to the UK.By Willis BH, Barton P, Pearmain P,Bryan S, Hyde C.No. 14Laparoscopic surgery for inguinalhernia repair: systematic review ofeffectiveness and economic evaluation.By McCormack K, Wake B, Perez J,Fraser C, Cook J, McIntosh E, et al.No. 15Clinical effectiveness, tolerability andcost-effectiveness of newer drugs forepilepsy in adults: a systematic reviewand economic evaluation.By Wilby J, Kainth A, Hawkins N,Epstein D, McIntosh H, McDaid C, et al.No. 16A randomised controlled trial tocompare the cost-effectiveness oftricyclic antidepressants, selectiveserotonin reuptake inhibitors andlofepramine.By Peveler R, Kendrick T, Buxton M,Longworth L, Baldwin D, Moore M, et al.No. 17Clinical effectiveness and costeffectivenessof immediate angioplastyfor acute myocardial infarction:systematic review and economicevaluation.By Hartwell D, Colquitt J, LovemanE, Clegg AJ, Brodin H, Waugh N, et al.No. 18A randomised controlled comparison ofalternative strategies in stroke care.By Kalra L, Evans A, Perez I,Knapp M, Swift C, Donaldson N.No. 19The investigation and analysis ofcritical incidents and adverse events inhealthcare.By Woloshynowych M, Rogers S,Taylor-Adams S, Vincent C.No. 20Potential use of routine databases inhealth technology assessment.By Raftery J, Roderick P, Stevens A.No. 21Clinical and cost-effectiveness of newerimmunosuppressive regimens in renaltransplantation: a systematic review andmodelling study.By Woodroffe R, Yao GL, Meads C,Bayliss S, Ready A, Raftery J, et al.No. 22A systematic review and economicevaluation of alendronate, etidronate,risedronate, raloxifene and teriparatidefor the prevention and treatment ofpostmenopausal osteoporosis.By Stevenson M, Lloyd Jones M, DeNigris E, Brewer N, Davis S, Oakley J.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 23A systematic review to examinethe impact of psycho-educationalinterventions on health outcomesand costs in adults and children withdifficult asthma.By Smith JR, Mugford M, HollandR, Candy B, Noble MJ, Harrison BDW,et al.No. 24An evaluation of the costs, effectivenessand quality of renal replacementtherapy provision in renal satellite unitsin England and Wales.By Roderick P, Nicholson T, ArmitageA, Mehta R, Mullee M, Gerard K, et al.No. 25Imatinib for the treatment of patientswith unresectable and/or metastaticgastrointestinal stromal tumours:systematic review and economicevaluation.By Wilson J, Connock M, Song F,Yao G, Fry-Smith A, Raftery J, et al.No. 26Indirect comparisons of competinginterventions.By Glenny AM, Altman DG, Song F,Sakarovitch C, Deeks JJ, D’Amico R,et al.No. 27Cost-effectiveness of alternativestrategies for the initial medicalmanagement of non-ST elevation acutecoronary syndrome: systematic reviewand decision-analytical modelling.By Robinson M, Palmer S, SculpherM, Philips Z, Ginnelly L, Bowens A, et al.No. 28Outcomes of electrically stimulatedgracilis neosphincter surgery.By Tillin T, Chambers M, Feldman R.No. 29The effectiveness and cost-effectivenessof pimecrolimus and tacrolimus foratopic eczema: a systematic review andeconomic evaluation.By Garside R, Stein K, CastelnuovoE, Pitt M, Ashcroft D, Dimmock P, et al.No. 30Systematic review on urine albumintesting for early detection of diabeticcomplications.By Newman DJ, Mattock MB,Dawnay ABS, Kerry S, McGuire A,Yaqoob M, et al.No. 31Randomised controlled trial of the costeffectivenessof water-based therapy forlower limb osteoarthritis.By Cochrane T, Davey RC,Matthes Edwards SM.No. 32Longer term clinical and economicbenefits of offering acupuncture care topatients with chronic low back pain.By Thomas KJ, MacPhersonH, Ratcliffe J, Thorpe L, Brazier J,Campbell M, et al.No. 33Cost-effectiveness and safety ofepidural steroids in the managementof sciatica.By Price C, Arden N, Coglan L,Rogers P.No. 34The British Rheumatoid OutcomeStudy Group (BROSG) randomisedcontrolled trial to compare theeffectiveness and cost-effectiveness ofaggressive versus symptomatic therapyin established rheumatoid arthritis.By Symmons D, Tricker K, RobertsC, Davies L, Dawes P, Scott DL.No. 35Conceptual framework and systematicreview of the effects of participants’and professionals’ preferences inrandomised controlled trials.By King M, Nazareth I, Lampe F,Bower P, Chandler M, Morou M, et al.No. 36The clinical and cost-effectiveness ofimplantable cardioverter defibrillators:a systematic review.By Bryant J, Brodin H, Loveman E,Payne E, Clegg A.No. 37A trial of problem-solving bycommunity mental health nurses foranxiety, depression and life difficultiesamong general practice patients. TheCPN-GP study.By Kendrick T, Simons L,Mynors-Wallis L, Gray A, Lathlean J,Pickering R, et al.No. 38The causes and effects of sociodemographicexclusions from clinicaltrials.By Bartlett C, Doyal L, Ebrahim S,Davey P, Bachmann M, Egger M, et al.No. 39Is hydrotherapy cost-effective?A randomised controlled trial ofcombined hydrotherapy programmescompared with physiotherapy landtechniques in children with juvenileidiopathic arthritis.By Epps H, Ginnelly L, Utley M,Southwood T, Gallivan S, Sculpher M,et al.No. 40A randomised controlled trial andcost-effectiveness study of systematicscreening (targeted and totalpopulation screening) versus routinepractice for the detection of atrialfibrillation in people aged 65 and over.The SAFE study.By Hobbs FDR, Fitzmaurice DA,Mant J, Murray E, Jowett S, Bryan S,et al.No. 41Displaced intracapsular hip fracturesin fit, older people: a randomisedcomparison of reduction and fixation,bipolar hemiarthroplasty and total hiparthroplasty.By Keating JF, Grant A, Masson M,Scott NW, Forbes JF.No. 42Long-term outcome of cognitivebehaviour therapy clinical trials incentral Scotland.By Durham RC, Chambers JA,Power KG, Sharp DM, Macdonald RR,Major KA, et al.No. 43The effectiveness and cost-effectivenessof dual-chamber pacemakers comparedwith single-chamber pacemakers forbradycardia due to atrioventricularblock or sick sinus syndrome: systematicreview and economic evaluation.By Castelnuovo E, Stein K, Pitt M,Garside R, Payne E.No. 44Newborn screening for congenital heartdefects: a systematic review and costeffectivenessanalysis.By Knowles R, Griebsch I,Dezateux C, Brown J, Bull C, Wren C.No. 45The clinical and cost-effectiveness ofleft ventricular assist devices for endstageheart failure: a systematic reviewand economic evaluation.By Clegg AJ, Scott DA, Loveman E,Colquitt J, Hutchinson J, Royle P, et al.No. 46The effectiveness of the HeidelbergRetina Tomograph and laser diagnosticglaucoma scanning system (GDx) indetecting and monitoring glaucoma.By Kwartz AJ, Henson DB, HarperRA, Spencer AF, McLeod D.No. 47Clinical and cost-effectiveness ofautologous chondrocyte implantationfor cartilage defects in knee joints:systematic review and economicevaluation.By Clar C, Cummins E, McIntyre L,Thomas S, Lamb J, Bain L, et al.205© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to date206No. 48Systematic review of effectiveness ofdifferent treatments for childhoodretinoblastoma.By McDaid C, Hartley S, BagnallA-M, Ritchie G, Light K, Riemsma R.No. 49Towards evidence-based guidelinesfor the prevention of venousthromboembolism: systematicreviews of mechanical methods, oralanticoagulation, dextran and regionalanaesthesia as thromboprophylaxis.By Roderick P, Ferris G, Wilson K,Halls H, Jackson D, Collins R, et al.No. 50The effectiveness and cost-effectivenessof parent training/educationprogrammes for the treatmentof conduct disorder, includingoppositional defiant disorder, inchildren.By Dretzke J, Frew E, Davenport C,Barlow J, Stewart-Brown S, Sandercock J,et al.Volume 10, 2006No. 1The clinical and cost-effectiveness ofdonepezil, rivastigmine, galantamineand memantine for Alzheimer’sdisease.By Loveman E, Green C, Kirby J,Takeda A, Picot J, Payne E, et al.No. 2FOOD: a multicentre randomised trialevaluating feeding policies in patientsadmitted to hospital with a recentstroke.By Dennis M, Lewis S, Cranswick G,Forbes J.No. 3The clinical effectiveness and costeffectivenessof computed tomographyscreening for lung cancer: systematicreviews.By Black C, Bagust A, Boland A,Walker S, McLeod C, De Verteuil R, et al.No. 4A systematic review of the effectivenessand cost-effectiveness of neuroimagingassessments used to visualise the seizurefocus in people with refractory epilepsybeing considered for surgery.By Whiting P, Gupta R, Burch J,Mujica Mota RE, Wright K, Marson A,et al.No. 5Comparison of conference abstractsand presentations with full-text articlesin the health technology assessments ofrapidly evolving technologies.By Dundar Y, Dodd S, Dickson R,Walley T, Haycox A, Williamson PR.No. 6Systematic review and evaluationof methods of assessing urinaryincontinence.By Martin JL, Williams KS, AbramsKR, Turner DA, Sutton AJ, Chapple C,et al.No. 7The clinical effectiveness and costeffectivenessof newer drugs forchildren with epilepsy. A systematicreview.By Connock M, Frew E, Evans B-W,Bryan S, Cummins C, Fry-Smith A, et al.No. 8Surveillance of Barrett’s oesophagus:exploring the uncertainty throughsystematic review, expert workshop andeconomic modelling.By Garside R, Pitt M, Somerville M,Stein K, Price A, Gilbert N.No. 9Topotecan, pegylated liposomaldoxorubicin hydrochloride andpaclitaxel for second-line or subsequenttreatment of advanced ovarian cancer:a systematic review and economicevaluation.By Main C, Bojke L, Griffin S,Norman G, Barbieri M, Mather L, et al.No. 10Evaluation of molecular techniquesin prediction and diagnosisof cytomegalovirus disease inimmunocompromised patients.By Szczepura A, Westmoreland D,Vinogradova Y, Fox J, Clark M.No. 11Screening for thrombophilia in highrisksituations: systematic reviewand cost-effectiveness analysis. TheThrombosis: Risk and EconomicAssessment of ThrombophiliaScreening (TREATS) study.By Wu O, Robertson L, Twaddle S,Lowe GDO, Clark P, Greaves M, et al.No. 12A series of systematic reviews to informa decision analysis for sampling andtreating infected diabetic foot ulcers.By Nelson EA, O’Meara S, Craig D,Iglesias C, Golder S, Dalton J, et al.No. 13Randomised clinical trial, observationalstudy and assessment of costeffectivenessof the treatment ofvaricose veins (REACTIV trial).By Michaels JA, Campbell WB,Brazier JE, MacIntyre JB, Palfreyman SJ,Ratcliffe J, et al.No. 14The cost-effectiveness of screening fororal cancer in primary care.By Speight PM, Palmer S, Moles DR,Downer MC, Smith DH, Henriksson M,et al.No. 15Measurement of the clinical and costeffectivenessof non-invasive diagnostictesting strategies for deep veinthrombosis.By Goodacre S, Sampson F,Stevenson M, Wailoo A, Sutton A,Thomas S, et al.No. 16Systematic review of the effectivenessand cost-effectiveness of HealOzone ®for the treatment of occlusal pit/fissurecaries and root caries.By Brazzelli M, McKenzie L, FieldingS, Fraser C, Clarkson J, Kilonzo M, et al.No. 17Randomised controlled trials ofconventional antipsychotic versusnew atypical drugs, and new atypicaldrugs versus clozapine, in people withschizophrenia responding poorly to, orintolerant of, current drug treatment.By Lewis SW, Davies L, Jones PB,Barnes TRE, Murray RM, Kerwin R,et al.No. 18Diagnostic tests and algorithms usedin the investigation of haematuria:systematic reviews and economicevaluation.By Rodgers M, Nixon J, Hempel S,Aho T, Kelly J, Neal D, et al.No. 19Cognitive behavioural therapy inaddition to antispasmodic therapy forirritable bowel syndrome in primarycare: randomised controlled trial.By Kennedy TM, Chalder T,McCrone P, Darnley S, Knapp M,Jones RH, et al.No. 20A systematic review of theclinical effectiveness and costeffectivenessof enzyme replacementtherapies for Fabry’s disease andmucopolysaccharidosis type 1.By Connock M, Juarez-Garcia A,Frew E, Mans A, Dretzke J, Fry-Smith A,et al.No. 21Health benefits of antiviral therapy formild chronic hepatitis C: randomisedcontrolled trial and economicevaluation.By Wright M, Grieve R, Roberts J,Main J, Thomas HC, on behalf of theUK Mild Hepatitis C Trial Investigators.No. 22Pressure relieving support surfaces: arandomised evaluation.By Nixon J, Nelson EA, Cranny G,Iglesias CP, Hawkins K, Cullum NA, et al.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 23A systematic review and economicmodel of the effectiveness and costeffectivenessof methylphenidate,dexamfetamine and atomoxetinefor the treatment of attention deficithyperactivity disorder in children andadolescents.By King S, Griffin S, Hodges Z,Weatherly H, Asseburg C, Richardson G,et al.No. 24The clinical effectiveness and costeffectivenessof enzyme replacementtherapy for Gaucher’s disease: asystematic review.By Connock M, Burls A, Frew E,Fry-Smith A, Juarez-Garcia A, McCabe C,et al.No. 25Effectiveness and cost-effectivenessof salicylic acid and cryotherapy forcutaneous warts. An economic decisionmodel.By Thomas KS, Keogh-Brown MR,Chalmers JR, Fordham RJ, Holland RC,Armstrong SJ, et al.No. 26A systematic literature review of theeffectiveness of non-pharmacologicalinterventions to prevent wandering indementia and evaluation of the ethicalimplications and acceptability of theiruse.By Robinson L, Hutchings D, CornerL, Beyer F, Dickinson H, Vanoli A, et al.No. 27A review of the evidence on the effectsand costs of implantable cardioverterdefibrillator therapy in differentpatient groups, and modelling of costeffectivenessand cost–utility for thesegroups in a UK context.By Buxton M, Caine N, Chase D,Connelly D, Grace A, Jackson C, et al.No. 28Adefovir dipivoxil and pegylatedinterferon alfa-2a for the treatment ofchronic hepatitis B: a systematic reviewand economic evaluation.By Shepherd J, Jones J, Takeda A,Davidson P, Price A.No. 29An evaluation of the clinical and costeffectivenessof pulmonary arterycatheters in patient management inintensive care: a systematic review and arandomised controlled trial.By Harvey S, Stevens K, Harrison D,Young D, Brampton W, McCabe C, et al.No. 30Accurate, practical and cost-effectiveassessment of carotid stenosis in theUK.By Wardlaw JM, Chappell FM,Stevenson M, De Nigris E, Thomas S,Gillard J, et al.No. 31Etanercept and infliximab for thetreatment of psoriatic arthritis: asystematic review and economicevaluation.By Woolacott N, Bravo Vergel Y,Hawkins N, Kainth A, Khadjesari Z,Misso K, et al.No. 32The cost-effectiveness of testing forhepatitis C in former injecting drugusers.By Castelnuovo E, Thompson-CoonJ, Pitt M, Cramp M, Siebert U, Price A,et al.No. 33Computerised cognitive behaviourtherapy for depression and anxietyupdate: a systematic review andeconomic evaluation.By Kaltenthaler E, Brazier J,De Nigris E, Tumur I, Ferriter M,Beverley C, et al.No. 34Cost-effectiveness of using prognosticinformation to select women with breastcancer for adjuvant systemic therapy.By Williams C, Brunskill S, Altman D,Briggs A, Campbell H, Clarke M, et al.No. 35Psychological therapies includingdialectical behaviour therapy forborderline personality disorder: asystematic review and preliminaryeconomic evaluation.By Brazier J, Tumur I, Holmes M,Ferriter M, Parry G, Dent-Brown K, et al.No. 36Clinical effectiveness and costeffectivenessof tests for the diagnosisand investigation of urinary tractinfection in children: a systematicreview and economic model.By Whiting P, Westwood M, Bojke L,Palmer S, Richardson G, Cooper J, et al.No. 37Cognitive behavioural therapyin chronic fatigue syndrome: arandomised controlled trial of anoutpatient group programme.By O’Dowd H, Gladwell P, RogersCA, Hollinghurst S, Gregory A.No. 38A comparison of the cost-effectivenessof five strategies for the preventionof nonsteroidal anti-inflammatorydrug-induced gastrointestinal toxicity:a systematic review with economicmodelling.By Brown TJ, Hooper L, Elliott RA,Payne K, Webb R, Roberts C, et al.No. 39The effectiveness and cost-effectivenessof computed tomography screeningfor coronary artery disease: systematicreview.By Waugh N, Black C, Walker S,McIntyre L, Cummins E, Hillis G.No. 40What are the clinical outcome and costeffectivenessof endoscopy undertakenby nurses when compared with doctors?A Multi-Institution Nurse EndoscopyTrial (MINuET).By Williams J, Russell I, Durai D,Cheung W-Y, Farrin A, Bloor K, et al.No. 41The clinical and cost-effectiveness ofoxaliplatin and capecitabine for theadjuvant treatment of colon cancer:systematic review and economicevaluation.By Pandor A, Eggington S, Paisley S,Tappenden P, Sutcliffe P.No. 42A systematic review of the effectivenessof adalimumab, etanercept andinfliximab for the treatment ofrheumatoid arthritis in adults andan economic evaluation of their costeffectiveness.By Chen Y-F, Jobanputra P, Barton P,Jowett S, Bryan S, Clark W, et al.No. 43Telemedicine in dermatology: arandomised controlled trial.By Bowns IR, Collins K, Walters SJ,McDonagh AJG.No. 44Cost-effectiveness of cell salvage andalternative methods of minimisingperioperative allogeneic bloodtransfusion: a systematic review andeconomic model.By Davies L, Brown TJ, Haynes S,Payne K, Elliott RA, McCollum C.No. 45Clinical effectiveness and costeffectivenessof laparoscopic surgeryfor colorectal cancer: systematic reviewsand economic evaluation.By Murray A, Lourenco T, de VerteuilR, Hernandez R, Fraser C, McKinley A,et al.No. 46Etanercept and efalizumab for thetreatment of psoriasis: a systematicreview.By Woolacott N, Hawkins N,Mason A, Kainth A, Khadjesari Z, BravoVergel Y, et al.No. 47Systematic reviews of clinical decisiontools for acute abdominal pain.By Liu JLY, Wyatt JC, Deeks JJ,Clamp S, Keen J, Verde P, et al.No. 48Evaluation of the ventricular assistdevice programme in the UK.By Sharples L, Buxton M, Caine N,Cafferty F, Demiris N, Dyer M, et al.207© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to date208No. 49A systematic review and economicmodel of the clinical and costeffectivenessof immunosuppressivetherapy for renal transplantation inchildren.By Yao G, Albon E, Adi Y, Milford D,Bayliss S, Ready A, et al.No. 50Amniocentesis results: investigation ofanxiety. The ARIA trial.By Hewison J, Nixon J, Fountain J,Cocks K, Jones C, Mason G, et al.Volume 11, 2007No. 1Pemetrexed disodium for the treatmentof malignant pleural mesothelioma:a systematic review and economicevaluation.By Dundar Y, Bagust A, Dickson R,Dodd S, Green J, Haycox A, et al.No. 2A systematic review and economicmodel of the clinical effectivenessand cost-effectiveness of docetaxelin combination with prednisone orprednisolone for the treatment ofhormone-refractory metastatic prostatecancer.By Collins R, Fenwick E, Trowman R,Perard R, Norman G, Light K, et al.No. 3A systematic review of rapid diagnostictests for the detection of tuberculosisinfection.By Dinnes J, Deeks J, Kunst H,Gibson A, Cummins E, Waugh N, et al.No. 4The clinical effectiveness and costeffectivenessof strontium ranelate forthe prevention of osteoporotic fragilityfractures in postmenopausal women.By Stevenson M, Davis S, Lloyd-JonesM, Beverley C.No. 5A systematic review of quantitative andqualitative research on the role andeffectiveness of written informationavailable to patients about individualmedicines.By Raynor DK, BlenkinsoppA, Knapp P, Grime J, Nicolson DJ,Pollock K, et al.No. 6Oral naltrexone as a treatment forrelapse prevention in formerly opioiddependentdrug users: a systematicreview and economic evaluation.By Adi Y, Juarez-Garcia A, Wang D,Jowett S, Frew E, Day E, et al.No. 7Glucocorticoid-induced osteoporosis:a systematic review and cost–utilityanalysis.By Kanis JA, Stevenson M,McCloskey EV, Davis S, Lloyd-Jones M.No. 8Epidemiological, social, diagnostic andeconomic evaluation of populationscreening for genital chlamydialinfection.By Low N, McCarthy A, Macleod J,Salisbury C, Campbell R, Roberts TE,et al.No. 9Methadone and buprenorphine for themanagement of opioid dependence:a systematic review and economicevaluation.By Connock M, Juarez-Garcia A,Jowett S, Frew E, Liu Z, Taylor RJ, et al.No. 10Exercise Evaluation RandomisedTrial (EXERT): a randomised trialcomparing GP referral for leisurecentre-based exercise, community-basedwalking and advice only.By Isaacs AJ, Critchley JA, See TaiS, Buckingham K, Westley D, HarridgeSDR, et al.No. 11Interferon alfa (pegylated and nonpegylated)and ribavirin for thetreatment of mild chronic hepatitisC: a systematic review and economicevaluation.By Shepherd J, Jones J, Hartwell D,Davidson P, Price A, Waugh N.No. 12Systematic review and economicevaluation of bevacizumab andcetuximab for the treatment ofmetastatic colorectal cancer.By Tappenden P, Jones R, Paisley S,Carroll C.No. 13A systematic review and economicevaluation of epoetin alfa, epoetinbeta and darbepoetin alfa in anaemiaassociated with cancer, especially thatattributable to cancer treatment.By Wilson J, Yao GL, Raftery J,Bohlius J, Brunskill S, Sandercock J,et al.No. 14A systematic review and economicevaluation of statins for the preventionof coronary events.By Ward S, Lloyd Jones M, Pandor A,Holmes M, Ara R, Ryan A, et al.No. 15A systematic review of the effectivenessand cost-effectiveness of differentmodels of community-based respitecare for frail older people and theircarers.By Mason A, Weatherly H, SpilsburyK, Arksey H, Golder S, Adamson J, et al.No. 16Additional therapy for youngchildren with spastic cerebral palsy: arandomised controlled trial.By Weindling AM, Cunningham CC,Glenn SM, Edwards RT, Reeves DJ.No. 17Screening for type 2 diabetes: literaturereview and economic modelling.By Waugh N, Scotland G, McNameeP, Gillett M, Brennan A, Goyder E, et al.No. 18The effectiveness and cost-effectivenessof cinacalcet for secondaryhyperparathyroidism in end-stage renaldisease patients on dialysis: a systematicreview and economic evaluation.By Garside R, Pitt M, Anderson R,Mealing S, Roome C, Snaith A, et al.No. 19The clinical effectiveness and costeffectivenessof gemcitabine formetastatic breast cancer: a systematicreview and economic evaluation.By Takeda AL, Jones J, Loveman E,Tan SC, Clegg AJ.No. 20A systematic review of duplexultrasound, magnetic resonanceangiography and computedtomography angiography forthe diagnosis and assessment ofsymptomatic, lower limb peripheralarterial disease.By Collins R, Cranny G, Burch J,Aguiar-Ibáñez R, Craig D, Wright K,et al.No. 21The clinical effectiveness and costeffectivenessof treatments for childrenwith idiopathic steroid-resistantnephrotic syndrome: a systematicreview.By Colquitt JL, Kirby J, Green C,Cooper K, Trompeter RS.No. 22A systematic review of the routinemonitoring of growth in children ofprimary school age to identify growthrelatedconditions.By Fayter D, Nixon J, Hartley S,Rithalia A, Butler G, Rudolf M, et al.No. 23Systematic review of the effectiveness ofpreventing and treating Staphylococcusaureus carriage in reducing peritonealcatheter-related infections.By McCormack K, Rabindranath K,Kilonzo M, Vale L, Fraser C, McIntyre L,et al.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 24The clinical effectiveness and costof repetitive transcranial magneticstimulation versus electroconvulsivetherapy in severe depression: amulticentre pragmatic randomisedcontrolled trial and economic analysis.By McLoughlin DM, Mogg A, ErantiS, Pluck G, Purvis R, Edwards D, et al.No. 25A randomised controlled trial andeconomic evaluation of direct versusindirect and individual versus groupmodes of speech and language therapyfor children with primary languageimpairment.By Boyle J, McCartney E, Forbes J,O’Hare A.No. 26Hormonal therapies for early breastcancer: systematic review and economicevaluation.By Hind D, Ward S, De Nigris E,Simpson E, Carroll C, Wyld L.No. 27Cardioprotection against the toxiceffects of anthracyclines given tochildren with cancer: a systematicreview.By Bryant J, Picot J, Levitt G,Sullivan I, Baxter L, Clegg A.No. 28Adalimumab, etanercept and infliximabfor the treatment of ankylosingspondylitis: a systematic review andeconomic evaluation.By McLeod C, Bagust A, Boland A,Dagenais P, Dickson R, Dundar Y, et al.No. 29Prenatal screening and treatmentstrategies to prevent group Bstreptococcal and other bacterialinfections in early infancy: costeffectivenessand expected value ofinformation analyses.By Colbourn T, Asseburg C, Bojke L,Philips Z, Claxton K, Ades AE, et al.No. 30Clinical effectiveness and costeffectivenessof bone morphogeneticproteins in the non-healing of fracturesand spinal fusion: a systematic review.By Garrison KR, Donell S, Ryder J,Shemilt I, Mugford M, Harvey I, et al.No. 31A randomised controlled trial ofpostoperative radiotherapy followingbreast-conserving surgery in aminimum-risk older population. ThePRIME trial.By Prescott RJ, Kunkler IH, WilliamsLJ, King CC, Jack W, van der Pol M,et al.No. 32Current practice, accuracy, effectivenessand cost-effectiveness of the schoolentry hearing screen.By Bamford J, Fortnum H, BristowK, Smith J, Vamvakas G, Davies L, et al.No. 33The clinical effectiveness and costeffectivenessof inhaled insulin indiabetes mellitus: a systematic reviewand economic evaluation.By Black C, Cummins E, Royle P,Philip S, Waugh N.No. 34Surveillance of cirrhosis forhepatocellular carcinoma: systematicreview and economic analysis.By Thompson Coon J, Rogers G,Hewson P, Wright D, Anderson R,Cramp M, et al.No. 35The Birmingham RehabilitationUptake Maximisation Study (BRUM).Homebased compared with hospitalbasedcardiac rehabilitation in a multiethnicpopulation: cost-effectivenessand patient adherence.By Jolly K, Taylor R, Lip GYH,Greenfield S, Raftery J, Mant J, et al.No. 36A systematic review of the clinical,public health and cost-effectiveness ofrapid diagnostic tests for the detectionand identification of bacterial intestinalpathogens in faeces and food.By Abubakar I, Irvine L, Aldus CF,Wyatt GM, Fordham R, Schelenz S, et al.No. 37A randomised controlled trialexamining the longer-term outcomesof standard versus new antiepilepticdrugs. The SANAD trial.By Marson AG, Appleton R, BakerGA, Chadwick DW, Doughty J, Eaton B,et al.No. 38Clinical effectiveness and costeffectivenessof different modelsof managing long-term oral anticoagulationtherapy: a systematicreview and economic modelling.By Connock M, Stevens C, Fry-SmithA, Jowett S, Fitzmaurice D, Moore D,et al.No. 39A systematic review and economicmodel of the clinical effectivenessand cost-effectiveness of interventionsfor preventing relapse in people withbipolar disorder.By Soares-Weiser K, Bravo Vergel Y,Beynon S, Dunn G, Barbieri M, Duffy S,et al.No. 40Taxanes for the adjuvant treatment ofearly breast cancer: systematic reviewand economic evaluation.By Ward S, Simpson E, Davis S, HindD, Rees A, Wilkinson A.No. 41The clinical effectiveness and costeffectivenessof screening for openangle glaucoma: a systematic reviewand economic evaluation.By Burr JM, Mowatt G, HernándezR, Siddiqui MAR, Cook J, Lourenco T,et al.No. 42Acceptability, benefit and costs of earlyscreening for hearing disability: a studyof potential screening tests and models.By Davis A, Smith P, Ferguson M,Stephens D, Gianopoulos I.No. 43Contamination in trials of educationalinterventions.By Keogh-Brown MR, BachmannMO, Shepstone L, Hewitt C, Howe A,Ramsay CR, et al.No. 44Overview of the clinical effectiveness ofpositron emission tomography imagingin selected cancers.By Facey K, Bradbury I, Laking G,Payne E.No. 45The effectiveness and cost-effectivenessof carmustine implants andtemozolomide for the treatment ofnewly diagnosed high-grade glioma:a systematic review and economicevaluation.By Garside R, Pitt M, Anderson R,Rogers G, Dyer M, Mealing S, et al.No. 46Drug-eluting stents: a systematic reviewand economic evaluation.By Hill RA, Boland A, Dickson R,Dündar Y, Haycox A, McLeod C, et al.No. 47The clinical effectiveness andcost-effectiveness of cardiacresynchronisation (biventricular pacing)for heart failure: systematic review andeconomic model.By Fox M, Mealing S, Anderson R,Dean J, Stein K, Price A, et al.No. 48Recruitment to randomised trials:strategies for trial enrolment andparticipation study. The STEPS study.By Campbell MK, Snowdon C,Francis D, Elbourne D, McDonald AM,Knight R, et al.209© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to dateNo. 49Cost-effectiveness of functionalcardiac testing in the diagnosis andmanagement of coronary arterydisease: a randomised controlled trial.The CECaT trial.By Sharples L, Hughes V, Crean A,Dyer M, Buxton M, Goldsmith K, et al.No. 50Evaluation of diagnostic tests whenthere is no gold standard. A review ofmethods.By Rutjes AWS, ReitsmaJB, Coomarasamy A, Khan KS,Bossuyt PMM.No. 51Systematic reviews of the clinicaleffectiveness and cost-effectiveness ofproton pump inhibitors in acute uppergastrointestinal bleeding.By Leontiadis GI, SreedharanA, Dorward S, Barton P, Delaney B,Howden CW, et al.No. 52A review and critique of modelling inprioritising and designing screeningprogrammes.By Karnon J, Goyder E, TappendenP, McPhie S, Towers I, Brazier J, et al.No. 53An assessment of the impact of theNHS Health Technology AssessmentProgramme.By Hanney S, Buxton M, Green C,Coulson D, Raftery J.Volume 12, 2008No. 1A systematic review and economicmodel of switching fromnonglycopeptide to glycopeptideantibiotic prophylaxis for surgery.By Cranny G, Elliott R, Weatherly H,Chambers D, Hawkins N, Myers L, et al.No. 4Does befriending by trained lay workersimprove psychological well-being andquality of life for carers of peoplewith dementia, and at what cost? Arandomised controlled trial.By Charlesworth G, Shepstone L,Wilson E, Thalanany M, Mugford M,Poland F.No. 5A multi-centre retrospective cohortstudy comparing the efficacy, safetyand cost-effectiveness of hysterectomyand uterine artery embolisation forthe treatment of symptomatic uterinefibroids. The HOPEFUL study.By Hirst A, Dutton S, Wu O, BriggsA, Edwards C, Waldenmaier L, et al.No. 6Methods of prediction and preventionof pre-eclampsia: systematic reviews ofaccuracy and effectiveness literaturewith economic modelling.By Meads CA, Cnossen JS, Meher S,Juarez-Garcia A, ter Riet G, Duley L,et al.No. 7The use of economic evaluations inNHS decision-making: a review andempirical investigation.By Williams I, McIver S, Moore D,Bryan S.No. 8Stapled haemorrhoidectomy(haemorrhoidopexy) for the treatmentof haemorrhoids: a systematic reviewand economic evaluation.By Burch J, Epstein D, Baba-AkbariA, Weatherly H, Fox D, Golder S, et al.No. 9The clinical effectiveness of diabeteseducation models for Type 2 diabetes: asystematic review.By Loveman E, Frampton GK,Clegg AJ.No. 12The clinical effectiveness and costeffectivenessof central venous catheterstreated with anti-infective agents inpreventing bloodstream infections:a systematic review and economicevaluation.By Hockenhull JC, Dwan K, BolandA, Smith G, Bagust A, Dundar Y, et al.No. 13Stepped treatment of older adults onlaxatives. The STOOL trial.By Mihaylov S, Stark C, McColl E,Steen N, Vanoli A, Rubin G, et al.No. 14A randomised controlled trial ofcognitive behaviour therapy inadolescents with major depressiontreated by selective serotonin reuptakeinhibitors. The ADAPT trial.By Goodyer IM, Dubicka B,Wilkinson P, Kelvin R, Roberts C,Byford S, et al.No. 15The use of irinotecan, oxaliplatinand raltitrexed for the treatment ofadvanced colorectal cancer: systematicreview and economic evaluation.By Hind D, Tappenden P, Tumur I,Eggington E, Sutcliffe P, Ryan A.No. 16Ranibizumab and pegaptanib forthe treatment of age-related maculardegeneration: a systematic review andeconomic evaluation.By Colquitt JL, Jones J, Tan SC,Takeda A, Clegg AJ, Price A.No. 17Systematic review of the clinicaleffectiveness and cost-effectivenessof 64-slice or higher computedtomography angiography as analternative to invasive coronaryangiography in the investigation ofcoronary artery disease.By Mowatt G, Cummins E, Waugh N,Walker S, Cook J, Jia X, et al.210No. 2‘Cut down to quit’ with nicotinereplacement therapies in smokingcessation: a systematic review ofeffectiveness and economic analysis.By Wang D, Connock M, Barton P,Fry-Smith A, Aveyard P, Moore D.No. 3A systematic review of the effectivenessof strategies for reducing fracture riskin children with juvenile idiopathicarthritis with additional data on longtermrisk of fracture and cost of diseasemanagement.By Thornton J, Ashcroft D, O’Neill T,Elliott R, Adams J, Roberts C, et al.No. 10Payment to healthcare professionals forpatient recruitment to trials: systematicreview and qualitative study.By Raftery J, Bryant J, Powell J,Kerr C, Hawker S.No. 11Cyclooxygenase-2 selective nonsteroidalanti-inflammatory drugs(etodolac, meloxicam, celecoxib,rofecoxib, etoricoxib, valdecoxib andlumiracoxib) for osteoarthritis andrheumatoid arthritis: a systematicreview and economic evaluation.By Chen Y-F, Jobanputra P, Barton P,Bryan S, Fry-Smith A, Harris G, et al.No. 18Structural neuroimaging in psychosis:a systematic review and economicevaluation.By Albon E, Tsourapas A, Frew E,Davenport C, Oyebode F, Bayliss S, et al.No. 19Systematic review and economicanalysis of the comparativeeffectiveness of different inhaledcorticosteroids and their usage withlong-acting beta 2agonists for thetreatment of chronic asthma in adultsand children aged 12 years and over.By Shepherd J, Rogers G, AndersonR, Main C, Thompson-Coon J,Hartwell D, et al.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 20Systematic review and economicanalysis of the comparativeeffectiveness of different inhaledcorticosteroids and their usage withlong-acting beta 2agonists for thetreatment of chronic asthma in childrenunder the age of 12 years.By Main C, Shepherd J, Anderson R,Rogers G, Thompson-Coon J, Liu Z,et al.No. 21Ezetimibe for the treatment ofhypercholesterolaemia: a systematicreview and economic evaluation.By Ara R, Tumur I, Pandor A,Duenas A, Williams R, Wilkinson A, et al.No. 22Topical or oral ibuprofen for chronicknee pain in older people. The TOIBstudy.By Underwood M, Ashby D, CarnesD, Castelnuovo E, Cross P, Harding G,et al.No. 23A prospective randomised comparisonof minor surgery in primary andsecondary care. The MiSTIC trial.By George S, Pockney P, Primrose J,Smith H, Little P, Kinley H, et al.No. 24A review and critical appraisalof measures of therapist–patientinteractions in mental health settings.By Cahill J, Barkham M, Hardy G,Gilbody S, Richards D, Bower P, et al.No. 25The clinical effectiveness and costeffectivenessof screening programmesfor amblyopia and strabismus inchildren up to the age of 4–5 years:a systematic review and economicevaluation.By Carlton J, Karnon J, Czoski-Murray C, Smith KJ, Marr J.No. 26A systematic review of the clinicaleffectiveness and cost-effectivenessand economic modelling of minimalincision total hip replacementapproaches in the management ofarthritic disease of the hip.By de Verteuil R, Imamura M, Zhu S,Glazener C, Fraser C, Munro N, et al.No. 27A preliminary model-based assessmentof the cost–utility of a screeningprogramme for early age-relatedmacular degeneration.By Karnon J, Czoski-Murray C,Smith K, Brand C, Chakravarthy U,Davis S, et al.No. 28Intravenous magnesium sulphateand sotalol for prevention of atrialfibrillation after coronary arterybypass surgery: a systematic review andeconomic evaluation.By Shepherd J, Jones J, FramptonGK, Tanajewski L, Turner D, Price A.No. 29Absorbent products for urinary/faecalincontinence: a comparative evaluationof key product categories.By Fader M, Cottenden A, Getliffe K,Gage H, Clarke-O’Neill S, Jamieson K,et al.No. 30A systematic review of repetitivefunctional task practice with modellingof resource use, costs and effectiveness.By French B, Leathley M, Sutton C,McAdam J, Thomas L, Forster A, et al.No. 31The effectiveness and cost-effectivnessof minimal access surgery amongstpeople with gastro-oesophageal refluxdisease – a UK collaborative study. Thereflux trial.By Grant A, Wileman S, Ramsay C,Bojke L, Epstein D, Sculpher M, et al.No. 32Time to full publication of studies ofanti-cancer medicines for breast cancerand the potential for publication bias: ashort systematic review.By Takeda A, Loveman E, Harris P,Hartwell D, Welch K.No. 33Performance of screening tests forchild physical abuse in accident andemergency departments.By Woodman J, Pitt M, Wentz R,Taylor B, Hodes D, Gilbert RE.No. 34Curative catheter ablation in atrialfibrillation and typical atrial flutter:systematic review and economicevaluation.By Rodgers M, McKenna C, PalmerS, Chambers D, Van Hout S, Golder S,et al.No. 35Systematic review and economicmodelling of effectiveness and costutility of surgical treatments for menwith benign prostatic enlargement.By Lourenco T, Armstrong N, N’DowJ, Nabi G, Deverill M, Pickard R, et al.No. 36Immunoprophylaxis against respiratorysyncytial virus (RSV) with palivizumabin children: a systematic review andeconomic evaluation.By Wang D, Cummins C, Bayliss S,Sandercock J, Burls A.Volume 13, 2009No. 1Deferasirox for the treatment of ironoverload associated with regularblood transfusions (transfusionalhaemosiderosis) in patients sufferingwith chronic anaemia: a systematicreview and economic evaluation.By McLeod C, Fleeman N, KirkhamJ, Bagust A, Boland A, Chu P, et al.No. 2Thrombophilia testing in people withvenous thromboembolism: systematicreview and cost-effectiveness analysis.By Simpson EL, Stevenson MD,Rawdin A, Papaioannou D.No. 3Surgical procedures and non-surgicaldevices for the management of nonapnoeicsnoring: a systematic review ofclinical effects and associated treatmentcosts.By Main C, Liu Z, Welch K, WeinerG, Quentin Jones S, Stein K.No. 4Continuous positive airway pressuredevices for the treatment of obstructivesleep apnoea–hypopnoea syndrome: asystematic review and economic analysis.By McDaid C, Griffin S, Weatherly H,Durée K, van der Burgt M, van Hout S,Akers J, et al.No. 5Use of classical and novel biomarkersas prognostic risk factors for localisedprostate cancer: a systematic review.By Sutcliffe P, Hummel S, Simpson E,Young T, Rees A, Wilkinson A, et al.No. 6The harmful health effects ofrecreational ecstasy: a systematic reviewof observational evidence.By Rogers G, Elston J, Garside R,Roome C, Taylor R, Younger P, et al.No. 7Systematic review of the clinicaleffectiveness and cost-effectivenessof oesophageal Doppler monitoringin critically ill and high-risk surgicalpatients.By Mowatt G, Houston G, HernándezR, de Verteuil R, Fraser C, CuthbertsonB, et al.No. 8The use of surrogate outcomes inmodel-based cost-effectiveness analyses:a survey of UK Health TechnologyAssessment reports.By Taylor RS, Elston J.No. 9Controlling Hypertension andHypotension Immediately Post Stroke(CHHIPS) – a randomised controlledtrial.By Potter J, Mistri A, Brodie F,Chernova J, Wilson E, Jagger C, et al.211© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to date212No. 10Routine antenatal anti-D prophylaxisfor RhD-negative women: a systematicreview and economic evaluation.By Pilgrim H, Lloyd-Jones M, Rees A.No. 11Amantadine, oseltamivir and zanamivirfor the prophylaxis of influenza(including a review of existing guidanceno. 67): a systematic review andeconomic evaluation.By Tappenden P, Jackson R, CooperK, Rees A, Simpson E, Read R, et al.No. 12Improving the evaluation oftherapeutic interventions in multiplesclerosis: the role of new psychometricmethods.By Hobart J, Cano S.No. 13Treatment of severe ankle sprain: apragmatic randomised controlled trialcomparing the clinical effectivenessand cost-effectiveness of three types ofmechanical ankle support with tubularbandage. The CAST trial.By Cooke MW, Marsh JL, Clark M,Nakash R, Jarvis RM, Hutton JL, et al.,on behalf of the CAST trial group.No. 14Non-occupational postexposureprophylaxis for HIV: a systematicreview.By Bryant J, Baxter L, Hird S.No. 15Blood glucose self-monitoring in type 2diabetes: a randomised controlled trial.By Farmer AJ, Wade AN, French DP,Simon J, Yudkin P, Gray A, et al.No. 16How far does screening women fordomestic (partner) violence in differenthealth-care settings meet criteria fora screening programme? Systematicreviews of nine UK National ScreeningCommittee criteria.By Feder G, Ramsay J, Dunne D,Rose M, Arsene C, Norman R, et al.No. 17Spinal cord stimulation for chronicpain of neuropathic or ischaemicorigin: systematic review and economicevaluation.By Simpson, EL, Duenas A, HolmesMW, Papaioannou D, Chilcott J.No. 18The role of magnetic resonanceimaging in the identification ofsuspected acoustic neuroma: asystematic review of clinical and costeffectivenessand natural history.By Fortnum H, O’Neill C, Taylor R,Lenthall R, Nikolopoulos T, LightfootG, et al.No. 19Dipsticks and diagnostic algorithms inurinary tract infection: developmentand validation, randomised trial,economic analysis, observational cohortand qualitative study.By Little P, Turner S, Rumsby K,Warner G, Moore M, Lowes JA, et al.No. 20Systematic review of respite care in thefrail elderly.By Shaw C, McNamara R, AbramsK, Cannings-John R, Hood K, LongoM, et al.No. 21Neuroleptics in the treatment ofaggressive challenging behaviour forpeople with intellectual disabilities:a randomised controlled trial(NACHBID).By Tyrer P, Oliver-Africano P, RomeoR, Knapp M, Dickens S, Bouras N, et al.No. 22Randomised controlled trial todetermine the clinical effectivenessand cost-effectiveness of selectiveserotonin reuptake inhibitors plussupportive care, versus supportive carealone, for mild to moderate depressionwith somatic symptoms in primarycare: the THREAD (THREshold forAntiDepressant response) study.By Kendrick T, Chatwin J, Dowrick C,Tylee A, Morriss R, Peveler R, et al.No. 23Diagnostic strategies using DNA testingfor hereditary haemochromatosis inat-risk populations: a systematic reviewand economic evaluation.By Bryant J, Cooper K, Picot J, CleggA, Roderick P, Rosenberg W, et al.No. 24Enhanced external counterpulsationfor the treatment of stable angina andheart failure: a systematic review andeconomic analysis.By McKenna C, McDaid C,Suekarran S, Hawkins N, Claxton K,Light K, et al.No. 25Development of a decision supporttool for primary care management ofpatients with abnormal liver functiontests without clinically apparent liverdisease: a record-linkage populationcohort study and decision analysis(ALFIE).By Donnan PT, McLernon D, DillonJF, Ryder S, Roderick P, Sullivan F, et al.No. 26A systematic review of presumedconsent systems for deceased organdonation.By Rithalia A, McDaid C, SuekarranS, Norman G, Myers L, Sowden A.No. 27Paracetamol and ibuprofen for thetreatment of fever in children: thePITCH randomised controlled trial.By Hay AD, Redmond NM, CostelloeC, Montgomery AA, Fletcher M,Hollinghurst S, et al.No. 28A randomised controlled trial tocompare minimally invasive glucosemonitoring devices with conventionalmonitoring in the management ofinsulin-treated diabetes mellitus(MITRE).By Newman SP, Cooke D, Casbard A,Walker S, Meredith S, Nunn A, et al.No. 29Sensitivity analysis in economicevaluation: an audit of NICE currentpractice and a review of its use andvalue in decision-making.By Andronis L, Barton P, Bryan S.Suppl. 1Trastuzumab for the treatment ofprimary breast cancer in HER2-positivewomen: a single technology appraisal.By Ward S, Pilgrim H, Hind D.Docetaxel for the adjuvant treatmentof early node-positive breast cancer: asingle technology appraisal.By Chilcott J, Lloyd Jones M,Wilkinson A.The use of paclitaxel in themanagement of early stage breastcancer.By Griffin S, Dunn G, Palmer S,Macfarlane K, Brent S, Dyker A, et al.Rituximab for the first-line treatmentof stage III/IV follicular non-Hodgkin’slymphoma.By Dundar Y, Bagust A, Hounsome J,McLeod C, Boland A, Davis H, et al.Bortezomib for the treatment ofmultiple myeloma patients.By Green C, Bryant J, Takeda A,Cooper K, Clegg A, Smith A, et al.Fludarabine phosphate for the firstlinetreatment of chronic lymphocyticleukaemia.By Walker S, Palmer S, Erhorn S,Brent S, Dyker A, Ferrie L, et al.Erlotinib for the treatment of relapsednon-small cell lung cancer.By McLeod C, Bagust A, Boland A,Hockenhull J, Dundar Y, Proudlove C,et al.Cetuximab plus radiotherapy for thetreatment of locally advanced squamouscell carcinoma of the head and neck.By Griffin S, Walker S, Sculpher M,White S, Erhorn S, Brent S, et al.Infliximab for the treatment of adultswith psoriasis.By Loveman E, Turner D, HartwellD, Cooper K, Clegg A.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 30Psychological interventions forpostnatal depression: clusterrandomised trial and economicevaluation. The PoNDER trial.By Morrell CJ, Warner R, Slade P,Dixon S, Walters S, Paley G, et al.No. 31The effect of different treatmentdurations of clopidogrel in patientswith non-ST-segment elevation acutecoronary syndromes: a systematicreview and value of informationanalysis.By Rogowski R, Burch J, Palmer S,Craigs C, Golder S, Woolacott N.No. 32Systematic review and individualpatient data meta-analysis of diagnosisof heart failure, with modelling ofimplications of different diagnosticstrategies in primary care.By Mant J, Doust J, Roalfe A, BartonP, Cowie MR, Glasziou P, et al.No. 33A multicentre randomised controlledtrial of the use of continuous positiveairway pressure and non-invasivepositive pressure ventilation in the earlytreatment of patients presenting to theemergency department with severeacute cardiogenic pulmonary oedema:the 3CPO trial.By Gray AJ, Goodacre S, NewbyDE, Masson MA, Sampson F, DixonS, et al., on behalf of the 3CPO studyinvestigators.No. 34Early high-dose lipid-lowering therapyto avoid cardiac events: a systematicreview and economic evaluation.By Ara R, Pandor A, Stevens J, ReesA, Rafia R.No. 35Adefovir dipivoxil and pegylatedinterferon alpha for the treatmentof chronic hepatitis B: an updatedsystematic review and economicevaluation.By Jones J, Shepherd J, Baxter L,Gospodarevskaya E, Hartwell D, HarrisP, et al.No. 36Methods to identify postnataldepression in primary care: anintegrated evidence synthesis and valueof information analysis.By Hewitt CE, Gilbody SM, BrealeyS, Paulden M, Palmer S, Mann R, et al.No. 37A double-blind randomised placebocontrolledtrial of topical intranasalcorticosteroids in 4- to 11-year-oldchildren with persistent bilateral otitismedia with effusion in primary care.By Williamson I, Benge S, Barton S,Petrou S, Letley L, Fasey N, et al.No. 38The effectiveness and cost-effectivenessof methods of storing donated kidneysfrom deceased donors: a systematicreview and economic model.By Bond M, Pitt M, Akoh J, MoxhamT, Hoyle M, Anderson R.No. 39Rehabilitation of older patients: dayhospital compared with rehabilitationat home. A randomised controlled trial.By Parker SG, Oliver P, PenningtonM, Bond J, Jagger C, Enderby PM, et al.No. 40Breastfeeding promotion for infants inneonatal units: a systematic review andeconomic analysisBy Renfrew MJ, Craig D, Dyson L,McCormick F, Rice S, King SE, et al.No. 41The clinical effectiveness and costeffectivenessof bariatric (weight loss)surgery for obesity: a systematic review andeconomic evaluation.By Picot J, Jones J, Colquitt JL,Gospodarevskaya E, Loveman E, BaxterL, et al.No. 42Rapid testing for group B streptococcusduring labour: a test accuracy studywith evaluation of acceptability andcost-effectiveness.By Daniels J, Gray J, Pattison H,Roberts T, Edwards E, Milner P, et al.No. 43Screening to prevent spontaneouspreterm birth: systematic reviews ofaccuracy and effectiveness literaturewith economic modelling.By Honest H, Forbes CA, Durée KH,Norman G, Duffy SB, Tsourapas A, et al.No. 44The effectiveness and cost-effectivenessof cochlear implants for severe toprofound deafness in children andadults: a systematic review andeconomic model.By Bond M, Mealing S, Anderson R,Elston J, Weiner G, Taylor RS, et al.Suppl. 2Gemcitabine for the treatment ofmetastatic breast cancer.By Jones J, Takeda A, Tan SC,Cooper K, Loveman E, Clegg A.Varenicline in the management ofsmoking cessation: a single technologyappraisal.By Hind D, Tappenden P, Peters J,Kenjegalieva K.Alteplase for the treatment of acuteischaemic stroke: a single technologyappraisal.By Lloyd Jones M, Holmes M.Rituximab for the treatment ofrheumatoid arthritis.By Bagust A, Boland A, HockenhullJ, Fleeman N, Greenhalgh J, Dundar Y,et al.Omalizumab for the treatment ofsevere persistent allergic asthma.By Jones J, Shepherd J, Hartwell D,Harris P, Cooper K, Takeda A, et al.Rituximab for the treatment of relapsedor refractory stage III or IV follicularnon-Hodgkin’s lymphoma.By Boland A, Bagust A, HockenhullJ, Davis H, Chu P, Dickson R.Adalimumab for the treatment ofpsoriasis.By Turner D, Picot J, Cooper K,Loveman E.Dabigatran etexilate for the preventionof venous thromboembolism in patientsundergoing elective hip and kneesurgery: a single technology appraisal.By Holmes M, C Carroll C,Papaioannou D.Romiplostim for the treatmentof chronic immune or idiopathicthrombocytopenic purpura: a singletechnology appraisal.By Mowatt G, Boachie C, CrowtherM, Fraser C, Hernández R, Jia X, et al.Sunitinib for the treatment ofgastrointestinal stromal tumours: acritique of the submission from Pfizer.By Bond M, Hoyle M, Moxham T,Napier M, Anderson R.No. 45Vitamin K to prevent fractures inolder women: systematic review andeconomic evaluation.By Stevenson M, Lloyd-Jones M,Papaioannou D.No. 46The effects of biofeedback for thetreatment of essential hypertension: asystematic review.By Greenhalgh J, Dickson R,Dundar Y.No. 47A randomised controlled trial of theuse of aciclovir and/or prednisolone forthe early treatment of Bell’s palsy: theBELLS study.By Sullivan FM, Swan IRC, DonnanPT, Morrison JM, Smith BH, McKinstryB, et al.Suppl. 3Lapatinib for the treatment of HER2-overexpressing breast cancer.By Jones J, Takeda A, Picot J, vonKeyserlingk C, Clegg A.Infliximab for the treatment ofulcerative colitis.By Hyde C, Bryan S, Juarez-Garcia A,Andronis L, Fry-Smith A.213© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment reports published to dateRimonabant for the treatment ofoverweight and obese people.By Burch J, McKenna C, Palmer S,Norman G, Glanville J, Sculpher M, etal.Telbivudine for the treatment ofchronic hepatitis B infection.By Hartwell D, Jones J, Harris P,Cooper K.Entecavir for the treatment of chronichepatitis B infection.By Shepherd J, Gospodarevskaya E,Frampton G, Cooper, K.Febuxostat for the treatment ofhyperuricaemia in people with gout: asingle technology appraisal.By Stevenson M, Pandor A.Rivaroxaban for the prevention ofvenous thromboembolism: a singletechnology appraisal.By Stevenson M, Scope A, Holmes M,Rees A, Kaltenthaler E.Cetuximab for the treatment ofrecurrent and/or metastatic squamouscell carcinoma of the head and neck.By Greenhalgh J, Bagust A, BolandA, Fleeman N, McLeod C, Dundar Y,et al.Mifamurtide for the treatment ofosteosarcoma: a single technologyappraisal.By Pandor A, Fitzgerald P, StevensonM, Papaioannou D.Ustekinumab for the treatment ofmoderate to severe psoriasis.By Gospodarevskaya E, Picot J,Cooper K, Loveman E, Takeda A.No. 48Endovascular stents for abdominalaortic aneurysms: a systematic reviewand economic model.By Chambers D, Epstein D, Walker S,Fayter D, Paton F, Wright K, et al.No. 52The clinical effectiveness ofglucosamine and chondroitinsupplements in slowing or arrestingprogression of osteoarthritis of theknee: a systematic review and economicevaluation.By Black C, Clar C, Henderson R,MacEachern C, McNamee P, QuayyumZ, et al.No. 53Randomised preference trial ofmedical versus surgical termination ofpregnancy less than 14 weeks’ gestation(TOPS).By Robson SC, Kelly T, Howel D,Deverill M, Hewison J, Lie MLS, et al.No. 54Randomised controlled trial of the useof three dressing preparations in themanagement of chronic ulceration ofthe foot in diabetes.By Jeffcoate WJ, Price PE, PhillipsCJ, Game FL, Mudge E, Davies S, et al.No. 55VenUS II: a randomised controlled trialof larval therapy in the management ofleg ulcers.By Dumville JC, Worthy G, SoaresMO, Bland JM, Cullum N, Dowson C,et al.No. 56A prospective randomised controlledtrial and economic modelling ofantimicrobial silver dressings versusnon-adherent control dressings forvenous leg ulcers: the VULCAN trialBy Michaels JA, Campbell WB,King BM, MacIntyre J, Palfreyman SJ,Shackley P, et al.No. 60Colour vision testing for diabeticretinopathy: a systematic review ofdiagnostic accuracy and economicevaluation.By Rodgers M, Hodges R, HawkinsJ, Hollingworth W, Duffy S, McKibbinM, et al.No. 61Systematic review of the effectivenessand cost-effectiveness of weightmanagement schemes for the underfives: a short report.By Bond M, Wyatt K, Lloyd J, WelchK, Taylor R.No. 62Are adverse effects incorporated ineconomic models? An initial review ofcurrent practice.By Craig D, McDaid C, Fonseca T,Stock C, Duffy S, Woolacott N.Volume 14, 2010No. 1Multicentre randomised controlledtrial examining the cost-effectiveness ofcontrast-enhanced high field magneticresonance imaging in women withprimary breast cancer scheduled forwide local excision (COMICE).By Turnbull LW, Brown SR, OlivierC, Harvey I, Brown J, Drew P, et al.No. 2Bevacizumab, sorafenib tosylate,sunitinib and temsirolimus for renalcell carcinoma: a systematic review andeconomic evaluation.By Thompson Coon J, Hoyle M,Green C, Liu Z, Welch K, Moxham T,et al.214No. 49Clinical and cost-effectiveness ofepoprostenol, iloprost, bosentan,sitaxentan and sildenafil for pulmonaryarterial hypertension within theirlicensed indications: a systematic reviewand economic evaluation.By Chen Y-F, Jowett S, Barton P,Malottki K, Hyde C, Gibbs JSR, et al.No. 50Cessation of attention deficithyperactivity disorder drugsin the young (CADDY) – apharmacoepidemiological andqualitative study.By Wong ICK, Asherson P, Bilbow A,Clifford S, Coghill D, R DeSoysa R, et al.No. 51ARTISTIC: a randomised trial ofhuman papillomavirus (HPV) testing inprimary cervical screening.By Kitchener HC, Almonte M,Gilham C, Dowie R, Stoykova B, SargentA, et al.No. 57Communication of carrier statusinformation following universalnewborn screening for sickle celldisorders and cystic fibrosis: qualitativestudy of experience and practice.By Kai J, Ulph F, Cullinan T,Qureshi N.No. 58Antiviral drugs for the treatment ofinfluenza: a systematic review andeconomic evaluation.By Burch J, Paulden M, Conti S,Stock C, Corbett M, Welton NJ, et al.No. 59Development of a toolkit and glossaryto aid in the adaptation of healthtechnology assessment (HTA) reportsfor use in different contexts.By Chase D, Rosten C, Turner S,Hicks N, Milne R.No. 3The clinical effectiveness and costeffectivenessof testing for cytochromeP450 polymorphisms in patientswith schizophrenia treated withantipsychotics: a systematic review andeconomic evaluation.By Fleeman N, McLeod C, Bagust A,Beale S, Boland A, Dundar Y, et al.No. 4Systematic review of the clinicaleffectiveness and cost-effectiveness ofphotodynamic diagnosis and urinebiomarkers (FISH, ImmunoCyt,NMP22) and cytology for the detectionand follow-up of bladder cancer.By Mowatt G, Zhu S, Kilonzo M,Boachie C, Fraser C, Griffiths TRL, et al.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8No. 5Effectiveness and cost-effectiveness ofarthroscopic lavage in the treatmentof osteoarthritis of the knee: a mixedmethods study of the feasibilityof conducting a surgical placebocontrolledtrial (the KORAL study).By Campbell MK, Skea ZC,Sutherland AG, Cuthbertson BH,Entwistle VA, McDonald AM, et al.No. 6A randomised 2 × 2 trial ofcommunity versus hospital pulmonaryrehabilitation, followed by telephone orconventional follow-up.By Waterhouse JC, Walters SJ,Oluboyede Y, Lawson RA.No. 7The effectiveness and costeffectivenessof behaviouralinterventions for the prevention ofsexually transmitted infections in youngpeople aged 13–19: a systematic reviewand economic evaluation.By Shepherd J, Kavanagh J, Picot J,Cooper K, Harden A, Barnett-Page E,et al.215© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8Health Technology AssessmentprogrammeDirector,Professor Tom Walley,Director, NIHR HTAprogramme, Professor ofClinical Pharmacology,University of LiverpoolDeputy Director,Professor Jon Nicholl,Director, Medical Care ResearchUnit, University of SheffieldMembersPrioritisation Strategy GroupChair,Professor Tom Walley,Director, NIHR HTAprogramme, Professor ofClinical Pharmacology,University of LiverpoolDeputy Chair,Professor Jon Nicholl,Director, Medical Care ResearchUnit, University of SheffieldDr Bob Coates,Consultant Advisor, NETSCC,HTADr Andrew Cook,Consultant Advisor, NETSCC,HTADr Peter Davidson,Director of Science Support,NETSCC, HTAProfessor Robin E Ferner,Consultant Physician andDirector, West Midlands Centrefor Adverse Drug Reactions,City Hospital NHS Trust,BirminghamProfessor Paul Glasziou,Professor of Evidence-BasedMedicine, University of OxfordDr Nick Hicks,Director of NHS Support,NETSCC, HTADr Edmund Jessop,Medical Adviser, NationalSpecialist, NationalCommissioning Group (NCG),Department of Health, LondonMs Lynn Kerridge,Chief Executive Officer,NETSCC and NETSCC, HTADr Ruairidh Milne,Director of Strategy andDevelopment, NETSCCMs Kay Pattison,Section Head, NHS R&DProgramme, Department ofHealthMs Pamela Young,Specialist Programme Manager,NETSCC, HTAMembersProgramme Director,Professor Tom Walley,Director, NIHR HTAprogramme, Professor ofClinical Pharmacology,University of LiverpoolChair,Professor Jon Nicholl,Director, Medical Care ResearchUnit, University of SheffieldDeputy Chair,Dr Andrew Farmer,Senior Lecturer in GeneralPractice, Department ofPrimary Health Care,University of OxfordProfessor Ann Ashburn,Professor of Rehabilitationand Head of Research,Southampton General HospitalObserversHTA Commissioning BoardProfessor Deborah Ashby,Professor of Medical Statistics,Queen Mary, University ofLondonProfessor John Cairns,Professor of Health Economics,London School of Hygiene andTropical MedicineProfessor Peter Croft,Director of Primary CareSciences Research Centre, KeeleUniversityProfessor Nicky Cullum,Director of Centre for Evidence-Based Nursing, University ofYorkProfessor Jenny Donovan,Professor of Social Medicine,University of BristolProfessor Steve Halligan,Professor of GastrointestinalRadiology, University CollegeHospital, LondonProfessor Freddie Hamdy,Professor of Urology,University of SheffieldProfessor Allan House,Professor of Liaison Psychiatry,University of LeedsDr Martin J Landray,Reader in Epidemiology,Honorary Consultant Physician,Clinical Trial Service Unit,University of OxfordProfessor Stuart Logan,Director of Health & SocialCare Research, The PeninsulaMedical School, Universities ofExeter and PlymouthDr Rafael Perera,Lecturer in Medical Statisitics,Department of Primary HealthCare, Univeristy of OxfordProfessor Ian Roberts,Professor of Epidemiology &Public Health, London Schoolof Hygiene and TropicalMedicineProfessor Mark Sculpher,Professor of Health Economics,University of YorkProfessor Helen Smith,Professor of Primary Care,University of BrightonProfessor Kate Thomas,Professor of Complementary &Alternative Medicine Research,University of LeedsProfessor David JohnTorgerson,Director of York Trials Unit,University of YorkProfessor Hywel Williams,Professor of Dermato-Epidemiology, University ofNottinghamMs Kay Pattison,Section Head, NHS R&DProgramme, Department ofHealthDr Morven Roberts,Clinical Trials Manager,Medical Research Council217© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment programmeMembersDiagnostic Technologies & Screening PanelChair,Professor Paul Glasziou,Professor of Evidence-BasedMedicine, University of OxfordDeputy Chair,Dr David Elliman,Consultant Paediatrician andHonorary Senior Lecturer,Great Ormond Street Hospital,LondonProfessor Judith E Adams,Consultant Radiologist,Manchester Royal Infirmary,Central Manchester &Manchester Children’sUniversity Hospitals NHS Trust,and Professor of DiagnosticRadiology, Imaging Scienceand Biomedical Engineering,Cancer & Imaging Sciences,University of ManchesterMs Jane Bates,Consultant UltrasoundPractitioner, UltrasoundDepartment, Leeds TeachingHospital NHS TrustDr Stephanie Dancer,Consultant Microbiologist,Hairmyres Hospital, EastKilbrideProfessor Glyn Elwyn,Primary Medical Care ResearchGroup, Swansea Clinical School,University of WalesDr Ron Gray,Consultant ClinicalEpidemiologist, Departmentof Public Health, University ofOxfordProfessor Paul D Griffiths,Professor of Radiology,University of SheffieldDr Jennifer J Kurinczuk,Consultant ClinicalEpidemiologist, NationalPerinatal Epidemiology Unit,OxfordDr Susanne M Ludgate,Medical Director, Medicines &Healthcare Products RegulatoryAgency, LondonDr Anne Mackie,Director of Programmes, UKNational Screening CommitteeDr Michael Millar,Consultant Senior Lecturer inMicrobiology, Barts and TheLondon NHS Trust, RoyalLondon HospitalMr Stephen Pilling,Director, Centre for Outcomes,Research & Effectiveness,Joint Director, NationalCollaborating Centre forMental Health, UniversityCollege LondonMrs Una Rennard,Service User RepresentativeDr Phil Shackley,Senior Lecturer in HealthEconomics, School ofPopulation and HealthSciences, University ofNewcastle upon TyneDr W Stuart A Smellie,Consultant in ChemicalPathology, Bishop AucklandGeneral HospitalDr Nicholas Summerton,Consultant Clinical and PublicHealth Advisor, NICEMs Dawn Talbot,Service User RepresentativeDr Graham Taylor,Scientific Advisor, RegionalDNA Laboratory, St James’sUniversity Hospital, LeedsProfessor Lindsay WilsonTurnbull,Scientific Director of theCentre for Magnetic ResonanceInvestigations and YCRProfessor of Radiology, HullRoyal InfirmaryObserversDr Tim Elliott,Team Leader, CancerScreening, Department ofHealthDr Catherine Moody,Programme Manager,Neuroscience and MentalHealth BoardDr Ursula Wells,Principal Research Officer,Department of HealthMembersPharmaceuticals PanelChair,Professor Robin Ferner,Consultant Physician andDirector, West Midlands Centrefor Adverse Drug Reactions,City Hospital NHS Trust,BirminghamDeputy Chair,Professor Imti Choonara,Professor in Child Health,University of NottinghamMrs Nicola Carey,Senior Research Fellow,School of Health and SocialCare, The University ofReadingMr John Chapman,Service User RepresentativeObserversDr Peter Elton,Director of Public Health,Bury Primary Care TrustDr Ben Goldacre,Research Fellow, Division ofPsychological Medicine andPsychiatry, King’s CollegeLondonMrs Barbara Greggains,Service User RepresentativeDr Bill Gutteridge,Medical Adviser, LondonStrategic Health AuthorityDr Dyfrig Hughes,Reader in Pharmacoeconomicsand Deputy Director, Centrefor Economics and Policy inHealth, IMSCaR, BangorUniversityProfessor Jonathan Ledermann,Professor of Medical Oncologyand Director of the CancerResearch UK and UniversityCollege London Cancer TrialsCentreDr Yoon K Loke,Senior Lecturer in ClinicalPharmacology, University ofEast AngliaProfessor Femi Oyebode,Consultant Psychiatristand Head of Department,University of BirminghamDr Andrew Prentice,Senior Lecturer and ConsultantObstetrician and Gynaecologist,The Rosie Hospital, Universityof CambridgeDr Martin Shelly,General Practitioner, Leeds,and Associate Director, NHSClinical Governance SupportTeam, LeicesterDr Gillian Shepherd,Director, Health and ClinicalExcellence, Merck Serono LtdMrs Katrina Simister,Assistant Director NewMedicines, National PrescribingCentre, LiverpoolMr David Symes,Service User RepresentativeDr Lesley Wise,Unit Manager,PharmacoepidemiologyResearch Unit, VRMM,Medicines & HealthcareProducts Regulatory Agency218Ms Kay Pattison,Section Head, NHS R&DProgramme, Department ofHealthMr Simon Reeve,Head of Clinical and Cost-Effectiveness, Medicines,Pharmacy and Industry Group,Department of HealthDr Heike Weber,Programme Manager,Medical Research CouncilDr Ursula Wells,Principal Research Officer,Department of HealthCurrent and past membership details of all HTA programme ‘committees’ are available from the HTA website (www.hta.ac.uk)

DOI: 10.3310/hta14080 Health Technology Assessment 2010; Vol. 14: No. 8MembersTherapeutic Procedures PanelChair,Dr John C Pounsford,Consultant Physician, NorthBristol NHS TrustDeputy Chair,Professor Scott Weich,Professor of Psychiatry, Divisionof Health in the Community,University of Warwick,CoventryProfessor Jane Barlow,Professor of Public Health inthe Early Years, Health SciencesResearch Institute, WarwickMedical School, CoventryMs Maree Barnett,Acting Branch Head of VascularProgramme, Department ofHealthMrs Val Carlill,Service User RepresentativeMrs Anthea De Barton-Watson,Service User RepresentativeMr Mark Emberton,Senior Lecturer in OncologicalUrology, Institute of Urology,University College Hospital,LondonProfessor Steve Goodacre,Professor of EmergencyMedicine, University ofSheffieldProfessor Christopher Griffiths,Professor of Primary Care, Bartsand The London School ofMedicine and DentistryMr Paul Hilton,Consultant Gynaecologistand Urogynaecologist, RoyalVictoria Infirmary, Newcastleupon TyneProfessor Nicholas James,Professor of Clinical Oncology,University of Birmingham,and Consultant in ClinicalOncology, Queen ElizabethHospitalDr Peter Martin,Consultant Neurologist,Addenbrooke’s Hospital,CambridgeDr Kate Radford,Senior Lecturer (Research),Clinical Practice ResearchUnit, University of CentralLancashire, PrestonMr Jim ReeceService User RepresentativeDr Karen Roberts,Nurse Consultant, Dunston HillHospital CottagesObserversDr Phillip Leech,Principal Medical Officer forPrimary Care, Department ofHealthMs Kay Pattison,Section Head, NHS R&DProgramme, Department ofHealthMembersChair,Dr Edmund Jessop,Medical Adviser, NationalSpecialist, NationalCommissioning Group (NCG),LondonDeputy Chair,Dr David Pencheon,Director, NHS SustainableDevelopment Unit, CambridgeDr Elizabeth Fellow-Smith,Medical Director, West LondonMental Health Trust, MiddlesexDr Morven Roberts,Clinical Trials Manager,Medical Research CouncilProfessor Tom Walley,Director, NIHR HTAprogramme, Professor ofClinical Pharmacology,University of LiverpoolDisease Prevention PanelDr John Jackson,General Practitioner, ParkwayMedical Centre, Newcastleupon TyneProfessor Mike Kelly,Director, Centre for PublicHealth Excellence, NICE,LondonDr Chris McCall,General Practitioner, TheHadleigh Practice, CorfeMullen, DorsetMs Jeanett Martin,Director of Nursing, BarnDocLimited, Lewisham PrimaryCare TrustDr Julie Mytton,Locum Consultant in PublicHealth Medicine, BristolPrimary Care TrustMiss Nicky Mullany,Service User RepresentativeProfessor Ian Roberts,Professor of Epidemiology andPublic Health, London Schoolof Hygiene & Tropical MedicineProfessor Ken Stein,Senior Clinical Lecturer inPublic Health, University ofExeterDr Ursula Wells,Principal Research Officer,Department of HealthDr Kieran Sweeney,Honorary Clinical SeniorLecturer, Peninsula Collegeof Medicine and Dentistry,Universities of Exeter andPlymouthProfessor Carol Tannahill,Glasgow Centre for PopulationHealthProfessor Margaret Thorogood,Professor of Epidemiology,University of Warwick MedicalSchool, CoventryObserversMs Christine McGuire,Research & Development,Department of HealthDr Caroline Stone,Programme Manager, MedicalResearch Council219© 2010 Queen’s Printer and Controller of HMSO. All rights reserved.

Health Technology Assessment programmeMembersExpert Advisory Network220Professor Douglas Altman,Professor of Statistics inMedicine, Centre for Statisticsin Medicine, University ofOxfordProfessor John Bond,Professor of Social Gerontology& Health Services Research,University of Newcastle uponTyneProfessor Andrew Bradbury,Professor of Vascular Surgery,Solihull Hospital, BirminghamMr Shaun Brogan,Chief Executive, RidgewayPrimary Care Group, AylesburyMrs Stella Burnside OBE,Chief Executive, Regulationand Improvement Authority,BelfastMs Tracy Bury,Project Manager, WorldConfederation for PhysicalTherapy, LondonProfessor Iain T Cameron,Professor of Obstetrics andGynaecology and Head of theSchool of Medicine, Universityof SouthamptonDr Christine Clark,Medical Writer and ConsultantPharmacist, RossendaleProfessor Collette Clifford,Professor of Nursing andHead of Research, TheMedical School, University ofBirminghamProfessor Barry Cookson,Director, Laboratory of HospitalInfection, Public HealthLaboratory Service, LondonDr Carl Counsell,Clinical Senior Lecturer inNeurology, University ofAberdeenProfessor Howard Cuckle,Professor of ReproductiveEpidemiology, Departmentof Paediatrics, Obstetrics &Gynaecology, University ofLeedsDr Katherine Darton,Information Unit, MIND – TheMental Health Charity, LondonProfessor Carol Dezateux,Professor of PaediatricEpidemiology, Institute of ChildHealth, LondonMr John Dunning,Consultant CardiothoracicSurgeon, Papworth HospitalNHS Trust, CambridgeMr Jonothan Earnshaw,Consultant Vascular Surgeon,Gloucestershire Royal Hospital,GloucesterProfessor Martin Eccles,Professor of ClinicalEffectiveness, Centre for HealthServices Research, University ofNewcastle upon TyneProfessor Pam Enderby,Dean of Faculty of Medicine,Institute of General Practiceand Primary Care, University ofSheffieldProfessor Gene Feder,Professor of Primary CareResearch & Development,Centre for Health Sciences,Barts and The London Schoolof Medicine and DentistryMr Leonard R Fenwick,Chief Executive, FreemanHospital, Newcastle upon TyneMrs Gillian Fletcher,Antenatal Teacher and Tutorand President, NationalChildbirth Trust, HenfieldProfessor Jayne Franklyn,Professor of Medicine,University of BirminghamMr Tam Fry,Honorary Chairman, ChildGrowth Foundation, LondonProfessor Fiona Gilbert,Consultant Radiologist andNCRN Member, University ofAberdeenProfessor Paul Gregg,Professor of OrthopaedicSurgical Science, South TeesHospital NHS TrustBec Hanley,Co-director, TwoCan Associates,West SussexDr Maryann L Hardy,Senior Lecturer, University ofBradfordMrs Sharon Hart,Healthcare ManagementConsultant, ReadingProfessor Robert E Hawkins,CRC Professor and Directorof Medical Oncology, ChristieCRC Research Centre,Christie Hospital NHS Trust,ManchesterProfessor Richard Hobbs,Head of Department of PrimaryCare & General Practice,University of BirminghamProfessor Alan Horwich,Dean and Section Chairman,The Institute of CancerResearch, LondonProfessor Allen Hutchinson,Director of Public Health andDeputy Dean of ScHARR,University of SheffieldProfessor Peter Jones,Professor of Psychiatry,University of Cambridge,CambridgeProfessor Stan Kaye,Cancer Research UK Professorof Medical Oncology, RoyalMarsden Hospital and Instituteof Cancer Research, SurreyDr Duncan Keeley,General Practitioner (Dr Burch& Ptnrs), The Health Centre,ThameDr Donna Lamping,Research Degrees ProgrammeDirector and Reader inPsychology, Health ServicesResearch Unit, London Schoolof Hygiene and TropicalMedicine, LondonMr George Levvy,Chief Executive, MotorNeurone Disease Association,NorthamptonProfessor James Lindesay,Professor of Psychiatry for theElderly, University of LeicesterProfessor Julian Little,Professor of Human GenomeEpidemiology, University ofOttawaProfessor Alistaire McGuire,Professor of Health Economics,London School of EconomicsProfessor Rajan Madhok,Medical Director and Directorof Public Health, Directorateof Clinical Strategy & PublicHealth, North & East Yorkshire& Northern LincolnshireHealth Authority, YorkProfessor Alexander Markham,Director, Molecular MedicineUnit, St James’s UniversityHospital, LeedsDr Peter Moore,Freelance Science Writer,AshteadDr Andrew Mortimore,Public Health Director,Southampton City PrimaryCare TrustDr Sue Moss,Associate Director, CancerScreening Evaluation Unit,Institute of Cancer Research,SuttonProfessor Miranda Mugford,Professor of Health Economicsand Group Co-ordinator,University of East AngliaProfessor Jim Neilson,Head of School of Reproductive& Developmental Medicineand Professor of Obstetricsand Gynaecology, University ofLiverpoolMrs Julietta Patnick,National Co-ordinator, NHSCancer Screening Programmes,SheffieldProfessor Robert Peveler,Professor of Liaison Psychiatry,Royal South Hants Hospital,SouthamptonProfessor Chris Price,Director of Clinical Research,Bayer Diagnostics Europe,Stoke PogesProfessor William Rosenberg,Professor of Hepatologyand Consultant Physician,University of SouthamptonProfessor Peter Sandercock,Professor of Medical Neurology,Department of ClinicalNeurosciences, University ofEdinburghDr Susan Schonfield,Consultant in Public Health,Hillingdon Primary Care Trust,MiddlesexDr Eamonn Sheridan,Consultant in Clinical Genetics,St James’s University Hospital,LeedsDr Margaret Somerville,Director of Public HealthLearning, Peninsula MedicalSchool, University of PlymouthProfessor Sarah Stewart-Brown,Professor of Public Health,Division of Health in theCommunity, University ofWarwick, CoventryProfessor Ala Szczepura,Professor of Health ServiceResearch, Centre for HealthServices Studies, University ofWarwick, CoventryMrs Joan Webster,Consumer Member, SouthernDerbyshire Community HealthCouncilProfessor Martin Whittle,Clinical Co-director, NationalCo-ordinating Centre forWomen’s and Children’sHealth, LymingtonCurrent and past membership details of all HTA programme ‘committees’ are available from the HTA website (www.hta.ac.uk)

FeedbackThe HTA programme and the authors would like to knowyour views about this report.The Correspondence Page on the HTA website(www.hta.ac.uk) is a convenient way to publishyour comments. If you prefer, you can send your commentsto the address below, telling us whether you would likeus to transfer them to the website.We look forward to hearing from you.NETSCC, Health Technology AssessmentAlpha HouseUniversity of Southampton Science ParkSouthampton SO16 7NS, UKEmail: hta@hta.ac.ukwww.hta.ac.uk ISSN 1366-5278