Information Retrieval - The Campbell Collaboration

Main Searches: Decisions– Preparation of Search Strategies• What are the key concepts to be searched?• How are these represented in each discipline?• What are their related terms?• How are these key concepts represented in the controlledvocabulary within each database to be searched?C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Sample Research QuestionThe purpose of this review is to assess theimpacts of MST on out-of-home livingarrangements, crime and delinquency, andbehavioral and psychosocial outcomes foryouth and familiesC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Using a Thesaurus1. From the research question, determine the main concepts to besearched (usually there are three):– Multisystemic therapy– Multisystemic treatment– Research or evaluation or outcome2. Consult the main database to be searched.3. Look up each concept in the thesaurus for this database.– A thesaurus is an alphabetical listing of the controlled vocabulary (or descriptors) usedwithin a subject database– A hierarchical arrangement is used so that Broader, Narrower and Related headings maybe discovered– The user will be sent from invalid headings to valid headings.C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

• Construction of the Search StatementsMain Searches: Decisions– What terms should be searched as descriptors or as “free text”?– What Boolean operators should be used?– Where should truncation characters be used? (e.g. parent* will retrieveparent, parents, parental)– What limiting features are available to narrow results? (e.g. use ofPublication Type codes)?– What time period should be searched?C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Boolean OperatorsAND: Both terms must be present in order for a record to beretrieved. Used to combine different concepts. e.g parent participation AND achievementOR: Either term may be present in order for a record to be retrieved.Used to search for related terms or synonyms. e.g. parent OR familyNOT: Used between two terms to ensure that the second term will notappear in any of the results. e.g. literacy NOT adult(Parental involvement OR parent participation) AND academicachievement AND (elementary OR primary education)C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Next StepsRepeat these steps for each database to besearched.C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Electronic Search Strategies: Potential Problems• Publication bias– Known difference in statistical significance of published vs.unpublished studies• Hence, don’t rely on electronic searches alone• English language bias– Tendency for studies published in English language journals tohave stronger findings than studies published in other languages• Hence, don’t rely on works published in English alone• Inconsistent usage of terms across disciplines• Inconsistent labeling of methodologyC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Generic vs. Specialized Search Engines: Specific vs.sensitive searches• Generic search engines (e.g., Google) can sometimes be a goodresource for unpublished material– Far less efficient thoughPrimary BenefitPrimary DrawbackSpecialized db(Specific)↓ false hits↓ retrieval ofunpublished workGeneric searchengine(Sensitive)↑ retrieval ofunpublished work↑ false hits(dramatically)C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Inconsistent Labeling of Methodology• Reviewers only interested in experiments will sometimes usesearch strategies aimed at capturing design(After-school or afterschool)AND(random or evaluation or RCT)• Several studies have shown that this strategy will missrelevant studies (25-67%!)C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Hope for the Future• CONSORT statement– Specific elements of studies that ought to be reported• Standardizes terms• Adopted by major medical journals and the APA• Structured abstracts– Specific effort aimed at standardizing the information contained inabstractsC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Searching the Literature: Finding the Grey literature bySnowballing• Contacts & Solicitations– Email correspondence to active researchers– Postings on relevant listservs– Email to organizations that might conduct relevant research• Reviewer must be aware of these (again, subject matter expertise is critical)• Reference Treeing– Search the reference sections of all relevant studies and reviews for citationsto work not found• Databases of unpublished and ongoing research– Usually topic specific• Dissertations• Citation Searches– SSCI & Google ScholarC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Social Science Citation Index & Google Scholar• Can find …– works that cite a particular reference– works by a particular author– Note that for SSCI, the journal must be indexed by ThompsonScientific (about 4000, or roughly 20% of all journals in allacademic disciplines)– Example on next slide looks upC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Example of Non-electronic Search• Cooper et al. (2002) on Modified School Calendars– Wrote to the 100 largest school districts in the U.S.– Contacted all authors of included studies– Reference tree for all included studiesC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Information Retrieval:A Continuous Process– Preliminary Searches• Supports beginning steps: Definition of key concepts & research question• Use of standard reference tools and broad searches for review articles and keyprimary studies– Main Searches• Identification of primary studies through searches of online databases, printedindices, Internet, branching, hand-searches• Most difficult given a number of challenges– Final Searches• Occurs towards the end of the Review Process• Refine search terms and update original searchesC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Additional Databases• Consult with the C2 Information Retrieval MethodsGroup– Learn availability of Trials Search Coordinator or IRMG liaisonC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Managing Your Results• Export the results– Save as a Text file• Import into a bibliographic management software:– RefWorks,– Reference Manager,– EndNotes• Edit your inhouse database– Add Source code for each database searched (e.g ERIC1, PsycINFO1…)– Add notes to the records (e.g.includes vs excludes)• Compile a Search History document listing the original search strategies– Use of IRMG/ECG worksheet (to come)C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Key Decisions in Literature Retrieval• Important to have inter-rater agreement on key decisions– Does the study look like it might be relevant?• Based on reading titles and abstracts– If yes, retrieve full text of article– Is the study in fact eligible for inclusion?• Based on information in the full text of study– If yes, code study• When in doubt, double code– At least two trained raters working independently• Generally I double code everythingC2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org

Information Retrieval: Wrap Up“Shoestring-budget information retrieval is likely to introduce bias, and should be avoided.” (IRPolicy Brief, 2004)• Importance of information retrieval process– Not a “one-shot”deal– Requires expertise in the planning and implementation of searches– Consulting with the IRMG or an Information Specialist is highly recommended (Somecoordinating groups have Trial Search Coordinators)• Use of bibliographic management software– Store, manage and organize results• Ability to replicate review– Documentation of entire process, including search strategies used for each database,decisions taken, etc.C2 Training Materials – Oslo – June 2010 www.campbellcollaboration.org