Reproducibility in searching for grey literature

From HLWIKI Canada
Jump to: navigation, search
“The color of truth is grey” – André Gide
Are you interested in contributing to HLWIKI International? contact:

To browse other articles on a range of HSL topics, see the A-Z index.


Last Update

  • Updated.jpg 23 May 2017


See also Grey literature | Grey data ("hard to find" data) | Grey information and data | Grey literature searching in medicine | Grey literature - part II | Snowballing

How can you improve the reproducibility of your grey literature searches?

  • Reproducibility should be the goal for all searching; however, with the high variability of the search tools in grey literature, perhaps the more reasonable goal is transparency
  • It is important to read current papers on the challenges of grey literature searching such as CADTH, Haddaway et al (2017), and Hartling et al (2016) and Briscoe (2015).
  • Documentation of your searches is key; did you use a valid reporting standard? These will not correct poor reproducibility but will have a positive effect on transparency.
  • It helps to have some gold standard GL search examples at hand (or look for them).
  • In planning it will help to develop sound methodological and publication strategies that include considerations of reproducibility .
  • Negotiate copyright; authors should aim to retain their rights to make their research available; create a dedicated website for your paper, and include your data plans (and grey literature search report).
  • The EQUATOR Network is an international initiative that seeks to enhance reliability and value of medical research literature by promoting transparent and accurate reporting of research studies; see the principles of reporting searches in the PRISMA Statement
  • See for clinical trials registration
  • Develop your search protocol and register the entire protocol at PROSPERO
  • Moher D, Schulz KF, Simera I, Altman DG. Guidance for Developers of Health Research Reporting Guidelines. PLoS Med 2010.

Challenges in reproducibility

  • One pillar of the scientific method is reproducibility and being able to redo an experiment and get the same results. When doing field work, the collection of rare samples or observing rare natural events may not be reproducible.
  • Tracking variables in a lab (e.g. how someone uses pipettes or mixes reagents) is a challenge as there are variables similar to cooking; using the same recipe does not always give you the same result. See Munafò MR, Nosek BA, Bishop DV, Button KS, Chambers CD, du Sert NP, Simonsohn U, Wagenmakers EJ, Ware JJ, Ioannidis JP. A manifesto for reproducible science. Nature Human Behaviour. 2017 Jan 10;1:0021.
  • My goal is to raise awareness of how to bring the standards of searching, especially grey literature searching, to the level of the scientific method through my wiki, use of Twitter, holding workshops and my writing.
  • Moving grey literature towards greater reproducibility involves raising awareness of the issues but also publishing in the field, which is a constant challenge given a lack of research time in my position. Still, it is important enough as many SRs include searches for the grey literature.
  • Health librarians and search experts undertaking searches for the systematic review (SR) encounter many issues and challenges in terms of search reproducibility. One of the major challenges is that grey literature searching – arguably a critical part of the SR – is a more uncontrolled and iterative type of searching than that performed in the standard bibliographic databases such as MEDLINE, EMBASE and CINAHL, to name a few (Booth, Sutton and Papaioannou 2016; Godin, Stapleton, Kirkpatrick et al 2015).
  • The problem requires tailor-made solutions and broad information dissemination. So, why is grey literature searching a challenge? Lack of control, and replicability, in grey literature searching is related to poor indexing on websites where grey literature is located. Unique search interfaces at websites impose navigational challenges and limitations that require repeating searches for grey literature often in multiple places (Stansfield, Dickson, Bangpan 2016).
  • What are some of the other problems? Many grey literature websites do not permit the narrowing of topics easily or provide expected features such as sorting and saving. Some of the larger concerns expressed about the difficulty of using these websites are their inability to perform complex Boolean search queries and export citation results efficiently (Stansfield, Dickson, Bangpan 2016).
  • Health librarians and searchers can improve the reproducibility of their grey literature searching.
  • As noted in the literature, planning for grey literature searching using a search document and managing the results of searches are both critical to accurate and reliable reproducibility (Shaw, Booth, Sutton et al 2004).
  • Another reason that documentation is important is that the validity of the SR can be evaluated in part by how explicit searches for grey literature were performed (Hopewell, Clark, Mallett 2005). In the Internet era, several variables limit the reproducibility of web searching due to the unpredictability of tools and ongoing changes to web content and addresses (Stansfield, Dickson, Bangpan 2016; Briscoe 2015). All of these issues must be accounted for by the health librarian or expert searcher involved in SR searching.
  • Stansfield, Dickson and Bangpan (2016) explore some of the challenges of conducting website searches and emphasize systematic but practical recordkeeping. Key information about search methods, sources, search queries, and results should be recorded (McArthur 2016). Web searching should be reported to an extent that search strategies are transparent and reproducible; the aim is to include complete, detailed search strategies (which can be copied and pasted in some instances) and the number of records retrieved. Other details such as handsearching, contact with experts, reference lists and citation searching should also be included in planning documents


  1. Define grey literature and explain its characteristics vis a vis ‘published literature’
  2. Develop grey literature search strategies and approaches that are reproducible
  3. Identify key resources and websites to locate the most relevant grey literature
  4. Undertake comprehensive searches using diverse tools, engines and resources
  5. Use handsearching, snowballing and citation searching to increase sensitivity
  6. Document and report the search process to meet required standards
  7. Manage citations using Endnote, Mendeley, RefWorks or related tools
  8. Budget for and estimate the costs associated with major grey literature searching
  9. Understand review synthesis methods and communicate the value of grey literature to the research synthesis team (i.e., to help minimize publication and linguistic bias, to avoid distorted view of literature)
  10. Develop strategies to communicate effectively with the research team throughout the search process


  • Benzies, Karen M., Shahirose Premji, K. Alix Hayden, and Karen Serrett. 2006. "State‐of‐the‐evidence reviews: advantages and challenges of including grey literature." Worldviews on Evidence‐Based Nursing 3(2): 55-61.
  • Booth, Andrew, Anthea Sutton, and Diana Papaioannou. 2016. Systematic approaches to a successful literature review. Sage.
  • Booth, Andrew. 2016. "Searching for qualitative research for inclusion in systematic reviews: a structured methodological review." Systematic reviews 5(1): 1.
  • Booth, Andrew, and Christopher Carroll. 2015. "Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable?" Health Information and Libraries Journal 32(3): 220-235.
  • Bramer, Wichor M., Dean Giustini, and Bianca MR Kramer. 2016. "Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study." Systematic Reviews 5(1): 1.
  • Bramer, Wichor M., Dean Giustini, Gerdien B. de Jonge, Leslie Holland, and Tanja Bekhuis. 2016. "De-duplication of database search results for systematic reviews in EndNote." Journal of the Medical Library Association: JMLA 104 (3): 240.
  • Bramer, Wichor M., Dean Giustini, Bianca MR Kramer, and P. F. Anderson. 2013. "The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews." Systematic Reviews 2 (1): 1.
  • Briscoe, S. 2015. "Web Searching for Systematic Reviews: A Case Study of Reporting Standards in the UK Health Technology Assessment Programme." BMC Research Notes 8: 153.
  • CADTH Finding the Evidence: Literature Searching Tools in Support of Systematic Reviews
  • Chojecki, Dagmara and Lisa Tjosvold. 2016. “Documenting and reporting the search process.” HTAi Vortal. 20 December 2016. Accessed from:
  • Godin, Katelyn, Jackie Stapleton, Sharon I. Kirkpatrick, Rhona M. Hanning and Scott T. Leatherdale. 2015. "Applying Systematic Review Search Methods to the Grey Literature: A Case Study Examining Guidelines for School-Based Breakfast Programs in Canada." Systematic Reviews 4 (1): 138.
  • Grant, Maria and Andew Booth. 2009. “A typology of reviews: an analysis of 14 review types and associated methodologies”. Health Information and Libraries Journal 26: 91–108.
  • Hartling, Lisa, Robin Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. 2016. “The contribution of databases to the results of systematic reviews: a cross-sectional study”. BMC Med Res Methodol. 16(1):127.
  • Haddaway, Neal R., Alexandra M. Collins, Deborah Coughlin, and Stuart Kirk. "A rapid method to increase transparency and efficiency in web-based searches." Environmental Evidence 6, no. 1 (2017): 1.
  • McGowan, Jessie, Margaret Sampson, Douglas M. Salzwedel, Elise Cogo, Vicki Foerster, and Carol Lefebvre. 2016. "PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement." Journal of clinical epidemiology 75 (2016): 40e46.
  • Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Method. 2014;5(2):98-115.
  • Shaw, Rachel L., Andrew Booth, Alex J. Sutton, Tina Miller, Jonathan A. Smith, Bridget Young, David R. Jones, and Mary Dixon-Woods. 2004. "Finding qualitative research: an evaluation of search strategies." BMC Medical Research Methodology 4(1): 1.
  • Stansfield, Claire, Kelly Dickson, and Mukdarut Bangpan. 2016. "Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic?" Systematic Reviews 5(1): 191.
Personal tools