Grey literature searching in medicine

From HLWIKI Canada
Jump to: navigation, search
Search greys.png
Are you interested in contributing to HLWIKI International? contact:

To browse other articles on a range of HSL topics, see the A-Z index.


Last Update

  • Updated.jpg 17 February 2018


See also CADTH | Expert searching | Grey data ("hard to find" data) | Grey literature | Grey literature - part II | Snowballing

"...As a PhD student working in a reasonably cutting-edge area, grey literature is my life-blood. And yet when it comes to some key sources who take money from public coffers for their work, getting access to material that should be public domain is tricky at best..."' — The Thesis Whisperer
  • This entry is a highly selective bibliography of ~52 articles that address the retrieval of grey literature in various disciplines but with an emphasis on (bio)medicine.

A to H articles

A classic article in health librarianship that describes grey literature and the activities of the Italian Library Association Study Group. Using survey methods, definitions and how end-users use greylit are explored. Research was analyzed from 1987-1988 to determine number of articles citing GL, citations found in journals, various kinds of GL, technical reports cited and country of origin. Selected databases were searched to determine the presence of greylit during those same years.

Anderson says that some information not in traditional databases is easier to find in the grey literature. Addressing librarians and scholars, he describes access problems and efforts of the National Library of Medicine (NLM) to improve access to grey lit.

Auger describes grey literature as "difficult-to-define" publications not available through regular channels. This guide concentrates on identifying, tracing and acquiring publications. Contents include a discussion on the nature and production of grey lit and methods of bibliographic control and indexing. Individual chapters are devoted to aerospace, life sciences, business and economics, education, energy, and science and technology. A list of international organizations dealing in grey literature appears at the end.

Augusto et al showed that it was possible to gather valuable and reliable data from the grey literature. Making the most of unpublished sources of information is a slow process requiring a great deal of work (Conn et al, 2003; Batt et al, 2004), due to poor standardization and indexing of the bibliographic information (Schmidmaier, 1986). It took Augusto et al several years to identify all sources and to obtain access to them. Moreover, their integration in a normalized database was time-consuming because soil information was dispersed and had to be evaluated by an experienced soil scientist. Units used were sometimes not those of the international system and had to be converted before the plausibility of the final value could be assessed. Finally, most people are unaware of the potential value of this kind of literature, and we had to 'rescue’ in extremis a few discarded data sources.

The potential of open access to increase access to peer-reviewed literature is worth discussion. The challenge of providing access to the gray literature to complement peer-reviewed research is keeping up to it. We do not need to launch an open access movement to obtain this material due to its lack of commercial significance. The challenge is to develop resources of depth comparable to peer-reviewed scholarship.

The distinction between grey and non-grey (or white) literature will be less relevant over time as online options proliferate. In the meantime, the political success of the open access publishing movement has valuable lessons for proponents of increasing access to grey literature.

Evidence-based reviews are subject to bias. Grey literature, for its part, can be of poorer quality and may even be expensive to access. Search strategies need to be customized by topic to be effective. This paper complements a systematic review of costs of expanding immunization services in developing countries. Data on the effectiveness and cost-effectiveness of strategies to increase immunization coverage is shown to be similar across literatures, but the quality of information on costing is lower in the grey literature. After excluding poorer quality studies in this review we found the quantity of available evidence almost doubled, particularly for more complex health-system interventions and cost or cost-effectiveness analyses. Interventions in the grey literature are more current and cover different geographical areas. The conclusions of the published and grey literatures differ though the number of papers is still too low to account for differences across types of interventions. Searchers should consider using non-English keywords in their searches.

Health policy decision-makers turn to research-based evidence to support their decisions. Systematic reviews are useful for gathering, summarizing and synthesizing published and unpublished research. State-of-the-evidence reviews are broader than traditional systematic reviews and may include not only published and unpublished research. Decisions about whether to include this “grey literature” are a challenge and lead to many questions about whether the advantages outweigh the challenges. This article aims to describe what constitutes grey literature and methods to locate and assess it. It discusses the issues when making decisions to include grey literature in a state-of-the-evidence review. A recent state-of-the-evidence review is used as an exemplar to present advantages and challenges. The inclusion of grey literature was seen to be useful in validating the results of a research-based literature search. A checklist to assist in decision-making was created as a tool to assist researchers in determining whether it is advantageous to include grey literature in a review.

The inclusion of unpublished and grey literature minimizes publication bias for their inclusion provides a more accurate representation of the whole evidence base. Studies that show statistically significant “positive” results are more likely to be published than those that do not. If systematic reviews are limited to published studies, they risk excluding vital evidence and yielding inaccurate results -- which are likely to be biased to positive results. Previous research shows that meta-analyses excluding grey literature can lead to exaggerated estimates of intervention effects.

This is a verbatim speech delivered at a conference. The proliferation of electronic formats that support or used with standard metadata formats (e.g., Dublin Core) in tandem with a rising wave of requirements (e.g., regulations requiring metadata for government publications, metadata creation as part of e-dissertation submission) and visibility-driven incentives for publicizing resources (e.g., gaining a listing in Yahoo), there may be some hope that originating parties will do the metadata and grey literature will become less labour intensive for libraries.

Clinical trial results have been traditionally communicated through the publication of scholarly reports and reviews in biomedical journals. However, this dissemination of information can be delayed or incomplete, making it difficult to appraise new treatments, or in the case of missing data, evaluate older interventions. Going beyond the routine search of PubMed, it is possible to discover additional information in the “grey literature.” Examples of the grey literature include clinical trial registries, patent databases, company and industrywide repositories, regulatory agency digital archives, abstracts of paper and poster presentations on meeting/congress websites, industry investor reports and press releases, and institutional and personal websites.

An ever-growing amount of grey literature is available to nurses, including oral presentations, personal communication, leaflets, newspapers and magazines, unpublished research, internal reports and minutes of meetings. It can be an invaluable research resource.

In meta-analysis, researchers combine the results of individual studies to arrive at cumulative conclusions. Meta-analysts sometimes include "grey literature" in their evidential base, which includes unpublished studies and studies published outside widely available journals. Because grey literature is a source of data that might not employ peer review, critics have questioned the validity of its data and the results of meta-analyses that include it.

A systematic review into palliative care team effectiveness was undertaken which has, inherent in its methodology, grey literature searching. Over 100 letters were written to a systematically chosen range of service providers, commissioners, and experts in combination with requests for information in six UK national cancer/palliative care organization newsletters. In addition, the System for Information on Grey Literature (SIGLE ) database was searched. As a result, 25 document hard copies were received. The documents were, in all but one case (this one study was also highlighted by the SIGLE search), not relevant as they were predominated by annual reports, service descriptions, and needs assessments. In terms of obtaining unpublished studies for possible inclusion in the review, this comprehensive search was unsuccessful and, therefore, it would appear that grey literature searching is not a useful tool in palliative care systematic reviews.

The amount of gray literature published by nongovernmental organizations, governments, intergovernmental organizations, consultancies, private companies and individuals outnumbers peer-reviewed literature in ecology and conservation. This article explains why this is a problem in terms of quality, discovery, access and archiving of literature. Unpublished dissertations and theses are a vast resource in tropical biology. Internet search tools integrate access to information sources but only when producers of gray literature provide electronic access to these resources. Digital repositories can enable this accessibility and permanent archiving.

To identify reports in the grey literature in primary care and improve depression and anxiety outcomes, this article examines the reports for evidence of effectiveness. A systematic search was undertaken using primary health care research databases, community and professional websites, clearinghouse sources, government reports, and reports from the Australian General Practice Network. Reports were included if they related to programs targeting depression or anxiety, contained qualitative or quantitative data, and were published during 1995-2006. 642 reports were identified; 43 met the inclusion criteria; 30 were delivered in general practice, five in the community or residential care, three in schools, and five were Internet based. Nine programs were reported in the formal literature, but most were not. Limited data on effectiveness or patient outcomes was available in the grey literature. No single service exists that identifies, describes and catalogues mental health in Australia. There may be a role for a mental health information "clearinghouse" to facilitate dissemination for researchers, practitioners, consumers and policymakers. Innovative schemes to disseminate evidence-based models and to encourage collection of data are needed.

This paper discusses the importance of fugitive and/or grey literature in women's studies and the difficulty in providing bibliographic control and access to this material. It presents a methodology designed to improve bibliographic control and access for Rutgers' Women's File, a collection of grey or fugitive literature, and to suggest a system to enhance access to similar collections. This can serve as a model for access and control for other non-conventional print collections in libraries.

This article analyzes the concept of the Invisible Web and its implication for academic librarianship. It offers a guide to tools that can be used to mine the Invisible Web and discusses the benefits of using the Invisible Web to promote interest in library services. The article includes an expanded definition, literature review and suggestions for ways to incorporate the Invisible Web in reference work and library promotion.

This study reviews grey literature relating to effective interventions in three areas: promotion of mental health, improvement of food and nutrition and interventions to increase physical activity. Evidence was collected from a variety of non-traditional sources, and 36 pieces of evidence were selected for in-depth appraisal and review. Various interventions tend to have common outcomes, enabling identification of themes. These include improvements in participant well-being and identification of barriers to success. Most interventions demonstrate positive impact but some did not. This was particularly true for more objective measures such as physiological measurements, particularly when used to evaluate short-term interventions. Objective measurement as part of an intervention may act as a catalyst for future behavioural changes. Time is an important factor that promotes or impedes success. The importance of involving all stakeholders, including participants, when planning health interventions was established as an important indicator of success. This review suggests that large-scale interventions could be more efficient while outcomes relating to the implementation could provide a clearer picture of effectiveness.

This article is a brief overview of the New York Academy of Medicine's Grey Literature Report. A short description of GL is provided, and how the material is used by the Academy. The history of the report is covered and is followed by the reasons NYAM collects it, collection development, technical services issues, format for the report, subscriber base, problems and obstacles involved in its production, and possibilities for future development.

Julia Gelfand is an American applied sciences librarian who writes about grey literature. In this paper, she describes progress made in exposing grey literature, and making it more accessible and visible to scholars. She describes the impact of the web on scholarly publishing and its implications for grey literature and scholarly communication.

This book chapter is a good introduction to the topic of grey literature for nurses in graduate programmes doing theses or dissertations.

This article demonstrated a feasible and seemingly robust method for applying systematic search strategies to identify web-based resources in the grey literature. The search strategy developed and tested is amenable to adaptation to identify other types of grey literature from other disciplines and answering a wide range of research questions. This method should be further adapted and tested in future research syntheses.

H to Z articles

Google Scholar collates results from across the internet and is free to use. GS has received attention as a method for searching particularly for grey literature, as required by systematic reviews. Reliance on GS as a standalone resource has been debated, and yet its efficacy in grey literature searching has not been fully investigated. Using systematic review cases from environmental science, we investigated GS in searches for grey literature. GS results contain moderate amounts of grey literature, with the majority found on average at page 80. The majority of literature identified using Web of Science was also found using GS. Our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10–67%), and that GS missed important literature in a handful of studies. A general GS search failed to find grey literature from a case study that involved manual searching of organisations’ websites. If used in systematic reviews for grey literature, we recommend searches of article titles focus on the first 200 to 300 results. Google Scholar can find much grey literature and specific, known studies, but should not be used alone for systematic review searches. It forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews.

Methods for reviewing research, such as systematic reviews and syntheses, are becoming increasingly common in conservation. It is widely recognised that grey literature, research not published in traditional academic journals, forms a vital part of the evidence base of these reviews. To date guidance and practice in searching for and including grey literature in conservation reviews has taken a broad approach, involving searching of a wide variety of resources. We argue that there are two distinct forms of grey literature and that each must be considered separately in order to assess potential importance and an appropriate searching strategy for every review undertaken. ‘File drawer’ research is as yet unpublished academic research that is important for countering possible publication bias and can be targeted via specific repositories for preprints, theses and funding registries, for example. ‘Practitioner-generated research’ includes organisational reports, government papers and monitoring and evaluation reports, and is important for ensuring comprehensiveness in conservation reviews. By considering the relative importance and appropriate strategies for inclusion of both types of grey literature, reviewers can optimise resource efficiency and comprehensiveness, and minimise publication bias.
Antibiotic resistance is a major challenge in health care worldwide. Up to 90% of antibiotics are being prescribed for outpatients and recommendations for treatment of community-acquired infections are based on resistance findings for hospitalized patients. In the EU-project called "APRES - the appropriateness of prescribing antibiotic in primary health care in Europe with respect to antibiotic resistance" the aim was to gain information about resistance data in both the scientific and grey literature. A systematic review was performed including scientific and grey literature published between 2000 and 2010. Inclusion and exclusion criteria were defined and the review process followed recommendations. 17 scientific articles and 23 grey literature documents were found. In contrast to grey literature, scientific publications describe a part of the resistance in primary care in Austria. Half of these contain data from the ambulatory sector but this data is older than ten years and heterogeneous concerning time period. The grey literature yields comprehensive up-to-date information, is available in German and yet not easily accessible. The resistance can be summarized as stable over the last two years; for Escherichia coli e.g. the highest resistance is fluorochiniolones (19%) and trimethoprim/sulfamethoxazole (27%). Up-to-date resistance data of pathogens is presented and found in the grey literature and only a few published in peer-reviewed journals.

The inclusion of grey literature (i.e. literature that has not been formally published) in systematic reviews may help to overcome some of the problems of publication bias, which can arise due to the selective availability of data. This review shows that published trials tend to be larger and show an overall greater treatment effect than grey trials. This has important implications for reviewers who need to ensure they identify grey trials, in order to minimise the risk of introducing bias into their review.

White papers can be a valuable source of information for librarians and business practitioners. This article is an exploratory investigation of commercial white papers and if they can be found in online databases, aggregators and search engines as an alternative to retrieving the document directly from the website of the authoring organization. It also examines the extent of secondary references to these white papers and whether those white papers can be found in library collections.

The purpose of this paper is to provide a detailed account of one systematic review team's experience in searching for grey literature and including it throughout the review. We provide a brief overview of grey literature before describing our search and review approach. We also discuss the benefits and challenges of including grey literature in our systematic review, as well as the strengths and limitations to our approach. Detailed information about incorporating grey literature in reviews is important in advancing methodology as review teams adapt and build upon the approaches described.

Systematic reviews in mental health are useful tools for health professionals. To determine risk of bias in studies and to avoid bias in generalizing conclusions it is important to use a strict methodology in systematic reviews. One bias which may affect generalization of results is publication bias, which is determined by the nature and direction of the study results. To control or minimize for this, authors of systematic reviews should undertake comprehensive searches of medical databases and searches of grey literature (material which is not formally published). This paper shows the consequences (and risk) of generalizing the implications of grey literature in the control of publication bias (as proposed in a recent systematic review). By repeating the analyses for the same outcome from three different systematic reviews that included published and grey literature our results show that confusion between grey literature and publication bias may affect results of a concrete meta-analysis.

The inclusion of a subset of all available evidence in meta-analyses may introduce biases and threaten validity. This is also likely if the subset of studies included differ from those not. Thus, this paper makes the case to include published and grey literature (unpublished studies, with limited distribution). It examines whether exclusion of grey literature, compared with inclusion in meta-analysis, provides different estimates of effectiveness of interventions assessed in randomized trials. The exclusion of grey literature from meta-analyses can lead to exaggerated estimates of intervention effectiveness. In general, meta-analysts should attempt to identify, retrieve, and include all reports, grey and published, that meet predefined inclusion criteria.

Accessing grey literature has received a lot of attention in recent years. Grey literature cannot be found using traditional indexes; librarians turn to it with ‘grey questions.’ Librarians and other information professionals are experts in providing answers but many questions remain unanswered. This paper treats unanswered grey questions as valuable information and suggests they might be used to drive research. A model grey question repository is proposed which could be queried and manipulated as other bibliographic tools. It might lead to the filling of important gaps in knowledge.

Black and ethnic minorities show different pathways in and out of care services, and involve non-statutory services. To improve access and develop effective interventions, innovations are described but knowledge about how to improve pathways has not been synthesized (or formally published). This paper addresses this oversight and undertakes a review of the grey literature. The key components of effective pathways include specialist services for ethnic minorities, collaboration between sectors, facilitating referrals between services, outreach and facilitating access in and out of care. Services that support collaboration and improve access seem effective, but warrant further evaluation. Innovative services must ensure that their evaluation frameworks meet minimum quality standards if the knowledge gained is to be generalized, and if it is to inform policy.

Research communications today are largely conducted by digital means. At this time, however, only a small percentage of these digital communiques are archived and preserved for future use. This article provides an overview of the challenge of this digital grey literature, a brief overview of digital archiving, and the role librarians and researchers can play in facilitating the availability and accessibility of these resources for future generations.

The Irish government published its primary care strategy, Primary Care: A New Direction in 2001. Progress with the implementation of Primary care teams is modest. The aim of this paper is to map the Irish grey literature and peer-reviewed publications to determine what research has been carried out in relation to primary care teams, the reform process and interdisciplinary working in primary care in Ireland. This scoping review employed three methods: a review of Web of Science, Medline and Embase databases, an email survey of researchers across academic institutions, the HSE and independent researchers and a review of Lenus and the Health Well repository. N = 123 outputs were identified. N = 14 were selected for inclusion. A thematic analysis was undertaken. Common themes identified were resources, GP participation, leadership, clarity regarding roles in primary care teams, skills and knowledge for primary care team working, communication and community.There is evidence of significant problems that disrupt team formation and functioning that warrants more comprehensive research.

each year there are several conferences offering presentations and workshops addressing the latest in research, best practice, and innovation in continuing education in the health professions (CEHP). Each conference leaves behind artifacts. For example, CME Congress 2012 — held last spring in Toronto and the subject of two commentaries in this issue — yielded abstracts, handouts, posters, slide presentations, newsletter articles, blog entries, tweets, participant notes, and video recordings. Some will make it into the conventional published literature. The rest becomes part of the grey literature.

The author discusses 'gray' literature - what librarians used to call 'the vertical file' - on the web. Content that is not making its way into commercial databases should be enriched with descriptive tagging to increase subject access. Google as a search engine surpasses others in indexing web files and providing better access to electronic theses and dissertations (ETD) than library OPACs.

Systematic reviews aide the analysis and dissemination of evidence, using rigorous and transparent methods to generate empirically attained answers to focused research questions. Identifying all evidence relevant to the research questions is an essential component, and challenge, of systematic reviews. Gray literature, or evidence not published in commercial publications, can make important contributions to a systematic review. Gray literature can include academic papers, including theses and dissertations, research and committee reports, government reports, conference papers, and ongoing research, among others. It may provide data not found within commercially published literature, providing an important forum for disseminating studies with null or negative results that might not otherwise be disseminated. Gray literature may thusly reduce publication bias, increase reviews’ comprehensiveness and timeliness, and foster a balanced picture of available evidence.

This article emphasizes the importance of grey literature for medical researchers. Because of the delay between research and publication, and the potential that some research is never published, access to innovative information is a challenge. Grey literature is of value and fills a void. The authors define grey literature, explore its sources, and identify its major users and uses. The authors identify the range of grey literature, its advantages and disadvantages, the various outlets that produce it, and where it may be found.

Grey literature is an accepted form of scientific communication. Little is known about the use of grey literature in veterinary medicine; thus a systematic study was done to analyze citations appearing in 12 core veterinary journals. Citations from 2,159 articles published in twelve veterinary journals in 2000 were analyzed to determine the % of citations from grey literature. Citations were further analyzed and categorized according to type of publication. Citation analysis yielded 55,823 citations, of which 3,564 (6.38%) were considered to be grey. Four veterinary specialties, internal medicine, pathology, theriogenology, and microbiology, accounted for 70% of the total articles. Three small-animal clinical practice journals cited 2.5-3% grey literature, less than half that of journals with basic research orientations, where results ranged from almost 6% to 10% grey literature. 90% of the grey literature appeared as conferences, government publications, and corporate organization literature. These results corroborate other research about the incidence of grey literature as lower in medicine and biology than other fields, such as aeronautics and agriculture. Use of the Internet has expanded communication among vet professionals. The appearance of closed community email forums and discussion groups in veterinary science is an example of what is a new kind of grey literature.

"...the median time searching all resources was 471 minutes and this includes grey literature searching...".

Extended systematic search methods uncovered 29.2% of all items retrieved for systematic reviews. The search of specialized databases was the most effective method, followed by scanning of reference lists, communicating personally and hand searching. Although items identified through hand searching is small, these items would otherwise be missed. Systematic search methods are effective in uncovering material for the systematic review. The quality of items uncovered has yet to be assessed and key in evaluating the value of extended search methods.

  • Schöpfel J, Farace DJ. Grey literature. In: Encyc Libr Inform Sci. 2029-2039. CRC Press.

This is an overview of GL resources in the area of communication disorders. It is geared to practitioners, researchers, and consumers seeking reliable, freely available scientific information. Includes identification of the methods specialists use to obtain this valuable, yet often overlooked, literature. Access points and search tools for identifying grey literature on communication disorders are recommended. Commercial databases containing grey literature are not included.

In this comment and opinion piece, the authors describe the process of developing and applying a “modified citation analysis” that builds on existing methods of examining a research paper's impact in two key ways: (1) by deliberately including gray literature in the citation analysis search process, and (2) by including quantitative and qualitative methods of analysis to gain a better understanding of how a research paper was used. By broadening the search and deepening the level of analysis, we suggest this new approach can better assess the impact of a given research paper—both within and outside of traditional peer-reviewed venues. We begin with a review of gray literature and then describe current methods for analyzing the impact of a research paper. Finally, we use a specific example to describe our new approach, highlight its potential for evolving the field of citation and impact analysis, and discuss future refinements and evaluation.

This paper is the second which reviews the evidence for recovery in mental health. Over the last 4 years a vast amount has been written about recovery in mental health (60% of all articles). In the first review, the focus was on peer-reviewed evidence; this paper focuses on grey/non-peer-reviewed literature. A search yielded 3 books, 11 book chapters, 12 papers, 6 policy documents and 3 publications from voluntary organizations. Each publication was analysed and presented by publication group. The findings are presented as themse: social, historical and political critique; philosophy of hope for the individual; individual identity and narrative; models and guidance for mental health practice. There is a need for both empirical research into recovery and a clearer theoretical exposition of the concept.

This essay focuses on the writing and publication practices that developed in and around the Birmingham Centre for Contemporary Cultural Studies from the time of its founding in 1964 until the cessation of the journal. The Centre’s success is a testament to the virtues of architecting an apparatus of ‘scholarly’ communications around gray literature. The history we present shows how gray literature is not the "other" of rigorous intellectual writing but, in the case of cultural studies, a key condition of the success of the field. It also shows how the scholarly value (credibility, authority) of gray literature is relative and not absolute, sometimes shifting as new types of documents get introduced into an existing publishing repertoire.

During the 2003-2004 academic year, library staff at the University of Rochester studied how faculty members find, use, and produce grey literature to do their scholarly work.

Grey lit is broadly defined as news clippings, reports, newsletters, listserv queries, consultations and personal contacts and periodicals not in databases. GL raises "grey questions" and fills in knowledge gaps left by peer-reviewed literature. Traditional databases do not index grey lit and leave researchers with a false impression that information cannot be found. The authors recommend a database of grey questions and networking among information providers and librarians to find answers.

A lot of public health information is considered gray and is not available through traditional venues. The aim of this project is to improve access to PH gray literature reports through established natural language processing (NLP) techniques. The paper summarizes development of a model for representing gray literature documents. Authors establish a model-based approach for analyzing and representing gray literature in public health through evaluation of gray literature from seven websites. Input from fifteen public health professionals assisted in developing the model and elements for NLP extraction.

This review summarizes existing European published and 'grey' literature re: effectiveness of school-based interventions to promote healthy diets in children (6-12 years old) and adolescents (13-18 years old). Eight electronic databases, websites and key journals were searched, reference lists were screened, and authors and experts were contacted. Studies were included if published between 1990 and 2007 and reported on topic. Forty-two studies met inclusion: twenty-nine in children and thirteen in adolescents. In children, strong evidence was found for multicomponent interventions on fruit and vegetable intakes. Limited evidence of effect was found for educational interventions on behaviour, and for environmental interventions on fruit and vegetable intakes. Interventions that specifically targeted children from lower socio-economic groups showed limited evidence of effect on behaviour. In adolescents, moderate evidence of effect was found for educational interventions on behaviour and limited evidence of effect for multicomponent programmes on behaviour. In children and adolescents, effects on anthropometrics were not measured, and evidence was lacking or inconclusive. Finally, evidence was found for effectiveness of multicomponent interventions promoting healthy diet in school-aged children in EU countries on self-reported dietary behaviour.

Back to main page

Personal tools