Are you interested in contributing to HLWIKI International? contact Levels I through III are the "data extraction" components of the SR, and require robust software for inclusion/exclusion process
To browse other articles on a range of HSL topics, see the A-Z index.
- 15 February 2018
See also Bibliographic citation software | Expert searching | Search filters & hedges | Hand-searching | Network meta-analysis | RefWorks | Systematic reviews | Systematic review librarian
What is the purpose of a systematic review (SR)?
"...a systematic review seeks to identify as many potentially relevant studies as possible that meet the research question for a given topic" — NHS Centre for Reviews and Dissemination
Before starting your systematic review
- Before undertaking a systematic review, read a good introduction or monograph on the topic to acquaint yourself with the issues and complexities
- In planning your review, consider registering the protocol first at the Cochrane Collaboration (or, its equivalent such as PROSPERO). Registering your protocol will improve the transparency of all research, and prevent unnecessary duplication.
- Researchers should speak to a statistician before using any of the programs listed below. Once you have discussed your statistical needs with an expert, consider using one of the following tools to manage your citations and inclusion/exclusion processes:
- There are a number of tools available to help you manage the various stages of a systematic review--in particular, the process of title/abstract screening, full text screening, data extraction and quality assessment. Below are a few such tools. Generally, these are all used in conjunction with a citation management software such as Endnote, Mendeley or Zotero.
- community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. The resource aims to help reviewers find appropriate tools based on how they provide support for the systematic review process. Users can perform a simple keyword search (i.e. Quick Search) to locate tools, a more detailed search (i.e. Advanced Search) allowing users to select various criteria to find specific types of tools and submit new tools to the database.
- An online tool for the task of citation screening for systematic reviews.
- Developed by the Center for Evidence-based Medicine (Brown School of Public Health)
- Assists with abstract screening process and statistical analysis
- Import abstracts from NLM using PMIDs; transfer abstracts from Reference Manager or EndNote. Allows for collaborative screening of abstracts.
- CADIMA https://www.cadima.info/index.php supports the conduct of systematic reviews and evidence/systematic maps by the provision of a freely available online tool that: 1. guides review authors through the evidence synthesis process, 2. facilitates the coordination of cooperating team members, 3. eases steps with considerable workload and 4. guarantees for its thorough documentation. The evidence synthesis tool was established and is further developed in a close collaboration between the Julius Kühn-Institut and the Collaboration for Environmental Evidence
- an open-access database; a non-profit internet portal provided and maintained by the Julius Kühn-Institut aiming to increase the transparency and traceability of information in agricultural and environmental sciences.
- Developed as part of EU-funded projects and works to enhance public access to information and data from ongoing and future projects in biosafety research.
- With the Collaboration for Environmental Evidence (CEE), researchers have developed an open-access online tool called CADIMA that facilitates the conduct and assure for a detailed documentation of systematic reviews and systematic/evidence maps.
See also: Kohl C, McIntosh EJ, Unger S, Haddaway NR, Kecke S, Schiemann J, Wilhelm R. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environmental Evidence. 2018 Dec;7(1):8.
- Note: Subscription model in May 2016 means it is no longer free to use. More information can be found here: https://covidence.groovehq.com/knowledge_base/topics/subscription-plans-launching-in-may
- Web-based platform streamlining entire! production of systematic reviews, including Cochrane Reviews. Citation screening, Full text review, Risk of Bias assessment, Extraction of study characteristics and other study data, Export of data into RevMan. Nonprofit organization, open source software
- Covidence improves healthcare evidence synthesis by improving the efficiency and experience of creating and maintaining systematic reviews
- Perform quality assessment, data extraction and final decisions to include and exclude studies online in real-time. Formerly known as ReGroup.
- Discuss in the forums http://tech.cochrane.org/revman/other-resources
- Price: Free for one review, unlimited use is $20/month.
- DistillerSR is the the world’s most used / most popular systematic review software, and comes with unlimited expert support; web based reference screening, data extraction and reporting solution for the entire systematic review process.
- Further, DistillerSR is an online application designed specifically for screening and data extraction phases in the systematic review https://www.youtube.com/channel/UC2CXvErVKiZspdnarJqVDIw
- DistillerSR provides customizable forms for inclusion/exclusion criteria; researchers can import search results and track included and excluded references; create customizable forms to extract study data; export extracted data to other programs for analysis; in other words, DSR provides a simple way to view and screen citations, abstracts and full-text; perform data extraction; generate tables in Excel; DSR has capacity for handling citations
- DSR allows you to import your article citations from PubMed or an EndNote Library (RIS format files) and attach full-text articles to the citations. Once articles have been entered you can develop a series of forms that walk you through appraisal; put forms in sequence, and adjust the visible questions based on responses provided. For example, start with a title/abstract review form to determine if articles meet your inclusion criteria, and then move to a form that walks you through the quality appraisal piece. There are report options to help you create tables needed for your article based on the data you entered.
- DSR provides user permission settings to allow each individual to access only the parts of the project they need for their level of participation. So you could direct one team member with less experience to work on only the piece of the process they are qualified for, while another person could do more detailed evaluation work. You can also view each team member's progress through their delegated part of the project and easily see the project's current status.
- Costs vary from $125USD a month see here. DistillerSR's pricing model is per license (user) not per review. There is no limit to the number of reviews you can conduct with any DistillerSR license, except in the Student FREE package. Students can use DistillerSR for free to complete schoolwork for 1 semester; students can use DistillerSR indefinitely on an unlimited number of projects for $15/month; previous $30/month package has been replaced by the two programs above
- Distiller pricing for Cochrane users is $30/month for unlimited projects: https://www.evidencepartners.com/cochrane-pricing/
- Student Program URL is: https://www.evidencepartners.com/pricing/student-pricing/
- Facilitates collaboration; application designed for screening and data extraction phases of systematic reviews
- In summary, a powerful, popular but pricey tool for systematic review management; allows for the management of the entire systematic review process with multiple independent reviewers.
- Login to Distiller is now: https://login.evidencepartners.com
- EndNote Web is able to run duplicate checks (slow); split library into 3 for screening and marking results and used EPPI for analysis
- fine for 100K records as long as PDFs were not attached; library of 40,000 was slow
- not recommended for inclusion/exclusion process
- EpiGear International www.epigear.com is home to MetaXL, a free Microsoft Excel for Windows add-in for meta-analysis.
- MetaXL supports all major meta-analysis methods, and in addition implements two alternatives to the flawed random effects model.
- MetaXL also supports a simple yet powerful way to implement network meta-analysis.
- EPPI-Reviewer 4 is maintained by the EPPI-Centre, Social Science Research Unit, Institute of Education, University of London
- support given is good; recommended tool as it carries out whole process; de-duplicating is slow
- subscription fee payable; flexibility to handle data collecting and multiple reviewers accessed by signing online
- frustrations with EPPI; handbook not detailed enough so you must learn by trial and error (they have Youtube videos)
- data extraction can take some time and only after you've used EPPI reviewer are you aware of some of the quirks of how it will present data
- open to human error - if a particularly box or subcategory isn't ticked then the data extracted on that variable won't show up in tables. It gives no warning and its hard to spot when you've made this mistake particularly if you're having to click dozens of boxes. This is particularly problematic if you're trying to make statements like this many studies looked at this behaviour or this many studies used this type of intervention etc. as its hard to be confident the numbers you're getting are accurate.
- claims it has functionality to compare data extraction between two reviewers extracting in duplicate I found this really unhelpful and that it was much easier to compare reviewers data extraction manually
- poor with dealing with studies where there are multiple publications for one dataset - it provides little help
- workshop at the Oct Cochrane Colloquium (EROS dialogues with RevMan: data extraction, quality assessment and more )
- Web-based software designed specifcially to perform the first stages of a systematic review.
- Discuss in the forums http://tech.cochrane.org/revman/other-resources
- Price: Free
- Excel (Microsoft Excel) is the simpest tool for managing the screening and data extraction stages of the systematic review process.
- Customized workbooks and spreadsheets can be designed for the review process, and lists of references can be exported from citation managers into Excel format for screening.
- Excel Workbooks for Systematic Reviews & Corresponding Handouts: Excel Workbooks for SRs http://libguides.sph.uth.tmc.edu/excel_workbook_home
- Import.io https://import.io/ us a web-based platform for extracting data from websites without writing code
- Users create an API using their point and click interface; users navigate to websites and teach the app to extract data by highlighting examples of data from the page; learning algorithms then generalize from the examples to work out how to get all the data on the website
- The data users collect is stored on import.io’s cloud servers and can be downloaded as CSV, Excel, Google Sheets or JSON and shared :*Users can generate an API from the data allowing them to easily integrate live web data into their own applications or third party analytics and visualization software
- import.io offers real-time data retrieval through JSON REST-based and streaming APIs, integration with several common programming languages and data manipulation tools, as well as a federation platform which allows up to 100 data sources to be queried
- Mendeley is not a recommended software tool to use for the SR inclusion/exclusion process
- Despite the social capabilities and the ability to work as a group from anywhere, Mendeley is not truly capable of importing and deduping large #s of records <100,000)
- Mendeley is now owned by Elsevier and, despite being free, introduces all kinds of network challenges for users
- A free and open-source add-on for conducting meta-analyses within the statistical software environment R. It consists of a collection of functions that allow users to calculate various effect size or outcome measures, fit fixed-, random-, and mixed-effects models to such data, carry out moderator and meta-regression analyses, and create various types of meta-analytical plots.
- Parsifal is an online tool designed to support researchers to perform systematic literature reviews within the context of Software Engineering. Geographically distributed researchers can work together within a shared workspace, designing the protocol and conducting the research. As well as providing a way to document the whole process, the tool will help you remind what is important during a systematic literature review. During the planning phase, Parsifal will help you with the objectives, PICOC, research questions, search string, keywords and synonyms, selecting the sources, the inclusion and exclusion criterias. Will also provide mechanisms to build a quality assessment checklist and data extraction forms. During the conducting phase, you will be able to import bibtex files and select the studies, find duplicates among all the different sources, execute the quality assessment and extract data from the papers.
SDSR – systematic review data repository http://srdr.ahrq.gov/ (free)
- The Systematic Review Data Repository (SRDR) is produced by the Agency for Healthcare Research and Quality (AHRQ)
- It's a powerful and easy-to-use tool for the extraction and management of data for systematic reviews or meta-analyses: it is open and searchable
- As a web-based tool for extraction and management of data, it is open and searchable; users create an archive of systematic reviews
- With the searchable archive of systematic reviews (and the data extracted from studies for the meta-analysis), data management is easier
- SDSR allows you to create your own "abstraction form" with instant validation checks; SDSR does not actually analyze the data
- Price: Free
- Rayyan is a free web application that helps systematic reviewers to perform their job in a quick, easy and enjoyable fashion.
- Authors create systematic reviews, collaborate on them, maintain them over time and get suggestions for article inclusion.
- Rayyan is a completely web-based, collaborative application to support systematic reviews; it includes a mobile app for screening studies on the go.
- Some librarians have found Rayyan difficult to navigate once files are uploaded; separate files from the different databases are difficult to compile in one file.
- Some librarians transfer citations into Zotero (or other tool) and then export data into Excel. Rayyan did not calculate kappa statistics for inter-rater reliability.
- See Couban R. Covidence and Rayyan and Rayyan: a systematic reviews web app for exploring and filtering searches for eligible studies for Cochrane Reviews
Review Manager (RevMan)
- Review Manager (RevMan) (software used for preparing and maintaining Cochrane Reviews)
- Review Manager (RevMan) software is used for developing Cochrane systematic reviews. It's free and anyone can use it. It can help structure a review, build tables of included and excluded studies, and perform meta-analysis of study data
- YouTube Introduction to RevMan tutorials
Reference Manager and Microsoft Access in combination
- not recommended as ‘messy using 2 systems’
- not recommended for inclusion/exclusion process
- A machine learning system that automatically assesses bias in clinical trials.
- SWIFT-Review (SWIFT is an acronym for “Sciome Workbench for Interactive computer-Facilitated Text-mining”), is a freely available interactive workbench which provides numerous tools to assist with problem formulation and literature prioritization.
- See http://systematicreviewtools.com/tool.php?ref=SWIFT-Review
- Systematic Review Database SRDB.PRO™ is a client-based software tool for managing and aiding systematic reviews (hosted or enterprise)
- teams can cooperate and share results i.e., exclusion and inclusion criteria for the screening
- data extraction grids with treatments, outcomes and various data columns
- population details information fields
- study details and critical appraisal questions, which can be open, single or multi-choice
- hints can be input and data types chosen to help the extraction process
- report template can be created for the project or chosen from those available
- Price: requires purchase http://www.srdb.pro/pricing
SUMARI (System for the Unified Management, Assessment and Review of Information) https://www.jbisumari.org/faq.html
- QARI (Qualitative Assessment and Review Instrument), which is designed to facilitate critical appraisal, data extraction and meta-aggregation of the findings of qualitative studies;
- MAStARI (Meta Analysis of Statistics Assessment and Review Instrument), which is designed to conduct the meta-analysis of the results of comparable cohort, time series and descriptive studies using a number of statistical approaches;
- ACTUARI (Analysis of Cost, Technology and Utilisation Assessment and Review Instrument), which is is designed to facilitate critical appraisal, data extraction and synthesis of economic data, and:
- NOTARI Narrative, Opinion and Text Assessment and Review Instrument), which is designed to facilitate critical appraisal, data extraction and synthesis of expert opinion texts and of reports.
University of Texas School of Public Health Library Excel Workbooks
Helena Vonville MLIS, MPH has created a series of Excel workbooks for systematic reviews, which have been used in SRs around the world.
- The workbooks are free.
- Assistance on their use is free.
- Perfect for student projects or reviews with 1 or 2 screeners/reviewers.
- Could be easily adapted to 3 or 4 screener/reviewers.
- The handouts provide step-by-step guidance on each of the workbooks.
- They follow the PRISMA guidelines and enhance the ability of the PI to report their findings.
- They aren’t cloud-based so the screeners & reviewers aren’t dependent on having access to the Internet, just a computer.
- Not so good for large enterprise reviews.
- Requires someone with Excel skills to manage.
- One person is responsible for their upkeep and other duties may prevent from making changes.
- They aren’t cloud-based so it does require the workbooks to be sent back and forth.
See comparisons in Gough D, Oliver S, Thomas J. An introduction to systematic reviews. Sage Publications, London, 2012.
- "Provide authors and users of The Cochrane Library with clear and transparent expectations of review conduct and reporting"
- The PRISMA guidelines replace QUORUM and offer a checklist for preparing the manuscript reporting a systematic review. The PRISMA Flow Diagram Generator is also useful.
- Consensus-based extension to PRISMA that offers a checklist for "condensing [a] systematic review into the essentials for an abstract."
- Guide developed by the McGill Life Sciences Library to assist researchers in preparing manuscripts for publication and selecting venues for submission.
- Bax L, Yu LM, Ikeda N, Moons KG. A systematic comparison of software dedicated to meta-analysis of causal studies. BMC Med Res Methodol. 2007 Sep 10;7:40.
- Cohen A, Hersh W, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. JAMIA. 2006;13(2):206–19.
- Couban R. Covidence and Rayyan. JCHLA / JABSC. 2016;37:124–126.
- Gough D, Oliver S, Thomas J. An introduction to systematic reviews. London: SAGE, 2012. (p103‐105).
- Henderson LK, Craig JC, Willis NS, Tovey D, Webster AC. How to write a Cochrane systematic review. Nephrology. 2010;15:613-624.
- Khabsa M, Elmagarmid AK, Ilyas IF, Hammady H, Ouzzani M. Learning to identify relevant studies for systematic reviews using random forest and external information. Mach Learn. 2016;102(3):465–482.
- King R, Hooper B, Wood W. Using bibliographic software to appraise and code data in educational systematic review research. Medical Teacher. 2011;33(9):719-23.
- Kitchenham B. Guidelines for performing systematic literature reviews in software engineering, Version 2.3, EBSE Technical Report EBSE-2007-01, Keele University and University of Durham, 2007.
- Kohl C, McIntosh EJ, Unger S, Haddaway NR, Kecke S, Schiemann J, Wilhelm R. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environmental Evidence. 2018 Dec;7(1):8.
- Lorenzetti DL, Ghali WA. Reference management software for systematic reviews and meta-analyses: an exploration of usage and usability. BMC Med Res Methodol. 2013 Nov 15;13:141.
- Moore Z. Writing for publication: the essential literature review. In: Writing for Publication in Nursing and Healthcare: Getting it Right. 2012: 104.
- Nur S, Adams CE, Brailsford DF. Using built-in functions of Adobe Acrobat Pro DC to help the selection process in systematic reviews of randomised trials. Syst Rev. 2016 Feb 18;5(1):33.
- Olofsson H, Brolund A, Hellberg C, Silverstein R, Stenström K, Österberg M, Dagerhamn J. Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan. Res Synth Methods. 2017 Sep;8(3):275-280.
- O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5.
- Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3(1):74