Systematic reviews, scoping reviews and other evidence syntheses

Data extraction, which follows full-text screening, is the systematic process of transcribing key information from primary studies included in the review. Extraction is done by two reviewers independently to reduce bias and increase accuracy.

• Data is extracted to:
  • Objectively and accurately summarize studies in a common format to facilitate synthesis,
  • Identify numerical data if a meta-analysis is to take place, and
  • Obtain information to objectively assess the risk of bias in, and applicability of, studies.
• To enhance reliability, all reviewers must use the same, standardized form to extract and record data:

"The data collection form is a bridge between what is reported by the original investigators (e.g. in journal articles, abstracts, personal correspondence) and what is ultimately reported by the review authors." -- Rationale for data collection forms, Cochrane Handbook for Systematic Reviews of Interventions

• A fair procedure for resolving discrepancies should be established

Data extraction tools

Data can be extracted and recorded using:

• pen and paper
• Excel spreadsheet
• Data Extraction Form (Cochrane Review Group) -- edit this form to suit the needs of your systematic review
• Software - particularly useful if you are reviewing a large number of studies or planning to do a meta-analysis
  • Covidence (available to Brock users) facilitates data extraction
    • Users must create an account to utilize Brock's access to this tool.
    • From Covidence: Practical guide to data extraction for systematic reviews
  • RevMan - provided free from Cochrane Collaboration
  • DistillrSR - paid product; free for short-term student use
• See Considerations in selecting data extraction tools from the Cochrane Handbook for Systematic Reviews of Interventions

What information to record

Plan carefully to ensure the data you extract will allow you to answer your research question. Pilot test your data extraction form to ensure it is clear and comprehensive.

• Use the review's research question and inclusion and exclusion criteria as a guide
• Elements recorded will vary according to your project, but standard elements include:

• full article citation
• study objectives
• study design
  • include how the study attempted to avoid bias
• participant characteristics and numbers
• intervention
• comparison
• results/outcome
• comments related to study quality

If published data is missing or unclear, try contacting the study authors for details.

Study characteristics table

• Use data extraction forms to produce a summary table of study characteristics
• Include the summary table in the Results section of the report of the systematic review, either as an appendix or in the text

Critical appraisal is a key component of many evidence syntheses reviews.

"Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context" (Burls, 2009). 

Generally, the critical appraisal process will consider:

• Question - Does this study address a clearly focused question?
• Methodological quality - Did the study use valid methods to address this question? 
• Bias -- To what extent do the study design and conduct eliminate the potential for systematic error (bias)?
• Precision – What is the likelihood of random errors? (Often depicted as the confidence interval around the result)
• External validity - Are these valid, important results applicable to my patient or population?

To determine if your evidence synthesis review requires critical appraisal, consult this chart which outlines 14 types of review articles and indicates whether quality appraisal is typically included. Table 1: Main review types characterized by methods used (Grant & Booth, 2009).

Critical appraisal tools

When reading the full text of each article identified for inclusion in the review as part of the data extraction process, apply a Critical Appraisal Tool to each study selected for inclusion. Select a critical appraisal tool appropriate for the study designs you are reviewing.

Collections of critical appraisal tools

References

Systematic Reviews in Health by Murray Turner at University of Canberra

Systematic Reviews by Lydia Thorne at University of Ontario Institute of Technology

Systematic reviews and other syntheses by Sandra McKeown, Queen's University Library