This is a summary, written by members of the CITF Secretariat, of:
Perlman-Arrow S, Loo N, Bobrovitz N, Yan T, Arora RK. A real-world evaluation of the implementation of NLP technology in abstract screening of a systematic review. Res Synth Methods. 2023 May 25. doi: https://doi.org/10.1002/jrsm.1636.
The results and/or conclusions contained in the research do not necessarily reflect the views of all CITF members.
A CITF-funded study, published in Research Synthesis Methods, found that implementing a natural language processing (NLP) tool for abstract screening in a living systematic review of SARS-CoV-2 prevalence was feasible and beneficial in a real-world context. The NLP-assisted abstract screening tool provided text inclusion recommendations, keyword highlights, and visual context cues. The quality improvement assessment of screening with and without the tool evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. Users provided positive feedback overall and gave a mean satisfaction score of 4.2 out of 5. This study was led by SeroTracker’s Dr. Rahul Arora (University of Calgary).
- The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter-reviewer conflict rates from 8.3% to 3.6%.
- When one human abstract reviewer was replaced with the NLP tool’s votes, the abstract screening time process was reduced by 70%, with both recall and precision being similar.
- The summary statistics of the seroprevalence estimates were similar when using and not using the tool.
- The inclusion recommendation feature of the tool was frequently used and voted most useful by reviewers (mean rating of 4.7 out of 5), while keyword highlights were rated the lowest in usefulness.
This real-world evaluation comprehensively confirms the effectiveness of the NLP tool in abstract screening for systematic reviews, offering significant time savings while maintaining accuracy.