Abstract
Systematic review (SR) is a popular research method in software engineering(SE). However, conducting an SR takes an average of 67 weeks. Thus, automatingany step of the SR process could reduce the effort associated with SRs. Ourobjective is to investigate if Large Language Models (LLMs) can acceleratetitle-abstract screening by simplifying abstracts for human screeners, andautomating title-abstract screening. We performed an experiment where humansscreened titles and abstracts for 20 papers with both original and simplifiedabstracts from a prior SR. The experiment with human screeners was reproducedwith GPT-3.5 and GPT-4 LLMs to perform the same screening tasks. We alsostudied if different prompting techniques (Zero-shot (ZS), One-shot (OS),Few-shot (FS), and Few-shot with Chain-of-Thought (FS-CoT)) improve thescreening performance of LLMs. Lastly, we studied if redesigning the promptused in the LLM reproduction of screening leads to improved performance. Textsimplification did not increase the screeners' screening performance, butreduced the time used in screening. Screeners' scientific literacy skills andresearcher status predict screening performance. Some LLM and promptcombinations perform as well as human screeners in the screening tasks. Ourresults indicate that the GPT-4 LLM is better than its predecessor, GPT-3.5.Additionally, Few-shot and One-shot prompting outperforms Zero-shot prompting.Using LLMs for text simplification in the screening process does notsignificantly improve human performance. Using LLMs to automate title-abstractscreening seems promising, but current LLMs are not significantly more accuratethan human screeners. To recommend the use of LLMs in the screening process ofSRs, more research is needed. We recommend future SR studies publishreplication packages with screening data to enable more conclusiveexperimenting with LLM screening.