Abstract
Predicting postoperative risk can inform effective care management &planning. We explored large language models (LLMs) in predicting postoperativerisk through clinical texts using various tuning strategies. Records spanning84,875 patients from Barnes Jewish Hospital (BJH) between 2018 & 2021, with amean duration of follow-up based on the length of postoperative ICU stay lessthan 7 days, were utilized. Methods were replicated on the MIMIC-III dataset.Outcomes included 30-day mortality, pulmonary embolism (PE) & pneumonia. Threedomain adaptation & finetuning strategies were implemented for three LLMs(BioGPT, ClinicalBERT & BioClinicalBERT): self-supervised objectives;incorporating labels with semi-supervised fine-tuning; & foundational modellingthrough multi-task learning. Model performance was compared using the AUROC &AUPRC for classification tasks & MSE & R2 for regression tasks. Cohort had amean age of 56.9 (sd: 16.8) years; 50.3% male; 74% White. Pre-trained LLMsoutperformed traditional word embeddings, with absolute maximal gains of 38.3%for AUROC & 14% for AUPRC. Adapting models through self-supervised finetuningfurther improved performance by 3.2% for AUROC & 1.5% for AUPRC Incorporatinglabels into the finetuning procedure further boosted performances, withsemi-supervised finetuning improving by 1.8% for AUROC & 2% for AUPRC &foundational modelling improving by 3.6% for AUROC & 2.6% for AUPRC compared toself-supervised finetuning. Pre-trained clinical LLMs offer opportunities forpostoperative risk predictions with unseen data, & further improvements fromfinetuning suggests benefits in adapting pre-trained models to note-specificperioperative use cases. Incorporating labels can further boost performance.The superior performance of foundational models suggests the potential oftask-agnostic learning towards the generalizable LLMs in perioperative care.