This study demonstrates that the deep-learning segmentation model COALA for patients with CRLM is user-friendly when used within the radiological PACS. The evaluation sessions with expert abdominal radiologists provided valuable insights for further optimization and integration of COALA in the clinical setting, such as the decreased segmentation performance in smaller lesions, heterogeneous lesions and lesions at the border of the liver. Key lessons for successful implementation of AI models for future researchers were a multidisciplinary approach, proposing a robust method prior to model development with external validation and organizing feedback sessions with the end-users early in the development phase of the model, to meet their demands and needs.

To facilitate the clinical implementation of automatic segmentation models, seamless integration in the daily workflow of clinical radiologists is crucial. By involving the end-user (i.e., abdominal radiologists) in the development process and testing the model in their daily workflow, essential improvements can be made. Some radiologists evaluated the performance of the model as moderate, but did report that they would be confident in using the model for clinical decision-making. The reason for this is that, especially in small lesions, the performance was reported to be lower. In large lesions, however, the performance was deemed accurate by most radiologists. In current practice, only the two largest CRLM are used for RECIST1.1 assessment, explaining the seemingly contradictory outcome in evaluated model performance and confidence in clinical decision-making. The possibility of automating manual radiological evaluations, such as response evaluation using RECIST1.1 criteria through automated measurements, stands out as a promising application. Additionally, the model has the potential to advance TTV as an imaging biomarker in clinical practice, currently hindered by a labor-intensive manual segmentation process.

A strength of this study is that the patients included for this clinical evaluation resembled the clinical practice better than the patients used for model development. The latter group consisted of patients enrolled in an RCT, all meeting highly specific inclusion criteria to be enrolled in the trial. Patients from the CAIRO5 trial suffered from initially unresectable liver-only CRLM and had never undergone local treatment (surgery and/or ablation) before. Patients in this study were randomly identified from the CRLM MDT, some of them suffering from recurrent CRLM after local treatment, resulting in livers with operation clips and/or post-ablation zones. Testing COALA on such patients ascertains the generalizability of the model by gaining insights into how to optimize COALA’s performance on patients with small lesions and previously treated livers.

Various other segmentation models have been developed for patients with CRLM. None of the earlier developed models were, however, evaluated in the radiological workflow or clinically implemented. Most of the models never surpass the development phase and are only internally validated [16,17,18]. One group did externally validate their automatic segmentation model, resulting in a significant decrease in performance in the external cohort [19]. These findings are in concordance with findings by other research groups focusing on the clinical implementation of models. This emphasizes the critical need for external validation and clinical evaluation to ultimately facilitate clinical implementation [20, 21].

This study has several limitations, such as the ongoing development of radiological features like automatic tumor diameter measurements and lesion tracking over time. Both the model and the application need to be further developed to include these features. A more seamless integration in the hospital IT infrastructure is possible with the use of further containerization, streaming, and by deploying a web-API. Overall, the robustness of the software code could increase with the use of automatic deployments. Furthermore, the model is integrated into the PACS of the current vendor of our academic hospital. Adaptation to a different PACS could potentially pose challenges, although the produced results are DICOM conform. Lastly, an impediment to the complete integration of the model in the radiological workflow is the absence of a clear triggering system for the model. At the moment, communication between EHR, Radiology Information System, PACS and HPC cluster is fragmented, and therefore, a manual trigger to run the model for a specific patient remains necessary. Several institutes have taken effort to describe and build such a dedicated AI deployment infrastructure [22]. Standards-based guidelines for AI integration developed by Integrating the Healthcare Enterprise (IHE) Radiology AI Workflow are currently in trial implementation [23].

Future directions include a thorough assessment of the model’s segmentation and volumetric assessment capabilities during MDTs, specifically focusing on its ease of use, interpretability, and, foremost, clinical relevance.

In summary, this study demonstrates the successful integration of an externally validated automatic segmentation model for CRLM (COALA) into the radiological PACS, highlighting the model’s potential for enhancing radiological assessments. Valuable insights for further optimization and integration of COALA in the clinical setting were gathered. Key lessons for successful implementation for future researchers were a multidisciplinary approach, proposing a robust method prior to model development and organizing feedback sessions with the end-users early in the development.