Background Modern natural language tools have potential to improve clinical workflows, but few have been successfully deployed in practice. Here, we present the development, deployment, and evaluation of an AI language tool for generating preliminary clinical sign-outs in a urine drug testing service.

Methods Large language models (LLMs) were used to extract substance use patterns from 83,553 urine drug test interpretations. We then trained an AI model using these data to predict substance use from qualitative and quantitative urine testing results. Predicted substance use patterns were used to create preliminary clinical sign-out statements, which were then integrated into an existing clinical workflow. Pre- and post-deployment user studies were performed to evaluate model performance and user experience within this workflow.

Results LLM-based extraction of substance-use patterns was 99.9% accurate, outperforming human labelling. Substance use prediction was similarly accurate, with area under the ROC curve > 0.99 across 33 drug categories. Workflow integration reduced clinical sign-out times by 65s per case (51% efficiency gain), with the greatest benefits seen for less experienced users.

Conclusions AI-based interpretation of urine drug testing was fast and accurate, providing significant efficiency gains to the clinical service. This demonstrates that natural language tool integration can provide substantial clinical benefit, without comprising quality of care.

The authors have declared no competing interest.

This study was not supported by external funding

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This research was undertaken using an ethics and research protocol approved by the University of Washington Medicine Institutional Review Board, under a waiver of informed consent due to minimal risk (ID: STUDY00019458). Model deployment was performed under Clinical Laboratory Improvement Amendments (CLIA) and College of American Pathology (CAP) regulatory guidelines for custom software and laboratory information system tools, and after creation of an ongoing verification and monitoring plan.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Due to restrictions on sharing protected health information, patient level data cannot be shared.