While large language models (LLMs) have shown promise in medical text processing, their real-world application in self-hosted clinical settings remains underexplored. Here, we evaluated five state-of-the-art self-hosted LLMs for automated assignment of International Classification of Diseases for Oncology (ICD-O-3) codes using 21,364 real-world pathology reports from a large German hospital. For exact topography code prediction, Qwen3-235B-A22B achieved the highest performance (micro-average F1: 71.6%), while Llama-3.3-70B-Instruct performed best score at predicting the first three characters (micro-average F1: 84.6%). For morphology codes, DeepSeek-R1-Distill-Llama-70B outperformed other models (exact micro-average F1: 34.7%; first three characters micro-average F1: 77.8%). Large disparities between micro- and macro-average F1-scores indicated poor generalization to rare conditions. Although LLMs demonstrate promising capabilities as support systems for expert-guided pathology coding, their performance is not yet sufficient for fully automated, unsupervised use in routine clinical workflows.

The authors have declared no competing interest.

J.Keyl is supported by a German Research Foundation (DFG)-funded clinician scientist program (FU 356/12-2). The work of Bahadir Eryilmaz and Mikel Bahn was funded by a PhD grant from the DFG Research Training Group 2535 Knowledge- and data-based personalization of medicine at the point of care (WisPerMed).

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This study was approved by the Ethics Committee of the Medical Faculty of the University of Duisburg-Essen (approval number 23-11557-BO). Due to the study's retrospective nature, the requirement of written informed consent was waived by the Ethics Committee of the Medical Faculty of the University of Duisburg-Essen. All methods were carried out in accordance with relevant guidelines and regulations.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes