Samir Cayennesacayenn@utmb.eduJohn Sealy School of Medicine

Natalia Penalozanapenalo@utmb.eduJohn Sealy School of Medicine, The University of Texas Medical Branch, Galveston, TX

Anne C. Chanacchan@utmb.eduJohn Sealy School of Medicine, The University of Texas Medical Branch, Galveston, TX

M.I. Tahashildermdtahash@utmb.eduOffice of Biostatistics, The University of Texas Medical Branch, Galveston, TX

Rodney C. Guiseppircguisep@utmb.eduDepartment of Ophthalmology & Visual Sciences, The University of Texas Medical Branch, Galveston, TX

Touka Banaeetobanaee@utmb.eduDepartment of Ophthalmology & Visual Sciences, The University of Texas Medical Branch, Galveston, TX

Purpose: ChatGPT-3.5 has the potential to assist ophthalmologists by generating a differential diagnosis based on patient presentation.
Methods: One hundred ocular pathologies were tested. Each pathology had two signs and two symptoms prompted into ChatGPT-3.5 through a clinical vignette template to generate a list of four preferentially ordered differential diagnoses, denoted as Method A. Thirty of the original 100 pathologies were further subcategorized into three groups of 10: cornea, retina, and neuroophthalmology. To assess whether additional clinical information affected the accuracy of results, these subcategories were again prompted into ChatGPT-3.5 with the same previous two signs and symptoms, along with additional risk factors of age, sex, and past medical history, denoted as Method B. A one-tailed Wilcoxon signed-rank test was performed to compare the accuracy between Methods A and B across each subcategory (significance indicated by P < 0.05).
Results: ChatGPT-3.5 correctly diagnosed 51 out of 100 cases (51.00%) as its first differential diagnosis and 18 out of 100 cases (18.00%) as a differential other than its first diagnosis. However, 31 out of 100 cases (31.00%) were not included in the differential diagnosis list. Only the subcategory of neuro-ophthalmology showed a significant increase in accuracy (P = 0.01) when prompted with the additional risk factors (Method B) compared to only two signs and two symptoms (Method A).
Conclusion: These results demonstrate that ChatGPT-3.5 may help assist clinicians in suggesting possible diagnoses based on varying complex clinical information. However, its accuracy is limited, and it cannot be utilized as a replacement for clinical decision-making.

Keywords: Artificial Intelligence, ChatGPT, Cornea, Neuro-ophthalmology, Retina

1. Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, et al. Assessing the utility of ChatGPT throughout the entire clinical workflow. MedRxiv 2023:2023.02.21.23285886.

2. Potapenko I, Boberg-Ans LC, Stormly Hansen M, Klefter ON, van Dijk EH, Subhi Y. Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT. Acta Ophthalmol 2023;101:829–831.

3. Feldman B. The EyeWiki initiative. Virtual Mentor 2010;12:922–924.

4. Hirosawa T, Harada Y, Yokose M, Sakamoto T, Kawamura R, Shimizu T. Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: A pilot study. Int J Environ Res Public Health 2023;20:3378.

5. Cappellani F, Card KR, Shields CL, Pulido JS, Haller JA. Reliability and accuracy of artificial intelligence ChatGPT in providing information on ophthalmic diseases and management to patients. Eye 2024;38:1368–1373.

6. Delsoz M, Raja H, Madadi Y, Tang AA, Wirostko BM, Kahook MY, et al. The use of ChatGPT to assist in diagnosing glaucoma based on clinical case reports. Ophthalmol Ther 2023;12:3121–3132.

7. Vaishya R, Misra A, Vaish A. ChatGPT: Is this version good for healthcare and research? Diabetes Metab Syndr 2023;17:102744.

8. Teebagy S, Colwell L, Wood E, Yaghy A, Faustina M. Improved performance of ChatGPT-4 on the OKAP examination: A comparative study with ChatGPT-3.5. J Acad Ophthalmol 2023;15:e184–7.

9. Ting DS, Tan TF, Ting DS. ChatGPT in ophthalmology: The dawn of a new era? Eye 2024;38:4–7.

10. Tao BK, Hua N, Milkovich J, Micieli JA. ChatGPT-3.5 and Bing Chat in ophthalmology: An updated evaluation of performance, readability, and informative sources. Eye 2024;38:1897–1902.