Advanced general-purpose Large Language Models (LLMs), including OpenAI’s Chat Generative Pre-trained Transformer (ChatGPT), Google’s Gemini and Anthropic’s Claude, have demonstrated capabilities in answering clinical questions, including those with image inputs. The Thai National Medical Licensing Examination (ThaiNLE) lacks publicly accessible specialist-confirmed study materials. This study aims to evaluate whether LLMs can accurately answer Step 1 of the ThaiNLE, a test similar to Step 1 of the United States Medical Licensing Examination (USMLE). We utilized a mock examination dataset comprising 300 multiple-choice questions, 10.2% of which included images. LLMs capable of processing both image and text data were used, namely GPT-4, Claude 3 Opus and Gemini 1.0 Pro. Five runs of each model were conducted through their application programming interface (API), with the performance assessed based on mean accuracy. Our findings indicate that all tested models surpassed the passing score, with the top performers achieving scores more than two standard deviations above the national average. Notably, the highest-scoring model achieved an accuracy of 88.9%. The models demonstrated robust performance across all topics, with consistent accuracy in both text-only and image-enhanced questions. However, while the LLMs showed strong proficiency in handling visual information, their performance on text-only questions was slightly superior. This study underscores the potential of LLMs in medical education, particularly in accurately interpreting and responding to a diverse array of exam questions.

The authors have declared no competing interest.

The author(s) received no specific funding for this work.

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Exempted–no humans involved

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Researchers seeking additional information may contact the corresponding author to discuss potential collaborations or restricted access under specific agreements.