An overview of the experimental process is schematically summarized in Fig. 1. As shown in this figure, we compared the accuracy of cancer staging across three groups: REK+/RAG+ (NotebookLM with REK), REK+/RAG− (Gemini 2.0 Flash with REK), and REK−/RAG− (Gemini 2.0 Flash without REK). Among these, the REK+/RAG+ group represents our proposed method, while the other two groups served for comparison. In the REK+/RAG+ group, the REK was uploaded to the NotebookLM web system, which automatically retrieved only the excerpts relevant to the user input for staging and presented them together with the model’s classification outputs. In the REK+/RAG− group, the entire REK text was manually provided to the LLM without retrieval, so the model processed the full document directly. In the REK−/RAG− group, no REK was given to the LLM, and staging relied on the CT findings together with the model’s pretraining knowledge. At the time of the experiment, the LLM integrated into NotebookLM was Gemini 2.0 Flash; thus, the underlying language model was the same across all three groups. For Gemini 2.0 Flash, the temperature parameter was set to zero to promote output consistency.

Fig. 1The alternative text for this image may have been generated using AI.

An overview of the experimental process. Radiologists from our team generated CT findings for 100 fictional pancreatic cancer patients. NotebookLM with REK (REK+/RAG+), Gemini 2.0 Flash with REK (REK+/RAG−), and Gemini 2.0 Flash without REK (REK−/RAG−) conducted cancer staging based on the CT findings in response to Tasks 1–5 (see Table 2). REK = reliable external knowledge, RAG = retrieval-augmented generation

2.1 Data preparation

Two radiologists (in their 6th and 11th years of clinical practice, respectively; the latter is board-certified) from our team generated CT findings for 100 fictional pancreatic cancer patients, along with staging components (TNM classification, local invasion factors, and resectability classification) based on the latest pancreatic cancer staging guidelines in Japan—the eighth edition of the Japanese classification of pancreatic carcinoma [13]. The CT findings and staging components were subsequently reviewed and confirmed by four additional radiologists (two in their 7th year, one board-certified in the 12th year, and one board-certified abdominal imaging specialist in the 21st year of clinical practice) and one gastroenterologist (with 8 years of clinical practice, board-certified). A breakdown of the staging components for the 100 fictional pancreatic cancer patients is provided in Table 1. The number of cases is comparable to prior clinical studies [14, 15], but at this sample size, replicating real-world distributions inevitably leads to the absence of certain staging components. To address this, we designed the dataset to reflect the general imbalance observed in clinical practice—particularly in T-factors and local invasion patterns—while also ensuring coverage of the full range of categories. Specifically, we intentionally included a small number of rare cases so that no staging component was entirely absent. Six radiologists confirmed that the distribution reasonably reflects clinical practice. All the CT findings with staging components are available in Supplementary file 1. Below is an example from the dataset, presenting the first of the 100 cases:

Case 1 CT findings: A nodular pancreatic cancer measuring 20 mm is observed in the body of the pancreas. No local invasion factors are noted. Lymph node metastases are identified in two nodes at station 10 and two nodes at station 11d. No other metastases are observed.

Case 1 staging components: T factor: T1c; N factor: N1b; M factor: M0; Local invasion factors: CH0, DU0, S0, RP0, PV0, A0, PL0, OO0; Resectability classification: R.

Table 1 Breakdown of staging components for the 100 fictional pancreatic cancer patients2.2 Preparation of REK and user input

We attempted to use an open-access paper [13], which summarizes the current pancreatic cancer staging guidelines in Japan, as REK for NotebookLM and Gemini 2.0 Flash. However, since the full text of the paper (5,418 words, covering Chaps. 1 to 8, including figure captions and tables) could not be entered into the prompt field at once, we adopted REK consisting of 4,376 words from the paper (Chaps. 1 to 6), omitting the final two chapters that were not relevant to staging.

To enable the LLM to perform pancreatic cancer staging, we provided Tasks 1–5, shown in Table 2, as user input in the prompt field, followed by the CT findings for each case. In the REK+/RAG+ group, we uploaded the REK to the NotebookLM web system for RAG processing. In the REK+/RAG− group, we manually entered the REK into the prompt field before providing Tasks 1–5. In the REK−/RAG− group, we entered a prompt instructing adherence to the Japanese Classification of Pancreatic Carcinoma, Eighth Edition by the Japan Pancreas Society before providing Tasks 1–5. To assess potential variability in staging results, identical inputs were submitted twice for each of the 100 cases under each LLM setting across independent runs, and the resulting staging classifications were consistent.

Table 2 User input for LLMs to perform pancreatic cancer staging2.3 Evaluation

For a given case, cancer staging was regarded as correct if all staging components—TNM classification, local invasion factors, and resectability classification—were simultaneously determined correctly. Staging accuracy was then defined as the proportion of correctly staged cases among the 100 examined. Staging accuracy was compared across the three groups: REK+/RAG+, REK+/RAG−, and REK−/RAG−. For staging accuracy, the exact McNemar’s test was applied. Based on the observed concordant and discordant outcomes per case, we tested the null hypothesis that the population proportions of correctly staged cases were equal between groups, and calculated the corresponding p-values. Specifically, pairwise comparisons were performed between the REK+/RAG + and REK+/RAG− groups, between the REK+/RAG − and REK−/RAG− groups, and between the REK+/RAG + and REK−/RAG− groups. Additionally, for each staging component (T, N, M factors, and resectability), accuracy was defined as the proportion of cases (out of the 100 examined) in which the component was correctly classified. The TNM classification was considered correct only when all T, N, and M factors were accurately classified. For local invasion factors, a case was considered correct only when all eight factors (CH, DU, S, RP, PV, A, PL, and OO) were simultaneously classified correctly. For both the TNM classification and local invasion factors, accuracy was defined as the proportion of correctly classified cases among the 100 examined.

In the REK+/RAG+ group, retrieved excerpts from REK via NotebookLM were available for reference. Therefore, we examined these excerpts for each case and evaluated retrieval accuracy. Retrieval was considered accurate if the excerpts contained sufficient information to correctly classify all of the staging components. This definition is based on the context recall measure in the RAGAS framework [16,17,18], a recently developed suite of metrics for evaluation of RAG. Specifically, let \(\:C\) denote the set of all chunks (i.e., smaller text units) obtained from REK by NotebookLM. For each case \(\:i=1,\:\dots\:,\:100\), let \(\:_\subset\:C\) be the retrieved set, let \(\:_^,\:\dots\:,\:_^\}\) be the set of claims of the correct answer, and let

$$\:_\left(_^\right)=\left\1\:\:\:\:\text\:_^\:\text\:_\\\!\!\!\!\!\!\!0\:\:\:\:\text\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\end\:\:\:\:\:\:\:\:\left(j=1,\dots\:,k\right).\right.$$

Then, the context recall for case \(\:i\) is defined as

$$ \:}_ = \frac\sum\limits_}^ \left( ^ } \right)} $$

Since \(\:\text}_<1\) indicates insufficient retrieval, we applied the floor function \(\left\lfloor \cdot \right\rfloor\) to obtain a stricter, more conservative evaluation and defined the retrieval accuracy for the overall 100 cases as

$$ \:} = \frac}\sum\limits_}^} }_ } \right\rfloor } $$

        

For each case \(\:i\), the definition of \(\:_\) was independently reviewed by one radiologist (with 11 years of clinical practice, board-certified), one gastroenterologist (with 8 years of clinical practice, board-certified), and one engineer with medical training, to ensure consistency in claim attribution. Occasional discrepancies due to minor misunderstandings were observed, but all were resolved through mutual review, resulting in inter-rater agreement on the final definition of every \(\:_\).

The LLM’s answers for each case in the three groups (REK+/RAG+, REK+/RAG−, and REK−/RAG−), along with the retrieved excerpts in the REK+/RAG+ group, are provided in Supplementary file 2 to enable transparent verification by readers. Additionally, the case-wise accuracy of each staging component across the three groups is presented in Supplementary file 3, while the case-wise retrieval accuracy in the REK+/RAG+ group is provided in Supplementary file 4. Furthermore, Supplementary file 5 summarizes, for each model, the number of correctly classified cases for every staging component.