Keywords
Endoscopic ultrasonography - Biliary tract - Fine-needle aspiration/biopsy - Tissue diagnosis
Introduction
Cholangiocarcinoma (CCA) is a malignancy of the biliary epithelium, classified by tumor location into intrahepatic (iCCA), perihilar (pCCA), and distal (dCCA) subtypes [1]. Notably, CCA is often diagnosed at an advanced stage, limiting curative treatment options [2]
[3]
[4]. Only 15% of patients are eligible for curative resection [5]
[6]
[7]. Likelihood of resection is influenced by tumor characteristics such as vascular involvement, local extension, and presence of metastases [8].
Lymph node metastases (LNM) have a profound impact on survival, particularly in iCCA and pCCA, reducing median survival from 60 months to 20 months for iCCA and from 31 to 42 months to 13 months for pCCA [9]
[10]
[11]
[12]. Therefore, accurate preoperative evaluation of LN involvement is essential. Cross-sectional imaging modalities like computed tomography and magnetic resonance imaging have limited sensitivity for detecting LNM [13]. Endoscopic ultrasound (EUS), however, offers the advantage of tissue acquisition (EUS-TA) through fine-needle aspiration or biopsy, potentially preventing unnecessary invasive laparotomies [14]
[15]
[16]
[17]
[18].
Several predictors of malignancy among pancreato-biliary malignancies have been proposed, including round shape, homogeneity, size > 10 millimeter (mm), sharply demarcated borders, and hypoechogenicity [19]
[20]. However, interobserver variability among endosonographers in assessing these characteristics is poorly understood, particularly in CCA [21]
[22]. This study aimed to assess interobserver agreement in EUS-based LN evaluation in patients with iCCA and pCCA and to determine the accuracy of identifying malignant LNs based on morphology.
Methods
Study design
This study employed a cross-sectional survey design involving endosonographers from the QUEST consortium (QUality in EndoSonography Team, established in 2015 [23]) as well as endosonographers participating in the ongoing Preoperative Evaluation of Lymph nodes in Cholangiocarcinoma (POELH) trial (Clinicaltrials.gov NCT05678218). Study reporting was checked according to the Checklist for Reporting Of Survey Studies (CROSS) (Supplementary File) [24]. The study protocol was approved by the Medical Ethics Committee at the Erasmus MC University Medical Center (MEC-2023–0281, date of approval: 05–06–2023). A total of 145 endosonographers from both peripheral and academic centers were invited to participate, all with experience in using EUS to evaluate LNs. Because CCA treatment is centralized in the Netherlands in a total of seven academic hospitals, only academic endosonographers performed preoperative LN staging in the setting of CCA. All participants were sent a personal link (single-use link) with access to a website to review EUS still images of LNs and complete the survey through the web-based database Castor [25]. The survey was open for completion during a time period of approximately 3 months, without prior sample size calculation.
EUS image set
EUS images were collected during the aforementioned POELH trial and informed consent was retrieved from these patients. All EUS examinations were performed using the EG38-J10-UT Pentax scope. In total, 42 images from both benign and malignant LNs from different patients were selected. All images were then de-identified, and were randomly ordered in the survey.
In the POELH trial, which enrolled patients in six academic hospitals in the Netherlands and one center in Belgium, patients with presumed resectable perihilar, intrahepatic, and mid-common bile duct (CBD) CCA underwent preoperative EUS focusing on LN staging. To do so, a strict protocol was followed to systematically check both regional and extraregional LNs. In addition, all LNs with a short-axis > 5 mm were punctured, whenever safely possible. Images were taken of all identified LNs during EUS assessment. Regional and extraregional LN were defined according to the 8th edition of the American Joint Committee on Cancer staging manual [26]. In short, for pCCA, regional LNs consisted of LNs at the liver hilum, cystic duct, common bile duct, hepatic artery, and portal vein, and extraregional LNs consisted of LNs at the peri-aortic region, peri-caval region, superior mesenteric artery, and celiac trunk. For iCCA, regional LNs were defined differently than for iCCA located in the left or right hemi-liver. Extraregional LNs for both iCCA sides were LN at the peri-aortic region, peri-caval region, and celiac trunk.
Survey
The survey consisted of two sections (Supplementary File). In the first part, background information about the endosonographer was collected by requesting the following information: healthcare institution of employment (general or academic), total number of EUSs performed independently per year, and experience with EUS in the context of CCA. Level of expertise was classified as a composite endpoint of > 1000 procedures with > 200 to 300 EUSs per year with > 5 years of experience, mostly in line with the European Society of Gastrointestinal Endoscopy guidelines on EUS training curriculum [27].
In the second part, 42 EUS images were shown. Examples are shown in [Fig. 1] and [Fig. 2]. Information was provided about the type of CCA, short-axis of the LN in mm, and location of the LN station, according to the 3rd edition of the Japanese Society of Hepato-Biliary-Pancreatic Surgery classification of biliary tract cancers [28]. Participants had to give their opinion about six LN characteristics for each particular LN (Supplementary File). The following characteristics were requested in multiple choice: demarcation of the LN, echogenicity, homogeneity, shape, whether they suspected the LN to be malignant, and if so, whether they would perform tissue acquisition or not.
Fig. 1 Lymph node station 12P1 at the portal vein. Surgery showed iCCA with benign LN.
Fig. 2 Lymph node station 9 at the celiac trunk. Presumed pCCA with EUS-TA (blue LN) showing LNM precluding surgery.
Objectives
The primary objective of this study was to assess the level of agreement among endosonographers regarding EUS-based evaluation of LN characteristics in patients with presumed resectable iCCA or pCCA. The secondary objective was to evaluate the accuracy of endosonographers in differentiating between benign and malignant LNs based on EUS characteristics. Analysis was stratified by participant experience and institutional setting.
Statistical analysis
Interobserver agreement was quantified using Light’s kappa statistics with 95%
confidence intervals (CI) calculated for each LN characteristic [29]. Subgroup analyses were performed to compare agreement between experienced and less
experienced endosonographers, as well as between academic and general hospital
practitioners. Inter-rater reliability (IRR) was performed to evaluate consistency in
assigning LN characteristics [30]. Kappa values for each coder pair were averaged to provide an overall IRR index,
with bootstrapping employed to calculate 95% CIs using 1000 replicates.
To calculate a kappa value, it is essential that each rater demonstrated variability in their responses. When a rater provided repetitive answers across all questions (i.e. all LNs were rated as having sharp demarcation), the rater was excluded from the analysis, in accordance with the assumptions of kappa analysis. Kappa values were computed for each pair of raters, and these values were subsequently averaged to provide an overall index of IRR [29]. To quantify uncertainty of the estimated kappa statistics, a bootstrapping method was employed generating a 95% CI using 1000 bootstrap replicates. Kappa statistics and bootstrap replicates were calculated using the “vcd” and “boot” package in R, version 4.2.2. Non-response error was not addressed, as only completely filled in surveys were used.
Kappa statistics were interpreted according to Landis and Koch’s criteria: value < 0.4 indicated fair agreement, 0.4 to 0.6 moderate agreement, 0.6 to 0.8 substantial (good) agreement, and values > 0.8 almost perfect agreement [31].
Overall accuracy of identifying malignant LNs was calculated by constructing 2×2 tables comparing suspicious/non-suspicious findings with pathologically confirmed malignant/benign outcomes. Responses from endosonographers who answered "I don’t know" were excluded. All responses with “Other” were assigned to suspicious/non-suspicious accordingly. Sensitivity, specificity, and individual endosonographer accuracy were also calculated. LNs without pathological confirmation were excluded from the accuracy analysis.
Results
Characteristics of endosonographers and EUS images
Of the 145 endosonographers invited to participate, 24 (16.6%) completed the survey. Details regarding their experience and expertise are summarized in [Table 1]. Half of the endosonographers conducted their procedures in academic hospitals, and 13 (54.2%) reported that they had independently performed more than 1000 EUS.
Table 1 Baseline characteristics of participating endosonographers.
|
Characteristics
|
n (%)
|
|
CCA, cholangiocarcinoma; EUS, endoscopic ultrasound; LN, lymph node.
|
|
Endoscopist sex
|
|
|
15 (62.5)
|
|
|
9 (37.5)
|
|
Type of hospital
|
|
|
12 (50.0)
|
|
|
12 (50.0)
|
|
Experience with EUS in setting of CCA
|
|
|
12 (50.0)
|
|
|
12 (50.0)
|
|
Number of EUSs performed per year
|
|
|
3 (12.5)
|
|
|
6 (25.0)
|
|
|
8 (33.3)
|
|
|
4 (16.7)
|
|
|
3 (12.5)
|
|
Lifetime number of EUSs performed independently
|
|
|
2 (8.3)
|
|
|
9 (37.5)
|
|
|
13 (54.2)
|
Characteristics of the 42 collected LNs are described in [Table 2]. The majority of patients had iCCA or pCCA without underlying liver disease. Only 9.5% of patients had CCA in the context of primary sclerosing cholangitis (PSC). Of all reported LNs, 54.8% was defined as regional and 45.2% as extraregional. Median short-axis diameter of all LNs was 6.0 mm (interquartile range [IQR] 4.6–8.0). Only two LNs were >10 mm. Pathology results were available for 29 of 42 LNs (69.1%).
Table 2 Characteristics of included LNs.
|
Characteristics
|
Total
(n = 42)
|
pCCA
(n = 24)*
|
iCCA
(n = 17)
|
|
EUS-TA, endoscopic ultrasound-guided tissue acquisition; iCCA, intrahepatic cholangiocarcinoma; IQR, interquartile range; LNM, lymph node metastasis; pCCA, perihilar cholangiocarcinoma; PSC, primary sclerosing cholangitis.
*One patient with two LNs was included in the survey.
†One LN had benign EUS-TA, but surgery showed malignancy.
|
|
Regional (vs. extraregional) – n (%)
|
23 (54.8)
|
17 (70.8)
|
6 (35.3)
|
|
PSC diagnosis – n (%)
|
4 (9.5)
|
2 (8.3)
|
2 (11.8)
|
|
Short-axis diameter <5 mm – n (%)
|
13 (31.0)
|
8 (33.3)
|
5 (29.4)
|
|
Short-axis diameter >10 mm – n (%)
|
2 (4.8)
|
0 (0)
|
2 (11.8)
|
|
Short-axis diameter – median (IQR), mm
|
6.0 (4.6–8.0)
|
5.6 (4.3–8.0)
|
5.9 (4.7–8.2)
|
|
EUS-TA performed – n (%)
LNM
|
20 (47.6)
10 (23.8)
|
7 (29.2)
1 (4.2)
|
13 (76.5)
9 (21.4)
|
|
Surgical confirmation – n (%)
LNM
|
10 (23.8)
1 (2.4)
|
9 (37.5)
1 (4.2)
|
1 (5.9)
0 (0)
|
|
Combined confirmation – n (%)
LNM
|
29 (69.0)†
11 (26.2)
|
15 (62.5)†
2 (8.3)
|
14 (82.4)
9 (52.9)
|
Interobserver agreement
Overall diagnostic Light’s kappas were as follows: 0.24 (95% CI 0.17–0.34) for demarcation, 0.45 (95% CI 0.38–0.54) for shape, 0.38 (95% CI 0.32–0.46) for echogenicity, 0.52 (95% CI 0.42–0.64) for homogeneity, 0.36 (95% CI 0.27–0.45) for suspicion for malignancy, and 0.27 (95% CI 0.21–0.33) for the decision to perform an EUS-TA. For the kappa associated with “echogenicity”, two endosonographers were excluded due to lack of variability in their assessments. Similarly, for the kappa related to performing an EUS-TA, one endosonographer was excluded for the same reason. Sensitivity analysis for LN in which pathology was available revealed no significant differences in kappa values, as shown in the Supplementary File.
Subgroup analyses
Subgroup analyses comparing experienced vs. non-experienced endosonographers and endosonographers from academic vs. general hospital are presented in [Fig. 3] and [Fig. 4], respectively. These analyses indicate a trend in which experienced endosonographers demonstrated higher kappa values across the six lymph node characteristics, a pattern that was similarly reflected in the comparison between endosonographers working in academic versus general hospital settings.
Fig. 3 Kappa values per LN characteristic of experienced vs non-experienced endosonographers. CI, confidence interval.
Fig. 4 Kappa values per LN characteristic of endosonographers working in academic and general hospitals. CI, confidence interval.
Suspicion
Overall accuracy in correctly identifying a malignant LN was 62%, ranging from 44% to 75%. Overall sensitivity was 60%, whereas individual endosonographers exhibited a sensitivity range with a lower bound of 29% and an upper bound of 90%. Likewise, overall specificity was 64%, with a range of 28% to 89%.
Discussion
Preoperative LN assessment using EUS is of utmost importance for patients with presumed resectable CCA because LNM significantly worsen prognosis, even when curative surgery is attempted. Accurate preoperative LN evaluation, therefore, can help prevent unnecessary surgical interventions that might not improve survival and could potentially diminish patient quality of life. This study showed limited interobserver agreement among endosonographers regarding assessment of LN morphology in CCA patients, as well as in accurate identification of malignant LNs.
Findings from this study align with previous research examining EUS performance in LN evaluation and description of important characteristics. For instance, Takasaki et al. reported fair to moderate interobserver agreement for shape, demarcation, and echogenicity, which is consistent with results from this study [21]. Similarly, other studies on LN assessment, although not specifically focused on CCA, have shown moderate and fair agreement on the same LN features [32].
Subgroup analysis revealed slightly higher but not statistically significant agreement among experienced endosonographers compared with their less experienced counterparts. This finding is consistent with previous studies, which also reported that even endosonographers with > 5000 EUS procedures were not unanimous in their assessment [22]. This underscores the need for standardized and unified approaches to LN assessment, because current variations in interpretation may result in different LNs being targeted for tissue acquisition, depending on the endosonographer.
In addition to low agreement about LN characteristics, estimation of whether a specific LN is malignant exhibited only moderate accuracy. Overall accuracy for identifying malignant LNs was approximately 64%, with sensitivity ranging from 27% to 91% among individual endosonographers. Given that surgical resection remains the only curative option for patients with iCCA and pCCA [5]
[6]
[7], accurately identifying malignant LNs is crucial for determining resectability. Although characteristics such as size (> 10 mm), round shape, sharp demarcation, and homogeneity are considered indicative of malignancy in mediastinal LNs, these criteria have not been conclusively validated for pancreato-biliary malignancies, including CCA [19]
[20].
The present study further supports the notion that relying solely on LN morphology in EUS assessment is insufficient to reliably identify malignant LNs. Furthermore, the anatomical location of LNs is a significant factor in determining resectability. Presence of extraregional LNM generally warrants palliative treatment, whereas regional LNM may require a case-by-case evaluation regarding appropriateness of surgery [8]. However, distinguishing regional from extraregional LNs during EUS can be challenging, and this ambiguity complicates decisions about which LNs should be targeted for tissue acquisition. One objective feature that is less prone to interobserver variability is LN size. In light of this, LNs with a diameter > 10 mm should always be considered for biopsy [19]
[20]. Further research, such as that being conducted in the POELH trial, is necessary to clarify whether LNs measuring between 5 and 10 mm should also be targeted. The potential application of artificial intelligence (AI) in EUS-guided LN evaluation remains underexplored but could represent a future solution to current challenges in LN characterization [33]. Studies about AI for EUS-guided LN evaluation should take into account interobserver agreement by training the system on videos of LNs in which EUS-TA was performed, not based on assessment of individual endosonographers.
Several limitations should be acknowledged. First, pathological confirmation of LN status was available for only a subgroup of patients. Although the POELH trial protocol stipulated puncturing all LNs with a short-axis > 5 mm, fewer than half of these LNs were biopsied due to procedure constraints. This may have led to underestimation of the number of malignant LNs. In addition, use of static images for evaluating LN characteristics does not fully capture the dynamic nature of real-time EUS, potentially impacting accuracy of LN assessments. Follow-up studies on interobserver agreement on LN should aim to include videos and not still images. Finally, the low response rate (16.6%) and high proportion of expert participants may limit generalizability of the findings. It is likely that this selection bias affects the results and interpretability of this study. However, this may have resulted in an underestimation of interobserver variability rather than an overestimation, thereby reinforcing the robustness of the study conclusions, but not the findings themselves.
Conclusions
In conclusion, this study underscores the significant challenges associated with visual interpretation of abdominal LN characteristics in patients with iCCA and pCCA. Even among expert endosonographers, interobserver variability was high, and accuracy in identifying malignant LNs based on morphology was moderate. These findings underscore the need for standardized criteria to guide LN tissue acquisition, and further research is needed to evaluate the role of AI in improving reliability of EUS-guided LN evaluation.
Bibliographical Record
David Michaël de Jong, Daniëlle Roosterman, Marco J. Bruno, Lydi M.J.W. van Driel, Wim J. Lammers. Interobserver variability in lymph node evaluation with endoscopic ultrasonography in cholangiocarcinoma. Endosc Int Open 2025; 13: a25775449.
DOI: 10.1055/a-2577-5449
