Abstract
Background/Aim: To evaluate efficacy of the AIxURO system, a deep learning-based artificial intelligence (AI) tool, in enhancing the accuracy and reliability of urine cytology for diagnosing upper urinary tract cancers. Materials and Methods: One hundred and eighty-five cytology samples of upper urine tract were collected and categorized according to The Paris System for Reporting Urinary Cytology (TPS), yielding 168 negative for High-Grade Urothelial Carcinoma (NHGUC), 14 atypical urothelial cells (AUC), 2 suspicious for high-grade urothelial carcinoma (SHGUC), and 1 high-grade urothelial carcinoma (HGUC). The AIxURO system, trained on annotated cytology images, was employed to analyze these samples. Independent assessments by a cytotechnologist and a cytopathologist were conducted to validate the initial AIxURO assessment. Results: AIxURO identified discrepancies in 37 of the 185 cases, resulting in a 20% discrepancy rate. The cytotechnologist achieved an accuracy of 85% for NHGUC and 21.4% for AUC, whereas the cytopathologist attained accuracies of 95% for NHGUC and 85.7% for AUC. The cytotechnologist exhibited overcall rates of roughly 15% and undercall rates of greater than 50%, while the cytopathologist showed profoundly lower miscall rates from both undercall and overcall. AIxURO significantly enhanced diagnostic accuracy and consistency, particularly in complex cases involving atypical cells. Conclusion: AIxURO can improve the accuracy and reliability of cytology diagnosis for upper urine tract urothelial carcinomas by providing precise detection on atypical urothelial cells and reducing subjectivity in assessments. The integration of AIxURO into clinical practice can significantly ameliorate diagnostic outcomes, highlighting the synergistic potential of AI technology and human expertise in cytology.
Upper urinary tract conditions such as upper urinary tract urothelial carcinoma (UUTUC) have presented unique challenges in clinical diagnosis and management. UUTUC is a rare malignancy, accounting for 5% of all urothelial cancers, with 60% of cases being invasive at the time of diagnosis (1, 2). The carcinoma in situ in the upper urinary tract is associated with a poor prognosis in patients undergoing radical nephroureterectomy for UUTUC (3). UUTUC is a rare malignancy often diagnosed at more advanced stages that emphasizes an unmet medical need in early detection for improved clinical outcomes. Recently, the utility of upper urinary tract urine cytology, especially in conjunction with the Paris system (TPS) criteria, has been shown to improve the correlation with surgical resection and aid in the assessment of cytologic features associated with malignancy (4, 5).
The management of patients with atypical urinary cytology of the upper urinary tract is often complex due to the potential development of UUTUC (6). Voided urine cytology traditionally used for diagnosing lower urinary tract malignancies has limitations in detecting high-grade urothelial carcinoma (HGUC) in the upper urinary tract due to poor cellular preservation and mechanical distortion (4). Studies have demonstrated the effectiveness of 5-aminolevulinic acid-induced fluorescent selective upper tract urinary cytology in enhancing the sensitivity for diagnosing UUTUC, irrespective of tumor stage and grade (7). In addition, upper urinary tract washings have been shown to outperform voided urine specimens in the diagnosis of HGUC in the upper tract (8). Nonetheless, clinical evaluation of atypical urine cytology progression to malignancy has demonstrated that upper tract specimens diagnosed as atypical elicit the highest probability of progression to high-grade carcinoma (9). The necessity of intraoperative retrograde upper urinary tract cytology examination in patients with suspicious or positive upper tract urine cytology has thus an unsolved hurdle in managing upper urinary malignancy (10).
On the wave of the new AI era, different AI algorithms are increasingly utilized in medicine, including the development of medical robots, disease diagnosis, management of personalized treatment and healthcare (11, 12). In urinary health management, AI-associated healthcare products like Yodoc-m and Toilet Tracker enable self-diagnosis and monitoring of urination patterns using AI and mobile technology (13). For urinary disease diagnosis, AI models such as deep learning and convolutional neural networks are also increasingly employed to assist in identifying conditions like ureteral stricture and urolithiasis through imaging data analysis (14, 15). Recently, AI in combination with urine component detection was shown to enhance analysis accuracy, particularly in diagnosing urothelial carcinoma (16, 17). Previously, we developed an AI algorithm (AIxURO) that demonstrates advances in assistance from AI for improved diagnosis on bladder cancers (18). In this study, we aim to corroborate the efficacy of the AIxURO algorithm in enhancing effectiveness, accuracy and reliability of urine cytology for the diagnosis of upper urinary tract cancers. Given the limitations of conventional cytology, such as subjectivity and variability, the current study substantiates the potential of AI in overcoming these barriers.
Materials and Methods
Sample collection and categorization. We utilized The Paris System (TPS) (5), an internationally recognized system for reporting urinary cytology as our guideline and criteria in standardizing categorization of collected urinary cytology samples, which were categorized into four distinct groups: negative for high-grade urothelial carcinoma (NHGUC), atypical urothelial cells (AUC), suspicious for high-grade urothelial carcinoma (SHGUC), and high-grade urothelial carcinoma (HGUC). Collection of 185 urine cytology samples from patients of upper urinary tract conditions and all experimental protocols were conducted in accordance and approval by the Ethics Review Board of Veteran’s General Hospital Taichung (Institutional Review Board no. CF23221C). Of these 185 specimens, 168 were classified by our senior cytopathologist (CS. Yang) as NHGUC, and 17 were identified with atypical urothelial cells and potential malignant carcinomas.
Cytology slide digitization process. The protocols and procedures employed to convert cytology slides into digital format have been described previously (18, 19). Briefly, a pre-scanning quality check was conducted by a senior cytotechnologist (SJ. Lin) on all slides to confirm proper staining and the absence of any defects, such as cracks, bubbles, or other artifacts, ensuring conditions were optimal for digitization. The slides were then scanned to create whole slide images (WSIs) using the Aperio AT2 scanner (Leica Biosystems, Wetzlar, Germany) equipped with a 40× magnification lens, focusing on a single Z-plane. These WSIs were saved in the proprietary scan scope virtual slide format (SVS) provided by Aperio.
Utilization of a urinary-specific AI algorithm for analyzing WSIs. To ensure the quality of WSIs meets the standard required by the artificial intelligence (AI) algorithm inference, each WSI underwent a thorough cytotechnologist review by focusing on critical image quality parameters, such as clarity and color accuracy. Next, a dedicated deep-learning instance segmentation model we previously developed (18, 19) (AIxMed, Inc.) was specifically tailored for urinary specimen diagnosis (AIxURO). AIxURO is adept at detecting and locating both higher-risk suspicious cells and lower-risk atypical cells, in addition to calculating their cumulative numbers within each WSI (Figure 1A) (18). The integrated viewer at the AIxURO platform visually presents the identified cells while offering detailed quantitative analysis, including metrics such as the nuclear-to-cytoplasmic (N/C) ratio (Figure 1B) and the nuclear area (Figure 1C) for each cell across the entire slide to serve as vital tools for further in-depth evaluation and diagnostic decisions by cytopathologists and cytotechnologists.
Representative image of the AIxURO interface for the analysis of atypical urothelial cells. (A) This interface presents the analytical results from a total of sixteen fields conducted by AIxURO from digitalized atypical urothelial cells (AUC) whole-slide image (WSI). Atypical urothelial cells were automatically highlighted in each WSI. Cells, including those of atypical and suspicious morphology, were quantitatively analyzed based on nuclear-to-cytoplasmic (N/C) ratio (B), and nucleus area (C) (in μm2), as calculated and presented on the right panels of the AIxURO platform. Total cell counts of atypical and suspicious cells are depicted by histogram and bar graphs in yellow and red, respectively.
AIxURO performance evaluation. The clinical and cytopathologic findings of all slide samples were first independently reviewed by our expert panel composed of one board-certificated senior cytopathologist (CS. Yang) and one cytotechnologist (SJ. Lin), both with at least 20 years of experience in cytology analysis. The preparation and digitization procedures for all slides that passed image quality control for testing the performance of the AIxURO were described above. A multi-step evaluation strategy was next employed to assess the effectiveness of AIxURO in assisting cytology analysis. The evaluation process for each sample involved three key steps: after initial analysis by AIxURO, any cases with discrepancies were independently reviewed by expert panelists under a multi-head microscope to attain a unanimous consensus diagnosis. These independent reviews were conducted by an experienced cytotechnologist (SJ. Lin), followed by final validation by a senior cytopathologist (CS. Yang).
Results
AIxURO system leverages a cutting-edge deep learning algorithm trained on a comprehensive collection of annotated cytology images. In the current study, we utilized a rare collection of 185 upper urinary specimens to investigate the capability of AIxURO in identifying subtle anomalies from normal cellular patterns for assisting clinical cytological evaluations with unparalleled accuracy. The significance of nuclear morphometry parameters, such as nuclear size, area, perimeter, diameter, axes and nuclear/cytoplasmic (N/C) ratio in differentiating benign and malignant cells is renowned in cytology analysis (20, 21). Our representative AIxURO system interface showed sixteen fields of cells that were AI-analyzed based on N/C ratios, which ranked the cells as low- or high-risk when the values were 0.5 to 0.7 or above 0.7, respectively (Figure 1A). The average values of N/C ratios were plotted against the number of cells detected to distinguish atypical cells from suspicious cells (Figure 1B). In addition to N/C rations, nuclear area was also utilized to assist determination of atypical or suspicious urothelial cells (Figure 1C).
Conventional diagnosis method diverges from AIxURO on AUC and above. The irregular nuclear contours in upper urinary tract cytology has been the clinical hurdle in predicting high-grade urothelial carcinoma (4) that emphasizes the predictive value and the role of nuclear morphometry in accurate prediction for urothelial carcinoma in tumor recurrence. To ascertain whether AIxURO improves urinary cytology analysis over traditional manual evaluation methods by cytotechnologists and cytopathologists, AIxURO was employed to systematically analyze all of the 185 upper urinary tract specimens. Our data revealed discrepant cytology results in a total of 37 specimens, accounting for a 20% discrepancy rate (Table I). For NHGUC, AIxURO identified a total of 20 discrepant specimens versus 168 identified by the conventional cytology method (11.9% discrepancy rate). Of note, AIxURO demonstrated 100% discrepancy rate to AUC (14 cases), SHGUC (2 cases) and HGUC (1 case) categories (Table I), suggesting a higher degree of difficulty in diagnosing urinary cells that are atypical or suspicious.
Diagnostic discrepancy between conventional cytology and AIxURO analysis on 185 upper urinary specimens.
AIxURO greatly elevates diagnosis accuracy on AUC. To take full advantage of AIxURO system, cells that were recognized as atypical or suspicious by AIxURO from these 37 discrepant cases were subjected to undergoing re-evaluation by a cytotechnologist and a cytopathologist. Independent re-evaluations assisted by AIxURO on the 37 cases performed by a cytotechnologist demonstrated an accuracy rate of 85% on NHGUC cases (17 out of 20), 21.4% on AUC cases (3 out of 14), 50% on SHGUC (1 out of 2) and 100% on HGUC case (1 out of 1) as compared to the ground-truth diagnoses (Table II). Re-evaluations conducted by the cytopathologist, moreover, obtained a 95% of accuracy on NHGUC (19 out of 20), 85.7% on AUC (12 out of 14), 100% on SHGUC and HGUC (2 out of 2 and 1 out of 1, respectively) (Table II). These data indicates that AIxURO-assisted diagnosis conducted by both the cytotechnologist and cytopathologist both reached at least 85% accuracy on NHGUC. Nonetheless, the cytopathologist obtained an 85.7% accuracy versus only 21.4% in the case of the cytotechnologist on AUC, implicating the level of difficulty and extensive experience required in identifying ambiguous AUC from upper urinary tract specimens.
Accuracy of AIxURO-assisted diagnosis from cytotechnologist and cytopathologist.
The miscall rates of the cytopathologist are profoundly improved under assistance of AIxURO. The significantly lowered diagnostic accuracy on AUC by the cytotechnologist was corroborated further by digging into miscall rates of each TPS category. Notably, the cytotechnologist’s undercall cases were elevated to 64.3% for AUC (undercalled as NHGUC), and only 14.3% of AUC were overcalled as SGUC (Table III). As shown in Table III, other miscall cases include 1 SHGUC undercalled as AUC and 3 NHGUC overcalled as AUC by the cytotechnologist. In contrast, there were only 2 AUC cases that had been undercalled and 1 NHGUC case overcalled by the cytopathologist (14.3% and 5.0% miscall rate, respectively). These data demonstrate that the undercall rate on AUC by the cytopathologist was significantly lower than that by the cytotechnologist (14.3% versus 64.3%). Notably, there was no overcall case on AUC, nor undercall cases on SHGUC and HGUC by the cytopathologist, suggesting a significant difference in diagnostic accuracy under AIxURO between the two roles.
Miscalled rates by cytotechnologist and cytopathologist under AIxURO assistance.
Discussion
Despite recent advances in cytology analysis using 5-aminolevulinic acid for UUTUC (7), there is currently no AI algorithm that has been successfully employed to assist diagnosis in these complex cases. The AIxURO system demonstrates the significant potential of AI in improving the accuracy of urinary cytology analysis. By leveraging a deep learning algorithm trained on a comprehensive dataset of annotated cytology images, AIxURO excels in identifying subtle cellular anomalies, which are crucial in distinguishing between benign and malignant cells. In the current study of upper urinary tract specimens, this is manifested by only 20% of discrepancy rate across 185 specimens (that is 80% accuracy based on the 148 consistent cases) from the initial diagnostic assessments (Table I), indicating a superior capability of AIxURO as compared to higher discrepancy rates seen with traditional manual evaluations (18). In line with this, in cervical cancer, a recent study advocates AI as the most effective method for diagnosing low- or high-grade squamous intraepithelial lesion (22).
Notably, the discrepancy rates were significantly higher in the categories of AUC, SHGUC and HGUC, where AIxURO revealed 100% discrepancy in these complex cases. This observation is largely attributed to distinguishing technical artifacts from malignant urothelial cells that still remains a major challenge in urinary cytology, leading to a high number of cases classified as atypical cells and consequently diagnostic uncertainty (23).
The demand for minimally invasive and accurate cytopathological diagnoses has been on the rise at clinics (24). Nonetheless, misinterpretation of atypical cells, especially in cases with morphological similarities between malignant and non-malignant cells, further complicates the diagnostic process (25). To circumvent the misinterpretation problem, we further examined the discrepant 37 cases by performing independent re-evaluations from a cytotechnologist and cytopathologist who were assisted with the AIxURO system, which analyzes nuclear morphometry parameters, such as nuclear size, area, perimeter, diameter, axes, and nuclear/cytoplasmic (N/C) ratios that are essential in cytology for differentiating cell types. Our data demonstrates that diagnostic accuracy of cytopathology successfully achieved at least 85.7% for atypical and suspicious cells in the categories of AUC and above (Table II). In contrast, although the cytotechnologist achieved a comparable accuracy rate of 85% for NHGUC and the cytopathologist a rate of 95%, only 21.4% of accuracy was observed for AUC cases, indicating that extensive experience and expertise significantly influences the diagnostic outcomes when using AI-assisted tools. This disparity highlights the critical role of professional experience in interpreting AI-assisted diagnostics and suggests a potential need for additional training or calibration for cytotechnologists to fully utilize AIxURO. In fact, the misdiagnosis rate was reported as high as 27.6% from an annual average of 57 million cases misdiagnosed in China (26). Thus, both medical imaging analysis and disease diagnosis can depend on personal experience and subjective judgment of the pathologists and clinicians, warranting the need for future potential synergy between AI technology and human expertise.
Moreover, AIxURO also demonstrated a profound impact on reducing miscall rates, particularly in the AUC category. The cytotechnologist exhibited a high undercall rate of 64.3% for AUC cases, significantly higher than the 14.3% undercall rate observed by the cytopathologist (Table III). Lower miscall rates and absence of overcall cases by the cytopathologist further affirmed the robustness of AIxURO when paired with expert analysis. For overcalls, the cytotechnologist miscalled NHGUC as AUC in 15.0% of cases, while the cytopathologist had a 5.0% overcall rate. In fact, a recent study reports a Precision Urine Cytology AI Solution (PUCAS), which analyzes liquid-based urothelial carcinoma images and elicits significant improvements in the sensitivity of urine cytology at detecting malignancy within atypical urothelial cells, thereby reducing the need for invasive ureterorenoscopy examinations. (17). Similarly, another recent study showed the utility of AI (VisioCyt®) in improving diagnosis of bladder carcinoma using voided urine cytology (27). Although these observations are consistent with our previous report that demonstrates the ability of AI in identifying atypical urothelial cells from HGUC cases (18), the present study investigated the most commonly used approach of ureterorenoscopic examinations at clinics and emphasizes the diagnostic sensitivity under AI assistance. Despite recent advances in AI-assisted urine cytology, identifying atypical and suspicious urinary cells with voided urine cytology diagnosis methods can still be challenging due to the complexity and variability of cellular morphology that leads to diagnostic uncertainty in urine cytology (28, 29). In short, our present findings from the integration of AI systems like AIxURO into clinical practice address these challenges via the complementary interaction between AI technology and human expertise, which enhances diagnostic accuracy and reliability especially in potentially malignant cases from upper urine tracts.
Conclusion
The profoundly ameliorated miscall rate by the cytopathologist but not the cytotechnologist highlights the importance of human ability to validate AI-generated findings effectively. Of note, the complementary nature of human expertise remains crucial as cytopathologists occasionally recognize atypical cells initially missed by AI technology. Future research should focus on refining AI algorithms and enhancing training for clinicians to utilize these advanced tools effectively. The promising results from this study advocate for the broader adoption of AI-assisted diagnostics in cytology, potentially transforming urinary cytology evaluations.
Acknowledgements
The Authors would like to thank Tien-Jen Liu, Wei-Lei Yang, Ming-Yu Lin, Cheng-Hung Yeh and Shih-Wen Hsu from AIxMed Inc. (Taiwan) for their expertise and assistance in AI-associated analyses. This study was supported by TCVGH-1115801D from Taichung Veterans General Hospital Taiwan.
Footnotes
Authors’ Contributions
Conceptualization: KYC, CSY and CJC; methodology: JYL, SJL, JRL, TJL; formal analysis and investigation: KYC, CSY and CJC; data curation: WLY, MYL, CHY, SWH and CJC; original draft preparation: KYC, CSY and CJC; writing, reviewing, and editing: JYL, KYC, CSY and CJC. All Authors read and approved the final manuscript.
Conflicts of Interest
The Authors declare no conflicts of interest in relation to this study.
- Received July 10, 2024.
- Revision received August 12, 2024.
- Accepted August 17, 2024.
- Copyright © 2024 The Author(s). Published by the International Institute of Anticancer Research.
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY-NC-ND) 4.0 international license (https://creativecommons.org/licenses/by-nc-nd/4.0).
