Abstract
Background/Aim: In this study, we introduce an innovative deep-learning model architecture aimed at enhancing the accuracy of detecting and classifying organizing pneumonia (OP), a condition characterized by the presence of Masson bodies within the alveolar spaces due to lung injury. The variable morphology of Masson bodies and their resemblance to adjacent pulmonary structures pose significant diagnostic challenges, necessitating a model capable of discerning subtle textural and structural differences. Our model incorporates a novel architecture that integrates advancements in three key areas: Semantic segmentation, texture analysis, and structural feature recognition. Materials and Methods: We employed a dataset of whole slide imaging from 20 patients, totaling 100 slides of OP, segmented into training, validation, and testing sets to reflect real-world application scenarios. Our approach utilizes a modified multi-head self-attention mechanism combined with ResUNet for semantic segmentation, enhanced by superpixel concepts. This method facilitates the generation of representative token features through iterative super-token blocks, creating high-resolution token maps that leverage local and high-level feature information for improved accuracy. Results: Benefiting from token features and distribution for enhanced texture alignment with fewer false-positives, the super-token transformer (STT) model achieved a mean intersection over union (mIOU) of 72.42%, with a sensitivity of 47.81%, specificity of 99.83%, positive predictive value of 64.03%, and negative predictive value of 99.94%, highlighting superior efficacy in Masson body segmentation in complex cross-tissue analyses. Conclusion: Our team developed an iterative learning model based on the STT approach, emphasizing token features of super token, including texture and distribution, that enable enhanced alignment with the unique textures of Masson bodies to improve sensitivity and mIOU, The development of this STT model presents a significant advancement in the field of medical image analysis for OP that offers a promising avenue for improving diagnostic precision and patient outcomes in pulmonary pathology.
Organizing pneumonia (OP), also known as cryptogenic organizing pneumonia, is a pulmonary disorder that affects individuals across all age groups (1). The incidence of this condition is estimated to be around 1-2 cases per 100,000 individuals annually (2, 3). The diagnostic process for OP can be challenging due to the overlap of its clinical and radiological features with other respiratory conditions (4).
Masson bodies, which are histomorphological features in OP, form in the alveolar space in response to lung injury, leading to damage to the epithelial basal laminae (5). However, diagnosing Masson bodies pathologically presents challenges due to morphological variations in their shape and the inclusion of diverse tissue types, making differentiation from adjacent tissues complex (6, 7). This complexity underscores the need for more robust detection methodologies to enhance accuracy in identifying Masson bodies during pathological examination.
Diagnosis of OP typically involves a comprehensive approach including clinical evaluation, radiological imaging, and histopathological examination of lung tissue obtained through biopsy (8). High-resolution computed tomography scans often reveal distinct patchy or consolidative opacities in the lungs, which are characteristic findings of this condition (9, 10). Nonetheless, variabilities in clinical assessments have presented unsolved medical challenges in accurately identifying complex histomorphology features like Masson bodies, emphasizing the importance of a comprehensive diagnostic approach to ensure timely and appropriate management of this condition. Recently, the breakthrough of transformers in natural language processing has inspired their application in computer vision (11, 12).
The integration of transformers in computer vision tasks is a rapidly evolving field. Recent research has shown the successful integration of vision transformers (ViTs) into various computer vision tasks, demonstrating strong global information-capturing abilities in facilitating traditional image classification tasks (13). ViTs process sequential inputs without recurrent or convolutional operations has also enabled the use of transformers for image encoding by treating images as sequences of patches (14). Recently, transformer models have been adapted for different vision tasks such as image classification, segmentation, and semantic segmentation, achieving promising results beyond conventional classification methods (15). While convolutional neural networks (CNNs) are known for capturing locality and translation invariance, transformers have also shown promise in modeling long-range dependencies, making them suitable for tasks such as object re-identification (16). Further, the self-attention mechanism as well as the convolution-free design of transformers have led to their superior performance and increased applications in computer vision (17). Nevertheless, current mainstream ViT models largely rely on square patch-based tokens, of which can often be limited by the shape and mixture feature. Due to this limitation of token generation at the level of pixel features, minority features are often overlooked and lead to informational constraints. In this scenario, semantic objects separated by tokens often lack a full distribution or shape that can be disadvantageous in the resulting segmentation of OP.
In the current study, inspired by the concept of superpixeling, which groups similar pixels into coherent units that often align with object boundaries, we aimed to apply this concept in token generation. Superpixel methods (2, 3) are currently hindered by their reliance on initial, manually set hyperparameters, which may not be universally applicable. We therefore adopted a learnable super-token structure that iteratively updates pixel feature extraction weights to enhance clustering and allow self-attention maps to incorporate simple texture feature and distribution information, thereby building high-resolution token maps to facilitate diagnosis and differentiation of Masson bodies in OP from other pulmonary conditions.
Materials and Methods
Patient samples. This study received formal approval and support from the Institutional Review Board of Taichung Veterans General Hospital, with a commitment to rigorous adherence to ethical guidelines and the preservation of patient privacy (Institutional Review Board number: CE23165A). The microscopic examination of surgical specimens for consensus diagnosis of OP involved reviews by two pathologists (JTF and YHT).
Simple and high-resolution token maps. The super-pixel concept of a single element containing single texture feature was employed to develop the iterative token blocks in reference to super-token vision transformer (STViT) (18). After processing token blocks, token maps emerged as a low-resolution pixelate image. Due to the importance of distribution of features in image information, an affinity map was utilized to generate each token distribution feature and concatenate the distribution feature and texture feature as a token feature as follows:
In the final step of the token-generation process, the super token encapsulates a single texture and distribution feature, with its visual range spanning across a single square token in the high-resolution token map.
Reconstruction of image structure. The STT model utilizes multi-head self attention (MHSA) and integrates elements of ResNet’s architecture to construct a comprehensive image structure. Different from previous models that either processed token features directly through MHSA without considering larger regions of interest or dispersed token features via CNNs that potentially led to loss of representative of token feature, the STT backbone maintained token continuity and aggregated tokens at higher levels for a holistic analysis to preserve tokens as basic elements of image, allowing reconstruction of image structure to proceed with higher information efficiency and lower noise.
Segmentation head. Since semantic segmentation of OP, depends not only on classification of the tissue texture level, but also on the identification of tissue level, in the segmentation head, we employed ResNet and U-Net (19) as encoder and decoder of the feature pyramid network for multi-scale super-token classification. xcls refers to a special classification token used in the token classifier in the transformer-based model to aggregate information from all tokens for a final classification decision.
The token classifier produces class logits for all tokens except xcls:
The classification result was then filled in with the self-attention distribution map from the super-token layer established (20). For pixel p=[u,v]:
Where y(s) is the result of token classifier, and qs (p) is the affinity between the token and the pixel represented as a probability. In semantic segmentation of OP, since the tissue texture features are similar, and led to similar probability, we utilized the softmax function to enhance the clustering the probabilities.
Results
Patient demographics and clinical features of the study population such as radiological findings, lung function, comorbidity, and other potential risk factors for each patient were documented (Table I), showing most participants were between 51 to 70 years old, about half had associated malignancy and a minority had a smoking history. Various lung function statuses were represented, with half of the participants having normal function and nodular being the prevalent pattern, followed by consolidation and ground-glass opacity. To overcome current challenges in Masson body segmentation, a variety of normal histological structures aside from Masson body, such as pleura, pulmonary artery, bronchial wall and bronchiolar mucosa were included (Figure 1). The principal aim of our investigation was to evaluate the performance of our STT model in Masson body segmentation, a pivotal task in the pathological diagnosis of diverse pulmonary conditions including OP.
Patient demographics and pathological characteristics.
Representative images of ambiguous normal histological texture features that result in challenges for Masson body segmentation and analysis. (A) Masson body, (B) pleura, (C) pulmonary artery, (D) bronchial wall, and (E) bronchiolar mucosa. Magnification was 20×.
Firstly, whole-slide images of OP specimens containing Masson bodies were utilized as input images with dimensions and three-color RGB channels (H, W, 3). Initial images were then divided into smaller square regions or ‘tokens’ (s, s, 3), where s was the token size. Next, an average pixel feature in each square region to retrieve a representative token feature. Next, by using the representative token feature and pixel feature in the neighbor tokens (3×3), we generated an affinity map which represented the similarity distribution of tokens in probability, for the probability map to be used at the end of segmentation head. A backbone base compared segmenter with spatial tokens on MHSA and residual networks with U-Net (ResUNet) (21) was used as multi-scale token classification and was fed with tokens, generating output as a multi-channel feature map with an increase in depth from dimensions (H, W, 9, C) as the neural network architecture learns a richer representation of the data. The classification results from the ResUNet were then combined with the similarity map to fill in or refine the prediction masks, with a binary or segmented image used to indicate areas of interest within the original histopathology image (Figure 2).
Super-token transformer workflow for attention-based image segmentation in histopathological analysis of Masson bodies. Schematic representation of our attention-based segmentation pipeline for histopathological image analysis. The process initiates with a standard color image (H, W, 3), which is subdivided into smaller, square tokens (s, s, 3). Firstly, an average of pixel feature in each token is used to obtain a representative token texture feature. Secondly, an attention mechanism between pixel and token creates a similarity map and informs the pixel-token attention process. By concatenating token texture features and distribution features generated from similarity map, a simple and high-resolution token map emerges, and is subsequently processed through multi-head self attention (MHSA) and Residual U-Net (ResUNet) architecture to yield a multi-channel feature map. The outputs of the ResUNet are the multi-scale classification of the token that eventually utilizes the similarity map to enhance the precision of the segmentation. The culmination of this process is a segmented binary image that highlights areas of interest within the original histopathological image from whole-slide images of organizing pneumonia, demonstrating the model’s predictive segmentation capabilities.
To quantitatively assess the efficacy of our STT model, a suite of robust performance metrics was comparatively analyzed against the STViT reference model (18). We first attempted to assess the extent of overlap between the STT-generated segmentations and the ground truth annotations examined by pathologists, as indicated by the mean intersection-over-union (mIOU). As shown in Table II, the mIOU score for Masson body segmentation from STT was 72.42%, implicating the ability of the STT model to align its predictions with manual assessments by pathologists. The robust ability of STT was comparative to that of the STViT model (mIOU of 71.42%).
Comparative performance metrics of super-token transformer (STT) and super-token vision transformer (STViT).
Sensitivity, another critical metric in medical image analysis, which assesses a model’s ability to accurately identify true-positive cases was evaluated. In the context of Masson body segmentation, our STT model exhibited a commendable sensitivity of 47.81%, as compared to 35.82% of that scored by the reference STViT model (Table II). This result indicates effective detection of a significant proportion of actual Masson bodies within the images, mitigating the risk of false-negatives and ensuring the inclusion of relevant pathological features. To validate this finding, model specificity, a pivotal performance metric that assesses the ability of a model to accurately identify negative cases, was determined. In this evaluation, the STT model demonstrated a noteworthy specificity of 99.83% while STViT scored 99.82% (Table II), reflecting the ability to discern non-Masson body regions with a high degree of accuracy from both models.
Moreover, the positive predictive value (PPV) is a well-recognized critical measure of a model’s precision in identifying key features within medical images (22). Our STT model attained a PPV of 64.03%, signifying the ability of STT to identify the presence of Masson bodies with a high likelihood of accuracy. In comparison, the reference STViT model scored only 60.14%. By contrast, the negative predictive value (NPV) is used to evaluate whether a model is capable of confidently identifying areas where the key feature of an image is absent. In our study, the STT model demonstrated a robust NPV of 99.94%, indicating a similar ability to identify the absence of Masson bodies to that of STViT (Table II).
Discussion
Masson bodies play a pivotal role in OP that arises within the alveolar space and yet their identification in pathological diagnosis remains a notable challenge (22). To date, their recognition of Masson bodies during pathological examination is still complicated by their morphology, which manifests considerable variations from case to case (23, 24). These structural complications are further confounded by texture features akin to nearby pulmonary artery and pulmonary pleurae, which intensify the complexity of accurately distinguishing them from adjacent tissues (9, 24). In response to these challenges, our present study thus endeavored to develop a computer vision model for Masson body semantic segmentation, with the aim of developing a model that adeptly harnesses tissue texture characteristics to discriminate structural features across diverse OP tissue specimens.
Our STT model architecture integrated learnable iterative super-token blocks, pixel and token self-attention mechanisms, and image structure reconstruction components. STT was established via systemic training, validation, and testing using a curated dataset comprising 19 OP cases and 100 specimens (Table I).
To preserve image information from token features, maintain higher resolution through distribution information, and collect information beyond token boundaries, the STT model contrasts with traditional CNNs that may lose information during square region convolution. Mounting evidence argues that transformers exhibit robustness compared to CNNs (25). In fact, the self-attention mechanism utilized in STT potentially contributed to improved performance in identifying Masson body features. This is in line with recent studies that demonstrate self-attention-based architectures such as ViTs can outperform CNN models in various vision tasks while being computationally efficient (26, 27), providing plausible explanation for enhanced performance in detection of Masson body by STT in metrics such as mIOU, sensitivity and PPV over STViT (Table II). Further, the high specificity and high NPV also suggests this model is able to distinguish Masson bodies and its mimickers precisely. In short, STT development has brought about a paradigm shift in the field of medical image analysis by enabling improved connections between individual pixels within the image data, generating a comprehensive probability map. This study provides an innovative computer vision model STT that addressed limitations inherent in conventional models, thereby enhancing the diagnosis of OP via accurate detection of Masson bodies to revolutionize pathological diagnosis in diverse pulmonary conditions.
Acknowledgements
This study was supported by TCVGH-1115801D from Taichung Veterans General Hospital Taiwan.
Footnotes
Authors’ Contributions
Conceptualization: JTF, YST and CJC; methodology: JTF, YST and PCC; formal analysis: JTF, YST, PCC and YHT; investigation: YST, PKF and CJC; data curation: JTF, YST, PCC, PKF and CJC; writing the original draft preparation: JTF, YST and CJC; writing, reviewing, and editing: JTF, YST and CJC. All Authors read and approved the final article.
Conflicts of Interest
The Authors declare that they have no conflicts of interest in relation to this study.
- Received April 19, 2024.
- Revision received June 12, 2024.
- Accepted June 26, 2024.
- Copyright © 2024 The Author(s). Published by the International Institute of Anticancer Research.
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY-NC-ND) 4.0 international license (https://creativecommons.org/licenses/by-nc-nd/4.0).
References
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.