基于结合Transformer和卷积神经网络的生成对抗网络在磁共振成像中分割胎盘组织（英文）
PDF下载 (228)叶正洁,王玉涛,徐 建,金 炜.基于结合Transformer和卷积神经网络的生成对抗网络在磁共振成像中分割胎盘组织（英文）[J].宁波大学学报(理工版),2023,36(1):22-34.DOI:
YE Zhengjie,WANG Yutao,XU Jian,JIN Wei.Transformer and CNN combined generative adversarial network for segmentation of placental tissue in MR images[J].Journal of Ningbo University(Natural Science & Engineering Edition),2023,36(1):22-34.DOI:
| Title: | Transformer and CNN combined generative adversarial network for segmentation of placental tissue in MR images |
| 作者: | 叶正洁, 王玉涛, 徐 建, 金 炜 |
| Author(s): | YE Zhengjie, WANG Yutao, XU Jian, JIN Wei |
| 关键词: | 深度学习; 胎盘组织分割; 生成对抗网络; 核磁共振成像; 注意力机制 |
| Keywords: | deep learning; placental tissue segmentation; adversarial network; MRI; attention mechanism |
| 分类号: | TP391.4 |
| 文献标识码: | A |
| 摘要: | 磁共振成像(MRI)胎盘组织的准确分割对于研究妊娠和分娩并发症具有重要意义, 但传统放射科医师的人工标注难以保证分割准确性和客观性, 且费时费力. 为了开发用于MRI中胎盘组织自动分割的深度学习模型, 提出了结合Transformer和卷积神经网络(CNN)的生成对抗网络(TCGANet). 将特征嵌入模块与跳跃连接相结合, 缓解传统特征融合方法带来的信息丢失. 在此基础上引入内容提取模块, 采用Transformer的自注意力机制捕捉全局依赖关系, 有效表示MRI的全局和局部信息. 此外, 鉴于传统分割方法难于精确界定MR影像胎盘组织边缘的问题, 运用判别网络对胎盘组织分割的生成网络监督, 以提高胎盘边缘界定的精度. 结果表明, 该模型在定量指标和边界定位精度方面显著优于现有分割方法，其中准确度为0.993±0.003, 灵敏度为0.903±0.093, 特异度为0.996±0.003, Dice相关性系数为0.861±0.141. 对模型不同结构的消融实验验证了网络结构设计的合理性, 大部分性能指标明显优于现有方法(P<0.05). 该模型能够实现自动且准确地分割MRI中胎盘组织. |
| Abstract: | Accurate segmentation of placental tissue in magnetic resonance (MR) images is of great significance for the study of pregnancy and childbirth complications. However, manual annotation by radiologists is difficult to ensure the accuracy and objectivity of the results, and it is time-consuming and labor-intensive. To develop a deep learning model for automated placental tissue segmentation in MR images, we propose a generative adversarial network (TCGANet) that combines transformer and convolution neural network (CNN) to realize the segmentation purposes. It combines the feature embedding module with the skip connection which avoids the information loss caused by the traditional feature fusion method. On the basis of this technique, the context extractor module is introduced, and the self-attention mechanism of the transformer is adopted to capture the global dependencies, which can effectively represent the global and local information of MR images. In addition, to improve the precision of placental tissue edge segmentation, the generative adversarial architecture is introduced to supervise the generative network of placental tissue segmentation by the discriminator network. The results show that the proposed model is superior to other comparison methods in terms of quantitative metrics and boundary positioning accuracy, the accuracy of which reads 0.993±0.003, sensitivity reaches 0.903±0.093, specificity is 0.996±0.003, and Dice similarity coefficient is found to be 0.861±0.141. Ablation experiments on different structural units of the TCGANet have validated the network structure design, and most of the performance metrics are far better than those of the existing methods (P<0.05). The proposed TCGANet enables automated and accurate segmentation of placental tissue in MR images. |
| 参考文献 /References: | [1] Jansen C H J R, Kastelein A W, Kleinrouweler C E, et al. Development of placental abnormalities in location and anatomy[J]. Acta Obstetricia et Gynecologica Scandinavica, 2020, 99(8):983-993. [2] Hecht J L, Baergen R, Ernst L M, et al. Classification and reporting guidelines for the pathology diagnosis of placenta accreta spectrum (PAS) disorders: Recommendations from an expert panel[J]. Modern Pathology, 2020, 33(12):2382-2396. [3] S?rensen A, Hutter J, Seed M, et al. T2*-weighted placental MRI: Basic research tool or emerging clinical test for placental dysfunction?[J]. Ultrasound in Obstetrics & Gynecology, 2020, 55(3):293-302. [4] Maher M A, Abdelaziz A, Bazeed M F. Diagnostic accuracy of ultrasound and MRI in the prenatal diagnosis of placenta accreta[J]. Acta Obstetricia et Gynecologica Scandinavica, 2013, 92(9):1017-1022. [5] Chen Y, Hu F, Wang Y R, et al. Hybrid-attention densely connected U-Net with GAP for extracting livers from CT volumes[J]. Medical Physics, 2022, 49(2):1015-1033. [6] Bayramoglu N, Kannala J, Heikkil? J. Deep learning for magnification independent breast cancer histopathology image classification[C]//2016 IEEE 23rd International Conference on Pattern Recognition (ICPR), Cancun, Mexico, 2017:2440-2445. [7] Xing J, Chen C, Lu Q Y, et al. Using BI-RADS stratifications as auxiliary information for breast masses classification in ultrasound images[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 25(6):2058-2070. [8] Zheng S Y, Guo J P, Cui X N, et al. Automatic pulmonary nodule detection in CT scans using convolutional neural networks based on maximum intensity projection[J]. IEEE Transactions on Medical Imaging, 2020, 39(3):797-805. [9] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[M]//Lecture Notes in Computer Science. Cham: Springer International Publishing, 2015:234-241. [10] Alom M Z, Yakopcic C, Hasan M, et al. Recurrent residual U-Net for medical image segmentation[J]. Journal of Medical Imaging, 2019, 6(1):014006. [11] Oktay O, Schlemper J, Folgoc L L, et al. Attention U-Net: Learning where to look for the pancreas[EB/OL]. [2022-06-10]. https://arxiv.org/abs/1804.03999. [12] Araújo J D L, da Cruz L B, Ferreira J L, et al. An automatic method for segmentation of liver lesions in computed tomography images using deep neural networks[J]. Expert Systems with Applications, 2021, 180:115064. [13] McKinley R, Wepfer R, Aschwanden F, et al. Simultaneous lesion and brain segmentation in multiple sclerosis using deep neural networks[J]. Scientific Reports, 2021, 11:1087. [14] Lei Y, Dong X, Tian Z, et al. CT prostate segmentation based on synthetic MRI-aided deep attention fully convolution network[J]. Medical Physics, 2020, 47(2):530-540. [15] Valanarasu J M J, Oza P, Hacihaliloglu I, et al. Medical transformer: Gated axial-attention for medical image segmentation[C]//International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Cham, 2021:36-46. [16] Zhao H S, Shi J P, Qi X J, et al. Pyramid scene parsing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017:2881-2890. [17] Chen L C, Papandreou G, Kokkinos I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[EB/OL]. [2022-06-10]. https://arxiv.org/abs/1412.7062. [18] Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651. [19] Li X M, Dou Q, Chen H, et al. 3D multi-scale FCN with random modality voxel dropout learning for inter- vertebral disc localization and segmentation from multi- modality MR images[J]. Medical Image Analysis, 2018, 45:41-54. [20] Li K Q, Qi X Q, Luo Y W, et al. Accurate retinal vessel segmentation in color fundus images via fully attention- based networks[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 25(6):2071-2081. [21] Shahedi M, Dormer J D, Anusha Devi T T, et al. Segmentation of uterus and placenta in MR images using a fully convolutional neural network[C]//SPIE Medical Imaging: Computer-Aided Diagnosis, Houston, Texas, USA, 2020, 11314:411-418. [22] Creswell A, White T, Dumoulin V, et al. Generative adversarial networks: An overview[J]. IEEE Signal Processing Magazine, 2018, 35(1):53-65. [23] Luc P, Couprie C, Chintala S, et al. Semantic segmentation using adversarial networks[EB/OL]. [2022-06-10]. https://arxiv.org/abs/1611.08408. [24] Yang X Y, Zhang Y, Lo B, et al. DBAN: Adversarial network with multi-scale features for cardiac MRI segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 25(6):2018-2028. [25] Xue Y, Xu T, Zhang H, et al. SegAN: Adversarial network with multi-scale L1 loss for medical image segmentation [J]. Neuroinformatics, 2018, 16(3):383-392. [26] Dai W, Dong N Q, Wang Z Y, et al. SCAN: Structure correcting adversarial network for organ segmentation in chest X-rays[C]//Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision, Springer, Cham, 2018:263-273. [27] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[EB/OL]. [2022-06-10]. https://www.proceedings.com/content/034/034099webtoc.pdf. [28] Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[M]. Cham: Springer International Publishing, 2020:213-229. [29] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[EB/OL]. [2022-06-10]. https://arxiv.org/abs/2010.11929. [30] Chen J N, Lu Y Y, Yu Q H, et al. TransUNet: Transformers make strong encoders for medical image segmentation[EB/OL]. [2022-06-10]. https://arxiv.org/abs/2102.04306. [31] Gao Y H, Zhou M, Metaxas D N. UTNet: A hybrid transformer architecture for medical image segmentation [M]//Medical Image Computing and Computer Assisted Intervention - MICCAI 2021. Cham: Springer International Publishing, 2021:61-71. [32] Zhang Y D, Liu H Y, Hu Q. TransFuse: Fusing transformers and CNNs for medical image segmentation [M]//Medical Image Computing and Computer Assisted Intervention - MICCAI 2021. Cham: Springer International Publishing, 2021:14-24. [33] Korkmaz Y, Dar S U H, Yurt M, et al. Unsupervised MRI reconstruction via zero-shot learned adversarial transformers[J]. IEEE Transactions on Medical Imaging, 2022, 41(7):1747-1763. [34] Dalmaz O, Yurt M, Cukur T. ResViT: Residual vision transformers for multimodal medical image synthesis[J]. IEEE Transactions on Medical Imaging, 2022, 41(10):2598-2614. [35] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016:770-778. [36] Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017:936-944. [37] Pang S C, Du A N, Orgun M A, et al. Tumor attention networks: Better feature selection, better tumor segmentation[J]. Neural Networks, 2021, 140:203-222. |
| 备注/Memo: | 收稿日期: 2022－07－19. 宁波大学学报（理工版）网址: http://journallg.nbu.edu.cn/基金项目: 浙江省自然科学基金（LY20H180003）; 宁波市自然科学基金（2019A610104）; 宁波市公益类科技计划项目（202002N3104）.第一作者: 叶正洁（1996－）, 女, 浙江丽水人, 在读硕士研究生, 主要研究方向: 医学影像. E-mail: 1303698906@qq.com*通信作者: 金炜（1969－）, 男, 浙江金华人, 教授, 主要研究方向: 数字图像处理. E-mail: xyjw1969@126.com 宁波大学学报（理工版）网址：http://journallg.nbu.edu.cn/ |
