Deep learning based medical image segmentation and visual question answering

Abstract

This work proposes methods for 3D/2D medical image segmentation and medical visual question answering.

For 2D segmentation, we develop a self-learning correction paradigm for semisupervised biomedical image segmentation. Our coarse-to-fine strategy adopts lesion inpainting as a self-supervised pretext task for unlabeled data, which enhances network representations for improved segmentation. This approach leverages unlabeled data to derive additional supervision signals that guide the network to learn better feature representations.

For 3D segmentation, we develop 3D UNeXt, a hybrid CNN-MLP network based on U-Net. 3D UNeXt balances complexity and performance by combining convolutional local feature extraction with MLP blocks for global context propagation. The convolutional layers capture low-level local features while the MLP blocks model long-range dependencies, enabling more effective 3D segmentation.

For medical visual question answering, we introduce a cross-modal self-attention module to selectively capture long-range visual-linguistic contextual relevance for effective feature fusion. More importantly, we reformulate image feature pre-training as a multi-task learning paradigm to make features more applicable for multimodal fusion and question answering. This multi-task pre-training enables image features to encode richer contextual clues that are beneficial for visual question answering.

In summary, we propose 2D/3D segmentation methods and multi-task pre-training with cross-modal self-attention for medical visual question answering. Our techniques achieve superior performance while balancing model complexity, demonstrating their potential to improve segmentation, 3D analysis and multimodal learning for medical imaging applications. The proposed frameworks represent steps towards more comprehensive computational models for medical image analysis.