Addressing Out-of-Distribution Challenges in Image Semantic Communication Systems with Multi-modal Large Language Models

Abstract

Semantic communication is a promising technology for next-generation wireless networks. However, the out-of-distribution (OOD) problem, where a pre-trained machine learning (ML) model is applied to unseen tasks that are outside the distribution of its training data, may compromise the integrity of semantic compression. This paper explores the use of multi-modal large language models (MLLMs) to address the OOD issue in image semantic communication. We propose a novel 'Plan A - Plan B' framework that leverages the broad knowledge and strong generalization ability of an MLLM to assist a conventional ML model when the latter encounters an OOD input in the semantic encoding process. The novel framework integrates the anti-OOD ability of MLLMs with the domain expertise of ML models in tasks they have been trained for, thus enhancing the accuracy in the semantic encoding process. Further, at the receiver side of the communication system, we put forth a 'generate-criticize' framework that allows one MLLM to challenge the image generated by another MLLM, which then revises the generated image in the next iteration. The joint effort of the two MLLMs significantly enhances the reliability of image reconstruction.