Language models in NLP : from architecture design to downstream application

Abstract

Natural language processing (NLP) is a rapidly evolving field, and language models (LMs) play a critical role in advancing research in various NLP tasks, such as language generation, machine translation, sentiment analysis, and question-answering. This thesis presents our contributions toward advancing the research in language models from two perspectives: the design of language model architecture and downstream application.

In the first part of the thesis, we aim to enhance the ability of pre-trained language models by discovering an efficient and powerful architecture. Instead of resorting to manual design, we pioneer an approach to automatically discover novel pre-trained language model (PLM) backbone within a flexible search space. To this end, we introduce an efficient Neural Architecture Search (NAS) method, termed OP-NAS, which concurrently optimizes the search algorithm and the evaluation of prospective models. The architecture discovered through this process, referred to as AutoBERT-Zero, significantly surpasses the performance of BERT and its variants across various downstream tasks, while also exhibiting exceptional transfer and scaling abilities.

In the second part of this thesis, we explore the practical applications of language models, drawing upon their recent success in the field. Specifically, we examine two primary directions: effective downstream adaptation and the extension of language models to broader domains beyond natural language processing (NLP). In particular, we first introduce SunGen, a novel framework that enables the efficient adaptation of pre-trained language models (PLMs) to downstream tasks. SunGen enhances the quality of PLM-generated data, allowing for the training of a compact task-specific model with substantially fewer parameters. This approach not only achieves superior performance to that of the original PLM but also offers greater efficiency during training and inference. Then, we demonstrate the potential of language models beyond NLP by presenting a novel unpaired cross-lingual method for generating image captions. This method enables captioning tasks to be performed for languages without any caption annotations, effectively bridging the gap between vision and language understanding across different languages. Overall, this thesis contributes to realizing the full potential of language models and provides new insights for future research in this rapidly evolving field.