Enhanced neural machine translation with external resources

Abstract

Neural machine translation (NMT) is the task of translating a source language sentence into a target language with neural networks. The NMT has achieved superior performance to conventional machine translation approaches and become the dominative commercial translation system due to its impressive performance as well as advantages in training and deployment. However, the NMT still faces several challenges. First, due to its end-to-end nature, it is hard to incorporate lexical constraints, namely pre-specified translation fragments into NMT, which is useful for domain adaptation and interactive NMT. Second, as a data-driven approach, adequate training of the NMT model requires large-scale parallel datasets. Therefore, for low-resource language pairs where parallel sentences are limited, the NMT performance degrades significantly. In this thesis, we aim at enhancing NMT with external resources such as lexical constraints and a multilingual pretrained encoder. We discuss the following topics thoroughly: (1) Incorporating the lexical constraints into NMT. We propose two lexically constrained NMT methods either through data augmentation or the designs of novel decoding algorithms. The former approach trains the NMT model with augmented training data using sampled target phrases as constraints, while the latter method modifies the decoding algorithm using more accurate word alignment. (2) Incorporating the multilingual pretrained encoder into NMT. For this research, we first propose SixT, a zero-shot multilingual NMT model which incorporates a multilingual pretrained encoder into NMT with a position-disentangled encoder and capacity-enhanced decoder. SixT is trained with a novel two-stage transferability-enhanced training framework. Then we extend SixT to multilingual fine-tuning and propose SixT+. SixT+ is not only a multilingual NMT model, but can also serve as a pretrained model for different downstream cross-lingual text generation tasks, such as unsupervised machine translation for extremely low-resource languages and zero-shot cross-lingual abstractive summarization. We compare the proposed models against baselines on various datasets and language pairs with extensive experiments. The results show that our approaches significantly outperform the baselines, demonstrating that we effectively incorporate external resources for improving neural machine translation. Moreover, we conduct a series of in-depth analyses to better understand our proposed methods. All codes for these works are publicly available.