Lexically constrained text generation

Abstract

Teaching machines to generate high-quality natural texts comparable to human writing, has been a longstanding challenge in natural language processing. For a long time, statistical methods dominated natural language generation. In the past decade, we have witnessed many landmark developments in neural text generation, such as the encoder-decoder structure, attention mechanism, transformer and large-scale pretrained language models, which have made neural text generation far superior to statistical text generation. Pre-trained language models, such as BART and GPT-3, have garnered attention from numerous researchers in recent years due to their remarkable performance in various natural language generation tasks. Nowadays, pre-trained language models have become the standard paradigm for text generation.

Unfortunately, even though large-scale pre-trained language models have shown promising text generation capabilities, controlling the attributes of the generated text, which is usually referred to as controllable text generation, is not an easy task for users. The controllable aspects of text generation can take different forms, ranging from text sentiments (such as positive, negative, and neutral), text topics (such as sports, entertainment, and politics), text formats (such as poems and couplets), or even the identity of the person writing the text (such as gender and age).

In this thesis, we focus on lexically constrained text generation, which falls under controllable text generation. The task involves incorporating pre-specified keywords into generated outputs while maintaining the generation quality. The ability to generate text that meets these criteria has broad implications for downstream tasks, including but not limited to generating dialog responses, crafting stories, composing product advertisements, and creating meeting summaries based on key phrases. This thesis proposes models to improve the generation quality and reduce the inference latency for lexically constrained text generation. We compare our approaches with previous models on different datasets, such as One-Billion-Word, Yelp, CommonGen and Oxford. Extensive experiment results demonstrate the effectiveness of our proposed methods.

The contributions of this thesis to lexically constrained text generation include: (1) designing a two-step approach, “Predict and Revise”, which improves the generation quality by introducing a predictor to guide the model to refine the candidate outputs; (2) inventing Constrained BART (CBART), which accelerates the inference process by refining multiple tokens of the candidate output in parallel and improves the generation quality with the pre-trained model, BART; (3) proposing metric-guided distillation, which enables the retriever and ranker to select more relevant sentences by distilling knowledge from the metric to the ranker and retriever, and applying it to the retrieve-and-generate pipeline for commonsense generation (CommonGen), a much more challenging task related to lexically constrained text generation; (4) introducing dictionary example sentence generation, a useful application related to lexically constrained text generation, developing a controllable target-word-aware model and several baselines, releasing a new dataset, proposing two automatic evaluation metrics for this task, and exploring how to control the readability of the generated examples.