Prompt-Based Out-of-Distribution Intent Detection

Abstract

Recent rapid advances in pre-trained language models, such as BERT and GPT, in natural language processing (NLP) have greatly improved the efficacy of text classifiers, easily surpassing human level performance in standard datasets like GLUE. However, most of these standard tasks implicitly assume a closed-world situation, where all testing data are supposed to lie in the same scope or distribution of the training data. Out-of-distribution (OOD) detection is the task of detecting when an input data point lies beyond the scope of the seen training set. This is becoming increasingly important as NLP agents, such as chatbots or virtual assistants, have been being deployed ubiquitously in our daily lives, thus attracting more attention from the research community to make it more accurate and robust at the same time. Recent work can be broadly categorized into two orthogonal approaches - data generative/augmentative methods and threshold/boundary learning. In this work, we follow the former and propose a method for the task based on prompting, which is known for its zero and few-shot capabilities. Generating synthetic outliers in terms of prompts allows the model to more efficiently learn OOD samples than the existing methods. Testing on nine different settings across three standard datasets used for OOD detection, our method with adaptive decision boundary is able to achieve competitive or superior performances compared with the current state-of-the-art in all cases. We also provide extensive analysis on each dataset as well as perform comprehensive ablation studies on each component of our model.