Informer: Irregular traffic detection for containerized microservices RPC in the real world

Abstract

Containerized microservices have been widely deployed in the industry. Meanwhile, security issues also arise. Many security enhancement mechanisms for containerized microservices require predefined rules and policies. However, it is challenging when it comes to thousands of microservices and a massive amount of real-time unstructured data. Hence, automatic policy generation becomes indispensable. In this paper, we focus on the automatic solution for the security problem: irregular traffic detection for RPCs. We propose Informer, a two-phase machine learning framework to track the traffic of each RPC and automatically report anomalous points. We first identify RPC chain patterns using density-based clustering techniques and build a graph for each critical pattern. Next, we solve the irregular RPC traffic detection problem as a prediction problem for attributed graphs with time series by leveraging spatial-temporal graph convolution networks. Since the framework builds multiple models and makes individual predictions for each RPC chain pattern, it can be efficiently updated upon legitimate changes in any graphs. In evaluations, we applied Informer to a dataset containing more than 7 billion lines of raw RPC logs sampled from a large Kubernetes system for two weeks. We provide two case studies of detected real-world threats. As a result, our framework found fine-grained RPC chain patterns and accurately captured the anomalies in a dynamic and complicated microservice production scenario, which demonstrates the effectiveness of Informer. Furthermore, we extensively evaluated the risk of adversarial attacks for our prediction model under different reality constraints and showed that the model is robust to such attacks in most real-world scenarios.