Quantile index regression and time series applications in machine learning

Abstract

Modern technologies have enabled the collection of massive complex and high-dimensional data, which calls for more powerful data analysis tools. One direction is to extend the flexibility and applicability of traditional statistical models. And the other direction is to resort to neural network models, which are more flexible but less interpretable. Some statistical theories can be applied to improve people's understanding of those black-box models and to propose theory-inspired improvements. This thesis studies a new high-dimensional and nonlinear quantile regression model and two neural network architectures inspired by time series analysis. Firstly, a novel partially parametric quantile regression model is proposed, called the quantile index regression. Quantile regression is a class of flexible regression models that can be used to study the structures of response at any quantile level. It is usually difficult for quantile regression to make inference at levels with sparse observations, such as high and low quantiles, and the situation becomes more serious for high-dimensional data. Our model circumvents the data sparsity problem by alternatively conducting the estimating procedure at levels with rich observations and then extrapolating the fitted structures to levels with sparse observations. Asymptotic properties are derived for the case with fixed dimensions, and non-asymptotic error bounds are established for high-dimensional data. Simulation experiments and a real data example demonstrate the usefulness of the proposed model over existing methods. Secondly, this thesis studies the focal problem of the ability of recurrent neural networks (RNNs) to learn long-term dependency. Among RNN variants, the LSTM network was proposed to overcome the difficulty in learning long-term dependence and has made significant advancements in applications. However, we prove that RNN and LSTM do not have long memory from a statistical perspective. As a result, a new definition for long memory networks is further introduced, and it requires the model weights to decay at a polynomial rate. To verify the theoretical results, RNN and LSTM are converted into long memory networks by making minimal modifications, and their superiority is illustrated in modeling the long-term dependence of various datasets. Lastly, this thesis utilizes time series modeling techniques and proposes a recurrent layer aggregation module to improve deep convolutional neural networks for computer vision tasks. A concept of layer aggregation is introduced to describe how information from previous layers can be reused to better extract features at the current layer in a CNN. By making use of the sequential structure of layers in a deep CNN, this thesis proposes a very light-weighted module, called recurrent layer aggregation (RLA). The RLA module is compatible with many mainstream deep CNNs, including ResNets, Xception and MobileNetV2, and its effectiveness is verified by extensive experiments on image classification, object detection and instance segmentation tasks.