Nonparametric and adaptive tests for high-dimensional data

Abstract

Study of population mean is of great interest to researchers especially for high-dimensional multivariate data in recent years. But due to the curse of dimensionality, traditional methods developed for mean testing are rendered useless since the covariance matrix of high-dimensional data is singular and thus not invertible. In this thesis, a two-sample test as well as a K-sample test (K>=3) are proposed for testing high-dimensional mean equality.

For the two-sample tests of high-dimensional mean vectors, the classical methods are often particularly designed to test sparse or dense mean differences. However, the sparsity level of mean differences is often unknown. Also, the mean differences can have varying magnitudes, while they are often assumed to be equal in the existing literature. It is reasonable to develop a robust test capable of performing relatively well without the assumption of dense or sparse mean differences and the assumption of equal magnitude for each component. For this purpose, this thesis proposes a new test consisting of two steps: dynamically allocating weights onto components with varying magnitudes and combining multiple weighted component tests (WCTs) to be adaptive to different sparsity levels of mean differences. The proposed adaptive weighted component test (AWCT) can be viewed as a generalization of the generalized component test (GCT) that puts equal weight on each component. Also, the AWCT shares the idea similar to the adaptive sum of powered score (ASPU) test by optimizing the power among a class of tests. The asymptotic properties of the proposed test are studied, and both the simulation studies and real examples demonstrate that the proposed test can achieve an overall good performance for a variety of signal sparsity. As a comparison, existing approaches often cater for a particular situation where signals are either sparse or dense.

For the K-sample tests of mean equality especially for the time-ordered data, most existing methods are designed for low-dimensional data and not applicable to the high dimension, low sample size regime. Therefore, this thesis proposes a new K-sample test under the high-dimensional setting. The proposed test relies on neither the normality assumption nor the computationally expensive permutation method. Furthermore, with the utilization of the forward searching algorithm complemented by the adaptive lasso method, the proposed method is also capable of detecting the position and duration of signal(s) robust to the noise accumulated from the high dimensional covariates. The superior performance of the proposed method is illustrated by comparison with other existing methods both in numerical simulation and a real dataset collected from a semi-conductor manufacturing process.