Variable selection and prediction for local polynomial regression

Abstract

Local polynomial has been used for nonparametric regression with a long history and is generalised to many other regression models. However, it inescapably suffers from the curse of dimensionality since the size of local neighborhood, which is controlled by bandwidths, decreases exponentially as covariate dimension increases. To solve the problem, we propose a bandwidth regularization scheme to remove irrelevant variables.

We first discuss the proposed procedure in the general nonparametric regression setting with convex loss for high dimensional data. Selection consistency as well as the asymptotic property of the local linear estimator based on the optimised bandwidths are established.

Then, a modified Nadaraya--Watson estimator is proposed for variable selection under a nonparametric setting, in the presence of missing data, where a covariate may be missing either because its value is hidden from the observer or because it is inapplicable to the particular subject being observed. The method allows for information sharing across different missing patterns without affecting consistency of the estimator. Unlike other conventional methods such as those based on imputations or likelihoods, our method requires only mild assumptions on the model and the missing mechanism. For prediction we focus on finding relevant variables for predicting mean responses, conditional on covariate vectors subject to a given type of missingness.

The final problem is dimension reduction. The above procedure is extended to bandwidth matrix optimization to perform variable selection, dimension reduction and optimal estimation at the oracle convergence rate, all in one go. Compared to most existing methods, the new procedure does not require explicit bandwidth selection or an additional step of dimension determination using techniques like cross validation or principal components. The selected model is guaranteed to have non-inferior convergence rate compared to that of the oracle model.