Learning sparse structural associations from data

Abstract

Building statistical models to explain the structural association between/among responses (outputs) and predictors (inputs) is important in many real applications, especially for various settings such as bioinformatics, financial prediction and computer vision. A proper choice of the structure which reveals the underline mechanics may be crucial for boosting the performance of prediction models. In this thesis, I propose several methods for automatically learning the latent structural associations for different scenarios. In this thesis I first revisit the problem of multi-response regression and propose an efficient method for feature selection which defined on which we termed the intra- and inter- group sparsities, i.e. for inter-group sparsity I assume only small number of predictors are related to the responses, and for intra-group sparsity each predictor may relate to multiple responses with different sparsity levels. Existing methods fail to model the intra-group sparsity well by either assuming uniform regularization on each group, i.e. each input feature relates to similar number of response features, or requiring prior knowledge of the relationship of predictor and response features. With the new sparsity definition, my method is capable of learning the intra-group sparsity automatically. Second, as responses may not be independent in reality for multi-response prediction, a promising direction is to predict related responses together. However, not all responses have the same degree of relatedness. Sparse Gaussian conditional random field (SGCRF) was developed to learn the degree of relatedness from data without any prior knowledge. However, in real cases, features are not arbitrary, but are dominated by a (smaller) set of related latent factors, e.g. clusters. SGCRF does not capture these latent relations, thus more accurate associations can be explored if we model these latent factors. I propose a novel (mixed membership) hierarchical Bayesian model, namely M2GCRF, to capture this phenomenon in terms of clusters. Third, I explore Bayesian networks (BNs) for modeling conditional distributions of variables and causal relationships. Existing BN structure learning algorithms separately treat those variables with similar tendency. I propose a grouped sparse Gaussian BN (GSGBN) structure learning algorithm which generates the optimal BN with three assumptions: (i) variables follow a multivariate Gaussian distribution, (ii) the network only contains a few edges (sparse), (iii) similar variables have less-divergent sets of parents, while not-so-similar ones should have divergent sets of parents (variable grouping). I make the learned network sparse by L1 regularization, and apply another term to incorporate shared information among variables (Laplacian distributions on the differences of variables’ parents). For similar variables, GSGBN tends to penalize the differences of similar variables’ parent sets more, compared to those not-so-similar variables’ parent sets. With the new definition of the optimal Bayesian network, I obtain the edges of the network and also the similarity of variables from the data, without prior knowledge. Plenty of experiments on both simulated and real data show the above methods have substantially superior prediction performance compared to its competitors in many aspects. (484 words)