The linear model with sparsity-favouring prior on the coefficients has been the focus of much work in Statistics and Machine Learning, and has seen successful applications in many different domains. The importance of the linear model is reflected in the myriad of names attached to it in neighboring fields, for example multiple linear regression, least-squares SVM. It is an essential ingredient in Kalman smoothing and linear dynamical systems, and different coefficient priors lead to unsupervised techniques such as PCA and ICA. Direct extensions lead to (multiple) logistic regression, ordinal regression, point process models, and others.
Full Bayesian approaches employ expensive and hard-to-use MCMC techniques. MAP point estimation is frequently used (Lasso, L_1 regularization), but does not come with essential Bayesian features such as posterior variances and correlations, Bayesian optimal design, or Bayes factors for model comparison or hyperparameter fitting. Tipping’s SBL is maybe the most widely used approximate Bayesian technique available.

We propose a novel approximate framework, based on the expectation propagation method, which is efficient and accurate even in strongly underdetermined situations. We show how to implement Bayesian optimal design in this framework, addressing gene regulatory network identification from micro-array expression data. We also derive an approximation to the marginal likelihood, which is used in order to learn sparsity codes for natural images, in what amounts to a Bayesian variant of ICA. We compare our framework directly to Tipping’s SBL. Further extensions to non-negative linear models, sparse generalized linear models, or large-scale applications are also motivated.

Registration required:

No

Free:

Yes