On the Strange a Posteriori degeneracy of Normal Mixtures, and Related Reparameterization Theorems
Report ID: TR-541-96Author: Yianilos, Peter N. / Ristad, Eric Sven
Date: 1996-12-00
Pages: 16
Download Formats: |Postscript|
Abstract:
This short paper illuminates certain fundamental aspects of the nature of normal (Gaussian) mixtures. Thinking of each mixture component as a class, we focus on the corresponding a posteriori class probability functions. It is shown that the relationship between these functions and the mixture's parameters, is highly degenerate -- and that the precise nature of this degeneracy leads to somewhat unusual and counter-intuitive behavior. Even complete knowledge of a mixture's a posteriori class behavior, reveals essentially nothing of its absolute nature, i.e. mean locations and covariance norms. Consequently a mixture whose means are located in a small ball anywhere in space, can project arbitrary class structure everywhere in space. The well-known expectation maximization (EM) algorithm for Maximum Likelihood (ML) optimization may be thought of as a reparameterization of the problem in which the search takes place over the space of sample point weights. Motivated by EM we characterize the expressive power of similar reparameterizations, where the objective is instead to maximize the a posteriori likelihood of a labeled training set. This is relevant to, and a generalization of a common heuristic in machine learning in which one increases the weight of a mistake in order to improve classification accuracy. We prove that EM-style reparameterization is not capable of expressing arbitrary a posteriori behavior, and is therefore incapable of expressing some solutions. However a slightly different reparameterization is presented which is almost always fully expressive -- a fact proven by exploiting the degeneracy described above.