On the Strange a Posteriori degeneracy of Normal Mixtures, and Related Reparameterization Theorems

Abstract:

This short paper illuminates certain fundamental aspects of the nature
of normal (Gaussian) mixtures. Thinking of each mixture component as
a class, we focus on the corresponding a posteriori class
probability functions. It is shown that the relationship between
these functions and the mixture's parameters, is highly degenerate --
and that the precise nature of this degeneracy leads to somewhat
unusual and counter-intuitive behavior. Even complete knowledge of a
mixture's a posteriori class behavior, reveals essentially
nothing of its absolute nature, i.e. mean locations and covariance
norms. Consequently a mixture whose means are located in a small ball
anywhere in space, can project arbitrary class structure
everywhere in space.
The well-known expectation maximization (EM) algorithm for Maximum
Likelihood (ML) optimization may be thought of as a reparameterization
of the problem in which the search takes place over the space of
sample point weights. Motivated by EM we characterize the expressive
power of similar reparameterizations, where the objective is instead
to maximize the a posteriori likelihood of a labeled training
set. This is relevant to, and a generalization of a common heuristic
in machine learning in which one increases the weight of a mistake in
order to improve classification accuracy. We prove that EM-style
reparameterization is not capable of expressing arbitrary a
posteriori behavior, and is therefore incapable of expressing some
solutions. However a slightly different reparameterization is
presented which is almost always fully expressive -- a fact proven by
exploiting the degeneracy described above.