MACHINE LEARNING TECHNIQUES - LASA

24
2.1.5 Probabilistic PCA
Until now, all variants of PCA we have seen had a deterministic nature. However, implicitly
through the computation of the mean and covariance matrix of the data, one assumed that the
data followed a distribution that could be parameterized through these two parameters. A
distribution parameterized with this is the Gaussian distribution.
Here we will see how the standard PCA procedure can be extended with a probabilistic model.
This rewriting will provide a first step toward introducing a series of methods based on so-called
latent variables, which will see later on.
Latent variables correspond to unobserved variables. They offer a lower dimensional
representation of the data and their dependencies. Fewer dimensions result in more
parsimonious models. Probabilistic PCA (PPCA) is then PCA through projection on a latent
space.
Formalism:
N
Asssume a N-dimenionsal data set X ∈° . X corresponds to the observations. Probabilistic
PCA starts with the assumption that the data X were generated by a Gaussian latent variable
model of the form:
x= Wz + µ + ε
(2.11)
where z ∈ °
W is a N×
q matrix
µ ∈ °
N
q
are the q-dimensional Latent Variable,
is a vector of parameters
( ε )
N
ε ∈ ° is the noise and follows a zero mean Gaussian distribution ε =Ν 0, ∑
Probabilistic PCA hence differs from PCA by assuming that a) the linear transformation through
the matrix W goes from the latent variables to the observables; b) the transformation is no longer
deterministic and is affected by random noise.
Note that the noise is a random variable with zero mean and fixed covariance. If one further
assumes that the covariance matrix of the noise ∑ is diagonal, i.e. that the noise along each
dimension is uncorrelated to the noise along the other dimensions, this leads to a conditional
independence on the observables given the latent variables. In other words, the latent variables z
encapsulate the correlations across the variables. Such conditional independence on the
observables is advantageous for further processing, e.g. to proceed to an estimation of the
likelihood of the model given the data. In this case, one can then simply take the product of the
likelihood of the data for each dimension separately.
Probabilistic PCA consists then in estimating the density of the latent variable z. PPCA does so
through maximum likelihood.
ε
© A.G.Billard 2004 – Last Update March 2011