MACHINE LEARNING TECHNIQUES - LASA

25
Algorithm:
If one further assume that the noise follows an isotropic Gaussian distribution of the
2
form Ν (0, I) , i.e. that its variance σ is constant along all dimensions. The conditional
σ ε
probability of the observables X given the latent variables px ( | z)
is given by:
2
ε
2
( µσ ε )
px ( | z) =Ν Wz+ , I
(2.12)
The marginal distribution can then be computed by integrating out the latent variable and one
obtains:
If we set B=WW
T
z
T 2
( ) ( µ , σ ε )
p x =Ν WW + I
(2.13)
2
+ σ ε
I, one can then compute the log-likelihood:
M
−1
L ( B, σ
ε
, µ ) =− { N ln( 2π) + ln B + tr ( B C)
}
(2.14)
2
1
where C = x − x −
M
M
i
i
∑( µ )( µ )
i=
1
1 M
{ x x }
M datapoints X= ,..., .
T
is the covariance matrix of the complete set of
The parameters B, µ and σ can then be computed through maximum likelihood, i.e. by
maximizing the quantity L ( B, ε
, )
ε
maximum estimate of µ turns out to be the mean of the dataset.
The maximum-likelihood estimates of B and σ are then:
1
* 2
=
2
q
Λ −
*
( σ∈)
2
( q
σ
ε )
B W I R
1
=
N − q
N
j= q+
1
j
σ µ using expectation-maximization. Unsurprisingly, the
2
∈
(this is also called the residual)
where W is the matrix of eigenvectors of C and the λ are the associated eigenvalues.
q
∑
λ
As in PCA, the dimension N of the original dataset, i.e. the observable X, is reduced by fixing the
dimension q< N of the latent variable. The conditional distribution of the latent variable given
the observable is:
T 2
( )
where B= W W+ I.
σ ε
1 −1 2
( µ σ )
−
( ) ( )
j
p z| x =Ν B W x− , B ε
(2.15)
Finally note that, in the absence of noise, one recovers standard PCA. Simply observe that:
T
−1
(( W)
W) W( x µ )
sets A = ( )
T
( ) − 1
W W W
− is an orthonormal projection of the zero mean dataset, and hence if one
, one recovers the standard PCA transformation.
© A.G.Billard 2004 – Last Update March 2011