MACHINE LEARNING TECHNIQUES - LASA

64
4 Regression Techniques
There is a growing interest in machine learning to design powerful algorithms for performing nonlinear
regression. We will here consider a few of these. The principle behind the technique we will
present here is the same as the one used in most other variants we find in the literature and
hence this offers a good background for an interested reader.
Consider a multi-dimensional (zero-mean) variable
N
x∈° and a one-dimensional
variable y ∈ ° , regression techniques aim at approximating a relationship f between y and x by
building a model of the form:
y= f ( x)
(4.1)
4.1 Linear Regression
The most classical technique that the reader will probably be familiar with is the linear regressive
model, whereby one assumes that f is a linear function parametrized by
( , )
T
y f x w x w
w∈ °
N
, that is:
= = (4.2)
For a given instance of the pair x and y one can solve explicitly for w . Consider now the case
i
where we are provided with a set of M observed instances X = { x } and { } i
Y y
i=
1
i=
1
M
M
= of the
variables x and y such that the observation of y has been corrupted by some noise which may
or not be a function of x, i.e. ε ( x)
:
( )
T
y x w ε x
= + (4.3)
Classical means to estimate the parameters w is through mean-square, which we review next.
4.2 Partial Least Square Methods
Adapted from R. Rosipal and N. Kramer, Overview and Recent Advances in Partial Least Squares, C. Saunders et
al. (Eds.): SLSFS 2005, LNCS 3940, pp. 34–51, 2006.
Partial Least Squares (PLS) refer to a wide class of methods for modeling relations between sets
of observed variables by means of latent variables. It comprises regression and classification
tasks as well as dimension reduction techniques and modeling tools. As the name implies, PLS is
i
based on least-square regression. Consider a set of M pairs of variables { }
i=
1
M
i
Y = { y } , least-square regression looks for a mapping w that sends X onto Y, such that:
i=
1
.
X
M
= x and
© A.G.Billard 2004 – Last Update March 2011