Linear Transformation Examples Matrix Eigenvalue problems

Md53
Linear Transformation Examples
● Linear transformations from R2 into R2 :
i.e. linear transformations in the Cartesian plane
⎡0
1⎤
⎡1
0 ⎤ ⎡−1
0 ⎤ ⎡a
0⎤
() 1 ⎢ , ( 2)
, ( 3)
, ( 4)
.
⎣1
0⎥
⎢
⎦ ⎣0
−1⎥
⎢
⎦ ⎣ 0 −1⎥
⎢
⎦ ⎣0
1⎥
⎦
Md54
where (1) represents a reflection in the line x 2 = x 1, (2) represents
a reflection in the x 1-axis, (3) represents a reflection in the origin,
and (4) reflects a stretch (a > 0) or contraction (a < 0) in x 1-direction.
Similarly,
⎡cosθ
−sinθ⎤
⎢
,
⎣sinθ
cosθ
⎥ represents an anticlockwise rotation in the plane.
⎦
−1
If a matrix Ais nonsingular, then A exists and gives the inverse
−1
transformation, thus : x = A y, where y = Ax.
Matrix Eigenvalue problems
● Matrix eigenvalue problems: non-trivial solutions to: Ax = λx,
given square (n ×× n) matrix A, unknown scalar λ, unknown vector x
clearly the trivial x = 0 is a solution, but of no interest;
the solutions x ≠ 0 are called the eigenvectors of A;
these solutions only exist for certain values of λ, called eigenvalues,
or characteristic values.
● Eigenvalue problems are of the greatest importance in engineering,
although this is not immediately obvious from the equation
Ax = λλx
but the range of important applications is incredibly large!
● For example:
principal components analysis in pattern recognition and clustering,
markov processes in probabilistic decision probems,
models of population growth,
mechanical vibration problems,
stability analysis in control problems, ...

Md55
Eigenvalues and Eigenvectors
● Terminology:
Ax = λx matrix eigenvalue problem
A value of λλ for which x (≠ 0) is a solution eigenvalue,
(also known as characteristic value)
Solutions x (≠ 0) corresponding to a λ called eigenvectors
The set of eigenvectors is called the spectrum of A
Largest of absolute values of eigenvalues is spectral radius of A
● Determination of eigenvalues and eigenvectors:
Ax = λx = λIx, where I is the identity matrix
(A - λI)x = 0, homogeneous linear system with non-trivial solution
(x ≠ 0) if and only if D(λλ) = det(A - λI) = 0
Md56
Example
Ax = x A =
x
x
x
x x x
x x x
− ⎡
⎢
⎣
⎤
− ⎥ =
⎦
⎡
● Determination of eigenvalues :
5
λ ,
2
2
1 ⎤
, ⎢ ⎥,
so that
2 ⎣ 2 ⎦
− 5 1 + 2 2 = λ 1
2 1 − 2 2 = λ 2
− − x + x =
A− I x =
x + − − x =
D = A− I = −
( 5 λ)
1 2 2 0
( λ ) 0,
2 1 ( 2 λ)
2 0
and
5−λ ( λ) det( λ )
2
2
= 0
−2−λ Characteristic polynomial
2
D(
λ) = ( −5−λ)( −2−λ) − 4 = λ + 7λ + 6= 0
Whence, ( λ + 1)( λ + 6) = 0, so that λ = −1, − 6; i.e. λ =− 1, λ =−6.
● Determination of an eigenvector:
1 2
For
− + =
λ = λ = − :
− =
and these equations are linearly dependent
with a solution : = . This determines an eigenvector corresponding to λ = −
up to a scalar multiple. If we choose x 1 = , we obtain eigenvector x = e1
= .
⎡ ⎤ ⎡
⎢ ⎥ = ⎢
⎣ ⎦ ⎣
⎤
4x1 2x2 0
1 1
2x1 x2
0
x2 2x1 1 1
x1
1
1
x ⎥
2 2⎦

Md57
General Case
● Determination of other eigenvector:
+ =
For λ = λ = − :
and these equations are linearly dependent
+ =
with a solution : =− / with arbitrary . This determines an eigenvector
corresponding to λ =− up to a scalar multiple. If we choose x 1 = , we obtain
eigenvector x = e2 = . So that : λ , e1
⎡ ⎤
⎢ ⎥
⎣ ⎦
= ⎡ ⎤
⎢
⎣−
⎥ =− =
⎦
⎡
x1 2x2 0
2 6
2x1 4x2 0
x2 x1 2
x1
2 6 2
x1
2
1
1 1
x2
1
⎢
⎣2
⎤
⎡ 2 ⎤
⎥ ; λ2 =− 6,
e2 = ⎢
⎦
⎣−
⎥.
1⎦
● Eigenvalues:
The eigenvalues of a square matrix A are the roots of the characteristic
equation D(λ) = det(A - λI) = 0. Hence an n × n matrix has at least one
eigenvalue and at most n numerically different eigenvalues.
● Eigenvectors:
If x is an eigenvector of a matrix A corresponding to an eigenvalue λλ, so
is kx with any k ≠ 0. [since Ax = λx implies k(Ax) = λ(kx)].
Md58
Characteristic Polynomial
For n× n matrix A, Ax = λx, whence ( A− λI)
x = 0,
This homogeneous linear system
of equations has a nontrivial solution if and only if D( λ) = det( A− λI)
= 0 :
a11 − λ a12 ... a1n
D( λ) = det( A− λI)
=
a21 .
a22 − λ
.
...
...
a2n
.
= 0
a a ... a − λ
n1 n2 nn
which develops a polynomial of nth degree in λ,
called the characteristic polynomial of A :
n
n−1
D(
λ) = λ + bn−1λ + . . . + b1λ + b0
= 0,
which has at least one root
(solution for eigenvalue λ), and at most n numerically different roots, λ.
n
e.g. ( λ − λ0) = 0,
whence λ = λ0
or ( λ −λ1)( λ −λ2) . . . ( λ − λn) = 0, whence λ = λi,
i = 1,
. . . , n
Note that the roots can be real or complex, but if matrix A has real coefficients,
then any complex roots occur in complex conjugate pairs, e.g. λ = c+ jd λ = c− jd
k , k+
1

Md59
Multiple eigenvalues
⎡−2
2 −3⎤
−2−λ2 −3
Matrix, A= ⎢
2 1 −6
⎥
has characteristic polyomial D(
λ)
= 2 1−λ−6 = 0
⎢
⎥
⎣⎢
−1 −2
0 ⎦⎥
−1 −2 −λ
3 2
D(
λ) =− ( 2+ λ){( 1−λ)( −λ) −12} −2{ −2λ −6} −3{ − 4+ ( 1− λ)} =−λ − λ + 21λ + 45 = 0
2
which factorises to D(
λλ) = ( λ − 5)( λ + 3) = 0, whence roots λ1 = 5, λ2 = λ3
= −3
To find eigenvectors, we apply Gauss elimination to the system ( A− λI)
x = 0,
first with λ = 5 and then with λ = −3.
T
For λ = 5, we obtain an eigenvector, e1
= [ 1 2 −1]
For λ =−3
the characteristic matrix
⎡ 1
A − λI
= A+ 3I
=
⎢
2
⎢
⎣⎢
−1 2
4
−2
−3⎤
−6
⎥
⎥
3 ⎥⎦
⎡1
which row reduces to
⎢
0
⎢
⎣⎢
0
2
0
0
−3⎤
0
⎥
⎥
0 ⎦⎥
Hence it has rank 1. From x + 2x − 3x = 0 we have x = − 2x + 3x
.
Md60
1 2 3 1 2 3
Complex Eigenvalues
For the case λ =− 3, we have x1
=− 2x2 + 3x3
:
Choosing x2 = 1, x3 = 0 and then x2 = 0, x3
= 1 obtains two linearly
independent eigenvectors of A corresponding to λ =−3,
thus :
T T
e = −2
1 0 , and e 3 0 1;
so that µ e νe
is an eigenvector solution.
[ ] = [ ] +
2 3 2 3
● Complex eigenvalues example:
⎡
A= ⎢
⎣−
⎤
⎥
⎦
A− I =
j ie j j jx x
jx x x
T
e
T
j e j
−
0
1
1
0
λ
has det( λ )
−1 1 2
= λ + 1= 0
− λ
Eigenvalues ± , . . λ1 = , λ2
= − ; Eigenvectors from − 1 + 2 = 0
and 1 + 2 = 0 respectively, so e.g. choose 1 = 1 to obtain :
1 = [ 1 ] , 2 = [ 1 − ] . More generally, these are the eigenvectors of :
⎡ a
A = ⎢
⎣−b
b⎤
for real a b with eigenvalues a jb
a⎥
, , ± .
⎦

Md61
Stretching elastic membrane
2 2
Elastic membrane in xx 1 2 − plane with boundary circle x1 + x2
= 1
is stretched so that point P x1 x2 goes over into point Q y1 y2
by :
y1
5 3 x1
y = Ax
y2
3 5 x2
⎡ ⎤ ⎡ ⎤
⎢ ⎥ = = ⎢ ⎥
⎣ ⎦ ⎣ ⎦
⎡
:( , ) :( , )
⎤
x2 ⎢ ⎥
⎣ ⎦
● The problem is to find the principal
directions of position vector x of P for
which the direction of position vector
y of Q is the same or exactly opposite.
We are looking for vectors x such that y = λ x,
and
since y = Ax we have Ax = λ x eigenvalue problem.
Principal
directions
D = with solutions
T
Eigenvectors e corresponding to
T
e corresponding to
These vectors make 45 and angles with the positive x direction.
− 5 λ
( λ)
3
3
2
= ( 5−λ) − 9= 0, λ = 2, 8
5−λ
1 = [ 1 1] λ1= 2, 2 = [ 1 −1]
λ2=
8.
o o
135
1 − The eigenvalues
show that the membranes are stretched by factors 8 and 2 in the principal directions.
Md62
Vibrating masses on springs
Differential equations :
y1′′
=− 5y1 + 2y2
y2′′ = 2y1 −2y2
where y1, y2
are displacements
of the masses from rest. y′′ = Ay
Trial vector solution :
ωt
y = xe
2 ωt ωt
Whence, ω xe = Axe
Divide by
ωt
e
2
and set ω = λ,
Whence Ax = λ x,
with eigenvalues
λ =−1, − 6 so that ω =± j, ± j 6
Eigenvectors,
T
x1 = [ 1
T
2] , x2
= [ 2 −1]
.
General vector solution,
y = x ( a cost + b sin t) + x ( a cos 6t + b sin 6t).
1 1 1 2 2 2
y 1 = 0
y 2 = 0
y 1
y 2
K 1 = 3
m 1 = 1
K 2 = 2
m 2 = 1
System in
static
equilibrium
y 1
y 2
System in
motion
x 1
Net
change in
spring
length
= y 2 -y 1

Md63
Props of Eigenvalues & Eigenvectors
● (a) Real and complex eigenvalues.
If A is real, its eigenvalues are real or complex conjugates in pairs.
● (b) Inverse.
A-1 exists iff 0 is not an eigenvalue of A. It has the eigenvalues 1/λ1 , . . . , 1/λn .
● (c) Trace.
The sum of the main diagonal entries is called the trace of A. It equals the
sum of the eigenvalues.
● (d) Spectral Shift.
A - kI has the eigenvalues λ1- k, . . . , λn- k, and the same eigenvectors as A.
● (e) Scalar multiples, powers.
kA has the eigenvalues kλ 1 , . . . , kλ n . A m (m = 1, 2, . . . ) has the eigenvalues
λ 1 m , . . . , λn m . The eigenvectors are those of A.
● (f) Spectral Mapping Theorem.
Md64
The polynomial matrix p(A) = k m A m + k m-1 A m-1 + . . . +k 1 A + k 0 I has the
eigenvalues p(λλ j ) = k m λ j m + km-1 λ j m-1 + . . . +k1 λ j + k 0 , where j = 1, . . . , n.
Special Real Square Matrices
● Symmetric
AT = A, so that akj = ajk. ● Skew-symmetric
AT = -A, so that akj = -ajk, and aii = 0 for all i.
● Orthogonal (e.g. rotation matrix)
AT = A-1 , so that transposition gives the inverse.
symmetric skew - symmetric orthogonal
2 1 2
⎡−3
1 5 ⎤ ⎡ 0 9 −12⎤
⎡ 3 3 3 ⎤
⎢
−
⎥ ⎢
−
⎥ ⎢ 2 2 1
1 0 2
9 0 20
−
⎥
⎢
⎥ ⎢
⎥ ⎢ 3 3 3 ⎥
1 2 2
⎣⎢
5 −2
4 ⎦⎥
⎣⎢
12 −20
0 ⎦⎥
⎣⎢
3 3 − 3⎦⎥
● Any real square matrix A may be written as the sum of a symmetric
matrix R and a skew-symmetric matrix S, where:
1
T 1
T
R = ( A+ A ) and S = ( A − A ) so that A = R + S
2
2

Md65
Orthogonal Transformations
Orthogonal transformations are transformations y = Ax with an orthogonal matrix A.
e.g. y = Ax with
⎡cosθ
A=
⎢
⎣sinθ
−sinθ⎤
⎥,
rotation matrix.
cosθ
⎦
2
In fact any orthogonal transformation in spaces R or
3
R is a rotation
possibly combined with a reflection in a straight line or plane.
● Invariance of inner product:
Md66
An orthogonal transformation preserves the value of the inner product of vectors,
T
a• b = a b,
where a and b are column vectors.
i.e. if u = Aa and v = Ab, where A is orthogonal, then u• v = a•b. Hence, an orthogonal transformation also preserves the length or norm of a vector :
a = a• a =
T
a a,
since a is given as an inner product.
T T T T T T
Proof : u• v = u v = ( Aa) Ab = a A Ab = a Ib = a b = a•b, since A .
T −1
A= A A= I
Props of Orthogonal Matrices
● Orthonormality of column and row vectors:
A real square matrix is orthogonal if and only if its column vectors
a 1, a 2, ... , a n (and also its row vectors) form an orthonormal system :
if j k
T ⎧0
≠
that is aj • ak = aj ak=
⎨
for all j, k
⎩1
if j = k
● The determinant of an orthogonal matrix is +1 or -1
● The eigenvalues of: a symmetric matrix are real; and of an orthogonal
matrix are real or complex conjugate in pairs with absolute value 1.
2 1 2
⎡ 3 3 3 ⎤
The orthogonal matrix
⎢ 2 2 1 −
⎥
3 , has characteristic polynomial
⎢ 3 3 ⎥
:
1 2 2
⎣⎢
3 3 − 3⎦⎥
3 2 2 2
− λ + 3 λ + 3 λ−
1= 0.
Now at least one of the eigenvalues must be real,
and hence + 1 or −1. We find that −1is
an eigenvalue, so that dividing
2 5
1
by ( λ+1) obtains λ − λ+ 1= 0, from which λ = ( 5± j 11), −1.
3
6

Md67
Md68
Similarity of Matrices
● Eigenvectors and their properties:
The eigenvectors of an (n × n) matrix A may or may not form a
basis for Rn . If they do (e.g. case of n distinct eigenvalues), then
they can be used for “diagonalising” A - i.e. transforming A into
diagonal form with the eigenvalues on the main diagonal.
● Similarity transformation:
n× n A√ n× n A A√ −1
matrix is similar to matrix if = P AP
for some nonsingular n× n matrix P.
Then A√ has the same eigenvalues as A, and if x is an eigenvector of A,
−1
then P x is an eigenvector of A√
corresponding to the same eigenvalue.
● Basis of eigenvectors:
If λ1, λ2, ..., λk,
are distinct eigenvalues of a matrix, then the corresponding
eigenvectors x1, x2, ..., xkform a linearly independent set. If n× n matrix
A has n distinct eigenvalues, then A
n
has a basis of eigenvectors for R .
● Basis of eigenvectors
Diagonalization
A symmetric matrix always has an orthonormal basis of eigenvectors for
e.g. has orthonormal basis of eigenvectors 1
n
R .
A = ,
2
1
;
2
corresponding to eigenvalues , and respectively.
⎡
⎢
⎣
⎤
⎥
⎦
⎡
⎢
⎣
⎤
5
3
3
5
1
1⎥
⎦
⎡ 1 ⎤
⎢
⎣−1⎥
⎦
λ = 8 λ = 2
● Diagonalization of a matrix
1
If n× n matrix A has a basis of eigenvectors, then D= X AX is diagonal,
with the eigenvalues of A as the entries on the main diagonal,
and where X is the matrix with these eigenvectors as column vectors.
m m
Also D = X A X m =
e.g. A= has X and X giving X AX
⎡
−1
−1
, 2, 3,
...
5
⎢
⎣1
4⎤
⎡4
⎥ , =
2 ⎢
⎦ ⎣1
1 ⎤
⎡−
−1 1 1
− ⎥ , =
1⎦
−5
⎢
⎣−1
−1⎤
−1
⎡6
⎥ ,
=
4
⎢
⎦
⎣0
0⎤
1⎥
⎦
2

Md69
Diagonalization of Quadratic forms
● Quadratic forms
e.g. Let
● Diagonalization
Md70
T
x Bx = [ x
⎡
x ] ⎢
⎣−
− ⎤ x
⎥ x x x x
⎦ x
x x
x T T
x Ax where A B B symmetric).
x
⎡ ⎤
⎢ ⎥ = − +
⎣ ⎦
= [
⎡
] ⎢
⎣−
− ⎤
⎥
⎦
⎡
1
17
2
20
10 1
2
2
17 1 30 1 2 17 2
17 2
1
17
2
15
15 1 ⎤
1
⎢ ⎥ = , = 2 [ + ] (
17 ⎣ 2 ⎦
T
Let Q= x Ax, where we can assume n× n A is symmetric.
Then A has an orthonormal basis of n eigenvectors,
−1
T
and matrix X with these column vectors is orthogonal, so that X = X .
−1
Now D= X AX
T T T
so that A= XDX giving Q = x XDX x.
T
[since XDX
−1
T
= XX AXX = IAI = A].
T T T T
2
Set X x = y, so that x = Xy; also x X = y . Then Q= y Dy = λ y + ... + y .
2
Transformation to Principal Axes
● Principal components example
1 1
λ n n
T −1 −1
T T T T
Q= x Ax, D= X AX with X = X so that A= XDX giving Q = x XDX x.
T T T T
2 2
Set X x = y, so that x = Xy; also x X = y . Then Q= y Dy = λ1y1+ ... + λnyn.
⎡ 17 −15⎤
e.g. A = ⎢
⎣−
⎥ , with eigenvalues λ1 = 2, λ2
= 32.
15 17 ⎦
Then Q = − + = = + =
+ = −
⎡ − ⎤
= ⎢ ⎥
⎣ ⎦
=
2
2
2 2
17x130x1x2 17x2 128 becomes Q 2y132y2128 (an ellipse),
2 2
y1y2 i.e. 1.
Direction of principal axes in xx
2 2 1 2 coordinates from normalised
8 2
1 1 1 ⎡cos
π
4 −sin
π
4⎤
eigenvectors, which are columns of : X
2 1 1 ⎢
⎥
⎣sin
π
4 cos π
4 ⎦
so that principal axes transformation x = Xy represents a 45 degree rotation.
[see " Stretching elastic membrane" example on slide Md65].