Spectral Analysis and Time Series - max planck research school ...

A. Lagg – Spectral Analysis
Spectral Analysis and Time Series
Andreas Lagg
Part I: fundamentals
on time series
classification
prob. density func.
auto­correlation
power spectral density
cross­correlation
applications
pre­processing
sampling
trend removal
Part II: Fourier series
definition
method
properties
convolution
correlations
leakage / windowing
irregular grid
noise removal
Part III: Wavelets
why wavelet
transforms?
fundamentals:
FT, STFT and
resolution problems
multiresolution
analysis: CWT
DWT
Exercises

d
e
t
e
r
m
A. Lagg – Spectral Analysis
Basic description of physical data
deterministic: described by explicit mathematical relation
xt=X cos
k t t
n
non o deterministic: no way to predict an exact value at a future instant of time
n

A. Lagg – Spectral Analysis
Classifications of deterministic data
Deterministic
Periodic
Nonperiodic
Sinusoidal
Complex Periodic
Almost periodic
Transient

A. Lagg – Spectral Analysis
Sinusoidal data
xt=X sin2 f 0
t
T =1/ f 0
time history
frequency spectrogram

A. Lagg – Spectral Analysis
Complex periodic data
xt=xt±nT
n=1,2,3,...
xt = a 0
2 ∑ a n
cos 2n f 1
t b n
sin 2 n f 1
t
(T = fundamental period)
time history
frequency spectrogram

A. Lagg – Spectral Analysis
Almost periodic data
xt=X 1
sin2 t 1
X 2
sin3t 2
X 3
sin50 t 3
no highest common divisor ­> infinitely long period T
time history
frequency spectrogram

A. Lagg – Spectral Analysis
Transient non­periodic data
all non­periodic data other than almost periodic data
xt={ Ae−at t≥0
0 t0
xt={ Ae−at cos bt t≥0
0 t0
xt={ A c≥t≥0
0 ct0

A. Lagg – Spectral Analysis
Classification of random data
Random Data
Stationary
Nonstationary
Ergodic
Nonergodic
Special classifications
of nonstationarity

A. Lagg – Spectral Analysis
stationary / non stationary
collection of sample functions = ensemble
data can be (hypothetically) described by
computing ensemble averages (averaging over
multiple measurements / sample functions)
mean value (first moment):
x
t 1
= lim
N ∞
1
∑ N
N k=1
autocorrelation function (joint moment):
R x
t 1,
t 1
= lim
N ∞
x k
t 1
1
∑ N
N k=1
x k
t 1
x k
t 1
stationary: x
t 1
= x
, R x
t 1,
t 1
=R x
weakly stationary: x
t 1
= x
, R x
t 1,
t 1
=R x

A. Lagg – Spectral Analysis
ergodic / non ergodic
Ergodic random process:
properties of a stationary random process
described by computing averages over only one
single sample function in the ensemble
mean value of k­th sample function:
x
k=lim
T ∞
autocorrelation function (joint moment):
T
1
∫ T
0
x k
tdt
R x
, k=lim
T ∞
T
1
∫ T
0
x k
t x k
tdt
ergodic:
x
k= x
, R x
, k=R x

Basic descriptive properties of random data
mean square values
propability density function
autocorrelation functions
power spectral density functions
(from now on: assume random data to be stationary and ergodic)
A. Lagg – Spectral Analysis

A. Lagg – Spectral Analysis
Mean square values
(mean values and variances)
describes general intensity of random data: 2 x
= lim
T ∞
rout mean square value:
x
rms
= x
2
T
1
∫ T
0
x 2 tdt
often convenient:
static component described by mean value:
x
= lim
T ∞
T
1
∫ T
0
xtdt
dynamic component described by variance:
x
2
= lim
T ∞
1
∫ T
T
0
[ xt− x
] 2 dt
2
standard deviation: x
= x
= x 2 − x
2

A. Lagg – Spectral Analysis
Probability density functions
describes the probability that the data will assume a value within some defined
range at any instant of time
for small x :
Prob[ xxt≤x x] = lim
T ∞
Prob[ xxt≤x x] ≈ p x x
probability density function
px =
lim
x 0
probability distribution function
P x
= Prob[xt≤x]
x
= ∫ pd
−∞
Prob[ xxt≤x x]
x
T x
= lim
x 0
k
T , T x = ∑
i=1
1
x [ lim
T ∞
t i
T x
T ]

Illustration: probability density function
sample time histories:
sine wave (a)
sine wave + random noise
narrow­band random noise
wide­band random noise
all 4 cases: mean value x
=0
A. Lagg – Spectral Analysis

A. Lagg – Spectral Analysis
Illustration: probability density function
probability density function

A. Lagg – Spectral Analysis
Autocorrelation functions
describes the general dependence of the data values at one time on the
values at another time.
R x
= lim
T ∞
T
1
∫ T
0
xt xtdt
x
= R x
∞ 2 x
= R x
0 (not for special cases like sine waves)

A. Lagg – Spectral Analysis
Illustration: ACF
autocorrelation functions (autocorrelogram)

A. Lagg – Spectral Analysis
Illustrations
autocorrelation function of a rectangular pulse
xt xt−
autocorrelation function

A. Lagg – Spectral Analysis
Power spectral density functions
(also called autospectral density functions)
describe the general frequency composition of the data in terms
of the spectral density of its mean square value
f ,
f f
mean square value in frequency range :
x 2 f , f = lim
T ∞
T
1
∫ T
0
definition of power spectral density function:
G x
f =
lim
f 0
x 2 f , f
f
xt , f , f
2 dt
portion of x(t) in (f,f+∆f)
f)
= lim
f 0
x 2 f , f ≈ G x
f f
1
f [ lim
T ∞
T
1
∫ T
0
important property: spectral density function is related to the
autocorrelation function by a Fourier transform:
G x
f = 2∫
−∞
∞
R x
e −i 2 f d = 4∫
0
∞
xt , f , f 2 dt]
R x
cos 2 f d

A. Lagg – Spectral Analysis
Illustration: PSD
power spectral density functions
Dirac delta function at f=f 0
sine wave
sine wave
+ random noise
narrowband
noise
“white” noise:
spectrum is uniform over
all frequencies
broadband
noise

A. Lagg – Spectral Analysis
Joint properties of random data
until now: described properties of an
individual random process
Joint probability density functions
joint properties in the amplitude domain
Cross­correlation functions
joint properties in the time domain
joint probability measurement
Cross­spectral density functions
joint properties in the frequency domain

A. Lagg – Spectral Analysis
Cross­correlation function
describes the general dependence of
one data set to another
R xy
= lim
T ∞
T
1
∫ T
0
xt ytdt
similar to autocorrelation function
R xy
=0
functions are
uncorrelated
cross­correlation measurement
typical cross­correlation plot (cross­correlogram):
sharp peaks indicate the existence of a correlation
between x(t) and y(t) for specific time displacements

A. Lagg – Spectral Analysis
Applications
Measurement of time delays
2 signals:
different offset
different S/N
time delay 4s
often used:
'discrete' cross correlation coefficient
lag =l, for l >=0:
R xy
l =
N −l
∑
k=1
∑ N
k=1
x k
−xy kl
−y
N
x k
−x 2 ∑ y k
−y 2
k=1

A. Lagg – Spectral Analysis
Applications
Detection and recovery from
signals in noise
3 signals:
noise free replica of the signal
(e.g. model)
2 noisy signals
cross correlation can be used to
determine if theoretical signal is
present in data

A. Lagg – Spectral Analysis
Pre­processing Operations
sampling considerations
trend removal
filtering methods
sampling
cutoff frequency (=Nyquist
frequency or folding frequency)
f c
= 1
2 h

A. Lagg – Spectral Analysis
Trend removal
often desirable before performing a
spectral analysis
Least­square method:
time series: ut
desired fit
(e.g. polynomial):
K
u=∑
k=0
N
Lsq­Fit: minimize Qb=∑
set partial
derivatives to 0:
∂Q
n=1
∂ b l
=∑
n=1
K
K+1 equations: ∑
k=0
N
bk nh k
u n
−u n
2
n=1,2,... , N
2u n
−u n
[−nh l ]
N
b k ∑ nh kl
n=1
N
= ∑
n=1
u n
nh l

Digital filtering
A. Lagg – Spectral Analysis

end of part I ...
A. Lagg – Spectral Analysis

A. Lagg – Spectral Analysis
Spectral Analysis and Time Series
Andreas Lagg
Part I: fundamentals
on time series
classification
prob. density func.
auto­correlation
power spectral density
cross­correlation
applications
pre­processing
sampling
trend removal
Part II: Fourier series
definition
method
properties
convolution
correlations
leakage / windowing
irregular grid
noise removal
Part III: Wavelets
why wavelet
transforms?
fundamentals:
FT, STFT and
resolution problems
multiresolution
analysis: CWT
DWT
Exercises

A. Lagg – Spectral Analysis
Fourier Series and Fast Fourier Transforms
Standard Fourier series procedure:
if a transformed sample record x(t) is periodic with a period T p
(fundamental
frequency f 1
=1/T p
), then x(t) can be represented by the Fourier series:
where
∞
xt = a 0
2 ∑ a q
cos 2q f 1
t b q
sin 2q f 1
t
T
q=1
a q
= 2 ∫ xtcos 2q f
T 1
t dt q=0,1,2,...
0
T
b q
= 2 T ∫ 0
xtsin 2q f 1
t dt
q=1,2,3,...

A. Lagg – Spectral Analysis
Fourier series procedure ­ method
sample record of finite length, equally spaced sampled:
x n
=xnh
n=1,2,... , N
Fourier series passing through these N data values:
xt = A 0
N /2
∑
q=1
A q
cos 2q t
T p
N /2−1
∑
q=1
B q
sin 2qt
T p
Fill in particular points: t=nh , n=1,2,... , N , T p
=Nh , x n
=xnh = ...
coefficients A q
and B q
: A 0
= 1 ∑ N n=1
N
x n
= x A N /2
= 1 N ∑ n=1
N
x n
cos n
N
A q
= 2 N ∑ n=1
N
B q
= 2 N ∑ n=1
x n
cos 2q n
N
x n
sin 2q n
N
inefficient & slow => Fast Fourier Trafos developed
q=1,2,... , N 2 −1
q=1,2,... , N 2 −1

A. Lagg – Spectral Analysis
Fourier Transforms ­ Properties
Linearity
DFT
{x n
} ⇔
DFT
{y n
} ⇔
a{x n
}b{y n
}
DFT
⇔
{X k
}
{Y k
}
a{X k
}b{Y k
}
Symmetry {X k
} = {X −k
R{X k
} is even
∗ }
I{X k
} is odd
Circular time shift
DFT
{x n−n0
} ⇔
{e i k n DFT
0
y n
} ⇔
{e −i k n 0
X k
}
{Y k−k0
}

A. Lagg – Spectral Analysis
Using FFT for Convolution
r * s ≡ ∫
−∞
∞
r st−d
Convolution Theorem:
r * s ⇔ FT R f S f
Fourier transform of the convolution
is product of the individual Fourier
transforms
discrete case:
N /2
r * s j
≡ ∑
k=−N /21
Convolution Theorem:
N /2
∑
k=−N /21
s j−k
r k
s j−k
r k
⇔ FT R n
S n
(note how the response function for negative times is wrapped
around and stored at the extreme right end of the array)
constraints:
duration of r and s are not the same
signal is not periodic
original data
response function
convolved data

A. Lagg – Spectral Analysis
Treatment of end effects by zero padding
constraint 1:
constraint 2:
simply expand response function to length N by padding it with zeros
extend data at one end with a number of zeros equal to the max.
positive / negative duration of r (whichever is larger)

A. Lagg – Spectral Analysis
FFT for Convolution
1. zero­pad data
2. zero­pad response function
(­> data and response function have N elements)
3. calculate FFT of data and response function
4. multiply FFT of data with FFT of response function
5. calculate inverse FFT for this product
Deconvolution
­> undo smearing caused by a response function
use steps (1­3), and then:
4. divide FFT of convolved data with FFT of response
function
5. calculate inverse FFT for this product

A. Lagg – Spectral Analysis
Correlation / Autocorrelation with FFT
definition of correlation / autocorrelation see first lecture
Corr g , h = g* h = ∫
−∞
∞
Correlation Theorem:
Corr g , h ⇔ FT
Auto­Correlation:
G f H * f
Corr g , g ⇔ FT ∣G f ∣ 2
discrete correlation theorem:
Corr g , h j
≡
N −1
∑
k=0
⇔ FT G k
H k
*
g jk
h k
gthd

A. Lagg – Spectral Analysis
Fourier Transform ­ problems
spectral leakage
T =nT p
T ≠nT p

A. Lagg – Spectral Analysis
reducing leakage by windowing (1)
Applying windowing (apodizing) function to data record:
xt = xtwt (original data record x windowing function)
x n
= x n
w n
No Window:
Bartlett Window:
Hamming Window:
wt=1−
∣t−T /2∣
T /2
wt=0.540.46∗cost
Blackman Window:
Hann Window:
Gaussian Window:
wt =
0.420.5∗cost
0.08 cos2 t
wt=cos 2 t
2 wt=exp−0.5a t 2

A. Lagg – Spectral Analysis
reducing leakage by windowing (2)
without windowing
with Gaussian windowing

A. Lagg – Spectral Analysis
No constant sampling frequency
Fourier transformation requires constant sampling (data points at
equal distances)
­> not the case for most physical data
Solution: Interpolation
linear:
inear interpolation between y and y k k+1
IDL> idata=interpol(data,t,t_reg)
quadratic:
quadratic interpolation using y k­1 , y k
and y k+1
IDL> idata=interpol(data,t,t_reg,/quadratic)
least­square quadratic
least­square quadratic fit using y k­1 , y k
, y k+1 and y k+2
IDL> idata=interpol(data,t,t_reg,/lsq)
spline
IDL> idata=interpol(data,t,t_reg,/spline)
IDL> idata=spline(t,data,t_reg[,tension])
important:
interpolation changes
sampling rate!
­> careful choice of
new (regular) time grid
necessary!

Fourier Transorm on irregular gridded data ­ Interpolation
original data: sine wave + noise
FT of original data
irregular sampling of data (measurement)
interpolation: linear, lsq, spline, quadratic
're­sampling'
FT of interpolated data
A. Lagg – Spectral Analysis

A. Lagg – Spectral Analysis
Noise removal
Frequency threshold (lowpass)
make FT of data
set high frequencies to 0
transorm back to time
domain

A. Lagg – Spectral Analysis
Noise removal
signal threshold for
weak frequencies (dB­threshold)
make FT of data
set frequencies with amplitudes
below a given threshold to 0
transorm back to time domain

A. Lagg – Spectral Analysis
Optimal Filtering with FFT
normal situation with measured data:
underlying, uncorrupted signal u(t)
+ response function of measurement r(t)
= smeared signal s(t)
+ noise n(t)
= smeared, noisy signal c(t)
st = ∫
−∞
∞
r t−ud
ct = stnt
estimate true signal u(t) with:
U f = C f f
R f
f ,t = optimal filter
(Wiener filter)

A. Lagg – Spectral Analysis
Calculation of optimal filter
reconstructed signal and uncorrupted signal should be close in least­square sense:
­> minimize
­> minimize
∫−∞
∞
∣ ut−ut∣ 2 dt = ∫
−∞
∞
∣ U f −U f ∣ 2 df
⇒
∂
∣ ∂ f
⇒ f =
[S f N f ] f
R f
∣S f ∣ 2
∣S f ∣ 2 ∣N f ∣ 2
− S f
R f ∣2
= 0
additional information:
power spectral density can often
be used to disentangle noise
function N(f) from smeared
signal S(f)

A. Lagg – Spectral Analysis
Using FFT for Power Spectrum Estimation
discrete Fourier transform of c(t)
­> Fourier coefficients:
N−1
C k
=∑
j=0
c j
e 2 i j k / N
k=0,... , N−1
­> periodogram estimate of
power spectrum:
P0 = P f 0
= 1 N 2 ∣C 0 ∣ 2
Periodogram
P f k
=
1
N 2 [∣C k ∣ 2 ∣C N−k ∣ 2 ]
P f c
= P f N /2
= 1 N 2 ∣C N /2 ∣ 2

end of FT
A. Lagg – Spectral Analysis

A. Lagg – Spectral Analysis
Spectral Analysis and Time Series
Andreas Lagg
Part I: fundamentals
on time series
classification
prob. density func.
auto­correlation
power spectral density
cross­correlation
applications
pre­processing
sampling
trend removal
Part II: Fourier series
definition
method
properties
convolution
correlations
leakage / windowing
irregular grid
noise removal
Part III: Wavelets
why wavelet
transforms?
fundamentals:
FT, STFT and
resolution problems
multiresolution
analysis: CWT
DWT
Exercises

A. Lagg – Spectral Analysis
Introduction to Wavelets
why wavelet transforms?
fundamentals: FT, short term FT and resolution
problems
multiresolution analysis:
continous wavelet transform
multiresolution analysis:
discrete wavelet transform

A. Lagg – Spectral Analysis
Fourier: lost time information
6 Hz, 4 Hz, 2 Hz, 1 Hz 6 Hz + 4 Hz + 2 Hz + 1 Hz

Solution: Short Time Fourier Transform
(STFT)
perform FT on 'windowed' function:
example: rectangular window
move window in small steps over data
perform FT for every time step
STFT f ,t ' = ∫[ xtt−t ']e −i 2 f t dt
t
A. Lagg – Spectral Analysis

A. Lagg – Spectral Analysis
Short Time Fourier Transform
STFT
STFT­spectrogram shows both time and
frequency information!

A. Lagg – Spectral Analysis
Short Time Fourier Transform: Problem
narrow window function ­> good time resolution
wide window function ­> good frequency resolution
Gauss­functions as
windows

A. Lagg – Spectral Analysis
Solution: Wavelet Transformation
time vs. frequency resolution is intrinsic problem (Heisenberg Uncertainty Principle)
approach: analyze the signal at different frequencies with different resolutions
­> multiresolution analysis (MRA)
Continuous Wavelet Transform
similar to STFT:
signal is multiplied with a
function (the wavelet)
transform is calculated
separately for different segments
of the time domain
but:
the FT of the windowed signals are
not taken
(no negative frequencies)
The width of the window is changed
as the transform is computed for
every single spectral component

A. Lagg – Spectral Analysis
Continuous Wavelet Transform
CWT x = x , s = 1
∣s∣ ∫ t
... translation parameter , s ... scale parameter
t ... mother wavelet = small wave
*
t−
xt
s
dt
mother wavelet:
finite length (compactly
supported) ­>'let'
oscillatory ­>'wave'
functions for different
regions are derived from
this function ­> 'mother'
scale parameter s replaces frequency in STFT

A. Lagg – Spectral Analysis
The Scale
similar to scales used in maps:
high scale = non detailed global view (of the signal)
low scale = detailed view
in practical applications:
low scales (= high frequencies)
appear usually as short bursts or
spikes
high scales (= low frequencies) last
for entire signal
scaling dilates (stretches out) or
compresses a signal:
s > 1 ­> dilation
s < 1 ­> compression

A. Lagg – Spectral Analysis
Computation of the CWT
signal to be analyzed: x(t), mother wavelet: Morlet or Mexican Hat
start with scale s=1 (lowest scale,
highest frequency)
­> most compressed wavelet
shift wavelet in time from t 0
to t 1
increase s by small value
shift dilated wavelet from t 0
to t 1
repeat steps for all scales

A. Lagg – Spectral Analysis
CWT ­ Example
signal x(t)
axes of CWT: translation and
scale (not time and frequency)
translation ­> time
scale ­> 1/frequency

A. Lagg – Spectral Analysis
Time and Frequency Resolution
every box corresponds to a value of the wavelet
transform in the time frequency plane
all boxes have constant area
Δf Δt t = const.
low frequencies: high resolution
in f, low time resolution
high frequencies: good time
resolution
STFT: time and frequency
resolution is constant (all boxes
are the same)

A. Lagg – Spectral Analysis
Wavelets: Mathematical Approach
WL­transform:
CWT x = x , s = 1
∣s∣ ∫ t
*
t−
xt
s
dt
Mexican Hat wavelet:
t =
−t 2
1
s e
2 2
3
t2
2 −1
Morlet wavelet:
t = e i a t e − t2
2
inverse WL­transform:
xt =
1 2 ∫∫ x
, s 1
c s s t−
2 s
d ds
admissibility
condition:
c
={2∫
−∞∞
∣∣ 2
∣∣
1/2
d } ∞ with
FT
⇔
t

A. Lagg – Spectral Analysis
Discretization of CWT: Wavelet Series
­> sampling the time – frequency (or scale) plane
advantage:
sampling high for high frequencies (low scales)
scale s and 1 rate N 1
sampling rate can be decreased for low
frequencies (high scales)
scale s 2
and rate N 2
N 2
= s 1
s 2
N 1
N 2
= f 2
f 1
N 1
continuous wavelet
discrete wavelet
, s
= 1 s t−
s t = s − j/2 −
j , k 0
s j 0
t−k 0
, j , k
x
j , k =∫
t
xt = c ∑
j
*
xt j , k
tdt
∑
k
x
j , k j , k
t
orthonormal
WL­transformation
reconstruction of
signal

A. Lagg – Spectral Analysis
Discrete Wavelet Transform
(DWT)
discretized continuous wavelet transform is only a sampled version of the CWT
The discrete wavelet transform (DWT) has significant advantages for
implementation in computers.
excellent tutorial:
http://users.rowan.edu/~polikar/WAVELETS/WTtutorial.html
IDL­Wavelet Tools:
IDL> wv_applet
Wavelet expert at MPS:
Rajat Thomas

end of Wavelets
A. Lagg – Spectral Analysis

A. Lagg – Spectral Analysis
Exercises
Part I: Fourier Analysis
(Andreas Lagg)
Instructions:
http://www.linmpi.mpg.de/~lagg
Part II: Wavelets
(Rajat Thomas)
Seminar room
Time: 15:00

A. Lagg – Spectral Analysis
Exercise: Galileo magnetic field
data set from Galileo magnetometer
(synthesized)
file: gll_data.sav, contains:
total magnetic field
radial distance
time in seconds
your tasks:
Which ions are present?
Is the time resolution of
the magnetometer
sufficient to detect
electrons or protons?
Background:
Tips:
restore,'gll_data.sav'
use IDL­FFT
remember basic plasma
physics formula for the
ion cyclotron wave:
gyro
= q B
m ,
f gyro = gyro
2
If the density of ions is high enough they will excite ion cyclotron waves
during gyration around the magnetic field lines. This gyration frequency
only depends on mass per charge and on the magnitude of the magnetic
field.
In a low­beta plasma the magnetic field dominates over plasma effects.
The magnetic field shows only very little influence from the plasma and
can be considered as a magnetic dipole.
http://www.sciencemag.org/cgi/content/full/274/5286/396

A. Lagg – Spectral Analysis
Literature
Random Data: Analysis and Measurement Procedures
Bendat and Piersol, Wiley Interscience, 1971
The Analysis of Time Series: An Introduction
Chris Chatfield, Chapman and Hall / CRC, 2004
Time Series Analysis and Its Applications
Shumway and Stoffer, Springer, 2000
Numerical Recipies in C
Cambridge University Press, 1988­1992
http://www.nr.com/
The Wavelet Tutorial
Robi Polikar, 2001
http://users.rowan.edu/~polikar/WAVELETS/WTtutorial.html