Introduction to the Frequency Domain

Thomas Breuel (UniKL) Introduction to the Frequency Domain 2 / 1

Fourier transforms use periodic boundary conditions (we had talked aboutthis before). So, in order to make our signal looks nice and periodic, wemodify the image so that it goes smoothly towards zero around the edgesof the image. In practice, we don’t need to do something as drastic asthis, but in the example, the intermediate results look nicer.Thomas Breuel (UniKL) Introduction to the Frequency Domain 3 / 1

1 image = mean ( imread (" bridge . jpg "),axis =2)2 image = image - mean ( image )3 w,h = image . shape4 ys ,xs = meshgrid ( range (-h//2 ,h-h //2) ,range (-w//2 ,w-w //2) )5 r = minimum (h//2 - abs (ys),w//2 - abs (xs))6 r = 1.0 - exp (-r **2/80.0**2)7 image = r* image8 imrow ( image )Thomas Breuel (UniKL) Introduction to the Frequency Domain 4 / 1

1 plot ( image [200 ,:])Thomas Breuel (UniKL) Introduction to the Frequency Domain 5 / 1

Thomas Breuel (UniKL) Introduction to the Frequency Domain 6 / 1

The Fourier transform is a linear, invertible transformation of the imagethat yields a complex result. The Fourier transform for images isimplemented by the fft2 function, and its inverse by the ifft2 function.In principle, ifft2(fft2(image)) should be the identity function, butifft2 returns complex numbers; the imaginary part is nearly zero(roundoff error), so we are just interested in the real part.Note that the Fourier transformation is not shift invariant.Thomas Breuel (UniKL) Introduction to the Frequency Domain 7 / 1

1 f = fft2 ( image )2 imrow ( image , real ( ifft2 (f)))3 print amax ( abs ( imag ( ifft2 (f))))6.03959719156e-13Thomas Breuel (UniKL) Introduction to the Frequency Domain 8 / 1

What is this mysterious Fourier transform f?We can’t really easily tell, because it is complex. However, the magnitudeis kind of instructive. Since the magnitude spans a very large range, wecompress the range with a γ = 0.2 .Thomas Breuel (UniKL) Introduction to the Frequency Domain 9 / 1

1 imrow ( image , abs (f),abs (f) **0.2)Thomas Breuel (UniKL) Introduction to the Frequency Domain 10 / 1

Let’s try some simple test images. The most obvious one is the impulse.Thomas Breuel (UniKL) Introduction to the Frequency Domain 11 / 1

1 impulse = zeros ((256 ,256) )2 impulse [128 ,128] = 1.03 spectrum = abs ( fft2 ( impulse )) **0.24 imrow ( impulse , spectrum )Thomas Breuel (UniKL) Introduction to the Frequency Domain 12 / 1

Didn’t that work? Actually, it did, but the output is constant.Thomas Breuel (UniKL) Introduction to the Frequency Domain 13 / 1

1 plot ( spectrum [10 ,:])Thomas Breuel (UniKL) Introduction to the Frequency Domain 14 / 1

1 f = fft2 ( impulse ) [50 ,50:100]2 plot (f. real )3 plot (f. imag )Thomas Breuel (UniKL) Introduction to the Frequency Domain 16 / 1

Maybe something slightly bigger than an impulse gives us a better idea.Thomas Breuel (UniKL) Introduction to the Frequency Domain 17 / 1

1 step = zeros ((256 ,256) )2 step [112:144 ,112:144] = 1.03 spectrum = abs ( fft2 ( step )) **0.24 imrow (step , spectrum )Thomas Breuel (UniKL) Introduction to the Frequency Domain 18 / 1

That’s still pretty mysterious.Actually, it turns out that the signal with the simplest 2D Fouriertransform is a sine wave. This will become clearer when we look at thisagain in processing of sound signals later.Thomas Breuel (UniKL) Introduction to the Frequency Domain 19 / 1

1 ys ,xs = meshgrid ( range (300) ,range (300) )2 wave = sin (( xs +2.0* ys)*pi /15.0)3 spectrum = abs ( fft2 ( wave )) **0.24 imrow (wave , spectrum , spectrum [:50 ,:50])Thomas Breuel (UniKL) Introduction to the Frequency Domain 20 / 1

Notice that there is just a single pixel at (10,20). Why is it there?Roughly speaking, because there are 30 repeats in the y direction, and 60repeats in the x direction.We can now mix multiple of these signals and the Fourier transform willresolve these components for us.Thomas Breuel (UniKL) Introduction to the Frequency Domain 21 / 1

1 wave = sin (( xs +2.0* ys)*pi /15.0) + cos ((3.0* xs+ys)*pi /15.0) + sin (( xs+ys)*pi /15.0) +3.0* sin (xs*pi /150.0) +3.0* sin (ys*pi /150.0)2 spectrum = abs ( fft2 ( wave )) **0.23 imrow (wave , spectrum , spectrum [:40 ,:40])Thomas Breuel (UniKL) Introduction to the Frequency Domain 22 / 1

Fourier transforms model images as being composed of sine waves that areshifted and added.This turns out not to be a very good statistical model of real-worldimages. It is, however, a good model for audio and radio signals, which iswhere most of the theory comes from. And the theory works even if theimages are not naturally represented using the basis functions of the filters.Thomas Breuel (UniKL) Introduction to the Frequency Domain 23 / 1

Thomas Breuel (UniKL) Introduction to the Frequency Domain 24 / 1

The basic idea behind frequency domain filtering is to perform a Fouriertransform, select a subset of the frequencies–those we are interestedin–and then transform back.Now, the low frequencies happen to be in the corners of the Fouriertransform (this is just a convention).So, we take the original Fourier transform and just copy values from itscorners, then transform back. And let’s also transform back everything butthe values we selected.Thomas Breuel (UniKL) Introduction to the Frequency Domain 25 / 1

1 f = fft2 ( image )2 f2 = zeros (f. shape , dtype = complex )3 d = 304 f2 [:d ,:d] = f[:d ,:d]5 f2 [:d,-d:] = f[:d,-d:]6 f2[-d:,:d] = f[-d:,:d]7 f2[-d:,-d:] = f[-d:,-d:]8 imrow ( abs (f) **0.2 , abs (f2)**0.2 , abs (f-f2) **0.2)9 imrow ( image , real ( ifft2 (f2)),real ( ifft2 (f-f2)))Thomas Breuel (UniKL) Introduction to the Frequency Domain 26 / 1

What we can see here is that if we just select low frequencies (a low passfilter), we get a filter that looks like a smoothing filter.If we just select high frequencies (a high pass filter, we get a filter thatlooks like a derivative filter.However, there is a lot of ringing in these images, the little wavy blacklines surrounding edges in the filtered images. Strict low-pass or high-passfilters show this effect.Thomas Breuel (UniKL) Introduction to the Frequency Domain 27 / 1

Filters in the Frequency Domain

Speedup from FFTThe Fourier transform is a very special kind of filter, because it transformsconvolutions into multiplications.Let’s write F[s] for the Fourier transform of the signal s . Then,F[s ∗ t] = F[s] · F[t] (1)ors ∗ t = F −1 [F[s] · F[t]] (2)This matters a great deal because using the fast Fourier transformalgorithm (FFT), we can compute the Fourier transform of a signal withN elements in time N log N .So, a naive implementation of convolution of two sequences of length Ntakes time O(N 2 ) , while a convolution implemented with Fouriertransforms takes time O(N log N) . Radios, disk drive, cell phones, andmany other modern devices take advantage of this.Thomas Breuel (UniKL) Introduction to the Frequency Domain 29 / 1

Let us explore this issue computationally a bit more by looking at theGaussian filter in the image domain and the frequency domain.1 def impulse (w,h):2 image = zeros ((w,h))3 image [w//2 ,h //2] = 1.04 return imageThomas Breuel (UniKL) Introduction to the Frequency Domain 30 / 1

Let’s compute the impulse response of the Gaussian filter.Thomas Breuel (UniKL) Introduction to the Frequency Domain 31 / 1

1 from scipy . ndimage import filters2 gaussian = filters . gaussian_filter ( impulse (64 ,64) ,8.0)3 imrow ( gaussian )Thomas Breuel (UniKL) Introduction to the Frequency Domain 32 / 1

We can convolve with that directly to obtain the following image.Thomas Breuel (UniKL) Introduction to the Frequency Domain 33 / 1

1 result1 = filters . convolve ( image , gaussian )2 imrow ( result1 )Thomas Breuel (UniKL) Introduction to the Frequency Domain 34 / 1

Alternatively, we can multiply with the Fourier transform of the impulseresponse.(We recompute the Fourier transform because the fft2 function wantsthe impulse response centered on (0,0), while the convolve functionwants it centerd on (w//2,h//2). Furthermore, fft2 requires an impulseresponse padded with zeros to the size of the image.)Thomas Breuel (UniKL) Introduction to the Frequency Domain 35 / 1

1 gaussian0 = filters . gaussian_filter ( impulse (* image . shape ) ,8.0)2 gaussian = roll2 ( gaussian0 . copy () ,w//2 ,h //2)3 imrow ( gaussian0 , gaussian )Thomas Breuel (UniKL) Introduction to the Frequency Domain 36 / 1

1 result2 = real ( ifft2 ( fft2 ( image )* fft2 ( gaussian )))2 imrow ( result2 )Thomas Breuel (UniKL) Introduction to the Frequency Domain 37 / 1

These two are the same.Since we’re going to look at more spectra (magnitudes of Fouriertransforms), let’s define a function that makes this more convenient. Thisfunction also shifts the low frequencies to the center of the image, wherethey are easier to see.Thomas Breuel (UniKL) Introduction to the Frequency Domain 38 / 1

1 def spectrum ( image , mode = lambda x:x **0.2 , eps =1e -6) :2 w,h = image . shape3 return roll2 ( mode ( maximum (eps , abs ( fft2 ( image )))),w//2 ,h //2)Thomas Breuel (UniKL) Introduction to the Frequency Domain 39 / 1

1 imrow ( image , spectrum ( image ))Thomas Breuel (UniKL) Introduction to the Frequency Domain 40 / 1

So, the spectrum of the original image has a lot of high frequencycomponents. These are organized in “spikes”, and these are actuallyperpendicular to the largest edges found in the image.Let’s look at the filtered outputs.Thomas Breuel (UniKL) Introduction to the Frequency Domain 41 / 1

1 imrow ( result1 , spectrum ( result1 ))Thomas Breuel (UniKL) Introduction to the Frequency Domain 42 / 1

The above filter was implemented by the convolve function. It got rid ofmost of the high frequency components, although there are smallishstreaks still visible.Thomas Breuel (UniKL) Introduction to the Frequency Domain 43 / 1

1 imrow ( result2 , spectrum ( result2 ))Thomas Breuel (UniKL) Introduction to the Frequency Domain 44 / 1

That filter was implemented by Fourier transform. The image looks thesame, but it really got rid of all high frequency components. Thisdifference is actually very slight and probably related to the tructation inthe convolve function. It is only visible because we are using γ = 0.2 fordisplay.Now, we multiply the image with the Fourier transform of the Gaussian.What does that look like?Thomas Breuel (UniKL) Introduction to the Frequency Domain 45 / 1

1 imrow ( roll2 ( gaussian ,w//2 ,h //2) , spectrum ( gaussian ))Thomas Breuel (UniKL) Introduction to the Frequency Domain 46 / 1

The smooth falloff of the Gaussian is really important. Let’s construct alowpass filter of exactly the same size as the Gaussian and compare thetwo.Thomas Breuel (UniKL) Introduction to the Frequency Domain 47 / 1

1 f = fft2 ( gaussian )2 mag = abs (f)3 lowpass = where (mag >0.5 , f/mag ,0.0)4 imrow ( roll2 ( abs (f),w//2 ,h //2) ,roll2 ( abs ( lowpass ),w//2 ,h //2) )Thomas Breuel (UniKL) Introduction to the Frequency Domain 48 / 1

1 imrow ( real ( ifft2 ( fft2 ( image )*f)),real ( ifft2 ( fft2 ( image )* lowpass )))Thomas Breuel (UniKL) Introduction to the Frequency Domain 49 / 1

As we can see above, the Gaussian-filtered image looks smoothly blurred,while the lowpass filtered image has roughly the same amount of blur, butlots of additional “structure” due to ringing.Thomas Breuel (UniKL) Introduction to the Frequency Domain 50 / 1