Dan Werthimer - CASPER - University of California, Berkeley

Collaboration for Radio
Astronomy Instrumentation
Dan Werthimer and CASPER Collaborators
http://casper.berkeley.edu

CASPER
Collaboration for Radio Astronomy
Signal Processing and Electronics Research
Collaborators
Xilinx, Fujitsu, HP, Sun/Oracle, Nvidia, NSF, NASA, NRAO, NAIC,
CFA (Havard/Smithsonian), Haystack (MIT), Caltech, Cornell, CSIRO/ATNF,
JPL/DSN, South Africa KAT, Manchester/Jodrell Bank, GMRT (India),
Oxford, Bologna, Metsahovi Observatory/Helsinki University,
University of California, Berkeley; Swinburne University (Australia),
Seti Institute, University of California, Santa Barbara;
University of California, Los Angeles; CNRS (France), University of Maryland
Nancay Observatory, Univerity of Cape Town (South Africa),
ASTRON (Netherlands), Academica Sinica (Taiwan), Cambridge,
Brigham Young University, Rhodes University (South Africa)

The Problem with the Traditional
Hardware Development Model
• Takes 5 to 10 years
• Cost Dominated by NRE because of
custom Boards, Backplanes, Protocols
• Antiquated by the time it’s released.
• How to buy the hardware at the last
minute
• Each observatory designs from scratch

Solution:
• Modular General Purpose Hardware
– Low number of board designs
– Can be upgraded piecemeal or all together
– Reusable
– Standard signal processing model which
is consistent between upgrades.

CASPER Real-time Signal Processing Instrumentation
• Low NRE, shared by the community
• Rapid development
• Open-source, collaborative
• Reusable, platform-independent gateware
• Modular, upgradeable hardware
• Industry standard communication protocols
• Use switches to solve correlator interconnect
• Low Cost

Collaboration (not turn key instruments)
• Share Open Source Libraries
• Workshops (Tamara)
• Video’s and Doc’s on Tool Flow, Libraries
• Wiki, Mailing List
• Open Source Boards (available from vendors)

Tutorials
• 1: Introduction to Simulink, Roach and Borph
2: 10GbE
3: basic spectrometer (400MHz, 2k channels)
4: 4-input pocket correlator (400MHz, 1k ch)
5: ADC ROACH CPU/GPU
• 6: GPU tutorials (Richard Edgar, David Kirk)
Jason Manley, Terry Filiba, Mark Wagner,
Wesley New, Andrew Marten, Danny Price,
Jack Hickish, Griffin Foster

Roach Motel (Roach Nest) (KAT)

Current CASPER ADC Boards
ADC2x1000-8 (dual 1GSa/sec, single 2Gsps, 8 bit)
ADC1x3000-8 (3GSa/sec, 8 bit) ADC
(6Gsps interleaved)
64ADCx64-12 (64x 64MSa/sec, 12 bit)
ADC4x250-8 (quad 250MSa/sec, 8 bit)
katADC (dual 1.5GSa/sec, 8 bit, with gain, atten, synth)
ADC2x550-12 (dual 550 Msps, 12 bit)
ADC2x400-14 (dual 400 Msps, 14 bit)
ADC1x5000-8 (1x5Gsps,2x2.5Gsps,4x1.25G sps – Taiwan)
ADC1x1000-12 (optically isolated 12 bit 1Gsps – JPL)

Upcoming Hardware
• Roach II (Virtex 6 – South Africa team)
• Rhino (Spartan 6, ARM CPU, FMC connect)
• Roach III (Virtex 7)
• 20 to 26 Gsps ADC board

Board Interconnect - Upgradable
• Problem: Backplanes are short lived
(S100, Multibus, VME, ISA, EISA, PCI, PCIx, PCIe,
compactPCI, compactPCIe, ATCA…)
• Solution: Use 10Gbit Ethernet
(10Gbe, Infiniband, Myrinet, Xaui, Aurora)
Copper CX4 (40 meters max) or Optical

Beowulf Cluster Like General Purpose Architechture
Dynamic Allocation of Resources, need not be FPGA based
Polyphase
Filter Banks
Reconfigurable
Compute Cluster
ADC
PFB
FPGA DSP
Module
ADC
PFB
FPGA DSP
Module
.
.
.
FPGA DSP
Module
Correlator
.
.
.
Commercial off-the-shelf
Multicast 10 Gbps (10GE
or InfiniBand) Switch
FPGA DSP
Module
FPGA DSP
Module
Beamformers/
Spectrometers
FPGA DSP
Module
.
.
.
Pulsar timer
.
.
.
ADC
PFB
General-purpose CPUs

Software
Hardware
BORPH Operating System – Hayden So
• An extended version of
Linux operating system
– Treats FPGAs = CPUs
• FPGA applications execute
as hardware processes
• HW/SW communication
– UNIX file I/O
• Benefits
– Easy to understand for
novice/experienced users
– Remote control+monitor
SW SW SW
file
pipe
Device Driver
Hardware Platform
(Network, UART, HD…)
User Library
BORPH Kernel
IPC
Hardware User Library
HW
FPGA
ioreg
HW
Poster Session 3 P3_09
(11am):
File System Access From
Reconfigurable FPGA
Hardware Processes in
BORPH
socket
FPGA

Simulink-based Design Tool Flow
• Simulink Xilinx System Generator Library
• Custom BEE2 Library Blocksets
• Software programmable registers
• BEE Platform Studio

FFT controls
Simulink Library – Aaron Parsons, David MacMahon
Verilog Library – Jeff Mock
• Transform length
• Bandwidth
• Complex or Real
• Number of Polarizations
• Input bit width and output bit width
• twiddle coefficient bit width
• Run-time programmable down-shifting
• Decimate option

PFB vs. FFT

Digital Down-Converter
• Selectable # of FIR taps
• On-the-fly programmable mix frequency
• Selectable FIR coeff
• Agile sub-band selection.

X-Engine Correlation Architecture
(Lynn Urry, Aaron Parsons)

Hardware and Software Libraries
legend:

Applications

Applications
• VLBI Mark 5B data recorder – Haystack, NRAO – 512 MHz
• Beamforming – ATA, SMA –
• SETI – Arecibo (UCB)
JPL/UCB DSN (Preston, Gulkis, Levin, Jones)
• Correlators and Imagers:
ATA (Aaron Parsons, Mel Wright)
PAPER (Reionization Experiment)
Carma Next Gen
MeerKAT/SKA South Africa
GMRT next gen correlator
Bologna (SKA), FASR
Pulsar Timing and Searching, Transient
Greenbank, Allen Telescope Array, VLA,
Swinburne (Parkes), meerKAT, Nancay

SETI Spectrometers
• Parkes Southern SERENDIP
• ALFA SETI Sky Survey (300 MHz x 7 beams)
• JPL DSN Sky Survey (eventually 20 GHz bandwidth)
Radio Astronomy Spectrometers
• GALFA Spectrometer – Arecibo Multibeam Hydrogen Survey
• Astronomy Signal Processor – ASP – Don Backer, Ingrid
Stairs, et al(pulsars)
• Antenna Holography, ATNF, China
• Gavert (DSN education, outreach) – 8 GHz BW –G. Jones
• CMB Bolometer Readout – Caltech, UCB
• Fast Readout Spectrometers (Parkes, NRAO, ATA...)

ATA Fly’s Eye Transient Instrument
44 fast readout spectrometers
3 weeks to build
Geoff Bower, Jim Cordes, Griffin
Foster, Joeri van Leeuwen, Peter
McMahon, Andrew Siemion, Mark
Wagner, Dan Werthimer

Undergraduate Radio Astronomy Course

4096 channel Mars spectrometer
“Chip in a day” FPGA to ASIC

CASPER Correlator Collaboration
Allen Telescope Array (90 uS imaging – G. Jones)
PAPER (Epoch of Reionization)
Carma Next Generation
MeerKAT/SKA South Africa
GMRT next gen
Bologna
ISI (Infrared) – 6 Gsps
SKADS (Oxford)
SMA next gen (CFA, ASIAA)
FASR, Baryon Acoustic Oscillation

CASPER FX Architecture
F Engine 0
X Engine 0
F Engine 1
X Engine 1
. . .
. . .
. . .
10GbE Switch
F Engine N-1
X Engine N-1

CASPER FXB Correlator/Beamformer
(correlator needed to calibrate beamformer)
F Engine 0
X Engine 0
F Engine 1
X Engine 1
. . .
. . .
. . .
10GbE Switch
F Engine N-1
X Engine N-1

Correlators and Beamformers
• Globally Asynchronous (like a computer cluster)
• Data is time stamped with 1 PPS at ADC
• Locally Synchronous, Globally Asynchronous
• Solve problem of correlator/beamformer
interconnect problem by using 10 Gbe switches
(for both interconnect and fast readout)
• No need for high density complex boards
• Use Fifo’s to align data before correlation or
beamforming…

Correlator Comparison - 2009 benchmarks
GPU:
CPU:
1.3 MHz per GPU, Greenhill et al, http://www.scigpu.org
2.0 MHz per 8 core computer, Roy, Gupta, et al, optimized code
FPGA: 10.4 MHz per FPGA (CASPER : Xilinx XC5VSX95T roach boards)
Power (critical for SKA)
GPU: 150 watts/MHz (including GPU, CPU/motherboard, P.S.)
CPU:
FPGA:
ASIC:
150 watts/MHz
6 watts per MHz (including digitizers, P.S., CPU, motherboard)
2 watts per MHz (estimate)
(For 32 antenna, dual polarization, 500 MHz correlator)

Packetized FPGA FX Correlator Cost
(assuming non hierarchical)
Cost (2010) = Bandwidth/GHz ( $1200 N + $5 N^2 )
2010 $128M for 4,000 antenna, dual pol, 1 GHz bandwidth
2012 $ 64M
2014 $ 32M
2016 $ 16M
2018 $ 8M
2020 $ 4M
2022 $ 2M
• N = number of dual polarization receivers (full stokes)
• Moores Law Foundry Prediction through 22nm
• ITRS roadmap predicts slow down (but they always do)

Astronomy Signal Processor
Terry Filiba, Peter McMahon

Parkes Pulsar Discoveries
Bailes, Filiba, McMahon et al
• BPSR is 13 beams, 1024 channels, 64 us.
• 50 pulsars
• 11 millisecond pulsars
• a magnetar (Levin et al.)
• a pulsar with a planetary-mass companion.
• large number of RRATs due to increased
dynamic range over previous generation
filterbanks.

1960 – First Radio Astronomy Digital Correlator
21 lags
300kHz clock
discrete transistors
$19,000
Sandy
Weinreb

Correlator processing power
SKA
GFlops
10 6 DXB
10 5
10 7 10 3
ALMA
LOFAR
SMA
EVLA
.
10 9
10 4
EVN/WSRT
10 3
VLA
10 6
10 2
10
1
DLB
DCB
DAS
70 75 80 85 90 95 2000 05 10 2015
source: Arnold van Ardenne

Moores Law – Instruments using FPGA’s: 2X per year
(1,000,000 over 20 years)

Future Spectrometers
2015 4 THz 400 beams
10 GHz each
2020 128 THz 12,800 beams
2025 4000 THz 40,000 beams
2030 128,000 THz 1M beams

Cost of FPGA integer computing
2010 $200 per 1E11 MAC/sec
2012 $100
2014 $50
2016 $25
2018 $13
2020 $6
2022 $3
• XC6SLX150T FPGA with PCB, DRAM, P.S., Cooling…
• Moores Law Foundry Prediction through 22nm
• ITRS roadmap predicts slow down (but they always do)

CASPER the Friendly...
• Group Helping Open-source Signalprocessing
Technology (GHOST)
– Goal to help develop signal processing
instrumenation and libraries for the
community.
– Open source hardware, gateware, and
software.
– Provide training and tutorials
– Not so much delivering turn-key instruments
– Promote Collaboration