Status, Capabilities, and Usage of the ATA Beamformer - CASPER

Status, Capabilities, and Useof the ATA BeamformerBilly BarottJuly 8, 20081

OutlineIntroductionBasic UseHardwareFirmwareCalibration TheorySoftwareConcluding Thoughts2

Major Milestones2007 August: Prototype controls shown2007 October: 4-antennas2007 November: 24-antennas2008 March: First obs with BAPP2008 April: 42-antenna, two-pol, 1-beam2008 June: Second dual-pol b.f.Current: Upgrades, engineering tests…3

System LayoutThe ATA Synchronous Time-DomainBeamformerBeamformer: The whole thing, with onemaster software controllerBeam Processor: Independent FPGA treeBEE2: FPGA processing machineFPGA: Element of the BEE2 (one of 4)iBob: Processor with 4 ADC inputs4

Current CapabilitiesTwo Independent Dual-pol beamformers• BF1: 42-ants, 18 FPGA (two x9 FPGA b.p.)• BF2: 36-ants, 16 FPGA (two x8 FPGA b.p.)Beam processors independently steeredWired for X and Y beam processors• Calibration on X/Y – No CP yet50 MHz analog, 72 MHz Digital BandwidthIn Progress…• Let user modify coefficients• Better solar/ RFI handling in calibration5

Using the Beamformer*Detailed usage information onlog.hcro.org6

The Very Basic:InitializeCalibrateObserveScripts should be run on tumulus or user17

Initializing the Beamformer1. Verify ATA engineering status• Only observe with good feeds• Verify LNA / PAM / Focus settings2. Configure iBobs: bfibob• Walsh, Rearm, Sky, Autoatten3. Configure beamformer: bfinit.rb• -b [beamformer] -a [antlist]• bfinit.rb –b 1 –a 1ax,1bx,1cx8

Calibrating the Beamformer1. Strong Source Delay [--caldelay]• Flattens instrumental delays• Suggested: Cas-A, 3c274 (high SNR desired)2. Unresolved Phase [--calphase]• Aligns instrumental phase offsets• Suggested: 3c84, 3c48, 0927+3903. (Optional) Wideband Cal [--calfreq]• Calculates derivative of phase with frequency• Same source requirements as calphase• Slowly change --freq over wide range9

When To Calibrate*: Only if –calfreq is not used**: Not tested, but recommendedPower cycle or iBob rearmCall to bfinitMajor Link GlitchMinor Link Glitch Calibrate phase:• After potential drift• When moving “out ofcalibration” range Calibrate everything:• After potential linkdisruptionsChanging --freq (< 500 MHz)*Changing --freq (> 500 MHz)**Elapsed time (hours?)10

Is it good? (--calphase report)Both phase and RMS error should be small(Phase < 5 degrees, RMS < 30 degrees)Overflow should always be zero,but might go above zero duringinitial calibrationsPeak voltage output clips at 4096.Link diagnostics: Not fully implemented here,but will indicate trouble with XAUI links11

Steering a Beam (bftrackephem) 1. Pointing Information• Requires fixed az/el (--azel) or ATA-style ephemeris (--ephem) 2. Observing Information• Duration (-d), frequency (-f), and beamformer (-b) 3. Everything Else…• More details in logbook… Offset pointing, sidebandselection, observing with 16k-channel spectrum analyzer After obs, beam is “left on” with last values Pointing is not constrained by primary beam12

AccuracyDelay:• 5°-10° RMS typical with caldelay• F.D. FIR 0.02-smp steps (< 2° over 104 MHz)Phase:• 10°-30° RMS typical with calphase• F.D. worst error is 4° across 70% bandwidthAmplitude:• F.D. worst error is 8% (0.7 dB), most < 0.4 dB• Currently no balancing for sensitivity• Autoatten error is about 0.5 dB13

Bandwidth Filters limit bandwidth iBob DDC Passband• 78% to 3-dB points Fine Delay FIR Filter• Designed for 70% (Phase /amp degradation point)• 72 MHz (Digital)Discard DC channel• 50 MHz (Analog, shifted)Discard DC, 26.2144 MHz104 MHzFull Nyquist BandwidthiBob DDC FIR FilterFine Delay FIR Filter(Upconverted)Analog Low Pass81 MHz72 MHz50 MHz14

Output Specifications Digital: 10-Gbps XAUI Analog: XAUI into iDAC• 50 MHz …. Peak-peak max• Ordinary +/- 1.0 V pk (2v pp)• Clipping at about 1.25V peak Calibration: Isolated to B.P.• No guaranteed delay/phasebetween X and Y beamprocessors• No guaranteed cal betweendifferent beam formersNoise Output: 0.5 Volts per div.15

Things to Watch ForHigh RMS on calibration• Antenna problems or solar interferenceVery low input and AGCUpdate rates• ~5 sec: Software coefficient update• ~1024 * fringe rate = fringe update rate• ~13.5 MHz fringe table16

User Scripts(More Detail)17

User-Level Scriptsbfibob• Configures beamformer iBobs• Sets walshing, source select, attemplifier, etc.bfinit• Initializes & alters hookup• Loads firmware into FPGAsbftrackephem• Loads data for pointing and obs-time config• Used for calibration18

fibobControls the beamformer ibobs for testingand observingWalshing (on/off)Source (Sky/sinewave/noise)Reset (1pps, seed)Attemplifiers (autoatten)19

finitInitializes and creates hookupSets default peak headroomCommon Switches• -a: Antenna list• -b: Beamformer• -h: Headroom (defaults to 0.75)20

ftrackephemGeneral pointing and settingsCalibration (delay, phase, agc)Returns diagnostic informationSets atasetskyfreqSoftware-synchronous register updatesPointing Switches:• --ephem: Loads ephemeris file• --azel: Uses fixed az,el• --offset[az,el]: Offsets ephemeris file21

ftrackephem Common Switches:• -b: Beamformer select• -f: Sky frequency (MHz)• -d: Data table time (seconds)• -n: Update cycles (req. for cal) Other Switches:• -i: Integration time (seconds)• --cal[xxxx]: Calibration cycle• --write: Correlates without calibration• --sideband: Hilbert transform setting• --agcdb: Automatic gain controller level22

Exit Status ReportingImplemented:• Most script failuresComing soon:• Calibration status (Succeeded, failed,marginal)• Automatic weighting to eliminate trial-anderrordropping of inputs from hookup23

Bftrackephem StartupConnect to UserServer [-b beam]Process PointingDataHilbert Mode[--sideband sb]Sky Freq[-f MHz]AzEl GeoPoint(1s resolution)*Adjust Gain[--gainadjustXX]Stop Obs TimersPointing Offsets[--offsetaz/el]**GeoPointPointing TablesLoad Tables inBEE ServersStart Obs Timers[--timer sec]Set DataDirectoryIf Timerswere setLoad Pointing for[-d seconds]SynchronizeClocksObserving LoopExit24

Bftrackephem Observing LoopWait 2 SecondsIntegrate[-i seconds]NormalObservingWrite outputsnapshotsLook-AheadWrite (10s)Yes-n used?NoWrite “Future”Register ValuesLoop Until Count[-n counts]Loop Until Time[-d seconds]Load at “Future”Time +/- 0.1sLoop OrExit25

Bftrackephem Calibration LoopWait 2 SecondsLoad Pointing for[-d seconds]Calibration CycleProcess PointingDataLook-AheadWriteAdjust AGCIntegrate[-i seconds]Write outputsnapshotsApply CalibrationCorrectionsLoop Until Count[-n counts]Exit26

Hardware27

One Beamformer 24 ADC iBobs• 96 inputs at 104 MS/s 5 BEE2s• 4 FPGAs per BEE2• Clocked from 52 MHziBob output All hardware issynchronously clocked28

The CASPER BEE2 5 FPGAs (4 used) Embedded PowerPC• Running BORPH Linux Supports four 10 GbpsXAUI links per FPGA29

FPGA Use30Initial RV13 RB13Antennas 12 4 8Beams 2 1 1FIR Taps 2 8* 6Autorotator 0 4 8 (Muxed)FFT Bins 0 256 128I/O Registers 127 112 60Slices Used 84% 99% 100% I/O registers also affect load time 12 antennas is the absolute maximum per-BEE2-FPGA

iBobs 24 ADC iBobs (each b.f.)• Using 84 of 96 inputs• Up to 48 antennas withoutany additional hardware Overheating• Heat sink at 80C+ w/o fans• 3m XAUI links fail, but 1m ok• Use 1m copper, 3m fiber,speed rate…?• Link problem only – theFPGAs are not in danger.31

SynchronizationHeat-induced link errors cause “slip”Formerly-calibrated data streams losesynchronization!32

Synchronized (example)Sample ValueSample Value0.150.10.050-0.05-0.10.150.10.050-0.05-0.1Time Series: Synchronized0 10 20 30 40 50 60 70 80 90 1000 10 20 30 40 50 60 70 80 90 100Sample Number (Time)Time SeriesCross Correlation Phase150100500-50-100-150Cross Correlation Phase Spectra: Synchronized0 10 20 30 40 50 60 70 80 90 100Frequency BinX-CorrPhase Spectra33

5-Sample Slip (example)Sample ValueSample Value0.150.10.050-0.05-0.10.150.10.050-0.05-0.1Time Series: 5 Samples Off0 10 20 30 40 50 60 70 80 90 1000 10 20 30 40 50 60 70 80 90 100Sample Number (Time)Time SeriesCross Correlation Phase150100500-50-100-150Cross Correlation Phase Spectra: 5 Samples Off0 10 20 30 40 50 60 70 80 90 100Frequency BinX-CorrPhase Spectra34

Synchronizer Solves (most) SlipsiBob: 10ms and 1pps out-of-band signalsBEE2: Self-adjusting circuits keep o.o.b.pulses aligned• Reference link coupled to Ref-antAvailable Diagnostics• Link Offset… Times Adjusted…• XAUI “Down” and “Empty” status counts• Excessive increases indicate problems!35

Firmware Architecture36

i1.bfai2.bfaB1B2X0X1X2b01.bfaFPGA 1X-pol / Ant1X3Beam Processor TreeC33i3.bfai4.bfaB7B4X0X1X2b01.bfaFPGA ` 2X-pol / Ant2X3B40X0X1X2b01.bfaFPGA 4X-pol / Sum1X3LeafNodesi5.bfai6.bfai7.bfai8.bfai9.bfai10.bfaB5B6B3B45B8B12X0X1X2X0X1X2X0X1X2b01.bfaFPGA 3X-pol / Ant3b02.bfaFPGA 1X-pol / Ant4b02.bfaFPGA ` 2X-pol / Ant5X3X3X3B39C37B46X0X1X2BranchNodesb02.bfaFPGA ` 4X-pol / Sum2X3C39C38X0X1X2b05.bfaFPGA 1X-pol / Sum3Sub-Beam24 AntsX3B20To DAC48 Antsid1.bfa374 Antsi11.bfai12.bfaB13B14X0X1X2b02.bfaFPGA 3X-pol / Ant6X3B29Sub-beambeam8 Ants

Cascaded Data FlowLeaf Nodes (input from iBobs)• Apply beam-forming corrections• Form sub-beam• Correlate antennas with reference antennaBranch Nodes (input from leaf nodes)• Form sub-beam• Correlate leaf-node reference antennas38

Why Correlate? Early design only maximized antennas & beams• 12 antennas per leaf node, 2 beams Calibration method undefined• Iterative software adjustments (slow, inaccurate)• Copy data from external correlatorBeam-formerRF PathTuningIF ConversionADCCorrelator39

Separate Signal Paths ATA Beam-former and Correlator use separatesignal paths• Different instrumental offsets in down converter• Different power-up clock offsets in ADCs Calibration must be contained in Beam-former Current limits: 8 ants x 1 beam (and stuffed!)TuningIF ConversionADCBeam-formerRF PathTuningIF ConversionADCCorrelator40

Inside the FPGAs Reference antennaselected frominputs Later stagescorrelate referenceantennas (not subbeams)ADCADCADCADCADCADCADCADCCorrectionsCorrectionsCorrectionsCorrectionsAntenna-levelFPGA on BEECorrectionsCorrectionsCorrectionsCorrectionsReferenceAntennaSubbeamRef. Ant.SubbeamRef. Ant.SubbeamRef. Ant. ATA beamformerhas 3 stages for 42antennasADCADCAntenna-levelFPGA on BEECorrectionsCorrectionsReferenceAntennaSumming-levelFPGA on BEEReferenceAntennaADCCorrectionsADCCorrectionsAntenna-levelFPGA on BEEReferenceAntenna41

Leaf Node Signal PathPeak &Overflow1024 SmpBulk DelayFIR FilterFine DelayMult.GainXAUI OutgainiADC OutInput RegisterOutput Registerdb 0 b 2 b 4b 1 b 3 b 5Coefficientsare real onlyFringeRotatorF/X Correlator128-binMain SignalωφRef. Ant. or Selfauto Bulk Delay: 1024-sample buffer memory Fine Delay: 6-tap Real FIR Filter (12-bit coefficients) Fringe Rotator: Programmable phase angle and rate42

Branch Node Signal PathXAUI InBeamSummerGainF/X Correlator256-binRef. Ant. or SelfgainOutputSpectraautoMult.Fs / 4ShifterselectHilbertTransformselectAutoCorrelate16k-binInput RegisterOutput RegisterMain SignalPeak &OverflowXAUI OutOutputSpectraWv13/ June 2008ATA Branch Nodefirmware/ OBM Fs/4 Shifter: Moves usable spectrum to sideband Hilbert Transform: Selects real sideband output43

Output Data Format Upper half of channelis reserved for futureuse• More beams?• Correlator Output? Passes iADC out-ofbandsignals fromreference channel• 1pps / 10ms tick Signal is compatiblewith iBob output44

Fringe Rotator45

Fringe Rotator 10-bit starting angle, 32-bit fringe rate 0..6400 Hz in 1.49 uHz steps46• Register = Fringe(Hz) * 2 46 / 104.8675e6 One multiplexed table per leaf

Fractional Delay47

Sub-Sample (Fractional) Delay6-Tap, 12-Bit FIR FilterGLS Coefficients• Good for small filtersAccuracy Depends OnFilter Design!• Taps, Bandwidth, DelayX[n]b 0b 1z -1 b 2z -1 b 3z -1 b 4z -1z -1 b 5Y[n]48

Phase Error vs. Design BandwidthMaximum Phase Error (Degrees)1816141210864Phase Error for Fine Delay FilterWe Are Here6 Taps8 Taps10 Taps12 Taps24900.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9Operating Bandwidth (Normalized to 104 MHz)

Amplitude Error vs. Design BandwidthAmplitude Error for Fine Delay FilterMaximum Voltage Amplitude Error (Ratio)1.11.091.081.071.061.051.041.031.021.01We Are Here6 Taps8 Taps10 Taps12 Taps1500.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9Operating Bandwidth (Normalized to 104 MHz)

Amplitude Errors Depend on Delay1.1FD FIR / GEN. LS DESIGN L=6 wp=0.7FluctuationsDepend on DelayMAGNITUDE (Voltage)10.90.80.70.6510.50 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1NORMALIZED FREQUENCY

In practice…52

FIR Summary: 6-Taps / GLSWorst errors near the band edge70% Bandwidth Design• Worst gain error of 8% (0.7dB)• Worst phase error of 4 degrees50% Bandwidth Design (possible)• Worst gain error of 1.5% (0.13dB)• Worst phase error of 1 degree• With Spectrum Shifter and Hilbert Transform:Approach full-50 MHz output to Prelude53

F s /4 Spectrum ShifterWithout shift: Fine delaybandwidth limits usefulanalog output rangeWith shift: Non-usefulspectrum moved todiscarded sideband54* “DC” at 26.2144 MHz must be discarded

Sideband Select NotesSpectrum shifter is always on withsideband select (--sideband)Sky frequency (--freq) appears at 26 MHz• With (--sideband upper), frequencies above 26MHz are at sky frequencies above (--freq).• With (--sideband lower), the spectrum isreversed so frequencies above 26 MHz are atsky frequencies below (--freq).55

Calibration56

Three Calibration Modes Delay (--caldelay)• Strong source (casa)• Solves instrumental delay Phase (--calphase)• Unresolved source (3c84)• Assumes “zero-delay”,solves instrumental phase Frequency (--calfreq)• Assumes “zero-delay”,“zero-phase” (at F 0 )• Solves dφ/dF57

Iterative Calibrations Calibration is refined with each iteration Zero-slope forcing on weak calibrationseliminates false-turns from noisy spectra• Delay is assumed to already be zero from delay-cal58

Delay Calibration GeoPoint Delay andDelay Offset producethe total FPGA delay Delay error is estimatedby phase slope of crosscorrelation spectraGeoPointDelayDelayOffsetDelay ErrorTotal Delay59

Phase Calibration Fringe correctionsdetermined fromGeoPoint Phase,Phase Offset, Phaserate vs. Frequency,and current skyfrequency F 0 set to F sky oneach calphase (butnot obs. or calfreq) Phase error is vectormean phasePhase ErrorGeoPointPhasePhaseOffsetdΦ/dFF sky – F 0FringeStarting AngleCal TermPointing DataddtFringe RateData FlowCal FeedbackTo FPGAFrom FPGA60

Frequency Calibration Phase vs. Freq isassumed to belinear Major componentis “unknown delay”in system• Delays at skyfreqcontribute to fringeangle, delays at IFdo not.Cal TermPointing DataTo FPGAData FlowCal FeedbackFrom FPGA61

Amplitude Calibration (in progress) Goal: Weight allinputs for best beam Problem: Attemplifiersetting is noise only Requires Sensitivity:(from correlator) Two modes planned:• Equal Signal: To show“Ideal” beam pattern• MRC: To realize thebest output SNRReferencePSDSolve forErrorGeoPointAmplitudeAmplitudeGainSensitivitySelf PSDCal TermPointing DataTo FPGAFIR ScalingData FlowCal FeedbackFrom FPGA62

In SoftwarePick One: --caldelay --calphase --calfreqAlso Use: -d -i -n -f --ephem(Delay Calibration Cycle)(Phase Calibration Cycle)(Frequency Calibration Cycle)(Ephemeris to pre-load)(Integration time, seconds)(Number of iterations)(Sky frequency, MHz)(Ephemeris file)63

Examplebftrackephem -b 1 -f 1420 -i 20 -d300 --ephem “casa.ephem” -n 4--caldelaySets beamformer 1 to a skyfreq of 1420 MHz, andstarts a delay calibration cycle on CasA. Thecycle uses four iterations of 20-seondintegrations and preloads 5 minutes worth ofephemeris before each cycle.64

Potential LimitationsPhase of Cross Correlation During integration:• Fringe angle is updatedby hardware• Delay is not updated bysoftware Very long integrationswill smear results• Phase offset “ok” “Worst:” for 300m e-wbaseline w/ refant• 1 turn in 130 seconds65

Software Design66

Written for Portability A generic, cascaded-FPGA beam processor Uses Ruby for fast development cycle Only user scripts are ATA specific (Hopefully) Reusable architecture67

01.bfa b02.bfa b03.bfa b04.bfa b05.bfaSoftware Daemonsboot268

Abstraction LayersUser Scripts• Observing level, ephemeris, frequency, etc.Master User Server• Coordinate user requests• Organize antennas by hookup and by hardware• Create data tables69

Abstraction LayersBEE User Servers• Hardware only: BEE2, FPGAs, XAUIs, ADCs• Data tables (register vs. time)• Originally designed to be “Fast” on power-PCBEE Process Servers• Single data pipe from user server• Allows multiplexed access to corner FPGAs• Most writes are buffered to limit DRb calls70

Concluding Thoughts andEngineering Tests71

Simulated Beam Shape (iBob Noise)0-2Beam pattern using iBob noise (ATA-35), Az = 40 El = 80MeasuredPredictedOutput Power Relative to On-Target (dB)-4-6-8-10-12-14-16-1872-20-10 -5 0 5 10Offset Elevation (Arc Minutes)

Simulated Array Gain (iBob Noise)Signal Power (dB) vs. Single Input3530252015105Total Output power vs. Array Size, iBob NoiseIdeal GainPeak Detector15 MHz Marker25 MHz Marker073-50 5 10 15 20 25 30 35Number of Enabled Antennas

Solar Interference in Crosses74

Remaining…Speed and performance (always)Very long observation responseCircular PolarizationNull beamRFI rejectionAmplitude balancing75

Software is NOT Real-Time!Task Time (seconds)0.90.80.70.60.50.40.30.20.1Variations in software-firmware task time (Sorted)Register Write (b01)Register Load (b01)Register Write (b03)Register Load (b03) 2 sec preloadallowance Variation in loadtask affectsfringe accuracy *Recentlyimproved, butillustratesvulnerability00 10 20 30 40 50 60 70 80 90 100Trial Number76

Thoughts for the futurePower PC is slow and not real-time• C is faster, but Ruby shortens development• For 0.1 Hz fringe rate, 3.6 deg / 100ms• Include “data tables” and RTC in firmware?FFTs required for correlator, and 16multiplies per time-domain path• In retrospect, a frequency domain beamformermakes a lot of sense here…77

Questions…?78