Documentation of Wideband VLBA Software - National Radio ...

Abstract
1 Introduction
VLBA Sensitivity Upgrade Memo 3x
VLBA Wideband Software
Walter Brisken
17 Oct 2011 draft v.1
This document serves as an overview of the software being added to VLBA operations as part of the VLBA
Sensitivit Upgrade Project. New software that will be installed as part of the C-band upgrade project
will also be included here. The top-level sections are organized by the devices or computers on which
the software is installed. The second-level of the heirarchy represents natural installation units, which can
include multiple individual programs. In most cases these units are Redhat Packages. Many of the larger
software subsystems have extensive documention which will not be repeated here but this document will
serve as primary documentation for many of the smaller or more obscure utilities. This document is meant
to be useful for a wide audience including operators, software engineers and systems administrators. It is
expected that this document will evolve with time. At the time of writing this documented’s Latex source
is stored in subversion at https://svn.aoc.nrao.edu/repos/VLBA/DigitalBackend/trunk/doc/memos/
upgradesoftware . Hopefully some effort will be made to prevent this document from getting too far out
date.
2 Mark5C
The following sections describe software running on NRAO Mark5C units that are deployed at VLBA
stations, the EVLA, GBT and the test rack. All NRAO Mark5 units run RedHat Enterprise Linux (version
5 as of this writing). Except for DRS, all of the software is currently packaged as Redhat Packages (.rpm
files) which allows computing staff to easily deploy upgrades and configure a clean unit.
Each VLBA station has at least one Mark5C unit. The primary Mark5C unit at each station will have
hostname xx-mark5c-1, where xx is the standard two letter station code. At the EVLA and GBT, the two
letter codes are ea and gb respectively. Provisions for a second Mark5C unit with name xx-mark5c-2 have
been in plans since the beginning. Pie Town and Mauna Kea each have a Mark5C unit specifically installed
to support daily observations made by the US Naval Observatory. These units are called pt-mark5c-usno
and mk-mark5c-usno respectively.
2.1 Streamstor SDK
Package streamstor-9.1-1.i386.rpm
SVN path https://svn.aoc.nrao.edu/repos/VLBA/mark5/trunk/src/streamstor
Requires 32-bit Linux operating system 1
Provides StreamStor driver (windrvr6.ko)
StreamStor firmware (*.xfb)
include files (xlrapi.h, xlrdbcommon.h, xlrtypes.h)
libraries (libssapi.a, libwdapi1021.so)
streamstor.pc
utilities (dumpram, sflash, sslog, ssopen, ssprintVersions, sstest)
For the most part this is a repackaging of the Software Development Kit (SDK) made by conduant. The
file streamstor.pc is added to aid in detection of the library in the compilation of packages that depend on
this library.
1

The included utilties do not make use of the locking mechanism that is used by the NRAO and DiFX
Mark5 software and by DRS. Thus it is very important to make sure that no program is currently using the
StreamStor card when running any of these programs. This includes stopping mk5daemon which wakes up
the StreamStor card every 10 seconds for a status update.
Please see the official Conduant documentation for more information about any of the above software
components.
2.1.1 Firmware upgrades
Note that with the Amazon cards that are used in Mark5C units, upgrading this package almost certainly
requires a manual firmware upgrade. This is done with the ssflash utility. Running ssflash requires a
module to be installed in bank A (one without critical data is suggested) and no other software accessing
the StreamStor card can be running. The recommended procedure to upgrade firmware on an NRAO Mark5
unit is:
1. Run StopDRS and StopMK5D (see §??)
2. Run ssprintVersions and record the current versions of the firmware and API components
3. Upgrade the package: rpm -Uvh streamstor-X.i386.rpm
4. Reboot the unit
5. Run ssprintVersions to verify an upgrade in the API version
6. cd into $STREAMSTOR BIB PATH (/usr/share/streamstor/bib)
7. Run ssflash -u SDK9.1.ssf (replacing 9.1 with the new version)
8. Upon completion, reboot the unit
9. Verify firmware upgrade with ssprintVersions
10. Start DRS or mk5daemon as desired
2.2 DifxMessage
Package difxmessage-2.0.2-5.i386.rpm
SVN path https://svn.atnf.csiro.au/difx/libraries/difxmessage
Provides shared libraries (libdifxmessage.so, libmark5ipc.so)
static libraries (libdifxmessage.a, libmark5ipc.a)
include files (difxmessage.h, mark5ipc.h)
difxmessage.pc
utilities (mk5mon, cpumon, . . .)
The DifxMessage package provides libraries that implement the DiFX software correlator messaging system
and the Mark5 locking mechanism. While several utilities are included with this library most are not of
much use at a station Mark5 unit.
Please see the DiFX documentation for more information about the DifXMessage software.
2

2.3 Mark5Access
Package mark5access-1.4.3-1.i386.rpm
SVN path https://svn.atnf.csiro.au/difx/libraries/mark5access
Provides shared library (libmark5access.so)
static library (libmark5access.a)
include file (mark5access.h)
mark5access.pc
utilities (m5d, m5fold, m5pcal, m5spec, zerocorr, . . .)
This library is for the decoding of VLBI baseband data. It currently supports Mark4, VLBA and Mark5B
formats fairly completely. Single thread variants of VDIF are handled at a minimal level.
Please see DiFX documentation for more information.
2.4 Mk5Daemon
Package mk5daemon-2.0.2-9.i386.rpm
SVN path https://svn.atnf.csiro.au/difx/applications/mk5daemon
Requires streamstor, difxmessage, mark5access
Provides mk5daemon
utilities (fileto5c, mk5cp, mk5dir, mk5erase, mk5map, record5c, recover, testmod, vsn, . . .)
The program mk5daemon was originally written to aid in the transition from the hardware to the software
correlator during which time Mark5 units were capable of feeding both systems. It was responsibile for starting
and stopping the Mark5A program and maintaining current state of modules for the software correlator.
This software has evolved to do much more and now supports a VSI-S interface that matches the capabilities
of DRS fairly closely. DRS cannot be run at the same time as mk5daemon as both want to accept messages
on the VSI-S port (2620) and both need access to the StreamStor card 2 .
The VSI-S interface supported by mk5daemon is almost equivalent to that of DRS. Please see the DRS
documentation, but note the following differences:
• DTS id replies with system type mk5daemon 3 instead of Mark5C
• record? queries will return the data rate as an additional parameter
• error? queries are handled very differently. FIXME: document this
• scan set, scan check, and data check are not yet supported
The architecture of mk5daemon is very different than that of DRS. The key differences are:
• A single programming language is used (C++) rather than two (python, C++)
• The StreamStor card is not kept open; only occasional peeks at the modules are made
• All major module functionality is handled through helper programs record5c, mk5erase, recover,
vsn, . . .
• mk5daemon will block VSI-S commands for up to 3 seconds on record stop to ensure consistency in
state after recording.
• Logging infrastructure is simpler but more consistent
• Facilities for extracting S.M.A.R.T. data from modules is deeply integrated
2 Note however that DRS can be set to a personality that does not lock the StreamStor.
3 mk5daemon can be compiled without StreamStor support; the system type parameter will report mk5daemon noss in this case
and will only support a small subset of the VSI-S commands.
3

The differences between the two programs is hoped to aid in the stabilization of the Mark5C recorder. Having
two independent code bases can expediate isolation of problems. The choice of which software to ultimately
use has not been made yet.
Most of the included utilities are explained in the DiFX documentation. See §?? for details on starting
and stopping mk5daemon
2.5 DRS
SVN path svn+ssh://wbrisken@vault.haystack.mit.edu/svnrepos/DRS
Requires streamstor
Provides drs
drs client
The Data Recording System (DRS) is the standard software for recording data on a Mark5C unit. This
software has extensive documentation and won’t be described in detail here. See §?? for details on starting
and stopping DRS on NRAO Mark5C units.
A Redhat package (.rpm) for DRS will become the primary distribution mechanism as soon as python
2.5 becomes standard with RedHat Enterprise Linux.
2.6 Mark5C MIB Emulator
Package mib mark5c-0.7-1.i386.rpm
SVN path https://svn.aoc.nrao.edu/repos/VLBA/mib_mark5c
Provides mib mark5c
Configuration file mib mark5A pt.xml
The Monitor Interface Board (MIB) is a single board computer monitor & control interface used by the
EVLA. The software on these boards allows high-level communication via XML over TCP or UDP. Archive
data and alerts are multicast for other software on the local network to intercept. Several subsystems have
software that emulates the functionality of a MIB. The VLBA Sensitivity Upgrade Project is adopting the
EVLA control system and thus the MIB infrastructure.
The Mark5C MIB is a piece of software that runs on the Mark5C unit. It connects to the VSI-S
interface of DRS or mk5daemon. It periodically (ranging between seconds and hours) makes queries. When
commanded to do so, the MIB emulator will also send VSI-S commands. This software is also responsible
for implementing auto-erase of modules based on their Disk Module State and bank switching on module
fill-up.
To start this program from the command line, you must include a command line parameter that points
to the path where the configuration file lives:
mib mark5c -p/etc
Logging information will be sent to stdout and stderr.
See §?? for details on starting and stopping mib mark5c on NRAO Mark5C units.
2.7 Mark5C logging software
Package logmark5c-0.10-4.i386.rpm
SVN path https://svn.aoc.nrao.edu/repos/VLBA/logmark5c
Provides logmark5c (and StartDRS, StopDRS, & RestartDRS)
logmk5d (and StartMK5D, StopMK5D, & RestartMK5D)
logmibmark5c (and StartMIB, StopMIB, & RestartMIB)
start-up scripts nrao-mark5, nrao-mk5d, & nrao-mibmark5c
A system of programs that provide logging and boot-persistent start/stop were created to increase the
simplicity and consistency of software that must be running on Mark5 units for proper operation. This
package has this logging software for DRS, mk5daemon and mib mark5c.
4

2.7.1 logmark5c
logmark5c is a program that is started on boot by nrao-mark5c, a script that lives in /etc/init.d . On
start-up it begins monitoring a watch file /var/spool/mark5/logmark5c.watch. The contents of the first
line of this file are used to control the state of DRS. If the line contains a positive integer, DRS will be started,
if not already active; if zero or negative, it will be stopped if not already stopped. Two key properties this
system provides are:
• Ability for any user to control program running as root in a secure way.
• Prevent more than one instance of the program from being run.
The watch file is most conveniently managed by three scripts:
1. StartDRS: Start the program
2. StopDRS: Stop the program
3. RestartDRS: Stop and restart the program
Note that RestartDRS recognizes that the MIB software likely needs to be restarted after restarting DRS, so
it first checks the state of the /var/spool/mark5/logmibmark5c.watch; if it is on, then the MIB software
is first stopped and then started. Note also that the scripts know that DRS and mk5daemon cannot coexist,
so StartDRS will first ensure mk5daemon is stopped (and vice-versa).
Logging facilities collect stdout and stderr coming from DRS. Native time-tag information from these
text streams are removed and DEBUG level messages are dropped. The logging software reformats the text
with just one time-tag per second (maximum). Example log output (from 2011 302 18 00 00.drs.log):
2011/302 18:01:58
INFO Input: status?;
INFO Output: [1, ’!status? 0 : 0x00300001 ;\n’]
INFO Input: bank_set?;
OTHER arg passed was A
OTHER Getting the status for Bank 0
OTHER 0 status was successful
OTHER XLR_LABEL_LEN is 100
OTHER label NRAO+313/8000/1024
OTHER state 2 sel 1 pwr_req 1
OTHER pwr_en 1 media_status 1 wr_prot 0 err_code 0 err_data 0
OTHER len 264a0a0aeb8 total_cap 74701830 total_cap_bytes 74701830000
OTHER arg passed was B
OTHER Getting the status for Bank 1
OTHER 1 status was successful
OTHER XLR_LABEL_LEN is 100
OTHER label LABEL NOT AVAILABLE
OTHER state 0 sel 0 pwr_req 0
OTHER pwr_en 0 media_status 0 wr_prot 0 err_code 0 err_data 0
OTHER len 0 total_cap 0 total_cap_bytes 0
INFO Output: [1, ’!bank_set? 0 : A : NRAO+313/8000/1024 : - : - ;\n’]
INFO Input: record?;
INFO Output: [1, ’!record? 0 : off : 500 : TESTPFB_AT_No0001 ;\n’]
Log data is written into a log directory /export/home/mark5/vlbamon/logs into files that rotate every
6 hours. After each file is completed (a new 6 hour portion of the day is started), the log file is gziped. The
log directory is monitored by tmpwatch with files older than 60 day automatically deleted.
5

2.7.2 logmk5d
This program is completely analogous to logmark5c, but controls mk5daemon instead. Example log output
(from 2011 302 18 00 00.mk5d.log):
2011/302 18:00:01
VSI-S received: status?;
VSI-S responding: !status? 0 : 0x02340001;
VSI-S received: bank_set?;
VSI-S responding: !bank_set? 0 : A : UNAM+010 : B : NRAO+398;
VSI-S received: record?;
VSI-S responding: !record? 0 : off : 117 : BR152G_PT_No0072;
VSI-S received: bank_info?;
VSI-S responding: !bank_info? 0 : A : 12371057209816 : B : 16003099193344;
2.7.3 logmibmark5c
This program is completely analogous to logmark5c, but controls mib mark5c instead. Example log output
(from 2011 301 00 00 00.mib mark5c.log):
2011/301 00:00:03
rx_msg @: 55862.000034722223: get mark5c.ConnectionState
rx_msg @: 55862.000034722223: get mark5c.Online
rx_msg @: 55862.000034722223: get MARK5C.RecordState.msg
rx_msg @: 55862.000034722223: get MARK5C.ConnectionState
rx_msg @: 55862.000034722223: get bank0.TotalBytesRecorded
rx_msg @: 55862.000034722223: get bank0.ModuleUsed
rx_msg @: 55862.000034722223: get bank1.ModuleUsed
rx_msg @: 55862.000034722223: get bank1.TotalBytesRecorded
rx_msg @: 55862.000034722223: get mark5c.BankASelected
rx_msg @: 55862.000034722223: get mark5c.BankBSelected
rx_msg @: 55862.000034722223: get mark5c.BankAReady
rx_msg @: 55862.000034722223: get mark5c.BankBReady
rx_msg @: 55862.000034722223: get mark5c.Online
sent: status?;
recv: !status? 0 : 0x02340001;
sent: bank_set?;
recv: !bank_set? 0 : A : UNAM+010 : B : NRAO+398;
3 RDBE
Each VLBA station and the GBT will have two RDBEs at the completion of the VLBA Sensitivity Upgrade
Project with hostnames xx-mark5c-1 and xx-mark5c-2, where xx is the standard two letter station code.
The EVLA will not have any RDBEs as the WIDAR correlator will provide equivalent functionality.
Unlike the other Linux-based components of the VLBA, the RDBE natively runs a small Debian-based
distribution and thus Redhat Packages are not used for software distribution. FIXME: explain what is
actually done.
6

3.1 Server
SVN path svn+ssh://vault.haystack.mit.edu/svnrepos/RDBE
Provides
3.2 RDBE MIB
SVN path https://svn.aoc.nrao.edu/repos/VLBA/mib_rdbe
Provides
4 Control Computer
Each VLBA station and the GBT has a new Linux-based Control Computer with hostname xx-cc, where xx
is the standard two letter station code. The EVLA will make use of existing computers at the site to host a
relevant subset of software.
4.1 Observe Script Monitor
Package ObservScriptMonitor-0.0.8-2.x86 64.rpm
SVN path https://svn.aoc.nrao.edu/repos/VLBA/ObservScriptMonitor
Provides ObservScriptMonitor-0.0.8.jar
utilities (observDebug.sh, observInfo.sh, & observMon.sh
This program is started upon boot of the control computer and monitors file /export/home/cc/vlba/scripts/observ.tx
for upcoming observations. When it is time to begin an observation, this program sends a request to a running
Exector to start the file. Some example lines from the observ.tx file follow:
TD023A 2011OCT06 13h25m00s 2011OCT06 13h54m15s > pt
TD023B 2011OCT06 13h55m00s 2011OCT06 14h25m15s > pt
TD023C 2011OCT06 14h27m00s 2011OCT06 16h17m15s > pt
The format of this file is identical to that of observ.tx that runs on the legacy system, allowing one file
to be used for both. By default, entries in observe.tx start 5 minutes before the actual observation (as
specified in the vex file) start to allow slewing to the first source. The end time of the entry is 5 seconds after
nominal end of observation to provide time for finalization. It seems that in practice this simplification will
not be exploited as a script (see §??) will automatically update the observ.tx file at each VLBA control
computer as necessary.
4.2 Executor
Package observe-0.5.0k-3.x86 64.rpm
SVN path https://svn.aoc.nrao.edu/repos/FIXME
Provides vlba-observe-with-dependencies.jar
The executor (also called the observe executor or observation executor) is the software that reads a python
control script and generates the commands to cause an observation to happen. Currently the executor only
controls the new equipment installed as part of the VLBA Sensitivity Upgrade Project (Mark5C, RDBE
and 4x4 matrix switch). Ultimately the executor will completely replace the vxWorks-based legacy control
system that currently controls all of the antenna motion, tuning, etc.
FIXME: What installs execAdd, execInfo, & execKill?
7

4.3 Executor Logger
Package logExecutor-0.3-1.x86 64.rpm
SVN path https://svn.aoc.nrao.edu/repos/VLBA/logExecutor
Provides logExecutor
control scripts (StartExecutor, StopExecutor, & RestartExecutor)
start-up script (nrao-executor)
The executor logger is completely analogous to logmark5c (§??). Log data is stored in /export/home/cc/vlba/logs.
Example log data from :
2011/302 20:55:00
Oct 29, 2011 8:55:00 PM edu.nrao.evla.observe.Executor addJob
INFO: 20:55:00 Oct29::Executor->Add job :’N1302’; script: /export/home/cc/vlba/scripts/N1302.PT.py
Oct 29, 2011 8:55:00 PM edu.nrao.evla.observe.Executor start
INFO: 20:55:00 Oct29::Executor->Job ’N1302’ started...
N1302 55863 20:55:00 Problem setting TAI-UTC in L350
Oct 29, 2011 8:55:00 PM edu.nrao.evla.observe.Array println
INFO: N1302 55863 20:55:00 Problem setting TAI-UTC in L350
MatrixSwitch - Antenna name abbreviation: pt
Matrix Switch name: pt-t450
Oct 29, 2011 8:55:00 PM edu.nrao.evla.observe.MatrixSwitch
INFO: Matrix 4x4 Switch socket address: pt-t450/10.81.106.176:7000
4.4 Remote Monitor Data Server
Package FIXME!
SVN path https://svn.aoc.nrao.edu/repos/VLBA/RemoteMonitorDataServer
Provides RemoteMonitorDataServer.final.jar
The Remonte Monitor Data Server listens for multicast XML messages. Two classes of messages are collected:
alerts and archive monitor data. The data is collected and periodically compressed and FTPed to the operations
center where it is ingested into the monitor database. FIXME What installs nrao-RemoteMonitorDataServer
into /etc/init.d?
4.5 Alerts Server
SVN path https://svn.aoc.nrao.edu/repos/EVLA/alerts-server
Provides vlba-alerts-server.war
The VLBA Alerts Server is a near clone of the same tool used at the EVLA. It is a web service that listens
for multicast alerts from MIB or MIB emulator equipped devices and stores a table of active alerts. All
active alerts can be requested from a web portal FIXME: http address goes here at each VLBA site.
4.6 RDBE Switched Power Receiver
Package rspr-0.1-1.x86 64.rpm
SVN path https://svn.aoc.nrao.edu/repos/VLBA/rspr
Requires logExecutor
Provides rspr
logger (logrspr)
control scripts (StartRSPR, StopRSPR, & RestartRSPR)
start-up script (nrao-rspr)
utilities (switchedpowerfake)
8

The RDBE Switched Power Receiver program runs on the control computer at each VLBA site (and GBT)
and listens for multicast switched power measurements from the RDBEs at each site. It collects observation
setup metadata from the RDBEs and labels the measurements according to frequency and polarization in
a text file. This text file is periodically (nominally 4 times daily) gziped and sent to central operations by
the Remote Monitor Data Server. The data are left as comperessed files on a machine within direectory
FIXME: TBD.
The utility switchedpowerfake can be used as a software development tool. It broadcasts semirealistic
switched power data in the same manner the RDBE would.
5 Central Operations
5.1 Operator GUIs
SVN path https://svn.aoc.nrao.edu/repos/VLBA/opspack
The GUI can be spawn via web start from http://www.aoc.nrao.edu/asg-internal/software/vlba/ .
5.2 vex2difx utilities
SVN path https://svn.atnf.csiro.au/difx/applications/vex2difx
Requires difxmessage, difxio
Provides vex2script
queueVex
goUSNO
several other DiFX tools
5.2.1 vex2script
The new VLBA software system is controlled by a .vex file. The program �vex2script converts one station
at time the .vex file to python control files.
5.2.2 queueVex
queueVex is a script that runs vex2script, copies the python control files to the VLBA stations, and updates
the observ.tx file appropriately.
5.2.3 goUSNO
goUSNO is a special purpose script that is used to schedule daily UT1-UTC observations for the USNO. The
program goes through the following steps:
1. generate a schedule (.key) file
2. run sched to generate .vex and crd files
3. run db2vex to add predicted EOPs and clocks to generate a .vex.preobs file
4. run vex2script to generate executor control files
5. copy executor Mark5C control files to VLBA stations
6. update the Mark5C observ.tx file on the new systems at VLBA stations
7. email a confirmation to a special purpose mailing list
9

6 Acronym Salad
API Application Programming Interface (essentially a set of function calls)
DiFX Distributed FX, the software correlator used by the VLBA
DRS Data Recording System (Mark5C)
DTS Data Transmission System
EOP Earth Orientation Parameter
EVLA Expanded Very Large Array
GBT Greenbank Telescope
MIB Monitor Interface Board (here usually used to mean MIB emulator)
NRAO National Radio Astronomy Observatory
RDBE ROACH Digital Back End
ROACH Reconfigurable Open-Architecture Computing Hardware
RPM Redhat Package Manager: the software package system for Redhat Linux
RSPR RDBE Switched Power Receiver: a program that collects data for Tsys
SDK Software Development Kit
SS StreamStor
SVN Subversion, a software version control system
TCP Transmission Control Protocol, a network communication type
Tsys
System Temperature
USNO US Naval Observatory
UDP Unreliable Datagram Protocol, a network communication type
UT1 Universal Time as identified by the phase of Earth rotation
UTC Coordinated Universal Time (atomic time)
VDIF VLBI Data Interchange Format, a new format for baseband data
VLBA Very Long Baseline Array
VLBI Very Long Baseline Interferometry
VSI-S VLBI Standard Interface for Software, a query/control protocol
XML Extensible Mark-up Language
10