IDC Development LAN Operator's Guide - ATM - Comprehensive ...

IDC/DEVLAN/OG-ATMPage 2Document historyVersion Date Author Description0.1 7 June 2004 Alexander Boresch draft document structure0.2 12 August 2004 Marian Harustak partial draft procedures in chapters 3 and 40.3 16 September 2004 Alexander Boresch complete draft version0.4 15 October 2004 Alexander Boresch revised draft version1.0 28 November 2004 Alexander Boresch final SHI version1.1 16 August 2005 Alexander Boresch baseline for expanded version (SHI, RN, ATM)1.2 7 February 2006 Henrik Edlund ATM chapters added1.3 14 December 2006 Gertrud Waich added autoconf build procedures, updated SHIchapters2.0 5 January 2007 Gertrud Waich,Alexander Boresch3.0 20 May 2008 Vladimir Gelashvili,Jan Wüster (review)full SHI, RN and ATM versionCreated a separate volume for ATM software;restructured the document to make it consistent withSHI and RN volumesMay-08

IDC/DEVLAN/OG-ATMPage 4List of FiguresFigure 1. IDC technologies and data processing systems ........................................................5Figure 2. ATM Software Control Flow Diagram...................................................................11May-08

IDC/DEVLAN/OG-RNPage 51. INTRODUCTION1.1. Identification and Purpose of the DocumentThis document describes the procedures used to operate and maintain the AtmosphericTransport Modelling (ATM) software version 2.0 on the IDC Development LAN. It providesan overview of the existing infrastructure and also describes the hardware, software andenvironment baseline, which is needed to properly operate the Development LAN software.The addressed audience is primarily the Development LAN operators, but also includes alldevelopers, testers, users and administrators of the Development LAN hardware and software.There are two companion documents, [IDC/DEVLAN/OG-SHI] and [IDC/DEVLAN/OG-RN], which describe the Development LAN’s Operations and Maintenance procedures for thewaveform technologies processing software and the radionuclide data software, respectively.The current document is based on general framework provided by the following documents:• IDC Operational Manual, [IDC-OM]: Describes the general operational framework andmission of the IDC. Available at H:\Conference Documents\Official Documents\PMOsand AG\WGA\WGB\Operational Manuals\IDC OM in the CTBTO Intranet.• Draft Procedures of the IDC Configuration Control Board (CCB), [IDC-CCB]: Describesthe software and configuration change procedures for the IDC Testbed and OperationsLANs. Available at: http://idc030.idc.ctbto.org:8000/Docs/CCB-Documentation.html inthe CTBTO Intranet.IDC Software Documentation Orientation Booklet, [IDC-SDO]: Introduces new users to theIDC work environment including the IDC computer infrastructure and software. It is availableat http://intranet.ctbto.org/stafforientation02mar20.pdf .1.2. Purpose and Role of the Development LANThe Development LAN was created in July 2002 and serves as a development and testplatform for the IDC software. This includes processing software for all CTBT monitoringtechnologies Seismic, Hydro-acoustic and Infrasound (SHI) data processing, Radionuclide(RN) data processing and Atmospheric Transport Modelling (ATM).Figure 1. IDC technologies and data processing systemsThe Development LAN is owned by the IDC Software Integration unit and is under lessrigorous configuration control than other IDC LANs (Testbed and Operations LANs). Whennew or modified software becomes available and has successfully passed unit testing by theMay-08

IDC/DEVLAN/OG-ATMPage 6developers, it is installed on the Development LAN to be tested in the integrated IDC dataprocessing system for the relevant technology. Such integration testing helps to identify anyunintended effects of the installed software change on other processing software. If sucheffects are found, they are analyzed and resolved before the software change is implementedon the Testbed LAN for operational testing, and finally on the Operations LAN for use inregular IDC operations. Thus the Development LAN has a major quality assurance role in theIDC software development cycle.1.3. Document OverviewChapter 1. provides an introduction to this document.Chapter 2. contains general information, which is common for all processing technologies. Itis organized in sections describing roles and responsibilities, the hardware and infrastructurebaselines, the general directory structure, the role and use of the configuration managementsystem ClearCase at the IDC, and generic change procedures. The information in this chapteris of general interest for all Development LAN users. Chapter 2. is identical for all threevolumes of the Development LAN Operator’s Guide. To support maintainability it isonly contained in [DEVLAN/OG-SHI] and omitted here.Chapter 3. contains all ATM-specific information including the baseline inventory andprocedures for routine operations, system monitoring, and system maintenance andtroubleshooting. The primary audience for this chapter are the operators of the ATMprocessing software. However, the baseline information and troubleshooting procedures mayalso be of interest for developers and testers.1.4. Typographical ConventionsElement Font Exampleapplication, process,italicWorkFlow, Perlprogramming languagedatabase account, database table, UPPERCASE.bold.normal IDCX.interval.intvliddatabase attributeconfiguration parameter, variable,literal value, command, computercode, machine nameCourier New 11 ssh cmss@eldey,$(CMS_CONFIG)1.5. Status of document releaseThis ATM volume is released as a work in progress. It is expected to be updated as moreinformation becomes available and experience is gathered with ATM on the DevelopmentLAN. Specific gaps in available information are marked with [TBD] (to be determined).Information in need of review and possibly update and revision is marked with [TBV] (to beverified). The last section of the document DESIRABLE ENHANCEMENTS contains a to-dolist.May-08

IDC/DEVLAN/OG-RNPage 72. GENERAL BASELINES AND PROCEDURES FOR ALL TECHNOLOGIESThis chapter is identical to the three volumes of the Development LAN Operator’s Guide. Tofacilitate document maintenance it is omitted here. See [IDC/DEVLAN/OG-SHI] to see thecontents.3. ATM PROCESSING3.1. IntroductionThe Atmospheric Transport Modelling (ATM) software uses wind field data, in relation withidentification of specific nuclides at Radionuclide (RN) stations, to derive likely geographicalorigins for potential releases of radionuclides into the earth’s atmosphere. It calculates varioustypes of Source-Receptor-Sensitivity (SRS) matrices, Fields of Regard (FOR) and PossibleSource Regions (PSR), to be visualized, attached to the Reviewed Radionuclide Report(RRR) and further analyzed in connection with the Radionuclide (RN) products.Since its first version the components of the ATM software have been grouped into four socalledlayers. Though currently this division does not play such an important role as before, itprovides a convenient logical framework to presenting the software’s functionality.• Layer 1 is responsible for pre-processing activities: the retrieval of suitablemeteorological input data required for global dispersion modelling. There is a set ofapplications that retrieves data from the European Centre for Medium-Range WeatherForecasts (ECMWF), and a set of applications that retrieves data from the U.S.National Centre for Environmental Prediction (NCEP).• Layer 2 contains the software for SRS modelling, which links hypothetical sourceregions to the likely destinations of radionuclides originating in the source regions bymodelling the dispersion of hypothetically released radionuclides under influence ofthe observed wind fields.• Layer 3 contains an application to convert the SRS matrices to FORs. [TBV]• Layer 4 combines several applications to produce the ATM products and to providethem to all authorized users via the IDC products Web site and to external subscribersvia the subscription subsystem.The ATM software is required to operate both at the IDC and at National Data Centres(NDCs). The current ATM 2.0 system, developed and supported by the Software ApplicationsSection of the IDC, replaces the ATM 1.0 system completely. ATM 2.0 was developed toimprove the portability and maintainability of the software and to bring it in compliance withcurrent IDC software development standards.ATM 2.0 serves as the basis for an in-house Research&Development project at the CTBTOProvisional Technical Secretariat (PTS) with the objective to develop a so-called EnsembleDispersion Modelling (EDM) system [B+07], which will make use of backward modelling.May-08

ComponentLocal::ATM::Acquisition::ScheduleLocal::ATM::Acquisition::SFTPLocal::ATM::Acquisition::MirrorLocal::ATM::ModellingLocal::ATM::Modelling::FLEXPARTLocal::ATM::Modelling::HYSPLITLocal::ATM::VisualisationIDC/DEVLAN/OG-RNPage 9FunctionalityThis component handles acquisition ofremote files that become available on ascheduled basis.Perl module specific to data retrieval usingSFTP protocolThis component handles acquisition ofremote files and can use parts of the remotename to create a local directory and afilename. [TBV]Perl module that handles common modellingtasks.This component handles pre-modelling andpost-modelling tasks which are specific to theFLEXPART model.This component handles pre-modelling andpost-modelling tasks which are specific to theHYSPLIT model.Perl module that handles visualisation tasks,like the production of plots and theproduction of the RRR attachment (plots,web page and link to SRS field data file).The following processing components provide functionality to launch or to support the abovementionedprocessing components:ComponentFunctionalityatm Perl script that is run to launch the ATM 2.0systemLocal::ATMThis component acts as the catch-all for anyuncaught error that occurs in other (called)components so that these can be properlylogged. It also logs the command linearguments given when atm is called.Local::ATM::DispatchPerl module implementing the dispatchmethod, used to determine what major ATMcomponent (acquisition, modelling orvisualisation) to launch and then launch it.Local::ATM::StationThe component encapsulates all informationabout one station and makes this accessiblevia various methods.Local::ATM::SampleThe component encapsulates all informationabout one sample and makes this accessiblevia various methods.Local::ATM::ConstantsThe component provides importableconstants for other components.May-08

IDC/DEVLAN/OG-ATMPage 10ComponentLocal::ATM::DefaultLocal::ATM::PreprocessLocal::ATM::TypeFunctionalityThe component provides functions whichdetermine default parameter values for othercomponents.The component provides functions forconverting raw configuration parametersvalues (strings) to an internal representation(Perl data).The component provides several commonlyused functions which are used for parametervalidation by other components.The following Perl modules constitute the interface components to the logging functionality,database and various types of configuration and data files:ComponentFunctionalityLocal::ATM::ConfigThe component acts as a superclass formany other components and provides aconstructor (new) that will initialize anobject’s attributes from a configurationsource.Local::ATM::Config::SingletonThe component loads configurationfrom configuration files via commandline parametersLocal::ATM::Config::StationThe component provides a constructor(new) that will initialize the object’sattributes from a configuration source.Local::ATM::Config::Station::Singleton The component loads configurationfrom configuration files specified viathe provided configuration attributes.Local::ATM::Config::SampleThe component provides a constructorthat will initialize the object’s attributesfrom a sample parameter source.Local::ATM::Config::Sample::Singleton The component loads configuration fora sample via a default (stationconfiguration) or otherwise configuredsample configuration source.Local::ATM::LogThe component acts as a superclass formany other components and provides anumber of methods to be inherited forlogging.Local::ATM::Log::BaseThe component provides sharedmethods for the logging componentsLocal::ATM::Log::EmailThe component provides logging viaemailLocal::ATM::Log::FileThe component provides logging viafileMay-08

ComponentLocal::ATM::ConfigLocal::ATM::Log::ScreenLocal::ATM::Log::SyslogLocal::ATM::StateLocal::ATM::State::DatabaseIDC/DEVLAN/OG-RNPage 11FunctionalityThe component acts as a superclass formany other components and provides aconstructor (new) that will initialize anobject’s attributes from a configurationsource.The component provides logging viascreen (e.g., terminal).The component provides logging viaSyslog.The component acts as a superclass formany other components and provides anumber of methods to be inherited forstate tracking.This component provides statereporting to a database.A simplified general control flow diagram is displayed below in Figure 2Figure 2. ATM Software Control Flow Diagram.The diagram omits components of supportive nature (Local::ATM::Constants,Local::ATM::Default,Local::ATM::Dispatch::Session,Local::ATM::Preprocess, Local::ATM::Station, Local::ATM::Sample and Local::ATM::Type), as wellas all interfaces (configuration, logging, state information) and all external components.Considering the software division in layers described in the previous section, the mappingbetween actual software modules and the layers is done in the following way:• Layer 1 processing is handled mainly by Local::ATM::Acquisition (andsubcomponents);• Layer 2 and Layer 3 functionality is organized mainly byLocal::ATM::Modelling module (and subcomponents) ; [TBV]• Layer 4 processing is handled by Local::ATM::Visualisation;May-08

IDC/DEVLAN/OG-ATMPage 12Other software components fall outside the layer division or can be used by several layers.All ATM software applications are located below the directory /dvl/software/atm. AllATM specific Perl modules reside in /dvl/software/atm/lib. See section 3.2.5 for moredetailed information on the directory structure.The automatic ATM processing software is run by the UNIX/Linux user atmops. The useratmops is maintained by the Development LAN operator.The automatic part of the software is controlled by cron jobs.Like the other components of the IDC software, the ATM software is subject to configurationmanagement, using the IDC’s configuration management system ClearCase as a tool. TheDevelopment LAN Operator maintains the application software in ClearCase and in theDevelopment LAN runtime system. The ClearCase administrator maintains the IDC softwarebaseline for other LANs and assigns the Development LAN software patch labels. Refer tosection 2.6 of the SHI part of the Operator's Guide for additional information on requestingand implementing changes to the Development LAN application software.The following ClearCase VOB contains the ATM software at the IDC:/vobs/idc/external.In case of the ATM software, only the core applications and all configuration data files areunder ClearCase control, while currently the Perl modules belonging to the ATM softwarepackage reside only in the run-time system. This is expected to change in the near future.3.2.3 Software locationAll ATM software is located below the directory /dvl/software/atm./dvl/software/atm/bin contains symbolic links pointing to the binaries in thesubdirectory /dvl/software/atm/bin/Linux-x86_64-RHEL5/, which is specific to theOperating System RedHat 5.1. The indirection to an OS-specific directory is introduced inorder to support the use of the mechanism for inhomogeneous data processing systems,although the ATM software exists in the version for RH5.1 only. The use of isaexec.sh isdiscussed in [IDC/DEVLAN/OG-SHI] section 3.2.5.2.For more detailed information about the directory structure please check section 3.2.5 of thecurrent document.3.2.4 Software configuration baselineAll configuration files for the ATM processing system are located in the directory/dvl/software/atm/config. More detailed description on the directory content is givenin section 3.2.5.4.May-08

IDC/DEVLAN/OG-RNPage 133.2.5 Development LAN ATM directory structureThe general top-level directory structure for the Development LAN is described in chapter 2. .The ATM-specific top level directories follow this general structure and are listed in the tablebelow. Directory levels are counted from the top level (level 1), which is $(lan)=/dvl forthe Development LAN. In the ATM area, the subdirectory depth goes to 8.3.2.5.1 Top level ATM directories$(lan)/software/atm$(lan)/logs/atm$(lan)/products/atm$(lan)/data/atmTop level (Level 3) ATM directoriesAll ATM processing software.ATM log file areaATM data productsATM data3.2.5.2 atm directoryLevel 4: ATM data processing directories$(lan)/software/atmAll ATM processing software../binATM binaries and scripts [TBV]./configATM processing configuration files./libATM Perl modules3.2.5.3 bin directoryLevel 5: ATM executables/dvl/software/atm/binContains symbolic links to ATM binaries andscripts [TBV] of the corresponding OS../Linux-x86_64-RHEL5/ Contains executables for RedHat 5.13.2.5.4 config directory$(lan)/software/atm/config/ is the upper level directory containing ATM, systemand station parameters. It contains the following sub-directories:Level 5-7: ATM processing parameter configuration$(lan)/software/atm/configContains ATM, logging, database and stationparameters../atm/Root directory for ATM configuration files./atm/acquisitionConfiguration files relevant to dataacquisition./atm/logConfiguration files relevant to variouslogging functionality./atm/modelling/flexpartFLEXPART configuration./atm/modelling/hysplitHYSPLIT configuration./atm/stateConfiguration files for the software thathandles state reporting for the entire ATMsystemMay-08

IDC/DEVLAN/OG-ATMPage 14Level 5-7: ATM processing parameter configuration./atm/visualisationVisualisation software./static/Processing state reporting configuration forthe entire ATM system./stations/Station specific parameters./system/System specific parameters3.2.5.5 lib directoryLevel 5-8: ATM Perl modules$(lan)/software/atm/libContains all Perl modules relevant to theATM processing../perlContains Local subdirectory./LocalRoot directory for Perl modules./Local/ATMRoot directory for Perl modules specific toATM processing./Local/ATM/AcquisitionPerl modules specific to data acquisition./Local/ATM/ConfigPerl modules specific to./Local/ATM/Config/SamplePerl modules specific to sampleconfiguration./Local/ATM/Config/StationPerl modules specific to station configuration./Local/ATM/DispatchPerl modules specific to dispatch [TBV]./Local/ATM/LogPerl modules specific to logging activities./Local/ATM/ModellingPerl modules specific to ATM modeling./Local/ATM/StatePerl modules specific to processing statereporting3.2.6 Development LAN ATM user environment3.2.6.1 User environment for automatic processingThe following environment variables have to be set in order to operate the automatic ATMprocessing software: [TBV]GRIBLIB=TBDMETLIBHOME=TBDECMWF_LOCAL_TABLE_PATH=\/dvl/software/site/.rhel5/usr/lib64/gribtables/LOCAL_DEFINITION_TEMPLATES=\/dvl/software/site/.rhel5/usr/lib64/gribtemplates/GRIB_DEFINITION_PATH=\/dvl/software/site/.rhel5/usr/share/grib_api/definitionsGRIB_TEMPLATES_PATH=\/dvl/software/site/.rhel5/usr/share/grib_api/templatesMay-08

3.2.6.2 User environment for interactive processingIDC/DEVLAN/OG-RNPage 15There is a standard ATM user environment based on csh, which should be used by all userswho want to develop, test or use any ATM software on the Development LAN. Theenvironment variables are defined in the file .cshrc in the home directory of the atmopsuser. These variables are automatically read and used by the ATM system when run via cronthe next time.The relevant environment variables are presented below in alphabetical order with their valueon the Development LAN, information in which software layers they are used and a shortdescription: [TBV]Variable Value Layers DescriptionATMMODHOME /dvl/software/atm/\atm-layer2/SRSM-MODEL3, 4 home of the currentdispersion model softwareATMOPERATOR "atmops@idc.ctbto.org" 1, 2, 4 development LAN specific,used where possible insteadof hard-coded emailaddresses in scriptsECMWFDATAHOME /dvl/software/atm/\atm-layer1/ECMWFDATA1, 2 home of the ECMWFretrieval software layer 1GRIBLIB /dvl/software/atm/\atm-layer2/SRSM-MODEL/\emosdir_0002201, 2, 3 home for emos, required forbuilding softwareMETLIBHOME/dvl/software/atm/\atm-layer2/SRSM-MODEL/\metlib1, 2, 3 home for metlib, required forbuilding softwareNCARG_ROOT /usr/local/ncarg 4 home of the NCAR GraphicssoftwareNCEPDATAHOME /dvl/software/atm/\atm-layer1NCEPDATA1, 2 home of the NCEP retrievalsoftware in layer 1PATH $PATH:$NCARG_ROOT/bin 4 the binary directory for theNCAR Graphics software isincluded in the PATHPLOTFORHOME /dvl/software/atm/\atm-layer4/plotforPOSTPROCHOME /dvl/software/atm/\atm-layer34 home for the plottingsoftware in layer 43, 4 home for the post-processingsoftware in layer 33.2.7 ATM data acquisitionATM data acquisition works by regularly checking availability and by pulling the most recentwind-field data off specified, publicly available URLs at two meteorological institutions,ECMWF and NCEP.The csh script get-ecmwf.csh launches the data acquisition; it is under control of the crondemon.Wind field data is retrieved as 3- or 6-hourly snapshots. Retrieval takes place at configuredintervals based on crontab entries and parameter files depending on data availability. ForMay-08

IDC/DEVLAN/OG-ATMPage 16example NCEP GDAS consists of 4 data files per day (00, 06, 12, 18 hours analysedsnapshots).3.2.8 ATM automatic processingLayer 2 software (modelling) is run automatically under control of cron. It. consists of twoparticle trajectory models (FLEXPART_5.1 [SS02]; HYSPLIT_4.7 [DSRT05]). Output of themodelling is sample specific SRS fields. Modelling takes place backwards over 14 or 21 dayswith 8, 12 or 24 hours of releases.Layer 3 software (result conversion) is run automatically under control of cron. It providespost-processing of modelling results, converting SRS fields into several Fields of Regard(application SRS2FOR, processing part) and PSR products.Layer 4 software (visualization) is partly run automatically under control of cron and partlyinteractively on demand. The automatic part performs visualization of the layer 3 output inbatch mode (SRS2FOR, plotting part).3.2.9 ATM interactive processingVisualization can be performed interactively on demand using the application WEB-GRAPE.The WEB-GRAPE stands for Web-Connected Graphics Engine. This software has beendeveloped by the PTS as an end-user tool to interpret the SRS fields provided by the IDC.This software can be used both to calculate and to interpret graphically the FOR and PSRproducts. It is not routinely used on the Development LAN.For more details please refer to WEB-GRAPE documentation. [TBD]3.2.10 ATM post-analysis processingNot applicable to ATM.May-08

IDC/DEVLAN/OG-RNPage 173.3. Operations ProceduresUntil now the Development LAN operators have not gathered much experience with the ATMsoftware. This section will be enhanced and enlarged as more experience is gathered3.3.1 Starting the ATM processing systemThe ATM processing system is controlled by the cron jobs listed below. If one or several ofthese have been disabled and the corresponding process is thus not being run, it can beactivated by removing the comment sign (crochet #) from the respective line in the crontabfile.# Acquisition Schedule ECMWF05,20,35,50 * * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/acquisition/schedule/ecmwf/\ecmwf.par atm_session_id=1# Modelling FLEXPART ECMWF00,30 08-13 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/modelling/flexpart/ecmwf/\ecmwf.par atm_session_id=11 \atm_modelling_stations=VIP00,ARP01,ARP02,ARP03,AUP04,\AUP05.AUP06,AUP07,AUP08,AUP09,AUP10,BRP11,BRP12,CMP1300,30 08-13 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/modelling/flexpart/ecmwf/\ecmwf.par atm_session_id=12 \atm_modelling_stations=CAP14,CAP15,CAP16,CAP17,CLP18,\CLP19,CNP20,CNP21,CNP22,CKP23,ECP24,ETP25,FJP26,FRP2700,30 08-13 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/modelling/flexpart/ecmwf/\ecmwf.par atm_session_id=13 \atm_modelling_stations=FRP28,FRP29,FRP30,FRP31,FRP32,\DEP33,ISP34,IRP36,JPP37,JPP38,KIP39,KWP40,LYP4100,30 08-13 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/modelling/flexpart/ecmwf/\ecmwf.par atm_session_id=14 \atm_modelling_stations=MYP42,MRP43,MXP44,MNP45,NZP47,\NEP48,NOP49,PAP50,PGP51,PHP52,PTP53,RUP54,RUP5500,30 08-13 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/modelling/flexpart/ecmwf/\ecmwf.par atm_session_id=15 \atm_modelling_stations=RUP56,RUP57,RUP58,RUP59,RUP60,\RUP61,ZAP62,SEP63,TZP64,THP65,GBP66,GBP67,GBP6800,30 08-13 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/modelling/flexpart/ecmwf/\May-08

IDC/DEVLAN/OG-ATMPage 18ecmwf.par atm_session_id=16 \atm_modelling_stations=USP70,USP71,USP72,USP73,USP74,\USP75,USP76,USP77,USP78,USP79# Visualisation FLEXPART00,30 18-23 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/visualisation/flexpart/\flexpart.par atm_session_id=11 \atm_modelling_stations=VIP00,ARP01,ARP02,ARP03,AUP04,\AUP05,AUP06,AUP07,AUP08,AUP09,AUP10,BRP11,BRP12,CMP1300,30 18-23 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/visualisation/flexpart/\flexpart.par atm_session_id=12 \atm_modelling_stations=CAP14,CAP15,CAP16,CAP17,CLP18,\CLP19,CNP20,CNP21,CNP22,CKP23,ECP24,ETP25,FJP26,FRP2700,30 18-23 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/visualisation/flexpart/\flexpart.par atm_session_id=13 \atm_modelling_stations=FRP28,FRP29,FRP30,FRP31,FRP32,\DEP33,ISP34,IRP36,JPP37,JPP38,KIP39,KWP40,LYP4100,30 18-23 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/visualisation/flexpart/\flexpart.par atm_session_id=14 \atm_modelling_stations=RUP56,RUP57,RUP58,RUP59,RUP60,\RUP61,ZAP62,SEP63,TZP64,THP65,GBP66,GBP67,GBP6800,30 18-23 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/visualisation/flexpart/\flexpart.par atm_session_id=15 \atm_modelling_stations=RUP56,RUP57,RUP58,RUP59,RUP60,\RUP61,ZAP62,SEP63,TZP64,THP65,GBP66,GBP67,GBP6800,30 18-23 * * * /dvl/software/atm/bin/atm \par=/dvl/software/atm/config/atm/visualisation/flexpart/\flexpart.par atm_session_id=16 \atm_modelling_stations=USP70,USP71,USP72,USP73,USP74,\USP75,USP76,USP77,USP78,USP793.3.2 Shut down of the ATM processing systemIn order to disable the automatic processing it is sufficient to comment out the relevantcrontab entries on the processing machines.May-08

IDC/DEVLAN/OG-RNPage 193.4. Monitoring ProceduresUntil now the Development LAN operators have not gathered much experience with the ATMsoftware. This section will be enhanced and enlarged as more experience is gathered3.4.1 Routine monitoring tasksThe ATM processing software is able to report processing errors via various methods. Theseare: log file, Syslog, screen (interactive in case of interactive processing) and email. Alldebug, info, notice, warning and error messages are extensively being reported to allconfigured destinations depending on the configured reporting minimum and maximumlevels.3.4.2 Layer 1 monitoringLayer 1 software is responsible for data retrieval. So the main monitoring task consists inchecking that configured wind field data is successfully being retrieved. This is done byregularly observing the contents of the /dvl/data/atm directory for all configuredacquisitions. Also WorkFlow can be used to spot processing failures.3.4.3 Layer 2 monitoringMonitor that all configured modelling is running [TBD, how] and producing SRS products in/dvl/products/atm. WorkFlow also provides useful information for this software layer.3.4.4 Layer 3 monitoring[TBD]3.4.5 Layer 4 monitoringMonitor that all configured visualizations are running and produce RRR attachments to theReviewed Radionuclide Report (RRR) webpage using the SRS data products from layer 2.Also, the /dvl/products/atm directory content should contain relevant products[examples, TBD]. WorkFlow provides a graphical representation of the level 4 processingstatus.3.4.6 Log filesATM software has various logging capabilities. This includes notifications via e-mail, onscreenoutput, logging information into a log file and Syslog.For logging, Local::ATM::Log class plays the central role. All other classes wishing to dologging use this class as a super class. Logging properties are configured in the ATM parfiles. The default global log configuration is inconfig/log/log.par,config/log/file/file.par,config/log/syslog/syslog.par,config/log/email/email.par,andconfig/log/screen/screen.par.3.4.7 Maintenance proceduresAt the time of writing no maintenance procedures exist. It is not known if there are anyrecurrent tasks that require maintenance procedures.May-08

IDC/DEVLAN/OG-ATMPage 203.5. Troubleshooting ProceduresUntil now the Development LAN operators have not gathered much experience with the ATMsoftware. This section will be enhanced and enlarged as more experience is gathered3.5.1 Interpreting log and error messages• The logging area for all ATM processing applications is /dvl/logs/atm. Someapplications write log files into subdirectories there, while others send status or errormessages by email to the configured ATM operator address instead of writing to log files.cron jobs automatically remove log files after a certain delay, usually 14 days.• Some applications use log directories, which are separate from the common log directorydescribed above. Most of these applications have their own log file management. Theselogs can be found either in the data directory or in the software directory of theapplication.• Routine checks and monitoring:Check if the log files expected for each application exist for the current date and also forthe previous few days. A missing daily log file usually indicates that something waswrong with the corresponding run of the application.Check if the partitions (logs and software) where the log files reside have sufficient freespace for the next daily log files. Under normal conditions the log files are automaticallypurged, such that the total used space on these partitions does not grow. However, a fulllog or software partition will typically lead to widespread logging problems in the entireprocessing system.To find the log entries for a particular processing interval, see the monitoring section ofthis document. The log files are normal text files and can be read using any text editor orviewer.• Interpreting error messages and resolving problems:The specific log and error messages vary for each application. Refer to the ATMdocumentation for the application or subsystem to find more application-specificinformation.Error messages may indicate configuration problems, e.g. if a variable is not found orcannot be read. In this case check the application configuration file and correct theconfiguration.Error messages may also indicate the unavailability of system resources, e.g. if the fileserver is unavailable, directories or files cannot be accessed, the network connection isunavailable, etc. Such problems may be transient, i.e. they will vanish after a short time,or they may be persistent. Manually check the availability of the relevant systemresources. For example, if a file cannot be read check if the file exists and whatpermissions are set:ls -l /All applications are run by atmops, thus this user needs to have the relevantread/write/execute permissions. If a persistent problem with accessing system resources isdetermined contact the system administrator to diagnose the specific system problem.May-08

IDC/DEVLAN/OG-RNPage 21Error messages may indicate the inability of an application to process specific data. In thiscase the problem can be data-related (e.g. corrupt data file) or software-related. To furtherdetermine the nature of the problem try to reproduce it for different data and determine thespecific features of the data that may trigger the problem. If the problem is clearly datarelated,the problem and the relevant data should be logged and reported for furtherassessment. If the problem is related to a software bug report the software problem. If theproblem can be related to specific features, which may routinely occur in regular dataintervals, report an enhancement request.In addition to the application log files the Syslog files on the central file store may behelpful in specific cases. For example the cron log file can be of particular interest todiscover cron-related problems. These files are maintained by the system administratorand are not further described in this document.3.5.2 Solving common problemsAt the time of writing no common problems relevant to the ATM software have beenobserved in the Development LAN. [TBD]3.5.3 Handling unknown problemsFor individual failed processing runs, analyse the log files for the particular days to determinethe type of the problem. Try re-processing the interval if meteorological data is still available.There is a chance that the problem was of a transient nature.Check the scripts that were called by cron, and try to execute the relevant commands fromthese on the command line as the atmops user.The ATM scripts can also be run in debug mode with verbose output on what commands areexactly executed. This way it may be easier to discover where the problem occurs. This isdone by executing the scripts through the shell with the –x argument, i.e. for a C Shell script:csh –x ${scriptname}.cshMay-08

IDC/DEVLAN/OG-ATMPage 22TERMINOLOGYGlossaryClearCase branchClearCase labelClearCaseversion treeClearCase viewClearCase VOBcron jobdataDevelopmentLANA set of sequentially numbered versions within the version tree of a ClearCaseobject. All version trees have a main branch and can have additional brancheswith unique branch names.A label that is applied to an individual version of a ClearCase object. Variouslabels are used for the Development LAN to define versions, which are part ofa software baseline or patch as well as to define versions that are currentlyinstalled in the runtime system.The set of all versions of a ClearCase object (i.e. a directory or file), which arestored in ClearCase. The version tree starts with version 0 on the main branchand can have additional branches each holding sequentially numbered versions.At the IDC the ClearCase version tree holds the history of files and directorieson all IDC LANs as well as all development versions. A ClearCase versionstring consisting of branch names and the version number uniquely defineseach version in the version tree.A configured set of rules in ClearCase to select unique versions from allversion trees of objects in a given VOB. Individual views are identified byview names. The views used for the Development LAN select versions basedon their branch names or on specific labels.A logical area in ClearCase to manage a set of ClearCase objects including alltheir versions. The IDC ClearCase VOBs contain high-level system directoriesincluding all their subdirectories, files and links. Each ClearCase VOBcorresponds to a directory on the runtime system.Cron is a utility of the UNIX/Linux operating systems to automatically runindividual commands, scripts or applications at configurable regular times. AllIDC scripts and applications, which are not controlled by the DACS or byother scripts or applications, are controlled by cron. Individual cron jobs canbe configured in the local crontab file for individual authorised users oneach machine.Files, which are written by the IDC processing system, are considered data ifthey are re-used by the system for further processing. Otherwise they areconsidered products. This categorisation is used in the Development LANdirectory structure. Data and product files are not under ClearCase versioncontrol.The computer hardware and infrastructure used to integrate and test softwareand configuration changes as well as new software at the IDC beforepromoting the changes to the Testbed LAN for operational testing. Physicallyseparate from the Operations and Testbed LANs and under less rigorousconfiguration control.May-08

IDC/DEVLAN/OG-RNPage 23devlan branchdevlan_rn_viewdevlan_solaris_viewThe devlan_view is configured to automatically create a new version on thedevlan branch if a directory or file is modified and checked-in in thedevlan_view. The latest sequential version on the devlan branch isalways the latest version that has been created under the devlan_view. Ifthe version tree of a ClearCase object does not have a devlan branch thisClearCase object has never been modified and checked-in in thedevlan_view.The devlan_rn_view is the ClearCase view to be used to modify andcheck-in versions of Radionuclide software-related ClearCase objects for theDevelopment LAN. It selects versions based on the same branch names as thestandard devlan_view. However, the devlan_rn_view enables users tocheck-out and check-in versions in the Radionuclide software ClearCase VOB.The devlan_solaris_view is exclusively used to build Linux compatiblesoftware on Solaris platforms for backwards compatibility reasons. It isconfigured to check-in versions on the devlan_solaris branch. Sincesource code is uniformly maintained on the devlan_branch for both Solarisand Linux platforms the view will select the latest source code version on thedevlan branch if this branch exists in the version tree of the ClearCaseobject. If no devlan branch exists for a particular objet the view will use thesame selection rules as the standard devlan_view. Only ClearCase objectswhich are installed via the Solaris build procedure in this view are supposed tohave a devlan_solaris branch.devlan_view The devlan_view is the standard ClearCase view to be used to modify andcheck-in versions of ClearCase objects for the Development LAN. It isconfigured to select versions based on their branch names. It will select thelatest version on the devlan branch if this branch exists in the version tree ofthe ClearCase object. It will select the latest version on the R3_tst branch ifthe devlan branch does not exist. If neither a devlan branch nor anR3_tst branch exist in a given version tree the devlan_view will selectthe latest version of the ClearCase object on the main branch. This latter caseoccurs for files or directories that have never been modified either on theDevelopment LAN or on the Testbed.DEV_LAN labelDEV_LAN viewOperations LANThe DEV_LAN label defines ClearCase versions, which are currently installedin the Development LAN runtime system. It can only be applied to a singleunique version in the version tree of each ClearCase object. The DEV_LANlabel is manually applied when a new version is promoted from ClearCase tothe Development LAN runtime system.The DEV_LAN view is configured to select only versions, which have theDEV_LAN label applied. If the DEV_LAN labels are correctly set the DEV_LANview will show exactly the same versions as the Development LAN runtimesystem directories.The computer hardware and infrastructure used for operational data processingat the IDC.May-08

IDC/DEVLAN/OG-ATMPage 24productsruntime systemTestbed LANField of RegardPossible SourceRegionSource ReceptorSensitivity Fields(SRS fields)Source ReceptorMatrixFiles, which are written by the IDC processing system, are considered productsif they are not further processed and are made available to (external) users.Otherwise they are considered data. This categorisation is used in theDevelopment LAN directory structure. Data and product files are not underClearCase version control.The directories and files, which are mounted on the machines of an IDC LANand which are used to operate the IDC processing software. All directories andfiles in the runtime system, which are under version control, have a counterpart in ClearCase. The Development LAN runtime versions correspond toClearCase versions that have the DEV_LAN label applied. Data files, productsand log files are not under version control and exist only on the runtimesystem.The computer hardware and infrastructure used to test new stations andsoftware changes in an operational environment before installation in theOperations LAN. A close copy but physically separate and independent fromthe Operations LAN.Sample specific ATM Product attached to the RRR. The different FORs(binary, quantitative, differential, integral) describe a set of predefined viewson the SRS fields. For a detailed definition see [BWD04, Annex II].Measurement Event (Scenario) specific ATM Product. For the PSR generationan inversion problem is solved whereby source hypotheses are tested across theglobe and a certain period of interest. The inversion algorithm combines thoseSRS fields relevant to the source hypothesis formulated and returns a timedependent global distribution of correlation coefficients.Sample specific field that contains the time dependent sensitivity of themeasurement to potential sources in the measurement stations environmentduring and n days prior to the collection stop time. n is currently 14 (butplanned to become 21 for full scale ATM operations). As the sample specificsensitivity fields are stored within a standard 3-hourly-time stagger, they canbe combined to a source receptor matrix valid and daily updated for the wholeIMS RN network.Event specific matrix consisting of those source receptor sensitivities that areassembled for the IMS RN network according to the actual source hypothesesformulated. This event specific matrix constitutes the RN analogue to the"ground truth" information assembled for the location of seismic events.May-08

IDC/DEVLAN/OG-RNPage 25AbbreviationsATMCCBCDSCICINCPANCSCICRPCTBTODACSDBIDCRDev LANECMWFEDMFORFTPIDCIMSLANLEBNCARNCEPNDCPSRPTSREBRNRRRSELSFTPSHISISRSAtmospheric Transport ModellingConfiguration Control BoardContinuous Data SubsystemComputer Infrastructure sectionChange Implementation NoteComprehensive Perl Archive NetworkComputer Software Configuration ItemChange Request ProposalComprehensive Nuclear-Test-Ban Treaty OrganisationDistributed Application Control SubsystemDatabase InterfaceDevelopment LAN Change RequestDevelopment LANEuropean Centre for Medium-Range Weather ForecastsEnsemble Dispersion ModellingField of RegardFile Transfer ProtocolInternational Data CentreInternational Monitoring SystemLocal Area NetworkLate Event BulletinNational Center for Atmospheric ResearchU.S. National Centers for Environmental PredictionNational Data CentrePossible Source RegionProvisional Technical SecretariatReviewed Event BulletinRadio-Nuclide technologyReviewed Radionuclide ReportStandard Event ListSecure File Transfer ProtocolSeismic, Hydro-acoustic and Infrasound technologiesSoftware Integration unitStandardized Source-Receptor SensitivityMay-08

IDC/DEVLAN/OG-ATMPage 26TBDTBVVOBWEB-GRAPEWMOTo be determined, to be doneTo be verifiedVersioned Object BaseWeb-Connected Graphics EngineWorld Meteorological OrganizationMay-08

IDC/DEVLAN/OG-RNPage 27REFERENCES[ATM-batchmodeATM_1.0-SIP] batchmodeATM 1.0 Software Installation Plan.[ATM-NCEPDATA_V2.0] ATM NCEPDATA 2.0 Software User Guide.[ATM-plot-FOR_2.0] ATM plot-FOR 2.0 Software User Guide.[ATM-SRSM_1.0] ATM SRSM-Model 1.0 Software User Guide.[B+07] Andreas Becker et al., Global backtracking of anthropogenic radionuclides by means of areceptor oriented ensemble dispersion modelling system in support…, AtmosphericEnvironment, 2007, doi:10.1016/j.atmosenv.2006.12.048.[BWD04] Andreas Becker, Gerhard Wotawa and Lars-Erik De Geer, Review on New PTS modellingcapabilities supporting the emerging CTBTO-WMO response system including a proposal forstandardised model intercomparison. World Meteorological Organization, Geneva,Switzerland, March 2004, CBS/ERA-CG/INF.1/Doc.8.3 (8.III.2004),http://www.wmo.ch/web/www/ERA/Meetings/ERACG-Geneva2004/Doc8-3.pdf.[DSRT05] Roland Draxler, Barbara Stunder, Glenn Rolph, and Albion Taylor, HYSPLIT4 User'sGuide, Version 4.7, Air Resources Laboratory, National Oceanic & AtmosphericAdministration, 1 November 2005,http://www.arl.noaa.gov/data/web/models/hysplit4/win95/user_guide.pdf.[Edw03] Tryggvi Edwald, Using syslog at CTBTO, Version 0.9, International Data Centre,Provisional Technical Secretariat, CTBTO, Vienna, Austria, 30 May 2003.[IDC/DEVLAN/OG-RN] Development LAN Operator’s Guide -RN[IDC/DEVLAN/OG-SHI] Development LAN Operator’s Guide - SHI[IDC-CCB] IDC CCB Procedures, http://idc030.idc.ctbto.org:8000/Docs/CCB-Documentation.html.[IDC-OM] IDC Operational Manual, CTBT/WGB/TL[IDC-SCD] Software Checklist Development LAN, Version 2.1, IDC/CI, 2004[Pre02b] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization, IDCSoftware Documentation Framework, International Data Centre, Provisional TechnicalSecretariat, CTBTO, Vienna, Austria, 29 March 2002.[Pre02c] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,CTBTO Editorial Manual, Conferences Services Section, Division of Administration,Provisional Technical Secretariat, CTBTO, Vienna, Austria, 10 April 2002.[Pre03] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization, NewATM software layer 2: Software package SRSM-Model Version 1.0, International DataCentre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 10 February 2003,http://kuredu.ops.ctbto.org/librarybox/idcdoc/downloads/SRSM-MODEL_1_0.pdf.[Pre04a] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization, NewATM software layer 1: Software package NCEPDATA Version 2.0, International Data Centre,Provisional Technical Secretariat, CTBTO, Vienna, Austria, 16 April 2004,http://kuredu.ops.ctbto.org/librarybox/idcdoc/downloads/NCEPDATA_2_0.pdf.[Pre05b] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,New ATM software layer 4: Software package plot-FOR Version 2.1, International DataCentre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 8 November 2005,May-08

IDC/DEVLAN/OG-ATMPage 28http://kuredu.ops.ctbto.org/librarybox/idcdoc/downloads/PLOTFOR_2_1.pdf.[Pre07b] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,ECMWFDATA V2.0 on demand Software User Tutorial, CTBT/ECMWFDATA V2.0/SUT,International Data Centre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 27January 2007,J:\idc projects\DOCUMENTS\Radion_Devel\ATM\layer1\ECMWFDATA_2.0\SUTondemand.doc.[Pre07c] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,ECMWFDATA V2.0 ops Software User Tutorial, International Data Centre, ProvisionalTechnical Secretariat, CTBTO, Vienna, Austria, 27 January 2007,J:\idc projects\DOCUMENTS\Radion_Devel\ATM\layer1\ECMWFDATA_2.0\SUT_ops.doc.[Pre07e] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,WEBGRAPE_1.1.2beta Software Installation Plan, IDC/WEBGRAPE_1.1.2_beta/SIP,International Data Centre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 22February 2007,J:\idc projects\DOCUMENTS\Radion_Devel\ATM\layer4\WEB_GRAPE\WEBGRAPE_1.1.2beta\WEBGRAPE_1.1.2beta\docs\WEBGRAPE_SIP_20070222.pdf.[SS02] Andreas Stohl and Petra Seibert, The FLEXPART Particle Dispersion Model – Version 5.0 –User Guide, 9 August 2002, pp. 48–64.[W+03] Gerhard Wotawa et al., Atmospheric transport modelling in support of CTBT verification –overview and basic concepts, Atmospheric Environment, 37 (18), pp. 2529–2537.May-08

IDC/DEVLAN/OG-RNPage 294. DESIRABLE ENHANCEMENTS (TODO-LIST)1. Add references to Web-Grape documentation.2. Add WorkFlow snapshots and WorkFlow configuration information for the foursoftware layers to chapter 3.4.5.3. Add descriptions of the cron jobs, chapter 3.3.1.4. Add information on ATM software maintenance, chapter 3.4.7.5. Add examples of troubleshooting scenarios, chapter 3.5.1.6. Write section 3.4.47. Insert log file examples in section 3.5.1May-08