CERES Data Management Overview and Status - NASA
CERES Data Management Overview and Status - NASA
CERES Data Management Overview and Status - NASA
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>CERES</strong> <strong>Data</strong> <strong>Management</strong> ActivityPresented to <strong>CERES</strong> Science Team MeetingSeptember 16, 2010Jonathan GleasonJonathan.l.gleason@nasa.govThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Topics To Be Covered• DMT <strong>Overview</strong>• Edition 2 <strong>Status</strong> Update• Edition 3 Approach <strong>and</strong> Schedule• NPP Update• Optimization Efforts• DMT/ASDC Process Improvement• Scheduling ActivitiesThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
<strong>Data</strong> <strong>Management</strong>Team MembersInstrument:Denise CooperThomas GrepiotisBrain MagillRichard SpivakMark TimcoeNelson HillyerERBElike:Dale WalikainenJeremy L<strong>and</strong>eConvolution:Walter MillerIgor AntropovTISA Gridding:Raja RajuRon ParrishTISA Averaging:Cathy NguyenBetty LockClouds:Sunny Sun-MackRicky BrownYan ChenLiz HeckertRita SmithDianne SnyderProduction /Optimization:Lisa ColemanCarla GruneInversion:Victor SothcottSSAI:Lee BoddenSARB:Tom CaldwellFM5:Jim ClossConfiguration<strong>Management</strong>:Tammy AyersJoanne SaundersThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
<strong>CERES</strong> OrganizationScience• Derives & refinesalgorithms• Validates algorithms• Validates <strong>CERES</strong> datasets• Writes Quality Summary<strong>Data</strong><strong>Management</strong>Team(DMT)• Implements algorithms• Maintains software• Verifies data• Assists in validation• Provides CM <strong>and</strong>documentation supportAtmosphericSciences<strong>Data</strong> Center(ASDC)• Ingests data• Places operationalsoftware in production• Produces data sets• Distributes data sets• Archives data• Provides UserServicesAlgorithm Development Algorithm Implementation <strong>Data</strong> ProcessingThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
<strong>CERES</strong> Subsystems• <strong>CERES</strong> is made up of 7 Working Groups- Instrument - SOFA- ERBElike - SARB- Clouds - TISA- Inversion or ADM• Code organized into 12 Subsystems- Each subsystem tied to 1 or more working groups• Each Subsystem made up of 1 or moreProduct Generation Executives (PGEs)- Currently there are 75 active PGEsThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
<strong>Data</strong> from other Instruments used by <strong>CERES</strong>• <strong>CERES</strong> Instrument/ERBElike only subsystems that canprocess when only <strong>CERES</strong> data available.• <strong>CERES</strong> directly uses the following MODIS data sets:• MYD02SS1/MOD02SS1* (19 channel radiance subset of everyother pixel every other scanline)• MYD03/MOD03* (geolocation)• MYD04_L2/MOD04_L2 (5 min 10 km aerosol swath)• MYD08_D3/ MOD08_D3 (daily 1 deg aerosol)• Critical data sets; must have matched pairs to process.• Additionally <strong>CERES</strong> uses, Geostationary satellite data:• MET-5, MET-6, MET-7, MET-8, MET-9• GOES-8, GOES-9, GOES-10, GOES-11, GOES-12• GMS-5, MTSAT-1RThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
<strong>CERES</strong> Processing SoftwareSubsystemNumberSubsystemNameLOC(to nearest 1K)Publicly Available<strong>Data</strong> ProductsProduct FrequencyComments1 Instrument/Pre-Processor<strong>CERES</strong>lib 133K All Satellites4KThe Science Directorate at <strong>NASA</strong>’s Langley Research CenterNPP only1 Instrument 306K BDS 1/day All Satellites2 ERBE-like/ Inversion 30K ES-8 1/day All Satellites3 ERBE-like/ TSA 12K ES-9, ES-4 1/month All Satellites12 MOA 15K Run monthly4.1 – 4.4 Clouds 359K All Satellites4.5 – 4.6 Inversion 227K SSF 1/hour All Satellites5 SARB 164K CRS 1/hour All Satellites6 & 9 TISA-Gridding 60K FSW, SFC,ISCCP-D2like-Day/Nit60/month, 36/month,1/monthAll Satellites11 GGEO 172K ISCCP-D2like-GEO 1/month Geostationary7.2 Synoptic SARB 47K All Satellites7.1 & 810TISA-Averaging 249K SYN, AVG, ZAVGSRBAVGTOTAL LOC 1,778K1/day, 1/month, 1/month5/monthAll Satellites
Edition 2 <strong>Data</strong> SetsProduct Edition 2 AvailablethroughBDS, ES8, ES9,ES4SSFSFCSRBAVGCRSFSWSYN, AVG, ZAVGEdition2 (Terra,Aqua)Edition2G (Terra)Edition2D (Aqua)Edition2G (Terra)Edition2D (Aqua)Edition2D (Terra)Edition2A (Aqua)Edition2G (Terra)Edition2C (Aqua)Edition2G (Terra)Edition2C (Aqua)Edition2C (Terra)Edition2B (Aqua)Feb ‘10Feb ‘10Sept ‘09Feb ‘10Aug ’09Oct ’05Dec ’09Dec ’07Nov ’09Dec ‘07Oct ’05CommentsGains DeliveredThrough 7/1/2010Aqua processingongoingNeed GGEOCoefficientsNeed GGEOCoefficientsThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Edition 3 ProcessingApproachEdition 3BDS/IESES8 ES4 ES9Edition 2SSFSSFSFCEdition 2MOAEdition 2GGEOTSI(Terra + Aqua)Single InstrumentXtrk InstrumentsTerra And AquaMATCHSYNI(Terra + Aqua)SYN1deg-3HourSYN1deg-MonthSYN1deg-M3HourInput Product(Terra + Aqua)The Science Directorate at <strong>NASA</strong>’s Langley Research Center
Edition 3 Production Schedule2010 2011Sept Oct Nov Dec Jan Feb Mar April May JuneTerra SSFTerra SFCAqua SSF9/39/209/305/3010/4 11/1 12/6 1/10 2/7 3/7 4/4 5/2 5/30 6/134/18Aqua SFCTSISYNI10/110/1810/2411/15 12/20 1/24 2/21 3/2111/15 12/8H O L I D A Y S11/5 12/10 1/14 2/11 3/111/9 2/2 3/2 3/304/18 5/16SYN1deg10/2912/151/6 1/13 2/8 3/8 4/5GGEO Ed29/3010/18 11/5 12/3The Science Directorate at <strong>NASA</strong>’s Langley Research Center
Relevant NPP Issues• Imager input data required for Climate <strong>Data</strong> Record(CDR) must be of climate quality <strong>and</strong> consistentlycalibrated over entire period.– In NPP era, L<strong>and</strong> PEATE provides <strong>CERES</strong> aggregated radiance<strong>and</strong> geolocation files <strong>and</strong> sub-sampled data files using <strong>CERES</strong>provided code. L<strong>and</strong> PEATE also provides AOT files thatcorrespond to sub-sampled radiance/geolocation.– For Terra/Aqua, MODAPS provides radiance, geolocation, <strong>and</strong>aerosol files from a collection that begins at covers open.• NPP <strong>CERES</strong> made use of already existinginterfaces.– Cost savings by using existing infrastructure.– L<strong>and</strong> PEATE already getting VIIRS data. Agreed to also obtain<strong>CERES</strong> RDRs.– Network between L<strong>and</strong> PEATE <strong>and</strong> ASDC exists for Terra/Aqua.The Science Directorate at <strong>NASA</strong>’s Langley Research Center
FM5 Code Development• Instrument only subsystem modified– Preprocessor will convert data to format consistentwith Terra <strong>and</strong> Aqua– Instrument subsystem currently implemented withAda– Convert Ada to C++ <strong>and</strong> deliver preprocessor for Adaas schedule risk mitigation• Seven total code deliveries– Delivered 11 Ada PGEs (Risk Mitigation)– Delivered 5 C++ PGEs (will replace Ada)• VIIRs subsetter updates delivered to L<strong>and</strong> PEATE<strong>and</strong> test output from L<strong>and</strong> PEATE verifiedThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Process Optimization Effort• Proposed in 2009 <strong>CERES</strong> Senior Review– Migrate off SGI to AMI (DPO)– Optimize <strong>CERES</strong> production software– Communicate Production Requests• Production Request <strong>Data</strong>base– Automate PRs (Currently via .pdf)– Run scripts can read required input– Online for real time review by subsystem teamsThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Computing Platform MigrationWarlock (SGI) Magneto (P4) AMI-P (P6 & x86)• Instrument• ERBElike• Clouds/Inversion (Ed2Terra)• Clouds/Inversion Ed2(Aqua)• Inversion Ed3• TISA Gridding Ed2 &Ed3• Inst. SARB Ed2• Synoptic SARB Ed2• TISA Averaging (Ed2)• GGEO• MOA• Instrument• Synoptic SARB (Ed3)• TISA Averaging (Ed3)• ISCCP-D2likeThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Production Request<strong>Data</strong>base• PR requests a specified range of datadates be processed for a <strong>CERES</strong> PGE inthe production environment• Provides data dates, input/output filenames, configuration code, targetplatform, PGE, etc• Used for operational testing <strong>and</strong>productionThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Example PRThe Science Directorate at <strong>NASA</strong>’s Langley Research Center16
Production Request<strong>Data</strong>base• Production Request <strong>Data</strong>base Tool (PRDBT)– PRs submitted, reviewed, updated <strong>and</strong>approved via web interface– Public view <strong>and</strong> Restricted write– Capability for scripting to automate production• Preliminary design planning meeting <strong>and</strong>Requirements document• Planned PRDBT Completion December 2010The Science Directorate at <strong>NASA</strong>’s Langley Research Center
Operational ProcessImprovement Kaizen• ASDC <strong>and</strong> <strong>CERES</strong> DMT• Current State• Ideal State• Future State• First of FourThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Operational ProcessImprovement KaizenThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Questions & CommentsView <strong>and</strong> Order <strong>CERES</strong> <strong>Data</strong>http://ceres.larc.nasa.gov/order_data.phpThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
<strong>CERES</strong> Processing SoftwareApril 2010SubsystemNumberSubsystemNameLOC(to nearest 1K)Publicly AvailableDate ProductsProduct FrequencyComments<strong>CERES</strong>lib 132K All Satellites1 Instrument/Pre-Processor4KNPP only1 Instrument 110K BDS 1/day All Satellites2 ERBElike/ Inversion 33K ES-8 1/day All Satellites3 ERBElike/ TSA 16K ES-9, ES-4 1/month All Satellites12 MOA 10K Run monthly4.1 – 4.4 Clouds 359K All Satellites4.5 – 4.6 Inversion 153K SSF 1/hour All Satellites5 SARB 178K CRS 1/hour All Satellites6 & 9 TISA-Gridding 43K FSW, SFC,ISCCP-D2like-Day/Nit60/month, 36/month,1/monthAll Satellites11 GGEO 172K ISCCP-D2like-GEO 1/month Geostationary7.2 Synoptic SARB 47K All Satellites7.1 & 810TISA-Averaging 211K SYN, AVG, ZAVGSRBAVG1/day, 1/month, 1/month5/monthAll SatellitesTOTAL LOC 1,468KThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Edition 3ProcessingApproachThe Science Directorate at <strong>NASA</strong>’s Langley Research Center
Change language