The ALICE guide to Shift Leadership - Alice - CERN

Start of pp PHYSICS run5/31/2011 ALICE SL guide 5/

Calibration run during LHCramp-up or flat topSystem Run Frequency Duration When DCS stateHMPID calibration fill 2 min pre ramp/ramp up HMPBEAM_TUNINGTOF Noise fill 3 min pre ramp**/ramp up/flat top*repeat if FERO_NOT_READY** try to perform it at pre-ramp to avoid possible failures due to clock oscillationsTOF BEAM_TUNINGZDC Pedestal fill 4 min ramp up ZDC READYSSD pedestal fill 6.5 min ramp up SSD READYSDDPedestal,Pulser,Injector,INIT_FEROfill 6.5 min ramp up SDD READYM_TRK pedestal* fill 2.5 min ramp up MCHBEAM_TUNINGPMD pedestal fill 5 min flat top/squeeze PMDBEAM_TUNING5/31/2011 ALICE SL guide 6/

Calibration runs during LHCramp-downsystem run frequency duration when DCS stateT0 amp_cal fill 20 min. ramp down T00 READY, Right BM_TRK elec. cal. fill 25 min. ramp down/adjust MCH BEAM_TUNINGM_TRK pedestal* fill 2.5 min. ramp down/adjust MCH BEAM_TUNINGEMCAL pedestal fill 1 min. ramp down/adjust EMC READYM_TRG calibration fill 5 min. ramp down MTR READY* repeat if FERO_NOT_READY5/31/2011 ALICE SL guide 7/

How to deal with calibration runs●●●●●●SL should check that all central calibration runs are donecheck the calibration run page in the logbookECS shifter monitors the correct ending of the runsIf error try once more. If error, give up and call expertStand alone calibration runs may be carried out on request,at an appropriate timeDo not wait to start a run for systems which run standaloneNote that now RAMP down time is very short; runs mayhave to be stopped if a new handshake is issued: in thatcase, DO NOT WASTE ANY TIME and bring ALICE toSAFE immediately.5/31/2011 ALICE SL guide 9/

In case of Problems●●●●●●●a detector’s busy time substantially slows down the run: stop, exclude,and restart. Then call expert if neededa detector is producing too much data: exclude it in the next chance:stop, exclude, and restart. Then call expertTPC, ITS, MCH and MTR should be included asapfor other detectors, try to group several actions in one single break(exclude/include detectors, pedestal runs, lumi adjustment), even if ittakes much longer than 1 hour to stopconsider to run a MCH and/or PMD pedestal during breaksproblems with main system: shifter calls on-call expert if needed.Consult with PRC and RC if problem persistsa trigger detector or the TPC is not operable: call expert; consult withPRC about how to proceed5/31/2011 ALICE SL guide 10/

Current issuesenter value andclick to changetarget lumi●Lumi: we aim at a 500 Hz V0ANDrate per colliding bunchN bunches ⋅ 500Hz= N54.2mbbunches ⋅ 6.944 ⋅ 10 −3 ub s −1where N bunches is the number of colliding bunches●Publish this value via DCS: LHC-> Target Lumi●If needed, guide CCC to reach our lumi(either through target lumi or through V0AND rate)●●●●restart run after all lumi scans finishedLumi 10 times larger for high lumi fills; see PRC daily instructionsSPD alignment to be checked by shifter at each runif wrong, stop and start a new runALI_DCS-> LHC_DCS->ALILUMINOSITY●Try to run as efficiently and for as long as possible - avoid short runs●Try to document (EOS) the reasons and time lost due to problems●Do not hesitate to add log entries (even associated to a run) along the shift; no need to wait till the EOS report●Please make sure the messages in the DCS big screen are meaningful and up-to-date (this is web-broadcasted!)5/31/2011 ALICE SL guide 11/

L2a rates, dead times●●●●●●●●For a given L0 rate (e.g. V0AND in DCS for the ALL cluster) and the correspondingdead time fraction ,l, of this cluster, the rate of accepted triggers, L2a, is given byL2a = (1-l) L0l = t·L2awhere t is the average busy time of the cluster, determined by the slowest detector.ThereforeL2a = L0/(1+t·L0)i.e. Poissonian Arrivals See Time Averages (PASTA)With this simple formula you can check the consistency of the event recording rateand how far from saturation is the system. For example, if V0AND = 3 kHz and t =0.001 s,L2a = 3000/(1+3) = 750 Hz● l = 75%●Tip: use your pocket calculator: the numbers should come out quite exact5/31/2011 ALICE SL guide 12/

Tips●●●●●●●look at the recovery procedure of TPC in case of trips (s. TPC: ERROR_RECOVER) before ithappens, so you are preparedWe go to supersafe when machine announces beam losses, loss maps, injection tests,collimator set-upsDAQ shifter should issue CHECK_FERO to HMPID, PMDClock switches automatically from LOCAL to BEAM1 (and back)do not delay the injection permit despite TOF going MIXED during clock changeIf a run doesn’t really stop, ask DAQ shifter to select ‘Force timeout’ in File tab in PCA, orsee if the option ‘Abort’ is active in Run Control panelAnticipate the action: make sure all shifters know what to do before it comes● Proton Physics beam modes: INJECTION – (PREPARE) RAMP – FLAT TOP – SQUEEZE –ADJUST – STABLE BEAMS – (ADJUST) – DUMP – RAMP DOWN●●You can use the computers at your desk: shift shift.123If ECS errors mention machines alidcscom026-027,alidcsvme001-008, or alidcsvme0017,then this is a CTP problem; DAQ machines are aldaqxxxx5/31/2011 ALICE SL guide 13/

Handshakes●●●●●●●LHC must go through the handshake procedure to get the ALICE injection permits.If LHC asks for injection permits without handshake the SL must contact the(P)RC before any action is taken.All handshakes (INJECTION, ADJUST, DUMP or TI2_SETUP) produce an audible gong atthe DCS console. Make sure the DCS shifter promptly reacts to it (two actions, one clickeach).ALICE must be safe before proceeding with a handshake; do NOT delay the handshake forany other reason but ALICE safe.If a handshake is received but one or more subsystems cannot be moved toSAFE/SUPER_SAFE the DCS operator must issue REPORT_PROBLEM. The SL mustcontact CCC to inform them and contact RC to decide. The Run Coordinator will decide onthe further procedure.If a handshake (INJECTION, TI2_SETUP) is active but the injection permits are notgiven/restored the SL has to check that ALICE is safe/super-safe and ask the DCS operator togive the injection permits manually.Before manually removing permits (if it does not happen by post-mortem or dumphandshake) the SL must contact CCC.5/31/2011 ALICE SL guide 14/

SMAQ plots●●●●●●●SMAQ is a tool that the CTP shifter should run every now and then,certainly every time there is a new filling scheme.It should be run as soon asInjection is finished; trigger on one PBTX and check filling schemethe first PHYSICS run in a fill is started; trigger on 0SMH (0MSL forpp) and check alignmentAnticipate: make sure the CTP shifter knows and understands the toolbefore hand. Otherwise s/he should call the oncall.The shifter should set up as many zoomed windows as there arecolliding bunches. The time buckets where to expect each bunch is forexample in the CTP configuration in ACT (please find it)Make sure these plots are put in the logbook at least once per fill.5/31/2011 ALICE SL guide 15/

SMAQ plotsAll (relevant) triggers should be aligned.Note the tails of V0 and the two groups of4 BCs for SPD (all BC must fall in one of the 4 phasesof the first SPD group).In this example there is a problem with theclock alignment of the SPD: 3 groups!SPD expert should be called5/31/2011 ALICE SL guide 16/

Trigger DQM•CTP shiftershould befamiliar withthis; you too.•Examine the‘main’ tab(here theALLNOTRD tabis selected)amore -d TRI -m MyModule15/31/2011 ALICE SL guide 17/

Technical runs●During periods with no or non-stable beams, SL organises for TECHNICAL runs●this allows us to check if the experiment is ready for data taking●configure partitions (PHYSICS_1, PHYSICS_2, …) CTP (without trigger detectors) as, e.g., CTP_BCMASKACWU●ALICE is safe if needed●During the day or if Physics imminent, call experts if problems●if busy, let the expert investigate before stopping the run●no change in ECS partition●recording OFF (for now), mark run as ‘bad’●consider to make a short run with recording ON●(run a PMD PEDESTAL calibration at the beginning of the session)●you may run all calibration runs as during a fill●you may stop the run to take new pedestals for MCH●remember to load the ‘good’ configuration before Physics!●remember to turn on recording in ACT/ECS before Physics!●in supersafe, currently only FMD is unable to run5/31/2011 ALICE SL guide 18/

Ramping up/down the magnets●●●●●●When scheduled, SL asks CCC to ramp down the ALICE solenoid and/or the dipolecheck before that DCS brings detectors to the right state (HMPID, T0, MCH, PMD)There is a dedicated tab ‘magnet safe’ in the central DCSinform CCC that you will turn on the interlocks (keys next to the magnet console)When scheduled, SL asks CCC to switch polaritiesWhen scheduled, or if ready, SL informs that ALICE is ready for the magnets to beramped up, and disables the interlocks (keys in the back side of the control room)● In case of problems call the magnet piquet 162082●●●●Before starting a run:T00 should be configured properly wrt. the magnet state.No need to notify HMPID OnCall before runSame applies for ramp up after magnet trips5/31/2011 ALICE SL guide 19/

Run coordinationhomepage & linksShift instructions5/31/2011 ALICE SL guide 20/

Shift crew - all present?incl. on-call, SLIMOS and RPESL ticks present shiftersShifters must do their full shifts,but you should tick them within the first 2 hours.5/31/2011 ALICE SL guide 21/

LHC page 1, 35/31/2011 ALICE SL guide 22/

Magnets, beam permit,detector status5/31/2011 ALICE SL guide 23/

LHC beam data:intensities, rates, luminosity5/31/2011 ALICE SL guide 24/

Statistics of Physics runs5/31/2011 ALICE SL guide 25/

Log entrieswrite SL EOS report!5/31/2011 ALICE SL guide 26/

Example of SL EOS report5/31/2011 ALICE SL guide 27/

ACT - overviewconfigure partitions (this view)configure CTP (next slide)5/31/2011 ALICE SL guide 28/

ACT - CTP globalACT -> Run Coordination -> CTPtechnical runs -> DAQRANDOMphysics runs -> pp11apply filter if adetector is notavailable astrigger inputfilling schemefrom LHCleave the other settings as they are5/31/2011 ALICE SL guide 29/

change the trigger configuration for this partitionwithout reconfiguring the detectorsACT - PHYSICS_1technical runs -> ERP_2011cosmics -> CTP_tofCosmicscollisions -> pp11mb015/31/2011 ALICE SL guide 30/

Trigger overview screenaverage busy times in microseconds0SMB - SPD min bias0VBA - V0 A-side0VBC - V0 C-side0BPA - beam pickup A-side0BPC - beam pickup C-side0MSL - muonINT1/2 - interaction triggers5/31/2011 ALICE SL guide 31/

Useful phone numbers and links●●●●●●ALICE run coordinationhttp://aliweb.cern.ch/Run_Coordination/Run09/index.htmlALICE logbookhttps://alice-logbook.cern.ch/logbook/ALICE Configuration Tool (ACT)https://alice-logbook.cern.ch/act/ALICE DCS monitoringhttp://alicedcs.web.cern.ch/AliceDCS/monitoring/ALICE trigger pagehttp://alidcscom188/ALICE shift management systemhttps://alicesms.cern.ch/● LHC page 1http://op-webtools.web.cern.ch/opwebtools/vistar/vistars.php?usr=LHC1CCC7760070480RC (Paolo) 163757PRC(MinJung)163889Coolingpiquet72201Gas piquet 162516Magnetpiquet162082Depannage 72201Guardians 78878GLIMOS 162237●LHC logbookhttps://ab-dep-op-elogbook.web.cern.ch/ab-dep-opelogbook/elogbook/eLogbook.php5/31/2011 ALICE SL guide 32/

TPC: ERROR_RECOVERIn case of TPC ERROR in the DCS due to HV trip, the TPC DCS statewill change to ERROR_RECOVER.●●●●●●The run will be paused automatically and there is no need to stop therun.Notify the TPC HV oncall immediately who will have 20 minutes torecover.If TPC is recovered within the time window, the run will restartautomatically.If the trip is not recovered within the time window, the run will stopautomatically.A new run should start once the TPC state goes to READY.The DCS shifter should be very alert to monitor this error.5/31/2011 ALICE SL guide 33/

TRDIn order to avoid possible trips due to beam losses:- TRD has to go SUPERSAFE before theinjection*- TRD has to go back to SAFE at the pre-ramp** keep TRD in the safety matrixmove TRD to supersafe/safe acting only on TRDin the safety matrixThis should be done only for Injection Physics Beam(not for Injection Probe Beam)

HLT●●●●●●●If the HLT times out during engaging or stopping, before calling the on-call expert,the shift crew should try the following procedure:Do not release the HLT lock from ECS.Go to the HLT screens and click on the big red "ABORT HLT" button. (If the abortbutton is missing try the "Start abort button" instructions below).It will change to "ABORTING..." and takes about 20 seconds maximum.After the abort procedure is done the button should read "ABORT HLT" again andthe HLT state in the ECS should be INITIALIZED or OFF.At this point try start the run again as per normal.If the run fails a second time due to HLT then call the HLT on-call expert and thistime release the HLT lock, plus put HLT into mode A.5/31/2011 ALICE SL guide 35/

Detector FSM statesin SAFE and SUPER-SAFE5/31/2011 ALICE SL guide 36/