TMS Performance Monitoring, Evaluation, & Reporting

Transportation ManagementSystems‣ The deployed form of ITS, along withhuman resource contribution, towardtransportation management‣ Includes computer hardware, software,communications, surveillance technology‣ TMC is a physical facility that housescentral equipment, software, andpersonnel that operate TMS

Performance Monitoring,Evaluation, & Reporting‣ Monitoring is ongoing internal processwhere system conditions are examinedthrough collected data‣ Evaluation is process where collecteddata is analyzed and compared to setbenchmarks.‣ Reporting provides the results of theevaluation for the stakeholders.

Performance Measures‣ Provide quantifiable indicators of programeffectiveness and efficiency‣ Help to determine progress towardspecific program goals and objectives‣ Help to determine priority projects,goal/objective refinements, and fundallocation

Performance Measure Types‣ Input• Supply of resources‣ Output• Delivery of transportation programs, projects,and services‣ Outcome• Degree to which transportation system meetspolicy goals and objectives

Performance Measure TypesMeasureTypeInputTraditional CapacityCapital projects budgetMaintenance and OperationsOrientedPerson-hours of incident responsepatrolsOutput Miles of roadway built Response time to incidentsOutcome Reduced miles of congestion Change in incident-related delay

Criteria for Defining PerformanceMeasures‣ Purpose‣ Validity‣ Precision‣ Accuracy‣ Cost-effectiveness

TMS/TMC FunctionsTMSFreeway TMSTraffic SurveillanceTraffic ControlIncident ManagementSensorsCommunicationCCTVTMC SoftwareWireless TechnologySpecial EventsInformation Sharing/DisseminationOverall TMS OutcomesIndividual Hardware ComponentsArterial TMSArterial ManagementWork ZoneSpecial EventsOverall OutcomesSensorsTraffic Signal ControlControllersPre-emption (Receivers)CommunicationTransit TMSInformation SharingTransit OperationsOverall Outcomes

Freeway TMS PerformanceMeasures‣ Incident Management –General MeasuresMetricTypeSupplementary NotesCalculationNumber of incidents, byseverity (e.g., fatal, injury), bytype (e.g., crash, stalledvehicle)externaln/an/aPerson hours working forTMS/TMC incidentmanagement systeminputBoth infield and TMCNo. People x workinghours per personNumber of respondedcrashes versus total numberof crashesoutputResponded crashes are crashesresponded by state safety patrolor freeway incident responseteamNo. of respondedcrashes over total no.of crashes reportedResponse time to incidentsoutputn/an/a

Performance MeasurementProgram‣ Performance measurement• Use of quantifiable indicators to determineprogress made toward agency goals andobjectives‣ Program is organized set of measuresthat combine to quantify and evaluateTMS goals and objectives

Performance MeasurementProgram Benefits‣ Accountability‣ Efficiency‣ Effectiveness‣ Communications‣ Clarity‣ Improvement

Establishing PerformanceMeasurement Program‣ Key steps• Identify the vision, goals, and objectives of the agency• Identify intended uses and audiences• Develop TMS performance measures and relate to respectiveprograms.• Identify performance benchmarks.• Collect complete, accurate, and consistent data• Analyze and evaluate data• Report data to stakeholders• Identify areas for improvement/change and report tostakeholders

Denver PerformanceMeasurement Program‣ Regional Transportation District established in1969‣ 3-tiered performance measurement program• Service Standards• Quarterly Progress Report• Annual Report‣ Measures reviewed using establishedstandards‣ Collect economic and customer satisfactionmeasures

Performance Measurement &Data‣ Data is important aspect of performancemeasurement program• Quantity• Quality• Coverage‣ Without “good”data, performancemeasurement program cannot beeffective

Data‣ Success of performance measurementplan relies heavily on quality of data‣ Data requirements must be defined‣ Requirements specify the types of dataneeded for an application, domain, orcomponent

Data Requirement Issues‣ Multiple concurrent incidents‣ Local economy‣ Data should be• Relevant• Timely• Cost-effective‣ Sensor coverage

Data Categories‣ Facility use and performance• Determine if TMS is operating at fulleffectiveness‣ Staff activities and resource use• Measure the efficient use of agencyresources‣ Events and incidents that affect normaltraffic conditions

Data Collection‣ Data obtained from 3 sources• Data archives• Modeling/estimation• Manual/automatic data collection

Data Archiving‣ Helps to make long-term evaluationpossible with regard to these categories.‣ Reasons to archive:• Greater and more accurate data• Cost-effective• Cheaper than manual collection• Adheres to current business practices

Data Collection/Archiving Issues‣ Availability‣ Completeness‣ Coverage‣ Quality‣ Standards‣ Reliability‣ Variability

Data Collection/Archiving Issues(Cont’d)‣ Aggregation level‣ Experimental design‣ Storage‣ Metadata‣ Institutional/data sharing

Data Archiving Best Practice‣ PeMS• Freeway performance management systemcreated by Caltrans and UC-Berkeley• Gathers raw, real-time freeway data fromparticipating districts• Established process for processing data• http://transacct.eecs.berkeley.edu

Performance Monitoring‣ Using performance measurement tovisualize system status‣ Immediate decisions based thisinformation‣ Long-term monitoring can assist withdecision-making• Maintenance• Future deployment

Performance Monitoring Levels‣ System operators typically focus on dayto-dayoperations of one corridor orroadway‣ Supervisors generally focus on severalcorridors or entire region‣ Managers generally focus on entire TMSat high level with daily/weekly reports

Monitoring Examplehttp://www.dot.ca.gov/dist11/d11tmc/sdmap/mapmain.html

Performance Evaluation‣ Analysis of data about the TMS‣ Results compared to benchmarkmeasures‣ Used to determine effectiveness ofstrategies, policies, systems, etc.‣ Helps identify areas to improve and justifythe need for additional resources

Performance Evaluation‣ Allows for the following:• Determination of actual improvement inperformance• Identification of problems that result ininefficiencies• Analysis and prioritization of alternativesolutions• Estimation of the benefits and costs of TMS

Evaluation Techniques‣ Prior to selecting a tool to evaluate theTMS, agencies need to consider thefollowing:• Analysis context (planning, design, ops, etc.)• Geographic scope (corridor, region, etc.)• Capability of modeling facilities (freeway,HOV lanes, etc.)• Ability to analyze various modes

Evaluation Techniques (Cont’d)‣ Also must consider• Ability to analyze different managementstrategies (ramp metering, signalcoordination, etc.)• Ability to estimate traveler response tomanagement strategies (route diversion,mode shift, etc.)• Ability to output direct performance measures(safety, efficiency, etc.)• Tool/cost effectiveness

Before-and-After Evaluation‣ Most common method to evaluate TMSeffectiveness‣ Studies effects of particular managementstrategy by studying performancemeasurement results from before andafter the implementation

Before-and-After EvaluationIssues‣ Difficulty in distinguishing effects of oneimprovement when multiple ones weremade at once‣ Time needed for drivers to adjust tochange‣ Time-related factors‣ Regression to the mean

Best Practice‣ Before-and-after study of Phase I of I-10/I-17 FMS in Phoenix area‣ Studied several MOEs for 57 km offreeway fitted with ramp meters, VMS,loop detectors, CCTV cameras‣ 2-6% travel time improvement alongseven-mile stretch with ramp meters

Performance Reporting‣ Allows for communication withstakeholders about system performance‣ Helps in decision-making process‣ Allows for tracking of TMS progress‣ Creates sense of accountability

Reporting Trends‣ Using the media‣ Daily/weekly intranet postings‣ Quarterly/monthly public reports on theInternet‣ Formal biannual/annual reports forgovernment/business officials‣ “Notebooks”keep key decision-makers upto date on agency performance/goals

Reporting Best Practice‣ WSDOT’s “Gray Notebook”‣ Explains agency’s planning process andrationale behind decision-making‣ Assesses statewide conditions‣ Tracks assortment of reliability andeffectiveness measures for routine review‣ Has become important source for statelegislators and other agency stakeholders

Handbook Overview‣ Handbook serves as a technical referenceon performance monitoring, evaluation,and reporting‣ Provides ways for planning,implementing, and sustaining aperformance measurement program‣ Discusses issues related to datacollection and archiving

Handbook Overview (Cont’d)‣ Intended audience include representatives of• State DOTs• MPOs• Transit agencies• Enforcement agencies‣ Intended audience is also anyone with role inTMS/TMC performance monitoring, evaluation,and reporting

Chapter 3TMC FunctionsChapter 4TMS Goals andObjectivesChapter 3Budget andResourceAllocation,ProjectPrioritizationHigh-levelTasksChapter 3/4DefinePerformanceMeasures &Set BenchmarkChapter 3StakeholdersChapter 5DataRequirementsData-drivenTasksChapter 5Data CollectionandProcessingChapter 5Data ArchiveChapter 6PerformanceMonitoringChapter 6EvaluationChapter 3DecisionMakersMotivateChangesChapter 3TMS EfficiencyTasksChapter 6ReportPerformanceMeasuresPublic

Handbook At-A-Glance‣ Chapter 1 –Introduction. Defines thebackground, purpose, and scope of thehandbook and the intended audience.‣ Chapter 2 –Overview of TMS PerformanceMonitoring, Evaluation, and Reporting.Provides a high level overview of TMSperformance monitoring, evaluation, andreporting and how they relate to TMSs.‣ Chapter 3 –Performance MeasurementProgram. Discusses the purpose andimportance of, and need for a TMSperformance measurement program.

Handbook At-A-Glance‣ Chapter 4 –Agency Goals and PerformanceMeasures. Presents typical performancemeasurement goals of TMS-related agencies.Also provides high-level performance measuresby TMS functions and calculation methods ofsuch performance measures.‣ Chapter 5 –Data Requirements, Collectionand Archiving. Provides performance measuredata requirements and best practices for datacollection, evaluation, and reporting.

Handbook At-A-Glance (cont’d)‣ Chapter 6 –Performance Monitoring, Evaluation andReporting. Explains various monitoring and evaluationmethodologies and processes related to TMS performance.Discusses recommended reporting techniques, formats, andfrequencies for reporting TMS performance.‣ Chapter 7 –Self- Assessment. A checklist of questionsdrawn from case studies. Can be used by TMCs to assessthe status and performance of the TMS. Also includes somebest practice examples from selected agencies.‣ Appendix A –Survey Questionnaire and Results‣ Appendix B –Contact List of Traffic ManagementCenters

Other Subject Information‣ Fact sheet‣ FAQ‣ Tri-fold brochure‣ Primer‣ Available athttp://tmcpfs.ops.fhwa.dot.gov/Projects.htmFHWA-HOP-07-126