Evaluation of Intelligent Mobility Services - Connecting Research and Policy by Using Program Theory
Lone-Eirin Lervåg, SINTEF Technology and Society
An evaluation framework for assessment of intelligent mobility services is presented, introducing program theory as a beneficial method for development of appropriate study designs and in connecting research and policy.
This paper presents an evaluation framework for assessment of intelligent mobility services, introducing program theory as a beneficial method for development of an appropriate study design within the evaluation methodology.
The wide range and variety of intelligent transport system (ITS) services and the continuous development and improvement of technology maturity and solutions has proven to be a challenge for the evaluation assessment, as the nature and extent of impacts of such ITS services are fundamentally different from those of conventional road projects (Kulmala, 2010; Jenssen, 2010; Linder et al., 2007 and Newman-Askins et al., 2003). Effort has been put in development of common evaluation guidelines based on best practices, as the FESTA methodology offers a framework for assessment of driver assistance systems (FOT-Net, 2014), and the EasyWay initiative provides recommendations for evaluating the implementation of ITS projects (Tempo, 2008). However, there is yet no consensus on how to evaluate intelligent mobility services.
A framework for assessment of intelligent mobility services is developed based on a combination of general principles for scientific evaluation and specific ITS evaluation guidelines, focusing on providing research evidence and knowledge that is applicable and beneficial for policy making and practical deployment of ITS infrastructure and services. This framework is utilized and tested in two Norwegian evaluation studies and in a Nordic cooperation project on evaluation of ITS services.
The evaluation approach includes assessment of technical performance, impacts, user acceptance, socio-economic performance, financial performance and market potential. The evaluation process consecutively addresses a) technology readiness, b) problem analysis and needs assessment, c) establishment of a program theory based on objectives, d) formulation of evaluation criteria, e) development of a proper evaluation design, f) definition of data collection methods, g) field test execution and data collection, h) design of data analysis and i) specification of report content. Although using a common evaluation framework, it is recognized that the individual study designs will have to be tailor made for each individual service and test site.
Special emphasis is placed on establishment and use of program theory as a method for development of an appropriate evaluation study design. The program theory is described as an explicit theory or model of how an intervention contributes to a chain of intermediate results and finally to the intended or observed outcomes (Funnel and Rogers, 2011). The program theory describes the relationship between the ITS service and the expected outcome in details, including the causal mechanisms that are expected to trigger off the anticipated effects of the service. Thus, the program theory represents the logical foundation of the implementation of the intelligent mobility scheme; how the intervention is expected to lead to its effects and in which conditions it would do so.
Development of the program theory is built on problem analysis and needs assessment, including stakeholder analysis and definition of targets groups, user needs and interests. The result is an explicit model of the policy makers' expectations of the effects and impacts of the mobility service, and once the program theory is established; the appropriate research questions, hypothesis and performance indicators are easily derived from the model.
Using the program theory in development of evaluation studies has become increasingly prevalent within societal sciences as health, education and crime research (Karlsen and Jentoft, 2013; Funnel and Rogers, 2011), but scarcely any evidence is found on program theory as an explicit method within traffic or ITS studies. This paper discusses the experiences with program theory based evaluations in three case studies on intelligent mobility services. Benefits and limitations are presented, concluding that the program theory seems to be a promising method especially when it comes to connecting research to policy. In addition to offer an efficient evaluation tool, the program theory clarifies the stakeholders' expectations in the beginning of the project and makes sure the emphasis is put on the right research questions to obtain applicable knowledge. Finally, the program theory has also proven beneficial in achieving the correct interpretation of evaluation results.
Association for European Transport