Optimality and Efficiency Requirements for the Design of Stated Choice Experiments
J N Ibáñez, J Toner, ITS, University of Leeds, UK: A Daly, ITS, University of Leeds, and RAND Europe, UK
This paper discusses appropriate procedures to construct optimal stated preference designs, reviewing some differing results in the literature and building new insights into the comparison of design criteria and properties such as orthogonality.
To assess the value of aspects of transport alternatives researchers often apply survey techniques that allow them to explore public preferences for hypothetical scenarios. A widely-used standard survey technique for this purpose has been the contingent valuation method. In the last twenty years, stated preferences (SP) approaches have been used in several contexts of interest, including transport analysis. SP is a technique that can be used to assess values for attributes of market or non-market goods based on survey respondents' willingness to trade-off different bundles of these attributes. In a SP survey, respondents are presented with a set of scenarios that differ in terms of a series of attributes and are asked to rank or rate the alternative scenarios, or choose their most preferred. The scenarios differ by the levels of the different attributes and the methodology applied to obtain parameters that weight the contribution of each attribute to the preferences is discrete choice modelling (DCM). Most recently, the Stated Choice approach has been the most widely used of the Stated Preference methods, as it is believed to approximate most closely to actual consumer behaviour.
Because of cost considerations, sample sizes are often limited to the smallest that researchers consider necessary. By employing optimal survey design techniques, practitioners can increase the informational content of each observation, producing the equivalent effect of a larger sample size. In this context, the main goal of this paper is to describe a procedure to construct optimal SP designs that, given a fixed number of observations, will provide the most information possible about parameter estimators of interest, such as mean or median willingness to pay. In so doing, the paper aims to extend the existing literature on the optimal design of surveys to apply DCM in three ways: first, by arguing the limited applicability of the concepts of traditional conjoint analysis to build choice sets to apply DCM, second, by analysing the different and sometimes competing ways to define the optimality required, and third, by discussing the influence on design optimality of the ultimately more important issues of reliability and credibility of the responses.
The methods to be used in deriving the design results are first analytical, studying the points of optimality of the simpler designs. In more complex cases, use is made of optimisation algorithms, principally a guided search routine that explores over all choice set combinations to derive optimal choice sets, showing the limited applicability of some analytical procedures to derive optimal designs.
To support this discussion, and given that a number of different measures of the statistical efficiency of a design have been proposed in the literature (D-optimality, A-optimality, S-optimality, etc.), we show how these measures are not complementary, since they serve different purposes, and include quantitative measures of this difference. The approach explored as a more appropriate method for SC design is the consideration of a combination of different optimality measures so as to decrease the inherent uncertainty in SC designs, where prior beliefs about the parameters to be estimated have to be made explicit.
The paper also addresses with the dissimilar criteria that have been used in the literature to measure the covariance matrices of the parameters to be estimated. We assess the different ways to calculate these matrices and the assumptions on which their use is based, aiming to give practical advice to choice designers.
The type of DCM that is going to be applied over the responses collected also affects the process to design such survey. The criterion when a multinomial logit model is going to be applied is clear in the literature, though it is not so when the analyst is interested in more behaviourally plausible models such as nested or mixed logit. By means of numerical simulation of responses we offer numerical evidence of the magnitude of the loss arising when an experiment is designed on the basis of an inappropriate choice model. In doing so we also give proof of restrictive criteria adopted repeatedly in the literature when mixed logit is employed at the design stage.
We also address the issue of orthogonality in SP designs, a property often sought by designers, since when it applies the parameter estimates are independent in their explanation of the observed responses, which does not necessarily imply that these estimates will be significant. This issue is particularly relevant for larger and more diverse designs.
In conclusion, the paper offers a number of insights and results that will be of assistance to the designers of SP surveys in their attempts to maximise the effectiveness of survey budgets.
Association for European Transport