Publication Date
2019
Journal or Book Title
Under submission
Abstract
Expert elicitation, a method of developing probability distributions over unknown parameters, traditionally involves in-person interviews by a trained analyst. There is growing interest in using the internet to enable participation of larger, more distributed groups of experts. However, analysts have questioned the quality of judgements elicited online rather than in person. We systematically compare online and in-person elicitation modes, finding no significant difference between the two modes across multiple measures: the two modes are similar in accuracy, uncertainty ranges, number of surprises, fatigue, and the substance of qualitative comments. These findings have an important caveat: many elicitation questions were subject to problems in online administration that made it impossible to compare to in-person results. We conclude that, although online elicitations represent a less resource-intensive option for large expert elicitations, they may require a higher level of testing and quality control since there is no analyst to catch errors or clarify small misunderstandings.
License
UMass Amherst Open Access Policy
Funder
Sloan Foundation
Recommended Citation
Baker, Erin; Cruickshank, Claire; Jenni, Karen; and Davis, Steven, "Comparing in-person and online modes of expert elicitation" (2019). Under submission. 620.
Retrieved from https://scholarworks.umass.edu/mie_faculty_pubs/620