Predicting Crowd workers’ Performance as Human-Sensors for Robot Navigation

This paper provides and evaluates a new paradigm for collaborative human-robot operation in search and rescue-like settings with information asymmetry. In particular, we focus on settings where the human, a crowd worker in our case, is used as a sensor, providing the route-planning module with essential environmental information. In such settings, the ability to predict the expected performance of the collaborating crowd worker in real-time is instrumental for maintaining a continuously high level of performance. Through an extensive set of experiments with crowd workers recruited and interacted through Amazon Mechanical Turk, we show that effective online prediction is indeed possible, however only if distinguishing between two subpopulations of crowd workers, termed ”operators” and ”sensors”, applying a different prediction model to each. Furthermore, we show that even the classification of crowd workers to the two types can be carried out successfully in real-time, based merely on the first two minutes of collaboration. Finally, we demonstrate how the above abilities can be used for a more effective workers’ recruiting process, resulting in a substantially improved overall performance.