Effective Operator Summaries Extraction

Ido Nimni, David Sarne

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

This paper proposes a heuristic algorithm for effectively summarizing the work of novice robot operators, e.g., ones recruited through crowdsourcing platforms, in search and rescuelike tasks. Such summaries can be used for many purposes, perhaps most notably for monitoring and evaluating an operator’s performance in settings where information gaps preclude automatic evaluation. The underlying idea of our method is dividing the task timeline into intervals, and extracting a subset of high-scoring and low-scoring segments within, using a heuristic scoring function. This results in a short effective summary of the operator’s work, based on which several other crowd workers can evaluate her performance. The effectiveness of the proposed method was extensively evaluated and compared to a large set of alternative methods through a series of experiments in Amazon Mechanical Turk. The analysis of the results reveals that the proposed method outperforms all tested alternatives. Finally, we evaluate the performance one may achieve with the use of machine learning for predicting the operator’s performance in our domain. While this approach manages to reach a performance level similar to the one achieved with summaries, it requires an order-of-magnitude greater effort for training (measured in terms of crowd workers time).

Original languageEnglish
Title of host publicationHCOMP 2020 - Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing
Editors Lora Aroyo, Elena Simperl
PublisherAssociation for the Advancement of Artificial Intelligence
Pages102-111
Number of pages10
ISBN (Print)9781577358480
DOIs
StatePublished - 2020
Event8th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2020 - Virtual, Online
Duration: 25 Oct 202029 Oct 2020

Publication series

NameProceedings of the AAAI Conference on Human Computation and Crowdsourcing
Volume8
ISSN (Print)2769-1330
ISSN (Electronic)2769-1349

Conference

Conference8th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2020
CityVirtual, Online
Period25/10/2029/10/20

Bibliographical note

Publisher Copyright:
© 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Fingerprint

Dive into the research topics of 'Effective Operator Summaries Extraction'. Together they form a unique fingerprint.

Cite this