Crowdsourcing inference-rule evaluation

Naomi Zeichner; Jonathan Berant; Ido Dagan

Crowdsourcing inference-rule evaluation

Naomi Zeichner, Jonathan Berant, Ido Dagan

Department of Computer Science

Tel Aviv University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

30 Scopus citations

Abstract

The importance of inference rules to semantic applications has long been recognized and extensive work has been carried out to automatically acquire inference-rule resources. However, evaluating such resources has turned out to be a non-trivial task, slowing progress in the field. In this paper, we suggest a framework for evaluating inference-rule resources. Our framework simplifies a previously proposed "instance-based evaluation" method that involved substantial annotator training, making it suitable for crowdsourcing. We show that our method produces a large amount of annotations with high inter-annotator agreement for a low cost at a short period of time, without requiring training expert annotators.

Original language	English
Title of host publication	50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference
Pages	156-160
Number of pages	5
State	Published - 2012
Event	50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Jeju Island, Korea, Republic of Duration: 8 Jul 2012 → 14 Jul 2012

Publication series

Name	50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference
Volume	2

Conference

Conference	50th Annual Meeting of the Association for Computational Linguistics, ACL 2012
Country/Territory	Korea, Republic of
City	Jeju Island
Period	8/07/12 → 14/07/12

Cite this

@inproceedings{359e39736d884461bab65a739da16601,

title = "Crowdsourcing inference-rule evaluation",

abstract = "The importance of inference rules to semantic applications has long been recognized and extensive work has been carried out to automatically acquire inference-rule resources. However, evaluating such resources has turned out to be a non-trivial task, slowing progress in the field. In this paper, we suggest a framework for evaluating inference-rule resources. Our framework simplifies a previously proposed {"}instance-based evaluation{"} method that involved substantial annotator training, making it suitable for crowdsourcing. We show that our method produces a large amount of annotations with high inter-annotator agreement for a low cost at a short period of time, without requiring training expert annotators.",

author = "Naomi Zeichner and Jonathan Berant and Ido Dagan",

year = "2012",

language = "אנגלית",

isbn = "9781937284251",

series = "50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference",

pages = "156--160",

booktitle = "50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference",

note = "50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 ; Conference date: 08-07-2012 Through 14-07-2012",

}

Zeichner, N, Berant, J & Dagan, I 2012, Crowdsourcing inference-rule evaluation. in 50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference. 50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference, vol. 2, pp. 156-160, 50th Annual Meeting of the Association for Computational Linguistics, ACL 2012, Jeju Island, Korea, Republic of, 8/07/12.

Crowdsourcing inference-rule evaluation. / Zeichner, Naomi; Berant, Jonathan; Dagan, Ido.
50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference. 2012. p. 156-160 (50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference; Vol. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Crowdsourcing inference-rule evaluation

AU - Zeichner, Naomi

AU - Berant, Jonathan

AU - Dagan, Ido

PY - 2012

Y1 - 2012

N2 - The importance of inference rules to semantic applications has long been recognized and extensive work has been carried out to automatically acquire inference-rule resources. However, evaluating such resources has turned out to be a non-trivial task, slowing progress in the field. In this paper, we suggest a framework for evaluating inference-rule resources. Our framework simplifies a previously proposed "instance-based evaluation" method that involved substantial annotator training, making it suitable for crowdsourcing. We show that our method produces a large amount of annotations with high inter-annotator agreement for a low cost at a short period of time, without requiring training expert annotators.

AB - The importance of inference rules to semantic applications has long been recognized and extensive work has been carried out to automatically acquire inference-rule resources. However, evaluating such resources has turned out to be a non-trivial task, slowing progress in the field. In this paper, we suggest a framework for evaluating inference-rule resources. Our framework simplifies a previously proposed "instance-based evaluation" method that involved substantial annotator training, making it suitable for crowdsourcing. We show that our method produces a large amount of annotations with high inter-annotator agreement for a low cost at a short period of time, without requiring training expert annotators.

UR - http://www.scopus.com/inward/record.url?scp=84878186118&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:84878186118

SN - 9781937284251

T3 - 50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference

SP - 156

EP - 160

BT - 50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference

T2 - 50th Annual Meeting of the Association for Computational Linguistics, ACL 2012

Y2 - 8 July 2012 through 14 July 2012

ER -

Crowdsourcing inference-rule evaluation

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this