Improving reliability of word similarity evaluation by redesigning annotation task and performance measure

Oded Avraham; Yoav Goldberg

Improving reliability of word similarity evaluation by redesigning annotation task and performance measure

Oded Avraham, Yoav Goldberg

CS@BIU

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

9 Scopus citations

Abstract

We suggest a new method for creating and using gold-standard datasets for word similarity evaluation. Our goal is to improve the reliability of the evaluation, and we do this by redesigning the annotation task to achieve higher inter-rater agreement, and by defining a performance measure which takes the reliability of each annotation decision in the dataset into account.

Original language	English
Title of host publication	Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016
Publisher	Association for Computational Linguistics (ACL)
Pages	106-110
Number of pages	5
ISBN (Electronic)	9781945626142
State	Published - 2016
Event	1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Berlin, Germany Duration: 7 Aug 2016 → …

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016
Country/Territory	Germany
City	Berlin
Period	7/08/16 → …

Bibliographical note

Funding Information:
The work was supported by the Israeli Science Foundation (grant number 1555/15). We thank Omer Levy for useful discussions.

Publisher Copyright:
© 2016 Proceedings of the Annual Meeting of the Association for Computational Linguistics. All Rights Reserved.

Cite this

Avraham, O., & Goldberg, Y. (2016). Improving reliability of word similarity evaluation by redesigning annotation task and performance measure. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 (pp. 106-110). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL).

Avraham, Oded ; Goldberg, Yoav. / Improving reliability of word similarity evaluation by redesigning annotation task and performance measure. Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Association for Computational Linguistics (ACL), 2016. pp. 106-110 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{7669fcecd0b849a393f2786186665aa7,

title = "Improving reliability of word similarity evaluation by redesigning annotation task and performance measure",

abstract = "We suggest a new method for creating and using gold-standard datasets for word similarity evaluation. Our goal is to improve the reliability of the evaluation, and we do this by redesigning the annotation task to achieve higher inter-rater agreement, and by defining a performance measure which takes the reliability of each annotation decision in the dataset into account.",

author = "Oded Avraham and Yoav Goldberg",

note = "Publisher Copyright: {\textcopyright} 2016 Proceedings of the Annual Meeting of the Association for Computational Linguistics. All Rights Reserved.; 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 ; Conference date: 07-08-2016",

year = "2016",

language = "אנגלית",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "106--110",

booktitle = "Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016",

address = "ארצות הברית",

}

Avraham, O & Goldberg, Y 2016, Improving reliability of word similarity evaluation by redesigning annotation task and performance measure. in Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 106-110, 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, 7/08/16.

Improving reliability of word similarity evaluation by redesigning annotation task and performance measure. / Avraham, Oded; Goldberg, Yoav.
Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Association for Computational Linguistics (ACL), 2016. p. 106-110 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Improving reliability of word similarity evaluation by redesigning annotation task and performance measure

AU - Avraham, Oded

AU - Goldberg, Yoav

PY - 2016

Y1 - 2016

N2 - We suggest a new method for creating and using gold-standard datasets for word similarity evaluation. Our goal is to improve the reliability of the evaluation, and we do this by redesigning the annotation task to achieve higher inter-rater agreement, and by defining a performance measure which takes the reliability of each annotation decision in the dataset into account.

AB - We suggest a new method for creating and using gold-standard datasets for word similarity evaluation. Our goal is to improve the reliability of the evaluation, and we do this by redesigning the annotation task to achieve higher inter-rater agreement, and by defining a performance measure which takes the reliability of each annotation decision in the dataset into account.

UR - http://www.scopus.com/inward/record.url?scp=85121214618&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85121214618

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 106

EP - 110

BT - Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016

PB - Association for Computational Linguistics (ACL)

T2 - 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016

Y2 - 7 August 2016

ER -

Avraham O, Goldberg Y. Improving reliability of word similarity evaluation by redesigning annotation task and performance measure. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Association for Computational Linguistics (ACL). 2016. p. 106-110. (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Improving reliability of word similarity evaluation by redesigning annotation task and performance measure

Abstract

Publication series

Conference

Bibliographical note

Other files and links

Fingerprint

Cite this