Abstract
We suggest a new method for creating and using gold-standard datasets for word similarity evaluation. Our goal is to improve the reliability of the evaluation, and we do this by redesigning the annotation task to achieve higher inter-rater agreement, and by defining a performance measure which takes the reliability of each annotation decision in the dataset into account.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 106-110 |
| Number of pages | 5 |
| ISBN (Electronic) | 9781945626142 |
| State | Published - 2016 |
| Event | 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Berlin, Germany Duration: 7 Aug 2016 → … |
Publication series
| Name | Proceedings of the Annual Meeting of the Association for Computational Linguistics |
|---|---|
| ISSN (Print) | 0736-587X |
Conference
| Conference | 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 |
|---|---|
| Country/Territory | Germany |
| City | Berlin |
| Period | 7/08/16 → … |
Bibliographical note
Publisher Copyright:© 2016 Proceedings of the Annual Meeting of the Association for Computational Linguistics. All Rights Reserved.
Funding
The work was supported by the Israeli Science Foundation (grant number 1555/15). We thank Omer Levy for useful discussions.
| Funders | Funder number |
|---|---|
| Israel Science Foundation | 1555/15 |