Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark

Stephen Mayhew; Terra Blevins; Shuheng Liu; Marek Šuppa; Hila Gonen; Joseph Marvin Imperial; Börje F. Karlsson; Peiqin Lin; Nikola Ljubešić; L. J. Miranda; Barbara Plank; Arij Riabi; Yuval Pinter

Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark

Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Šuppa, Hila Gonen, Joseph Marvin Imperial, Börje F. Karlsson, Peiqin Lin, Nikola Ljubešić, L. J. Miranda, Barbara Plank, Arij Riabi, Yuval Pinter

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. The overarching goal of UNER is to provide high-quality, cross-lingually consistent annotations to facilitate and standardize multilingual NER research. UNER v1 contains 19 datasets annotated with named entities in a cross-lingual consistent schema across 13 diverse languages. In this paper, we detail the dataset creation and composition of UNER; we also provide initial modeling baselines on both in-language and cross-lingual learning settings. We will release the data, code, and fitted models to the public.

Original language	English
Title of host publication	Long Papers
Editors	Kevin Duh, Helena Gomez, Steven Bethard
Publisher	Association for Computational Linguistics (ACL)
Pages	4322-4337
Number of pages	16
ISBN (Electronic)	9798891761148
State	Published - 2024
Externally published	Yes
Event	2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024 - Hybrid, Mexico City, Mexico Duration: 16 Jun 2024 → 21 Jun 2024

Publication series

Name	Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024
Volume	1

Conference

Conference	2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024
Country/Territory	Mexico
City	Hybrid, Mexico City
Period	16/06/24 → 21/06/24

Bibliographical note

Publisher Copyright:
©2024 Association for Computational Linguistics.

Cite this

Mayhew, S., Blevins, T., Liu, S., Šuppa, M., Gonen, H., Imperial, J. M., Karlsson, B. F., Lin, P., Ljubešić, N., Miranda, L. J., Plank, B., Riabi, A., & Pinter, Y. (2024). Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark. In K. Duh, H. Gomez, & S. Bethard (Eds.), Long Papers (pp. 4322-4337). (Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024; Vol. 1). Association for Computational Linguistics (ACL).

Mayhew, Stephen ; Blevins, Terra ; Liu, Shuheng et al. / Universal NER : A Gold-Standard Multilingual Named Entity Recognition Benchmark. Long Papers. editor / Kevin Duh ; Helena Gomez ; Steven Bethard. Association for Computational Linguistics (ACL), 2024. pp. 4322-4337 (Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024).

@inproceedings{8867514e82a84b8494198d3eb28f9419,

title = "Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark",

abstract = "We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. The overarching goal of UNER is to provide high-quality, cross-lingually consistent annotations to facilitate and standardize multilingual NER research. UNER v1 contains 19 datasets annotated with named entities in a cross-lingual consistent schema across 13 diverse languages. In this paper, we detail the dataset creation and composition of UNER; we also provide initial modeling baselines on both in-language and cross-lingual learning settings. We will release the data, code, and fitted models to the public.",

author = "Stephen Mayhew and Terra Blevins and Shuheng Liu and Marek {\v S}uppa and Hila Gonen and Imperial, {Joseph Marvin} and Karlsson, {B{\"o}rje F.} and Peiqin Lin and Nikola Ljube{\v s}i{\'c} and Miranda, {L. J.} and Barbara Plank and Arij Riabi and Yuval Pinter",

note = "Publisher Copyright: {\textcopyright}2024 Association for Computational Linguistics.; 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024 ; Conference date: 16-06-2024 Through 21-06-2024",

year = "2024",

language = "אנגלית",

series = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024",

publisher = "Association for Computational Linguistics (ACL)",

pages = "4322--4337",

editor = "Kevin Duh and Helena Gomez and Steven Bethard",

booktitle = "Long Papers",

address = "ארצות הברית",

}

Mayhew, S, Blevins, T, Liu, S, Šuppa, M, Gonen, H, Imperial, JM, Karlsson, BF, Lin, P, Ljubešić, N, Miranda, LJ, Plank, B, Riabi, A & Pinter, Y 2024, Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark. in K Duh, H Gomez & S Bethard (eds), Long Papers. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024, vol. 1, Association for Computational Linguistics (ACL), pp. 4322-4337, 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024, Hybrid, Mexico City, Mexico, 16/06/24.

Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark. / Mayhew, Stephen; Blevins, Terra; Liu, Shuheng et al.
Long Papers. ed. / Kevin Duh; Helena Gomez; Steven Bethard. Association for Computational Linguistics (ACL), 2024. p. 4322-4337 (Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Universal NER

T2 - 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024

AU - Mayhew, Stephen

AU - Blevins, Terra

AU - Liu, Shuheng

AU - Šuppa, Marek

AU - Gonen, Hila

AU - Imperial, Joseph Marvin

AU - Karlsson, Börje F.

AU - Lin, Peiqin

AU - Ljubešić, Nikola

AU - Miranda, L. J.

AU - Plank, Barbara

AU - Riabi, Arij

AU - Pinter, Yuval

PY - 2024

Y1 - 2024

N2 - We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. The overarching goal of UNER is to provide high-quality, cross-lingually consistent annotations to facilitate and standardize multilingual NER research. UNER v1 contains 19 datasets annotated with named entities in a cross-lingual consistent schema across 13 diverse languages. In this paper, we detail the dataset creation and composition of UNER; we also provide initial modeling baselines on both in-language and cross-lingual learning settings. We will release the data, code, and fitted models to the public.

AB - We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. The overarching goal of UNER is to provide high-quality, cross-lingually consistent annotations to facilitate and standardize multilingual NER research. UNER v1 contains 19 datasets annotated with named entities in a cross-lingual consistent schema across 13 diverse languages. In this paper, we detail the dataset creation and composition of UNER; we also provide initial modeling baselines on both in-language and cross-lingual learning settings. We will release the data, code, and fitted models to the public.

UR - http://www.scopus.com/inward/record.url?scp=85199571930&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85199571930

T3 - Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024

SP - 4322

EP - 4337

BT - Long Papers

A2 - Duh, Kevin

A2 - Gomez, Helena

A2 - Bethard, Steven

PB - Association for Computational Linguistics (ACL)

Y2 - 16 June 2024 through 21 June 2024

ER -

Mayhew S, Blevins T, Liu S, Šuppa M, Gonen H, Imperial JM et al. Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark. In Duh K, Gomez H, Bethard S, editors, Long Papers. Association for Computational Linguistics (ACL). 2024. p. 4322-4337. (Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024).

Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark

Abstract

Publication series

Conference

Bibliographical note

Other files and links

Fingerprint

Cite this