Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs

William Sheffield; Kanishka Misra; Valentina Pyatkin; Ashwini Deo; Kyle Mahowald; Junyi Jessy Li

doi:10.18653/v1/2025.findings-acl.1117

Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs

William Sheffield
, Kanishka Misra
, Valentina Pyatkin
, Ashwini Deo
, Kyle Mahowald
, Junyi Jessy Li

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Discourse particles are crucial elements that subtly shape the meaning of text. These words, often polyfunctional, give rise to nuanced and often quite disparate semantic/discourse effects, as exemplified by the diverse uses of the particle just (e.g., exclusive, temporal, emphatic). This work investigates the capacity of LLMs to distinguish the fine-grained senses of English just, a well-studied example in formal semantics, using data meticulously created and labeled by expert linguists. Our findings reveal that while LLMs exhibit some ability to differentiate between broader categories, they struggle to fully capture more subtle nuances, highlighting a gap in their understanding of discourse particles.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	ACL 2025
Editors	Wanxiang Che, Joyce Nabende, Ekaterina Shutova, Mohammad Taher Pilehvar
Publisher	Association for Computational Linguistics (ACL)
Pages	21704-21715
Number of pages	12
ISBN (Electronic)	9798891762565
DOIs	https://doi.org/10.18653/v1/2025.findings-acl.1117
State	Published - 2025
Externally published	Yes
Event	63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025 - Vienna, Austria Duration: 27 Jul 2025 → 1 Aug 2025

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025
Country/Territory	Austria
City	Vienna
Period	27/07/25 → 1/08/25

Bibliographical note

Publisher Copyright:
© 2025 Association for Computational Linguistics.

Access to Document

10.18653/v1/2025.findings-acl.1117

Cite this

Sheffield, W., Misra, K., Pyatkin, V., Deo, A., Mahowald, K., & Li, J. J. (2025). Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs. In W. Che, J. Nabende, E. Shutova, & M. T. Pilehvar (Eds.), Findings of the Association for Computational Linguistics: ACL 2025 (pp. 21704-21715). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2025.findings-acl.1117

Sheffield, William ; Misra, Kanishka ; Pyatkin, Valentina et al. / Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs. Findings of the Association for Computational Linguistics: ACL 2025. editor / Wanxiang Che ; Joyce Nabende ; Ekaterina Shutova ; Mohammad Taher Pilehvar. Association for Computational Linguistics (ACL), 2025. pp. 21704-21715 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{3b1571f9edca45fab3d1dd5a879afeac,

title = "Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs",

abstract = "Discourse particles are crucial elements that subtly shape the meaning of text. These words, often polyfunctional, give rise to nuanced and often quite disparate semantic/discourse effects, as exemplified by the diverse uses of the particle just (e.g., exclusive, temporal, emphatic). This work investigates the capacity of LLMs to distinguish the fine-grained senses of English just, a well-studied example in formal semantics, using data meticulously created and labeled by expert linguists. Our findings reveal that while LLMs exhibit some ability to differentiate between broader categories, they struggle to fully capture more subtle nuances, highlighting a gap in their understanding of discourse particles.",

author = "William Sheffield and Kanishka Misra and Valentina Pyatkin and Ashwini Deo and Kyle Mahowald and Li, \{Junyi Jessy\}",

note = "Publisher Copyright: {\textcopyright} 2025 Association for Computational Linguistics.; 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025 ; Conference date: 27-07-2025 Through 01-08-2025",

year = "2025",

doi = "10.18653/v1/2025.findings-acl.1117",

language = "אנגלית",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "21704--21715",

editor = "Wanxiang Che and Joyce Nabende and Ekaterina Shutova and Pilehvar, \{Mohammad Taher\}",

booktitle = "Findings of the Association for Computational Linguistics",

address = "ארצות הברית",

}

Sheffield, W, Misra, K, Pyatkin, V, Deo, A, Mahowald, K & Li, JJ 2025, Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs. in W Che, J Nabende, E Shutova & MT Pilehvar (eds), Findings of the Association for Computational Linguistics: ACL 2025. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 21704-21715, 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, 27/07/25. https://doi.org/10.18653/v1/2025.findings-acl.1117

Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs. / Sheffield, William; Misra, Kanishka; Pyatkin, Valentina et al.
Findings of the Association for Computational Linguistics: ACL 2025. ed. / Wanxiang Che; Joyce Nabende; Ekaterina Shutova; Mohammad Taher Pilehvar. Association for Computational Linguistics (ACL), 2025. p. 21704-21715 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs

AU - Sheffield, William

AU - Misra, Kanishka

AU - Pyatkin, Valentina

AU - Deo, Ashwini

AU - Mahowald, Kyle

AU - Li, Junyi Jessy

PY - 2025

Y1 - 2025

N2 - Discourse particles are crucial elements that subtly shape the meaning of text. These words, often polyfunctional, give rise to nuanced and often quite disparate semantic/discourse effects, as exemplified by the diverse uses of the particle just (e.g., exclusive, temporal, emphatic). This work investigates the capacity of LLMs to distinguish the fine-grained senses of English just, a well-studied example in formal semantics, using data meticulously created and labeled by expert linguists. Our findings reveal that while LLMs exhibit some ability to differentiate between broader categories, they struggle to fully capture more subtle nuances, highlighting a gap in their understanding of discourse particles.

AB - Discourse particles are crucial elements that subtly shape the meaning of text. These words, often polyfunctional, give rise to nuanced and often quite disparate semantic/discourse effects, as exemplified by the diverse uses of the particle just (e.g., exclusive, temporal, emphatic). This work investigates the capacity of LLMs to distinguish the fine-grained senses of English just, a well-studied example in formal semantics, using data meticulously created and labeled by expert linguists. Our findings reveal that while LLMs exhibit some ability to differentiate between broader categories, they struggle to fully capture more subtle nuances, highlighting a gap in their understanding of discourse particles.

UR - https://www.scopus.com/pages/publications/105028560720

U2 - 10.18653/v1/2025.findings-acl.1117

DO - 10.18653/v1/2025.findings-acl.1117

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:105028560720

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 21704

EP - 21715

BT - Findings of the Association for Computational Linguistics

A2 - Che, Wanxiang

A2 - Nabende, Joyce

A2 - Shutova, Ekaterina

A2 - Pilehvar, Mohammad Taher

PB - Association for Computational Linguistics (ACL)

T2 - 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025

Y2 - 27 July 2025 through 1 August 2025

ER -

Sheffield W, Misra K, Pyatkin V, Deo A, Mahowald K, Li JJ. Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs. In Che W, Nabende J, Shutova E, Pilehvar MT, editors, Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics (ACL). 2025. p. 21704-21715. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2025.findings-acl.1117

Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this