Abstract
Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products. The API vendors charge their users based on usage, more specifically on the number of “tokens” processed or generated by the underlying language models. What constitutes a token, however, is training data and model dependent with a large variance in the number of tokens required to convey the same information in different languages. In this work, we analyze the effect of this non-uniformity on the fairness of an API's pricing policy across languages. We conduct a systematic analysis of the cost and utility of OpenAI's language model API on multilingual benchmarks in 22 typologically diverse languages. We show evidence that speakers of a large number of the supported languages are overcharged while obtaining poorer results. These speakers tend to also come from regions where the APIs are less affordable to begin with. Through these analyses, we aim to increase transparency around language model APIs' pricing policies and encourage the vendors to make them more equitable.
| Original language | English |
|---|---|
| Title of host publication | EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings |
| Editors | Houda Bouamor, Juan Pino, Kalika Bali |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 9904-9923 |
| Number of pages | 20 |
| ISBN (Electronic) | 9798891760608 |
| DOIs | |
| State | Published - 2023 |
| Externally published | Yes |
| Event | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - Hybrid, Singapore, Singapore Duration: 6 Dec 2023 → 10 Dec 2023 |
Publication series
| Name | EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings |
|---|
Conference
| Conference | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 |
|---|---|
| Country/Territory | Singapore |
| City | Hybrid, Singapore |
| Period | 6/12/23 → 10/12/23 |
Bibliographical note
Publisher Copyright:© 2023 Association for Computational Linguistics.
Funding
We thank the members of the Tsvetshop and Noah’s ARK labs at the University of Washington for the valuable discussions and useful feedback. We thank Lucille Njoo for help with our illustrations. We also thank the reviewers and area chair for their valuable feedback. During the course of this study, S.K. was supported by Google Ph.D. Fellowship. We also gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, the Alfred P. Sloan Foundation Fellowship, and NSF grants No. IIS2125201, IIS2203097, and IIS2113530. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily state or reflect those of the United States Government or any agency thereof. We thank the members of the Tsvetshop and Noah's ARK labs at the University of Washington for the valuable discussions and useful feedback. We thank Lucille Njoo for help with our illustrations. We also thank the reviewers and area chair for their valuable feedback. During the course of this study, S.K. was supported by Google Ph.D. Fellowship. We also gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, the Alfred P. Sloan Foundation Fellowship, and NSF grants No. IIS2125201, IIS2203097, and IIS2113530. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily state or reflect those of the United States Government or any agency thereof.
| Funders | Funder number |
|---|---|
| National Science Foundation | IIS2142739 |
| Alfred P. Sloan Foundation | IIS2203097, IIS2125201, IIS2113530 |
| University of Washington |