Shared computational principles for language processing in humans and deep language models

Ariel Goldstein; Zaid Zada; Eliav Buchnik; Mariano Schain; Amy Price; Bobbi Aubrey; Samuel A. Nastase; Amir Feder; Dotan Emanuel; Alon Cohen; Aren Jansen; Harshvardhan Gazula; Gina Choe; Aditi Rao; Catherine Kim; Colton Casto; Lora Fanda; Werner Doyle; Daniel Friedman; Patricia Dugan; Lucia Melloni; Roi Reichart; Sasha Devore; Adeen Flinker; Liat Hasenfratz; Omer Levy; Avinatan Hassidim; Michael Brenner; Yossi Matias; Kenneth A. Norman; Orrin Devinsky; Uri Hasson

doi:10.1038/s41593-022-01026-4

Shared computational principles for language processing in humans and deep language models

Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia DuganLucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, Uri Hasson

Research output: Contribution to journal › Article › peer-review

41 Scopus citations

Abstract

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

Original language	English
Pages (from-to)	369-380
Number of pages	12
Journal	Nature Neuroscience
Volume	25
Issue number	3
DOIs	https://doi.org/10.1038/s41593-022-01026-4
State	Published - Mar 2022
Externally published	Yes

Bibliographical note

Funding Information:
We thank A. Goldberg, R. Goldstein, S. Michelmann, M. Meshulam, M. Kumar, M. Slaney and A. Huth for technical and conceptual assistance that motivated and informed this paper’s writing. This work was supported by the National Institutes of Health under award numbers DP1HD091948 (to A.G., Z.Z., A.P., B.A., G.C., A.R., C.K., F.L., A.F. and U.H.), R01MH112566 (to S.A.N.) and R01NS109367-01 to A.F., Finding A Cure for Epilepsy and Seizures (FACES) and Schmidt Futures Foundation DataX Fund.

Publisher Copyright:
© 2022, The Author(s).

Access to Document

10.1038/s41593-022-01026-4

Cite this

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., Jansen, A., Gazula, H., Choe, G., Rao, A., Kim, C., Casto, C., Fanda, L., Doyle, W., Friedman, D., ... Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3), 369-380. https://doi.org/10.1038/s41593-022-01026-4

@article{e700cfd75a2d4598b82c10930e7987a2,

title = "Shared computational principles for language processing in humans and deep language models",

abstract = "Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.",

author = "Ariel Goldstein and Zaid Zada and Eliav Buchnik and Mariano Schain and Amy Price and Bobbi Aubrey and Nastase, {Samuel A.} and Amir Feder and Dotan Emanuel and Alon Cohen and Aren Jansen and Harshvardhan Gazula and Gina Choe and Aditi Rao and Catherine Kim and Colton Casto and Lora Fanda and Werner Doyle and Daniel Friedman and Patricia Dugan and Lucia Melloni and Roi Reichart and Sasha Devore and Adeen Flinker and Liat Hasenfratz and Omer Levy and Avinatan Hassidim and Michael Brenner and Yossi Matias and Norman, {Kenneth A.} and Orrin Devinsky and Uri Hasson",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s).",

year = "2022",

month = mar,

doi = "10.1038/s41593-022-01026-4",

language = "אנגלית",

volume = "25",

pages = "369--380",

journal = "Nature Neuroscience",

issn = "1097-6256",

publisher = "Nature Publishing Group",

number = "3",

}

Goldstein, A, Zada, Z, Buchnik, E, Schain, M, Price, A, Aubrey, B, Nastase, SA, Feder, A, Emanuel, D, Cohen, A, Jansen, A, Gazula, H, Choe, G, Rao, A, Kim, C, Casto, C, Fanda, L, Doyle, W, Friedman, D, Dugan, P, Melloni, L, Reichart, R, Devore, S, Flinker, A, Hasenfratz, L, Levy, O, Hassidim, A, Brenner, M, Matias, Y, Norman, KA, Devinsky, O & Hasson, U 2022, 'Shared computational principles for language processing in humans and deep language models', Nature Neuroscience, vol. 25, no. 3, pp. 369-380. https://doi.org/10.1038/s41593-022-01026-4

TY - JOUR

T1 - Shared computational principles for language processing in humans and deep language models

AU - Goldstein, Ariel

AU - Zada, Zaid

AU - Buchnik, Eliav

AU - Schain, Mariano

AU - Price, Amy

AU - Aubrey, Bobbi

AU - Nastase, Samuel A.

AU - Feder, Amir

AU - Emanuel, Dotan

AU - Cohen, Alon

AU - Jansen, Aren

AU - Gazula, Harshvardhan

AU - Choe, Gina

AU - Rao, Aditi

AU - Kim, Catherine

AU - Casto, Colton

AU - Fanda, Lora

AU - Doyle, Werner

AU - Friedman, Daniel

AU - Dugan, Patricia

AU - Melloni, Lucia

AU - Reichart, Roi

AU - Devore, Sasha

AU - Flinker, Adeen

AU - Hasenfratz, Liat

AU - Levy, Omer

AU - Hassidim, Avinatan

AU - Brenner, Michael

AU - Matias, Yossi

AU - Norman, Kenneth A.

AU - Devinsky, Orrin

AU - Hasson, Uri

PY - 2022/3

Y1 - 2022/3

N2 - Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

AB - Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

UR - http://www.scopus.com/inward/record.url?scp=85126078487&partnerID=8YFLogxK

U2 - 10.1038/s41593-022-01026-4

DO - 10.1038/s41593-022-01026-4

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

C2 - 35260860

AN - SCOPUS:85126078487

SN - 1097-6256

VL - 25

SP - 369

EP - 380

JO - Nature Neuroscience

JF - Nature Neuroscience

IS - 3

ER -

Shared computational principles for language processing in humans and deep language models

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this