Abstract
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIGbench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI’s GPT models, Googleinternal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit “breakthrough” behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
| Original language | English |
|---|---|
| Journal | Transactions on Machine Learning Research |
| Volume | 2023 |
| State | Published - 1 May 2023 |
| Externally published | Yes |
Bibliographical note
Publisher Copyright:© 2023, Transactions on Machine Learning Research. All rights reserved.
Other files and links
Fingerprint
Dive into the research topics of 'Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver
}
In: Transactions on Machine Learning Research, Vol. 2023, 01.05.2023.
Research output: Contribution to journal › Article › peer-review
TY - JOUR
T1 - Beyond the Imitation Game
T2 - Quantifying and extrapolating the capabilities of language models
AU - Srivastava, Aarohi
AU - Rastogi, Abhinav
AU - Rao, Abhishek
AU - Md Shoeb, Abu Awal
AU - Abid, Abubakar
AU - Fisch, Adam
AU - Brown, Adam R.
AU - Santoro, Adam
AU - Gupta, Aditya
AU - Garriga-Alonso, Adrià
AU - Kluska, Agnieszka
AU - Lewkowycz, Aitor
AU - Agarwal, Akshat
AU - Power, Alethea
AU - Ray, Alex
AU - Warstadt, Alex
AU - Kocurek, Alexander W.
AU - Safaya, Ali
AU - Tazarv, Ali
AU - Xiang, Alice
AU - Parrish, Alicia
AU - Nie, Allen
AU - Hussain, Aman
AU - Askell, Amanda
AU - Dsouza, Amanda
AU - Slone, Ambrose
AU - Rahane, Ameet
AU - Iyer, Anantharaman S.
AU - Andreassen, Anders
AU - Madotto, Andrea
AU - Santilli, Andrea
AU - Stuhlmüller, Andreas
AU - Dai, Andrew
AU - La, Andrew
AU - Lampinen, Andrew
AU - Zou, Andy
AU - Jiang, Angela
AU - Chen, Angelica
AU - Vuong, Anh
AU - Gupta, Animesh
AU - Gottardi, Anna
AU - Norelli, Antonio
AU - Venkatesh, Anu
AU - Gholamidavoodi, Arash
AU - Tabassum, Arfa
AU - Menezes, Arul
AU - Kirubarajan, Arun
AU - Mullokandov, Asher
AU - Sabharwal, Ashish
AU - Herrick, Austin
AU - Efrat, Avia
AU - Erdem, Aykut
AU - Karakas, Ayla
AU - Roberts, B. Ryan
AU - Loe, Bao Sheng
AU - Zoph, Barret
AU - Bojanowski, Bartłomiej
AU - Özyurt, Batuhan
AU - Hedayatnia, Behnam
AU - Neyshabur, Behnam
AU - Inden, Benjamin
AU - Stein, Benno
AU - Ekmekci, Berk
AU - Lin, Bill Yuchen
AU - Howald, Blake
AU - Orinion, Bryan
AU - Diao, Cameron
AU - Dour, Cameron
AU - Stinson, Catherine
AU - Argueta, Cedrick
AU - Ramírez, César Ferri
AU - Singh, Chandan
AU - Rathkopf, Charles
AU - Meng, Chenlin
AU - Baral, Chitta
AU - Wu, Chiyu
AU - Callison-Burch, Chris
AU - Waites, Chris
AU - Voigt, Christian
AU - Manning, Christopher D.
AU - Potts, Christopher
AU - Ramirez, Cindy
AU - Rivera, Clara E.
AU - Siro, Clemencia
AU - Raffel, Colin
AU - Ashcraft, Courtney
AU - Garbacea, Cristina
AU - Sileo, Damien
AU - Garrette, Dan
AU - Hendrycks, Dan
AU - Kilman, Dan
AU - Roth, Dan
AU - Freeman, Daniel
AU - Khashabi, Daniel
AU - Levy, Daniel
AU - González, Daniel Moseguí
AU - Perszyk, Danielle
AU - Hernandez, Danny
AU - Chen, Danqi
AU - Ippolito, Daphne
AU - Gilboa, Dar
AU - Dohan, David
AU - Drakard, David
AU - Jurgens, David
AU - Datta, Debajyoti
AU - Ganguli, Deep
AU - Emelin, Denis
AU - Kleyko, Denis
AU - Yuret, Deniz
AU - Chen, Derek
AU - Tam, Derek
AU - Hupkes, Dieuwke
AU - Misra, Diganta
AU - Buzan, Dilyar
AU - Mollo, Dimitri Coelho
AU - Yang, Diyi
AU - Lee, Dong Ho
AU - Schrader, Dylan
AU - Shutova, Ekaterina
AU - Cubuk, Ekin Dogus
AU - Segal, Elad
AU - Hagerman, Eleanor
AU - Barnes, Elizabeth
AU - Donoway, Elizabeth
AU - Pavlick, Ellie
AU - Rodola, Emanuele
AU - Lam, Emma
AU - Chu, Eric
AU - Tang, Eric
AU - Erdem, Erkut
AU - Chang, Ernie
AU - Chi, Ethan A.
AU - Dyer, Ethan
AU - Jerzak, Ethan
AU - Kim, Ethan
AU - Manyasi, Eunice Engefu
AU - Zheltonozhskii, Evgenii
AU - Xia, Fanyue
AU - Siar, Fatemeh
AU - Martínez-Plumed, Fernando
AU - Happé, Francesca
AU - Chollet, Francois
AU - Rong, Frieda
AU - Mishra, Gaurav
AU - Winata, Genta Indra
AU - de Melo, Gerard
AU - Kruszewski, Germán
AU - Parascandolo, Giambattista
AU - Mariani, Giorgio
AU - Wang, Gloria
AU - Jaimovitch-López, Gonzalo
AU - Betz, Gregor
AU - Gur-Ari, Guy
AU - Galijasevic, Hana
AU - Kim, Hannah
AU - Rashkin, Hannah
AU - Hajishirzi, Hannaneh
AU - Mehta, Harsh
AU - Bogar, Hayden
AU - Shevlin, Henry
AU - Schütze, Hinrich
AU - Yakura, Hiromu
AU - Zhang, Hongming
AU - Wong, Hugh Mee
AU - Ng, Ian
AU - Noble, Isaac
AU - Jumelet, Jaap
AU - Geissinger, Jack
AU - Kernion, Jackson
AU - Hilton, Jacob
AU - Lee, Jaehoon
AU - Fisac, Jaime Fernández
AU - Simon, James B.
AU - Koppel, James
AU - Zheng, James
AU - Zou, James
AU - Kocon, Jan
AU - Thompson, Jana
AU - Wingfield, Janelle
AU - Kaplan, Jared
AU - Radom, Jarema
AU - Sohl-Dickstein, Jascha
AU - Phang, Jason
AU - Wei, Jason
AU - Yosinski, Jason
AU - Novikova, Jekaterina
AU - Bosscher, Jelle
AU - Marsh, Jennifer
AU - Kim, Jeremy
AU - Taal, Jeroen
AU - Engel, Jesse
AU - Alabi, Jesujoba
AU - Xu, Jiacheng
AU - Song, Jiaming
AU - Tang, Jillian
AU - Waweru, Joan
AU - Burden, John
AU - Miller, John
AU - Balis, John U.
AU - Batchelder, Jonathan
AU - Berant, Jonathan
AU - Frohberg, Jörg
AU - Rozen, Jos
AU - Hernandez-Orallo, Jose
AU - Boudeman, Joseph
AU - Guerr, Joseph
AU - Jones, Joseph
AU - Tenenbaum, Joshua B.
AU - Rule, Joshua S.
AU - Chua, Joyce
AU - Kanclerz, Kamil
AU - Livescu, Karen
AU - Krauth, Karl
AU - Gopalakrishnan, Karthik
AU - Ignatyeva, Katerina
AU - Markert, Katja
AU - Dhole, Kaustubh D.
AU - Gimpel, Kevin
AU - Omondi, Kevin
AU - Mathewson, Kory
AU - Chiafullo, Kristen
AU - Shkaruta, Ksenia
AU - Shridhar, Kumar
AU - McDonell, Kyle
AU - Richardson, Kyle
AU - Reynolds, Laria
AU - Gao, Leo
AU - Zhang, Li
AU - Dugan, Liam
AU - Qin, Lianhui
AU - Contreras-Ochando, Lidia
AU - Morency, Louis Philippe
AU - Moschella, Luca
AU - Lam, Lucas
AU - Noble, Lucy
AU - Schmidt, Ludwig
AU - He, Luheng
AU - Colón, Luis Oliveros
AU - Metz, Luke
AU - Senel, Lütfi Kerem
AU - Bosma, Maarten
AU - Sap, Maarten
AU - Hoeve, Maartje Ter
AU - Farooqi, Maheen
AU - Faruqui, Manaal
AU - Mazeika, Mantas
AU - Baturan, Marco
AU - Marelli, Marco
AU - Maru, Marco
AU - Ramírez Quintana, Maria Jose
AU - Tolkiehn, Marie
AU - Giulianelli, Mario
AU - Lewis, Martha
AU - Potthast, Martin
AU - Leavitt, Matthew L.
AU - Hagen, Matthias
AU - Schubert, Mátyás
AU - Baitemirova, Medina Orduna
AU - Arnaud, Melody
AU - McElrath, Melvin
AU - Yee, Michael A.
AU - Cohen, Michael
AU - Gu, Michael
AU - Ivanitskiy, Michael
AU - Starritt, Michael
AU - Strube, Michael
AU - Swedrowski, Michał
AU - Bevilacqua, Michele
AU - Yasunaga, Michihiro
AU - Kale, Mihir
AU - Cain, Mike
AU - Xu, Mimee
AU - Suzgun, Mirac
AU - Walker, Mitch
AU - Tiwari, Mo
AU - Bansal, Mohit
AU - Aminnaseri, Moin
AU - Geva, Mor
AU - Gheini, Mozhdeh
AU - Mukund Varma, T.
AU - Peng, Nanyun
AU - Chi, Nathan A.
AU - Lee, Nayeon
AU - Gur-Ari Krakover, Neta
AU - Cameron, Nicholas
AU - Roberts, Nicholas
AU - Doiron, Nick
AU - Martinez, Nicole
AU - Nangia, Nikita
AU - Deckers, Niklas
AU - Muennighoff, Niklas
AU - Keskar, Nitish Shirish
AU - Iyer, Niveditha S.
AU - Constant, Noah
AU - Fiedel, Noah
AU - Wen, Nuan
AU - Zhang, Oliver
AU - Agha, Omar
AU - Elbaghdadi, Omar
AU - Levy, Omer
AU - Evans, Owain
AU - Moreno Casares, Pablo Antonio
AU - Doshi, Parth
AU - Fung, Pascale
AU - Liang, Paul Pu
AU - Vico, Paul
AU - Alipoormolabashi, Pegah
AU - Liao, Peiyuan
AU - Liang, Percy
AU - Chang, Peter
AU - Eckersley, Peter
AU - Htut, Phu Mon
AU - Hwang, Pinyu
AU - Miłkowski, Piotr
AU - Patil, Piyush
AU - Pezeshkpour, Pouya
AU - Oli, Priti
AU - Mei, Qiaozhu
AU - Lyu, Qing
AU - Chen, Qinlang
AU - Banjade, Rabin
AU - Rudolph, Rachel Etta
AU - Gabriel, Raefer
AU - Habacker, Rahel
AU - Risco, Ramon
AU - Millière, Raphaël
AU - Garg, Rhythm
AU - Barnes, Richard
AU - Saurous, Rif A.
AU - Arakawa, Riku
AU - Raymaekers, Robbe
AU - Frank, Robert
AU - Sikand, Rohan
AU - Novak, Roman
AU - Sitelew, Roman
AU - Lebras, Ronan
AU - Liu, Rosanne
AU - Jacobs, Rowan
AU - Zhang, Rui
AU - Salakhutdinov, Ruslan
AU - Chi, Ryan
AU - Lee, Ryan
AU - Stovall, Ryan
AU - Teehan, Ryan
AU - Yang, Rylan
AU - Singh, Sahib
AU - Mohammad, Saif M.
AU - Anand, Sajant
AU - Dillavou, Sam
AU - Shleifer, Sam
AU - Wiseman, Sam
AU - Gruetter, Samuel
AU - Bowman, Samuel R.
AU - Schoenholz, Samuel S.
AU - Han, Sanghyun
AU - Kwatra, Sanjeev
AU - Rous, Sarah A.
AU - Ghazarian, Sarik
AU - Ghosh, Sayan
AU - Casey, Sean
AU - Bischoff, Sebastian
AU - Gehrmann, Sebastian
AU - Schuster, Sebastian
AU - Sadeghi, Sepideh
AU - Hamdan, Shadi
AU - Zhou, Sharon
AU - Srivastava, Shashank
AU - Shi, Sherry
AU - Singh, Shikhar
AU - Asaadi, Shima
AU - Gu, Shixiang Shane
AU - Pachchigar, Shubh
AU - Toshniwal, Shubham
AU - Upadhyay, Shyam
AU - Debnath, Shyamolima
AU - Shakeri, Siamak
AU - Thormeyer, Simon
AU - Melzi, Simone
AU - Reddy, Siva
AU - Makini, Sneha Priscilla
AU - Lee, Soo Hwan
AU - Torene, Spencer
AU - Hatwar, Sriharsha
AU - Dehaene, Stanislas
AU - Divic, Stefan
AU - Ermon, Stefano
AU - Biderman, Stella
AU - Lin, Stephanie
AU - Prasad, Stephen
AU - Piantadosi, Steven T.
AU - Shieber, Stuart M.
AU - Misherghi, Summer
AU - Kiritchenko, Svetlana
AU - Mishra, Swaroop
AU - Linzen, Tal
AU - Schuster, Tal
AU - Li, Tao
AU - Yu, Tao
AU - Ali, Tariq
AU - Hashimoto, Tatsu
AU - Wu, Te Lin
AU - Desbordes, Théo
AU - Rothschild, Theodore
AU - Phan, Thomas
AU - Wang, Tianle
AU - Nkinyili, Tiberius
AU - Schick, Timo
AU - Kornev, Timofei
AU - Tunduny, Titus
AU - Gerstenberg, Tobias
AU - Chang, Trenton
AU - Neeraj, Trishala
AU - Khot, Tushar
AU - Shultz, Tyler
AU - Shaham, Uri
AU - Misra, Vedant
AU - Demberg, Vera
AU - Nyamai, Victoria
AU - Raunak, Vikas
AU - Ramasesh, Vinay
AU - Prabhu, Vinay Uday
AU - Padmakumar, Vishakh
AU - Srikumar, Vivek
AU - Fedus, William
AU - Saunders, William
AU - Zhang, William
AU - Vossen, Wout
AU - Ren, Xiang
AU - Tong, Xiaoyu
AU - Zhao, Xinran
AU - Wu, Xinyi
AU - Shen, Xudong
AU - Yaghoobzadeh, Yadollah
AU - Lakretz, Yair
AU - Song, Yangqiu
AU - Bahri, Yasaman
AU - Choi, Yejin
AU - Yang, Yichi
AU - Hao, Yiding
AU - Chen, Yifu
AU - Belinkov, Yonatan
AU - Hou, Yu
AU - Hou, Yufang
AU - Bai, Yuntao
AU - Seid, Zachary
AU - Zhao, Zhuoye
AU - Wang, Zijian
AU - Wang, Zijie J.
AU - Wang, Zirui
AU - Wu, Ziyi
AU - Delgado, Ramón Risco
AU - Chen, Anna
AU - Mann, Ben
AU - Olsson, Catherine
AU - Telleen-Lawton, Tim
AU - Chi, Nathan
AU - Le Bras, Ronan
AU - Rivera, Clara
AU - Gray, Alex
N1 - Publisher Copyright: © 2023, Transactions on Machine Learning Research. All rights reserved.
PY - 2023/5/1
Y1 - 2023/5/1
N2 - Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIGbench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI’s GPT models, Googleinternal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit “breakthrough” behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
AB - Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIGbench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI’s GPT models, Googleinternal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit “breakthrough” behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
UR - https://www.scopus.com/pages/publications/105012408542
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???
AN - SCOPUS:105012408542
SN - 2835-8856
VL - 2023
JO - Transactions on Machine Learning Research
JF - Transactions on Machine Learning Research
ER -