Generative Spoken Dialogue Language Modeling

Tu Anh Nguyen; Eugene Kharitonov; Jade Copet; Yossi Adi; Wei Ning Hsu; Ali Elkahky; Paden Tomasello; Robin Algayres; Benoît Sagot; Abdelrahman Mohamed; Emmanuel Dupoux

doi:10.1162/tacl_a_00545

Generative Spoken Dialogue Language Modeling

Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, Emmanuel Dupoux

Research output: Contribution to journal › Article › peer-review

17 Scopus citations

Abstract

We introduce dGSLM, the first ‘‘textless’’ model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn taking compared to a text-based cascaded model.^1,2.

Original language	English
Pages (from-to)	250-266
Number of pages	17
Journal	Transactions of the Association for Computational Linguistics
Volume	11
DOIs	https://doi.org/10.1162/tacl_a_00545
State	Published - 14 Mar 2023
Externally published	Yes

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Funding

In this work, E.D. in his academic role (EHESS, ENS-PSL, CNRS) was supported by the Agence Nationale pour la Recherche (ANR-17-EURE-0017 Frontcog, ANR-10-IDEX-0001-02 PSL*, ANR-19-P3IA-0001 PRAIRIE 3IA Institute), a grant from CIFAR (Learning in Machines and Brains). B.S. was also supported by the Agence Nationale pour la Recherche (ANR-19-P3IA-0001 PRAIRIE 3IA Institute).

Funders	Funder number
ANR-10-IDEX-0001-02	PSL, ANR-19-P3IA-0001
ANR-17-EURE-0017
Canadian Institute for Advanced Research
Agence Nationale de la Recherche

Access to Document

10.1162/tacl_a_00545

Cite this

@article{9901dd58d36c4db1a1e5de64d2ea6d36,

title = "Generative Spoken Dialogue Language Modeling",

abstract = "We introduce dGSLM, the first {\textquoteleft}{\textquoteleft}textless{\textquoteright}{\textquoteright} model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn taking compared to a text-based cascaded model.1,2.",

author = "Nguyen, {Tu Anh} and Eugene Kharitonov and Jade Copet and Yossi Adi and Hsu, {Wei Ning} and Ali Elkahky and Paden Tomasello and Robin Algayres and Beno{\^i}t Sagot and Abdelrahman Mohamed and Emmanuel Dupoux",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.",

year = "2023",

month = mar,

day = "14",

doi = "10.1162/tacl_a_00545",

language = "אנגלית",

volume = "11",

pages = "250--266",

journal = "Transactions of the Association for Computational Linguistics",

issn = "2307-387X",

publisher = "MIT Press Journals",

}

TY - JOUR

T1 - Generative Spoken Dialogue Language Modeling

AU - Nguyen, Tu Anh

AU - Kharitonov, Eugene

AU - Copet, Jade

AU - Adi, Yossi

AU - Hsu, Wei Ning

AU - Elkahky, Ali

AU - Tomasello, Paden

AU - Algayres, Robin

AU - Sagot, Benoît

AU - Mohamed, Abdelrahman

AU - Dupoux, Emmanuel

PY - 2023/3/14

Y1 - 2023/3/14

N2 - We introduce dGSLM, the first ‘‘textless’’ model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn taking compared to a text-based cascaded model.1,2.

AB - We introduce dGSLM, the first ‘‘textless’’ model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn taking compared to a text-based cascaded model.1,2.

UR - http://www.scopus.com/inward/record.url?scp=85150992661&partnerID=8YFLogxK

U2 - 10.1162/tacl_a_00545

DO - 10.1162/tacl_a_00545

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:85150992661

SN - 2307-387X

VL - 11

SP - 250

EP - 266

JO - Transactions of the Association for Computational Linguistics

JF - Transactions of the Association for Computational Linguistics

ER -

Generative Spoken Dialogue Language Modeling

Abstract

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this