Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling

Itai Gat, Felix Kreuk, Tu Anh Nguyen, Ann Lee, Jade Copet, Gabriel Synnaeve, Emmanuel Dupoux, Yossi Adi

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

Generative Spoken Language Modeling research focuses on optimizing speech Language Models (LMs) using raw audio recordings without accessing any textual supervision. Such speech LMs usually operate over discrete units obtained from quantizing internal representations of self-supervised models. Although such units show impressive modeling results, their robustness capabilities have not been extensively investigated. This work focuses on improving the invariance of discrete input representations to non-spoken augmentations for generative spoken language modeling. First, we formally define how to measure the robustness of such representations to various signal variations that do not alter the spoken information (e.g., time-stretch). Next, we empirically demonstrate how current state-of-the-art representation models lack robustness to such variations. To overcome this, we propose an effective and efficient method to learn invariant discrete speech representation for generative spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and optimizing the model using an iterative pseudo-labeling scheme. Our method significantly improves over the evaluated baselines when considering encoding and modeling metrics. We additionally evaluate our method on the speech-to-speech translation task, considering Spanish-English and French-English translations, and show the proposed approach outperforms the evaluated baselines.

Original languageEnglish
Title of host publication20th International Conference on Spoken Language Translation, IWSLT 2023 - Proceedings of the Conference
EditorsElizabeth Salesky, Marcello Federico, Marine Carpuat
PublisherAssociation for Computational Linguistics
Pages465-477
Number of pages13
ISBN (Electronic)9781959429845
StatePublished - 2023
Externally publishedYes
Event20th International Conference on Spoken Language Translation, IWSLT 2023 - Hybrid, Toronto, Canada
Duration: 13 Jul 202314 Jul 2023

Publication series

Name20th International Conference on Spoken Language Translation, IWSLT 2023 - Proceedings of the Conference

Conference

Conference20th International Conference on Spoken Language Translation, IWSLT 2023
Country/TerritoryCanada
CityHybrid, Toronto
Period13/07/2314/07/23

Bibliographical note

Publisher Copyright:
© IWSLT 2023.All rights reserved.

Fingerprint

Dive into the research topics of 'Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling'. Together they form a unique fingerprint.

Cite this