Abstract
Natural conversation is multisensory: when we can see the speaker's face, visual speech cues improve our comprehension. The neuronal mechanisms underlying this phenomenon remain unclear. The two main alternatives are visually mediated phase modulation of neuronal oscillations (excitability fluctuations) in auditory neurons and visual input-evoked responses in auditory neurons. Investigating this question using naturalistic audiovisual speech with intracranial recordings in humans of both sexes, we find evidence for both mechanisms. Remarkably, auditory cortical neurons track the temporal dynamics of purely visual speech using the phase of their slow oscillations and phase-related modulations in broadband high-frequency activity. Consistent with known perceptual enhancement effects, the visual phase reset amplifies the cortical representation of concomitant auditory speech. In contrast to this, and in line with earlier reports, visual input reduces the amplitude of evoked responses to concomitant auditory input. We interpret the combination of improved phase tracking and reduced response amplitude as evidence for more efficient and reliable stimulus processing in the presence of congruent auditory and visual speech inputs.
| Original language | English |
|---|---|
| Pages (from-to) | 8530-8542 |
| Number of pages | 13 |
| Journal | Journal of Neuroscience |
| Volume | 40 |
| Issue number | 44 |
| DOIs | |
| State | Published - 28 Oct 2020 |
Bibliographical note
Publisher Copyright:© 2020 the authors
Funding
This work was supported by the Swiss National Science Foundation (Grants 139829, 145563, 148388, and 167836 to P.M.), the National Institute of Neurological Disorders and Stroke (Grant NS-098976 to M.S.B., C.E.S., and A.D.M.), and the Page and Otto Marx Jr Foundation to A.D.M. We thank the patients for their participation; and Erin Yeagle, Willie Walker Jr., the physicians and other professionals of the Neurosurgery Department and Neurology Department of North Shore University Hospital; and Itzik Norman and Bahar Khalighinejad for their assistance. Part of the computations for this work were performed at the University of Geneva on the Baobab cluster. This work was supported by the Swiss National Science Foundation (Grants 139829, 145563, 148388, and 167836 to P.M.), the National Institute of Neurological Disorders and Stroke (Grant NS-098976 to M.S.B., C.E.S., and A.D.M.), and the Page and Otto Marx Jr Foundation to A.D.M. We thank the patients for their participation; and Erin Yeagle, Willie Walker Jr., the physicians and other professionals of the Neurosurgery Department and Neurology Department of North Shore University Hospital; and Itzik Norman and Bahar Khalighinejad for their assistance. Part of the computations for this work were performed at the University of Geneva on the Baobab cluster. *C.E.S. and A.D.M. are senior authors and contributed equally to this work. The authors declare no competing financial interests. Correspondence should be addressed to Charles E. Schroeder at [email protected] or Ashesh D. Mehta at [email protected]. https://doi.org/10.1523/JNEUROSCI.0555-20.2020 Copyright © 2020 the authors
| Funders | Funder number |
|---|---|
| Neurosurgery Department and Neurology Department of North Shore University Hospital | |
| Page and Otto Marx Jr Foundation | |
| University of Geneva on the Baobab cluster | |
| National Institute of Neurological Disorders and Stroke | U01NS098976 |
| Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung | 145563, 167836, 148388, 139829 |
Keywords
- Audiovisual speech
- Broadband high-frequency activity
- Crossmodal stimuli
- Intracranial electroencephalography
- Neuronal oscillations
- Phase-amplitude coupling