Electrophysiological evidence for speech-specific audiovisual integration
Baart,M. Stekelenburg,J.J. Vroomen,J.
Baart,M.
Stekelenburg,J.J.
Vroomen,J.
Abstract
Lip-read speech is integrated with heard speech at various neural levels. Here, we investigated the extent to which lip-read induced modulations of the auditory N1 and P2 (measured with EEG) are indicative of speech-specific audiovisual integration, and we explored to what extent the ERPs were modulated by phonetic audiovisual congruency. In order to disentangle speech-specific (phonetic) integration from non-speech integration, we used Sine-Wave Speech (SWS) that was perceived as speech by half of the participants (they were in speech-mode), while the other half was in non-speech mode. Results showed that the N1 obtained with audiovisual stimuli peaked earlier than the N1 evoked by auditory-only stimuli. This lip-read induced speeding up of the N1 occurred for listeners in speech and non-speech mode. In contrast, if listeners were in speech-mode, lip-read speech also modulated the auditory P2, but not if listeners were in non-speech mode, thus revealing speech-specific audiovisual binding. Comparing ERPs for phonetically congruent audiovisual stimuli with ERPs for incongruent stimuli revealed an effect of phonetic stimulus congruency that started at ~200 ms after (in)congruence became apparent. Critically, akin to the P2 suppression, congruency effects were only observed if listeners were in speech mode, and not if they were in non-speech mode. Using identical stimuli, we thus confirm that audiovisual binding involves (partially) different neural mechanisms for sound processing in speech and non-speech mode. Keywords: N1, P2, Audiovisual speech, Sine-wave speech, Audiovisual integration
Description
Date
2014
Journal Title
Journal ISSN
Volume Title
Publisher
Research Projects
Organizational Units
Journal Issue
Keywords
Citation
Baart, M, Stekelenburg, J J & Vroomen, J 2014, 'Electrophysiological evidence for speech-specific audiovisual integration', Neuropsychologia, vol. 65, pp. 115-211. https://doi.org/10.1016/j.neuropsychologia.2013.11.011
License
info:eu-repo/semantics/closedAccess