Audiovisual Integration of Consonant Clusters

Tobias Andersen, Juan Camilo Gil Carvajal

Research output: Contribution to conferenceConference abstract for conferenceResearchpeer-review

128 Downloads (Pure)

Abstract

Seeing incongruent visual speech can alter the auditory phonetic percept. In
the McGurk fusion illusion the auditory percept is a single consonant
different from both the acoustic and the visual consonant. In the McGurk
combination illusion the auditory percept contains both consonants. It
remains unclear why some audiovisual stimuli elicit combination illusions. It
is also unexplored how actual consonant combinations integrate
audiovisually. Here we investigate the integration of audiovisually congruent
and incongruent combinations of /aba/, /aga/, /ada/, /abga/, and /abda/. We
found that visual stimuli containing a bilabial component (/aba/, /abga/ and
/abda/) all facilitated perception of both acoustic consonant clusters
regardless of audiovisual congruence. This is surprising because
incongruent visual stimuli usually lead to illusory, hence incorrect,
responses. The effect was most likely caused by the visual bilabial closure
as we found a general increase in bilabial responses. Visual consonant
clusters also produced combination illusions for auditory /aga/ and /ada/
and these responses were similar to the combination illusion induced by
visual /aba/. The velar and alveolar components of visual consonant
clusters did, however, also have an effect on auditory perception as they
influenced perception of auditory /aba/ in inducing novel combination
illusions where subjects perceived /abda/. Acoustic consonant clusters
dubbed onto visual velar or alveolar stimuli created novel illusions. For
example, acoustic dubbed /abga/ dubbed onto visual /aga/ created an
illusion of hearing /agda/ or /adga/. This illusion could be due to the acoustic
/b/ and visual /g/ creating a fusion illusion of hearing /d/ while leaving
perception of acoustic /g/ unaffected. This indicates that opening, closing
and release stages of consonants can integrate differentially. We
hypothesise that this may explain why some audiovisual combinations
produce combination illusions while others produce fusion or visual
dominance illusions.
Original languageEnglish
Publication date2018
Number of pages1
Publication statusPublished - 2018
Event19th Annual International Multisensory Research Forum - Toronto, Canada
Duration: 14 Jun 201817 Jun 2018
Conference number: 19

Conference

Conference19th Annual International Multisensory Research Forum
Number19
Country/TerritoryCanada
CityToronto
Period14/06/201817/06/2018

Fingerprint

Dive into the research topics of 'Audiovisual Integration of Consonant Clusters'. Together they form a unique fingerprint.

Cite this