Vibrotactile stimulation can enhance music for cochlear implant users, but it is complicated

Cochlear implants are a surgically implanted medical device that are highly effective at restoring speech perception for those with severe hearing loss. However, the signal processing that converts incoming sound signals to electrical pulses used for stimulation do not well represent certain aspects of sound important to music, such as pitch. In general, music perception is considered a challenge for cochlear implant users (Drennan and Rubinstein, 2008; Limb and Roy, 2014;McDermott, 2004). Since tactile vibrations have been shown to improve music listening experiences (Merchel and Altinsoy, 2014) and ratings of enjoyment (Hove et al., 2020) for normal-hearing listeners, haptic devices could enhance music listening for cochlear implant users as well. The research presented in this thesis aimed to investigate how music listening experiences can be enhanced by tactile vibrations in normal-hearing listeners and cochlear implant users.
The first study aimed to investigate the importance of congruence between the tactile and auditory components of melodies for music enjoyment in normal-hearing listeners. In most commercial (Actronika, 2023; SUBPAC, 2022;Woojer, 2020; bHaptics, 2023) and research music haptic devices (Haynes et al., 2021; Karam et al., 2010; Nanayakkara et al., 2009), the tactile stimulation is congruent in some way with the acoustic music signal. However the benefit of this congruence on the listener’s music enjoyment, if anything, is not known. A simple sine wave melody was presented to participants both through headphones and through a small haptic actuator on the fingertip. Participants were instructed to rate the melody from zero to 100 based on preference, alongside versions in which certain parameters of the tactile stimulus (frequency, intensity, and/or timing) were made incongruent with the audio. It was found that participants’ ratings were most consistently and positively influenced by tactile stimulation that was congruent in timing and intensity, while ratings were not affected by frequency congruence. Therefore, results suggest that music haptic devices can increase normal-hearing listener’s music enjoyment with intensity and timing congruent tactile stimulation.
A second study repeated the first, with a cohort of both young normal-hearing listeners and older cochlear implant users, and with additional melodies. While cochlear implant users can struggle with the perception of many music-related acoustic features, timing and intensity perception is often comparable to normal-hearing listeners. Therefore, it was hypothesized that intensity and timing congruent tactile stimulation would benefit cochlear implant users’ music enjoyment as much as in normal-hearing listeners. The audio-tactile congruence of intensity and timing cues continued to positively influence preference ratings for normal-hearing listeners, however fundamental frequency congruence also improved preference ratings for melodies with large pitch changes. Cochlear implant users were split- a portion of the participants rated the stimuli in a similar way as the normal-hearing listeners, however the ratings for other cochlear implant users were seemingly unaffected by congruence changes. The results suggest that at least some portion of cochlear implant users could have their music enjoyment increased with intensity and timing congruent tactile stimulation.
The third study followed a similar methodology as the first and second study, but utilized a torso-based harness called the Multimodal Harness with five tactile actuators situated vertically up the back. The previous study showed fundamental frequency congruence between tactile stimulation and music only benefits preference ratings in songs with large pitch changes. One possible reason is the poor tactile frequency discrimination abilities of the tactile system. It was therefore hypothesized that a positive impact on preference ratings would be observed by conveying acoustic fundamental frequency with the location of tactile stimulation on the listener’s back. With the Multimodal Harness, congruence between acoustic fundamental frequency and tactile height was tested alongside timing and intensity congruence. However, it was found that the congruence between fundamental frequency and height only inconsistently increased listener preference ratings, and only for melodies with predictable pitch changes. Therefore, the results did not suggest that multiple actuators congruent with fundamental frequency would reliably enhance music enjoyment.
A final experiment was designed to explore measurable benefits of a single tactile actuator on music perception other than an increase in music enjoyment. An established audio-tactile bias called the audio-tactile loudness bias (Schürmann et al., 2004) was tested to see if it could potentially be utilized in a haptic device to increase perceived dynamic range. The audio-tactile bias consists of a loudness matching task between two low intensity level auditory tones, while one is accompanied by a fixed-level tactile vibration. In the bias, the auditory tones accompanied by the vibration are judged as louder, even though the intensity of the two tones is the same. Previously, the bias has only been tested on the hands and torso. The final study replicated the effect with stimulation on the wrist. The study also tested the audio-tactile loudness bias by accompanying both reference and comparison tones with a tactile vibration. It was shown that the responses to the task were disrupted by manipulating the tactile level of the reference. In other words, increasing the intensity of the tactile reference stimulus increased the average matched level of the auditory comparison stimulus, despite participants being told to ignore the tactile stimulation.
Together, these studies investigated the feasibility and best practices for using tactile vibrations to enhance music for cochlear implant users. While vibrotactile music enjoyment enhancement and perceived dynamic range enhancement can occur with simple tactile stimulation with a single actuator, outcomes for cochlear implant users are still highly variable.