MultiQT: Multimodal Learning for Real-Time Question Tracking in Speech

Jakob D. Havtorn, Jan Latko, Joakim Edin, Lasse Borgholt, Lars Maaloe, Lorenzo Belgrano, Nicolai F. Jacobsen, Regitze Sdun, Zeljko Agic

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

56 Downloads (Orbit)

Abstract

We address a challenging and practical task of labeling questions in speech in real time during telephone calls to emergency medical services in English, which embeds within a broader decision support system for emergency call-takers. We propose a novel multimodal approach to real-time sequence labeling in speech. Our model treats speech and its own textual representation as two separate modalities or views, as it jointly learns from streamed audio and its noisy transcription into text via automatic speech recognition. Our results show significant gains of jointly learning from the two modalities when compared to text or audio only, under adverse noise and limited volume of training data. The results generalize to medical symptoms detection where we observe a similar pattern of improvements with multimodal learning.
Original languageEnglish
Title of host publicationProceedings of 58th Annual Meeting of the Association for Computational Linguistics
Publication date2020
Pages2370-2380
Publication statusPublished - 2020
Event58th Annual Meeting of the Association for
Computational Linguistics
- Onlibe event
Duration: 6 Jul 202010 Jul 2020
https://acl2020.org/

Conference

Conference58th Annual Meeting of the Association for
Computational Linguistics
LocationOnlibe event
Period06/07/202010/07/2020
Internet address

Fingerprint

Dive into the research topics of 'MultiQT: Multimodal Learning for Real-Time Question Tracking in Speech'. Together they form a unique fingerprint.

Cite this