Please use this identifier to cite or link to this item:
Title: Bio-Inspired Modality Fusion for Active Speaker Detection
Authors: Assunção, Gustavo 
Gonçalves, Nuno 
Menezes, Paulo 
Keywords: artificial neural networks; multi-modal perception; human–robot interaction
Issue Date: 28-Feb-2020
Publisher: MDPI AG
Project: FCT - scholarhip 2020.05620.BD 
Serial title, monograph or event: Applied Sciences (Switzerland)
Volume: 11
Issue: 8
Abstract: Human beings have developed fantastic abilities to integrate information from various sensory sources exploring their inherent complementarity. Perceptual capabilities are therefore heightened, enabling, for instance, the well-known "cocktail party" and McGurk effects, i.e., speech disambiguation from a panoply of sound signals. This fusion ability is also key in refining the perception of sound source location, as in distinguishing whose voice is being heard in a group conversation. Furthermore, neuroscience has successfully identified the superior colliculus region in the brain as the one responsible for this modality fusion, with a handful of biological models having been proposed to approach its underlying neurophysiological process. Deriving inspiration from one of these models, this paper presents a methodology for effectively fusing correlated auditory and visual information for active speaker detection. Such an ability can have a wide range of applications, from teleconferencing systems to social robotics. The detection approach initially routes auditory and visual information through two specialized neural network structures. The resulting embeddings are fused via a novel layer based on the superior colliculus, whose topological structure emulates spatial neuron cross-mapping of unimodal perceptual fields. The validation process employed two publicly available datasets, with achieved results confirming and greatly surpassing initial expectations.
ISSN: 2076-3417
DOI: 10.3390/app11083397
Rights: openAccess
Appears in Collections:FCTUC Eng.Electrotécnica - Artigos em Revistas Internacionais
I&D ISR - Artigos em Revistas Internacionais

Files in This Item:
Show full item record

Page view(s)

checked on Sep 18, 2023


checked on Sep 18, 2023

Google ScholarTM




This item is licensed under a Creative Commons License Creative Commons