How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
George Sterpu; Christian Saam; Naomi Harte · 2020 · arXiv
WASTE classifies this as Negative / Null Result Report · AI classification, approximate
The study found no significant effect — useful as a negative control or null benchmark for your own design.
Abstract (excerpt)
Audio-Visual Speech Recognition (AVSR) seeks to model, and thereby exploit, the dynamic relationship between a human voice and the corresponding mouth movements. A recently proposed multimodal fusion strategy, AV Align, based on state-of-the-art sequence to sequence neural networks, attempts to model this relationship by explicitly aligning the acoustic and visual representations of speech. This study investigates the inner workings of AV Align and visualises the audio-visual alignment patterns. Our experiments are performed on two of the largest publicly available AVSR datasets, TCD-TIMIT and
Excerpt shown for reference under fair use — read the full paper at the publisher.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
Aggregation Kinetics of Graphene Oxides in Aqueous Solutions: Experiments, Mechanisms, and Modeling
Negative / Null Result ReportThe reaction between metakaolin and limestone and its effect in porosity refinement and mechanical properties
Negative / Null Result ReportTuning Alginate Bioink Stiffness and Composition for Controlled Growth Factor Delivery and to Spatially Direct MSC Fate within Bioprinted Tissues
Negative / Null Result ReportEx-situ characterisation of gas diffusion layers for proton exchange membrane fuel cells
Negative / Null Result ReportCobb Angle Measurement of Spine from X-Ray Images Using Convolutional Neural Network
Negative / Null Result ReportDeterminants of residential water consumption: Evidence and analysis from a 10‐country household survey
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: arXiv