Seeing and hearing egocentric actions: how much can we learn?
Document typeConference report
Rights accessOpen Access
Our interaction with the world is an inherently multi-modal experience. However, the understanding of human-to-object interactions has historically been addressed focusing on a single modality. In particular, a limited number of works have considered to integrate the visual and audio modalities for this purpose. In this work, we propose a multimodal approach for egocentric action recognition in a kitchen environment that relies on audio and visual information. Our model combines a sparse temporal sampling strategy with a late fusion of audio, spatial,and temporal streams. Experimental results on the EPIC-Kitchens dataset show that multimodal integration leads to better performance than unimodal approaches. In particular, we achieved a5.18%improvement over the state of the art on verb classification.
© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
CitationCartas, A. [et al.]. Seeing and hearing egocentric actions: how much can we learn?. A: ICCVW - IEEE International Conference on Computer Vision Workshops. "2019 International Conference on Computer Vision ICCV 2019: proceedings: 27 October - 2 November 2019 Seoul, Korea". 2019, p. 4470-4480.