Wav2Pix: speech-conditioned face generation using generative adversarial networks
View/Open
Preprint (4,422Mb) (Restricted access)
Request copy
Què és aquest botó?
Aquest botó permet demanar una còpia d'un document restringit a l'autor. Es mostra quan:
- Disposem del correu electrònic de l'autor
- El document té una mida inferior a 20 Mb
- Es tracta d'un document d'accés restringit per decisió de l'autor o d'un document d'accés restringit per política de l'editorial
Cita com:
hdl:2117/167073
Document typeConference lecture
Defense date2019
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Rights accessRestricted access - publisher's policy
European Commission's projectINPhINIT - Innovative doctoral programme for talented early-stage researchers in Spanish host organisations excellent in the areas of Science, Technology, Engineering and Mathematics (STEM). (EC-H2020-713673)
Abstract
Speech is a rich biometric signal that contains information about the identity, gender and emotional state of the speaker. In this work, we explore its potential to generate face images of a speaker by conditioning a Generative Adversarial Network (GAN) with raw speech input. We propose a deep neural network that is trained from scratch in an end-to-end fashion, generating a face directly from the raw speech waveform without any additional identity information (e.g reference image or one-hot encoding). Our model is trained in a self-supervised approach by exploiting the audio and visual signals naturally aligned in videos. With the purpose of training from video data, we present a novel dataset collected for this work, with high-quality videos of youtubers with notable expressiveness in both the speech and visual signals.
CitationCardoso, A. [et al.]. Wav2Pix: speech-conditioned face generation using generative adversarial networks. A: IEEE International Conference on Acoustics, Speech, and Signal Processing. "2019 IEEE International Conference on Acoustics, Speech, and Signal Processing: proceedings: May 12-17, 2019: Brighton Conference Centre, Brighton, United Kingdom". Institute of Electrical and Electronics Engineers (IEEE), 2019, p. 8633-8637.
ISBN978-1-4799-8131-1
Publisher versionhttps://ieeexplore.ieee.org/document/8682970
Collections
- Doctorat en Teoria del Senyal i Comunicacions - Ponències/Comunicacions de congressos [94]
- GPI - Grup de Processament d'Imatge i Vídeo - Ponències/Comunicacions de congressos [310]
- CAP - Grup de Computació d'Altes Prestacions - Ponències/Comunicacions de congressos [692]
- VEU - Grup de Tractament de la Parla - Ponències/Comunicacions de congressos [412]
- Departament d'Arquitectura de Computadors - Ponències/Comunicacions de congressos [1.635]
- Departament de Teoria del Senyal i Comunicacions - Ponències/Comunicacions de congressos [3.066]
Files | Description | Size | Format | View |
---|---|---|---|---|
1903.10195.pdf![]() | Preprint | 4,422Mb | Restricted access |
All rights reserved. This work is protected by the corresponding intellectual and industrial
property rights. Without prejudice to any existing legal exemptions, reproduction, distribution, public
communication or transformation of this work are prohibited without permission of the copyright holder