Breakingnews: article annotation by image and text processing
Cita com:
hdl:2117/120594
Tipus de documentArticle
Data publicació2018-05-01
EditorInstitute of Electrical and Electronics Engineers (IEEE)
Condicions d'accésAccés obert
Llevat que s'hi indiqui el contrari, els
continguts d'aquesta obra estan subjectes a la llicència de Creative Commons
:
Reconeixement-NoComercial-SenseObraDerivada 3.0 Espanya
ProjecteROBINSTRUCT: INSTRUYENDO ROBOTS USANDO HABILIDADES COMUNICATIVAS NATURALES (MINECO-TIN2014-58178-R)
Abstract
Building upon recent Deep Neural Network architectures, current approaches lying in the intersection of Computer Vision and Natural Language Processing have achieved unprecedented breakthroughs in tasks like automatic captioning or image retrieval. Most of these learning methods, though, rely on large training sets of images associated with human annotations that specifically describe the visual content. In this paper we propose to go a step further and explore the more complex cases where textual descriptions are loosely related to the images. We focus on the particular domain of news articles in which the textual content often expresses connotative and ambiguous relations that are only suggested but not directly inferred from images. We introduce an adaptive CNN architecture that shares most of the structure for multiple tasks including source detection, article illustration and geolocation of articles. Deep Canonical Correlation Analysis is deployed for article illustration, and a new loss function based on Great Circle Distance is proposed for geolocation. Furthermore, we present BreakingNews, a novel dataset with approximately 100K news articles including images, text and captions, and enriched with heterogeneous meta-data (such as GPS coordinates and user comments). We show this dataset to be appropriate to explore all aforementioned problems, for which we provide a baseline performance using various Deep Learning architectures, and different representations of the textual and visual features. We report very promising results and bring to light several limitations of current state-of-the-art in this kind of domain, which we hope will help spur progress in the field.
Descripció
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
CitacióRamisa, A., Yan, F., Moreno-Noguer, F., Mikolajczyk, K. Breakingnews: article annotation by image and text processing. "IEEE transactions on pattern analysis and machine intelligence", 1 Maig 2018, vol. 40, núm. 5, p. 1072-1085.
ISSN0162-8828
Versió de l'editorhttps://ieeexplore.ieee.org/document/7964736/
Fitxers | Descripció | Mida | Format | Visualitza |
---|---|---|---|---|
1967-BreakingNe ... ge-and-Text-Processing.pdf | 4,492Mb | Visualitza/Obre |