Hybrid models for hateful memes classification

Abstract

Over recent years hate speech has rapidly spread over the internet, concretely this work centered on Hateful Memes dataset, a problem that requires both visual and linguistic understanding of the meme to classify it correctly. In order to detect hate speech easily, AI can help. This work has presented a method that can help with it. The approach consists of an architecture that starts in-painting the memes from text, then extracting relevant features (including race), and finally training two models based on transformers (BERT and ERNIE-VIl) in order to ensemble it and get the hatefulness of the meme. At the end of the work, the results are analyzed in order to help the community that deals with similar multimodal problems