Analysis and improvement proposal on self-supervised deep learning

Abstract

Self-supervised learning is an emerging deep learning paradigm that aims at removing the label-dependency problems suffered by most supervised learning algorithms. Instance discrimination algorithms have proved to be very successful as they have reduced the gap between supervised and self-supervised ones to less than 5%. While most instance discrimination approaches focus on contrasting two augmentations of the same image, Neighbour Contrastive Learning approaches aim to increase the generalization of deep networks by pulling together representations from different images (neighbours) that belong to the same semantical class. However, they are limited mainly by their low accuracy regarding the neighbour selection. They also suffer from reduced efficiency while using multiple neighbours. Instance discrimination algorithms have their own particularities in solving the learning problem, and combining different approaches, bringing in the best of algorithms, is very interesting. In this thesis, we propose a neighbour contrast learning method called Musketeer. This method introduces Self-attention operations to create single representations, defined as centroids, from the extracted neighbours. Directly contrasting these centroids increases the neighbour retrieval accuracy while avoiding any efficiency loss. Moreover, Musketeer combines its neighbour contrast objective with a feature redundancy reduction objective, forming a symbiosis that proves to be beneficial in the overall performance of the framework. Our proposed symbiotic approach consistently outperforms SoTA instance discrimination frameworks on popular image classification benchmarking datasets, namely, CIFAR-10, CIFAR-100 and ImageNet-100. Additionally, we build an analysis pipeline that further explores the quantitative and qualitative results, providing numerous insights into the explainability of instance discrimination approaches.