Application of deep learning general-purpose neural architectures based on vision transformers for ISIC melanoma classification

Abstract

The field of computer vision has for years been dominated by Convolutional Neural Networks (CNNs) in the medical field. However, there are various other Deep Learning (DL) techniques that have become very popular in this space. Vision Transformers (ViTs) are an example of a deep learning technique that has been gaining in popularity in recent years. In this work, we study the performance of ViTs and CNNs on skin lesions classification tasks, specifically melanoma diagnosis. We compare the performance of ViTs to that of CNNs and show that regardless of the performance of both architectures, an ensemble of the two can improve generalization. We also present an adaptation to the Gram-OOD* method (detecting Out-of-distribution (OOD) using Gram matrices) for skin lesion images. A rescaling method was also used to address the imbalanced dataset problem, which is generally inherent in medical images. The phenomenon of super-convergence was critical to our success in building models with computing and training time constraints. Finally, we train and evaluate an ensemble of ViTs and CNNs, demonstrating that generalization is enhanced by placing first in the 2019 and third in the 2022 ISIC Challenge Live. Leaderboard (available at \href{https://challenge.isic-archive.com/leaderboards/live/}{https://challenge.isic-archive.com/leaderboards/live/}).