Skip RNN: learning to skip state updates in recurrent neural networks
Visualitza/Obre
Estadístiques de LA Referencia / Recolecta
Inclou dades d'ús des de 2022
Cita com:
hdl:2117/124987
Tipus de documentText en actes de congrés
Data publicació2018-04-24
EditorBarcelona Supercomputing Center
Condicions d'accésAccés obert
Llevat que s'hi indiqui el contrari, els
continguts d'aquesta obra estan subjectes a la llicència de Creative Commons
:
Reconeixement-NoComercial-SenseObraDerivada 3.0 Espanya
Abstract
Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates whilepreserving,andsometimesevenimproving,theperformance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/.
CitacióCampos, V. [et al.]. Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks. A: BSC Severo Ochoa International Doctoral Symposium (5th: 2018: Barcelona). "Book of abstracts". Barcelona: Barcelona Supercomputing Center, 2018, p. 66-67.
Fitxers | Descripció | Mida | Format | Visualitza |
---|---|---|---|---|
28-SKIP RNN LEARNING.pdf | 849,0Kb | Visualitza/Obre |