Learning by Back-Propagation: a systolic algorithm and its transputer implementation

Abstract

In this paper we present a systolic algorithm for back-propagation, a supervised, iterative, gradient-descent, connectionist learning rule. The algorithm works on feedforward networks where connections can skip layers and fully exploits spatial and training parallelisms, which are inherent to back-propagation. Spatial parallelism arises during the propagation of activity -forward- and error -backward- for a particular input-output pair. On the other hand, when this computation is carried out simultaneously for all input-output pairs, training parallelism is obtained. In the spatial dimension, a single systolic ring carries out sequentially the three main steps of the learning rule -forward, backward and weight increments update. Furthermore, the same pattern of matrix delivery is used in both the forward and the backward passes. In this manner, the algorithm preserves the similarity of the forward and backward passes in the original model. The resulting systolic algorithm is dual with respect to the pattern of matrix delivery -either columns or rows. Finally, an implementation of the systolic algorithm for the spatial dimension is derived, that uses a linear ring of Transputer processors.