The decay of solutions in nonsimple elasticity with memory is addressed, analyzing how the decay rate is influenced by the different dissipation mechanisms appearing in the equations. In particular, a first order dissipation is shown to guarantee the asymptotic stability of the related solution semigroup, but is not strong enough to entail exponential stability. The latter occurs for a dissipation mechanism of the second order, that is, the same order as the one of the leading operator.