Training nonlinear RNNs with flat, parallelized O(1) gradients

The proposed method attacks BPTT’s sequential, unstable O(T) gradient path and reframes how expressive RNNs can be trained

1/ We have been training RNNs wrong for decades. Backpropagation through time (BPTT) forces sequential updates, creating unstable O(T) gradient paths. What if we could train highly expressive, non-linear RNNs with flat, parallelized O(1)