aaargh the "When recurrent models don't need to be recurrent" paper is so frustrating!
— (((ل()(ل() 'yoav)))) (@yoavgo) August 8, 2018
On the one hand it presents important technical results.
On the other, so many people interpret it as "yo lets replace all RNNs with FF nets". This is wrong. This is NOT the result.
Also, many people seem to hold both of the following beliefs at the same time:
— (((ل()(ل() 'yoav)))) (@yoavgo) August 8, 2018
- ha cool we can do language models with feed-forward nets instead of RNNs!
- if we do LM well we will model all of language and achieve AGI!
It doesn't work this way. These are conflicting.
you want to replace unlimited history with size-k history because you think it is a good enough approximation? by all means please do. but don't expect to magically capture also the long-range stuff. it doesn't work this way. it cannot work this way.
— (((ل()(ل() 'yoav)))) (@yoavgo) August 8, 2018
