Badly tuned LR decay schedules are an excellent way to silently shoot yourself in the foot. Models can often look like they are converging but it's just LR getting too low too fast. FixedLR (+optional warmup) with 1 manual decay of 10X on plateau is a safe strong baseline.
β Andrej Karpathy (@karpathy) August 27, 2021
It's time to stop making t-SNE & UMAP plots. In a new preprint w/ Tara Chari we show that while they display some correlation with the underlying high-dimension data, they don't preserve local or global structure & are misleading. They're also arbitrary.π§΅https://t.co/XkAOTKlOcs pic.twitter.com/dmFzD5RR6R
β Lior Pachter (@lpachter) August 27, 2021
When Netflix starting using machine learning to personalize thumbnails a few years ago, the algorithm learned to target by race. Now it's learned that trashy clickbait works. The only surprise is that it's taken so long. https://t.co/fhv8BRTsRm https://t.co/iU0tdx0y6z
β Arvind Narayanan (@random_walker) August 26, 2021
This.
β (((Ω()(Ω() 'yoav))))πΎ (@yoavgo) August 26, 2021
Not all uses of "NLP" in a socially sensitive area are automatically bad.
It really depends on the method you are using, how you are using it, and what you will do with the results. NLP is a *measurement tool* over text. What we measure, how we interpret: that's up to us. https://t.co/byiG2kb2YK
debiasing & similar approaches are about *mitigating* the impact of specific latent variables that are relevant to the current application and that's genuinely the best you can hope for
β Rachael Tatman (@rctatman) August 25, 2021
Back when we had an office @themarkup, our wifi password was βmath+tearsβ
β Julia Angwin (@JuliaAngwin) August 25, 2021
Todayβs investigation explains why. @eh_mah_nwel and @lkirchner use math and human stories to reveal entrenched racism in mortgage lending. https://t.co/ApkvJ61ve6
/1
One TTS Alignment To Rule Them All
β AK (@ak92501) August 25, 2021
pdf: https://t.co/yQu2GWB6uw
abs: https://t.co/cAAxzOuGGg
present an alignment framework that is broadly applicable to various TTS architectures, both autoregressive and parallel pic.twitter.com/e4tpCSNwWC
TIL π³π΅βπ«π±. This single line change sped up our data loader 10% pic.twitter.com/9gIzscrgTQ
β Andrej Karpathy (@karpathy) August 24, 2021
π Mistral: A framework for easy large-scale language model training, built with @huggingface π€ Transformers. Props to @laurel_orr1, @siddkaramcheti, and team mates. The initial technical work from the Stanford Center for Research on #FoundationModelshttps://t.co/RlmdMtij1D
β Stanford NLP Group (@stanfordnlp) August 24, 2021
How Can Increased Randomness in Stochastic Gradient Descent Improve Generalization?
β AK (@ak92501) August 24, 2021
pdf: https://t.co/Jsj1hpi3vB
abs: https://t.co/nEGOGZ2Z8v pic.twitter.com/weTbqMbQbF
Challenges and Opportunities in NLP Benchmarking
β Sebastian Ruder (@seb_ruder) August 23, 2021
Recent NLP models have outpaced the benchmarks to test for them. I provide an overview of challenges and opportunities in this blog post.https://t.co/NbVfcwGX8z
Our RemBERT model (ICLR 2021) is finally open-source and available in π€ Transformers.
β Sebastian Ruder (@seb_ruder) August 23, 2021
RemBERT is a large multilingual Transformer that outperforms XLM-R (and mT5 with similar # of params) in zero-shot transfer.
Docs: https://t.co/AKwV0UF6cT
Paper: https://t.co/TXF7qlJtUY pic.twitter.com/ytIiMOqVks
