Today we unveiled the latest version of Aristo, an AI system capable of scoring over 90% on an 8th grade science exam – this achievement is the result of years of research into machine reasoning and natural language processing.https://t.co/Bk9zWVHJYB
— Allen Institute for Artificial Intelligence (AI2) (@allen_ai) September 4, 2019
by @CadeMetz via @nytimes
It's here! I've written a (lengthy!) blog post on how to finetune GPT-2 and generate text using gpt-2-simple, along with a history of GPT-2 finetuning and its future. https://t.co/kqLMXdL9IE
— Max Woolf (@minimaxir) September 4, 2019
🌟Pytorch-Transformers 1.2.0 is out!🌟
— Thomas Wolf (@Thom_Wolf) September 4, 2019
A new architecture:
- DistilBERT from @huggingface 👉https://t.co/5ANj4OyJfL
Five new pretrained checkpoints:
- DistilBERT base & SQuAD
- GPT2 large 774M
- XLM multilingual 17 & 100 languages
So many fixes/improvements from the community! pic.twitter.com/1hCDKfJpwC
I see many people spent hours of compute training Bert to get worse results that they would get on fraction of cost using ULMFIT or even naive bias. Re IMDb ulmfit has 95% accuracy and 55m parameters, less when sentence-piece & qrnnnis used.
— Piotr Czapla (@PiotrCzapla) September 1, 2019
Dimensions of Dialogue
— hardmaru (@hardmaru) August 30, 2019
New writing systems are created by challenging two neural networks to communicate information using images. Results look really interesting!https://t.co/NtBjcyZWv4 https://t.co/HpsNkOppkG pic.twitter.com/CGhFqR45kg
Our DistilBert post links to papers to get you started: https://t.co/5ANj4Oh7Rb
— Thomas Wolf (@Thom_Wolf) August 30, 2019
It can be surprisingly difficult to get energy-efficient papers accepted, in part because of the current unreasonable focus on SOTA only.
I think it's time for conferences & workshop to tackle this
I'm so excited about our current projects on energy-efficient NLP!
— Thomas Wolf (@Thom_Wolf) August 30, 2019
Distilled models are very complementary to larger models. Training cost make headlines but as large-scale models reach production, inference time will likely account for most of a model's total environmental cost pic.twitter.com/K81ZeGg2Yy
Probing Representations Learned by Multimodal Recurrent and Transformer Models
— Thomas Lahore (@evolvingstuff) August 30, 2019
"while Transformer models achieve superior machine translation quality, representations from the RNNs perform significantly better over tasks focused on semantic relevance"https://t.co/SxSJDDsP7t
Now that we know it's possible to achieve comparable results to BERT using only 66M parameters, can someone find a way to train a 66M param model from scratch instead of distilling? https://t.co/ycJjMwSwsr
— Chip Huyen (@chipro) August 28, 2019
#EMNLP2019 paper "A Little Annotation does a Lot of Good: A Study in Bootstrapping Low-resource NER", examines cross-lingual transfer and active learning to create NER systems in new languages: https://t.co/HEiu8tb2X0 pic.twitter.com/gY1B9M5mF3
— Graham Neubig (@gneubig) August 27, 2019
Transfer Learning works great for Natural Language Processing, but sometimes its large models can be hard to handle. At NLP Town we used model distillation to train @spacy_io text classifiers that rival BERT for sentiment analysishttps://t.co/mLvJnYou0R #deeplearning #NLProc pic.twitter.com/lcl3bQ9S8F
— Yves Peirsman (@yvespeirsman) August 27, 2019
"Are We Safe Yet? The Limitations of Distributional Features for Fake News Detection," Schuster et al.: https://t.co/sD0sKh0OLB
— Miles Brundage (@Miles_Brundage) August 27, 2019
