"Whether to go with a decoder-only or encoder-decoder transformer?"
— Mostafa Dehghani (@m__dehghani) May 12, 2022
It turned out that this question on the architecture of the model is not actually that important!
You just need the right objective function and a simple prompting to switch mode during pretraining/finetuning. pic.twitter.com/lanaYmHynW
Autoregressive Search Engines: Generating Substrings as Document Identifiers
— AK (@ak92501) April 25, 2022
abs: https://t.co/eIR7VVbfQ7
github: https://t.co/PwAidm4h7g pic.twitter.com/KoRfty8s35
I missed that SalesForce has released a collection of code models (including weights!) ranging from 350M params all the way up to 16B params! The largest model outperforms Codex on the HumanEval dataset https://t.co/hIpg7aHTXN
— Brendan Dolan-Gavitt (@moyix) April 20, 2022
CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
— AK (@ak92501) April 19, 2022
abs: https://t.co/HirMgPmieI pic.twitter.com/L88ZBDZJZn
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
— AK (@ak92501) April 19, 2022
abs: https://t.co/wuzHfvfDHQ
github: https://t.co/dms3SfhNQo pic.twitter.com/PDd8Xfp1K9
mGPT: Few-Shot Learners Go Multilingual
— AK (@ak92501) April 19, 2022
abs: https://t.co/9uxHVoqRXO
introduces two autoregressive GPT-like models with 1.3 billion and 13 billion parameters trained on 60 languages
from 25 language families using Wikipedia and Colossal Clean Crawled Corpus pic.twitter.com/gDGX6qjv8A
InCoder: A Generative Model for Code Infilling and Synthesis
— AK (@ak92501) April 14, 2022
abs: https://t.co/qAbrJzgVkw
project page: https://t.co/Sp87l2oGix pic.twitter.com/U0iNz40ZWq
A Review on Language Models as Knowledge Bases
— AK (@ak92501) April 14, 2022
abs: https://t.co/C70a1YM8AX pic.twitter.com/Ce84fhz5yX
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
— AK (@ak92501) April 13, 2022
abs: https://t.co/Lk71qAPdzm
github: https://t.co/hIzImwwFoD pic.twitter.com/NSI294Gs7M
Chinchilla: A 70 billion parameter language model that outperforms much larger models, including Gopher. By revisiting how to trade-off compute between model & dataset size, users can train a better and smaller model. Read more: https://t.co/RaZGUclBYQ 1/3 pic.twitter.com/TNWI1RLloA
— DeepMind (@DeepMind) April 12, 2022
PaLM: Scaling Language Modeling with Pathways
— AK (@ak92501) April 6, 2022
abs: https://t.co/yWvL0NGyjB pic.twitter.com/ACu4cVqAGO
I noticed that DistilBERT loves movies filmed in India, but not in Iraq, so I plotted the result for each country: the resulting map is scary. #aibias pic.twitter.com/ayMjYEcOz6
— Aurélien Geron (@aureliengeron) March 20, 2022
