ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
— ML Review (@ml_review) September 27, 2019
Parameter reduction techniques:
(i) Vocabulary embedding matrix factorization. Separates hidden layers size from vocabulary embedding
(ii) cross-layer parameter sharinghttps://t.co/dUamPpHLt1 pic.twitter.com/ehcwdYN6mE
P.S. See also the ALBERT paper, which shows stronger results on these metrics for a similarly sized model, using a different approach (using parameters better to begin with vs. compressing a big model later): https://t.co/4VY2gbuFQu
— Miles Brundage (@Miles_Brundage) September 28, 2019
ALBERT is a new, open-source architecture for natural language processing that achieves state-of-the-art performance on multiple benchmarks with ~30% fewer parameters than #BERT. Learn all about it below: https://t.co/oYriFcwrQo
— Google AI (@GoogleAI) December 20, 2019
