Tweeted By @moi_anthony

New python v0.7.0 of 🤗 tokenizers is out, with:

- 🚀 Reduced memory usage by 70%
- 💪 Rock-solid offsets/alignments, working even with byte-level BPE
- ➕ And so much more!
And soon all of this in 🤗 transformers too!

pip install tokenizershttps://t.co/wONh4qZSlF pic.twitter.com/Io1kmLE63M

— Anthony MOI (@moi_anthony) April 17, 2020