Tweeted By @_inesmontani
Here's the built-in spaCy version for aligning different tokenization (e.g. 🤗's BERT tokens and spaCy tokens):
— Ines Montani 〰️ (@_inesmontani) July 17, 2019
```
from https://t.co/rsigVJnBju import align
cost, a2b, b2a, a2b_multi, b2a_multi = align(bert_tokens, spacy_tokens)
``` pic.twitter.com/tICtGXNSem