Tweeted By @karpathy

Good post on the use of BPE (byte pair encodings) for I/O of language models, pointing out subtle under-the-hood detail with unintuitive repercussions https://t.co/vZ5R5lqteP e.g. hello,Hello,HELLO all tokenize completely differently, and possibly also of different # tokens each

— Andrej Karpathy (@karpathy) July 28, 2020