中英文对照学习，效果更佳！
原课程链接：https://huggingface.co/course/chapter6/10?fw=pt

Tokenizers, check!

托克尼茨人，检查！

问一个问题

Great job finishing this chapter!

完成这一章的工作做得很好！

After this deep dive into tokenizers, you should:

在深入研究了令牌化之后，您应该：

Be able to train a new tokenizer using an old one as a template
Understand how to use offsets to map tokensâ€™ positions to their original span of text
Know the differences between BPE, WordPiece, and Unigram
Be able to mix and match the blocks provided by the 🤗 Tokenizers library to build your own tokenizer
Be able to use that tokenizer inside the 🤗 Transformers library

Transformer

#Course