Type text to tokenise!
Tokenisation is the process of splitting a stream of text into smaller pieces called tokens — words, subwords, characters or symbols. It's the first step in almost every NLP pipeline, because a model can't work with raw text directly: each token is mapped to a number before it ever reaches the network.
The choice trades off vocabulary size against token sequence length and how unseen words are handled
Byte Pair Encoding starts from individual characters and repeatedly merges the most frequent adjacent pair into a new token. Balances vocabulary size against sequence length, and falls back gracefully on rare words. Used by the GPT family.
A BERT-style subword method. Like BPE, but it picks the merge that most improves the likelihood of the
training data rather than the most frequent one. Continuation pieces are marked with ##.
Treats text as a raw character stream with no pre-tokenisation, encoding spaces as ▁. Fully
reversible and language-agnostic — great for scripts that don't separate words with spaces. Usually paired
with a Unigram model.
Simple, training-free splits. Word tokenisation breaks on spaces and punctuation —
intuitive, but the vocabulary balloons and unseen words become unknowns. Sentence
tokenisation splits at . ! ? (guarding against abbreviations), often as
a first pass. Character tokenisation makes every character a token — a tiny vocabulary with
nothing ever out-of-vocabulary, at the cost of very long sequences.