How does AI deal with languages?

Tokenisation

How do raw texts become tokens?

From Rule-based tokenisers to BPE, WordPiece, and SentencePiece.

Rule-based BPEWordPiece SentencePiece

Language Models

How do models predict the next word?

From Bag-of-words and N-grams to Word2Vec and Transformers.

Bag-of-words N-gramsWord2Vec Transformer

Evaluation

How do we measure model quality?

From precision, recall and F1 to perplexity, BLEU, ROUGE, LLM-as-judge and human evaluation.

P / R / F1Perplexity BLEU · ROUGELLM-judgeHuman eval