The same context for next-word predictions made by n-gram and word2vec models.
How each model captures, and where it differs from another model.
| Model | Represents context as | Word order? | Context length | Output | Key limitation |
|---|---|---|---|---|---|
| Bag-of-Words | Counting vector over vocab | No | Whole doc (unordered) | Sparse vector | Loses all order & meaning |
| N-grams | Counts of word sequences | Local | Previous n−1 words | Next-word probabilities | Sparse; tiny context |
| Word2Vec | Dense learned vectors | No | Training window only | One vector per word | Context-free word meaning |
| Transformer | Attention over all tokens | Yes | Whole sequence, direct | Contextual vectors / next token | Compute grows with length² |