Pick a model and see what it predicts.
The n-grams and Word2Vec are real models trained on a small news corpus.
A language model (LM) is an algorithm that assigns probabilities to sequences of words. Given some context, it answers a simple question: what is likely to come next?.
Formally, a language model estimates the probability of a sequence by the chain rule — the probability of each word given everything before it:
P(w₁, w₂, …, wₙ) = P(w₁) · P(w₂ | w₁) · P(w₃ | w₁, w₂) · … · P(wₙ | w₁ … wₙ₋₁)
Represent text as an unordered count vector over a vocabulary. Simple and fast, but skips word order completely — "dog bites man" and "man bites dog" look identical.
Predict the next word based on the previous n−1 words. Estimated by counting: P(next | context) = count(context, next) / count(context). Limited context and sparse for large n.
Learn a dense vector for every word so that words used in similar contexts have similar vectors. It captures meaning — but each word still has a single, context-free vector.
Each word learns its vector by weighting all other words in the sentence. This is the architecture of modern large language models.