Evaluation | Algorithms

Precision, Recall & F1 classification

Key idea: from the four confusion-matrix counts, precision measures how many of your positive predictions were correct, and recall measures how many of the real positives you caught. F1 is their harmonic mean.

From counts to scores

precision = TP / (TP + FP)        # of what I flagged, how much was right
recall    = TP / (TP + FN)        # of what existed, how much I found
F1        = 2 · P · R / (P + R)   # harmonic mean — punishes imbalance
accuracy  = (TP + TN) / total     # can mislead on skewed classes

Try it — edit the confusion matrix

Interactive · Precision / Recall / F1

True Pos

False Pos

False Neg

True Neg

Precision ↑Fewer false alarms

Recall ↑Fewer misses

Watch outAccuracy lies on imbalance

Perplexity probabilistic

Key idea: how surprised is the model by real text? Average the per-word surprisal (negative log-probability) and exponentiate.

Perplexity of a sentence

# Suppose the sentence is:
# "I love NLP"
H = 0
for each word wi in the sentence: # Model predicts the next word from previous words
    # Example:
    # P(I) = 0.50
    # P(love | I) = 0.20
    # P(NLP | I love) = 0.10
    H += -log2( P(wi | previous words) )
H = H / N # Average surprisal (cross-entropy)
PP = 2^H # Convert average surprisal into perplexity

Try it — sentence → per-word probabilities → perplexity

Type a sentence. A small toy model assigns each word a probability given the words before it, and those combine into a perplexity.

Interactive · Perplexity

Sentence

Per-word probability — P(word | context)

Range1 (perfect) → ∞

NeedsA language model, no reference text

Used forComparing language models

BLEU reference overlap

Key idea: reward a candidate for sharing n-grams with a reference (clipped so it can't game repetition), then penalise it for being too short. Precision-oriented — built for machine translation.

BLEU score

for n in 1..4:
    pₙ = clipped_matches(cand, ref, n) / total_ngrams(cand, n)

BP   = 1 if c > r else exp(1 − r/c)   # brevity penalty (c,r = cand/ref lengths)
BLEU = BP · exp( (1/4) · Σ log pₙ )      # geometric mean of precisions

function clipped_matches(cand, ref, n): # Count all n-grams in the candidate and reference.
    # Example (n = 1):
    # cand = "the the the"
    # ref  = "the cat the"
    # cand_counts = {"the": 3}
    # ref_counts = {"the": 2, "cat": 1}
    cand_counts = count_ngrams(cand, n)
    ref_counts  = count_ngrams(ref, n)
    matches = 0
    for each ngram in cand_counts:
        matches += min(cand_counts[ngram], ref_counts[ngram]) # "the": min(3, 2) = 2
    return matches

function total_ngrams(cand, n): # Count all candidate n-grams.
    return count_ngrams(cand, n).total_count()

Try it — candidate vs reference

Interactive · BLEU

Candidate

Reference

N-gram precisions & score

Leans onPrecision

Best forTranslation

Blind toMeaning, paraphrase

ROUGE reference overlap

Key idea: the recall-oriented version of BLEU — how much of the reference shows up in the candidate. ROUGE-N counts n-grams; ROUGE-L uses the longest common subsequence. Built for summarisation.

ROUGE-N and ROUGE-L

ROUGE-N recall = overlap_ngrams(cand, ref, n) / total_ngrams(ref, n)
ROUGE-N prec.  = overlap_ngrams(cand, ref, n) / total_ngrams(cand, n)

LCS = longest_common_subsequence(cand, ref) 
ROUGE-L recall = LCS / len(ref),  prec = LCS / len(cand)

Try it — candidate vs reference

Interactive · ROUGE

Candidate

Reference

Scores

Leans onRecall

Best forSummarisation

ROUGE-LRewards in-order overlap

LLM-as-judge model-graded

Key idea: hand a strong model a rubric and ask it to grade the candidate against the reference on each criterion. Unlike n-gram overlap it can reward valid paraphrases, and it correlates well with human judgement, but it has biases you shall be careful about.

A scoring-rubric judge

prompt = rubric + criteria + the_output_to_grade
scores = LLM(prompt)            # e.g. {relevance:4, accuracy:5, ...} on 1–5
overall = Σ wᵢ · scoreᵢ          # weighted aggregate

# pairwise variant: ask "is A or B better?" — average over both orderings to cancel position bias.

Try it — judge the candidate against the reference

A candidate answer graded against a reference. The prompt below is the real text an LLM would receive and updates as you type; the response is a frozen example for the default pair (we do not call a model).

Interactive · LLM-as-judge

Candidate

Reference

Judge prompt (sent to the LLM)

Judge response

BiasesPosition, verbosity, self-preference

MitigateSwap orderings, clear rubric, calibrate

Human evaluation human-graded

Key idea: humans remain the gold standard that every automatic metric is trying to approximate. Annotators score each output against a rubric; you average across annotators and check how much they agree.

Rubric aggregation

# each annotator scores every criterion from 1-5
overall = Σ wᵢ · scoreᵢ              # weighted mean, per annotator
final   = mean(overall over annotators)   # average across people
# also report inter-annotator agreement (e.g. Cohen's κ)

Try it — score it yourself

Play the annotator: rate the candidate against the reference on each criterion. Weights reflect a typical rubric. (Compare your verdict with the LLM judge's above — humans and models often differ.)

Interactive · Human rubric

CANDIDATE: a cat rested on the rug · REFERENCE: the cat sat on the mat

Relevance ×0.30

Faithfulness ×0.35

Completeness ×0.20

Fluency ×0.15

StrengthGold standard; captures nuance

CostSlow, expensive, subjective

CheckInter-annotator agreement (κ)

Evaluation Algorithms

Jump to

Precision, Recall & F1 classification

Perplexity probabilistic

BLEU reference overlap

ROUGE reference overlap

LLM-as-judge model-graded

Human evaluation human-graded