Pick a metric and run it.
A model is only as trustworthy as the number you use to judge it. Evaluation turns "this output looks good" into a measurable score you can compare, track and optimise. But there is no single right metric — the question you're asking determines which one is meaningful.
When the output is a discrete label, score it against the truth with Precision, Recall and their harmonic mean F1 — all derived from the confusion matrix of true/false positives and negatives.
Perplexity measures how well a language model predicts held-out text — lower means less surprised. It's the standard intrinsic metric for language modelling.
For generated text, compare against a human reference by counting shared n-grams. BLEU (precision-oriented, for translation) and ROUGE (recall-oriented, for summarisation) are the classics.
LLM-as-judge uses a strong model to score or rank outputs against a rubric — flexible and correlates with human judgement, but brings its own biases (favoring outputs generated by itself rather than by other models).