You evaluate a translation app against human reference translations.

How it works

When you have human reference text (ground truth), Foundry offers two kinds of textual similarity evaluators:

• Similarity is an LLM-judge metric. It measures the degree of semantic similarity between the generated text and its ground truth with respect to a query. Unlike overlap-based metrics, it focuses on the semantics of a response instead of simple token or n-gram overlap, so a paraphrase or synonym can still score as equivalent.
• BLEU is an algorithmic metric. Microsoft describes it as well suited to machine translation quality assessment, where exact n-gram matches between system output and human reference translations are a meaningful proxy for quality. Use the algorithmic evaluators (F1, BLEU, GLEU, ROUGE, METEOR) when exact token overlap is the criterion. In the evaluator reference, BLEU lists no required parameters (no judge model deployment) and returns a 0 to 1 float.

So the paraphrase-tolerant metric is Similarity, and the n-gram metric is BLEU.

The other options, and when you'd use them

• Groundedness: It checks the response against its grounding context. It focuses on precision, meaning the response doesn't contain content outside that context. It doesn't compare against a reference translation.
• Fluency: It measures the effectiveness and clarity of written communication. Like Coherence, it measures writing quality independent of factual correctness, without comparing the output to a reference.
• Coherence: It measures the logical and orderly presentation of ideas in a response. It also doesn't compare the output to a reference translation.

Exam tip

Reference + meaning = Similarity (LLM judge). Reference + exact n-grams = BLEU (algorithmic, translation). Context = Groundedness. Writing quality = Fluency / Coherence.

References

• Textual similarity evaluators
• Retrieval-Augmented Generation (RAG) evaluators
• General purpose evaluators