A company is comparing two foundation models (FMs) for a customer service AI assistant.

AWS documentation describes LLM-as-a-judge as an automated evaluation technique where a large language model is used to assess the outputs of another model based on qualitative criteria such as helpfulness, correctness, tone, and alignment with expectations. This approach enables scalable and repeatable evaluations without requiring human reviewers.

In this scenario, the company needs to compare two foundation models across subjective dimensions that are difficult to measure using traditional metrics. LLM-as-a-judge allows the evaluator model to score or rank responses using predefined evaluation prompts and criteria, ensuring consistent and automated assessment.

The other options do not meet the requirements. String matching and ROUGE focus on lexical similarity and are unsuitable for evaluating tone or helpfulness in customer service interactions. Retrieval Augmented Generation is an architectural pattern, not an evaluation technique.

AWS highlights LLM-as-a-judge as a practical approach for automated qualitative evaluation of generative AI outputs, making it the correct choice.