The evaluation metrics must map directly to the two stated requirements. Role-based access-control enforcement rate, tested against a red-team dataset, measures whether requests from unauthorized identities, roles, or access paths are consistently denied. It converts the privacy and authorization requirement into an observable security metric and exposes bypass conditions that ordinary functional tests may miss. Because the requirement says patient data must never reach unauthorized roles, the acceptance threshold should be effectively 100% enforcement across representative and adversarial cases.
Per-query token cost directly measures compliance with the financial constraint. The metric should include uncached input, cache reads and writes where applicable, output tokens, thinking tokens, and any other billable components attributed to the request. Results can then be compared with the defined ceiling across normal, boundary, and worst-case inputs.
Anthropic’s evaluation guidance identifies privacy preservation and price as distinct success criteria and emphasizes that criteria must be specific, measurable, and relevant to the application. Define Success Criteria and Build Evaluations
BLEU may assess textual similarity but does not test authorization or cost. Throughput and percentile latency are valuable operational metrics, yet neither directly answers the two requirements in the scenario.
Study Guide references/topics: Multidimensional evaluation; privacy preservation; authorization testing; red-team datasets; token-cost measurement; requirement-to-metric traceability.
===============
