Running a full model-as-judge evaluation on every user request adds latency and cost. Many high-throughput systems therefore split the work: lightweight checks on the live path, and deeper evaluation run asynchronously over logged traffic.
The Latency and Quality Trade-off
Checks on the live path should be small enough to fit the latency budget you agree for that system. Examples are pattern-based personal-data detection and simple classifiers. Deeper checks, such as citation fidelity, subtle bias, and policy alignment, are better run after the response is sent.
Asynchronous Evaluation on the Lakehouse
Inference events can be written to an append-only table in Microsoft Fabric or Azure Data. Scheduled jobs sample and score conversations for faithfulness, relevance, and policy compliance, and the results feed dashboards for engineering and governance teams. Sampling rates and score thresholds should be documented and reviewed.
Drift Alerts and Fallbacks
When evaluation scores fall below an agreed threshold after a data update or a model change, the system should alert a named owner and, where the design allows, switch to a safer fallback. For high-risk systems, this kind of post-market monitoring is a provider obligation under Article 72 of the EU AI Act.
