Agent Acceptance Tests & Quality Checklist

Before expanding use, make sure the agent passes these acceptance tests. Treat each item as pass/fail and record evidence.

Functional tests

• Representative scenarios: The agent completes the primary outcome in a set of 10 representative scenarios (include edge cases).
• Correctness threshold: Agreed metric (e.g., extraction accuracy > 90% or recommended actions agreed by human reviewers in 8/10 cases).
• Latency check: Responses meet the user-acceptable latency (e.g., < 3 seconds for assistant suggestions, < 30s for evidence summaries).

Safety, hallucination, and trust

• Hallucination test: For a set of 20 prompts, the agent cites verifiable sources or correctly says "I don't know" in at least 95% of cases required by policy.
• Never-confident rule: Agent must flag low-confidence answers and route to humans rather than asserting them as facts.
• Prohibited actions: Validate the agent never performs forbidden actions (financial commitments, legal advice, patient diagnosis) without explicit human approval.

Privacy, compliance, and data controls

• Data minimization: Agent only accesses fields needed for the task.
• PII/PHI handling: Sensitive data is masked or excluded when required; access logs are enabled.
• Consent & retention: Document any customer consent and data retention policies for logs and interaction history.

Operational readiness

• Logging and observability: All inputs, outputs, and decisions are logged with timestamps and request IDs.
• Rollback plan: A clear, tested rollback procedure exists (disable endpoint, re-route traffic to humans) and was executed in a dry run.
• Incident response: Responsible person and escalation path are documented and reachable during pilot hours.

User acceptance

• Pilot feedback: Pilot users rate the agent helpfulness > target (e.g., > 4/5) and provide qualitative feedback captured in 10+ entries.
• Training & docs: Short help text and examples are available for users and reviewers.

Record pass/fail for each item and required remediation before wider rollout. Use this checklist as part of governance and KPI huddles to keep quality visible.