Exercise an agent with simulated callers. Explore behavior before a live release. Include missing details, interruptions, unexpected answers, and unsuccessful tool responses.
For new agents and changed instructions.
How it works +
Describe the caller’s goal, starting context, and expected result for each scenario.
Include missing details, interruptions, unexpected answers, and unsuccessful tool responses.
Inspect transcripts and action results against the expected outcome before approving a release.
Version comparison
Compare different agent versions. See whether an update improves the task. Compare task completion, collected fields, escalation decisions, and tool outcomes.
For prompt and workflow changes.
How it works +
Keep the scenario set and evaluation criteria consistent across agent versions.
Compare task completion, collected fields, escalation decisions, and tool outcomes.
Investigate regressions in the underlying conversations, then rerun the affected scenarios.
Test at the moments that change the experience.
A new agent
Validate the main job and its limits before launch.
A prompt update
Compare behavior against the previous version.
A new action
Check successful, slow and unsuccessful responses.
Make testing part of the workflow.
List common requests and difficult variations. Give each scenario an outcome that can be checked.
How do we build a test plan that reflects real callers?+
Use representative requests from your support or operations team, then add interruptions, unclear dates, corrections, silence, background noise, and requests outside scope. Include examples for the languages and workflows you intend to launch.
How should we test actions that could change customer records?+
Use a test environment or controlled records where possible. Check the requested inputs, permissions, confirmation wording, and final system state. Include timeouts and duplicate requests so a retry does not accidentally repeat an action.
What acceptance criteria should leadership approve?+
Set task-specific thresholds for correct completion, accurate information, safe handoff, and response time. Keep critical failures separate from an average score; an unauthorized action should not be hidden by many easy passes.
How do we compare a revised agent with the current version?+
Run the same scenarios against both versions and review changes in outcomes, transcripts, tool results, and timing. Re-test known failures and confirm that fixing one case has not broken another.
Are simulations enough to approve a production launch?+
Pair simulations with supervised calls on the intended phone or browser channel. Real devices, carrier conditions, and callers can expose issues that a simulation misses. Begin with limited traffic and review the first live outcomes.
