arksim run \ --scenario refund_escalation \ --agent http://localhost:3000/api \ --turns 42 \ --users 5
01
Simulate
Generate multi-turn conversations with synthetic users who have distinct goals, personalities, and edge-case behaviors.
Open-source agent evaluation
Generate multi-turn conversations. Evaluate every turn. Know exactly where your agent breaks before users do.
I need to cancel my order but also want a refund for the shipping.
I can help with that. Let me pull up your order details and check our refund policy.
Actually wait, I changed my mind. Can I update the address instead?
Of course! I'll cancel the refund process and update your shipping address.
Trusted by teams at
95%
of AI agents never make it to production because teams cannot prove they work across real conversations.
Teams build agents with no way to simulate realistic multi-turn behavior before launch.
Context gets lost by turn five. Tool calls break. Contradictions appear. Static benchmarks catch none of it.
Without simulation data, teams ship on vibes, then scramble when things break in production.
arksim run \ --scenario refund_escalation \ --agent http://localhost:3000/api \ --turns 42 \ --users 5
Generate multi-turn conversations with synthetic users who have distinct goals, personalities, and edge-case behaviors.
Agent forgot user cancelled refund, re-initiated it
Called payment API with stale session token
Shared internal pricing without authorization
Surface context loss, tool misuse, and policy violations that only emerge across multiple conversation turns.
Score every response on helpfulness, coherence, relevance, faithfulness, and goal completion.
Set quality gates and readiness standards that agents must pass before reaching production.
ArkSim generates synthetic users, simulates conversations, evaluates every turn, and surfaces failures. Works with any agent: HTTP endpoint, A2A protocol, or Python class.
$ arksim run --scenario onboarding_flow ▸ Generating 5 synthetic users... ▸ Running 42-turn conversations... ▸ Evaluating responses... ┌─────────────────────────────────────┐ │ Simulation Complete │ ├─────────────────────────────────────┤ │ Turns evaluated 210 │ │ Avg helpfulness 91% │ │ Avg faithfulness 88% │ │ Failures caught 3 │ │ Gate status READY │ └─────────────────────────────────────┘
What is simulation-based evaluation?
Instead of scoring a static dataset, Arklex creates the test data for you. It generates multi-turn conversations between synthetic users and your agent, then evaluates how the agent handled each turn. The result is coverage for failure modes you would not catch with single-turn benchmarks.
How is this different from other evaluation tools?
Most tools need you to bring your own test conversations. Arklex generates them. That means you can test for scenarios that have not happened in production yet, including edge cases where users push back, change their mind, or ask unexpected follow-ups.
Why does multi-turn testing matter?
An agent can ace a single question and still fall apart in a real conversation. Context gets lost by turn five. Tool calls break when the user changes direction. The agent contradicts something it said two turns ago. These are the failures that reach production, and they only show up when you test across multiple turns.
What agents and frameworks are supported?
Any agent, any framework. If it exposes an HTTP endpoint, speaks the A2A protocol, or is a Python class, Arklex can test it. The platform handles the simulation and evaluation regardless of how your agent is built.
Can I integrate this into my development workflow?
Arklex works as a CI/CD quality gate that runs on every code change, and as a standalone platform for testing, governance, and deployment approval. Teams typically start with ad-hoc testing during development and add CI gates once they have a baseline.
Is my data secure?
Workspaces are fully isolated with separate data storage. The platform can run on your infrastructure, keeping all conversations and evaluation data in your environment. Private cloud deployment is available for enterprise customers.
Simulate. Evaluate. Ship with evidence.