Concept2026
Agent Checkpoint
Provider-Agnostic Agent Evaluation
Provider-agnostic evaluation tooling for conversational agents, built around master test agents, adaptive campaigns, generated episodes, and a CI-integrated release gate. Designed and built solo, from the campaign engine through the release-gate scoring logic.
Next.js
TypeScript
Prisma
PostgreSQL
Redis
OpenAI
ElevenLabs
Salesforce Agentforce
ZodProduct Walkthrough
The Whole Loop, Start to Finish
The workflow runs inside one surface: connect a target, create or generate an examiner, run a campaign, review what broke, then refine the target and rerun to confirm the fix actually held.
An Evaluation Loop That Adapts
Simulated Users That Adapt
Master test agents that probe and adjust mid-conversation
A Master Test Agent Studio defines how a simulated user should probe and adapt during a conversation, instead of replaying a fixed transcript that stops testing anything the moment the target's behavior changes.
A Campaign Engine That Reacts
Every follow-up is a live decision
A campaign generates first turns in batch, then makes structured continuation decisions and adapts each follow-up from the target's actual responses and tool evidence, so a campaign explores rather than just replays.
Works With Any Target
One evaluation loop, four kinds of agent
Provider-agnostic adapters cover generic HTTP, OpenAI, ElevenLabs text, and Salesforce Agentforce targets, running on a BullMQ queue that falls back to inline execution when Redis isn't configured.
A Release Gate, Not Just a Score
Critical Families Can Hard-Block
A good average score doesn't save a bad release
Failures in critical coverage families — adversarial security, privacy and authorization, safety and harm, business policy — can block a release outright, regardless of how well the overall campaign scored.
Regression-Aware Thresholds
Measured against a moving baseline
Minimum average score and pass rate are configurable, but so are the maximum allowed regression against a pinned or previous baseline. A release can pass an absolute bar and still get blocked for regressing.
CI-Readable by Design
Pass, warn, or block — mapped to a real CI conclusion
Each release-gate decision resolves to a pass/warn/block status and a success/neutral/failure conclusion a CI pipeline can consume directly, so agent quality can gate a deploy the same way a test suite does.
Coverage Glossary
Nine Families Map the Whole Surface
The coverage glossary lays out all nine test families — from adversarial security and privacy/authorization to bias and fairness — each tagged with a probe count, so a thin family is visible before it becomes a blind spot.
Every Report Is Inspectable
Nine coverage-family scorecards per report
A campaign report includes nine coverage-family scorecards, full transcript replay, turn-by-turn breakdowns, raw JSON inspection, and trace-payload inspection — so a failure is something a team can actually look at, not just a percentage.
