Get started
The quality gate for AI agents

Evaluate any agent.
Ship with confidence.

A modular orchestrator that calls your target AI API, scores responses against YAML rubrics, and returns structured pass/fail results — ready for CI, red-teaming, and production QA. Cloud LLMs or local LLM using Ollama for the judge.

Start in 60 seconds
Any API
HTTP & multipart agents
Cloud + local
Cloud LLMs or local LLM using Ollama
Batch + CI
Parallel evals, exit codes
POST /eval PASS · 0.89 
POST /eval

{
  "prompt": "Extract invoice fields",
  "agent": { "url": "https://your-agent/ask" },
  "rubric_name": "default",
  "judge_type": "llm"
}

→ {
  "overall_score": 0.89,
  "overall_verdict": "pass",
  "passed": true,
  "latency_ms": 812
}
0.95
Accuracy
0.88
Relevance
1.00
Safety
LLM judge

Cloud LLMs or local LLM using Ollama

Pick cloud LLMs (Anthropic, OpenAI, Gemini, Qwen) or local LLM using Ollama — no cloud API key when running local. The interactive launcher walks you through provider and API base setup.

# Start Eval Gate — cloud or local LLM judge
./scripts/run_agent.sh
# Cloud: anthropic, openai, google, qwen
# Local: openai_compatible → Ollama at localhost:11434/v1
# → model: qwen2.5:3b (or any pulled model)
Why Eval Gate

Quality gates for the agents you build

Define what "good" means in a rubric, point Eval Gate at your service, and get reproducible scores every time.

Agent-agnostic

Works with any JSON-returning HTTP API. Configure URL, method, prompt field, and auth headers — no SDK on the target side.

YAML rubrics

Weighted dimensions — accuracy, safety, schema compliance — with per-dimension pass thresholds. Drop new rubrics in configs/ with zero code changes.

Pluggable judges

Cloud LLMs or local LLM using Ollama (openai_compatible) via LiteLLM — plus Anthropic, OpenAI, Gemini, and Qwen. Rule judge for regex and schema checks. Composite chains both.

Batch evaluation

Run hundreds of test cases in parallel. One failure doesn't stop the batch — each case returns its own structured result.

Document & file agents

Multipart support for TXT, PDF, and image extraction. Upload via POST /eval/with-file or set context.file_path server-side.

Safety & red-team

Test jailbreak resistance, harmful prompt refusal, and phishing detection. Combine rule-based checks with LLM safety scoring.

Architecture

How it works

Eval Gate is an orchestrator — it never replaces your agent. It calls your API, then judges the response.

Animated demo: send prompt, call agent, score with rubric, get pass/fail results
Send a prompt → call your agent → score with a YAML rubric → get structured results.
1

Send a prompt

POST to /eval with your test input and agent config.

2

Call your agent

The HTTP runner forwards the request and extracts the response field.

3

Score with rubric

The judge evaluates weighted dimensions using ground truth or schemas.

4

Get results

Overall score, per-dimension breakdown, pass/fail verdict, and latency.

Use cases

Built for real evaluation workflows

From document extraction QA to production regression suites — one service covers the full eval lifecycle.

Document AI

Structured extraction QA

Validate invoice, receipt, and contract extraction against JSON Schema and ground-truth fields.

  • Text + schema multipart mode
  • File upload with response_schema
  • Composite rule + LLM judging
Safety

Red-team & refusal testing

Probe agents with adversarial prompts and verify they refuse harmful requests consistently.

  • Jailbreak and phishing scenarios
  • expected_behavior hints for judges
  • Batch red-team suites
RAG & QA

General agent quality

Score chatbots and RAG pipelines on accuracy, relevance, coherence, and groundedness.

  • Default and RAG-specific rubrics
  • Ground truth in context
  • Swagger UI at /docs
Ecosystem

Integrations & tooling

Run locally, in Docker, or from your pipeline. Multiple UIs and clients ship with the repo.

Cloud LLMs or Ollama (local LLM) FastAPI + Uvicorn LiteLLM (multi-provider) Anthropic · OpenAI · Gemini · Qwen Docker Compose Python client SDK Vue browser UI Zero-build CDN client OpenAPI / Swagger

Start evaluating in under a minute

Interactive scripts walk you through cloud LLMs or local LLM using Ollama — provider, API key, and port — no manual config required.

Quick start 
./scripts/run_agent.sh          # start the eval service
./scripts/run_vue_cdn_client.sh  # open browser UI at :5500
curl http://localhost:8000/health