Know exactly how your LLM performs.
Vectory is a precision evaluation toolkit that combines AI judges, deterministic metrics, RAG evaluation, and human review so you can measure quality, find failure modes, and ship with confidence.
- evaluation metrics
- 9+
- evaluation metrics
- AI judge providers
- 2
- AI judge providers
- dataset formats
- 3
- dataset formats
- open source
- 100%
- open source
| Trace | Input | Judge | BLEU | Faith. | Verdict |
|---|---|---|---|---|---|
| trace-0192 | Summarize the Q3 earnings call transcript… | 4.8 | 0.71 | 0.96 | pass |
| trace-0193 | Extract action items from meeting notes… | 4.5 | 0.64 | 0.91 | pass |
| trace-0194 | Answer: What is our refund policy for… | 2.1 | 0.32 | 0.48 | fail |
| trace-0195 | Translate the product description into… | 4.9 | 0.83 | 0.98 | pass |
| trace-0196 | Generate SQL for monthly active users… | 3.2 | 0.55 | 0.77 | review |
Aggregate scores
Failure modes
Capabilities
Evaluation methods for quality, behavior, and proof.
Stop stitching together scripts and spreadsheets. Vectory unifies automated, deterministic, and human evaluation into a single workflow.
LLM-as-Judge
Use current OpenAI or Anthropic models to grade output quality at scale with customizable evaluation criteria and rubrics.
Rule-based metrics
Deterministic scoring with BLEU, ROUGE, exact match, Levenshtein similarity, and custom regex patterns.
RAG evaluation
Measure retrieval precision, recall, MRR, NDCG, context relevance, faithfulness, and answer completeness.
Error analysis
Discover failure modes with open coding, axial coding, taxonomy dashboards, and AI-assisted taxonomy suggestions.
Human evaluation
Rate outputs manually with customizable scales, criteria tags, and exportable annotation workflows.
Proof grounding
Score whether agent claims are linked to evidence, source references, closed obligations, and passing checker results.
Dataset management
Upload JSON, CSV, or PDF files with automatic column detection and role mapping.
MTEB leaderboard
Browse embedding model benchmarks from the Massive Text Embedding Benchmark directly in the app.
Custom benchmarks
Prepare organization datasets and run optional local model comparisons on your own data.
Evaluators
Six ways to measure quality.
Automated grading with frontier models
Point current OpenAI or Anthropic models at your outputs with custom criteria: accuracy, helpfulness, tone, safety, and task fit. Get structured scores with reasoning for every sample in a batch.
# Configure judge and criteriajudge = LLMJudge(provider="anthropic")judge.criteria = ["accuracy", "faithfulness"]results = judge.evaluate_batch(dataset)→ 512/512 samples · avg score 4.31/5
Workflow
From raw outputs to shipping decisions in three steps.
- 01
Upload your data
Drop in JSON, JSONL, CSV, or PDF files. Vectory detects column roles such as input, output, and expected answer, or lets you map them manually.
- 02
Run evaluations
Pick your evaluators: AI judges, deterministic metrics, RAG scoring, or human review. Run large-scale batch evaluations across models.
- 03
Analyze & export
Visualize performance across models, drill into failure-mode taxonomies, and export results, annotations, and summary reports as CSV or JSON.
Proof Grounding
Make verified claims different from confident claims.
Vectory Benchmark now supports proof-sensitive traces. The policy proof sample shows how an agent can translate claims into explicit obligations, validate them with checker results, attach source references, and keep unresolved text visible as a warning.
Policy check
Passing checker output is scored as evidence, not assumed from the final answer.
Source refs
Claims can point to source-reference artifacts and checker spans.
Warnings
NO_TRANSLATIONS is preserved as unproved content, not silently counted as validated.
- 1final claim
- 2evidence reference
- 3source document or checker output
- 4closed proof obligation
- 5passing checker result
# run the public sample
$ vectory benchmark data/vectory_benchmark/example_policy_proof_submission.json
Pass@1: 100.0%
Pathologies: none detected
# negative fixture
$ vectory benchmark data/vectory_benchmark/example_policy_proof_failure_submission.json
Flags open obligations and failed checkers
LLM judge
Can rate persuasive prose, but may not know which claims translated into policy variables.
Keyword check
Can find VALID or source-reference, but cannot prove the rule supports the conclusion.
RAG metric
Can measure retrieval quality, but does not prove generated content complies with a policy.
Vectory proof grounding
Scores the claim, evidence, obligation, checker, and untranslated-warning chain together.
Field notes
For teams evaluating real AI systems.
Your eval suite has a variance problem before it has a score problem
How many trials to run, when to report pass@k instead of pass^k, and how to tell a broken task or a saturated suite from a real regression.
The Prompting Playbook: Better Prompts Start With Evals
Use evals to diagnose prompt failures, prevent regressions, and decide when a tool or workflow change is the better fix.
How to fully evaluate software engineering agents
A software-agent benchmark is only credible when it scores the entire work loop: repository discovery, tool use, edits, tests, recovery, evidence, and release judgment.
Benchmark pages should show the work
A serious benchmark page should expose the task, environment, grader, reference boundary, score slices, checkpoints, and artifacts behind the headline number.
Static datasets are over
Agent evaluation now depends on live environments, production-derived traces, expert task design, and redaction that preserves why the work is hard.
Unfalsifiable stories of agent progress
Agent progress claims should be written so they can fail: named suite, score threshold, evidence requirement, budget, and known pathology budget.
Agent evaluation that catches bad work before it ships
A practical framework for finding low-value loops, unsupported completion claims, and weak recovery in agent traces.
Fix retrieval quality before blaming the model
Why search rounds, evidence diversity, and synthesis checkpoints matter for RAG and research agents.
Building evaluations from production traces
A grounded way to convert incidents, escalations, and expert review into durable AI regression tests.
Private evaluations
Bring your real traces, datasets, and failure modes.
Vectory can help teams convert messy production workflows into confidential benchmark suites, regression checks, and executive-ready reliability reports.
- Trace ingestion and redaction guidance for private data.
- Custom pathology rules for your tools, approval paths, and operating boundaries.
- Baseline comparisons across models, prompts, retrieval systems, and agent harnesses.
Open Source · MIT License
Run it locally in under a minute.
No account is required for local use. Launch the app with a Python CLI and evaluate on your own infrastructure.
# install from PyPI
$ pip install vectoryai
# launch the local UI
$ vectory app