AI EVALUATION · LLM & AGENT EVALUATION
Twenty evaluators. Any dataset. Every release.
Offline runs catch regressions before your customers do; online runs score what production is doing right now. Task completion, tool use, quality, groundedness, reference match and safety — each with its own pass threshold.
An evaluation in Kosmoy is a reusable definition: a subject, a method and a dataset. The subject can be an assistant driven end to end with its real tools, retrieval and guardrails; a model behind one of your gateways; or a registered external agent. Running it produces a numbered run, so you can compare today’s behaviour with last month’s — after a model swap, a prompt change, a new retriever.
Because the definition is reusable, the second run costs one click. That is the point: evaluation only changes behaviour when re-running it is easier than not bothering.
What it does.
Offline evaluation
The regression harness. Kosmoy drives your assistant over every row exactly as a user would experience it, then scores the responses. Failing rows are isolated, never fatal.
Online evaluation
Cases reconstructed from production traffic and scored with reference-free evaluators — relevance, coherence, groundedness, tool use, safety. What is it doing right now, for real users?
Agentic evaluation
Five tool-use evaluators: did the agent pick the right tools, pass the right arguments, succeed, and use what came back — or invent an answer?
RAG evaluation
Groundedness checks every response against the retrieved context — the direct measure of answering from sources versus imagination.
Curated benchmarks
AIME 2025, MuSR, TruthfulQA and a prompt-injection corpus ship pre-configured. Import MMLU, GSM8K, HellaSwag, ARC or any HuggingFace dataset alongside them.
Scores you can defend
Rule-based evaluators are deterministic and free; AI judges explain every score in writing. Not-applicable cases are reported, never counted as failures.
Module questions, answered straight.
What LLM evaluation metrics does Kosmoy support?
Twenty built-in evaluators across six dimensions: task completion (task completion, task adherence, intent resolution, navigation efficiency), tool use (call accuracy, call success, tool selection, input accuracy, output utilization), quality (relevance, coherence), groundedness against retrieved context, reference match (exact, multiple-choice, numeric — each normalising sensibly), and safety (injection resistance, toxicity, bias, PII leakage, attack success). Every score runs 0 to 1; each evaluator gets its own pass threshold.
Does Kosmoy use LLM-as-a-judge evaluation?
Where it is the right tool, yes. Deterministic evaluators — exact match, multiple choice, numeric match — score by rule, reproducibly and free. Qualities rules cannot capture, such as relevance, coherence, groundedness and policy compliance, are scored by an AI judge that explains every score in writing. The explanation travels into the results table, the spreadsheet export and the PDF report.
How does agent evaluation work?
You provide tasks that require tools, together with the tools the agent should have used. Kosmoy drives the agent end to end and scores whether it chose the right tools, passed the right arguments, actually used the results rather than inventing an answer, and finished the task efficiently — five tool-use evaluators plus the task-completion set.
Can I evaluate RAG quality and hallucinations?
Yes. The groundedness evaluator checks each response against the context the system retrieved — the direct measure of whether a RAG system answers from its sources or from imagination. Pair it with relevance and coherence for full RAG evaluation, offline on a question set or online against production traffic.
Which benchmarks ship built in?
Four arrive pre-configured with the right scorer attached: AIME 2025 (competition mathematics), MuSR (multi-step soft reasoning), TruthfulQA (resistance to plausible misconceptions) and a real-world prompt-injection corpus. Any other public benchmark — MMLU and its domain variants, GSM8K, HellaSwag, ARC, multilingual and industry-specific suites — imports from HuggingFace in the Datasets section and scores with the same evaluators.
What happens when an evaluator can't judge a case?
It is reported as not applicable rather than counted as a failure, so scores are never quietly diluted by data that could not be judged. A row that fails to run is isolated and reported instead of taking the whole run down. If a call fails on a timeout or network error, the case is excluded from every score and reported separately.
Run your first evaluation this week.
Bring a benchmark, a spreadsheet of graded questions or your production traffic — leave with a scorecard.