Evaluation

Changing the model, the number of documents retrieved or the similarity threshold can quietly break answers that worked yesterday. An eval set turns "it seems fine" into numbers you can compare, so you change settings with evidence.

• Before switching to a cheaper or newer model.
• Before changing retrieval settings such as top k or minimum similarity.
• After adding or rewriting a lot of knowledge.
• When customers report wrong answers, to check the fix and keep it fixed.

Evaluations run on your own AI provider key: a retrieval-only run costs only query embeddings; a full run also generates answers and, if you choose, has a judge model score them. Only workspace owners and admins can create sets and start runs.

Choose New eval set and give it a name, for example Regression. Aim for 50 to 200 questions (up to 1000 per set), and make at least 15% of them questions your knowledge cannot answer, so you also test that the assistant declines instead of guessing.

There are four ways to add cases:

1. 01

   By hand

   Add case on the set page. Best for your most important questions.

2. 02

   From real conversations

   In a conversation, press Add to eval set under any reply. The question, earlier turns and the cited sources are filled in; remove personal data before saving. This is the best source of realistic questions.

3. 03

   Generate from your knowledge

   Generate cases writes questions from your documents with the bot's own model (default 30 answerable and 6 unanswerable), tagged synthetic. Review them: generated cases count half as much as hand-written ones in every score.

4. 04

   Import a JSONL file

   One case per line, up to 2 MB. Choose to add to the existing cases or replace them. Problems are shown line by line before anything is saved. Export gives you the same format.

{"key": "ship-canada", "question": "Do you ship to Canada?", "expected_answer": "Yes, in 5 to 8 business days.", "gold_documents": [{"external_ref": "https://help.yourshop.com/shipping", "must_contain": "5 to 8 business days"}], "answerable": true, "tags": ["shipping"]}
{"key": "refund-window", "question": "How many days do I have to return an item?", "gold_documents": [{"external_ref": "https://help.yourshop.com/returns"}], "tags": ["returns"]}
{"key": "no-crypto", "question": "Can I pay with Bitcoin?", "answerable": false, "tags": ["payments", "unanswerable"]}
# lines starting with # and blank lines are ignored

Only question is required. Tags use lower-case letters, digits and _ : . - (up to 20 per case). Unknown fields are rejected so typos do not pass silently.

1. 01

   Start run

   On the set page, press Start run. The run uses the bot's live settings. You can try different retrieval settings or temperature for this run only; the bot itself is never changed.

2. 02

   Choose the mode

   Retrieval only checks whether the right documents are found; it costs query embeddings only and finishes in seconds. A full run also writes answers and scores them.

3. 03

   Choose the judge

   For full runs, a judge model scores groundedness, relevance and correctness: the bot's own model, another key and model, or no judge. Keep the same judge for runs you want to compare.

4. 04

   Set concurrency

   How many questions run at once (default 4). Use 1 or 2 on low-tier keys to avoid rate limits and to see realistic response times.

Tip: When you change the model or retrieval settings on the Settings tab and the bot has a set with at least 20 cases, a Run eval first button appears. It runs the set with your unsaved changes and opens the comparison, without saving anything.

Scores are shown from 0 to 100% with a 95% confidence range and the number of cases behind them. A small set gives a wide range: treat small differences with care.

A PARTIAL badge means the judge could not score every case; judged numbers are then less reliable.

Tick two runs of the same set and press Compare selected, or use Compare with baseline on a run. You see both runs side by side, the change in each metric in green or red, and the cases that got worse.

A drop that stays inside the baseline's confidence range is shown as within range and does not fail. When a run is the new normal, use Accept as baseline and note why.

• Start small: 30 good questions from real conversations beat 300 generated ones.
• Always include unanswerable questions, at least 15% of the set.
• Add a gold document to every answerable case, or retrieval scores cannot be computed.
• Give cases a key, so runs can be compared case by case.
• Keep the judge the same across runs you compare.
• Add every wrong answer a customer reports as a case. Your set becomes your safety net.