chat-quality

1. 1
   Close the last loop

   Read the previous run’s report and check whether its branch was merged, so a case already written is not proposed twice.
2. 2
   Pull the verdicts and the log

   Read the thumbs verdicts and the logged exchanges from the live store, and join each verdict to the exchange it rates by its id. The agent only reads; it never writes there. Without read access it stops and asks.
3. 3
   Triage

   Every thumbs down is a candidate. The unrated answers get a skim too, for silent failures such as an invented path or a limit notice where a real answer was possible. A window with no real failures is a valid result, and the loop stops there.
4. 4
   Makes the golden-set casesWrite the cases first

   Each failure becomes a case in evals/chat/golden-set.json before it is fixed: the question in other words, the facts a good answer needs, the cheapest check that would have caught it, and the rules it covers in evals/chat/SPEC.md. A fact missing from the chat’s sources is the real bug, and the source is what gets fixed.
5. 5
   Run the eval once

   npm run eval:chat runs the whole set through the real chat route on a local dev server. It spends real API budget, so it runs once per loop. The answers are read one by one: the new cases should fail before the fix and pass after, and nothing that passed before should break.
6. 6
   Makes the local branchFix on a branch

   The new cases and the fixes they called for go on one fresh local branch. Fixes stay inside the chat’s answer pipeline: page prose, the persona and the lookup tools. A fix in the package’s chat engine changes every consumer’s chat, so it is flagged as such.
7. 7
   Makes the reportReport and hand off

   A plain-English report: the window, the verdict counts, each disliked answer and its cause, what was encoded and the eval result. Merging the branch approves it; deleting it rejects it. Nothing is pushed or deployed.

A visitor rates an answer with a thumb. /api/chat/feedback stores the verdict in Redis under the exchange’s id, beside the exchange that /api/chat logged, and both are deleted after 30 days. The agent reads the two and joins them. The disliked answer becomes a golden-set case written in other words, the eval runs against a dev server, and the fix waits on a local branch. The visitor’s own words stay in a local report that is never committed.

1 · On the live sitekept for 30 days

2 · Read, never writtenthe agent observes

3 · Encode, then testthe case comes before the fix

4 · For approvalnothing is pushed

thumbs_up_downSite chata thumb on each answer

forumChat route/api/chat

rate_reviewFeedback route/api/chat/feedback

storageRedisexchanges and verdicts

Joinverdict to exchange, by id

ruleTriageevery dislike, plus silent failures

descriptionLocal reportnever committed

gavelSpec ruleevals/chat/SPEC.md

playlist_addGolden-set caseevals/chat/golden-set.json

fact_checkCoverage checkscripts/validate-chat-coverage.mjs

dnsDev servernpm run dev:eval -w website

scienceEval runnpm run eval:chat, once

personOwnermerge approves, delete rejects

call_splitLocal branchchat/YYYY-MM-DD-<slug>

buildFix the causepage prose · persona · lookup tools

read onlyparaphrased, never quotedfails before, passes after

removeZoom out50%addZoom infit_screenFit the whole mapopen_in_fullExpand the map

The live data is read, never written. Only a paraphrase of the failure and its fix reach the repo, and only on a branch the owner approves.