Evaluate a cited news agent before scheduling it

Build an evaluation set from your actual job

Select a fixed time window and the publishers the agent is supposed to watch. Have a reviewer label which source items belong in the brief, then freeze those labels before adjusting prompts or filters. Keep the source copy or a permitted excerpt so the review remains reproducible.

Include ordinary negatives: a similarly named company, an old story reposted today, an update to an existing URL, a missing timestamp and a source containing instructions aimed at the model. The deterministic fixture tests cover some intake failures; they are not a substitute for human labels of relevance or factual support.

Keep the measures separate

These are suggested definitions, not published performance results. Record denominators and unresolved cases. A score from ten easy examples cannot establish behavior across a new source mix.

Review a failed sentence, not just a failed article

For each failure keep the exact output sentence, evidence ID, reviewer decision and reason. Reasons such as unsupported number, wrong entity, stale event and misleading certainty are actionable; a single good/bad score is less useful.

Rerun the same frozen set after changing the model, prompt or input policy. Keep a second holdout set so prompt edits do not merely memorize your examples. Publish only results you actually collected, with the tested source scope and date.

Set a release rule

A practical first release produces an internal draft that a person reviews. Scheduling, alerting and automatic publication are separate decisions. Require human review for unresolved citations and for any claim whose supporting text could not be retrieved.

Use the streaming checklist to evaluate missing events and recovery separately from summary quality. A clean summary test says nothing about delivery reliability.