Source filtering for an AI news agent

Allow publishers by parsed hostname

Compare the parsed hostname with an explicit set such as {"www.sec.gov"}. A substring check for sec.gov would also accept an unrelated host such as www.sec.gov.evil.example. Decide explicitly whether both the root host and a www host are allowed.

A trusted hostname still needs a safe retrieval process. If your agent follows redirects, validate every destination, block private network destinations and apply response-size and time limits. The runnable gate does not fetch URLs; these controls belong in the reader.

Preserve three different times

Synoptic source metadata can include sourceLink, sourceName and sourceTimestamp; individual fields can be absent. Do not replace a missing publication time with createdAt and describe the story as newly published.

Deduplicate without hiding updates

Start with exact canonical URLs. Remove known tracking fields, preserve parameters that identify an article and keep the original URL alongside the canonical one. A changed headline on the same URL may be an update, so a production deduplicator should record content hashes and retrieval versions rather than discarding every repeat forever.

For syndicated stories, two hosts can carry the same text under different URLs. A similarity model can suggest a cluster, but retain the individual sources and avoid calling them independent corroboration until their reporting origin is checked.

Make exclusions visible

Write down excluded publishers, language limits, missing time ranges and sources requiring a separate licence. A short digest is not a comprehensive news search. Explicit exclusions help a reader understand what silence means.

Verify publisher and usage rights in the actual stream agreement. This guide does not establish Bloomberg, Reuters or PR Newswire availability, and a metadata URL is not permission to republish the linked full text.