Web-reading AI: a three-point contract for blocked pages

A blocked fetch is a product decision

Your reader asks for a page and gets a refusal. The system must say whether it got the page, part of the page, or a different source instead.

Refusals are becoming ordinary. Automated systems, not people, now generate 57.5 percent of all HTTP requests. A web-reading feature will meet paywalls, consent walls, bot filters, and publisher choices.

The failure to avoid is the silent substitute: the model sees a block, pulls a cached copy, a summary, or a related article, and presents the result as if the original was read. Users can forgive a degraded answer. They cannot forgive a hidden one.

The contract turns a blocked request into a readable answer

First, classify the fetch before the model writes. Reached means the requested page returned usable content. Partial means some content came back, but the page was truncated, rendered without key sections, or served a challenge. Blocked means the request was refused, redirected to a gate, or returned an error that prevents reading the target.

Classification matters because publishers are tightening control over AI traffic. Cloudflare's new AI traffic controls give customers separate on/off control over crawlers used for search, AI training, or AI agents. For new domains, Cloudflare's September 15, 2026 default will deny access to bots classified as Training or Agent when the page shows ads, while keeping Search access permitted. Customers have a window before September 15, 2026 to decline Cloudflare's new AI-bot defaults.

Cloudflare's AI-training blocking configurations cover crawlers that serve both Search and Training purposes, including the legacy Block AI bots option. Under those controls, multi-purpose crawlers such as Googlebot, Applebot, and Bingbot are subject to the most restrictive setting, so they would be blocked if a publisher blocks training crawlers. A feature that labels itself as a search assistant may still be treated as an agent or training crawler by the site it visits.

Assuming every public page is readable will break on ordinary commercial sites. Treat the block as a signal about the source, not a prompt to improvise.

Second, disclose coverage, not confidence. The answer should say what the fetcher could and could not reach. Ask the fetcher to report which sections were read, which were missing, and whether the answer depends on a fallback.

The disclosure should be short. A line like I could not read the full page, so this answer uses a cached copy from earlier this month is better than a confident paragraph. The user needs enough information to decide whether the answer is safe to use.

Third, make the fallback source strategy part of the answer. A fallback should be a named choice: cached copy, related source, user-provided text, or no answer. The feature should decide the fallback before the model writes, not after the model has already produced a plausible answer.

A good fallback strategy respects the user's intent: for a specific page, it says what it is not; for a general answer, it can be broader but still names the source; for a fact that depends on the page's current state, it says the fact may be stale.

The checklist makes the contract testable

Run these checks before you ship a web-reading feature. Each one protects the user from a different trust failure.

• Label every fetch result as reached, partial, or blocked before the model composes the answer.
• Show the target URL, the status, and the reason the page was not fully read.
• State which parts of the answer came from the requested page and which came from a fallback source.
• If the fallback is a cached copy, say when it was captured and whether it may be stale.
• If the fallback is a related source, say why it is relevant and what it does not cover.
• Give the user a path to retry, open the original, or continue with the degraded answer.

Classification comes before composition. If the fetcher cannot say whether it reached the page, the answer cannot claim coverage; a bare error becomes a disclosure of the refusal, redirect, challenge, or missing element.

Fallback boundaries need to be explicit. A cached copy and a related source are not the live page or the requested claim, so the user can see the difference before relying on the answer.

The last checks catch drift and keep the path open. A news summary may be true in general and wrong for the specific page, while the user may have a session, a subscription, or a preference for a different source.