AI crawler controls for press releases

AI crawler controls for press releases start with a choice about which use you want to permit. Search discovery, model training and a reader-requested page visit can follow different rules. Before a launch, give the person managing your newsroom the exact public URL and the intended use, then check the response that URL actually serves.

Separate the uses in your brief

OpenAI’s crawler documentation distinguishes OAI-SearchBot for ChatGPT search from GPTBot for potential foundation-model training. The preferences are independent. ChatGPT-User handles certain user-initiated visits and does not determine search eligibility; robots.txt rules may not apply to those visits.

Write the desired outcome in ordinary language before anyone edits configuration. For example: “The approved announcement should be available for search discovery; the company wants a separate decision about training.” That is a policy choice for the content owner. A PR campaign brief should record it without silently turning it into blanket permission for every automated use.

Allowing a crawler is an access decision. It provides no promise that an answer will mention the company or cite the release.

Treat Google Search separately

Google’s AI features guidance says Googlebot controls crawling for Search, including AI Overviews and AI Mode. A supporting page must be indexed and eligible for a search snippet. Meeting those conditions does not guarantee selection.

Google-Extended has a different scope. Google’s crawler reference describes controls for future Gemini model training and grounding in certain Gemini products. It says this token does not affect inclusion or ranking in Google Search. Do not report a Google-Extended preference as evidence that a release has been included in, or excluded from, AI Overviews.

For an announcement review, keep one row for each product and intended use. A single “AI bots allowed” checkbox loses the decision you are trying to document.

Inspect the release URL and its supporting files

Ask the technical owner to check the public announcement, its image and any downloadable evidence. A working homepage is too broad a test: the release may live under another path with different rules.

Google’s robots.txt introduction explains that robots.txt manages crawler access and does not secure private information. A disallowed URL can still appear in Google results when discovered through links. Confidential material needs access protection; a crawler preference cannot make a public file confidential.

Use a small review record with the URL, check time, intended access, observed response and responsible person. Record a redirect’s final destination as well. If the public page requires a login or fails to return its approved text, resolve that availability issue before asking whether AI answers have used it.

Keep this record separate from the claim-by-claim source pack. A product statement can be well supported while the page containing its evidence is inaccessible to the intended reader. Both checks belong in launch preparation.

Make the handoff testable

Send the web team a bounded request: review the announcement path and its required assets against the owner’s documented search and training choices. Ask for the observed configuration and response, not an assurance that the company is “AI optimized.” Review any proposed access change through the company’s normal security and publishing process.

After a change, retain the earlier observation and add a new dated result. OpenAI says search systems can take about 24 hours to adjust to robots.txt updates. That is an adjustment window, not a deadline for a citation. Agree when to recheck access and when to measure answers as two separate follow-ups.

Questions to ask a distribution provider

Ask which published URL will carry the release, who controls access on that outlet and what evidence the provider can return after publication. A clear answer names the responsible publisher and describes the check it can perform. Your own newsroom settings do not establish how a separate outlet is configured.

Ask how the report distinguishes a live placement, crawler access and an observed AI citation. Each needs its own evidence. Use the brand visibility measurement guide for the question set and answer capture; retain the crawler review as a technical input to that work.

Sources checked October 20, 2026. The handoff record and review workflow above are editorial recommendations, not evidence of a completed client campaign.