For public marketing, product and technical content, generally no: being crawled is a precondition for being read and cited by an AI answer engine, and blocking removes that possibility while leaving competitors, distributor listings and trade directories to describe your products instead. The more useful question is which crawlers to allow for which purpose, and which material belongs behind real access control rather than a robots.txt request.
The decision separates cleanly once you divide content published to be found from content carrying commercial or contractual restrictions. Catalogue pages, specifications and application notes exist to be discovered. Distributor pricing, tender documents, licensed CAD libraries and unreleased product data belong behind authentication, where a crawler directive was never the right control.
An answer engine can use only what it has read. Blocking the crawlers on public content removes your pages from the candidate set while leaving distributor listings, trade directories, forum threads and competitor pages in it.
The result is usually worse than absence: the assistant still answers questions about your products, drawing on material you did not write and cannot correct. Sight of what is being said about you disappears along with the citation.
A robots.txt directive is respected voluntarily by crawlers that choose to honour it. It imposes no technical barrier, and content reachable without authentication stays reachable to anything disregarding the file.
Where material must be restricted, use authentication, entitlement checks and signed URLs. Treating a robots directive as protection for commercially sensitive documents is a control gap, and it tends to surface during a security review rather than before one.
Distributor and customer-specific pricing, quotation and tender documents, CAD and BIM libraries under licence terms, drawings for bespoke installations, unreleased product data, and anything covered by a customer confidentiality agreement.
Those belong in an authenticated area with access tied to an account. Our distributor portal page describes how that entitlement model is built, and the same boundary keeps the material away from every crawler at once.
Providers operate separate agents for separate purposes: one fetching pages to answer a live user query with attribution, another collecting data for model training, another indexing for a search product. Robots.txt can address them individually.
An organisation comfortable being cited in answers and uncomfortable contributing to training can allow the first and decline the second, accepting that the distinction relies on providers honouring their own agent names and that those lists change over time.
Where the objection is crawl load rather than principle — a large catalogue being fetched aggressively — the remedy is rate limiting, caching and CDN configuration. Blocking to solve a performance problem trades a visibility asset for an infrastructure fix that has cheaper answers available.
A faceted catalogue generating many URL combinations is usually the underlying cause, and reducing that crawlable space often does more than exclusion would. Our website performance page covers the delivery side.
Before setting a policy, establish what the assistants currently say about your company, your products and your competitors, and which sources they draw on. That baseline usually reframes the question, because the common finding for industrial manufacturers is under-representation rather than over-exposure.
See AI search visibility audit for how it is measured, and how do you get cited by ChatGPT for what improves the position.
For public marketing, product and technical content, generally no: being crawled is a precondition for being read and cited, and blocking leaves distributor listings, directories and competitor pages to describe your products instead. Restricted material — pricing, tenders, licensed CAD, unreleased data — belongs behind authentication, where a crawler directive was never the right control.
It may reduce that use by crawlers honouring the directive, and it has no effect on an agent that disregards robots.txt or on copies of your content already held elsewhere. Treat it as a partial measure carrying a visibility cost, and weigh it explicitly against that cost rather than adopting it as a default posture.
No. It is a request respected voluntarily and imposes no technical barrier, so anything reachable without authentication stays reachable to a crawler that disregards the file. Restricted material needs authentication, entitlement checks and signed URLs. Relying on a robots directive for commercial documents is a control gap that surfaces during a security review.
Yes. Providers run separate agents for answering live queries, for search indexing and for training data collection, and robots.txt can address them individually. That supports a position such as permitting attributed answers while declining training. It depends on providers honouring their own agent names, and the lists change, so review the file periodically.
Handle it as an infrastructure problem. Rate limiting, caching and CDN configuration address aggressive crawling without giving up visibility, and a large faceted catalogue generating many URL combinations is usually the underlying cause. Reducing the crawlable URL space through canonical and parameter handling often achieves more than blocking would.
Take a baseline first. Establish what the assistants currently say about your company, your products and your competitors, and which sources they draw on. The common finding for industrial manufacturers is under-representation rather than over-exposure, and that usually reframes the question before a policy gets written.
Tell us what concerns you and we will establish what the assistants say about you now, before recommending a policy either way.