Search bots vs AI bots — what to allow?
- Search engines (Googlebot, Bingbot) should always be allowed for pages you want in Google and Bing.
- AI search bots fetch pages to answer questions in ChatGPT search, Perplexity and Claude — allowing them can bring visibility and visits.
- AI training bots collect content to train models. Blocking them is a business choice and does not remove you from Google Search.
- robots.txt is a request, not security — never rely on it to hide private data.
Learn more in our AEO & GEO guide.
Frequently asked questions
Where do I upload robots.txt and llms.txt?
Both files go in the root folder of your website, so they open at https://yourdomain.com/robots.txt and https://yourdomain.com/llms.txt.
Does blocking GPTBot remove me from ChatGPT?
GPTBot is used for training. ChatGPT search uses OAI-SearchBot, so you can block training while still allowing search access.
Is llms.txt required?
No. It is an optional proposal. Google has said it does not need AI text files for AI Overviews or AI Mode.
Want experts to do it for you?
SEOYug has helped businesses grow on Google, Maps, social media and AI search since 2007.
Run Free SEO AuditWhatsApp Us