Skip to content
Docs
CtrlK
Go to Dashboard
AI & Bot Governance

Control which AI bots and headless web scrapers can access your website content with easy category switches

Many modern website owners are concerned about automated AI companies scraping their proprietary articles, product photography, and unique catalog data without permission. In addition, aggressive scraping bots can make thousands of requests an hour, driving up your hosting bandwidth bills and slowing down your website for real human visitors.

AI and Bot Governance screen displaying category cards, AI crawler presets, and per-bot override dropdowns
Manage AI crawlers, headless scrapers, and bot rate limits on the AI & Bot Governance tab.

Types of bots WebOptiva manages#

  • AI Model Training Crawlers: Crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot, Bytespider, Google-Extended, Meta-ExternalAgent, Amazonbot, Cohere-AI, and Applebot-Extended that gather web data to train language models.
  • AI Search & Citation Bots: Real-time retrieval agents such as PerplexityBot, ChatGPT-User, Claude-Web, and OAI-SearchBot that fetch pages to answer live user questions and cite your content.
  • Scrapers & Headless Tools: Automated frameworks such as Diffbot, Scrapy, Headless Chrome, Playwright, Puppeteer, Selenium, and raw HTTP script libraries (python-requests, curl, Wget) that download pages in bulk.

Search and AI citation visibility#

AI Search & Citation bots are allowed by default, so your pages can still appear in AI-generated answers with source links. Every category can be changed at any time, and per-bot overrides let you fine-tune individual crawlers.

Step-by-step: how to control bot access#

  1. In Security operations, click the AI & Bot Governance tab in the sub-navigation bar.
  2. Choose your website domain name from the domain selector dropdown.
  3. Use the three category cards — AI Model Training Crawlers, AI Search & Citation Bots, and Scrapers & Headless Tools — and click Allow, Throttle, or Block for each group.
  4. To customize an individual bot (for example, if you want to block GPTBot but allow ClaudeBot), find its row in the Granular Bot Directory & Overrides table and choose an Override Rule from the dropdown.
  5. Optionally set the Bot Throttle Rate Limit (req/min) shared by throttled bots.
  6. Click Save Draft at the bottom of the card.
  7. Click Deploy changes to send your bot rules live across our global network.

Expected result#

Complete control over who reads and indexes your digital content, saving server bandwidth and protecting your hard-earned business data.

Back to AI & Bot Governance