Website Security Guide
AI & Bot Governance
Autonomous AI bots and competitor scrapers constantly crawl the web to scrape articles, product catalogs, and pricing data. WebOptiva gives website owners easy switches to control exactly which bots are allowed on their site.
How it works
- Control which AI bots and headless web scrapers can access your website content with easy category switches
- Understand when to throttle bots instead of blocking them completely to maintain gentle discovery without server overload
Good to know
- AI Search & Citation bots such as PerplexityBot, ChatGPT-User, and Claude-Web are allowed by default so your content can still be cited in AI-generated answers.
- Choosing 'Throttle' for a bot allows it to index your content at a calm, controlled pace without overloading your hosting server.