AI & Bot Governance
Understand when to throttle bots instead of blocking them completely to maintain gentle discovery without server overload
While completely blocking a bot is the safest option if you want to protect private content, there are scenarios where you might want a bot to index your pages without allowing it to monopolize your web server.
What does 'Throttle' do?#
When you set a bot's policy to THROTTLE, WebOptiva allows the bot to visit your website, but enforces your configured bot throttle rate limit (10 requests per minute by default). If the bot tries to crawl faster than this speed, WebOptiva gives it a gentle pause notice (HTTP 429 Too Many Requests).
Why use throttling?#
- Gentle Visibility: It allows marketing tools or new AI search engines to discover your content without putting heavy load on your hosting database.
- Cost Control: It caps the bandwidth consumed by automated tools so your monthly hosting bills stay predictable.
- Human Priority: Real shoppers always get 100% of your server's available speed, while automated bots are kept in the slow lane.
Every throttled or blocked bot visit is logged under your Telemetry view so you can see exactly which bots are visiting your site and how often.