Security Ninja can recognize legitimate crawlers so search engines, monitoring tools, and published AI bots are less likely to be blocked by mistake. Validation runs as part of the firewall visitor checks.
How validation works
The process depends on the type of crawler:
Classic search and SEO crawlers
For User-Agents that look like a known crawler (Googlebot, Bingbot, Ahrefs, and similar), Security Ninja may run a reverse-DNS lookup on the visitor IP and confirm the hostname matches a trusted suffix (for example .googlebot.com or .search.msn.com).
Normal browser traffic does not trigger this DNS lookup. The check runs only when the User-Agent looks like a crawler.
If the IP and hostname match, the IP is remembered so repeat visits skip the extra work. The remembered list is capped at 200 entries.
AI crawlers (OpenAI, Perplexity, Anthropic)
When the User-Agent matches published bots such as GPTBot, PerplexityBot, or ClaudeBot, Security Ninja checks the visitor IP against official published IP ranges from each vendor (JSON feeds), not hostname matching.
Supported examples include:
- OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
- Perplexity: PerplexityBot, Perplexity-User
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
Classic crawler hostname suffixes
These hostname patterns are used when reverse-DNS confirmation runs:
.crawl.baidu.com- Baidu search crawler.
.crawl.baidu.jp- Baidu Japan crawler.
.search.msn.com- Microsoft Bing crawler.
.google.com- Google crawlers.
.googlebot.com- Googlebot.
.crawl.yahoo.net- Yahoo crawler.
.yandex.ru,.yandex.net,.yandex.com- Yandex crawlers.
.petalsearch.com- Petal Search (Huawei).
applebot.apple.com- Applebot.
.ahrefs.com- Ahrefs SEO crawler.
.semrush.com- SEMrush crawler.
.duckduckgo.com- DuckDuckGo crawler.
facebookexternalhit.com- Facebook link preview fetcher.
.commoncrawl.org- Common Crawl.
.googleother.com- Other Google crawlers.
.google-inspectiontool.com- Google inspection tools.
.swiftype.com- Swiftype crawler.
.sogou.com- Sogou crawler.
.yahoo.com- Yahoo international crawlers.
.bing.com- Bing crawlers.
If a legitimate crawler is still blocked
- Update Security Ninja to the latest version.
- Check that the crawler’s IP resolves to a known hostname (classic bots) or falls in the vendor’s published range (AI bots).
- Whitelist the IP manually if needed.
- Contact support if you run a crawler with a standard validation method we do not cover yet.
