In Security Ninja, we have implemented a feature to automatically validate known crawlers to ensure they are legitimate and not malicious bots. This validation process helps in reducing false positives and ensures genuine crawlers can access your website without being blocked. By recognizing and allowing legitimate crawlers, we help maintain your site’s visibility on search engines and other essential services.
How Does It Work?
The code in Security Ninja validates the IP addresses of incoming requests against a list of known crawler hostnames. Here’s a detailed overview of how this process works:
- It first checks if the IP has already been validated. If the IP is found in the list of previously validated crawlers, it immediately returns
true. - If the IP has not been validated yet, the code retrieves the hostname associated with the IP address. This step translates the IP into a human-readable hostname.
- The retrieved hostname is then checked against a list of known valid crawler hostnames. This list includes popular and trusted search engines and crawlers.
- If a match is found, the function performs a reverse DNS lookup to verify that the IP address corresponds to the hostname. This ensures that the IP is legitimately associated with the trusted hostname.
- If the IP and hostname match, the IP is added to the list of validated crawlers. If not, the code marks the IP as from a not-validated crawler, and the security check of the request continues.
List of Automatically Validated Crawlers
Below is the list of crawlers that Security Ninja automatically validates:
.crawl.baidu.com- Baidu is a major Chinese search engine, and this domain is used for its web crawling activities.
.crawl.baidu.jp- Baidu’s Japanese search engine crawler.
.search.msn.com- Used by Microsoft’s Bing search engine to crawl the web and index pages.
.google.com- Used by Google’s various web crawlers, including those for indexing and other search-related tasks.
.googlebot.com- Specifically used by Googlebot, the web crawler for Google’s search engine.
.crawl.yahoo.net- Yahoo’s search engine crawler, used to index web pages for its search engine.
.yandex.ru- Yandex is a Russian search engine, and this domain is used for its web crawling activities.
.yandex.net- Another domain used by Yandex for its web crawling operations.
.yandex.com- Used by Yandex’s international web crawlers for indexing web pages outside of Russia.
.petalsearch.com- Petal Search (Huawei), used for its crawling activities.
applebot.apple.com- Applebot, used primarily for Siri and Spotlight Suggestions.
.ahrefs.com- Ahrefs SEO toolset crawler for indexing web pages.
.semrush.com- SEMrush SEO tool crawler for gathering data.
.duckduckgo.com- DuckDuckGo privacy-focused search engine crawler.
facebookexternalhit.com- Used by Facebook to scrape link previews when content is shared.
.commoncrawl.org- Common Crawl, non-profit open repository of web crawl data.
.googleother.com- Various other Google crawlers outside the primary Googlebot domain.
.google-inspectiontool.com- Google’s inspection tool for analyzing and crawling web pages.
.swiftype.com- Swiftype enterprise search crawler.
.sogou.com- Sogou Chinese search engine crawler.
.yahoo.com- Yahoo’s international search engine crawlers.
.bing.com- Bing (Microsoft) search engine crawlers for indexing.
Benefits of Automatically Validated Crawlers
This feature ensures that legitimate crawlers, such as search engine bots, are not blocked by the firewall, allowing them to index your website properly. By validating these crawlers, we prevent unnecessary blocks and ensure that your site remains visible and accessible to search engines.
Troubleshooting
If you encounter issues with legitimate crawlers being blocked, ensure that your Security Ninja plugin is up-to-date and that the crawler’s IP addresses are correctly resolving to the known hostnames listed above. You can manually add these IPs to the whitelist if necessary.
It’s important to note that not all search engine or robot crawler systems support automatic validation. In some cases, manual whitelisting might still be necessary. If you notice a crawler with an automatic validation system that is not supported, please contact our support so we can add the system to the list.
For more details on how to manage and troubleshoot crawler validation, visit our documentation on creating or updating database tables.