Filter – securityninja_social_crawler_user_agents

Table of contents

1. What this filter does
2. Default list
3. Usage examples
4. How to include on your website

What this filter does

When the firewall’s Block IP Network (Global Block Network) is enabled, some IP addresses are blocked for all sites using the network. Legitimate crawlers used for link previews (e.g. Facebook, LinkedIn, Twitter) sometimes use those same IP ranges. This filter lets you control which User-Agent strings are treated as “social crawlers” and are not blocked by the Block IP Network, so link previews keep working when you share your site.

You can add or remove User-Agent substrings. The firewall checks the request’s User-Agent; if it contains any of the strings in the list (case-insensitive), the request is not blocked by the Block IP Network, even if the IP is in the block list.

Default list

The default list includes: facebookexternalhit, Facebot, LinkedInBot, Twitterbot, Slackbot-LinkExpanding, Discordbot.

Usage examples

Add an extra crawler (e.g. Pinterest)

add_filter( 'securityninja_social_crawler_user_agents', 'my_social_crawlers' );

function my_social_crawlers( $user_agents ) {
	$user_agents[] = 'Pinterestbot';
	return $user_agents;
}

Remove one of the default crawlers

add_filter( 'securityninja_social_crawler_user_agents', 'my_social_crawlers' );

function my_social_crawlers( $user_agents ) {
	return array_diff( $user_agents, array( 'Discordbot' ) );
}

Disable the bypass (block everyone in the Block IP Network again)

add_filter( 'securityninja_social_crawler_user_agents', '__return_empty_array' );

Important

This filter only affects blocking by the Block IP Network. Requests can still be blocked by other firewall rules (e.g. the “suspicious request” checks that use the user_agent_items filter). If a User-Agent is in both the social crawler list and the user_agent_items blocklist, the blocklist wins and the request can be blocked.

How to include on your website

Not sure where to add this code? See How to include custom code on your website.

Written by Lars Koudal