Resolving Cloudflare Bot Protection Issues with AddSearch Crawler

Overview

Cloudflare’s security features, while excellent for protecting websites from malicious bots, can sometimes block legitimate crawlers like the AddSearch bot. This article explains how to identify and resolve issues when Cloudflare prevents our crawler from indexing your content.

Symptoms of Cloudflare Blocking

You may be experiencing Cloudflare blocking if:

• Your search index shows zero or very few documents
• Document count suddenly drops
• Documents aren’t updating despite confirmed changes to your site
• Search results are incomplete or outdated

Common solutions

Option 1: Whitelist the AddSearch Bot

If you prefer to keep Bot Fight Mode enabled, you can specifically whitelist our crawler:

1. Go to your Cloudflare dashboard
2. Navigate to Security > WAF > Tools
3. Create a new rule:
4. Set the field to “User Agent”
5. Set the operator to “contains”
6. Set the value to “AddSearchBot”
7. Set the action to “Allow” or “Skip”
8. Save and deploy the rule
9. For detailed instructions, see our whitelisting documentation here

Option 2: Disable Bot Fight Mode

The most straightforward solution is to disable Cloudflare’s Bot Fight Mode:

1. Log in to your Cloudflare dashboard
2. Navigate to Security > Bot Management
3. Locate the Bot Fight Mode setting and disable it
4. Wait 1-2 hours for changes to propagate

Troubleshooting steps

If you’re still experiencing issues after applying the solutions above:

• Identify the specific blocking rule:
• Use our Single Page Recrawl tool to trigger a crawl request
• Check your Cloudflare logs immediately after the request
• Look for entries related to “AddSearchBot” or blocked bot activity
• Note which specific rule is triggering the block

Check for firewall rules:

• Some custom firewall rules may need adjustment
• Look for rules based on IP ranges or request patterns

Temporarily pause other crawlers:

• In some cases, having multiple crawlers active simultaneously can trigger rate-limiting

Important notes

• After making changes in Cloudflare, it may take several hours for the changes to fully propagate
• You can verify if the fix worked by using the Single Page Recrawl tool and checking if content is successfully indexed
• If you continue experiencing issues after trying these solutions, please contact AddSearch support with details of your Cloudflare configuration

Frequently Asked Questions

Q: How can I be sure Cloudflare is the issue?

A: The quickest verification is to temporarily disable Bot Fight Mode. If your index begins to populate within a few hours, Cloudflare was likely the cause.

Q: Will whitelisting the AddSearch bot compromise my security?

A: No. Whitelisting only affects our specific crawler and doesn’t reduce security for other traffic.

Q: How long will it take for my index to rebuild after fixing the issue?

A: Depending on the size of your site, it may take anywhere from a few hours to 24 hours for a complete reindex.