The Crawler Status prompt displays the following information in the Dashboard page of the AddSearch dashboard.

  • Docs in index: the number of indexed URLs
  • Crawling queue: the number of URLs scheduled for crawling or recrawling

Picture of Crawler status prompt in the AddSearch Dashboard page

Some of the URLs in the crawl queue will not be indexed. For instance, robots.txt rules, server-side permissions, and redirects may prevent our crawlers’ from accessing the URLs. Also, some pages might be detected as duplicates and are left out from the index.

If the crawling queue is empty, your index is up-to-date. However, if the number of URLs in the crawl queue increases and doesn’t decrease within a couple of days, contact our support.

  • Crawler status:
    1. Crawling: the crawlers are crawling URLs
    2. Starting soon: the crawlers are not crawling URLs

AddSearch runs crawling in batches. Our crawlers check for pages queued for recrawling and update them regularly. As this process runs continuously in the background, usually the “Starting soon” status is displayed in the Dashboard.

When you start a full recrawl the “Starting soon” status is displayed until the next crawl batch starts.

Was this helpful?

Need more help?
We're always happy to help with code or other questions you might have. Search our documentation, contact support, or connect with our sales team.