The Crawler Status prompt displays the following information in the Dashboard page of the AddSearch dashboard.
Some of the URLs in the crawl queue will not be indexed. For instance, robots.txt rules, server-side permissions, and redirects may prevent our crawlers’ from accessing the URLs. Also, some pages might be detected as duplicates and are left out from the index.
If the crawling queue is empty, your index is up-to-date. However, if the number of URLs in the crawl queue increases and doesn’t decrease within a couple of days, contact our support.
AddSearch runs crawling in batches. Our crawlers check for pages queued for recrawling and update them regularly. As this process runs continuously in the background, usually the “Starting soon” status is displayed in the Dashboard.
When you start a full recrawl the “Starting soon” status is displayed until the next crawl batch starts.