Indexing

While AddSearch bots can easily crawl most sites, it may be that the content you’d like to crawl is behind a login. In such cases, AddSearch supports both indexing content behind authentication and securing your index. For more information on securing your index, take a look at our documentation on using JWT tokens.

For authenticated crawling, we support the following login mechanisms:

If you are using a login method not noted above, contact our Support channel to further discuss a solution.

To ensure that we have access to content behind your login, we recommend whitelisting our bots.

An alternative way of indexing private content is pushing it to the index with our Indexing API This allows you to control what is indexed.

Contact us for more information.  

Was this helpful?

Need more help?

We’re always happy to help with code or other questions you might have. Search our documentation, contact support, or connect with our sales team.