By default, AddSearch includes all pages of your website in your search. You might want to exclude certain parts of your site from the search in certain situations. Also, sometimes, you might want to make sure that some specific content on your page gets included in or excluded from the search.
If you don’t want certain site areas or pages to appear in the search, you can do that in multiple ways:
To exclude all pages located under /temp/, just enter /temp/ into the text field and select Exclude from the dropdown box. Changes, that are done in the Site Areas, take effect with the next recrawl of your site, which for large sites might take some time.
You can also use robots.txt, which is a text file that resides in the root folder of your website. To make AddSearch and other search engines exclude pages under /temp/ from search, insert the following lines into your robots.txt file:
To make only AddSearch exclude pages under /temp/, insert the following lines into your robots.txt file:
You can also use robots meta tags and the index/noindex and follow/nofollow rules. These need to be placed in the
HEAD section of your page.
To make AddSearch and other search engines exclude a page from the search, insert the following into the
HEAD section of that page:
<meta name="robots" content="noindex">
To make AddSearch and other search engines exclude all pages from the search that a page links to (i.e. so that the crawler won’t follow links from this page) insert the following into the
HEAD section of that page:
<meta name="robots" content="nofollow">
You can also combine instructions. To make AddSearch and other search engines follow links from this page, but exclude this page from the search, insert the following into the
HEAD section of the page:
<meta name="robots" content="follow, noindex">
If you want to have the robots meta tags apply to AddSearch only, change
name="robots"in the above examples to
Finally, you can use the
rel="nofollow" attribute in links, to exclude the page you’re linking to from the search. Include the
rel="nofollow" attribute in the link like this:
<a href="/content/excluded_page" rel="nofollow">Link to page that is excluded from search</a>
AddSearch detects automatically the main content in your pages and indexes only that main content. This makes the search results better, as superfluous content (such as sidebars, headers, footers, etc.) and duplicate content is not included in the search.
Sometimes you might want to manually make sure that certain content is included in or excluded from the search. You can insert HTML5 tags to your pages to mark such content.
To make AddSearch include a certain section in your page into the search, use the
data-addsearch="include" attribute in any element, typically a
<div data-addsearch="include">Content that is included in the search</div>
To make AddSearch exclude a certain section in your page from the search, use the
data-addsearch="exclude" attribute in any element, typically a
P element. Example:
<div data-addsearch="exclude">Content that is excluded from the search</div>
If you have multiple versions of a web page and want to tell the crawlers to index only one you can use canonical links. For detailed information on canonical links visit the here.
AddSearch allows you to define parameters that should be ignored when our crawlers collect new links. Read more about setting up the ignored parameters here