Does your website have a Robots.txt files? They are located off of your domains root, usually at at www.your-domain.com/robots.txt.
A robots.txt file provides direction to search engines like Google and Bing on which pages to access and index and which pages they should not. For example, your robots.txt file may tell search engines not to access or index your websites order confirmation page. Managing which pages search engines are indexing on your site can help with privacy and help with your site's SEO.
Search engines are constantly crawling the internet with programs often referred to as "spiders" or "robots". These crawling programs search websites and send information back to the search engines regarding the pages and their content. The robots.txt file has the ability to provide instructions to these crawling program.
Magento allows you to manage your robots.txt file to your website directly from the admin. To do so, within the admin, go to Stores > Configuration. Under the General Menu, choose the Design tab.
From the Design tab, you will see a number of options available related to instructions for search engine robots.
The first is the Default Robots dropdown box. It allows you to specify meta tags for crawling robots that will be included the HTML code on pages before the closing <HEAD> tag. There are several options available, they are:
"Index, Follow"; "NoIndex, Follow"; "Index, NoFollow"; and "NoIndex, NoFollow"
The Second option is the ability to Edit the robots.txt file on your website. It is within this area that you are able to provide very specific instructions to the robots crawling your website. You can specify directories that should not be crawled, call out the location of your website's xml sitemap, and more.
The final option, the Reset to Default, can be used to restore the robots.txt file to the systems default system settings. Be sure to select the Save Config button to save any changes made.