Have you ever come across the message Host Directive Not Given in Robots Text file? Possibly, if you are a webmaster and submitted your website or blog to Yandex webmaster tools. Also the different version of the same, we can’t say error, is applicable to Google Search Console too!
Though there is a section on Info Inspired for webmasters tagged Tips for Webmasters, I rarely post any thing under this as I deeply involved in Spreadsheets tutorials and Quotes. But some times I do write some helpful tips for webmasters and such tutorials are as a result of my own encounter with my blog. Recently I’ve come across this host directive not given in robots.txt file issue and here is the tips to how to solve it on Yandex as well as on Google Webmaster Console.
What is Host directive?
Host directive is to direct the search bot to your original mirror site from a group of your mirror site.
Suppose your main mirror is http://yoursite.com and not http://www.yoursite.com, you can direct the search bot using your robots.txt file. Different search bots behave differently and it’s up the decision-making algorithm of the bots whether to follow your instructions in robots.txt file or not.
How to Solve Host Directive Not Given in Robots Text File?
Yandex search bot is sensitive about Host Directive. So you can solve the host directive not given in Robots.txt issue for Yandex as below.
Solve Host Directive Issue: Yandex Webmaster Tool
On the very first line of your robots.txt file add the below line depending your main mirror site.
Solve Host Directive Issue: Google Webmaster Console.
The procedure to solve the host directive issue is different in Google.
On Google Webmaster Console, you might have already added your site. If you have added the “www” version of your site, again add the “non www” version to the webmaster console. Then on the settings menu set your preferred domain. Refer the screenshot below.
Once you click the Site Settings, you will be provided with preferred domain and crawl rate options. Set it as per your preference.
I didn’t mention how to add site to Yandex / Google / Bing webmaster tools realising you have already added your site to these search engines and that’s why you are here to find the solution to the above host directive issue.