By Michail Dimitriou*




A common issue that web designers are facing is the wayto prevent their webpage to be indexed by Google and other search engines. The basic techniques are either to use the noindex tag in the header of html code or to set the page as disallow in the robots.txt file. But there are differences between these two methods.

The noindex tag tells the Google bot who crawls a page not to index it and do not present it in google's results. So noindex tag in general tells to Google bot to read it but not to show it in organic results. The robots.txt file tells to Google bot to not visit a specific page at all. Not just not to index that page but not even to visit and read it. So since Google does not read a web page due to a disallow line in the robots.txt file it cannot read the links that exist on that webpage and these links do not counted in PageRank score.

But the main practical difference between noindex tag and robots.txt file is in Google AdWords, in case that we want a web page not to appear in organic results but only in Google sponcored results. For instance, such a page may be a landing page showing a seasonal offer. In that case, using robots.txt to avoid indexing, we're actually preventing Google bot to reading it. As a result is to be decreased the Quality Score and to be increases in cost per click (CPC), since the landing page itself works as a counting factor of the Quality Score. So the best practice in that case is the usage the of meta-tag noindex.

Optimal usage of noindex and robots.txt

In most cases, it is being suggested to use noindex. Such cases may be a login page, or a thank you page when a form is geting completed. Using robots.txt is being recommended for files that do not contain code, so it's the only solution to prevent them from being indexed. Such files can be image or text files such as txt, ms-word or pfd format.





*Michail Dimitriou is working in the field of iGaming as a digital marketing executive for an iGaming company in the exotic Malta as well as an SEO freelancer. He supports several advertising companies and digital agencies around the world.
Also he is smart and handsome.


18 June 2017