SEO Israel - Professional SEO Services

SEO & Digital Marketing


robots.txt file

» »

In order to control the way search robots process certain pages on a website, it is possible to use the robots tag. This tags controls the following factors:

  • Whether or not to include the page on search engine results.
  • Whether or not to follow outgoing links from the page.

The major problem with this method (as well as with others) is that there are robots that simply ignore the site owner's requests and scan despite specific prohibitions. However, these methods work great with the major search engines.

Where to place the robots meta tag

The robots tag is a meta tag, and like other meta tags, it should be placed at the first part of the HTML page that is called HEAD.

Code example:



<meta name="robots" content="noindex,nofollow">

<title>Blue Widgets</title>




Robots tag structure

If we examine the code above, we will see that the robots tag consists of two parts:

  • name: includes the text 'robots' and tells the robots that this command is directed at them.
  • content: the command's content.

The content includes two commands directed at the search engine, separated by a comma:

  • index/noindex: whether to save the content of the page in the search engine database.
  • follow/nofollow: whether to follow links that appear on the page to other pages.

Note: the robot tag is not case sensitive.

GoogleBot Unique Commands

Google's robot (which is also called ' Googlebot) recognizes two additional commands: NOARCHIVE and NOSNIPPET.

Usually Google stores the content of its scanned pages in an enormous database that is called Cache. The purpose of this cache is to allow access to the page's content even if it's not active currently, or that the content has changed since Google's last scan. In addition, this cache is used to create the short text (which is a part of the page text) that is displayed with the search results. This text is called Snippet.

In order to see the content of the page stored in Google's cache, the user should press the 'cache' link that appears near the search results.

In order to prevent storing your website's content in Google's cache, you can use the following code:



<meta name="robots" content="index,follow,noarchive">

<title>Blue Widgets</title>




In order to prevent the short text from your page text (Snippet) to appear in the search results, you can add the following code line:

<meta name="googlebot" content="nosnippet">

Please note that this line also cancels your website's storage in Google's cache.

A customer that couldn't find you is your competitor's customer..

Please fill in your site's details so that we can contact you as soon as possible and answer your questions:

SEO Israel, 1'st Hamada St., Rehovot 76703, Israel. (+972)-73-2240000
Contact SEO Israel
Info Center

Additional Resources:

SEO Blog (Hebrew)

Setting Up a New Site:

Domain Name Registration
Website Hosting

Search Engines Info:

Search Engines


Google Adsense
Internet Marketing
Link Development
Optimization Techniques
Web Analytics
Google AdWords

Our Company:

About SEO Israel
Contact Us
Our Clients
SEO/SEM Jobs in Israel
הסמכת גוגל AdWords הסמכת גוגל אנליטיקס
Site Map, Hebrew Site Valig XHTML 1.0
SEO Israel Facebook Page SEO Israel on LinkedIn Follow SEO Israel on Twitter SEO Israel on Google+