robots.txt file - SEO Israel


	robots.txt file

SEO Israel

Search Engines

robots.txt file

Where to place the robots meta tag
Robots tag structure
GoogleBot Unique Commands

In order to control the way search robots process certain pages on a website, it is possible to use the robots tag. This tags controls the following factors:

Whether or not to include the page on search engine results.
Whether or not to follow outgoing links from the page.

The major problem with this method (as well as with others) is that there are robots that simply ignore the site owner's requests and scan despite specific prohibitions. However, these methods work great with the major search engines.

Where to place the robots meta tag

The robots tag is a meta tag, and like other meta tags, it should be placed at the first part of the HTML page that is called HEAD.

Code example:

<html>

<head>

<title>Blue Widgets</title>

</head>

<body>

...

Robots tag structure

If we examine the code above, we will see that the robots tag consists of two parts:

name: includes the text 'robots' and tells the robots that this command is directed at them.
content: the command's content.

The content includes two commands directed at the search engine, separated by a comma:

index/noindex: whether to save the content of the page in the search engine database.
follow/nofollow: whether to follow links that appear on the page to other pages.

Note: the robot tag is not case sensitive.

GoogleBot Unique Commands

Google's robot (which is also called ' Googlebot) recognizes two additional commands: NOARCHIVE and NOSNIPPET.

Usually Google stores the content of its scanned pages in an enormous database that is called Cache. The purpose of this cache is to allow access to the page's content even if it's not active currently, or that the content has changed since Google's last scan. In addition, this cache is used to create the short text (which is a part of the page text) that is displayed with the search results. This text is called Snippet.

In order to see the content of the page stored in Google's cache, the user should press the 'cache' link that appears near the search results.

In order to prevent storing your website's content in Google's cache, you can use the following code:

<html>

<head>

<title>Blue Widgets</title>

</head>

<body>

...

In order to prevent the short text from your page text (Snippet) to appear in the search results, you can add the following code line:

Please note that this line also cancels your website's storage in Google's cache.

A customer that couldn't find you is your competitor's customer..

Please fill in your site's details so that we can contact you as soon as possible and answer your questions:

SEO Israel, 1'st Hamada St., Rehovot 76703, Israel. (+972)-73-2240000