SEO Israel - Professional SEO Services

SEO & Digital Marketing

 
 

robots.txt file

 
SEO Israel » Search Engines » robots.txt file

Additional Articles:
Search Engines
Walla
Advanced search
dmoz
Crawlers
Submitting a site
Anti Spam
robots.txt file
Robots Meta Tag
robots.txt file
Where to place the robots meta tag
Robots tag structure
GoogleBot Unique Commands

In order to control the way search robots process certain pages on a website, it is possible to use the robots tag. This tags controls the following factors:

Whether or not to include the page on search engine results.
Whether or not to follow outgoing links from the page.

The major problem with this method (as well as with others) is that there are robots that simply ignore the site owner's requests and scan despite specific prohibitions. However, these methods work great with the major search engines.

Where to place the robots meta tag

The robots tag is a meta tag, and like other meta tags, it should be placed at the first part of the HTML page that is called HEAD.

Code example:

<html>
<head>
<meta name="robots" content="noindex,nofollow">
<title>Blue Widgets</title>
</head>
<body>
...

Robots tag structure

If we examine the code above, we will see that the robots tag consists of two parts:

name: includes the text 'robots' and tells the robots that this command is directed at them.
content: the command's content.

The content includes two commands directed at the search engine, separated by a comma:

index/noindex: whether to save the content of the page in the search engine database.
follow/nofollow: whether to follow links that appear on the page to other pages.

Note: the robot tag is not case sensitive.

GoogleBot Unique Commands

Google's robot (which is also called ' Googlebot) recognizes two additional commands: NOARCHIVE and NOSNIPPET.

Usually Google stores the content of its scanned pages in an enormous database that is called Cache. The purpose of this cache is to allow access to the page's content even if it's not active currently, or that the content has changed since Google's last scan. In addition, this cache is used to create the short text (which is a part of the page text) that is displayed with the search results. This text is called Snippet.

In order to see the content of the page stored in Google's cache, the user should press the 'cache' link that appears near the search results.

In order to prevent storing your website's content in Google's cache, you can use the following code:

<html>
<head>
<meta name="robots" content="index,follow,noarchive">
<title>Blue Widgets</title>
</head>
<body>
...

In order to prevent the short text from your page text (Snippet) to appear in the search results, you can add the following code line:

<meta name="googlebot" content="nosnippet">

Please note that this line also cancels your website's storage in Google's cache.

A customer that couldn't find you is your competitor's customer..

Please fill in your site's details so that we can contact you as soon as possible and answer your questions:


SEO Israel, 1'st Hamada St., Rehovot 76703, Israel. (+972)-73-2240000
Site:
Email:
Phone:
Name:
Contact SEO Israel
Name:
Site:
E-mail:
Phone:
Phone:972-73-2240000
Fax:972-73-2240022
Info Center

Our Company:

About SEO Israel
SEO/SEM Jobs in Israel
Our Clients
Contact Us

Services:

Optimization Techniques
Link Development
Internet Marketing
Google Adsense
Web Analytics
Google AdWords

Search Engines Info:

Search Engines
Google

Setting Up a New Site:

Website Hosting
Domain Name Registration

Additional Resources:

SEO Blog (Hebrew)
הסמכת גוגל AdWords הסמכת גוגל אנליטיקס
Site Map, Hebrew Site Valig XHTML 1.0
SEO Israel Facebook Page SEO Israel on LinkedIn Follow SEO Israel on Twitter SEO Israel on Google+