{"id":8059,"date":"2019-07-24T10:44:26","date_gmt":"2019-07-24T07:44:26","guid":{"rendered":"https:\/\/www.seoisrael.co.il\/?post_type=kb&#038;p=8059"},"modified":"2023-12-13T11:23:04","modified_gmt":"2023-12-13T09:23:04","slug":"robots-txt-file","status":"publish","type":"kb","link":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file","title":{"rendered":"robots.txt file"},"content":{"rendered":"\n<p>A considerable part of the crawling robots that operate today will look for the robots.txt file in the main directory of your website (https:\/\/seoisrael.com\/robots.txt). This file should help them decide which parts of the site they should avoid entering.<\/p>\n\n\n\n<p class=\"has-text-align-left\"><strong>Please note<\/strong>: Prohibiting a search engine\u2019s access to a certain page will indeed prevent the collection of the page\u2019s&nbsp;<span style=\"text-decoration: underline;\">content<\/span>, but sometimes, if there are references to that page in pages where the search engines can enter, then the page&nbsp;<span style=\"text-decoration: underline;\">will appear<\/span>&nbsp;in the search results, but without the information about it (title, description, etc.) If you want to prevent the page from appearing completely, use the second method (<a href=\"https:\/\/www.seoisrael.com\/?kb=robots-meta-tag\">robots tag<\/a>).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How do you create a robots.txt file and what is it made of?<\/h2>\n\n\n\n<p>The file must be prepared in a regular text editor (notepad) and not in another software.<\/p>\n\n\n\n<p>In order to better understand how this file is structured, let&#8217;s examine a code example:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>User-agent: *<br>Disallow: \/cgi-bin\/<br>Disallow: \/images\/<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>If we look at the code, it seems that there are two parts:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><b>User-Agent<\/b>: This part defines to whom the instructions that will come immediately after it are addressed.<\/li>\n\n\n\n<li><b>Disallow<\/b>: Which parts of the site do we wish to prevent access from those defined in the User-Agent field.<\/li>\n<\/ul>\n\n\n\n<p>In the case above, we asked all search engines (the mark for all search engines together is *) to avoid access to the cgi-bin directory and the images directory.<\/p>\n\n\n\n<p>Let&#8217;s look at another code example:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>User-agent: *<br>Disallow: \/<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"direction: ltr;\">This example will prevent all search engines from accessing the entire site, meaning the search engines will not crawl the site at all. <\/p>\n\n\n\n<p>A more complicated example:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>User-agent: <a href=\"https:\/\/www.seoisrael.com\/?kb=google-crawling-and-indexing\" title=\"googlebot\">googlebot<\/a><br>Disallow: \/bonbons\/<br>Disallow: bonbons.htm<br><br>User-agent: bonboncrawler<br>Disallow: \/<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>In this example we gave instructions to two different robots. The Googlebot robot was instructed to avoid access to the bonbons directory, and to the bonbons.htm file. The second directive refers to the bonboncrawler robot, which we have banned from accessing the site altogether.<\/p>\n\n\n\n<p>And the last example:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>User-agent: googlebot<br>Disallow:User-agent: *<br>Disallow: \/<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Note that the first ban (Googlebot) is empty! So, it basically means that Googlebot <span style=\"text-decoration: underline;\">can crawl all the pages of the website<\/span>. The second ban prevents all robots from crawling the site. Seemingly a contradiction between the two instructions, but in fact when there is a sign *, which contradicts a more specific instruction, then the more specific instruction wins.<\/p>\n\n\n\n<p>The meaning of the previous code is actually &#8211; Googlebot can scan the entire site, while every other bot can&#8217;t.<a name=\"anchor1\"><\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">robots.txt for forum websites<\/h2>\n\n\n\n<p>A robots.txt file can help you prevent robots from browsing parts of your forum that do not include useful information. An example of pages that you should prevent access to is the user profile pages, the search page, the page for writing a new message and the login page. To prevent access to these pages, you can create a file like this:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>User-agent: *<br>Disallow: \/forum\/post.asp<br>Disallow: \/forum\/post.asp<br>Disallow: \/forum\/user_profile.asp<br>Disallow: \/forum\/search.asp<br>Disallow: \/forum\/password.asp<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Should I create a robots.txt file even if it is not needed?<\/h2>\n\n\n\n<p>In a video posted by Matt Cutts of Google, on August 19, 2011, he addresses this question. The short answer is this: &#8220;Think of it this way: Search engine robots in general, and Google in particular, are constantly accessing the path of the Robots.txt file on your site. What if the file isn&#8217;t there? Will the server return a server error response ( response code 500) or page not found response (response code 404)? How will the robot treat this response? Is the risk worth the effort involved in uploading the file to the server?&#8221;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A considerable part of the crawling robots that operate today will look for the robots.txt file in the main directory of your website. This file should help them decide which parts of the site they should avoid entering.<\/p>\n","protected":false},"author":4,"featured_media":9077,"parent":0,"menu_order":0,"template":"","format":"standard","meta":{"footnotes":""},"class_list":["post-8059","kb","type-kb","status-publish","format-standard","has-post-thumbnail","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>robots.txt file - SEO Glossary - SEO Israel<\/title>\n<meta name=\"description\" content=\"Want to know more about robots.txt file? Detailed information, instructions and explanations about robots.txt file from the SEO ISRAEL team of experts!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"robots.txt file - SEO Glossary - SEO Israel\" \/>\n<meta property=\"og:description\" content=\"Want to know more about robots.txt file? Detailed information, instructions and explanations about robots.txt file from the SEO ISRAEL team of experts!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file\" \/>\n<meta property=\"og:site_name\" content=\"SEO Israel\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/seoisrael.co.il\/\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-13T09:23:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.seoisrael.com\/wp-content\/uploads\/SEOLogonotext-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file\",\"url\":\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file\",\"name\":\"robots.txt file - SEO Glossary - SEO Israel\",\"isPartOf\":{\"@id\":\"https:\/\/www.seoisrael.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.seoisrael.com\/wp-content\/uploads\/master-1.jpg\",\"datePublished\":\"2019-07-24T07:44:26+00:00\",\"dateModified\":\"2023-12-13T09:23:04+00:00\",\"description\":\"Want to know more about robots.txt file? Detailed information, instructions and explanations about robots.txt file from the SEO ISRAEL team of experts!\",\"breadcrumb\":{\"@id\":\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#primaryimage\",\"url\":\"https:\/\/www.seoisrael.com\/wp-content\/uploads\/master-1.jpg\",\"contentUrl\":\"https:\/\/www.seoisrael.com\/wp-content\/uploads\/master-1.jpg\",\"width\":1200,\"height\":628,\"caption\":\"\u05d7\u05d3\u05e9\u05d5\u05ea \u05e7\u05d9\u05d3\u05d5\u05dd \u05d0\u05ea\u05e8\u05d9\u05dd \u05e9\u05dc SEO \u05d9\u05e9\u05e8\u05d0\u05dc\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"SEO Israel\",\"item\":\"https:\/\/www.seoisrael.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SEO Glossary\",\"item\":\"https:\/\/www.seoisrael.com\/?post_type=kb\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"robots.txt file\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.seoisrael.com\/#website\",\"url\":\"https:\/\/www.seoisrael.com\/\",\"name\":\"SEO Israel Technologies Ltd\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.seoisrael.com\/#organization\"},\"alternateName\":\"SEO Israel\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.seoisrael.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.seoisrael.com\/#organization\",\"name\":\"SEO Israel Technologies Ltd\",\"alternateName\":\"SEO Israel\",\"url\":\"https:\/\/www.seoisrael.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.seoisrael.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.seoisrael.com\/wp-content\/uploads\/SEOLogonotext.png\",\"contentUrl\":\"https:\/\/www.seoisrael.com\/wp-content\/uploads\/SEOLogonotext.png\",\"width\":350,\"height\":350,\"caption\":\"SEO Israel Technologies Ltd\"},\"image\":{\"@id\":\"https:\/\/www.seoisrael.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/seoisrael.co.il\/\",\"https:\/\/www.linkedin.com\/company\/seo-israel\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"robots.txt file - SEO Glossary - SEO Israel","description":"Want to know more about robots.txt file? Detailed information, instructions and explanations about robots.txt file from the SEO ISRAEL team of experts!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file","og_locale":"en_US","og_type":"article","og_title":"robots.txt file - SEO Glossary - SEO Israel","og_description":"Want to know more about robots.txt file? Detailed information, instructions and explanations about robots.txt file from the SEO ISRAEL team of experts!","og_url":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file","og_site_name":"SEO Israel","article_publisher":"https:\/\/www.facebook.com\/seoisrael.co.il\/","article_modified_time":"2023-12-13T09:23:04+00:00","og_image":[{"width":350,"height":350,"url":"https:\/\/www.seoisrael.com\/wp-content\/uploads\/SEOLogonotext-1.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file","url":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file","name":"robots.txt file - SEO Glossary - SEO Israel","isPartOf":{"@id":"https:\/\/www.seoisrael.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#primaryimage"},"image":{"@id":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#primaryimage"},"thumbnailUrl":"https:\/\/www.seoisrael.com\/wp-content\/uploads\/master-1.jpg","datePublished":"2019-07-24T07:44:26+00:00","dateModified":"2023-12-13T09:23:04+00:00","description":"Want to know more about robots.txt file? Detailed information, instructions and explanations about robots.txt file from the SEO ISRAEL team of experts!","breadcrumb":{"@id":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.seoisrael.com\/?kb=robots-txt-file"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#primaryimage","url":"https:\/\/www.seoisrael.com\/wp-content\/uploads\/master-1.jpg","contentUrl":"https:\/\/www.seoisrael.com\/wp-content\/uploads\/master-1.jpg","width":1200,"height":628,"caption":"\u05d7\u05d3\u05e9\u05d5\u05ea \u05e7\u05d9\u05d3\u05d5\u05dd \u05d0\u05ea\u05e8\u05d9\u05dd \u05e9\u05dc SEO \u05d9\u05e9\u05e8\u05d0\u05dc"},{"@type":"BreadcrumbList","@id":"https:\/\/www.seoisrael.com\/?kb=robots-txt-file#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"SEO Israel","item":"https:\/\/www.seoisrael.com\/"},{"@type":"ListItem","position":2,"name":"SEO Glossary","item":"https:\/\/www.seoisrael.com\/?post_type=kb"},{"@type":"ListItem","position":3,"name":"robots.txt file"}]},{"@type":"WebSite","@id":"https:\/\/www.seoisrael.com\/#website","url":"https:\/\/www.seoisrael.com\/","name":"SEO Israel Technologies Ltd","description":"","publisher":{"@id":"https:\/\/www.seoisrael.com\/#organization"},"alternateName":"SEO Israel","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.seoisrael.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.seoisrael.com\/#organization","name":"SEO Israel Technologies Ltd","alternateName":"SEO Israel","url":"https:\/\/www.seoisrael.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.seoisrael.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.seoisrael.com\/wp-content\/uploads\/SEOLogonotext.png","contentUrl":"https:\/\/www.seoisrael.com\/wp-content\/uploads\/SEOLogonotext.png","width":350,"height":350,"caption":"SEO Israel Technologies Ltd"},"image":{"@id":"https:\/\/www.seoisrael.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/seoisrael.co.il\/","https:\/\/www.linkedin.com\/company\/seo-israel\/"]}]}},"_links":{"self":[{"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=\/wp\/v2\/kb\/8059","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=\/wp\/v2\/users\/4"}],"version-history":[{"count":22,"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=\/wp\/v2\/kb\/8059\/revisions"}],"predecessor-version":[{"id":17083,"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=\/wp\/v2\/kb\/8059\/revisions\/17083"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=\/wp\/v2\/media\/9077"}],"wp:attachment":[{"href":"https:\/\/www.seoisrael.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8059"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}