SEO & Digital Marketing

	Additional Articles: Search Engines Advanced search Anti Spam Crawlers dmoz Robots Meta Tag robots.txt file Submitting a site Walla
Search Engines

• Crawler
• Page Index
• Search Interface
• Website that are not Search Engines

A search engine is a tool that enables Internet users to locate, on demand, relevant information on topics of their interest. The most popular search engine today is Google.

The term search engines refers to websites that contain three main components:

Crawler

The modern search engines use special programs to scan the web. The nickname given to these programs is "crawlers" because this program crawls from page to page through links that appear on the various pages, and gathers every page that it encounters.

The crawler transfers the collected pages to a page index - updating pages that already exist in the database and adding new pages that do not appear on the index.

Sometimes the crawler will perform a shallow scan: it will only scan the site's main pages, important pages and pages that are frequently updated. Sometimes the crawler will perform a deep crawl and scan all of the pages on a website.

Pages that are frequently scanned by a search engine have two major advantages:

• Some of the search engines give an advantage to pages that have been recently scanned in their search results.
• In case you want to submit a new site to a search engine's index, you can do it easily by adding a link to the new site from a page that is frequently scanned by search engines.

Page Index

The index is where a search engine stores the information collected by its robots on the Internet. When a search is performed on the search engine, it is actually performed on the index and not on the web; therefore the freshness of the search results depends on the freshness level of the engine's index.

The search engines' database is built to execute a quick search for keywords that appear in different pages or links that lead to these pages.

Search Interface

When a user is interested in finding material on a certain subject, he uses the part of the search engine that is called a "search interface". This part of the engine is used for locating the most relevant result for the search that the user performed on the search engine's index.

The search interface output is a list of webpages where the requested information should be found. These lists are called Search Engine Result Pages - SERPs.

The order by which the search engine displays the results is called a search algorithm, which is the actual core of the system.

Website that are not Search Engines

There are many kinds of websites that are also used for locating information, but are not defined as search engines. The most evident example is sites that serve as site indexes, where you can find a catalogue of links to websites that is sorted by a hierarchical topic tree. A new site is added to the index catalogue when it is suggested by one of its users. The visitor that proposes a website to the index lists the site's name and a short description of its content. Next, the site is scanned by one of the index's editors, and in case it is suitable, it will be added to the index. The information that is displayed on the index about the site is a combination of the description proposed by the submitting user and alterations by the index's editor. This kind of sites index is based on human editing and requires many resources in order to maintain the freshness of its information. An example for this kind of an index is dmoz.org.

Additional Information about Crawlers: Scan Robots (Crawlers)

© 2004-2013 SEO Israel Technologies Ltd.
All rights reserved to SEO Israel.