How Do Search Engines Work?

Crawler-based search engines have three major elements. First is the spider, or crawler. The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being "spidered" or "crawled." The spider returns to the site on a regular basis. to look for changes.

Second is the search engine index. All changes the spider finds goes into the index. The index, sometimes called the catalog, is like an immense repository containing a copy of every web page that the spider finds.

Proprietary search engine software is the third part of a search engine. This is the program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant.

Search engines go about determining relevancy by following a set of rules, known as an algorithm. One of the the main rules in a ranking algorithm involves the location and frequency of keywords on a web page.

There are many location considerations. Pages with the search terms appearing in the HTML title tag are often assumed to be more relevant than others to the topic. Search engines will also check to see if the search keywords appear near the top of a web page, such as in the headline or in the first few paragraphs of text. They assume that any page relevant to the topic will mention those words right from the beginning.

Frequency is the other major factor in how search engines determine relevancy. A search engine will analyze how often keywords appear in relation to other words in a web page. Those with a higher frequency are often deemed more relevant than other web pages.

Each search engine has developed their own algorithm, it is this difference that provides you with different results from one search engine to another, in addition to the fact that some search engines have a larger number of web pages in their repository than others may. Search engines may also penalize pages or exclude them from the index, if they detect search engine "spamming." An example is when a word is repeated hundreds of times on a page, to increase the frequency and propel the page higher in the listings.

Due to the importance of search engine ranking in the success of your web site, you may want to consider hiring a search engine promotion professional to assist you. A professional can help your site meet the search engine algorithm requirements necessary to obtain a high ranking for the terms most likely to be used when a potential visitor searches for a site similar to yours. At Digital Catapult you can talk to a search engine professional from 9 AM to 5 PM Eastern time at 1-866-SLINGME (754-6463), or fill out our request form if you think professional assistance may benefit your web site's rankings.