Before dealing with the topic of search engine optimization, it makes sense to ask yourself what exactly a search engine is and how it works. The question may sound trivial – but for an optimal search engine optimization you need profound knowledge about how search engines work.
What is a search engine?
Today, it is hard to imagine the Internet without search engines. 98% of web entries are made via a search engine. While in the 90s the Internet was still largely organized via Internet addresses, so-called domains, it is now unusual to enter a domain in its entirety. Nowadays, users even use search engines even though they know the name (and thus the domain of the address). We have become accustomed to searching for everything. The indexing of the Internet by human memory is too cumbersome and so the search engine is used for everything.
And when we talk about a search engine, we actually mean Google. Because Google now has a market share of 94.5%. At least in Europe, Google is THE search engine.
But what is a search engine actually? A search engine is a program that usually searches for content in the browser. We search for a so-called keyword and the search engine presents us the results according to a certain algorithm, which is inherent to the search engine. The goal of the optimal search engine is of course to present us the most relevant result. However, since the search engine does not know exactly what we are looking for, because we communicate with it only via syntax, the search engine usually delivers the results in list form, where of course the first position shows the most relevant result (according to the search engine algorithm) and then in descending order further hits and information are presented. This list is also called SERP (Search Engine Result Page).
How does a search engine work?
Search engines filter the Internet according to a certain algorithm. Strictly speaking, a search engine does not search the entire Internet, but only the World Wide Web. Other parts of the Internet are, for example, e-mail, FTP or Usenet.
Strictly speaking, it is not the World Wide Web that is searched, but only an image of this WWW. This image is called the search engine index. Search engines deliver search results in a fraction of a second – if a search were performed live, i.e. the World Wide Web had to be searched the moment I entered the search term, it would certainly take hours and days before I received a result.
Therefore, each search engine creates a so-called index. This is, so to speak, a card index that has categorized and indexed the Internet. This index is already on the search engine provider’s server (cache) – and enables results to be delivered in seconds. Therefore, the index of a search engine is never in real time. So you see here the stored version of the World Wide Web from the past. In the best case, of course, this stored version is only a few hours in the past.
But how is this index created? So-called crawlers are used for this – at Google, these crawlers are called Googlebots. These bots are computer programs that independently surf the Internet and scan websites. In the process, content is captured and stored in a cache.
Since Google of course does not have the capacity to store all websites completely, there is the page title, which should contain information about the website or its contents, as well as the so-called metdescription – a 156-character short description of the website. The website owner can either specify this short description himself in the HTML code of his website and make it available to the Googlebot – or, if the bot decides that the description is bad or irrelevant, the Googlebot writes its own description by setting content from the website directly as the metadescription.
How does the bot move through the Internet?
How does the Googlebot decide on which page to crawl or which page to omit? The Internet is of course a huge maze, with many crossroads and forks. How does the bot decide which way to go? For this, Google uses links in particular. That means the bot crawls a page and of course sees the links. It uses them to shimmy from page to page. Therefore, websites that are not or only very little linked on the Internet are hardly or only very slowly crawled and added to the index.
This data is now stored in the Google index – or in the search engine index in general. When a search query reaches the search engine, the search term (the keyword) is compared with the index of the search engine and the relevance of the individual entries is compared with the keyword according to a certain algorithm. Since there is of course not only one entry for the respective keyword, the results are output in list form – the most relevant result according to the algorithm is displayed in first place in this list of results.
The search engine providers constantly update and adjust the algorithm for outputting the relevant results – therefore, the shift of a website in the search results may occur. This is beyond the control of the respective website operator.