How do search engines know what websites are out there?
In: 6
In the earliest days of the Web, you could tell people about your website through non-electronic means (word-of-mouth, paper newsletters), or older forms of computer-based communication (email, BBS’s, Usenet).
Some people quickly started making lists of useful websites. Heck, there were so few websites that some people made lists of *every website they knew about*.
Anyone who has a website can put links to other websites on it.
So if you want to make a search engine, you start with a list of websites — maybe a list you made, or maybe a list made by somebody else.
Then you write a program to go to each website, check it for links, and follow those links to more websites. Those websites also have links, and you can follow them to even more websites.
If there’s a new website, and somebody posts a link to that website on any other website, odds are that your program will eventually find it by following a chain of links from one of the sites in your original list.
Visiting massive numbers of websites by following links is referred to as “crawling,” short for “crawling the web”. A program that crawls the web is called a “crawler” or “spider.”
Yahoo! (one of the earliest search engines) started in 1994 as a list of websites called “Jerry and David’s Guide to the World Wide Web.” That list of websites continued to be updated at least through 2014.
Latest Answers