As a web developer, just want to add my two cents. Not sure if Google still sells them, but in the mid 2000’s, we bought a Google Mini for our site. Every night, it would spider the entire site, in the morning it would send out a report on any broken links, missing images, etc, this was helpful for SEO, but it also indexed the whole site, when someone used the search box at the top of the page, it was using the google mini interface and found things very accurately, and very fast, now just scale that up to the web.
Secondly, about ten years ago I was developing an internal site for a large corporation, they wanted a, drop down display, results as you type, kind of thing. After every third letter of typing, it would “hit the server” to see if anything matched in cache, if so, send down the recordset and display in the dropdown, if not, hit the database with a %like% query, save the query in cache, send it down to the client browser, the cached recordsets were set to expire in an hour after the last use. It was fast for the clients but very taxing on the servers, it also got better as the day progressed, as the cache would be empty in the morning,
Having Windows spider your hard drive every night would not be too bad, but if you left your computer off during the night, would you want Windows to do it during the day while you are working? How much hard drive or RAM space are you willing to give up just to have a faster search that you don’t use that often?
Latest Answers