January 28, 2004

How can you keep a website out of a search engine?

It's an interesting problem. Usually, you want your site to be found, but there are cases where you'd rather not have your website show up in a search engine. There are many reasons for this: perhaps because google never forgets, or perhaps because what is on the website is truly private information: personal photos or business documents. There are several ways to prevent indexing of your site by a search engine. However, the only sure fire method is to password protect your site.

If you require some kind of username and password to access your website, it won't be indexed from by any search engine robots. Even if a search engine finds it, the robot doing the indexing won't be able to move past the login page, as they won't have a username and password. Use a .htaccess if you have no other method of authenticating, since even simple text authentication will stop search engine robots. Intranets and group weblogs will find this kind of block useful. However, if it's truly private information, make sure that you use SSL because .htaccess access control sends passwords in clear text. You'll be defended from search engines, but not from people snooping for interesting passwords.

What if you don't want people to be forced to remember a username and password? Suppose you want to share pictures of baby with Grandma and Grandpa, but don't want to either force them to remember anything, nor allow the entire world to see your child dressed in a pumpkin suit. In this case, it's helpful to understand how search engines work.

Most search engines start out with a given set of URLs, often submitted to them, and then follow all the links in a relentless search for more content (for more on this, see this excellent tutorial). Following the links means that submitters do not have to give the search engine each and every page of a site, as well as implying that any page linked to by a submitted site will eventually be indexed as well. Therefore, if you don't want your site to be searched, don't put the web sites URL any place it could be picked up. This includes archived email lists, Usenet news groups, and other websites. Make sure you make this requirement crystal clear to any other users who will be visiting this site, since all it takes is one person posting a link somewhere on the web, or submitting the URL to a search engine, for your site to be found and indexed. I'm not sure whether search engines look at domain names from whois and try to visit those addresses; I suspect not, simply because of the vast number of domains that are parked, along with the fact that robots have plenty of submitted and linked sites to visit and index.

It's conceivable that you'd have content that you didn't want searched, but you did want public. For example, if the information is changing rapidly: a forum or bulletin board, where the content rapidly gets out of date, or you're EBay. You still want people to come to the web site, but you don't want any deep links. (Such 'deep linking' has been an issue for a while, from 1999 to 2004.) Dynamic content (that is, content generated by a web server, usually from a relational database) is indexable when linked from elsewhere, so that's no protection.

There are, however, two ways to tell a search engine, "please, don't index these pages." Both of these are documented here. You can put this meta tag: <meta name="robots" content="none"> in the <head> section of your HTML document. This lets you exclude certain documents easily. You can also create a robots.txt file, which allows you to disallow indexing of documents on a directory level. It also is sophisticated enough to do user-agent matching, which means that you can have different rules for different search engines.

Both of these latter approaches depend on the robot being polite and following conventions, whereas the first two solutions guarantee that search engines won't find your site, and hence that strangers will have a more difficult time as well. Again, if you truly want your information private, password protect it and only allow logins over SSL.

Posted by moore at January 28, 2004 12:08 PM
Comments
© Moore Consulting, 2003-2006