INTERNET WATCH

Age: White House site prevents Iraq material being archived

The White House website effectively prevents search engines indexing and archiving material on the site related to Iraq.

The directories on a site which can be searched by the bots sent out by search engines can be limited by means of a file called robots.txt, which resides in the root directory of a site.

Adding a directory to robots.txt ensures that nothing in that folder will ever show up in a search and will never be archived by search sites.

The White House's robots.txt file lists a huge number of directories all related to Iraq.