Robots.txt Files and What They Reveal

Posted by Fran Jeanes in Web Design Jul 21st, 2009 | No Comments

While doing a little research on robots text file tips I Googled “robots.txt disallow tips” and what was the first listing I got?

robots-txt-file.gif

The White House’s robots.txt file!!! And it has a Page Rank of 4. This discovery had my little SEO brain wondering why on earth that specific page would rank #1 for that term. Obviously the page must have a good number of incoming links and/or some good PR juice flowing to it.

So then I was interested in who might be linking to the White House’s robots.txt file – I mean, that’s not very interesting web reading. Or is it? (more…)

Post to Twitter Tweet This Post

Page 1 of 11