Robots.txt Files and What They Reveal

While doing a little research on robots text file tips I Googled “robots.txt disallow tips” and what was the first listing I got?

robots-txt-file.gif

The White House’s robots.txt file!!! And it has a Page Rank of 4. This discovery had my little SEO brain wondering why on earth that specific page would rank #1 for that term. Obviously the page must have a good number of incoming links and/or some good PR juice flowing to it.

So then I was interested in who might be linking to the White House’s robots.txt file – I mean, that’s not very interesting web reading. Or is it?

Well the good old Beeb gave me some information that I liked to hear – that the new administration is open. And responsible for removing a whopping 2374 or so lines from the old file, and whittling the new robots.txt file to a trim 3 lines.

I wanted to see how many sites are linking to the WH’s famous, or infamous, file. So using “allinlink: whitehouse.gov/robots.txt” I learned Google had 9,390 indexed pages linking to the White House’s file.

Although older pages I read on the issue discussed the idiocy of laying out all your ‘secret’ folders for the world, and terrorist hackers, to read was news a few years ago, it certainly made me think about how careless we can be as website owners with divulging information – information you thought you didn’t want anyone [like your competition] to know about.

So you want to know what your rival is working on for their new product launch? Take a peek at their robots.txt file and if they hire web techs like the old White House did you might be privvy to some juicy, secret, information.

Said rivals, hurry – check your robots file now and make sure you aren’t giving away your secrets.

What we tend to forget is that just because you have a robots.txt file, it doesn’t mean the pages you list won’t be indexed.

Yes, that’s right.

That begs the question – what can you do to stop search engines indexing your top secret pages? Well the obvious thing to do is to use password protected directories, or simply don’t ever have a link to a page you want hidden. No linkie, no findie.

I won’t go into all the details here, but will let others, more learned in the old robots methodology explain it to you:
An aptly named info site
More on creating a robots.txt file
Managing robots access
Even more help

Now safely make your own robots.txt file to get over any virtual arachnophobia:
Google webmaster tools
Robots.txt generator

And remember – the spiders will be coming soon.

Post to Twitter Tweet This Post

SEO and the Basics of Onpage SEO

Search engines do not rank whole web sites, but individual pages within the site. When doing some Page Rank research you may see that pages within a web site can have very different rankings. Therefore, with single web pages in mind, start small to build up your rankings within as many pages of your site as possible. You need to use what is known as onpage SEO to give your web page the best advantage, and if you have many highly optimized pages they will help you more than just a homepage with a good ranking.

The most baic level onpage SEO techniques that you need to encompass within your web pages are as follows:

1. Page title
This is what appears in the blue bar at the very top of your browser window. This ‘title tag’ must list your keywords. If you leave your title out it will be listed by default as ‘Untitled’. This is to be avoided as your title tag lets the search engines know what your page is about. So, ‘untitled’ doesn’t tell them anything other than you omitted the title. In a search result the title tag is displayed so you want to keep it short and to the point using about 7 or 8 words. Make sure all the pages on your site have different title tags.

2. Metadata
This is coding found before your ‘head’ tag. Search engines used to use metadata for page rank much more than they do these days, but don’t overlook your meta tags – Google differentiates between your pages by using your metadata. List your keywords in your meta tags, do not list words that you don’t have in your content.

3. Web page content
Use a light peppering of keywords throughout your web page content but don’t overdo it and ’stuff’ your pages. The search engines will regard that as a negative. As long as your web copy is relevant to your title tags and keywords you are OK. Think big picture and ‘theme’ your web page. And try using synonyms as well as your keywords so that your text reads properly.

4. Links
Link to pages within your web site using your chosen keywords. This internal linking will inform search engines that the keyword has importance. Refrain from linking all over the place within your web copy as your theme will ‘bleed’. Keep things tight by controlling where your links go out to, and remember this means content links here, not your menu links.

5. Your domain name

If you want to improve in the search engine rankings it helps to have your keywords in your url. If your keyword is “weightloss tips” you should chose weightlosstips.com which will rank much higher for your keyword term than, say, healthyeating101.com. Of course you have to make sure each page is fully optimized too.

Remember as you optimize your web page content to make sure that it reads well. Naturally flowing text will make your visitors feel like sticking around, and later returning; and by using these few, easy to follow pointers, your onpage SEO will be a success.

Post to Twitter Tweet This Post

Keywords – Are You Choosing the Right Ones?

To get enough potential customers to your website you need to harness the power of search engines. Targeted customers are potential customers because they come to your site looking for something specific through the keywords that you have used, either in your web content or your pay per click campaign.

As you plan your keyword strategy remember that using the right keywords, ones that people actually use, is the name of the game. Your potential customers are those visitors who are interested in what you have to say as long as it is relevant to their search.

Back and the old days before people got really search engine savvy, a one word search term was common: insurance, mortgage, diet – you get the idea. Try getting the first place position in the Google with a one word phrase like those now – it’s a much steeper hill to climb, and an expensive one at that.

With most people these days using a three or four word search terms when using a search engine we can really see how specific and targeted searches are. So, for example, if you have a diet website you know there is a lot of competition out there – I just googled diet and there are 187,000,000 results – that’s a lot of competition.

How easy do you think it is to get a number one placement for the term ‘diet?’ Do you think you would have more success ranking number one for a slightly longer search term, such as ‘diet scams.’ You bet ya – ‘diet scams’ has only 4,480,000 listings. And what about ‘diet scams revealed,’ a three word search term that has only 65,300 listings! Do you think you could optimize your page to be number one out of just 65K or so? Sure you can!

Now that you can see the power of funneling down and using longer search terms as your keyword phrase it’s time to work out a strategy to evaluate your website’s keywords and phrases.

The first step is working out what your keywords and phrases are. This is actually sometimes more difficult than it sounds because most business owners think they know what those keywords and phrases are, but after a little research find out that they did not.

It’s very difficult to be objective when you are so close to your business or endeavor and put yourself in your customers’ shoes. What you would use for a search term is likely to be an industry term not one used by regular Joes on the street.

You will likely be surprised by what you find when you ask your customers what they see your business is about and how they understand it, as well as how they would go about searching for it on the Internet.

When you have a stack of keywords or phrases from your customers and you feel like the big picture is getting clearer for you then think about adding in your own keywords – but remember, if you are trying to gain new customers or clients, industry specific terms are not going to bring you conversions.

With your keyword list now you have to work out how relevant you think the terms are. If you are a web hosting company and your polled customers tell you ‘free web hosting’ is a search term they would use you should still think twice about whether it really is a search term for your business. Do you want to offer free web hosting? If not then do not entertain the thought – scrap that keyword phrase. By using it the only people you will bring to your site are those looking for free web hosting. Will they suddenly be looking for hosting they have to pay for when they land on your snazzy website? In a word: No.

You want to use keywords and phrases that people actually use to find things on the Net so choose popular words and terms. If no one searches for a term why use it, just in the unlikely event that someone, some day might?

How can you find out how popular terms are? There are lots of resources available that can give you this information if you want to find this out for yourself. A popular one is Wordtracker. [Full disclosure – that is my affiliate link to the site where you can sign up for a free trial.]

But popularity alone is not a certainty for bringing in web traffic. Your keywords and phrases need to be specific and relevant – remember ‘diet scams revealed?’ That term was not the most popular [that was diet] but if you have a website where you debunk diets then you are going to get traffic from that term.

And finally as you go through your list of keywords remember the why of it all – why is someone typing in those specific words in a search engine in the first place? Maybe if they typed in ‘diet scams revealed’ they have been thinking about a diet but want to know if it is really unhealthy, or dangerous in some way.

When you think you can locate the whys then you will have a fuller grasp of your visitors’ motivations and you can expand upon them. You may be able to offer more products or services once you have successfully pinned down specific customer motivations, which is a nice bonus.

Post to Twitter Tweet This Post

Page 1 of 11