Passing header to play nice with search bots

Thursday, May 29, 2008, 2:41 AM
Website Design by John

One of the lowlights of using a dynamically generated URL system on your website with htaccess is that your site can produce URLs that look OK, but yield pages with little or no content. These sorta orphaned URLs also dilute some of your search engine mojo, by causing funny little extra pages to show up in search.

In order to play nice with the search engine bots, it is wise to scan the output of of your URL and determine whether it is delivering real content or if it is just answering one more query.

Advertisements

In my system, for example, most of the pages I want to index well in search have URLs that combine the ID number of the database entry for that page, along with a constructed, scrubbed text version of the title.

Now, here's why this matters: the search bots, especially GoogleBot, pick and stick.

For example, in the classifieds section of my one website, there are currently four pages of classified ads. But, in the server logs, sure enough Google picks and sticks a 5.html request in after making a request for 4.html.

The thing is, this works against you if 5.html returns anything.

So, if we detect that 5.html doesn't have any real content, it is wise to tell the bots as much.

We do so like so:

header("HTTP/1.0 404 Not Found");

Make sure you send the header before echoing or printing any output. Now, Google notes that there isn't a page there and will not request in the future. More importantly, it will not include whatever output your CMS was belching out.

Handy, huh? Especially since a good website should return the best content possible to the search engine bots, right?

Mail article to a friend

Tools

Check Google PageRank

Recent articles

Welcome!

Wonder where to start with your web design business?

This blog follows along with my efforts to build and grow a website design business, Pro Content and Design.

The goal of this blog is to fill in blanks that may be empty as you get your business rolling.

This blog, particularly the source code section, is not intended for beginners. If you are not comfortable with databases, Ajax, DOM objects and other advanced methods, I strongly suggest you go take a look over at W3 Schools before even reading -- let alone tinkering with -- any of the code here.

I hope this blog has some value to web designers as they attempt to get their businesses going.

Good luck, and happy reading.

Thank you,
John Crawford
Pro Content and Design