Search engine friendly URLs

[homoludens]

Please forgive the confused newb musings I'm about to spout.

Google states that it doesn't like indexing dynamic pages, but I don't really want to develop my website in such a way that I need a static file published every time I update the content via the DB.

I imagine that google prefers indexing www.example.com/products/stetsons/ to: www.example.com/shop?product=stetsons and that if I can use the webserver to rewrite a request for www.example.com/products/stetsons/ to www.example.com/shop?product=stetsons I can please google and keep my website maintenance requirements to a minimum.

I imagine this, but then I know near to nothing about HTTP headers , the Apache webserver and webness in general. Am I right, and if so, what can do to perform the rewrite.

Thanks for any help you can give me, and apologies if this has already been covered.

D

[jason]

You are correct. Google will *sometimes* index pages that have query sting data (the stuff after the ?) in them, but it is somewhat restrictive on what it will index this way (and sorry, but I can't give you any details--they're pretty secretive about their methods). Some SE's won't index any page that has a query sting.

The most common way to get around this restriction is to do URL rewriting. On Apache servers this is most commonly done with mod_rewrite (which is available here).

You can read up on mod_rewite here and here (warning: they're both pretty technical). It might als help to do a search on regular expressions because you use them to map your rewrites.

Rewrite rules are put in a file called .htaccess that "lives" in your public_html directory. It is also possible to put .htaccess files in sub-directories of .htaccess, but it is most common to place your rewrite rules in the top-most .htaccess file. There is an .htaccess editor available in CPanel, or you can use your favorite text editor.

The basic format of the rewrite that you described would be something like this:

RewriteEngine on

RewriteRule ^products/(.*)/?$ http://www.example.com/shop?products=$1 [QSA,L]

(The above isn't tested, but should be at least close to working. ) Basically it is telling the server that, whenever it sees a URL that starts with "products/," it should take whatever appears after the "/" and stick it into a query string variable called products. The QSA in brackets tells the server taht we are modifying the query string (so it will be sure to process it) and the L tells the server that we've done everything we want to do. Without the L, if there were more RewriteRules after this one, those would also be processed and might do things to our new URL that we don't want.

I hope this helps you out.

--Jason

[homoludens]

Thanks for your reply Jason, and thanks for pointing me in the right direction.

It's not currently working (I'm getting a 404 error) but at least I knowin what direction to point my brain cells.

Thanks again,

D

[jason]

I just tried my example, rewriting output to a phpinfo() script (that dumps tons of debugging information so that I could see what was going on) and it worked just fine. Are you sure that the URL you are rewriting to is correct? Should it use something like "shop.php" or "shop.cgi" instead of just "shop?" Try going to eee.example.com/shop and see if you get a 404 error. If you do then you'll need to track that down before you worry about the rewrites.

--Jason

[homoludens]

Perhaps my question should have been closer to what I am actually doing.

I'm trying to rewrite

zzz.example.com/clients/clientarea/clientsubpage/

to

zzz.example.com/clients/clientarea/page.php?=clientsubpage

There was a typo in my example above (sorry); shop should indeed have been shop.php, although I correctly put page.php in my htaccess. I don't know if the extra subdir changes things any, although I don't see why it should.

What you wrote certainly seems to tally with the mod_rewrite page and the little I know of regular expressions. I'm off to gorge myself on pizza and I'll have another crack at it after that.

Thanks,

D

[Connie]

Here is what Google says http://www.google.com/webmasters/guidelines.html

Under the Technical Guidelines:

Allow search bots to crawl your sites without session IDs or arguments that track their path through the site. These techniques are useful for tracking individual user behavior, but the access pattern of bots is entirely different. Using these techniques may result in incomplete indexing of your site, as bots may not be able to eliminate URLs that look different but actually point to the same page.

Don't use "&id=" as a parameter in your URLs, as we don't include these pages in our index.


The SE friendly and user friendly way to do URLs is the way Jason mentioned.

[homoludens]

Jason,

It's working perfectly now. Who knows why it wasn't before. Thanks very much for the help.

D

[Vin DSL]

Heh! I love these 'search engine friendly' threads!

Check this out:

http://www.google.com/search?as_q=od...ghts=&safe=off

Goggle has indexed a half-million of the worst 'search engine unfriendly' dynamic links I've ever seen on my production site.

How can this be? No session IDs -- that's all!

Bottom line: Google doesn't like session IDs! Stay away from them and you'll be okay...

[homoludens]

I suspect you loaded the experiment there. Now do a search for:

http://www.google.com/search?q=site:...r=all&filter=0

The first result with a query string is waaaaaay down the list. So google may well index them, it just doesn't like them much. That's what I was thinking of. I'm not SEO-obssessed but why cripple yourself from the outset?

[Vin DSL]

I suspect you loaded the experiment there...

Fair enough -- I just woke up.

Let's try it this way -- results strictly from my (SEO unfriendly) ODP module:

http://www.google.com/search?q=site:...all&filter =0

I'm seeing 435,000...

[homoludens]

OK so on your new search the first result _without_ a query string is at 30. But I still think you're loading the experiment, as the most of the pages returned either have "name=ODP" in the URL or in an href somewhere. I reckon a page on your site with name=ODP in the content somewhere would quickly float to the top of such a search.

And I still don't think it proves your point, which is (I think) that querystrings (ignoring sessions IDs) don't matter a jot.

The first result for your search is for

http://www.lenon.com/modules.php?nam...age/Retailers/

but a search for lenon, computers, hardware, storage, retailers

http://www.google.com/search?q=lenon...&sa=N&filter=0

returns your page for computer storage retailers on page 2. It returns 4 of your pages right at the top (google clearly likes your site in general), but not the one I would have been looking for as a search engine user. To my mind, that's because those pages have computers, retailers etc in the content, and google is down-ranking the right page because of the query string issue.

Personally I want to agree with you that SE-friendly URLs is a load of bunk (and I did wholeheartedly some months ago), as I don't think we should have to design sites according to google's diktats. They should improve their algorithms. But the issue started nagging at me, and so I thought I'd ask for help on the matter here.

And I think clssam is right to point out that the URLs are more user-friendly too.

As you wake up on your side of the pond, so I go to sleep on mine ... ;o)

[Connie]

The site which earns my living (and 4 other people) is a static html EC site. This is not a hobby or part time thing for me.

I do everything for my visitors. Fortunately that is what the SEs want you to do. The bottom line is to get the click and convert the visitor.

If your page comes up in a search I think a visitor is more likely to click on
page/red.widget. than they are page/product?101.

The Guidelines or suggestions of Yahoo, and MSN are similar to Google's.

[homoludens]

If your page comes up in a search I think a visitor is more likely to click on
page/red.widget. than they are page/product?101.

I quite agree. People tend to scan webpages alot, and product?101 isn't going to catch anyone's eye. Whilst I was skeptical about the SE-friendly issue I'd like to say thanks for drawing my attention to the user-friendly issue.

The rest of your post seems to be suggesting that my comment "I don't think we should have to design sites according to google's diktats. They should improve their algorithms" is foolish, and if so I'm inclined to agree with you. I do, however, fail to see how a indexer can cope with parsing an entire document, but not a query string.

Probably because I know nothing about Information Retrieval. ;o)

Definitely going to bed now.

[Vin DSL]

As an aside, the fact of the matter is, nobody knows how Google works. And, when someone starts to figure them out, Google will change things to mess you up. This is done on purpose, to keep ppl from playing the system.

It's not really 'good business' to try and play Google. If 'they' catch you with your hand in the 'cookie jar', they'll knock you down to '0' in a heartbeat...

[gerilya]

The site which earns my living (and 4 other people) is a static html EC site. This is not a hobby or part time thing for me.

I am not sure what you are trying to say by that. There are many dynamic e-commerce sites employing more than 5 people

I do everything for my visitors. Fortunately that is what the SEs want you to do.

What Search Engines really want is that you provide a relevant content for a search term. Nothing more, nothing less. They wouldn't care about dynamic links, text in a title tag or anything else if they could use a crystal ball

The bottom line is to get the click and convert the visitor. If your page comes up in a search I think a visitor is more likely to click on
page/red.widget. than they are page/product?101.

As a general rule, I think that visitors are more likely to give a heavier weight to a page title and description than to the URL, but that doesn't mean URLs shouldn't be friendly as well.