If you want more control over how Yahoo spiders your site, then you need to read this article on wild cards.

If you are interested in keeping up with what is happening with Search Engines there are a few blogs that every webmaster should read.

Googles WebMaster Central

Matt Cutts Blog

Yahoos Blog

MSN Blog

If you want to keep up with general information about Google, and whats new then I would add the Google Blog

You will get a lot of helpful information that can help you with the SEs, if you read these blogs.

The term “search engine” is often used generically to describe both crawler-based search engines and human-powered directories. These two types of search engines gather their listings in radically different ways.

Crawler-Based Search Engines

Crawler-based search engines, such as Google, create their listings automatically. They “crawl” or “spider” the web, then people search through what they have found.

If you change your web pages, crawler-based search engines eventually find these changes, and that can affect how you are listed. Page titles, body copy and other elements all play a role.

Human-Powered Directories

A human-powered directory, such as the Open Directory, depends on humans for its listings. You submit a short description to the directory for your entire site, or editors write one for sites they review. A search looks for matches only in the descriptions submitted.

Changing your web pages has no effect on your listing. Things that are useful for improving a listing with a search engine have nothing to do with improving a listing in a directory. The only exception is that a good site, with good content, might be more likely to get reviewed for free than a poor site.

“Hybrid Search Engines” Or Mixed Results

In the web’s early days, it used to be that a search engine either presented crawler-based results or human-powered listings. Today, it extremely common for both types of results to be presented. Usually, a hybrid search engine will favor one type of listings over another. For example, MSN Search is more likely to present human-powered listings from LookSmart. However, it does also present crawler-based results (as provided by Inktomi), especially for more obscure queries.

The Parts Of A Crawler-Based Search Engine

Crawler-based search engines have three major elements. First is the spider, also called the crawler. The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being “spidered” or “crawled.” The spider returns to the site on a regular basis, such as every month or two, to look for changes.

Everything the spider finds goes into the second part of the search engine, the index. The index, sometimes called the catalog, is like a giant book containing a copy of every web page that the spider finds. If a web page changes, then this book is updated with new information.

Sometimes it can take a while for new pages or changes that the spider finds to be added to the index. Thus, a web page may have been “spidered” but not yet “indexed.” Until it is indexed — added to the index — it is not available to those searching with the search engine.

Search engine software is the third part of a search engine. This is the program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant

Search for anything using your favorite crawler-based search engine. Nearly instantly, the search engine will sort through the millions of pages it knows about and present you with ones that match your topic. The matches will even be ranked, so that the most relevant ones come first.

Of course, the search engines don’t always get it right. Non-relevant pages make it through, and sometimes it may take a little more digging to find what you are looking for. But, by and large, search engines do an amazing job.

As WebCrawler founder Brian Pinkerton puts it, “Imagine walking up to a librarian and saying, ‘travel.’ They’re going to look at you with a blank face.”

OK — a librarian’s not really going to stare at you with a vacant expression. Instead, they’re going to ask you questions to better understand what you are looking for.

Unfortunately, search engines don’t have the ability to ask a few questions to focus your search, as a librarian can. They also can’t rely on judgment and past experience to rank web pages, in the way humans can.

So, how do crawler-based search engines go about determining relevancy, when confronted with hundreds of millions of web pages to sort through? They follow a set of rules, known as an algorithm. Exactly how a particular search engine’s algorithm works is a closely-kept trade secret. However, all major search engines follow the general rules below.

What do the key words represent?

Key words – these are the words that the users enter in a given search engine to find services or products you offer. It is extremely important to know what key words they will use. Selecting the appropriate for your activity keyword is the first thing you have to do because all other strategies are built on their right choice.

How to select the appropriate for you key words?

Key words that will work best for you means to describe exactly what you offer and meanwhile to be well used by users. If your website is ranked number one in the search engines results, after using key words that nobody uses, your work will be useless.

1. Don’t count just on you when selecting the appropriate for your business key words. The users might have a completely different view about what to insert in the search engines to find you.




2. The best choice will be if the selected word isn’t so competitive and has a very good popularity.




3. To understand the way Wordtracker functions, use it’s free version, although its paid version offers much more functions.




4. That’s why always add to a key word, corresponding to business like yours, some additional, specific only for your business words to define and to specify the process.

You can get good keywords by using tools such as Good keywords, you can download it from:

http://www.goodkeywords.com/products/gkw/

There are many techniques one can follow for SEO but then which are the right ones, you should get the one set in your mind wrong techniques will cost you very dear. Avoid the following ways: 1.Doorway page- Creating a doorway page can jeopardize your website’s status as doorway pages are prohibited by search engines. Doorway pages are fraudulent method to manipulate search engine results but are never viewed by users. 2.Invisible Text- Placing invisible text i.e. white text on white background to gain high ranking is unethical & can get you banned from search engines. 3.SEO Ploy- SEO is done for high search engine ranking but using them in wrong methods can cost you removal from search engine list & also ruin your reputation for utilizing such practices. 4.Link Farm- Link farms are created for primary purpose of exploiting search engine ranking, but these farms do not provide any useful information but inversely spoil your web sites ranking. Search engines have very strict rules against these link farms.

We all need Maps for navigation similarly the bots which crawl your website also need a navigator which tells them the where abouts of the infomation stored on your website

Google Sitemaps Beta is a service by Google designed for website owners which enables spiders to proactively and automatically crawl information about your website to Google. This sitemap can help provide Google with more information about your website than usually crawled by the Google.If your site has dynamic content or pages that aren’t easily discovered by following links, you can use a Sitemap file to provide information about the pages on your site. This helps the spiders know what URLs are available on your site and about how often they change.

A Sitemap provides an additional view into your site. This program does not replace the normal methods adopted by Google of crawling the web. Google still searches and indexes your sites the same way it has done in the past whether or not you use this program. A Sitemap simply gives Google additional information that the search engines may not otherwise discover.

You can create Google Sitemap within a span of few minutes using the tool Google Sitemap Creator.

Google takes many facts into consideration before assigning page rank to your website. Add links to your website at regular intervals rather than adding them all at once.

The search engine observes the invention date of links, the life span of the link and the rate at which a new web site obtains links. This approach shows that Google is discounting fast link exchange strategies such as buying many links for your site. Instead, Google appears to consider a natural linking evolution as an indication that a site is authorized.

The specifics of a good linking strategy are a little difficult to nail down. Factors that might be considered include:

1 The anchor text of the link.

2 The invention date of the link.

3 The growth rate of links to your site.

4 The rate at which links to a page appear and disappear.

5 The age of links with older links gets more value.

6 Many links to a new site will be looked at as spam, unless some of the links are from highly valued sites.

7 Link growths that are constant are optimal.

8 Unforeseen bunches of new links will be degraded as spam.

Cheers.

IMPORTANCE OF META-TAGS Meta Tags are the one which intoduce your website to the serach engine , it should contain the all major information about your website. There are 3 types of meta-tags – * Title – This is the text that appears in the title-bar of your web browser. * Description – This is a quick summary of the information on the page. * Keywords – These are words that are important in the page In the recent years, there was a tremendous focus on meta-tags as a method of improving search engine rankings. . Webmasters could stuff all sorts of words into the meta-tags to improve their search engine rankings for the words they wanted to rank well for. (black days for Meta Tags) It wasn’t long before the search engines caught on to this little trick, and as a result the efficacy of “keyword-stuffed” meta-tags dwindled over time. Posts on Search Engine Optimization (SEO) boards heralded the demise of the meta-tag with posters claiming that Google all but ignored the meta-tags. It seemed that there was no use for meta-tags anymore – at least from the perspective of the online marketer. (golden days are here again) But then, there was a revolt against these BLACK-HAT SEO techniques. The search market started to fracture. Yahoo and MSN both spun up their own search engines and dropped Google’s results from their engines. Both Yahoo and MSN’s new engines seemed to use content from the Title and Description meta-tags to display in the SERPs (Search Engine Results Pages). . Many of the results on the SERPs of both search engines will use the content from the meta-tags heavily. Google is less predictable with regard to the use of meta-tags. Google often uses the title tag to determine the clickable link on its results pages, but only occasionally uses the description tag content in the page summary that it displays. Steps to improve meta-tags – As a general rule, Yahoo uses the first 25-30 words of your Meta Description tag in the site description it displays on your SERPs; MSN uses the first 15 or so. Write out a 30-word description of each page of your website that is broken up into two parts. The first 15 words need to get across what the page is about – this is all the MSN searchers will see. The second 15 words should support the first – this will be visible to Yahoo searchers. Google searchers will sometimes see the first 7 – 15 words as well (although you can’t count on it). With meta-tags you have to just focus on the things you can control, and right now Google is an unknown quantity so it’s probably best not to worry about optimizing for Google.

Search Engine Optimization.

For beginers Search engine Optimization is the best way to get your website listed in major search engines such as Google, Yahoo, MSN, what it means is to get you website listed on the first page of the search engine for a particular keyword. The emphasis is layed on the first page is because any surfer wont bother to go to second page he will only click on the links which show up on the first page as they are all in a hurry no one has time to go through all the pages.

(SEO) Search Engine Optimization uses a combination of techniques, tools, and technical know-how to get results. SEO is a set of methodologies aimed at improving the visibility of a Web Site in search engine listing. SEO involves the careful designing of corporate Web Site to increase their chances of visibility in the major search engines for e.g. Google, Yahoo, Alta-Vista etc. The higher a ranks in the results of a search, the greater the chance that that site will be visited by a user.

• Meta Tags

Meta Tags are very important for your Web Site’s visibility. Many search engines read them from your site when you submit it to them .Major Meta tags that you can use:

Meta Title the only resource type that is currently in use is “documenting” This is the only tag that you need to put in for indexing purposes.

Meta Description Depending on the search engine, this will be displayed along with the title of your page in an index. “Content” could be a word, sentence or even paragraph to describe your page. Keep this reasonably short, concise and to the point.

Meta Keywords Choose whatever keywords you think are appropriate, separated by commas.

Navigational links:

At the right or at the bottom but not on the left of the page. When the search engines “read” your site, they read from the top left to the bottom right. Search engines place an emphasis on the first 100 words or text on the site. You do not want these words to be navigational links or JavaScript. Ideally, you want to have your heading tags with your keywords in the beginning of your page. This being said, placing your links/JavaScript on the right or bottom of your page ensures the search engine spiders get to the text first, giving more weight to what’s important on your page.

Rich Content:

Keep your site very content rich. The more content you have on your site the more likely someone is to find what they’re looking. Have a lot of pages. Update the site as many times as possible in a day if you have the time. Updating your site does not mean that you copy paste content from other sites. That will only push your rankings further down. Don’t have any duplicate content. At the same time, dont spam multiple content across the same website. A search engine constantly views or reviews a website or at least every few months, it looks for specific characteristics found in the content of website pages.

Matt Cutts posted a summary yesterday  of things that have been taking place at Google over the last few months.

He provides a lot of links to previous post of interest.  If you read his post and the comments that follow it should give you some kind of idea about what is going on with Google.

One thing for sure is that recipicol links are a thing of the past if they are not related to your site.   Matts Update.