| Search engines| How Search Engines Rank Web Pages|Search Engine Design Tips |

Search Engine Design Tips

A search engine query often turns up hundreds or thousands of matching web pages. In most cases, only the 10 most "relevant" matches are displayed first.

Naturally, anyone who runs a web site wants to be in the "top ten" results. This is because most users will find a result they like in the top ten. Being listed 11 or beyond means that many people may miss your web site.

The tips below will help you come closer to this goal, both for the keywords you think are important and for phrases you may not even be anticipating.

Pick Your Strategic Keywords :How do you think people will search for your web page? The words you imagine them typing into the search box are your strategic keywords.

For example, say you have a page devoted to stamp collecting. Anytime someone types "stamp collecting," you want your page to be in the top ten results. Then those are your strategic keywords for that page.

Each page in your web site will have different strategic keywords that reflect the page's content. For example, say you have another page about the history of stamps. Then "stamp history" might be your keywords for that page.

Your strategic keywords should always be at least two or more words long. Usually, too many sites will be relevant for a single word, such as "stamps." This "competition" means your odds of success are lower. Don't waste your time fighting the odds. Pick phrases of two or more words, and you'll have a better shot at success.

Position Your Keywords:Make sure your strategic keywords appear in the crucial locations on your web pages. The page title is most important. Failure to put strategic keywords in the page title is the main reason why perfectly relevant web pages may be poorly ranked.

Keep in mind that tables can "push" your text further down the page, making keywords less relevant because they appear lower on the page. This is because tables break apart when search engines read them. For example, picture a typical two-column page, where the first column has navigational links, while the second column has the keyword loaded text. Humans see that page like this:

Home      Stamp Collecting
Page 1
Page 2    Stamp collection is worldwide experience.
Page 3    Thousands enjoy it everyday, and millions
Page 4    can be made from this hobby/business.

Search engines (and those with old browsers) see the page like this:

Home
Page 1
Page 2 
Page 3 
Page 4

Stamp Collecting

Stamp collection is worldwide experience.
Thousands enjoy it everyday, and millions
can be made from this hobby/business.

See how the keywords have moved down the page? There's no easy way around this, except to use meta tags. That helps for the search engines that use them. For the others, it may not be that big a problem. Consider how tables might affect your page, but don't necessarily stop using them. I like tables, and I'll continue to use them.

Large sections of JavaScript can also have the same affect as tables. The search engine reads this information first, which causes the normal HTML text to appear lower on the page. Place your script further down on the page, if possible. As with tables, the use of meta tags can also help.

Have Relevant Content :Changing your page titles and adding meta tags is not necessarily going to help your page do well for your strategic keywords if the page has nothing to do with the topic. Your keywords need to be reflected in the page's content.

In particular, that means you need HTML text on your page. Sometimes sites present large sections of copy via graphics. It looks pretty, but search engines can't read those graphics. That means they miss out on text that might make your site more relevant. Some of the search engines will index ALT text and comment information, along with meta tags. But to be safe, use HTML text whenever possible. Some of your human visitors will appreciate it, also.

Be sure that your HTML text is "visible." Some designers try to spam search engines by repeating keywords in a tiny font or in the same color at the background color to make the text invisible to browsers. Search engines are catching on to these and other tricks. Expect that if the text is not visible in a browser, then it won't be indexed by a search engine.

Finally, consider "expanding" your text references, where appropriate. For example, a stamp collecting page might have references to "collectors" and "collecting." Expanding these references to "stamp collectors" and "stamp collecting" reinforces your strategic keywords in a legitimate and natural manner. Your page really is about stamp collecting, but edits may have reduced its relevancy unintentionally.

Avoid Search Engine Stumbling Blocks

Some search engines see the web the way someone using a very old browser might. They may not read image maps. They may not read frames. You need to anticipate these problems, or a search engine may not index any or all your web pages.

Have HTML links: Often, designers create only image map links from the home page to inside pages. A search engine that can't follow these links won't be able to get "inside" the site. Unfortunately, the most descriptive, relevant pages are often inside pages rather than the home page.

Solve this problem by adding some HTML hyperlinks to the home page, something that will help some of your human visitors, also. Put them down at the bottom of the page. The search engine will find them and follow them.

Also consider making a site map page with text links to everything in your web site. You can submit this page, which will help the search engines locate pages within your web site.

Frames can kill: Some of the major search engines cannot follow frame links. Make sure there is an alternative method for them to enter and index your site, either through meta tags or smart design. For more information, see the tips on using frames.

Dynamic Doorblock: Generating pages via CGI or database-delivery? Expect that some of the search engines won't be able to index them. Consider creating static pages whenever possible, perhaps using the database to update the pages, not to generate them on the fly. Also, avoid symbols in your URLs, especially the ? symbol. Search engines tend to choke on it.

Use Meta Tags :As mentioned above, meta tags can help you overcome problems with tables, frames and other trouble areas. Meta tags will also help you control your site's description in engines that support them. You should use meta tags, but keep in mind that they are NOT a guarantee that your site will appears first. Adding some meta tag code is not a magic bullet that cures your site of dismal rankings. For more information, see the tips on using meta tags. Site subscribers have access to extended information about meta tags.

Just Say No To Search Engine Spamming :For one thing, spamming doesn't always work with search engines. It can also backfire. Search engines may detect your spamming attempt and penalize or ban your page from their listings.

Also, search engine spamming attempts usually center around being top ranked for extremely popular keywords. You can try and fight that battle against other sites, but then be prepared to spend a lot of time each week, if not each day, defending your ranking. That effort usually would be better spent on networking and alternative forms of publicity, described below.

If those practical reasons aren't enough, how about some ethical ones? The content of most web pages ought to be enough for search engines to determine relevancy without webmasters having to resort to repeating keywords for no reason other than to try and "beat" other web pages. The stakes will simply keep rising, and users will also begin to hate sites that undertake these measures.

Consider search engine spamming against spam mail. No one likes spam mail, and sites that use spam mail services often face a backlash from those on the receiving end. Sites that spam search engines degrade the value of search engine listings. As the problem grows, these sites may face the same backlash that spam mail generates.

Submit Your Key Pages: Most search engines will index the other pages from your web site by following links from a page you submit to them. But sometimes they miss, so it's good to submit the top two or three pages that best summarize your web site.

Don't trust the submission process to automated programs and services. Some of them are excellent, but the major search engines are too important. There aren't that many, so submit manually, so that you can see if there are any problems reported.

Also, don't bother submitting more than the top two or three pages. It doesn't speed up the process. Submitting alternative pages is only insurance. In case the search engine has trouble reaching one of the pages, you've covered yourself by giving it another page from which to begin its crawl of your site.

It can take up to a month to two months for your "non-submitted" pages to appear in a search engine, and some search engines may not list every page from your site.

Verify And Maintain Your Listing :Once your pages are listed in a search engine, monitor your listing every week or two. Strange things happen. Pages disappear from catalogs. Links go screwy. Watch for trouble, and resubmit if you spot it.

Keep in mind that a number of the major search engines are now providing country-specific versions of their directories. These mainly work filtering sites by domain. For example, a British edition of a major search engine might only list web sites with British domains, such as ".co.uk." A British site ending in a non-British domain, such as .com, would be filtered out. If this type of situation applies to your site, you may need to message the search engine so that they can manually include your site.

Resubmit your site any time you make significant changes. Search engines should revisit on a regular schedule. However, some search engines have grown smart enough to realize some sites only change content once or twice a year, so they may visit less often. Resubmitting after major changes will help ensure that your site's content is kept current.

Beyond Search Engines :It's worth taking the time to make your site more search engine friendly, because some simple changes may pay off with big results. Even if you don't come up in the top ten for your strategic keywords, you may find an improvement for strategic keywords you aren't anticipating. The addition of just one extra word can suddenly make a site appear more relevant, and it can be impossible to guess what that word will be.

You should also consider negotiating reciprocal links with sites that do appear in the top ten lists, if you are having no luck. Perhaps some of these sites might be considered "competitors," but you'd be surprised how many are happy to link to your site in return for a link back. After all, your site may appear first when slightly different keywords are used. Links are what the web was built on, and they remain one of the best ways for people to find your site.

Also, remember that while search engines are a primary way people look for web sites, but they are not the only way. People also find sites through word-of-mouth, traditional advertising, the traditional media, newsgroup postings, web directories and links from other sites. Many times, these alternative forms are far more effective draws than are search engines.

Finally, know when it's time to call it quits. A few changes may be enough to make you tops in one or two search engines. But that's not enough for some people, and they will invest days creating special pages and changing their sites to try and do better. This time could usually be put to better use pursuing non-search engine publicity methods.

Don't obsess over your ranking. Even if you follow every tip and find no improvement, you still have gained something. You will know that search engines are not the way you'll be attracting traffic. You can concentrate your efforts in more productive areas, rather than wasting your valuable time.

Search Engine Features For Webmasters :

The search engine features chart below is designed primarily for webmasters who care about how search engines index their sites. It provides a summary of important factors and features that can affect how sites are indexed and ranked. Full explanations of items can be found immediately below the comparison chart.

This chart is as of Oct. 1, 1999. It covers AltaVista, Excite, Go (Infoseek), Google,  Inktomi, Lycos and Northern Light. Unless specifically named, Inktomi covers AOL Search and HotBot, while AltaVista covers the MSN Search.

Crawling: This section covers factors related to how well search engines crawl web sites.

Deep Crawl :The search engines shown doing deep crawls will list many pages from a web site, even if the pages are not explicitly submitted to them. The others will usually list far fewer pages from a site. In general, the larger a search engine's index is, the more likely it will many pages per site.

Instant Indexing :At an instant indexing search engine, usually any page you submit will appear within a day or two after submission.

Frames Support: This shows which search engines can follow frame links. Those that can't will probably miss listing much of your site.

Image Maps:This shows which search engines can follow client-side image maps. As with frames, those search engines that can't follow image maps will probably missing listing much of your site.

robots.txt : The robots.txt file is a means for webmasters to keep search engines out of their sites. Site subscribers have access to a page that explains this in more detail. More information about robots.txt can also be found on the Robots Exclusion Standard page, located at:

http://info.webcrawler.com/mak/projects/robots/exclusion.html

Meta Robots Tag :This is a special meta tag that allows site owners to specify that a page shouldn't be indexed. It is ideal for those who cannot create a robots.txt file.

To keep spiders out, simply add this text between your header tags on each page you don't want indexed:

<META NAME="ROBOTS" CONTENT="NOINDEX">

You do not need to use variations of this tag to help your page get indexed. They are unnecessary. Nor do you need to use this tag if you already use a robots.txt file.

Link Popularity Helps Deep Crawl: All search engines can determine the popularity of a page by analyzing how many links there are to it from other pages. Some engines use this as a means to determine which pages they will include in the index. This is NOT the same as ranking a page better for having good link popularity. That is explained further below.

Learns Frequency:A number of search engines can learn how often pages change. Pages that change often may be visited more frequently.

URL Status Check :There are various ways to find whether an exact page is listed at the different search engines. 

Indexing:This section explains what gets indexed when search engines spider a page.

Full Body Text: All of the major search engines say they index the full visible body text of a page, though some will not index stop words or exclude copy deemed to be spam (explained further below).

Stop Words:Some search engines either leave out words when they index a page or may not search for these words during a query. These "stop words" are excluded as a way to save storage space or to speed searches.

Meta Description & Meta Keywords:Shows which search engines support the meta description and meta keywords tags. This does NOT mean that using these tags gives pages a ranking boost. That is covered in a separate section, below.

ALT Text / Comments:This shows which search engines index ALT text associated with images or text in comment tags.

Stemming:Some search engines will search for variations of a word based on its stem. For example, entering "swim" might also find "swims" and maybe "swimming," depending on the search engine.

Ranking:Most search engines use the location and frequency of keywords on a web page as the basis of ranking it in response to a query. The exact mechanism is slightly different for each engine.

In addition to location and frequency, some engines may give a page a relevancy boost based on other factors. These usually can help a little, but they don't guarantee a boost to the top. Some major factors are listed below.

Meta Tags Boost Ranking: Some search engines that support the meta description and keywords tag will also give pages an extra boost if search terms appear in these areas. Not all search engines that support the tags also give a ranking boost.

Reviewed Status Boosts Rankings : Some search engines also review sites or list them in an associated directory. They may also give a boost to sites that have been listed in this way.

Link Popularity Boosts Rankings: As described above, all search engines can determine the popularity of a page by analyzing how many links there are to it from other pages. Some engines give pages with lots of links, or links from important web sites, a relevancy boost.

Direct Hit Boosts Rankings:Direct Hit is a system that measures what users click on from search results in order to refine relevancy rankings. This shows which search engines use this as a factor.

Spam : All major search engines penalize sites that attempt to "spam" the engines in order to improve their position. One common technique is "stacking" or "stuffing" words on a page. This is where a word is repeated many times in a row. If the search engines spot a spamming technique, they may downgrade a page's ranking or exclude it from listings altogether. The items below cover design elements that could cause a spam penalty.

Meta Refresh : Some site owners create target pages that automatically take visitors to different pages within a web site.

The meta refresh tag is one typical way of doing this. Some search engines will refuse to index a page with a high meta refresh rate. Go will not index pages with any redirection, whatsoever.

Google doesn't worry much about meta refresh tags or the items below because its link popularity ranking system pretty much defeats spam attempts.

Invisible Text: This is the technique of placing text on a page in the same color as the background, making it invisible to human viewers. Many search engines either refuse to index this text or will not index any page containing invisible text.See the soursce of this page to see my invisible text about search engine!!! ;0)

Tiny Text: This is the technique of placing text on a page in a small font size. Pages that are predominantly heavy in tiny text may be dismissed as spam. Or, the tiny text may not be indexed. As a general guideline, try to avoid pages where the font size is predominantly smaller than normal.

| Search engines| How Search Engines Rank Web Pages|Search Engine Design Tips |