Why Search Engine Spiders Can’t Index Your Pages

HTML Problems Can Prevent Indexing
This is a common problem. Your web designer has created a beautiful page with nice graphics and great Flash animations. Unfortunately, it seems that search engines won’t list your website no matter what you do.

This can have several reasons:

1. The HTML Code Of Your Web Pages Is Meaningless To Search Engines
   Search engines use very simple software programs to index your web pages. A web page that looks great to the human eye can be totally meaningless to search engines.
   Search engines cannot see content that is presented in images (GIF, JPEG, PNG, etc.), Flash elements, JavaScript and other script languages or other multimedia file formats.
   If you use JavaScript links for your website navigation then search engines might not be able to find your website page.
   Solution: Check your web pages with a search engine spider simulator. Spider simulators will show you how search engines see your web site.
   If you use IBP’s search engine spider simulator, you can even enter the name of the spider, so that you can test if Google and Yahoo get different results when they visit your web pages.
2. The HTML Code Of Your Web Page Contains Errors
   Some HTML errors can prevent search engine spiders from indexing your web pages. While most search engine spiders can deal with minor code errors, some of them will send faulty information to search engine spiders.
   For example, your web page could contain a tag at the top of the page that tells search engines “the web page ends here” although your main content has not been reached.
   Solution: Check the HTML code of your web pages with an HTML validator tool. You can find an HTML validator in the free IBP trial version.
3. The HTML Code Of Your Web Pages Doesn’t Contain The Right Elements
   If you want to get high rankings for a special keyword then this keyword must appear in the right places on your web page. For example, it usually helps to use the keyword in the web page title.
   There are many other elements that are important if you want to have high rankings.
   Solution: Check the HTML code of your web page with IBP’s Top 10 Optimizer. IBP’s Top 10 Optimizer compares your web page with the top ranked pages and it will tell you in detail how to change your web page so that it can get top 10 rankings.
4. Your Web Server Sends The Wrong Status Codes
   Some web servers send wrong status codes. When a search engine spider requests a web page from your site then your server sends a response code. This should be the “200 OK” code.
   

Unfortunately, some servers send a “302 moved” or even a “404 not found” response code to the search engine spiders although the web page can be displayed in a normal web browser.

In that case, search engines will think that the web page doesn’t exist and they won’t index the page.
Solution: Use the search engine spider simulator to find out which response code your web server returns to search engines. If the response code is not “200 OK”, IBP’s search engine spider simulator will return a warning message. ( Editor’s Note: You can also check your status code with our free HTTP Server Response Code Checker ) .

If you want to get top rankings on Google and other search engines then you must make sure that search engines don’t misunderstand your pages. Use the tips above to make sure that search engines see what you want them to see.

Happy trying, good luck

Source: Free Articles