Search Engine Optimization : SEO BOOK.
Chapter
9
Adding Your Site To the Indexes and Directories
Getting your Pages into The Search Engines: After you have built your
web pages, the next question is how do you get them into the search
systems? Many site owners are confused about how they get their Web
sites into search engines. They assume that to submit or register their
sites, they go to a search engine's Web site, provide the URL to one
of their pages, and wait for the search engine to come along and index
the pages.
Why Wont they Index My Pages:
It's frustrating when you cant seem to get search engines to index your
site. With some search engines you have to pay before you can submit
your URL. Other search engines enable you to submit your URL for free,
but doing so doesn't seem to make any difference; your site is not picked
up. Here are a few reasons why search engines are not as helpful as
we'd like them to be:
· Most search engines have fairly small indexes: Although Google,
AlltheWeb and Inktomi are the biggest systems, with around 3 billion
pages in their indexes, the other major systems are relatively small
- altavista holds "only" a billion pages, and Teoma only about
1.5 billion . And most of the sites are relatively more small. The smaller
systems don't care about getting every page they can find into the index;
that's simply not the game they are playing.
· Some systems want you to pay get into the search engine: they
either don't accept URLs for free or actively discourage people to pay.
· Some systems, in particular Google, believe that they can maintain
the quality of their indexes by focusing on Web pages that have links
pointing to them. These search engines would rather find your pages
themselves than have you submit the pages.
Linking Your Site For Inclusion : The more links you have pointing
at your site, the more important the search engines will think your
pages are. Links are also important because they provide a way for the
search engines to find your site. Without links. You might get your
site into the search engine's indexes, but chances are you won't . And
getting into the most important index, Google, is much quicker through
linking than through registration. I've had sites picked up and indexed
by Google within two or three days o having put up a link on another
site. If you simply submit your URL to Google, you may have to wait
weeks for it to index your site. But one or two links sometimes aren't
enough for some search engines. They are for Google, however. If Google
sees a link to your site on another site that it's already indexing,
It will probably visit your site and index that, too. But the smaller
systems are far choosier and may not visit your site even if they do
see a link to it. So the more links you have pointing to your site the
better. Eventually, the smaller systems will get back sick of tripping
over links to your site and come see what the fuss is all about. Don't
think your work is done when you create a site and then submit a page
or two to the search engines. Building links from other sites is an
essential part of the process.
Submitting Directly to the Major Systems
Are you sure it wont do any harm?
Will submitting directly to the search engines, or submitting many pages
over and over , harm your search engine position? Imagine that your
Rodent Racing Web site is sitting at the top of page two for a popular
search engine. You feel this ranking isn't good enough, so you go to
the search engine, run the search, and then grab the URLs of all the
pages that turn up to on the first results page. You then submit those
URLs to the search engine over and over again, perhaps using a submission
program. You keep doing this for days, until the search engines get
so annoyed they remove all those pages, pushing your page to the top!
Systems that are repeating submission to an egregious degree maybe blocked.
The search engine may identify your system's IP address and then block
you from submitting or even using the search engine. If the IP address
used by the program submitting URLs is the same as the IP address at
which your Website is hosted, your site could be penalized. These situation
are rare, though, because the IP numbers used by such submission programs
and the associated Websites usually different. What is far more common
is that the search engines simply block the submission program from
submitting more than a certain number of URLs at a time.
Submitting For Free
The following systems accept free URL submissions. In some cases they
wouldn't accept more than a specific number of submission: one a day,
three a day, or whatever. The system may suggest that you don't provide
more than one URL. In that case give them the URL of a page that is
linked well throughout your site, and the search engine will find the
other pages when they crawl the site. Here are some systems where you
can submit your site for free:
· Altavista.com:addurl.altavista.com/addurl/new
· Google: www.google.com/addurl.html
· Inktomi(via the Lycos site): insite.lycos.com/searchservices/lite_stepl.asp
· Inktomi(via the MSN site): submitit.bcentral.com/msnsubmit.htm
· FAST/Allthe Web: www.alltheweb.com/add_url.php
· MSN: At the timeof writing, MSN uses Inktomi, but it's preparing
to drop Inktomi and replace it with its own index. MSNbot is currently
crawling the Web, building this index, but not accepting direct submissions.
You can find information at search.msn.com/msnbot.htm.
· Zeal(if you have a noncommercial Website): zeal.com/website/add_start.jhtml
You may want to check out grub, too. The goal of the grub project is
to reach a stage where distributed crawling can update an index containing
every page on the web, every day. If a search engine doesn't provide
a way for you to submit a URL, that doesn't mean you cant get into the
search engine. You need lots of links pointing to your site so the search
engine will find it, and a fair amount of patience, but it definitely
does happen. And just because the search engine does provide a way for
you to submit a URL doesn't mean you will get into the search engine.
You need lots of links pointing to the site.
Using Paid Inclusion
Paid-inclusion programs provide you with the privilege of paying to
have a search engine index your pages. You pay the search engine a fee
- $15 a year, $49 a year, or whatever per URL - and the search engine
guarantees that your pages will remain in the index for the full term
of the agreement. I don't like the idea of paid inclusion; it strikes
as a bit of scam. The Federal Trade Commission doesn't like it much
either, and consumer-advocate complaints have targeted paid inclusion
based on the federal law that "prohibits unfair or deceptive acts
or practices in or affecting commerce such as a representation, omission,
or practice that is likely to mislead the consumer acting reasonably
in the circumstances,
to the consumer's detriment."
The argument is over whether paid inclusions are, in effect ads. The
search engines say they are not; you are simply paying for an improved
service in getting your URLs into the index, thereby improving the index.
For dynamic sites, this is tru8e because it is nearly impossible to
crawl a dynamic site. But from the consumer's perspective, paid-inclusion
ads are misleading. If a search engine ahs a complex algorithm designed
to provide accurate, relevant search results but then allows sites to
partially bypass the algorithm, how is that not misleading to searchers?
The search engines say that the paid-inclusion pages still have to compete
on the same terms with all the other pages in the index and are not
ranked any higher based on their payment status. That maybe true, but
paid-inclusion pages definitely are going to rank higher than the pages
that are not in the index because their owners haven't paid! Some paid
inclusion programs allow site owners to add enhancements to their listings.
Alta Vista lets you add a logo to your listing for $100 a year, a small
icon for $50, a customer tagline for $100 a year, and extra links to
additional pages in your site for $50.
Deciding whether to use paid inclusion
If you are trying to decide whether or not to use paid inclusion, consider
the following points:
· Teoma/Ask Jeeves, AltaVista. Inktomi and Fast/AlltheWeb all
have paid-inclusion programs.
· To submit a single URL to all preceding system's paid inclusion
programs, the cost is $182 a year.
· To submit the second and subsequent URL, the cost is $120 a
year the details in "The Paid-inclusion systems,"
· You may get stuck paying the search engines their paid inclusion
fees year after year.
· If you have a really good linking strategy, you may get these
search engines to index your site without having to pay them. A large
number of high-quality in the index, but it doesn't affect your position
in the index, so you may be paying to be on a page that few people read.
Some people like paid inclusion because of the speedy updates. You
can tweak pages and see the results in your search engine rank within
a few days. But again, each paid inclusion program accounts for only
a small proportion of the overall search engine results anyway. All
the major search engines that use paid inclusion programs also add pages
to the index for free. Three of the four accept free submissions pages,
they try to steer you toward paid inclusion - and all their searchbots
add pages that they find as they are crawling the Web. If these large
systems limited their indexes to only paid inclusions, they would quickly
go out of business. So a few important questions about paid inclusion:
· If you pay to have your pages indexed, what happens when the
agreement period expires? The search engines remove your pages. They
have to, or the paid inclusion program more or less collapses, doesn't
it? That means the search engines, in effect, hold you hostage, removing
pages that might otherwise have been indexed automatically by their
searchbots.
· What's to stop you from indexing a single page using paid inclusion,
a page that contains lot of links to your other pages, to make sure
the search engines find all your pages? The search engines want you
to submit multiple pages via the paid-inclusion program, so it would
be against their interests to do this. Are the search engines disabling
the normal algorithm for paid-inclusion pages to ensure they aren't
picked up for free?
· If in its normal course of travel, the searchbot runs across
pages that you have paid to have included, will it index those pages
and flag them as no longer requiring payment?
The Paid-Inclusion Systems
If you interested in paid inclusion here is the little information about
the major players:
· Inktomi: 72 hour addition to the index,48 hour reindexing .
For one year, you pay $39 for the first URL and $25 for each one there
after. Sign up at www.inktomi.com/products/web_search/submit.html.
· AltaVista: 48hour addition to the index, daily reindexing.
For six months, you pay $39 for the first URL, $29 each for two to ten
URLs and $19 for each one thereafter. Sign up at www.altavista.com/web/express_incl.
· FAST/AlltheWeb:72 hour addition to the index, 48 hour reindexing.
For one year, you pay $35 for the first URL and $19 foreach one thereafter.
Signup at www.alltheweb.com/help/webmaster/submit_site.
· Teoma/Ask Jeeves: Pages added within a week and reindexed every
week. For one year, you pay $30 for the first URL and $18 for each one
thereafter. Sign up at ask.ineedhits.com.
The search engines using paid inclusion sell the service through resellers
who may have slightly different pricing and services. Some of the major
resellers are Position Technologies, Lycos and Network Solutions.
Using Trusted Feeds
Another form of paid inclusion is known as a trusted feed - a sort of
en masse paid inclusion. Trusted feeds are sometimes used by companies
with dynamic Web sites and are generally intended for large numbers
of pages, 500 or more. With a trusted-feed program, you don't pay to
have your pages included, but you do pay for every click on a link to
one of those pages, $0.25 or more per click. All four of the paid-inclusion
services, have trusted feed programs. You can get data into the search
engines two ways:
· Provide a list of URLs in a text file.
· Provide an Excel spreadhsheet containing, for each page, the
URL, page title, DESCRIPTION meta tag, KEYWORDS meta tag, and the page
content.
Providing the data in an excel spreadsheet allows you to submit data
to the search engines so they don't have to crawl your web site to pick
it up.It also allows you to submit the relevant data - ignoring irrelevant
page content - which raises the question.
Submitting to the Secondary Systems
You can also submit your site to smaller systems with perhaps a few
hundred million pages in their indexes - sometimes far fewer. The disadvantage
with these systems is that they are relatively little used, Compared
to the big systems. So if the iste is ranked in these systems, you have
less competition because they are so small. Here are a couple secondary
systems that are worth submitting to:
· ExactSeek: www.exactseek.com/ad.html
· Gigablast: www.gigablast.com/addurl
A list of additional search pages has been provided at the site www.SearchEngineBulletin.com,
and you can find more, including regional sites, listed on the following
pages:
· www.searchenginewatch.com/links/article.php/2156161
· www.dir-search.com
· www.allsearchengines.com
· www.searchengines.com/generalKids.html
· www.searchengines.com/worldUSCan.html
· dir.yahoo.com/Computers_and_Internet/Internet?World_Wide_Web/Searching_the_Web/Search_Engines_and_Directories/
Some of the smaller search engines try to encourage you to pay for a
submission. Don't. Unless you know for sure otherwise, you can safely
assume that the amount of traffic you are likely to get is probably
not worth the payment. Some of the smaller search sites don't have their
own searchbots; they get the data from other systems such as the Open
Directory Project. So sometimes you'll run across search sites to which
you cant submit directly. If you plan to submit your site to many search
engines, you may want to use a programmable keyboard or a text-replacement
utility or macro program such as Short keys, which can make entering
repetitive data much quicker:
· With a programmable keyboard, you can assign a string of text
- URL, email address and so on. - to a single key, so all you need to
do to enter your email address, for instance is to press F1.
· A text replacement utility replaces a short string of text
with something else. To enter your email address you might just type
em.
Using Registration Service and Software Programs
You can submit your pages to hundreds of search engines by using a submission
service or program. Many free service are available, but some of them
are out of date - They have been sitting out on the internet running
automatically for a few years now and haven't kept up with changes in
the search engine scene.Many of the free services are definitely on
the lite side. They are provided by companies that hope that you will
spring for a fee-based fullservice. E.g. one well-known service, Adpro,
provides free submission to the following search engines: ABACHO, Acoon,
Aewi, Cipinet, Entireweb, ExactSeek, Find Once, Fyberation, Google,
Infomark, Kunani,ScrubTheWeb,Searchit, SplatSearch, Subjex, SurfGopher,
Walhello and Wotbot. ABAHCO and Acoon are German searchengines and Aewi
is no longer in business. Adpro also has a paid service that submits
to 120 search engines. For $79.95 a year, it adds one URL to Inktomi
through the Inktomi paid inclusion system, manually submits your site
to AltaVista,AlltheWeb,Google and "3 other directories" and
uses a submission program to submit to120 other services.
Another popular submission system is provided by ineedhits, which submits
a URL to 300 seach engines for as little as $2.50. Many of these submission
services are available; search for search engine submission service.
Many software programs are available, as well. Here are a few of the
more popular ones:
· Dynamic Submission, which you can find at www.dynamicsubmission.com
· SubmitWolf, at ww.submitwolf.com
· WebPosition Gold, at www.webposition.com.
The big advantage of these software programs is that you only pay once
rather than pay for a service every time you use it.
