Business, economic development - the unofficial Chamber of Commerce

The robots.txt file is an exclusion standard required by all
web crawlers/robots to tell them what files and directories
that you want them to stay OUT of on your site. Not all
crawlers/bots follow the exclusion standard and will continue
crawling your site anyway. I like to call them “Bad Bots” or
trespassers. We block them by IP exclusion which is another
story entirely.

This is a very simple overview of robots.txt basics for
webmasters. For a complete and thorough lesson, visit
http://www.robotstxt.org/

To see the proper format for a somewhat standard robots.txt
file look directly below. That file should be at the root of
the domain because that is where the crawlers expect it to be,
not in some secondary directory.

Below is the proper format for a robots.txt file —–>

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /group/

User-agent: msnbot
Crawl-delay: 10

User-agent: Teoma
Crawl-delay: 10

User-agent: Slurp
Crawl-delay: 10

User-agent: aipbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: psbot
Disallow: /

——–> End of robots.txt file

This tiny text file is saved as a plain text document and
ALWAYS with the name “robots.txt” in the root of your domain.

A quick review of the listed information from the robots.txt
file above follows. The “User Agent: MSNbot” is from MSN,
Slurp is from Yahoo and Teoma is from AskJeeves. The others
listed are “Bad” bots that crawl very fast and to nobody’s
benefit but their own, so we ask them to stay out entirely.
The * asterisk is a wild card that means “All”
crawlers/spiders/bots should stay out of that group of files
or directories listed.

The bots given the instruction “Disallow: /” means they should
stay out entirely and those with “Crawl-delay: 10″ are those
that crawled our site too quickly and caused it to bog down
and overuse the server resources. Google crawls more slowly
than the others and doesn’t require that instruction, so is
not specifically listed in the above robots.txt file.
Crawl-delay instruction is only needed on very large sites
with hundreds or thousands of pages. The wildcard asterisk *
applies to all crawlers, bots and spiders, including
Googlebot.

Those we provided that “Crawl-delay: 10″ instruction to were
requesting as many as 7 pages every second and so we asked
them to slow down. The number you see is seconds and you can
change it to suit your server capacity, based on their
crawling rate. Ten seconds between page requests is far more
leisurely and stops them from asking for more pages than your
server can dish up.

(You can discover how fast robots and spiders are crawling by
looking at your raw server logs - which show pages requested
by precise times to within a hundredth of a second - available
from your web host or ask your web or IT person. Your server
logs can be found in the root directory if you have server
access, you can usually download compressed server log files
by calendar day right off your server. You’ll need a utility
that can expand compressed files to open and read those plain
text raw server log files.)

To see the contents of any robots.txt file just type
robots.txt after any domain name. If they have that file up,
you will see it displayed as a text file in your web browser.
Click on the link below to see that file for Amazon.com

http://www.Amazon.com/robots.txt

You can see the contents of any website robots.txt file that
way.

The robots.txt shown above is what we currently use at
Publish101 Web Content Distributor, just launched in May of
2005. We did an extensive case study and published a series of
articles on crawler behavior and indexing delays known as the
Google Sandbox. That Google Sandbox Case Study is highly
instructive on many levels for webmasters everywhere about the
importance of this often ignored little text file.

One thing we didn’t expect to glean from the research involved
in indexing delays (known as the Google Sandbox) was the
importance of robots.txt files to quick and efficient crawling
by the spiders from the major search engines and the number of
heavy crawls from bots that will do no earthly good to the
site owner, yet crawl most sites extensively and heavily,
straining servers to the breaking point with requests for
pages coming as fast as 7 pages per second.

We discovered in our launch of the new site that Google and
Yahoo will crawl the site whether or not you use a robots.txt
file, but MSN seems to REQUIRE it before they will begin
crawling at all. All of the search engine robots seem to
request the file on a regular basis to verify that it hasn’t
changed.

Then when you DO change it, they will stop crawling for brief
periods and repeatedly ask for that robots.txt file during
that time without crawling any additional pages. (Perhaps they
had a list of pages to visit that included the directory or
files you have instructed them to stay out of and must now
adjust their crawling schedule to eliminate those files from
their list.)

Most webmasters instruct the bots to stay out of “image”
directories and the “cgi-bin” directory as well as any
directories containing private or proprietary files intended
only for users of an intranet or password protected sections
of your site. Clearly, you should direct the bots to stay out
of any private areas that you don’t want indexed by the search
engines.

The importance of robots.txt is rarely discussed by average
webmasters and I’ve even had some of my client business’
webmasters ask me what it is and how to implement it when I
tell them how important it is to both site security and
efficient crawling by the search engines. This should be
standard knowledge by webmasters at substantial companies, but
this illustrates how little attention is paid to use of
robots.txt.

The search engine spiders really do want your guidance and
this tiny text file is the best way to provide crawlers and
bots a clear signpost to warn off trespassers and protect
private property - and to warmly welcome invited guests, such
as the big three search engines while asking them nicely to
stay out of private areas.

Copyright © August 17, 2005 by Mike Banks Valentine

Google Sandbox Case Study http://publish101.com/Sandbox2
Mike Banks Valentine operates http://Publish101.com
Free Web Content Distribution for Article Marketers and
Provides content aggregation, press release optimization
and custom web content for Search Engine Positioning
http://www.seoptimism.com/SEO_Contact.htm

“Owning your business category online
is more than just a few keywords…
 Is your website broken?”

Let’s face it, the Internet is a crowded place… and it’s becoming even more crowded as you read this. Your business can’t stand still online, and you can’t afford to look backwards. Qualified, targeted traffic generation is what we all desire. Owning your category online goes a long way in providing just that.

It’s not about just looking forward, but moving forward and taking some ownership. Every business online today is worried about being found in the Top 10 Search Engines and Directories for their products or services - and they should be! Statistics still make them (search engines) the number one stop for all surfers.

But, just for a moment let’s imagine that these search engines and directories provide the most relevant results. That’s right, let’s just say for the sake of this article, that they all work extremely well. O.K?

Now, since we believe that our products or services are way more relevant than those companies listed above us in (today’s) search engines results - we have to admit something is wrong… but remember, the search engines are perfect.

We all know you can buy your keywords in Google™ and Overture™ right? Does that make you the best? (not.) Do surfers know this? (Yes.) Does it make your business (your website) any more relevant? (No.) It does put businesses with a larger budget at the top of the pile again, that’s all. Is this why many of us don’t click on sponsored links? (Ahem…)

If the search engines were (are) perfect, then let’s get back to something else being wrong. Hmmm, my website must be broken! Wow, somebody is actually staying with me here.. My website must be broken?

Why else would all those “other” websites be showing up ahead of my business when people search for my products/services/information? Don’t these people know we are the experts? (nope.) Don’t they know our products are superior? (nope.) Can’t they see the skill we employ and our dedication to employee and customer service? (nope.) Haven’t they read the reviews? (probably…)

So, in an effort to help you “own your category online”, here’s some suggestions to assist your business in getting ahead of those “other” websites and putting your business where it rightly belongs! If you already employ all these practices, we commend you. But if you find yourself behind, these tips can help your company without having to become a Webmaster or a Search Engine Marketing Professional. Pass these on to your Marketing Team and get better results and recognition starting today!

Do: Research the search terms (keywords) that people would use in seeking out your offerings. You can start with Wordtracker™ and the search terms you first come up with.

Wordtracker™ will help you locate terms that are similar to your terms in meaning and search. It is a tool that gives you an *idea of what the general terms people are using, and how many searches are taking place on these terms… * that’s another book though. This is a great starting point and the trial really is free!

Tip: Do not just ‘look-up’ your company name, - prospective customers or vendors or partners are not looking you up by your ‘company name’ in Google™ - they don’t know it yet…? Surfer’s have become quite good at ’searching’. Be specific and general. You can also learn who you have to beat online! - Who’s kicking your butt when people search for your product or service online? That first page of results is your market share online!

Do: Make sure you have a *keyword specific “Title” for every webpage that’s relevant to the content on that page (*keyword specific, not keyword stuffed).

Page Title; Top of the screen, blue ‘title’ bar (in explorer). I still see thousands of websites with “Homepage” in the title bar, or “MyOneHotBusiness.com” and just that phrase or ‘Title’ is repeated throughout every page on their website! Each page in your website is (content) unique - make it’s ‘Title’ unique too! Or call your webmaster with new “titles” for each of your website pages. Different values are placed on this “page title” from the different search engines and directories.

Tip: Add “Chicago, Chicagoland, Illinois” to the end of your Title (your city, state). This helps in regional search queries.

Do: Have at least 200 words of (content specific) text on the pages of importance (homepage, category pages, etc.).
This is what the engines and directories DO read, these words and links are how the engines interpret what’s on this page. Images or pictures of words or “ads” are not “read” so-to-speak. Make this text content highly relevant to the topic (product or service) on this page.

Tip: Put your fancy Flash commercial on a link from your homepage and put up a content relevant page of your best text instead! Visitors will still look at your commercial if you tell us it’s cool, but don’t force us to watch it! - That’s not what we came for. This Flash file does almost nothing for search engines and directories.

Do: Have yourself or your professional staff write articles on their specific area of expertise in product or service. Make these articles a webpage with relevant links to the product or service directly.

Short of hiring a Search Engine Marketing Consultant or a Professional Copywriter, these detailed articles if rich and full of relevant content, are the exact pages that will generate you the most qualified and targeted visitor to your business! Once these ‘article pages’ are crawled (indexed) by the search engines you will be pleasantly surprised to see that (now) you are a more relevant search result.

(Not to be confused with ‘links’ or ‘reciprocal linking programs’ - your (article) webpages are much more “read” and effective in sending the reader to your site, or keeping them there, not to mention the branding opportunities)

These articles not only show your business as an authority, they prove it in words and information for the reader, me, a human. These concise type articles are an excellent way to gain exposure, recognition (branding), traffic, and yes, even sales. You said you were an authority, now prove it.

Tip: Submit these articles to other online and offline publications and Ezine’s. You make them available to other websites just for leaving your Author and link intact. There are businesses craving this type of content for their website’s success! Your business website will begin receiving quality traffic from every website that posts your company’s article(s)! Now there’s a strong statement.

Do: Include your URL (www.) in every piece of material you print - every piece of traditional advertising you already pay for!

Every week I read through different newspapers and magazines - I can’t believe how many businesses forget to put their web address in that Advertisement they paid for. You pay big bucks for a quarter page ad or whatever, and then never mention your website!?! Which is open 24/7/365 - and that Tribune is just around for two days! Start your marketing return on investment right there!

Tip: Offer a website only promotion or coupon in your Ads to generate further interest and traffic to your specific offer web pages!

Do: Ask visitors to “Bookmark your site” or “Tell a friend” or “Send this article to a friend”…

Your visitors don’t always think of it, so just ask them? Bookmarks not only increase return visitors, but they also give you a vote of confidence in providing something worthwhile… even if that’s a return trip! This does not have to be bold and obnoxious either, a simple button saying “Bookmark Us” will do on any business website.

Do: Be Patient. All search engines and directories have different timeframes for updates, changes and new submissions.
After making changes or additions to your site, allow 90 days after initial submission for realistic results to begin showing in the major search engines and search directories.
Owning your category online is more than just a few keywords. Don’t just say you do all these things!

Do: Go and check!

Tip: Go and check again!

Because if you do your best with these few suggestions - your website will show up higher in the search results for those relevant terms you selected. You’ll be hundreds of websites ahead, and getting more qualified visitors too. There’s much more to owning your category online, but every business should be employing these “level one” marketing concepts.

There, now we can stop imagining that the search engines are perfect.

Good Luck.

Written by and Copyright © Scott Sedwick
http://www.hyperformancemedia.com

About The Author

Scott is the Founder and Sr. Project Manager for
Hyperformance Media, Inc. a website marketing company
since 1996. His 24+ years experience in the computer
industry can help your business succeed online!

Copyright SS
ss@hyperformancemedia.com
http://www.hyperformancemedia.com

I admit it, our SEO services are pretty bad.

Speaking of bad, the Beatles couldn’t sing a tune, John Grisham has never written a decent novel, Dell makes crappy computer’s, TiVo’s and iPods suck, the Lord of the Rings movies lacked imagination and video killed the radio star. (OK, that last one might actually be true!)

Do you see where this is going? To somebody, one or more of the above statements are true. Not to the vast majority, but some people do feel that way. No matter what the product or service there will always be those that are dissatisfied. Have you ever heard of a company with a 100% customer satisfaction rate (outside of their own marketing gibberish)? I haven’t.

There have been a number of blog and forum posts and articles recently about what to do if you’re with a bad SEO company. All good advice, but sometimes it’s not the SEO that’s bad, its the client. Yeah, that’s right, sometimes the client’s are just plain clueless, have unreasonable expectations, or fall prey to the “grass is greener” scenario presented by competition. Yes, sometimes clients are bad.

It’s Not You, It’s Me!

Like every other company, product and service on the planet, search engine optimization firms will always have dissatisfied clients.

Our firm has a significant number of very satisfied clients since 1998. We have also had a handful of clients that walked away dissatisfied with our services. I’m sure the same is true with every other SEO company out there. Does that make them bad SEOs? No, not necessarily.

Can good SEOs sometimes produce bad results for their clients? Absolutely. Happens all the time. Let me give you a few examples.

We worked with one client for several years. In the early phases of their campaign they selected some really poor keywords. We let them know this but they were insistent that we move forward. We were able to get most of their keywords ranked very well on the search engines, but as we suspected, sales did not increase. Every few weeks the client would call us asking if there was anything else that can be done to increase sales. We often made suggestions on how to improve their site along with recommending they change their optimized keywords. The site improvements they considered and often made. They keyword changes were considered but never approved. Eventually they left us.

We often find clients changing their website causing our optimization efforts to be overwritten with un-optimized versions of their pages. Those clients will then call us wondering why their rankings dropped. Duh! When this happens repeatedly with the same client you start to realize that you’re just not going to get any good sustainable results. [Shameless plug: our new CodeMonitor tool notifies SEOs and/or clients when page changes are made.]

Still other clients have limited our ability to optimize their websites with all kinds of restrictions. Don’t change this, can’t change that. There is always a “good” reason, but when your hands are tied there is only so much that you can do.

So What’s the Problem?

I recently had a client leave dissatisfied because his rankings dropped for a couple of weeks and he felt we just were “not working for him” anymore. The worst part about it is that he just canceled the contract without talking to me or anyone on our team to find out what’s going on. I took me several emails and phone messages to find out why he left.

We often get calls and emails from clients asking why rankings dropped which affords us the opportunity to explain things and usually ask for patience as these type of fluctuations occur every now and then. One client did just that and once his rankings popped back into the top positions he emailed me just to say, “hey, you were right!”.

We get a lot of calls (and clients) that come to us from other SEO firms that just didn’t pan out for them. Before I go off telling them that their SEO company is bad I try to find out a few things. How long have they been optimizing your site? What changes have they made? Is their name Traffic Power? Etc., etc. A few such questions will usually give me what I need to know to determine if they have cause to jump ship or not. If they have been with the company only for a few months I usually tell them to hang tough and give them a chance to produce results. Unfortunately SEO does take time, especially when a site gets sandboxed, and you wont’ always know that you’re with a bad firm for quite some time.

Talk to Me, Baby!

Communication is the key. If you have difficulty getting calls and emails returned, or are not quite sure what your SEO company is doing for you, then there is a good chance you’re not with a great firm.

One company came to us from one of the largest SEO firms after their one-year contract expired. We checked their rankings only to find sub-standard results. There were no penalization or sandbox issues, just lack of good performance. I was able to determine that their current firm simply wasn’t doing a good enough job. Once this client signed with us we were able to get most of their keywords on the first page within the first two months.

nother firm came to us after their previous SEO got them banned from Google. Coming to us we couldn’t guarantee any more than we would fix their site to make it compliant with search engine guidelines and work aggressively to get them back into the index. After fixing the spam issues, and several months wait… and several phone calls asking “when”, Google finally re-included them, and with great rankings on top of it.

Denial is Often the First Sign of a Problem

Nobody wants to think they are a bad client, but here are some signs:

• Does not communicate concerns to their SEO firm
• Does not listen to advice given
• Refuses to change their site based on SEO recommendations
• Demands optimization for poorly targeted keywords
• Constantly overwrites optimization efforts with page changes
• Expects rankings to appear overnight
• Calls all the time to complain about lack of progress due to the above

The signs of a bad firm are pretty obvious, bad communication, spammy optimization, getting your site banned, etc. If you work with one of them then run away as soon and as fast as you can. It’s usually pretty easy to tell a good from a bad. Communicate with (don’t harass) your SEO firm. A trustworthy and respectable firm will keep open lines of communication. Just be ready to listen to what they have to say and implement recommendations that they provide, even if those recommendations are to be patient.

Maybe it’s you. Maybe it’s them. Get the facts before you do anything. You definitely don’t want to be stuck with a firm that’s just sucking your money away and doing very little work for it. On the other hand, you don’t want to jump ship mid-way in what will turn out to be a substantially rewarding campaign just because things didn’t happen fast enough. Before you dump your SEO, make sure the problem isn’t you!

Stoney deGeyter is president of Pole Position Marketing, a professional search engine optimization and marketing firm providing SEO services since 1998. Stoney is also a part-time instructor at the community College in Reno, Nevada where he resides as well as a moderator in the Small Business Ideas Forum. You can read his SEM blog at (EMP) E-Marketing Performance. You can email him with any questions at stoney@polepositionmarketing.com

Google now checks the year your domain name was first registered.

This just makes sense. Those that care about their domain and their brand will register their domain for a long time. This demonstrates commitment. Of course it’s not very hard for a spammer to do the same but the upfront costs are just that little bit higher and might help act as a deterrent. If anything a short domain registration period will be yet another flag in the Google system that will keep certain sites away from top rankings. Hopefully it will be the spammers that trigger the penalty when combined with all the other spam flags they trigger.

If you are anything like me and you like to register domains for an “idea” you have for the future there is no way you will be investing in a 10 year registration for something you may never pursue. An idea is an idea and I know half of the domains I buy amount to nothing. However buying the domain also signifies *some* commitment to the project and on many occasions is the motivational spark I need to get the website built and a new project off the ground. A one or two year registration is not a significant cost. Securing a domain for 10 years is. The easy workaround is to initially register a domain for the minimum period, if things take off then renew the domain for a longer period. Simple.

Google now places huge emphasis on links. They want to see a slow, gradual increase to the number of incoming links to your websites. Links need to have a variety of anchor text phrases. If all your anchor links are the same you could get de-listed or lose ranking position.

Ahh, natural linking patterns. I’ve mentioned this before - Google in many ways destroyed what it relies on to create such a great search engine. Before Google linking patterns were very natural, organic with sites linking to each other in many different ways, with different anchor text built up over a long period of time. Sites grew in popularity slowly and incoming links increased subsequently at similar rates of growth. Google stepped in and used these patterns in a ranking algorithm to accurately value sites. It worked, almost too well. Google became so popular that people began to study what it takes to rank highly. They learnt it was all about incoming links so they started an unnatural linking process, creating link farms and chasing incoming links with a vengeance.

Google wants natural linking back and will reward those sites that appear to be popular based on natural linking. This is not an exact science of course but if you do these things you are on the right track:

• Make sure your incoming links are not all carbon copies of the same keyword phrase. Vary the incoming anchor text with different phrases.
• Make sure you get links from many sites with varied PageRank. Assess link swaps based on the site (content, relevance) asking for the exchange, not solely on the green PR bar at the top of the site.
• Chase back links naturally, slowly increasing their number over time. If you go from 10 - 5000 backlinks in one month Google is going to think you are link farming and penalise you.
• Don’t get paranoid. Stressing over why your site won’t get listed and pressuring other sites for link exchanges can drive you crazy. Take it easy and work on building a great site slowly, tell the right people about it and they will spread the word for you.

Google is telling us that they look for relevant, quality content on your websites, which is no surprise, and that PageRank is a good indicator of a website’s “authority” and relevancy.

The key term here is authority. Those sites that have been online for a long time with established authority in their field wield the power. One link from an authority site can boost you to the top of the rankings but take it away and you can just as easily fade to the bottom of the rankings. Again the emphasis here is on establishing links from many sites with various rankings. If you build an amazing site eventually the authority sites in your field may just link to you anyway and won’t that make you feel special!

How do you build an authority site? With hard work of course. Authority sites don’t appear out of nowhere, they build their authority over time by consistently working on quality content and audience creation. Don’t expect overnight success, if you want a popular website you have to work at it for years, not weeks.

By Yaro Starak
http://www.entrepreneurs-journey.com

Do you want to profit from your own successful home based Internet business?

Learn from Yaro Starak, a young entrepreneur from Australia. He works part time from home on several web based business that generate between $2,000 and $8,000 per month. Get your free articles and audio now - visit his Internet Business Blog.