Warning: Creating default object from empty value in /home/thesitez/public_html/search-engine-cloaking/wp-includes/functions.php on line 292
Search Engine Cloaking » Webmaster World - I think I get It

Sunday January 22, 2006 JST

Webmaster World - I think I get It

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/thesitez/public_html/search-engine-cloaking/wp-includes/functions-formatting.php on line 76

At first, I didn’t get it. While some of us thought he had lost his mind, we knew that Brett Tabke was no dummy. You don’t build the most successful webmaster forum on the web without understanding a thing or two about a thing or two.

As you may already know, he is now writing a blog on his robots.txt file (which is pretty funny). Clearly, he is cloaking to send the real robots.txt file to the search engine bots. Google, Yahoo, and MSN all know this - they don’t care. Getting banned just for cloaking is a myth.

As I see it, the bandwidth from the real search engine bots is obviously an acceptable cost of doing business. It’s all those other bots from scraper sites that you want to get rid of: the ones hitting 4 million pages per day each and pirating your content.

This blog entry about bot behavior and his experiment was interesting:

# After alot of testing and bot busting, the current robots.txt is what was
# settled on. I felt exposing the code was the best way to explain it all
# (see the actual robots.txt above for the full story).
# Testing the bots code and the security code to get it all right took alot
# of time. In the end, we found:
# - A surprising 21 bots that were following all the active list posts on a
# daily basis and downloading that content.
# - About 45 trademark and other page monitoring services. The majority of
# those monitoring services obey robots.txt.
# - 15 bots would accept cookies.
# - 2 more web sites reselling WebmasterWorld content. One in China and one
# in the stans. both out of legal reach.
# Sorry Shak - China will continue to be viewed with suspicion as long as
# it is still the wild-wild-west out there with few legal controls to
# protect content.
# - about 30 people who don’t understand the concept that if you look like
# a bot with a spoofed agent name - you are a bot.
# - Some of the worst bot running offenders? A few choice SEO’s.
# These are the same folks that bring you click bots and scrapper sites.
# I think they have little respect for other peoples content. I also think
# they don’t appreciate just how impactful their actions can be.
# - http://www.ojr.org/ojr/stories/051213niles/
# Thanks to Yahoo and MSN for the permission to treat your bots as if they
# were a tough steak during the testing and coding phase. Cloaking stuff for
# testing went a long way to being able to figure out the right
# balance of settings.

I don’t think Tabke was banned, or hacked, and it wasn’t to see if he could do without search engine traffic out of pride. Real search engine bots he can deal with, it’s all those other bots that he wanted to eliminate. It was a strategic business decision to get rid of bandwidth leaching non search engine bots of people who were taking his paid content for free: one that will likely pay off in the long term


Nobody has left a comment!

Leave a Comment

Following tags allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>

Please note: Comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.


Useful Links

About Income Streams