Comments on: Collection of Robots.txt Files

By: Dan from Lawn Mower Reviews

Dan from Lawn Mower Reviews — Fri, 25 Mar 2011 05:00:40 +0000

Hi,
I use the All in one SEO plugin on most of my sites. Will this automatically insert a robot.txt file for me? I have heard about this before, but to be honest I kinda overlooked it and didn’t think it was too important, however now I am a little worried that I am missing a vital trick here.

I would guess that wordpress automatically create the robot.txt file for you, but just double checking?

Thanks

By: shabi

shabi — Mon, 16 Aug 2010 12:56:03 +0000

hi…nice to see this…but can anyone tell me…whatis the function of these lines in robots.txt…
“User-Agent: Googlebot
Disallow: /index.xml$
Disallow: /excerpts.xml$”
i have seen these lines used gizmodo and other bloggers..
should these 2 lines help in removing duplicate content??
pls i want the answer..
thanks

By: Free Web Directory

Free Web Directory — Wed, 03 Jun 2009 08:17:11 +0000

nice stuff, you make ideas for me to change my robots.txt

By: Elle

Elle — Sun, 10 May 2009 20:54:15 +0000

Should the /wp-content folder be included as well…where all the themes and plugins reside? I’ve not noticed it listed on any robot.txt file anywhere..

Thanks.

By: Bang Kritikus

Bang Kritikus — Mon, 02 Feb 2009 04:11:32 +0000

But, my blogspot’s robot.txt is not editable..

By: SEO Freelancer

SEO Freelancer — Sat, 08 Mar 2008 08:59:38 +0000

Nice collection – this will help me and new webmaster and web designer to create a robots.txt file as they want

By: John

John — Fri, 08 Feb 2008 08:59:30 +0000

Hello…!

Can anyone tell me the list of websites which archives the websites. Pandora, Internet archive’s Waybackmachine are the some of the examples, I want to know the entire web archiving websites, please…..

By: Sangesh

Sangesh — Thu, 03 Jan 2008 15:23:22 +0000

I got to know more about the “robots.txt” in this article.

Thanks.

By: ä¸œèŽžç½‘ç«™å»ºè®¾

ä¸œèŽžç½‘ç«™å»ºè®¾ — Tue, 20 Nov 2007 09:06:12 +0000

Very good learning

By: AskApache

AskApache — Thu, 09 Aug 2007 13:35:39 +0000

The real benefit to learning about the robots.txt file and how it works is it teaches you to think like the web crawlers. Especially when you start targeting different user-agents/bots…

webmasterworld is definately the coolest, and 2nd is of course askapache.com

By: Visitor413

Visitor413 — Thu, 19 Jul 2007 16:18:37 +0000

Your site found in Google:

By: Visitor367

Visitor367 — Thu, 19 Jul 2007 16:18:04 +0000

I have visited your site 623-times

By: Daniel

Daniel — Sat, 14 Jul 2007 16:16:49 +0000

The effect upon individual ranking of your pages should not be huge, so do not expect to go from the tenth page to the first page of Google just because of using a robots.txt file.

That said, your search engine traffic will probably increase a lot if many of your pages were in the supplemental hell. First and foremost because now you will be cover many more keywords and terms.

By: TechZilo

TechZilo — Sat, 14 Jul 2007 11:04:51 +0000

I’d like to echo Zath’s question, since my number of indexed pages has gone down too..will it affect SERPs?

By: vijay

vijay — Sun, 01 Jul 2007 14:05:18 +0000

Hmm. I haven’t thought yet to update my robot.txt
Its as simple as problogger.net no complications 😉
I actually avoided that part coz I am not that much aware of robot.txt file changes and its effects!
Soon will give some time for that.
Thanks for the advice anyway.

By: vijay

vijay — Sun, 01 Jul 2007 14:04:38 +0000

Hmm. I have not thinked yest to update my robot.txt
Its as simple as problogger.net no complications 😉
I actually avoided that part coz I am not that much aware of robot.txt file changes and its effects!
Soon will give some time for that.
Thanks for the advice anyway.

By: Zath

Zath — Sun, 01 Jul 2007 08:56:51 +0000

I recently set up a robots.txt file and have noticed that my supplemental links on Google have gone down from around 2000 pages to about 250.

I’m thinking that’s pretty good, but like others have said, I’m not quite sure how much of a difference it makes to my site rankings.

Will this give more search engine traffic going forward or increase the chances of a better Pagerank?

By: Matt Wardman

Matt Wardman — Fri, 15 Jun 2007 13:42:08 +0000

>Wow, I’m surprised that so many SEO experts don’t include a line for sitemap autodiscovery. It’s not like it’s difficult to implement or anything

If you have a Google sitemap plugin for WordPress it pingfs google every time you post anyway.

And:

The robots.txt for webmasterworld.com has a blog in it. Fun.

By: CypherHackz

CypherHackz — Fri, 15 Jun 2007 07:17:15 +0000

i have this, list of robots.txt links. you can see here: Big Websites with Big Robots

By: Pchere

Pchere — Fri, 15 Jun 2007 05:03:07 +0000

I am also tweaking my robots.txt to remove duplicate content in WordPress. It was very insightful to see how top sites are dealing with the issue.

By: Jordan McCollum

Jordan McCollum — Thu, 14 Jun 2007 22:16:26 +0000

What timing! I was just contemplating roboting out my category and archive pages. Thanks for this!

By: Patrix

Patrix — Thu, 14 Jun 2007 21:48:29 +0000

I have been tweaking my robots.txt file for quite some time now mostly to reduce duplicate content (get pages out of supplemental hell)but haven’t noticed any appreciable difference.

I have been checking a few A-bloggers blogs for their robots.txt files so thanks for doing this.

BTW why does Shoemoney disallow some directories with and without the forward slash? What is the difference?

By: Daniel

Daniel — Thu, 14 Jun 2007 16:10:56 +0000

Adnan, I will need to tweak mine as well. So far I am getting pretty good results with a minimalistic one though, just exclusing feeds, trackbacks and WP files.

By: Adnan

Adnan — Thu, 14 Jun 2007 16:06:35 +0000

Hey Daniel – thanks for that compilation – its very interesting to see how some SEO sites like SearchEngineJournal were minimal, but how SEOMoz has something different.
Now I need to decide on which one to choose 😉

By: Hugh | A Politically Incorrect Entrepreneur

Hugh | A Politically Incorrect Entrepreneur — Thu, 14 Jun 2007 15:14:30 +0000

While crawling around the interweb a few days ago, I found the robots.txt file for the whitehouse (whitehouse.gov/robots.txt)

I just thought it interesting the things they disallowed.

By: John Wesley

John Wesley — Thu, 14 Jun 2007 14:34:22 +0000

Very interesting post. I actually started using something similar to Chow’s after he published it on his blog last week. It seems to be adding a bit of Google traffic.

By: Daniel

Daniel — Thu, 14 Jun 2007 14:05:37 +0000

Nia, sorry for that I just updated the article with a link to an introductory post I wrote sometime ago:

By: Nia

Nia — Thu, 14 Jun 2007 14:01:19 +0000

This looks valuable except I don’t know how to use it yet. I’ve put it in my RSS shares and when I figure it out I’ll implement the lesson and post about it. Thanks. 😉

By: Pablo

Pablo — Thu, 14 Jun 2007 13:56:54 +0000

nice stuff, i already changed my robots.txt

By: Daniel

Daniel — Thu, 14 Jun 2007 13:54:29 +0000

Stephen, I don’t think the “autodiscovery” factor is related to how easy it is to implement.

The question is: will it bring tangible improvements?

By: Stephen

Stephen — Thu, 14 Jun 2007 13:50:14 +0000

Wow, I’m surprised that so many SEO experts don’t include a line for sitemap autodiscovery. It’s not like it’s difficult to implement or anything…