Comments on: Create a robots.txt file

By: Kamal Hasaon

Kamal Hasaon — Thu, 06 May 2010 11:56:00 +0000

Well the images folder that is being mentioned could be the ones like the icons and miscellaneous images.

By: Sue Husson

Sue Husson — Tue, 25 Mar 2008 09:12:00 +0000

I’m confused can someone just give me the correct text for a wordpress blog. Will uploading it from notepad work then?

By: Jonyon

Jonyon — Wed, 27 Feb 2008 16:15:00 +0000

gracias justo lo que necesitaba

By: Jonyon

Jonyon — Mon, 18 Feb 2008 16:35:00 +0000

Thanks for the post, very helpful.

By: Bollywood Actresson

Bollywood Actresson — Sat, 09 Feb 2008 15:26:00 +0000

Although this post didn’t discuss why and how a certain robots.txt file with some certain entries is best, you made sure people should understand what those codes mean. This is something I have never seen on any other site on my quest to find the best robots.txt file for better SEO. Thank you very much for that.

By: karimon

karimon — Fri, 12 Oct 2007 01:49:00 +0000

thanks for info

By: AskApacheon

AskApacheon — Thu, 09 Aug 2007 05:29:00 +0000

Its nice that people are beginning to think about controlling robots with robots.txt.. You may want to look at my updated wordpress robots.txt file on AskApache, especially regarding the digg mirror, way back archiver, etc..

By: Harshaon

Harshaon — Tue, 10 Jul 2007 01:02:00 +0000

I think there is no harm in allowing the Bots to assecc your images folders. It can bring traffic through the Google image search

By: SEO Thailandon

SEO Thailandon — Thu, 05 Jul 2007 01:52:00 +0000

Yes Great Post
But can you give me tips regarding how to use sitemap effectively

By: John Doeon

John Doeon — Tue, 26 Jun 2007 06:37:00 +0000

So many people with so many ideas!!! I like the ideas in general to block robots to avoid duplicate contents. But in my opinion duplicate contents should be avoided at the url level. Dont allow your cms to generate more then one url of the same post.

By: Danielon

Danielon — Thu, 21 Jun 2007 05:18:00 +0000

Derek, I see what you are asking now.

By: Derekon

Derekon — Thu, 21 Jun 2007 05:02:00 +0000

Thanks Daniel I know that the robots.txt file should still go in the root, I just want to know if I have to change the robots file to look inside my new blog installation directory
like
disallow: /blogfolder/wp-admin

By: Danielon

Danielon — Wed, 20 Jun 2007 07:26:00 +0000

If I am not wrong, even if the blog in on a sub-folder, the robots.txt file should still go on the root directory since it is the first thing a search bot will look for.

By: derekon

derekon — Tue, 19 Jun 2007 19:20:00 +0000

I was curious, my blog isnt installed in the root but instead a folder on my side.

By: Scibizon

Scibizon — Thu, 14 Jun 2007 09:07:00 +0000

The particual robot.txt file is an important choice from the SEO point of view. Thanks for the original approach to the problem!

By: John T. Pratton

John T. Pratton — Wed, 13 Jun 2007 09:27:00 +0000

you can’t upload robots.txt to the root directory using WordPress – it doesn’t have an FTP function. The only way it would be possible is if someone associated “robots.txt” as a theme associated file, and you could edit and save it in “theme editor” under the “presentation” tab. You wouldn’t think it would be too difficult to associate a file with a theme by modifying a little code, but presently I don’t know how to do it.

By: rtempon

rtempon — Wed, 13 Jun 2007 06:04:00 +0000

How do I upload the robots.txt to the root directory using WordPress?

By: Danielon

Danielon — Sat, 09 Jun 2007 04:43:00 +0000

John, regarding the first question: not all search bots recognize the * attribute. Some people argue that the Google Bot specifically does not interpret the * as a joker, it just ignores it. That is why I avoid using it. Plus, it should not be necessary to add it before a folder like */feed/.

By: John T. Pratton

John T. Pratton — Fri, 08 Jun 2007 11:48:00 +0000

Thanks for the post, very helpful. Is there any problem with doing it like this:
Disallow: */feed/
Disallow: */trackback/

By: SEOon

SEOon — Sat, 12 May 2007 09:15:00 +0000

Great article mate. I’m still unsure what needs to be excluded though. But there is some stuff that needs to be blocked to avoid duplicate content. Cheers

By: Danielon

Danielon — Tue, 20 Feb 2007 03:33:00 +0000

Ajay /trackback/ will disallow all the trackback pages, and I also think we should disallow comments, but I am not sure if /comment/ is the right attribute for that.

By: Ajayon

Ajayon — Mon, 19 Feb 2007 23:00:00 +0000

Daniel, that is not the case, rules of robots.txt are always followed no matter where the index is followed from.

By: engtechon

engtechon — Thu, 15 Feb 2007 13:45:00 +0000

One other thing to consider blocking is any duplicate content on your side. WordPress gives you about three thousand ways to access content (/page, /tag, direct links, etc). Blocking some of them might be a good idea.

By: Danielon

Danielon — Thu, 15 Feb 2007 09:48:00 +0000

Bes, if I am not the wrong the Google Image Bot will not need to crawl your image folder at all. It will crawl your pages, and it will index all the images on those pages (i.e. posts).

By: Bes Zon

Bes Zon — Thu, 15 Feb 2007 08:52:00 +0000

Thilak, unless I am mistaken, wouldn’t the Google Image Bot be following the rules of the robots.txt file regardless of where it starts crawling from?

By: Thilakon

Thilakon — Thu, 15 Feb 2007 03:10:00 +0000

I guess disalowing “wp-” will not affect Google Image bot from crawling your images because it crawls them from the post and not from the directory

By: Dawud Miracleon

Dawud Miracleon — Wed, 14 Feb 2007 18:26:00 +0000

Nice post. Great reminder of how you can easily protect folders and files on your server.

By: Danielon

Danielon — Wed, 14 Feb 2007 13:55:00 +0000

I am not sure if the GoogleImage bot tracks down images from the /images/ folder or directly from the posts where the images where inserted.

By: egonon

egonon — Wed, 14 Feb 2007 13:33:00 +0000

Same with WordPress. If you disallow “/wp-” then it’s not going to index any of your uploads like images since they are in your wp-content folder. I get quite a bit of traffic from Google Image Search.

By: Mac Utopiaon

Mac Utopiaon — Wed, 14 Feb 2007 13:05:00 +0000

I agree that robots.txt files are important, however I disagree with blocking the images folder. Some sites achieve some pretty good traffic numbers from google image search, blocking this directory will block your images from showing up in these results.