• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
Daily Blog Tips

Daily Blog Tips

How to make money from your blog

  • Home
  • Popular
  • Contributors
  • About
  • Contact / Advertise
  • Blogging
  • Marketing
  • Design
  • Money
  • Reviews
  • Productivity
  • Software
  • Technology
  • Guides

Create a robots.txt file

By Mark Felix 30 Comments Reading Time: < 1 minute

background image

The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.

A robots.txt is a simple text file that can be created with Notepad. If you are using WordPress a sample robots.txt file would be:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

“User-agent: *” means that all the search bots (from Google, Yahoo, MSN and so on) should use those instructions to crawl your website. Unless your website is complex you will not need to set different instructions for different spiders.

“Disallow: /wp-” will make sure that the search engines will not crawl the WordPress files. This line will exclude all files and foldes starting with “wp-“ÂÂ from the indexation, avoiding duplicated content and admin files.

If you are not using WordPress just substitute the Disallow lines with files or folders on your website that should not be crawled, for instance:

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /any other folder to be excluded/

After you created the robots.txt file just upload it to your root directory and you are done!

DBT icon

About Mark Felix

Mark is an digital entrepreneur from New York City who recently acquired DBT. He started his ventures buying growing then selling several businesses and content blogs on Flippa. A perfect night for him constitutes pizza, wine, slow jazz and WordPress! He's an SEO expert and a growth strategist. When he's not working on his businesses he enjoys sharing his learned lessons with the DBT community.

Filed Under: Blogging

Reader Interactions

Comments

  1. Kamal Hasaon says

    at

    Well the images folder that is being mentioned could be the ones like the icons and miscellaneous images.

    Reply
  2. Sue Husson says

    at

    I’m confused can someone just give me the correct text for a wordpress blog. Will uploading it from notepad work then?

    Reply
  3. Jonyon says

    at

    gracias justo lo que necesitaba

    Reply
  4. Jonyon says

    at

    Thanks for the post, very helpful.

    Reply
  5. Bollywood Actresson says

    at

    Although this post didn’t discuss why and how a certain robots.txt file with some certain entries is best, you made sure people should understand what those codes mean. This is something I have never seen on any other site on my quest to find the best robots.txt file for better SEO. Thank you very much for that.

    Reply
  6. karimon says

    at

    thanks for info

    Reply
  7. AskApacheon says

    at

    Its nice that people are beginning to think about controlling robots with robots.txt.. You may want to look at my updated wordpress robots.txt file on AskApache, especially regarding the digg mirror, way back archiver, etc..

    Reply
  8. Harshaon says

    at

    I think there is no harm in allowing the Bots to assecc your images folders. It can bring traffic through the Google image search

    Reply
  9. SEO Thailandon says

    at

    Yes Great Post
    But can you give me tips regarding how to use sitemap effectively

    Reply
  10. John Doeon says

    at

    So many people with so many ideas!!! I like the ideas in general to block robots to avoid duplicate contents. But in my opinion duplicate contents should be avoided at the url level. Dont allow your cms to generate more then one url of the same post.

    Reply
  11. Danielon says

    at

    Derek, I see what you are asking now.

    Reply
  12. Derekon says

    at

    Thanks Daniel I know that the robots.txt file should still go in the root, I just want to know if I have to change the robots file to look inside my new blog installation directory
    like
    disallow: /blogfolder/wp-admin

    Reply
  13. Danielon says

    at

    If I am not wrong, even if the blog in on a sub-folder, the robots.txt file should still go on the root directory since it is the first thing a search bot will look for.

    Reply
  14. derekon says

    at

    I was curious, my blog isnt installed in the root but instead a folder on my side.

    Reply
  15. Scibizon says

    at

    The particual robot.txt file is an important choice from the SEO point of view. Thanks for the original approach to the problem!

    Reply
  16. John T. Pratton says

    at

    you can’t upload robots.txt to the root directory using WordPress – it doesn’t have an FTP function. The only way it would be possible is if someone associated “robots.txt” as a theme associated file, and you could edit and save it in “theme editor” under the “presentation” tab. You wouldn’t think it would be too difficult to associate a file with a theme by modifying a little code, but presently I don’t know how to do it.

    Reply
  17. rtempon says

    at

    How do I upload the robots.txt to the root directory using WordPress?

    Reply
  18. Danielon says

    at

    John, regarding the first question: not all search bots recognize the * attribute. Some people argue that the Google Bot specifically does not interpret the * as a joker, it just ignores it. That is why I avoid using it. Plus, it should not be necessary to add it before a folder like */feed/.

    Reply
  19. John T. Pratton says

    at

    Thanks for the post, very helpful. Is there any problem with doing it like this:
    Disallow: */feed/
    Disallow: */trackback/

    Reply
  20. SEOon says

    at

    Great article mate. I’m still unsure what needs to be excluded though. But there is some stuff that needs to be blocked to avoid duplicate content. Cheers

    Reply
  21. Danielon says

    at

    Ajay /trackback/ will disallow all the trackback pages, and I also think we should disallow comments, but I am not sure if /comment/ is the right attribute for that.

    Reply
  22. Ajayon says

    at

    Daniel, that is not the case, rules of robots.txt are always followed no matter where the index is followed from.

    Reply
  23. engtechon says

    at

    One other thing to consider blocking is any duplicate content on your side. WordPress gives you about three thousand ways to access content (/page, /tag, direct links, etc). Blocking some of them might be a good idea.

    Reply
  24. Danielon says

    at

    Bes, if I am not the wrong the Google Image Bot will not need to crawl your image folder at all. It will crawl your pages, and it will index all the images on those pages (i.e. posts).

    Reply
  25. Bes Zon says

    at

    Thilak, unless I am mistaken, wouldn’t the Google Image Bot be following the rules of the robots.txt file regardless of where it starts crawling from?

    Reply
  26. Thilakon says

    at

    I guess disalowing “wp-” will not affect Google Image bot from crawling your images because it crawls them from the post and not from the directory

    Reply
  27. Dawud Miracleon says

    at

    Nice post. Great reminder of how you can easily protect folders and files on your server.

    Reply
  28. Danielon says

    at

    I am not sure if the GoogleImage bot tracks down images from the /images/ folder or directly from the posts where the images where inserted.

    Reply
  29. egonon says

    at

    Same with WordPress. If you disallow “/wp-” then it’s not going to index any of your uploads like images since they are in your wp-content folder. I get quite a bit of traffic from Google Image Search.

    Reply
  30. Mac Utopiaon says

    at

    I agree that robots.txt files are important, however I disagree with blocking the images folder. Some sites achieve some pretty good traffic numbers from google image search, blocking this directory will block your images from showing up in these results.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Popular

  • 28 Ways to Make Money with Your Website 514 Comments
  • 43 Web Design Mistakes You Should Avoid 474 Comments
  • 4 Steps to Increase Your Blog Traffic 188 Comments
  • How to Find Advertisers for Your Website 126 Comments
  • Top 25 SEO Blogs 243 Comments
  • 101 Blog Tips I learned 177 Comments
  • 30 Traffic Generation Tips 351 Comments
  • 6 Ways to Speed Up Your Site 51 Comments
  • 9 Hacks to Show Readers Your Best Stuff 21 Comments
  • 50 Simple Ways to Gain RSS Subscribers 145 Comments
  • The Best Website Taglines 127 Comments
  • What Is Success? 57 Comments
  • How to setup a 301 Redirect 52 Comments
  • The 7 Characteristics of Good Domains 76 Comments
  • 7 Ways to Promote your Site 133 Comments
  • Top 25 Celebrity Blogs 44 Comments
  • What Is A Blog? 81 Comments
  • Blog Setup: 40 Practical Tips 57 Comments
  • 10 Tips To Write Your Most Popular Post 79 Comments
  • The Blog Post Checklist 75 Comments
  • Interview: 12 Top Online Entrepreneurs 98 Comments
  • What Is Bounce Rate? 42 Comments
  • 20 SEO Terms You Should Know 28 Comments
  • How To Choose A Blog Name 15 Comments
  • 10 Tips to Sell Your Website on Flippa 37 Comments
  • Top 25 Web Design Blogs 80 Comments
As featured
CNBC copyblogger problogger Entrepreneur Lifehacker Hubspot Business Insider Wordpress Showcase
Online business done right. Take your marketing efforts to the next level with tips and resources to get visitors to your website.

All Content Copyright © 2023-2023. Daily Blog Tips unless otherwise noted or credited.

DBT is an independent website. The views expressed on this site may come from individual contributors and do not necessarily reflect the view of DBT or any other organization.

Privacy Policy| Terms of Service | About | Contact

777 Brickell Ave #500-14648, Miami, FL, 33131, US