X

How to Write a Good robots.txt File

Snippets by IsItWP

Are you looking for a way to replace keywords within the_content and the_excerpt using an associative array? While there’s probably a plugin for this, we have created a quick code snippet that you can use to replace keywords in the_content and the_excerpt in WordPress.

This is a good example of a robots.txt to use with your WordPress installation.

Instructions:

All you have to do is add this code to your theme’s robot.txt file.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Sitemap: http://smashingmagazine.com/sitemap.xml

User-agent: *
Disallow: /cc/
Disallow: /wp-admin/
Disallow: /tag/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php
Disallow: /*?
Disallow: /*comment-page

User-agent: MSIECrawler
Disallow: /

User-agent: psbot
Disallow: /

User-agent: Fasterfox
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: SiteSucker
Disallow: /

Note: If this is your first time adding code snippets in WordPress, then please refer to our guide on how to properly copy / paste code snippets in WordPress, so you don’t accidentally break your site.

If you liked this code snippet, please consider checking out our other articles on the site like: 24 best WordPress education themes and how to create a guest post submission form in WordPress.

Comments   Leave a Reply

  1. can you please give me a simple robot txt

    1. You can simply copy the above code to a .txt file and upload it to your site directory.

Add a Comment

We're glad you have chosen to leave a comment. Please keep in mind that all comments are moderated according to our privacy policy, and all links are nofollow. Do NOT use keywords in the name field. Let's have a personal and meaningful conversation.

WordPress Launch Checklist

The Ultimate WordPress Launch Checklist

We've compiled all the essential checklist items for your next WordPress website launch into one handy ebook.
Yes, Send Me the Free eBook!