X

Good robots.txt

This is a good example of a robots.txt to use with your wordpress installation. Bellow the first example is one from smashing magazine. Yes yes I know, I know, not the best robot drawing in the world! have a better one? send it along ill replace my robot with yours.



Code Snippet:


User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes


Code Snippet:


Sitemap: http://smashingmagazine.com/sitemap.xml

User-agent: *
Disallow: /cc/
Disallow: /wp-admin/
Disallow: /tag/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php
Disallow: /*?
Disallow: /*comment-page

User-agent: MSIECrawler
Disallow: /

User-agent: psbot
Disallow: /

User-agent: Fasterfox
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: SiteSucker
Disallow: /

Comments  Leave a Reply

Add a Comment

We're glad you have chosen to leave a comment. Please keep in mind that all comments are moderated according to our privacy policy, and all links are nofollow. Do NOT use keywords in the name field. Let's have a personal and meaningful conversation.

WordPress Launch Checklist

The Ultimate WordPress Launch Checklist

We've compiled all the essential checklist items for your next WordPress website launch into one handy ebook.
Yes, Send Me the Free eBook!