X

Good robots.txt



Code Snippet:


User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes


Code Snippet:


Sitemap: http://smashingmagazine.com/sitemap.xml

User-agent: *
Disallow: /cc/
Disallow: /wp-admin/
Disallow: /tag/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php
Disallow: /*?
Disallow: /*comment-page

User-agent: MSIECrawler
Disallow: /

User-agent: psbot
Disallow: /

User-agent: Fasterfox
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: SiteSucker
Disallow: /

Comments  Leave a Reply

Add a Comment

We're glad you have chosen to leave a comment. Please keep in mind that all comments are moderated according to our privacy policy, and all links are nofollow. Do NOT use keywords in the name field. Let's have a personal and meaningful conversation.

WordPress Launch Checklist

The Ultimate WordPress Launch Checklist

We've compiled all the essential checklist items for your next WordPress website launch into one handy ebook.
Yes, Send Me the Free eBook!