How to optimize WordPress robots.txt file
Below is a example for robots.txt file which can effectively use on a WordPress website / blog. Just open notepad and add below contents to it and just save it as robots.txt. Once created upload it to root of your server. Below format will give a overall idea for optimizing robots.txt for WordPress and you have to edit it according your needs.
User-agent: * Allow: / Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content Disallow: /e/ Disallow: /show-error-* Disallow: /xmlrpc.php Disallow: /trackback/ Disallow: /comment-page- Allow: /wp-content/uploads/ User-agent: Mediapartners-Google Allow: / User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / Sitemap: http://yoursite.com/sitemap.xml |
Tags: SEO, Web Design Tips