Web Design | SEO | Website Development

Search Engine Optimisation, Magento & Wordpress Specialist [Cranleigh | Surrey]

301 a whole site but exclude the robots.txt file

By In Domain Migration On 29/07/2014

I  recently realised that I was being stupid when moving a site and 301 redirecting the whole domain!

The code I had in the  .htaccess file (on Apache) was also redirecting the robots.txt and subsequently there was a good chance that Google wouldn’t be taking it into consideration when it spidered the old site as it was redirected to the new site!

I only realised this when I tried to get Majestic SEO to report on the old sites… it wouldn’t work without a live robots.txt file so I had to fiqure out how to exclude it from the sitewide 301 redirected .htaccess.

After much testing I have combined it with some other useful techniques and believe I have sorted the .htaccess code for redirecting both the non www and www version of any old url to the relevant new url (with a www.) whilst also being able to exclude files like the robots.txt and the Majestic SEO verification file, so that you can esily verify the site to analyse backlinks via This code is also useful because you can’t necessarily verify a redirected site for Webmaster Tools with a meta tag or with analytics – so now you can keep the redirections on and still verify with the verification file system.

Let me know how you get on!

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_HOST} ^ [OR]
RewriteCond %{HTTP_HOST} ^
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{REQUEST_URI} !^/verificationfile\.txt$
RewriteCond %{REQUEST_URI} !^/verificationfile\.html$
RewriteRule ^(.*)$ "http\:\/\/www\.newdomain\.com\/$1" [R=301,L]

About the Author

Dan Vince

Yet another Web Development Professional

Related Posts

1 Comment

  • Michael Rodrigues 1 YEAR AGO

    Saved me hours of research!


Leave a comment