Discussion Forum

jesse.glaves

Member

Registered: June 28, 2014

Posts: 124

Robots Tag Preventing Google Custom Search Index

Hi there. My site pages started being excluded from paid Google Custom Search results a while back. We're still in development, so it's not a huge deal yet, as we'll be switching domains at launch, and will have to reindex at that point anyway. It was working perfectly for quite some time. I emailed Google to see if the problem was on our end or theirs, and they responded with a curl request that shows that our site is returning a X-Robots-Tag: noindex, nofollow. This robots tag is not in any of the template files in the site, which leads me to believe that Bigtree must be introducing that information in the header response for each page somewhere. Any idea what might have changed? Below is the curl response.

curl -I http://bigtree.......edu/student_life/campus_safety/
HTTP/1.1 200 OK
Date: Tue, 25 Nov 2014 21:04:14 GMT
Server: Apache/2.2.15 (Red Hat)
X-Powered-By: PHP/5.3.3
Set-Cookie: PHPSESSID=oa6ttgtm524bovaqbtpv0q71h5; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Permitted-Cross-Domain-Policies: master-only
Connection: keep-alive, close
X-Robots-Tag: noindex, nofollow
Content-Type: text/html; charset=UTF-8

Last edited by jesse.glaves (November 25, 2014 4:14pm)

timbuckingham

Administrator

From: Baltimore, MD

Registered: April 2, 2012

Posts: 974

Re: Robots Tag Preventing Google Custom Search Index

That's strange -- BigTree should never pass the "nofollow" header except for in the admin (where it's actually a meta tag, still wouldn't be in your headers). The only time BigTree passes any X-Robots-Tag header at all is when you select the "Hide From Search Engines" checkbox in the SEO tab when editing a page, and that only passes "noindex" (not "nofollow").

Any chance you have a custom .htaccess setup or a vhost policy setting the headers?

jesse.glaves

Member

Registered: June 28, 2014

Posts: 124

Re: Robots Tag Preventing Google Custom Search Index

Looks like something got set in the .htaccess file, along with some other headers to secure the site. Not sure why the robots entry was in there. Thanks Tim.

Header set X-Content-Type-Options "nosniff"
Header set X-XSS-Protection "1; mode=block"
Header set X-Permitted-Cross-Domain-Policies "master-only"
Header set Connection keep-alive
Header set X-Robots-Tag "noindex, nofollow"