Digital Marketing Glossary
X-Robots-Tag
The X-Robots-Tag is an HTTP response header that gives search engines indexing instructions, such as noindex or nofollow, for a URL. It works like the robots meta tag but is sent by the server, so it can apply to non-HTML files such as PDFs and images. Crawlers must be able to fetch the URL to see it.
Fire Source Media Editorial Team Updated
Why X-Robots-Tag Matters
It lets site owners keep files like internal PDFs out of search results where an HTML meta tag cannot be used.
Misconfigured headers can accidentally remove important pages from search, so it is a key check in technical SEO audits.
How It Works
The server adds a header such as 'X-Robots-Tag: noindex' to responses. Rules can target specific crawlers or file types.
Do not block the URL in robots.txt if you need the noindex header seen; blocked URLs cannot be fetched. Check headers with browser developer tools or crawlers.
X-Robots-Tag Example
Hypothetical example: a company's downloadable price sheets appear in search results. Adding a noindex X-Robots-Tag to those PDFs removes them over time once recrawled.
X-Robots-Tag vs. Robots Meta Tag
The robots meta tag sits in an HTML page's head. X-Robots-Tag is a header and works for any file type.
Common Mistakes
A common mistake is leaving a staging site's noindex header active after launch.
Putting It Into Practice
Check response headers on key pages after every deployment or hosting change.
Use the header deliberately for files that should not appear in search. Explore SEO services for technical reviews.
Frequently Asked Questions
Does X-Robots-Tag stop crawling?
No. It controls indexing; robots.txt controls crawling.
Can it apply to images?
Yes, it works on any file the server delivers.