Skip to content
RED SAGSkills
easySEO Specialist interview question

What is the difference between robots.txt and a noindex tag?

What the interviewer wants to hear

  • robots.txt controls crawling, not indexing
  • Blocked URLs can still be indexed if linked
  • noindex controls indexing and must be crawlable to be seen
  • Use robots.txt for crawl control, such as filters
  • Use noindex for pages that should not appear in results
  • Neither is security

Model answer

robots.txt tells crawlers which URLs they should not crawl. It is mainly for managing crawling, for example stopping bots from wasting time on endless filter URLs. But it does not control indexing. If other pages link to a blocked URL, Google can still index it, usually without a description. A noindex tag, in the meta robots tag or an X-Robots-Tag header, tells Google not to show the page in results. For it to work, Google must be able to crawl the page and see the tag, so you should not block a noindexed page in robots.txt. I use noindex for things like thank-you pages or thin internal search results, and robots.txt for crawl control. Neither is a security tool, so private content needs a password or proper access control.

Can you answer it under time pressure?

Take a timed easy round and get scored on every answer.

Start mock interview

More easy SEO Specialist questions