hard20 pts · first answer countstechnical-seorobots-txtcrawling

When robots.txt returns 5xx

For several days, https://example.in/robots.txt returns HTTP 503. How does Google treat crawling of the site?

A.It treats the site as having no rules and crawls it all at onceB.It removes the whole site from the index at onceC.It ignores robots.txt and only crawls the sitemapD.It pauses crawling, then may use a cached copy later

Discussion 0

Join free to post in the discussion.

When robots.txt returns 5xx

Discussion 0

Related questions