Describe how you would use log file analysis on a large ecommerce site with millions of URLs.

Question

RED SAG · Accepted Answer

First I would get at least a few weeks of server or CDN logs and filter them to verified Googlebot, using reverse DNS or Google's IP ranges, because many bots fake the user agent. Then I would analyse which sections and page templates Googlebot crawls, how often, and which status codes it receives. On large ecommerce sites I usually find crawl waste: huge numbers of hits on filter and sort parameters, internal search pages, redirect chains or old URLs. Next I would compare the crawled URLs with our list of important, indexable pages, such as key categories and in-stock products, to find pages Google rarely or never visits. Fixes might include robots.txt rules for low-value parameters, better internal linking to deep pages, cleaner sitemaps, and fixing 5xx errors or slow responses. I would keep monitoring logs to confirm Googlebot shifts its attention to valuable pages.

Describe how you would use log file analysis on a large ecommerce site with millions of URLs.

What the interviewer wants to hear

Model answer

More hard SEO Specialist questions