What is toil, and how would you reduce it on your team?

What the interviewer wants to hear

• Definition: manual, repetitive, automatable, no lasting value, scales with load
• Measure toil and set a cap such as 50%
• Automate the most frequent or painful tasks first
• Fix root causes instead of repeating fixes
• Self-service tools for other teams

Model answer

Toil is operational work that is manual, repetitive, can be automated, has no lasting value and grows as the service grows, such as restarting a stuck job every night or creating user accounts by hand. Some toil is normal, but too much burns people out and stops engineering progress, so many SRE teams cap it at around 50% of their time. To reduce it, I first measure it, for example by tagging tickets and pages. Then I rank tasks by frequency and pain and automate the top ones, or better, fix the root cause so the task disappears. I also build self-service tools so other teams do not need to file tickets for routine requests.

More easy Site Reliability Engineer questions

• Explain the difference between an SLI, an SLO and an SLA, with an example.
• What is an error budget and how does a team use it?
• What are the four golden signals, and why are they useful for monitoring?
• You are on call and get paged for a high error rate. What are your first steps?
• What is the difference between monitoring and observability, and what are metrics, logs and traces each good for?
• Why do SRE teams manage infrastructure with code tools such as Terraform instead of clicking in a cloud console?
• Explain blue-green and canary deployments. When would you choose each?
• You are asked to build the main Grafana dashboard for a web API. What would you put on it, and how would you lay it out?
• What makes an on-call rotation healthy and sustainable for the engineers on it?