Which quality metrics would you track for a product team, and which would you avoid?
What the interviewer wants to hear
- Escaped defects and production incidents
- Defect severity trends and time to fix
- Test pass rates and flaky test rate
- Change failure rate and mean time to recovery
- Avoid vanity metrics like number of test cases or bugs per tester
- Use metrics to learn, not to punish
Model answer
I prefer metrics that reflect real customer impact and help the team improve. Escaped defects, meaning bugs found in production, broken down by severity, show how well we catch issues before release. Change failure rate and mean time to recovery, from DORA metrics, show release health. I track time to fix critical bugs, and in automation the flaky test rate and pipeline duration, because slow or unreliable feedback hurts everyone. Customer-facing signals like crash-free sessions and support tickets per release are also valuable. I avoid vanity metrics such as number of test cases written, bugs logged per tester or raw coverage percentage, because they are easy to game and can push people to log trivial bugs or write useless tests. I look at trends over time rather than single numbers, and use them in retrospectives to find improvements, never to blame individuals.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard QA Engineer questions
- A critical bug reached production even though QA tested the release. How do you handle it and what do you change afterwards?
- How would you test an integration with a third-party payment gateway that you do not control?
- Your team finds most bugs late, just before release. How would you shift quality earlier in the process?
- How would you test a feature whose output is not fixed, such as an AI chatbot or a recommendation engine?
- You join a company with no automation. How would you design a UI and API test automation framework that will still be healthy in three years?
- A product is built from 15 microservices owned by different teams. The shared end-to-end environment is slow and often broken. How would you design the testing strategy?
- You are not a penetration tester, but you are the only QA on the team. How do you include security testing in your everyday work?
- The full regression suite takes six hours, so the team releases only once a week. How would you get feedback down to under 30 minutes without losing confidence?
- How would you design the test strategy for releasing a mobile banking app on iOS and Android every two weeks?