How would you test a feature whose output is not fixed, such as an AI chatbot or a recommendation engine?

What the interviewer wants to hear

• Define quality criteria and acceptable ranges instead of exact outputs
• Build an evaluation set of representative and edge-case inputs
• Test properties: relevance, safety, format, no harmful or private data
• Use human review and automated checks or scoring
• Monitor in production and use A/B tests
• Track regressions across model or prompt versions

Model answer

Because the output changes, I test properties and quality levels rather than exact strings. First, with the product team, I define what good means: for a support chatbot, answers must be accurate, on-topic, polite, in the right language, and never reveal private data or make up refund policies. I build an evaluation set of a few hundred real and edge-case questions, including tricky and adversarial ones like prompt injection or abusive language. Each response is checked with automated rules, such as format and banned content, plus scoring by human reviewers or a judge model against a rubric, and we track the pass rate. Every new model or prompt version runs this set, so we catch regressions. I also test the fixed parts normally: APIs, fallbacks when the model times out, and handover to a human. In production, we monitor feedback and run A/B tests for impact.

More hard QA Engineer questions

• A critical bug reached production even though QA tested the release. How do you handle it and what do you change afterwards?
• Which quality metrics would you track for a product team, and which would you avoid?
• How would you test an integration with a third-party payment gateway that you do not control?
• Your team finds most bugs late, just before release. How would you shift quality earlier in the process?
• You join a company with no automation. How would you design a UI and API test automation framework that will still be healthy in three years?
• A product is built from 15 microservices owned by different teams. The shared end-to-end environment is slow and often broken. How would you design the testing strategy?
• You are not a penetration tester, but you are the only QA on the team. How do you include security testing in your everyday work?
• The full regression suite takes six hours, so the team releases only once a week. How would you get feedback down to under 30 minutes without losing confidence?
• How would you design the test strategy for releasing a mobile banking app on iOS and Android every two weeks?