Skip to content
RED SAGSkills
hardAutomation Test Engineer interview question

Your product adds an AI assistant built on a large language model. Its answers vary between runs. How would you automate testing for it?

What the interviewer wants to hear

  • Test deterministic parts normally: prompts building, tools, APIs, UI
  • Use evaluation sets with rubric or property checks, not exact matches
  • LLM-as-judge or human review with care and calibration
  • Track pass rates over many runs and set thresholds
  • Safety, prompt injection and regression on model changes

Model answer

I separate what is deterministic from what is not. Prompt building, tool calls, API handling, error states and the UI get normal tests, and in UI tests I mock the model so they are stable. For the model behaviour itself I build an evaluation set of realistic questions with expected properties instead of exact text: the answer mentions the refund policy, cites a source, stays under a length, uses the right tool, or refuses unsafe requests. Checks combine code-based rules with an LLM-as-judge using a clear rubric, calibrated against human ratings. Because outputs vary, I run each case several times and track pass rates against a threshold rather than single pass or fail. The eval suite runs whenever prompts or model versions change, and it includes prompt injection and data leakage cases so safety regressions are caught before release.

Can you answer it under time pressure?

Take a timed hard round and get scored on every answer.

Start mock interview

More hard Automation Test Engineer questions