How good are the answers in AI-led interviews?

Why answer quality is the right question

Open questions in employee surveys are known for short, general answers: “communication should improve”, “too many priorities”. A leadership team cannot act on them, because they do not say where, why or what it means in practice. The promise of an AI-led interview is the depth of an interview at the scale of a survey. That promise has to be measured, not assumed.

So we measured it. We took answers from real Insight Bridge rounds and scored each one on five quality criteria.

How the answers were scored

The data comes from 15 production rounds of Insight Bridge run between November 2025 and October 2026 in strategy execution, employee listening and leadership work. Together they contain more than 21,000 substantive open answers in five languages. We drew 40 answers at random from each round and analysed 570 of them. Background questions, scale ratings, closing summaries and bare yes-or-no answers were excluded.

Each answer was scored from 1 to 5 on five criteria:

• Relevance: does the answer address the question and what it set out to learn?
• Concreteness: does it describe specific experiences, actions or examples?
• Reasoning: does it explain how or why something happened or matters?
• Context: does it make the circumstances clear, such as unit, role, market or timing?
• Participant’s interpretation: does it convey what the issue means to the participant and how they judge it?

A criterion was marked not applicable when the question did not call for it. Not every question needs an example or an explanation. The scoring was done by an AI judge (Claude, by Anthropic), which had to justify every score with verbatim quotes from the answer. Each quote was checked mechanically against the answer text. Length was not to be rewarded, and agreeing with official company language was not to be penalized.

1. People answer the question and say what it means to them

Relevance was the strongest criterion: 84% of answers scored 4 or 5, and only 4% scored 1 or 2. Participants stay on the subject and answer what was asked rather than steering to something else.

They also take a position. In 60% of answers it was clear how the participant judges the issue: whether a change is working, whether a target feels realistic, what worries them. This is the part of an answer that a numeric rating flattens and a leadership team most needs to hear.

2. The first answer rarely explains why

Three criteria were clearly weaker: reasoning, concreteness and context. More than half of the first answers stated a view without explaining what lay behind it, and fewer than one in four gave a specific example.

This is not a weakness of the participants. It is how people answer an open question the first time, in a survey as much as in an interview. A first answer tells you what someone thinks. The why, the example and the circumstances usually come only when someone asks.

3. The follow-up is where the depth comes from

To measure what the follow-up adds, we scored 124 answers twice: the first answer on its own, and the first answer together with the reply to the follow-up.

The gains landed exactly where the first answers were weakest: context, reasoning and concreteness. Relevance and interpretation moved less, because they were already high.

Averages do not tell the whole story. Look at the answers that lacked a reason, an example or a sense of where the issue shows up. After the follow-up, more than half of them had the missing piece. When the participant answered the follow-up properly, more than four in five did.

Two things decide how much a follow-up adds. The first is the participant: when they answered the follow-up properly, the gain was about twice as large as after a short reply. The second is the starting point: the thinner the first answer, the more the follow-up added, while answers that were already comprehensive hardly changed. A second follow-up on the same topic roughly doubled the gain, but a third added nothing more.

This is the core difference between an AI-led interview and a survey. A survey takes the first answer as given. An interviewer, human or AI, can ask “what does that look like in your unit?” or “why do you think that happens?”, and the answer to that question is what a leadership team can act on.

4. The biggest lever: probing the thin answers

The data also shows where the interviewer can do better. Only 30% of thin first answers received a deepening follow-up, against 50% of mid-range and 44% of good answers.

Part of this is by design: an interviewer that presses every short answer starts to feel like an interrogation. But thin answers are exactly where a follow-up adds most, so this is the clearest lever in developing the interviewer further.

5. Longer is not better, but explanations take words

The judge was told not to reward length. Even so, longer answers were usually better reasoned, because explaining why takes more words than stating a view. Length alone did not guarantee quality, though: some answers of twelve words or fewer scored well. For the interviewer, the lesson is to ask for the explanation, not for a longer answer.

6. No sign of fatigue

A common worry about longer conversations is that answers get thinner towards the end. We did not see that: answers at the end of a conversation were on average as good as, or slightly better than, those at the start.

What this means for leadership teams

• You learn why, not just what. A survey tells you that people doubt a target. A follow-up tells you why they doubt it. When the interviewer asked, more than half of those who had not explained their view did so. That is the difference between knowing there is resistance and knowing what to fix.
• Expect findings you can locate. The follow-up lifted context, together with reasoning, the most: answers say in which unit, role or market an issue shows up. Ask for results at the level where the strategy is executed, within the anonymity limits.
• Treat participants’ own judgement as a signal of its own. In 60% of answers, participants said how they judge the issue: whether a change is working, whether a target feels realistic. That is evidence of belief and commitment, the hardest thing to see in strategy execution, and it should be reported on its own rather than averaged into a score.
• Narrow the scope at the start. When you agree what the leadership team needs to learn, fewer themes with room to go deeper give more than a long list of questions. A second follow-up on the same topic roughly doubled the gain.
• Ask to see the evidence. Every finding should be traceable to what people actually said. In this pilot every score is backed by a verbatim quote, and the same standard should apply to any finding presented to a board.

Limits of the pilot

This is a first, internal pilot, and the numbers should be read as such. All scores come from AI judges. A second, independent AI judge re-scored 100 of the answers, and its scores were very close to the first: none differed by more than one point. Agreement between two AI judges is still not the same as the judgement of human experts, so a blind set of 50 answers has been prepared for human raters as the next step.

A follow-up cannot make an answer worse, because the original answer is always included; the few cases where the score fell reflect normal variation between two scorings. Follow-ups were also not asked more often after weak answers, so that does not explain the gains. Short “don’t know” answers were left out before sampling, so the pilot says nothing about how often people decline to answer. No client names or answer texts are published here.

Where Insight Bridge fits

Insight Bridge is NAG’s AI platform for strategy execution. It runs AI-led conversations across a whole organization at once, with the speed of a survey and the depth of an interview, and every finding can be traced back to what participants actually said. This pilot is part of how we keep that claim honest: we measure the quality of the answers, publish what we find and improve the interviewer where the data says it falls short.