How to assess AI fluency in software engineers

To assess AI fluency in software engineers, watch them use an AI coding agent on a realistic, slightly ambiguous task and score four things: how they frame the problem for the agent, what they choose to delegate, how they verify what comes back, and when they override it. Self-reported usage, tool-name quizzes, and prompt-trivia questions measure familiarity, not fluency. Fluency only shows up in the work — so the assessment has to capture the work.

What is AI fluency for software engineers?

"AI fluency" has become a hiring requirement faster than it has become a definition. Zapier set a minimum AI fluency bar for every new hire and published a rubric that scores AI mindset, strategy, building, and accountability. In 2026, Business Insider reported that Google is piloting a code comprehension interview in which candidates work on existing code with Gemini and are assessed on "AI fluency, including prompt engineering, output validation and debugging skills."

Those are company-wide definitions. For an engineer, the term can be much sharper. An AI-fluent engineer reliably gets correct, maintainable software out of AI tools on problems that matter — including the problems where the AI is wrong. That breaks into four observable behaviours:

• Framing. Turning a vague requirement into an instruction an agent can act on: constraints, context, the definition of done.
• Delegation. Knowing which parts to hand off (boilerplate, a migration, a test scaffold) and which to keep (the data model, the tricky edge case, the security boundary).
• Verification.Reading diffs, running the code, writing or strengthening tests, and checking the agent's claims instead of trusting them.
• Override. Recognising when the agent has gone down the wrong path, stopping it, and either redirecting it or doing the work by hand.

AI fluency vs. AI literacy

Most of the confusion comes from mixing these up. AI literacy is knowing what the tools are and roughly how they work: which models exist, what a context window is, that LLMs hallucinate. AI fluency is using that knowledge under pressure to ship good work. A candidate can explain retrieval-augmented generation beautifully and still merge a 400-line diff they never read.

The industry hasn't settled this yet. In TestGorilla's State of Hiring for AI Fluency 2026 survey of roughly 2,000 US and UK hiring experts, 49% reported no internal agreement on what AI fluency should measure. It's a vendor survey, so weigh it accordingly, but it matches what you hear from hiring managers: everyone wants the skill and few can say how they'd recognise it.

Why most ways of measuring AI fluency in developers miss it

• Self-reports and résumé lines."Uses Copilot and Claude daily" tells you frequency, not quality. Heavy users include both the most and the least careful engineers.
• Multiple-choice AI quizzes.They test literacy. Knowing the right answer about prompt structure doesn't mean you check the output.
• Banning AI in the coding round, then asking about it. You observe the candidate without the tool and hear a story about the tool. Neither shows you the skill.
• Scoring only the final output. When an agent can produce a working solution in minutes, a passing submission says little about the human. Two candidates with the same final diff can have reached it in completely different ways — one by steering carefully, one by luck.

How to test AI fluency in a coding interview

The reliable approach is the obvious one: give the candidate the tools they would use on the job and watch the process, not just the result. Concretely:

1. Use a real codebase, not a blank file.Fluency shows up when the agent has to navigate existing code, conventions, and tests. Google's pilot uses existing code for the same reason.
2. Make the brief slightly ambiguous.If one prompt can solve the task, you're testing the model. Leave a requirement the candidate has to clarify, and an edge case the agent is likely to miss.
3. Plant a trap the agent will walk into.A misleading function name, a flaky test, a subtle data-integrity rule. Whether the candidate catches the agent's mistake is the single most informative moment in the session.
4. State the AI policy explicitly."Use the agent as you would at work" — otherwise cautious candidates underuse it and you end up scoring their nerve.
5. Capture the session, not just the submission.Prompts, agent actions, edits, test runs, and who did what. Without that record, scoring falls back to the interviewer's memory.
6. Close with a walkthrough. Ask the candidate to explain a piece of generated code they kept and one they rejected. Ownership is easy to fake in the doing and hard to fake in the explaining.

We've written more on session structure in how to interview engineers who use AI coding agents and, for the live variant with an interviewer present, in how to run an AI pair programming interview.

An AI fluency rubric for engineers

Score each dimension on evidence from the session, not on impression. A simple three-level scale is enough:

• Framing — Weak: pastes the brief verbatim. Solid: adds constraints and context. Strong: clarifies the ambiguity first, then gives the agent a scoped task with a definition of done.
• Delegation — Weak: hands everything off, or nothing. Solid: delegates routine work. Strong: delegates deliberately and keeps the risky decisions for themselves, and can say why.
• Verification — Weak: accepts output unread. Solid: runs the tests. Strong:reads the diff, questions the tests themselves, and checks the agent's claims against reality.
• Override — Weak: re-prompts the same instruction when output is wrong. Solid: rewrites the prompt with new information. Strong:notices the wrong path early, stops it, and takes over where that's faster.

Tie every score to a specific moment — a prompt, a rejected diff, a test run. A rubric that can't point at evidence is a vibe check with columns.

AI fluency interview questions (and their limits)

Behavioural questions are useful as a screen before the hands-on round, because candidates with no real agent experience struggle to answer them concretely:

• Tell me about a time an AI tool was confidently wrong. How did you notice?
• What do you never delegate to an AI agent, and why?
• How do you review a large AI-generated change before merging it?
• How has the way you use AI tools changed over the last six months?
• Walk me through a prompt that didn't work and what you changed.

Listen for specifics — a named bug, a concrete review habit, a boundary they hold. But treat answers as a filter, not a verdict. A practised story is easy to rehearse; the working session is not.

When this kind of assessment doesn't fit

Not every role needs the same bar. If the job is low-level systems work where AI tools are rarely used, or a role in an environment that prohibits AI tooling, weight fundamentals and test fluency lightly or not at all. And don't let fluency stand in for fundamentals: an engineer who can't reason about the code without the agent can't catch the agent's mistakes either. The strongest signal is both together.

Where DeftBench fits

DeftBenchis an AI-native coding interview platform built around exactly this kind of assessment. Candidates work in a browser IDE with a real coding agent; the session is captured as telemetry that attributes each action to the human or the agent; and the evaluation links every score to the moment that produced it, with a human making the final call. It's pre-GA. Other platforms now market AI-fluency assessment too — see how we compare in DeftBench vs HackerRank and DeftBench vs CoderPad. If you're rethinking how you assess engineers, get in touch.