How to interview engineers who use AI coding agents

If the engineers on your team use an AI coding agent every day, a whiteboard algorithm question tells you almost nothing about whether a candidate can do the job. The skill that now matters is direction: framing an ambiguous problem, prompting an agent through it, checking what it produces, and knowing when to take the wheel back. To interview for that, put the candidate in a real workspace with a real agent, give them open-ended work, and evaluate how they drove it — not whether they could reproduce a data structure from memory.

Why traditional coding interviews miss this

Most loops still optimise for a version of the job that's receding. Three failure modes show up again and again:

• The "no AI" interview.Candidates are told not to use AI, then use it anyway — off camera, unmeasured, ungraded. You've selected for people willing to work around the rule, and learned nothing about how they use the tool.
• The AI-allowed but shallow interview. A chat sidebar bolted onto a LeetCode clone. You get a pass or fail and no view of how they got there.
• The misaligned rubric.The real work is to direct an agent, verify its output, catch its mistakes, and override it when it's confidently wrong. None of that appears in a timed algorithm puzzle.

The result is that you find out whether someone can actually work this way six months after the offer — after the ramp, after the team already reorganised around the assumption that they could.

What to actually measure

Shift the rubric from "did they produce correct code" to "how did they work with the agent." Concretely, look for:

• Problem framing. Do they clarify the ambiguity before prompting, or fire off a vague request and accept whatever comes back?
• Prompt quality and iteration. Are prompts specific, scoped, and informed by the codebase — and do they adjust course when the first attempt is off?
• Verification. Do they read the diff, run the code, add a test, or check an edge case — or do they trust a plausible-looking answer?
• Override judgment. When the agent goes down a bad path, do they notice, and do they intervene with a correction or a hand-written fix?
• Delegation boundaries. Do they know which parts to hand off and which to keep — architecture decisions, security-sensitive code, the tricky 10%?

How to structure the interview

1. Give a real workspace.A browser IDE with an agent already wired in — the same class of tool they'd use on the job. No local setup eating the first fifteen minutes.
2. Give an ambiguous problem. Open-ended work that rewards research and judgment: extend an unfamiliar codebase, debug a failure with an unclear cause, integrate something under-documented. Not a puzzle with one known answer.
3. Capture the whole session. Every prompt, tool call, and accepted or rejected suggestion — as structured events, not a screen recording someone has to scrub through later.
4. Score against evidence. Each rubric score should point back to a specific moment in the session, so the debrief opens that moment instead of arguing from memory.

Scoring it fairly

The honest move — the one that makes the result defensible to a hiring committee and to the candidate — is to grade the process on record, not just the artifact. A candidate who reached a working solution by accepting three lucky guesses should not out-score one who caught the agent's mistake, explained why it was wrong, and redirected it. That distinction is only visible if the session is captured and the rubric rewards it.

Calibrate with your own engineers first: have two or three of them run the same problem, and use their sessions to anchor what "strong direction" looks like before you evaluate a candidate against it.

Common objections

"This just helps people cheat."Unsupervised AI use in a "no AI" interview is the cheating. Making the agent part of the assessment and recording how it's used is the control, not the loophole.

"We won't be able to compare candidates." You will, if the rubric is consistent and every score is evidence-linked — arguably more reliably than comparing two whiteboard performances graded from memory.

"Our team doesn't work this way yet." Then scope the problem and weight the rubric to match how you work now, and revisit as that changes. The structure holds either way.

Where DeftBench fits

DeftBenchis built to run exactly this interview: candidates direct a real agent through an ambiguous problem in a live browser IDE, the session is captured as telemetry with human and agent actions attributed separately, and every score in the evaluation links back to the moment that produced it. If you're weighing it against an AI-assisted coding test, the DeftBench vs CoderPad comparison lays out what each one captures and scores. Or talk to us about a pilot.