DeftBench
Book a call
AI-native coding interviews

Hire engineers who can actually build with AI.

Run your first assessment freeWatch the demo

Runs in the browser. Nothing for candidates to install.

Candidate #4471 · Payments API: duplicate webhook charges
handler.ts
1import { db } from "../db";
2import { charge } from "../billing/charge";
3import { verifySignature } from "./verify";
4
5export async function handleStripeEvent(req: Request) {
6  const evt = await verifySignature(req);
7−  await charge(evt.customer, evt.amount);
7+  // Stripe retries deliveries: charge once per event id.
8+  const seen = await db.events.find(evt.id);
9+  if (seen) return ok();
10+
11+  await db.transaction(async (tx) => {
12+    await tx.events.insert({ id: evt.id });
13+    await charge(evt.customer, evt.amount);
14+  });
15  return ok();
16}
06:40
42:18
Evaluation4.1 / 5 · Strong hire
Design judgment4.5

Asked for two designs with tradeoffs before any code, then chose the transactional one and said why.

↳ replay at 06:40

Used by engineers at

  • Coursera
  • Precisely
  • Broperks
Product demo
The problem

Your interview process is testing for a version of the job that's already gone.

  • unmeasured

    Candidates are told “no AI,” then use it anyway: off camera, unmeasured, ungraded.

  • shallow

    The ones who are allowed to use AI get a chat sidebar bolted onto a LeetCode clone. You get a pass/fail, and nothing about how they got there.

  • misaligned

    Meanwhile the actual job is to direct an agent, verify its output, catch what it gets wrong, and know when to override it. None of that shows up in a whiteboard algorithm question.

You're left guessing. Six months later you find out whether the hire could actually work this way, after the offer, after the ramp, after the org chart already moved.

On the record

The people running engineering already moved on.

What CEOs and founders have said in public about how their engineers work now. Every quote links to where it was said.

  • On Y Combinator asking founders to attach a coding-agent session they're proud of:

    “I seriously think this is just the beginning of being able to find the real builders.”
    Garry TanPresident & CEO, Y CombinatorX · Feb 2026
  • “taste is a new core skill”
    Greg BrockmanPresident & Co-Founder, OpenAIX · Feb 2026

  • 100%

    of NVIDIA's engineers work with AI coders

    “Every one of our engineers — 100% — is now assisted by AI coders. … Our productivity has gone up incredibly.”
    Jensen HuangPresident & CEO, NVIDIACNBC Squawk Box · Oct 2025

  • 0

    lines written by hand since December, by Spotify's best developers

    “When I speak to my most senior engineers, the best developers we have, they actually say that they have not written a single line of code since December. They actually only generate code and supervise it.”
    Gustav SöderströmCo-CEO, SpotifyQ4 earnings call · Feb 2026

  • 1 week

    for every Coinbase engineer to onboard AI coding tools

    “Some of them had a good reason, because they were just getting back from some trip or something. Some of them didn't, and they got fired.”
    Brian ArmstrongCo-Founder & CEO, CoinbaseCheeky Pint podcast · Aug 2025
  • “The obvious tactical thing is just get really good at using AI tools. Like when I was graduating as a senior from high school, the obvious tactical thing was get really good at coding, and this is the new version of that.”
    Sam AltmanCEO, OpenAIStratechery interview · Mar 2025

Public statements, quoted verbatim and linked to the source. None of the people or companies quoted are affiliated with DeftBench or endorse it.

Photos, cropped, via Wikimedia Commons: Garry Tan by Web Summit (CC BY 2.0); Greg Brockman by Simulation (CC BY 3.0); Sam Altman by TechCrunch (CC BY 2.0); Jensen Huang by Prime Minister's Office, India (GODL-India); Brian Armstrong by TechCrunch (CC BY 2.0).

A hiring signal you can trust, not a vibe you have to defend in the debrief.

Imagine handing your hiring manager a report that says, with a timestamp and a replay link: this is exactly the moment the candidate caught the agent's bug, here's the prompt that fixed it, here's where they didn't verify and it bit them.

Evaluation report — candidate #4471Strong hire signal
  • Caught an unsafe default the agent introduced, and verified the fix with a test before merging.14:02
  • Redirected the agent after a vague first prompt produced the wrong abstraction.03:41
  • Shipped without running the test suite once. Flagged for review.21:15

That's a decision, not a transcript dump.

How it works

Five steps, one evidence trail.

  1. 01

    Give candidates a real workspace

    A full browser-based IDE with an AI coding agent already wired in, the same class of tool they'll use on day one. No install, under 60 seconds to spin up.

  2. 02

    Give them an ambiguous problem

    Open-ended engineering work of the kind that rewards judgment, research, and tool selection rather than memorized algorithms.

  3. 03

    Capture everything that matters

    Every prompt, tool call, MCP invocation, and accepted or rejected suggestion, captured as structured telemetry rather than a video you have to scrub through.

    {
  "event": "suggestion_rejected",
  "tool": "edit_file",
  "reason_inferred": "missing_input_validation",
  "t": "00:14:02"
}
  4. 04

    Get an evidence-linked evaluation

    Every score traces back to a specific, replayable moment in the session, so a debrief can open the moment instead of arguing about it.

  5. 05

    Decide faster, with confidence

    A recruiter dashboard built for comparing candidates at a glance, instead of re-reading raw transcripts one at a time.

Why this, not that

The comparison that matters.

How DeftBench compares with traditional and bolt-on AI coding tests
CapabilityTraditional coding testsBolt-on "AI-allowed" testsDeftBench
Tests the actual job
Measures how AI was used
Evidence-linked scoring
Setup timeHoursHours< 5 min
Report you can defend in a debrief
What you get

Every score, traceable to the moment it came from.

Evidence, not a rating
Every score links back to a timestamped moment in the session you can open and read for yourself.
The whole session, structured
Prompts, tool calls, MCP invocations, and each accepted or rejected diff are captured as queryable telemetry.
Isolated by construction
Every session runs in its own container, provisioned fresh for the candidate and torn down afterwards.
Replay over transcripts
Step back through how the work actually unfolded instead of re-reading a wall of chat logs.

We'll publish measured latency numbers, and named customer references, once pilots close.

Who it's for

Built for every stage of hiring.

AI-native startups
Fast setup, no process overhead, hire for the tools you already use.
Scaleups
Standardized rubrics and dashboards, so every interviewer's signal means the same thing.
Enterprises
SSO, audit logs, data residency, BYOK.
Recruiting agencies
Multi-tenant client workspaces, white-labeled reports.
Frequently asked

Questions, answered.

No — it's measuring the thing you actually care about: whether they can direct, verify, and override an agent. Unsupervised AI use in a "no AI" interview is the actual blind spot.
The onboarding flow is built to get you from signup to your first assessment sent in a single sitting — no install, no infrastructure work, no scheduling call.
That's fine — assessments are configurable per role, and the report shows you exactly where a candidate's process holds up and where it doesn't, regardless of your current process.

Give your next candidate a problem worth solving.

We're onboarding teams in small batches. Book a 20-minute call and we'll walk you through a pilot.

Book a meetingContact us