AI vs Human Vishing: Who Gets People to Comply?
AI vs Human Vishing: About Half Complied Either Way
A client asked us a direct question: is a live human caller more dangerous than an AI voice agent running the same pretext? We ran an internal IT support pretext both ways and scored both on the same OSES™ ladder. Once a person was in the conversation, about half complied either way.
Human or AI, about half of the people who talked to the caller complied. What separated the two campaigns was who picked up.
Is an AI vishing agent as persuasive as a human caller?
We could not tell them apart. On the same internal IT support pretext, 7 of 13 people who talked to a live human caller complied, and 20 of 40 who talked to our AI voice agent complied: about half either way. Complied means agreed on the call. The real gap was how many people picked up.
People who conversed with the live human caller and complied
People who conversed with the AI voice agent and complied
Human-call compliers who then completed the remote support workflow, in about 7 and 8 minutes
Is a skilled human on the phone a bigger threat than an autonomous agent that never gets tired? Most of the debate about AI social engineering answers that with opinion. We answered it with two scored campaigns. The headline: once someone was on the line, we found no measurable difference between a live human caller and our AI voice agent. Whether the caller was a person or a model did not show up in the result.
That finding sits squarely on what we have seen across our AI vishing red team field findings: an expected, routine request from a plausible internal function is enough. The pretext needed no deepfake and no human.
How we ran the comparison
Both campaigns used the same pretext, internal IT support, delivered by voice. Both were scored under one rule on the OSES™ ladder, so the numbers mean the same thing on both sides of the table.
| Field | Human operator | AI operator |
|---|---|---|
| Operator | Live human caller, no AI voice | Autonomous conversational AI voice agent |
| Pretext | Internal IT support | Internal IT support |
| Vector | Outbound voice call | Outbound call, voicemail drop, inbound callback |
| Follow-on | Email with a remote support link, sent only to people reached live | None |
| The ask | Agree to a remote support session | Comply with a credential verification request |
Source: Breacher.ai OSES™ engagement data. Organizations, sectors and dates withheld.
The scoring rule was identical for both. Complying with the caller's request in conversation counts as L4 Action, and verbal compliance counts. Conversing without complying (refusing, verifying, or deferring) counts as L3 Engaged. L5 Escalation is reserved for a consequence that actually followed, and downstream activity was observed in both campaigns. No credential material was captured in either. In plain terms, complied means the person agreed on the call. It does not mean credentials changed hands.
Results: no measurable difference in persuasion
| Metric | Human operator | AI operator |
|---|---|---|
| Reached a live conversation | 13 of 37 | 40 of approx. 400 |
| Complied, over all targets | 7 of 37 | 20 of approx. 400 |
| Complied, of those who conversed | 7 of 13 (about half) | 20 of 40 (half) |
| Escalation (L5) | 0 of 7 compliant | 0 of 20 compliant |
| DEPTH | 45 | 45 |
| SPREAD | 30.0 (capped) | 8.1 |
| OSES™ SCORE | 75.0, High | 53.1, Medium |
Fisher exact test on compliance among those who conversed: p = 1.0, no statistical difference.
Read the third row first. Once someone was talking to the caller, the human got 7 of 13 to comply and the AI voice agent got 20 of 40. The test cannot tell those two results apart. That is not the same as proving they are identical: with only 13 human conversations, the human result is consistent with anything from roughly 3 in 10 to 3 in 4. So we will not publish a precise human-versus-AI rate as a benchmark. What the data supports is the plain version: about half of the people who talked to the caller complied, whether the caller was a person or our AI voice agent, on asks that were similar but not identical.
The score gap came from who picked up, not from what happened on the call.
Why the human campaign scored higher
The human campaign scored 75.0 (High) and the AI campaign scored 53.1 (Medium). Every point of that gap sits in SPREAD. DEPTH was identical at 45, because in both campaigns someone complied. SPREAD measures how much of the population went with the caller, and the human campaign got more people on the phone: 13 of 37 targets reached a live conversation, against 40 of approx. 400 for the AI campaign.
That looks like a human advantage until you check it against a second AI-operated campaign on the same IT support pretext, which reached a live person 46 of 126 times, close to the human operator's 13 of 37. One cross-check does not settle it, but it points at the target list and the organization rather than the caller. A hand-picked shortlist picks up differently from a full roster.
The human scored higher, so humans are the bigger threat. This reads a reach difference, driven by who was on the list, as a persuasion difference.
No measurable difference in conversation. The reach gap tracked the population, and a second AI campaign picked up at close to the human rate.
What the people who held did
The most useful part of the human campaign is not the 7 who complied. It is the 5 of 13 who independently verified the caller and refused, plus 1 who deferred to another channel. There was nothing synthetic for them to catch: the caller was a real person. What stopped those calls was a verification step, not a verdict on the voice.
That is the behavior the Control Invariance Thesis says to measure. A verification step does not read the voice, so it returns the same answer whether the caller is human, cloned or generated. What a more convincing caller changes is the pressure to skip the step, which is why we measure how often the step is skipped. It is why we argue for training the procedure and then testing it rather than asking people to judge how real a caller sounds.
The other side of the ledger shows why the procedure matters. Of the 7 who complied on the call, 2 went on to complete the remote support workflow, in about 7 and 8 minutes from the email arriving. Minutes, not hours. A control that depends on someone noticing later is too slow for that window.
What this means for your program
- Stop treating AI voice risk as a fidelity problem. The pretext needs no deepfake and no human. An expected, routine request is enough, so plan for plausibility, not realism.
- Bind consequential actions to verification. Remote support sessions and credential requests should require a callback to a known-good number, or another channel the caller does not control, before anyone acts.
- Measure reach and persuasion separately. DEPTH shows how far a caller got. SPREAD shows how many went with them. Reading one as the other is how the wrong conclusion gets drawn.
What this comparison does and does not show
We publish the limits alongside the result, because a score you cannot defend is not worth having.
A natural comparison, not a controlled experiment
Different organizations, different populations, run weeks apart. One campaign used a hand-picked shortlist and the other a full roster. That is why this result cannot say AI is more dangerous than humans, or the reverse.
Different asks under one rule
Agreeing to a remote session is not the same act as complying with a credential verification request. Both score L4 under one rule, but they are not identical, so treat the side-by-side as indicative.
The AI had an extra path
The AI voice agent could leave a voicemail and take the inbound callback. The human could not, and some of the 40 AI conversations came through that path. The next step that turns this into a causal result is a random split of one shortlist between a human operator and the AI agent, same pretext, same week, same ask.
How Breacher.ai runs AI vs human vishing tests
Breacher.ai runs orchestrated social engineering simulations under OSES™. Our autonomous AI voice agent places outbound calls, leaves voicemail, and handles the inbound callback live, so it tests the paths a real caller would use. When a client asked for a direct comparison, we ran a human operator against the same pretext and scored both on the same ladder. Findings are tied to the verification procedure that held or failed, not to the people who answered.
See how it works on the vishing simulation page, or explore the full simulation platform.
Measure your risk.
Train for what you find.
Prove it changed.
Find out whether your help desk verification procedure holds against a caller that never tires.
Book Your DemoFrequently asked questions
We could not tell them apart. On the same internal IT support pretext, 7 of 13 people who talked to a live human caller complied, and 20 of 40 who talked to our AI voice agent complied: about half either way. Complied means agreed on the call. The real gap was how many people picked up.
Because more people picked up. Both campaigns had the same DEPTH of 45, since someone complied in each. The human campaign reached a live conversation with 13 of 37 targets, against 40 of approx. 400 for the AI campaign. A second AI campaign reached 46 of 126, close to the human caller's pickup rate.
No. It compares two campaigns at two organizations, run weeks apart, with different asks. What it supports is narrower: in conversation, we found no measurable difference between an AI voice agent and a live human caller on the same IT support pretext. A random split of one target list is the next step toward a causal claim.
Test the procedure, not the voice
In our comparison, human or AI, about half of the people who talked to the caller complied. Find out whether your verification procedure stops the rest.
Book Your Demo
