For years, awareness training gave employees the same advice: look for the typo. Spelling mistakes, clumsy phrasing, and generic greetings were the tells that gave phishing away. That advice is aging fast.
In a controlled study by a team that included security expert Bruce Schneier, fully AI-automated spear-phishing emails earned a 54% click-through rate. That matched emails written by human experts and was 4.5 times the 12% rate of a control group. The study was small, with 101 participants, but its direction matches what investigators report in the field. This is what AI social engineering looks like in practice: cheap, fluent, and personal.
-
What Is AI-Powered Social Engineering?
AI-powered social engineering is the use of generative AI to research targets, write convincing messages, and imitate voices or faces in order to manipulate people into handing over money, access, or information. It targets the person rather than the software, and AI makes it cheaper, more personalized, and harder to spot.
-
Why "Spot the Typo" Training Is Losing Its Edge
Three things changed at once.
Fluency. Generative AI writes clean, on-brand prose in any language, so the grammar tell is gone.
Personalization. In the same study, the automated tool built accurate target profiles in 88% of cases, and the authors estimate that automation can raise attacker profitability by up to 50 times for large campaigns.
Relevance. This matters most. In a randomized trial at UC San Diego Health covering 19,500 employees, poorly worded lures drew few clicks, but emails about topics employees cared about, such as vacation or dress code policies, saw click rates soar. AI makes it cheap to get both the wording and the topic right.
That same trial found that training barely moved the needle. Embedded training reduced failure rates by only about 1.7% on average, completing annual training had no significant relationship with falling for a phish, and most people don't engage with the materials. It was a single health system, so treat it as a caution rather than a verdict. It still argues for defenses that don't depend on every employee spotting every lure.
| Old advice | Why it is weaker now | What works instead |
|---|---|---|
| Look for spelling and grammar errors | AI writes fluent, on-brand text | Verify the request, not the writing |
| Check for a generic greeting | Automated research personalizes at scale | Treat urgency plus money or access as the trigger |
| Trust a familiar voice or face | Both can be faked, even live | Call back on a known number; split approvals |
| Complete annual training | Little measurable effect in the UCSD trial | Rehearse the verification workflow |
-
Beyond Email: Voice, Text, and Video
Email is no longer the only front. Verizon's 2026 DBIR found phishers having more success reaching users by voice and text message than by email. Pretexting, where an attacker invents a scenario to manipulate a target, has also become a more common way into ransomware and extortion attacks. Groups such as Lapsus$ and ShinyHunters often work by phone.
The sharpest example is still Arup. In early 2024, a finance worker at the engineering firm's Hong Kong office joined a video call with people who looked and sounded like the CFO and other colleagues. According to Hong Kong police, all of them were deepfake re-creations. The worker put aside his doubts and sent HK$200 million, about $25.6 million, across 15 transactions.
The worker recognized the faces. That is the lesson: recognition is not a control. A process that lets one person's belief release money will eventually lose to a convincing enough fake.
-
How Big Is the Problem?
The FBI's 2025 Internet Crime Report logged more than 1 million complaints and $20.877 billion in losses, up 26% year over year. Business email compromise accounted for roughly $3 billion of that. The report also includes a dedicated AI section, with more than 22,000 AI-related complaints and nearly $900 million in losses.
Treat that figure as a floor. One analysis notes that only about $30 million of BEC losses carried an AI flag, a small slice of $3 billion. That likely understates AI's role, since victims often can't tell whether AI was involved.
-
What Actually Works Against AI-Enhanced Social Engineering
-
Verify the request, not the writing. For any payment change, urgent wire, or credential reset, confirm through a second channel using a number you already have on file, never the one in the message.
-
Split the approval. Require two people to approve transfers above a threshold, and add a delay for new payees, so no single call, voice, or face can release money.
-
Harden the help desk. Vishing often targets support staff for resets. Require identity verification before resetting passwords or enrolling MFA devices. Talos recommends help desk verification for MFA enrollment for this reason.
-
Make a stolen answer worthless. CISA rates phishing-resistant MFA as the strongest form of MFA. With FIDO/WebAuthn, even a flawless lure can't hand over a reusable code.
-
Make reporting easy and blameless, and track how quickly people report, not only who clicked. One fast report can protect everyone else.
AI can help on defense too. In the same research, Claude 3.5 Sonnet identified phishing emails with 97.25% accuracy and zero false positives across 381 emails when prompted to be suspicious. That is a lab result from one model and one dataset, not a product guarantee. It is a good reason to test AI-assisted email triage in your own environment.
-
Common Mistakes
-
Teaching typo-spotting as the main defense.
-
Treating a video call as proof of identity.
-
Running annual training and calling the job done.
-
Letting one person approve large transfers.
-
Measuring clicks without measuring reports.
Practice the Reflex Before You Need It
Reading about verification is easy. Doing it when the "CFO" is on a live call is not. Simulations Labs' Cyber Range lets security teams run live-fire exercises, so you can turn a scenario like the Arup call into a scored drill and see whether people verify, escalate, and freeze the payment, or wire first and ask later. For a recurring program, security team upskilling tracks help you repeat and measure it. The scenario library also includes OSINT challenges, which show analysts how much of a convincing target profile can be built from public information, the same reconnaissance AI now automates.
Want your team to rehearse a payment-fraud or help-desk vishing scenario before a real one arrives? Explore the Simulations Labs Cyber Range or request a demo.
-
Frequently Asked Questions
-
FAQ 1: How is AI changing phishing?
-
AI removes the old tells. It writes fluent, on-brand messages, automates research to personalize them, and extends the same tactics to voice and video. A controlled study found fully AI-automated spear phishing matched human experts at a 54% click-through rate.
-
FAQ 2: Can AI-generated phishing emails really beat human experts?
In one controlled study of 101 participants, fully automated AI emails matched human experts at 54% click-through, against 12% for a control group. The sample was small, so the result is directional, but it shows AI can reach expert-level quality at far lower cost.
-
FAQ 3: Does phishing awareness training still work?
Less than most programs assume. In a trial of 19,500 employees at UC San Diego Health, embedded training reduced failure rates by only about 1.7% on average, and completing annual training showed no significant relationship with falling for a phish. It was one health system, so it is a caution rather than proof. Training works best alongside process controls such as verification and split approvals.
-
FAQ 4: What is deepfake CEO fraud, and how can it be stopped?
It is fraud in which attackers use cloned voices or faces to impersonate an executive and request money or access. In the Arup case, a finance worker sent about $25.6 million after a video call with deepfaked colleagues. Stop it with process, not perception: out-of-band callbacks on known numbers, two-person approval above a threshold, and delays for new payees.
-
FAQ 5: How can you tell if an email was written by AI?
Often you can't, and that is the point. Instead of judging the writing, judge the request. Be cautious when a message pairs urgency with a request for money, credentials, or an MFA reset, and verify it through a second channel on a number you already have.
-
FAQ 6: How much is AI-enabled fraud costing?
The FBI's 2025 Internet Crime Report logged more than 22,000 AI-related complaints and nearly $900 million in losses. Analysts caution this is likely a floor, since victims often can't tell whether AI was involved.
