OneNym.idDiscuss deployment
English

THE HUMAN FIREWALL / Chapter 5

Deepfakes and AI-Powered Scams

Computers can now copy a familiar voice or face and write scam emails without a single mistake. This chapter explains what has changed, what has not, and why the old habit of pausing and verifying matters more than ever.

This page summarises the chapter. The full text is in the book.

What a deepfake actually is

The chapter opens with Rosa Mendes taking a call from her manager, Paul Edwards, asking her to push a payment to a new account before five and skip the usual sign-off. It sounds exactly like Paul, but it is a computer-generated copy of his voice. A deepfake is simply a modern forgery: a fake voice, picture or video built from real clips of a person, and a few minutes of public audio, a webinar recording or a voicemail greeting can be enough raw material.

The practical consequence is that recognising a voice or a face no longer proves who you are dealing with. Most calls are still genuine, but recognition is now the start of a check rather than the whole check. The book also points out that the copied person is only the costume; the real target is the person who can move money or grant access, so attention belongs on what the call is asking you to do.

The voice of the boss

Deepfake calls to back-office staff follow a recognisable recipe: a trusted voice, urgency, secrecy or isolation, and a specific action that moves money or hands over access. When those four stack up in one call, caution should rise however real the voice sounds. It is the same con as business email compromise, upgraded from an email to a phone call.

A voice is more disarming than text because it arrives in real time with emotion and expects an answer now. The criminal is working on your helpfulness and your dislike of conflict, not your judgement. You do not need to decide whether the voice is real; you only need to notice that the request wants to skip verification, and then not let it.

Stay calm, then call back

The defence is to say something polite, hang up, and reach the person through a channel you already trust, such as their number in the company directory, internal chat or their desk. If they cannot be reached, the payment waits; a short delay will not hurt a genuine payment but often stops a fraudulent one. The criminal controls the call you are on, not the channel you choose yourself.

The book is careful to say a call-back is a strong layer, not a guarantee, because numbers can be diverted and accounts compromised, so it should sit alongside approval procedures and separation of duties. Two extra habits help: an agreed safe word or rule for sensitive requests, and naming the situation to yourself out loud. In Rosa's story she rang Paul's directory number, he knew nothing about the payment, and the security lead later found the voice had probably been cloned from a recorded conference panel.

When the grammar gets good

For years people were taught that phishing emails were full of spelling mistakes. That shortcut has expired, because scammers now use the same AI writing tools as everyone else and can produce fluent, polished, even personalised messages in any language.

The replacement rule is to judge a message by what it asks you to do and how, not by how well it is written. A beautifully written request to change bank details, click a login link or buy gift cards is just as dangerous as a clumsy one. AI can polish the words, but the request for money, passwords or data does not change.

Video calls, scale and staying steady

Some frauds now happen on video meetings where one or more familiar faces on screen are fakes, and the social pressure of a live meeting makes refusing feel rude. A face on a screen is no more proof than a voice on a line, so confirm any important, irreversible request afterwards through a trusted channel. AI also lets criminals produce thousands of tailored messages and run automated chats, which is why scams can feel more frequent, but the same habit stops one fake or ten thousand.

The chapter ends by rejecting fear. Technology changed the costume, not the con: scams still impersonate someone you trust, manufacture urgency and push an action you cannot take back. You do not need to out-spot a computer, only to refuse to act on an important request until you have verified it through a channel you trust.

KEY POINTS

Key points

  • Treat recognising a voice or face as the start of a check, not proof of identity.
  • Watch for the pattern of a trusted voice, urgency, secrecy and a request to move money or grant access.
  • Hang up politely and call back on a number from the company directory before acting on any urgent request.
  • Let a payment wait if you cannot reach the requester through a trusted channel.
  • Judge emails by what they ask you to do, not by how well they are written.
  • Confirm important requests made in video meetings afterwards through a separate channel.

QUESTIONS

Common questions

What is a deepfake voice scam?
A criminal uses software trained on real recordings of someone, such as a manager, to generate new speech in that person's voice. They then phone a member of staff and ask for an urgent payment or a change of details. A short public clip, webinar or voicemail can be enough material to build the copy.
How can I tell if a call from my boss is a deepfake?
The book's advice is not to try. Instead, notice the pattern of a trusted voice, urgency, secrecy and a request to move money or grant access, then hang up and call the person back on a number you already have, such as the company directory. If you cannot reach them, the request waits.
Is calling back on a known number enough to stop deepfake fraud?
It is a strong layer but not a perfect guarantee, because phone lines can be diverted and accounts or directory entries can be tampered with. Use it together with your organisation's approval procedures, separation of duties and independent confirmation for anything that moves money or grants access. The aim is that no important action rests on a single unverified request.
Can I still spot phishing emails by their bad spelling and grammar?
Not reliably. Scammers now use AI tools that write fluent, correct and even personalised text in any language. Judge a message by what it asks you to do, such as a payment, a login, a password or sensitive data, and by whether it pushes urgency or secrecy.
Can a video call be faked?
Yes. Live video can be deepfaked in the same way as a voice, and some frauds use meetings where familiar colleagues on screen are fakes. Stiff movement, odd lighting or lips that lag the words can be clues, but do not rely on spotting them. Confirm any important, irreversible request afterwards through a separate trusted channel.
What is a safe word and should my team use one?
A safe word is a phrase or an agreed rule that a stranger cannot know and a recording cannot supply on demand. Some teams use a rule such as always confirming unexpected phone payment requests in person or by internal chat. Families use the same idea against cloned-voice calls from a relative asking for money.