AI and Scams are a Dangerous Combination

Cybercriminals’ schemes are getting better by the day. By combining AI and scams, they can create and launch targeted attacks that used to take days of research and planning in a matter of minutes. And they can target even more people with even more trustworthy-sounding ploys. With AI-powered tools like voice cloning and deepfakes , the line between real and fake is getting blurry. But keeping up is crucial. See AI Supercharges Scams with Brett Winterford for a complete transcript of the Easy Prey podcast episode. Brett Winterford is the vice president of Okta Threat Intelligence, a company that bills itself as the world’s identity company. They focus on helping employees access what they need to do their jobs securely, as well as supporting a lot of customer identification behind he scenes. They also study things like phishing , social engineering , credential stuffing , and other attacks to get better at prevention. One of Brett’s jobs is to take those observations and make sure customers adjust their system configurations and security awareness programs to be better protected from the latest threats. Five or ten years ago, setting up the infrastructure behind something like a targeted phishing campaign would have been challenging for an attacker. They would have needed to do some information gathering, understand what their target would respond to, and make sure they knew enough about them to maximize their chances of success. Today, AI makes scams much easier. Criminals can create targeted phishing infrastructure quickly. And they can also rent that infrastructure from other criminals who have already created it. Cybercriminals are very good at specializing and collectively they’re very efficient. AI makes every step of the process easier. It can gather data faster, create compelling messages, create well-written messages in languages the attacker doesn’t speak, and create front-end infrastructure in just seconds. Digital efficiency has let scammers go from zero to attack in hours instead of days, and do it at scale while still being targeted, effective, and looking polished. Everything we’ve done to enable efficiency in the digital world is something that can be abused by threat actors. When you’re trying to respond to these kinds of attacks, speed is essential. You have to be proactive, because if you’re reacting to what’s happening, you’re already behind. Threat actors’ creativity and ability to adapt to defenses is remarkable. And their goal is often disruption. When it happens, you don’t have a lot of time to respond. If you’re going to try and disrupt [threat actor] activity, you don’t have a lot of time to get the message out to the relevant people and help protect them. Large-scale phishing attacks are still out there. These are the kind that are a little generic and sent to a huge number of people. They’re a numbers game – if you send it to 100 million people and 0.01% of them bite, that’s still a pretty good payday. But with AI, it’s easier for scams to be highly targeted. They can send out personalized messages that people are more likely to fall for in huge quantities. In the past, scammers would spend days building campaigns to spearphish particular CEOs and well-known people. Now, they can do that for everybody. One of the biggest changes has been around malware . In the past, a lot of it was focused on banking, specifically trying to get your bank account information. But the world’s largest banks have gotten much better at detecting fraudulent activity. So a lot of criminals have moved to more generic info stealing. They try to infect devices and steal passwords, browser information, and more. They compromise whatever accounts they can and then sell those to people with all different kinds of motivations. So many of our online systems are now critical to our lives that they can sometimes get a lot with what seems like an insignificant account. The rise of crypto has changed targeting, too. Scammers like targeting crypto holders because the paydays can be huge. And because many people don’t understand crypto, they sometimes respond to crypto-based scam lures even if they don’t own any crypto. The total number of phishing campaigns that end up in inboxes is stabilizing. The number is very high, but it seems to have plateaued. It’s interesting to note that as email authentication has improved over time, scammers need new ways to get their messages into targets’ inboxes. They’re leaning into trusted senders to deliver scams. Brett used to see a lot of activity where threat actors went after anyone who had a bulk email sending account – accounts where they were considered a trusted sender and could send email to thousands of people. Once they compromised those accounts, they could use that to send messages that snuck through spam filters. This is also one of the reasons many scammers have switched to SMS . You don’t have to worry about email security if you’re not sending emails, and text message security is not nearly as good yet. The thing to note about this is that the scammers don’t necessarily want anything from the person who owns that account. They’re not trying to scam them or steal money or personal information from them (beyond what they need to get into the account). What they want is the account itself. They can then use that to perpetuate their schemes. You can be a target not just because you have something they want, but because you can access a capability that they can integrate into their attacks on other people. Now you’re a target, not just because you have something of value to an attacker, but … if you have access to a capability that they need to be able to phish other users. If a criminal gets into a company’s systems, they can use the company’s infrastructure to go after their customers, or others. This happens most in the tech sector, but can be in any industry. A company’s capabilities are still of interest to a criminal. Supply chain attacks can use any capacity a company has to send messages to people. In the last few years, many campaigns have used Microsoft Teams as an attack platform because of Teams’ messaging capabilities. Sometimes it will be from an attacker-controlled Teams client, other times a compromised one belonging to a legitimate business. Because they can deliver their phishing message through a Teams message that way, it has less scrutiny than with email. Brett recently became aware of a unique new attack that threat actors are using. They set up a Slack instance that looks like it was set up by the CEO of the company they’re targeting. It seems like the CEO is inviting people to join this Slack instance. The interesting bit is that the target didn’t have to fall for it and join the Slack instance to get the phishing message. If you invite someone to join a Slack instance and then message them before they join, that message is delivered as an email. That message can be a phishing message with a malicious link. But even though the message is from a scammer, because the email itself is coming from Slack, corporate IT isn’t going to block it. It’s a clever way to get around email authentication methods and hide the signals that a message is fraudulent. Interestingly, that campaign specifically told people not to use Okta FastPass to sign in. As far as Brett’s concerned, attackers telling everyone not to use Okta means you probably should! Even five years ago, you could generally spot a scam by looking for bad grammar, outdated logos, weird formatting, and typos. With generative AI, scams are much better, and that’s all out the window. Judging by some of the things Brett sees on a daily basis, it’s getting really hard to distinguish between phishing and real. It is getting very, very hard to distinguish between a phishing lure and a real sign-in page. Brett’s best advice is just to look at the domain . If you’re expecting to go to a particular domain and you aren’t, that’s suspicious. When you first start interacting with a particular service provider, it’s a good idea to download their app. There are some fraud apps out there, but once you’re authenticated with the legitimate app, that creates a type of trust between your device and the service provider Service providers need to do more to protect people, but all we can rely on at this point is intent. What are you being asked to do, and is this something you would expect from that service provider? Brett also encourages people to use passkeys . With a passkey, there’s no password or authentication code to hand over to a scammer. It’s taken a long time to get people to understand that they need multi-factor authentication, and some still struggle with it. But because of the boost generative AI is giving scams and attackers, we need to get more used to passkey technology. Just like a password manager , getting started is the hardest part. Once you get over the hurdle of enrolling, it’s much faster and easier. AI impersonation scams are most well-known as “grandparent scams” – where scammers call older adults pretending to be their grandkids in trouble and in need of money. But it also happens in the corporate world. Impersonating CEOs is the most common. Many CEOs speak at meetings or do TV appearances, and that provides a lot of voice clips attackers can use to make a great AI imitation. People get calls and messages all the time from the “CEO” asking them to do something. Deepfake videos are used in things like the North Korean fraudulent IT worker scheme. That scam uses a lot of AI, not just deepfakes, but occasionally include an AI overlay to change an appearance. It’s a little glitchy. One way of spotting those is to ask the person to wave their hand in front of their face. It glitches in a way that makes it clear it’s AI. But the progress tech has made even over the last six months is scary. Brett would like to think he would never be tricked by AI. But a month or two from now he might not be able to say that. A lot of AI advancements have, unfortunately, given more of an advantage to attackers than defenders and targets. The tools accessible to protect us from scams aren’t scaling as fast as the tools to scam us. AI is great at creating synthetic things, and we’re going to see more of that in the future. The advice people need to follow has to change as the tech evolves. Focus on finding multiple signals and ways to anchor trust in something that you can trust more than someone calling out of the blue. Advancements in AI … give asymmetrical advantage to the attacker and not the defender. Some of the old methods of spotting scams still hold in an age of AI. If someone won’t get on a video call, is trying to move you off the platform, the grammar seems weird, or the background noise on a call is looping, that’s a sign to be suspicious. And we still see those with AI and scams, too. Being cynical and thinking first is important. As a consumer, think about how you could verify a service provider if they contact you and ask you to do something. Service providers also need to get better at setting clear expectations and trusted methods of communication people can rely on. Criminals’ skills are adapting and improving at an exceptional rate. We can’t expect people to keep up with it. Every conversation helps. But we also need tools so if your bank calls you, you can verify that they are your bank. Log into your bank’s app and see if there’s a message in that inbox. We need to teach people to find an alternate channel to verify if a message is legitimate. The tradecraft of adversaries keeps adapting and improving at a rate [where] we can’t expect people’s knowledge to keep up. Right now, you have to take responsibility for your own security. See if any of your service providers offer passkeys and start experimenting. The initial investment getting it set up will save a lot of time in the future. Get used to double-checking everything before you trust it, especially if the person or message seems unusual or is asking for private information. We don’t have to throw away existing techniques right away, but we need to start transitioning. Scammers have an advantage right now. Attackers are leveraging AI to create highly-targeted scams and attacks and more synthetic material for much less cost. But Brett tries to remain optimistic. The number of companies embracing phishing-resistant passwordless authentication methods increases every year. That makes everybody a more difficult target. And many companies are recognizing the importance of the enrollment and recovery processes in threat protection. Many times, attackers have to get on the phone, which is more risk for them. Any time we can cut off the easy routes so the attackers have to take more risk of getting caught, that’s a win. Every time an attacker has to do something that’s risky for them, that is more likely to result in them getting caught and going to prison, I’m happy. In a consumer context, providers need to do more. Not just move over to passkeys, but get better at determining normal user activity and take appropriate action when the behavior pattern differs. There’s a lot more that can be done even to secure passwords. There’s also a lot they can do around identifying synthetic traffic, fraudulent account creation, and similar activity. Every now and then, Brett sees signals that a criminal group has to adapt and change, which is great – that means what we’re doing is working. He’s a big advocate for cross-industry groups. With the right people in the room, amazing things can happen. And he remains optimistic. Maybe with AI, we’ll create a generation of digital service users who trust messages less by default. Learn more about these and other topics in Okta’s newsroom at okta.com/newsroom . There are a lot of resources there to learn more about threat intelligence and get an overview of the threat landscape.