AI and Hacking: How This Tech Changes the Cybersecurity Game

AI and hacking are a match made in heaven – or in hell, depending on which side of the cybersecurity equation you’re working on. It’s changing the rules, revealing new vulnerabilities, sometimes behaving unpredictably, and exposing our tendency to trust technology too much. Cybersecurity is entering into uncharted territory. And it’s essential we don’t let our curiosity get too far ahead of our caution. See Hacking AI with Rich Smith for a complete transcript of the Easy Prey podcast episode. Rich Smith has been in cybersecurity for over twenty years, and currently leads offensive research at Mindgard. Before that, he held a number of positions, including CISO at Crash Override and Director of Security at Etsy, leading research labs at Gemini and Duo Security, and a lot of security consulting and red teaming all the way back to Hewlett-Packard Research Labs in the early 2000s. He had always been fascinated with computers and tech, but his interest in hacking sparked when he was eight years old and managed to bypass the password his older brother had put on his computer games’ floppy disk. Though he went to college for chemistry, Rich realized pretty quickly that he didn’t want to be a chemist, and pivoted back to his interest in computers and computer security. After getting a Master’s in information security, he planned to go on to a PhD. His PhD advisor got him a short-term job in the security department at HP Labs to earn some money before he went back to school. He never returned for his PhD, instead launching into a career in offensive security. It’s a passion and something he genuinely enjoys. He’s motivated to do it for the problem, not the paycheck. The majority of Rich’s career has been in the white hat world. People ask for his assistance trying to hack their systems and finding vulnerabilities. And through this, he’s found some interesting things. One example that he worked with was a major bank. They had updated their web banking and called him in for a standard assessment. Rich found a surprisingly trivial vulnerability with huge impact. Users were authenticated to the banking session within a cookie , which was serializing raw C# code into the session. That means you could serialize your own C# function in the cookie and have it executed on the server. That’s not the kind of thing you expect from top-tier banking systems. It goes to show that mistakes can still happen and it pays to have another set of eyes looking at things. Another interesting thing Rich found was with a gaming company. They were experiencing some cheating with online games, and wanted to understand the core problems cheating bots were taking advantage of so they could reduce them. The back end of the game was written in Python, which Rich is very experienced in. He reverse-engineered one of the bots to see how it worked and discovered the author was using some open-source code that Rich himself had developed. He had to report it citing his own toolkit as the problem. That demonstrated that open-source tools aren’t inherently good or bad, it all depends on how you use them. Open source software, as with many things, it can be used for good or bad. It’s really about the application. In a lot of ways, security professionals are still working out how AI has changed hacking and cybersecurity. A lot has changed, and it’s going to continue to change. Over the next year or two, we’re going to see big changes on both the security and privacy side. That includes new ways to protect ourselves, and new security and privacy risks. Over the next 12 to 24 months, there is going to be a rapid acceleration on both the security and privacy side, both improving and finding [new] problems. It’s a different space than it used to be, and that comes with new problems. A big one is the sometimes random nature of AI. In the past, testing a system’s security meant trying to attack it and seeing if it fell victim. It was binary. Either the attack would work or it wouldn’t. With generative AI, that’s not the case anymore. You have to send the same prompt multiple times to figure out if it’s actually vulnerable, and if so how badly. It’s not a binary anymore, and false positives and negatives become crucial. A lot of tools built over the last few decades don’t apply anymore. Mindgard is focused around finding problems in new technology and finding out if it requires new techniques. There’s been a huge amount of progress, but we still haven’t solved assessing risk and threat of AI capabilities. Some of these AI models even have emergent behavior, which is behavior that wasn’t programmed in but happened as a byproduct of the model or network. That’s amazing. But emerging capabilities comes with emerging risks. Just because you haven’t created an app to do something doesn’t mean it can’t be exploited to further an adversary’s goal. With emerging capabilities comes emerging risk. When testing whether an AI system is vulnerable to certain hacking attacks, Rich runs a variety of tests to help understand the system’s capabilities. One of those is Base64 encoding. All of the main AI models right now have a deep understanding of Base64, probably because it’s been prevalent in their data sets. If you send a query or prompt encoded in Base64, the majority of these models can decode it, understand it, and reply accordingly. You can also ask it to respond in Base64. This is a powerful tool for getting around input/output filters. But when you’re assessing a model’s Base64 capabilities, you have to be careful how you ask the question. If you just send a “Hello, world” program encoded in Base64, it’s likely the model has seen a lot of examples of those during training. It may be able to recognize it without needing to decode or understand it. The key is to have lots of carefully-chosen techniques to tease out what’s real and what’s memorization or hallucination. AI models are improving all the time, too. This adds a new layer of risk. Maybe the model didn’t have a particular capability a few months ago and was tested as safe, but it’s improved and now has that capability. It’s become a risk in a way that it wasn’t previously. It becomes a completely different way of assessing security, risk, and threats. A lot of it comes down to rigorous statistics. It takes a lot of overlapping disciplines to weave together objective truth from probabilistic systems. [AI leads] to a fundamentally different way of assessing the security, the risk, and the threat of a system. A lot of the focus in academia has been on AI model safety. Are the training sets polluted? Can we make the model act in a way that’s not in compliance with its system prompt? Is there a way it will give you dangerous, illegal, or offensive content? These are all important aspects, and they should be researched. At Mindgard, their AI focus is more on system safety. They don’t just look at the model, but also how it’s built into systems and integrated into tech stacks and data sets. They’re looking at what’s connecting to those models and giving them access to resources and tools. From a hacking perspective, those edges of the AI model are where there’s the most leverage and the most bugs. The more external areas you plug it into, the more that increases. The traditional attack method involves understanding resources, learning how you might take advantage of them, turn them into assets, and use them either to discover more resources or reach your goal. The fundamental model of hacking in the AI world is no different. Criminals want to understand the core capabilities of the model so they know how to exploit it. We’re plugging these models into systems built before AI was even conceived, so risks are in the edges. What is the model touching, what is it integrated with, and what assumptions are there? We can get a lot of value from integrating AI capabilities right now, and AI will probably be the future. But we have to make sure we’re not inadvertently opening a vulnerability to something we were previously secure against. The way you would attack an AI-based platform is different from the way you’d attack one without AI. There’s definitely some carryover. If you drew it as a Venn diagram, there would be some in the middle that work on both. But the new ways have no effect on a more traditional system. There’s a lot of overlap between different areas of expertise on the new systems. One of the things Rich appreciates most about Mindgard is they have such a variety of expertise. The vast majority of people there have PhDs. They have deep expertise on machine learning, cybersecurity and hacking, and even psychology and social engineering . These skills all come together to test applications. They need traditional offensive understanding of how to breach systems, deep understanding of how models work and why, and the psychology knowledge to talk their way around models. AI is almost like a bridge between the technology and the human, and that’s where social engineering comes in. Attackers are now brining these skills into their attacks from the very beginning of interacting with an AI model to try and get it to give them more usable tools. Bridging these disciplines is crucial for better security against attacks and hacking as we integrate AI into more and more technology. Now [attacks are] really bringing that social engineering in from the first instance of interacting with the [AI] model. There are polarizing perspectives on AI. But the truth lies in the middle. While AI seems magical in a lot of ways while you’re interacting with it, it’s just technology. Don’t lose sight of that. Every early tech has benefits and pitfalls, and AI has a lot of both. Every early phase technology comes with both potential and pitfalls. The potential on the AI side is huge. So are the potential pitfalls. When you interact with new technology, understand that it is new, and its uses outpace its security right now. For most consumers, you don’t need to be petrified, but do be mindful. There are a lot of privacy aspects to AI, including your inputs being used to train the models. If you’re using the models as therapists, they don’t have any of the privacy or confidentiality laws that human therapists have. Be aware of what you’re sharing and how it could be using your data. We need to interact with new technology understanding that it’s new, understanding that its applications may be outpacing its security right now. If you’re deploying AI as part of a company product or feature, make sure you test it like you would with other new tech. Have pen testers take it for a spin and see if anything comes up. People are going to attack it anyway, so it’s good to know in advance. Adopting new tech always has some risks. It’s important to be mindful and weigh the risks against the benefits. Learn more about Mindgard at mindgard.ai . You can also connect with Rich Smith on LinkedIn or email him at [email protected] . He’s happy to chat more and dig into any of these topics.