I’m coming down from spending a few days at Usenix Security, right here in Baltimore. This means that my days have been taken up with two kinds of conversation: first, explaining to colleagues why Baltimore isn’t actually like The Wire. And second: trying not to talk about AI.
Here I’m going to break both of those rules.
I have many worries about what AI means for our field, for various definitions of “field”. But in this post I want to focus on just one thing I’ve started worrying about, and it’s a perverse thing: specifically, I’m worried that AI is going to make softwaremuch too secure.
While that doesn’t sound so bad on the surface, there’s a consequence to this. I mean something very specific: I’m concerned that U.S. intelligence and law enforcement agencies are about to go dark, meaning lose a huge portion of their capability. And that this isn’t going to be simply a problem for those agencies, but also for those of us who value computer security and privacy in general.
To explain how we got here, we need to talk about recent history. Here we have a real excuse to reference The Wire, which embeds a realistic snapshot of what electronic surveillance looked like in 2002. The cops in that show are after payphones and burners, all used for voice calls. While the mobile phones were new, nothing in here would have surprised a cop from 1989.
Less than a decade later, everything was different.
The change started in the late 2000s with the rise of smartphones and texting. In 2010, Apple began encrypting iPhone data using a key derived from the user’s passcode, and Google followed behind them. In 2011, Apple deployed end-to-end encrypted text messaging. By 2014, WhatsApp had 600 million users worldwide, and by 2016 nearly a billion — and they were all using end-to-end encrypted messaging. The chart below gives a snapshot of how quickly the world changed between The Wire era and 2016:
The FBI and law enforcement agencies noticed the trend and took it very seriously. In 2014, Director Comey announced an initiative called Going Dark, which would launch a “national conversation”about what providers could do — or be compelled to do — to make these new communications media legible to law enforcement and counterintelligence.
In 2016, the agency stopped talking. When a terrorist attack left the FBI with the shooter’s locked iPhone, the agency ordered Apple togive them access. The companyrefused. What broke the stalemate — and, to some extent, ended “Going Dark” itself — was something that neither the FBI nor Apple expected. An outside company announced that there was no need for Apple’s assistance:they could simply hack the phone.
The Apple v. FBI case turned out to be microcosm of the whole debate. For the next decade, law enforcement and intelligence agencies continued to ask for exceptional access backdoors. But the urgency was gone: agencies and manufacturers knew that law enforcement could purchase targeted hacking tools if they needed them badly enough. Vendors like Apple and Google played a vigorous defense, closing vulnerabilities as soon as they learned about them. But commercial offensive vulnerability hunters consistently managed to keep the edge.
Anyway, that’s the history. And now it’s about to be over.
In April, Anthropic announced anew model called Mythosthat was optimized for software vulnerability finding. The U.S. governmenttemporarily blockedits export, restricting it to U.S. agencies. While the ban was dramatic and made for good PR, it was mostly pointless.OpenAI, along with Chinese open-weight model labs likeZ.aiandMoonshot, have since demonstrated that vulnerability finding isn’t anything that a single model can hold a monopoly on. The list ofserious vulnerabilitiesthat these modelshave found is getting scarier (or more impressive) by the day.