Name: How Twitter Survived Its Biggest Hack—and Plans to Stop the Next One
Brand: wired.com
SKU: wired-com-154
Availability: OnlineOnly

critic

How Twitter Survived Its Biggest Hack—and Plans to Stop the Next One

preview

at first, just another day for Parag Agrawal, the chief technology officer of . Everything seemed normal on the service: T-Pain’s fans were defending him in a spat with Travis Scott; people were upset that the London Underground had removed artwork by Banksy. Agrawal set up in his home office in the Bay Area, in a room that he shares with his young son. He started to hammer away at —integrating deep learning into Twitter’s core algorithms, keeping everything running, and countering the constant streams of .But by mid-morning on the West Coast, distress signals were starting to filter through the organization. , and they were good at it. They were , instructing them to reset their passwords. Many employees passed the messages onto the security team and went back to business. But a few gullible ones—maybe four, maybe six, maybe eight—were more accommodating. They went to a dummy site controlled by the hackers and entered their credentials in a way that served up their usernames and passwords as well as .Shortly thereafter, several Twitter accounts with short handles—@drug, @xx, @vampire, and more—became compromised. So-called the way that impressionist artwork is valued on the Upper East Side. Twitter knows this and views them internally as high priority. Still, the problem didn’t filter up to Agrawal just yet. Twitter has a dedicated Detection and Response Team that triages security incidents. DART had detected suspicious activity, but the needed response was limited. When you run a sprawling social network, with hundreds of millions of users, ranging from obscure bots to the leader of the free world, this kind of thing happens all the time. You don’t need to constantly harangue the CTO.But then, at 3:13 pm ET, the cryptocurrency exchange Binance sent an unlikely tweet announcing that it was “giving back” around $52 million of bitcoin to the community with a link to a fraudulent website. Over the next hour, 11 cryptocurrency accounts followed suit. And then, at 4:17 pm ET, @elonmusk tweeted a to his nearly 40 million followers. A few minutes later, @billgates did the same.Soon every single notification device that Agrawal had was buzzing: Slack, email, text, everything. Something was going horribly wrong. At 4:55 pm ET the tweets came faster: Uber, Apple, Kanye West. Jeff Bezos, Mike Bloomberg, and Elon Musk again. Twitter was under attack.The overwhelming feeling in those first moments was uncertainty, even fear. High-profile accounts were dropping like slasher-movie victims, with no sense of how or who might be next. The system had been compromised, and now Twitter had to figure out what to do next. Shut everyone out? Shut down some accounts? If the attack was coming from the inside, could anyone be trusted? Everyone at the company felt like they needed to respond, but no one was exactly sure how. “It was an unbounded amount of risk,” Agrawal says.Most PopularThat harrowing moment, and that harrowing day, also raised an even more harrowing prospect: What if someone hacked the platform to subvert American democracy? Since that moment, the company has embarked on an effort to harden its defenses before November 3, and it has been rolling out changes to better protect its systems, its users, and US democracy itself. Today, in fact, a series of new security protocols, mandatory employee trainings, and policy shifts. To understand why, it’s important to go back to July 15 and the chaos that engulfed Twitter. followed the bitcoin Tweets were some of the most chaotic in Twitter’s history, both on the platform and within the company. The first order of business: Stop the scam.Ideally, automated systems would have identified which Twitter reps were changing all those email addresses in such a short amount of time. But a former Twitter security employee says the company had been slow to invest in that kind of early warning technology and that a culture of trust had blinkered it to potential internal threats.Because it didn’t know where the attack was coming from, Twitter couldn’t predict what celebrity might fall next. Turning the service off altogether wasn’t practical; according to one former executive, it’s not even clear that Twitter could easily do that if it wanted to. But by 6:18 pm ET the team opted for the next-harshest thing: Block all verified accounts from tweeting. They placed further restrictions on any accounts that had changed their password in the previous weeks."We had to assume everyone was untrustworthy."Damien Kieran, TwitterChaos ensued, with many of those who could still tweet of the “blue checks.” But it also created an information bottleneck. The National Weather Service couldn’t out a tornado advisory, and media companies, including WIRED, were unable to tweet news about the hack, leaving the official Twitter Support account as the primary reliable source of information on the platform. The updates trickled out over one long thread that would ultimately extend into September, with Twitter sharing what it knew essentially in real-time. And what it knew was this: At least one of those phishing phone calls had worked.Inside Twitter, Agrawal and his team frantically worked through the tradeoffs of their potential courses of action. The tighter you shut down the internal network, the less able you are to counter the scam. You also lose the ability to track the perpetrators or figure out who on your team has been compromised. So they settled on a moderate first step: They would kick everyone—truly everyone—off the internal VPN. They didn’t want to do it all at once because they didn’t want the security response team to lose access, or to potentially overwhelm the system as everyone rushed to log back in. To stagger the process, they cut off access to one data center at a time. If you were suddenly disconnected from a meeting, it was your turn to reset.Next, they began the process of getting employees to log in to what security professionals call an environment of “zero trust.” Starting with CEO Jack Dorsey, and then going down the organizational chart, every single person needed to get onto a video conference with their supervisor and manually change their passwords in front of them. It was the Covid-era version of requiring everyone to get in a line outside the IT desk. Agrawal was soon in a meeting with the entire executive team, not to plan the response, but to confirm that everyone was who they said they were.Most Popular“We had to assume everyone was untrustworthy,” says Damien Kieran, Twitter’s global data protection officer. Each manager had to take each employee through a script and a series of password changes through the company’s internal software.To some outsiders, this reaction was a bit much. Alex Stamos, the former chief security officer of Facebook, says he’s surprised that a phishing scheme of customer service reps could lead to a total shutdown. Based on his understanding of the public record, it would have been much better for Twitter to just analyze its logs and shut down the accounts causing all the trouble. “These are the kinds of steps you take if you have the Ministry of State Security inside your Active Directory,” he says, referring to the home of China’s elite state-sponsored hackers.Another former senior Twitter employee says roughly the same thing: “There was a systems-level failure. The whole thing should not have happened. The issue isn’t that someone got phished; it’s that once they got phished, the company should have had the right systems in place.” widespread account takeovers before; Jack Dorsey himself of @jack a little over a year ago. Those incidents, though, have predominantly stemmed from vulnerabilities in third-party apps or, in Dorsey’s case, that transfer someone’s phone number to a hacker’s device. The hack of July 15 was different because it affected Twitter’s own systems. And because its alleged mastermind was a Florida teen.According to charges filed by the Justice Department and the Hillsborough County State Attorney’s Office, the scheme was orchestrated by Graham Ivan Clark, a 17-year-old from Tampa, Florida, who had previously specialized in scamming people on Minecraft. Clark had previously fallen in with the SIM-swapping community, which has . But Clark was also familiar with OGUsers, an online community that obsesses over short, common handles. And while the Twitter hack would end with 130 accounts being targeted, it allegedly started much smaller. Or as the chat recorded in his later indictment with one of his potential partners, , went:Clark: “Yo”Fazili: “Hey”Clark: “I work for Twitter / I can claim any @ for you / let me know / don’t tell anyone.”Fazeli: “Lol. Prove it.”With the help of Fazeli and another intermediary, Clark allegedly charged thousands of dollars for direct access to accounts. He had quickly graduated from scamming in Minecraft to controlling the accounts of people worth around a trillion dollars.According to prosecutors, Clark at some point that day upgraded his initial plan: taking over @kanyewest is more interesting than taking over @SC. Soon he allegedly took over those of Musk, Gates, Jeff Bezos, Joe Biden, and more, netting around $117,000 in his rudimentary bitcoin scam. Clark pleaded not guilty to 30 charges in all on August 4. Federal agents are also investigating a Massachusetts teen in connection with the hack.Twitter seems unlikely to fall victim to this exact same attack again, at least not any time soon. The OGUsers are laying low, says Allison Nixon, chief research officer of security firm Unit 221B, which assisted the FBI in its investigation. But that doesn’t mean the company can rest easy. “Presumably the attack burned this method,” Nixon says. “As far as the election goes, there’s going to be so much chaos going on caused by all the different bad actors participating in it, I just don’t know.”Most PopularNeither does Twitter. But if a teenager with access to an admin panel can bring the company to its knees, just imagine what Vladimir Putin could do. a month for Twitter to float back to something like normal, as employees gradually regained the tools they had been denied in the initial response. But not all of them, and not always at the level of access they had in the before time. If you’re going to run a social media company, you need to have some people with some access to some accounts. Lady Gaga may indeed forget her password. Elon may lose his phone. Someone may violate the company’s terms of service and have to be banned, which means that someone needs to be able to ban them. As executives at the company point out, doing right by your users can conflict with keeping the platform safe.But one of the first things Twitter realized in the immediate aftermath was that too many people had too much access to too many things. “It’s more about how much trust you’re putting in each individual, and in how many people do you have broad-based trust,” Agrawal says. “The amount of access, the amount of trust granted to individuals with access to these tools, is substantially lower today.”One of the biggest changes the company has implemented is to require all employees to . Twitter had already started distributing physical security keys to its employees prior to the hack, but stepped up the program’s rollout. Within a few weeks, everyone at Twitter, including contractors, will have a security key and be required to use it. This change fits well into a framework that Stamos suggested in a call with WIRED. There are, he says, primarily three ways you can authenticate someone: with their user-name and password, with two-factor authentication, and with a company-supplied device that you can trace. “For most stuff, you should have two of those things,” he says. “For critical things, you should have all three.”As the US presidential election nears, the most haunting aspect of the Twitter hack remains . Twitter’s investigation determined that the attackers accessed the direct messages of 36 of the 130 targets. They downloaded “Your Twitter Data” information for eight victims, which includes every tweet they’ve sent—private direct messages included—when and where they were at the time, and what devices they use Twitter from. A hacker more interested in espionage than cryptocurrency would love that kind of access.There’s also the possibility of more direct disruption: Someone interested in electoral chaos could cause plenty with a well-timed tweet from Joe Biden’s account. Or with something like the hack-and-leak operations that Russia pulled off and the . Or maybe someone will combine those schemes: hack an account, and then dump a repository of stolen, truthful, confidential information from the account’s own handle. How would Twitter handle that?Most PopularTwitter is navigating these threats without a chief security officer; it hasn’t had one since December. Still, the company has planned for the apocalypse. Between March 1 and August 1, Twitter rehearsed the above scenarios and more in a series of tabletop exercises, scripting out its plans for when things inevitably go haywire, vetting and streamlining options so that its security team isn’t stuck downriver on a fishing boat when the dam next breaks. And of course it has to game-plan, too, what happens if discord on the platform isn’t caused by a hacker, but rather by a politician or president who just feels like shitposting.July 15 shows, though, that not every crisis can be rehearsed. One way to overcome the limits of imagination is to make structural changes. In addition to the physical authentication keys that Twitter will soon require its own employees to use, the company has strengthened its internal training regimen. Employees will all undergo enhanced background checks, and they are all now required to take courses in understanding privacy and avoiding phishing. It’s not clear, meanwhile, what happened to the employees who fell for the scam back in July. To protect their privacy, and because of the ongoing DOJ investigation, the company won’t say who they are. To this day only a handful of people at Twitter know.The company has also looked outside itself, placing stricter password requirements on at-risk users like politicians, campaigns, and political journalists. It encourages, but does not require, those user accounts to enable two-factor authentication. It also remains unclear the extent to which Twitter is building in extra internal safeguards, and for what accounts. “If you have the possibility for an insider attack, which they definitely do and , you’re probably going to want a two-person sign-off policy,” says Rachel Tobac, cofounder of SocialProof security, which focuses on social engineering. Also known as a four-eyes principle, that step would mean that at least two employees would have to sign off on critical actions; if Bob has been hacked, ideally Sally hasn’t.Former Twitter security engineer John Adams that measure should apply to any account with more than 10,000 followers. A Twitter spokesperson confirmed only that “different customer support workflows require different levels of approvals based on actions/support needed.” Another former Twitter security employee says that the company protects a select number of accounts—primarily sitting world leaders—by keeping them in a separate server set, with permissions accessible only to a handful of Twitter employees. If the circle is very small indeed, it could explain why Donald Trump was spared this summer—but Elon Musk and Joe Biden were not.Twitter’s job on and around November 3 is not to avoid an attack. No target that big could. The test, instead, will be whether the structures it has put in place this year—steadily at first, then with urgency after July 15—will be enough to help it contain the impact. It needs to snuff out the flaming arrows before they turn into bonfires. There’s no guarantee that it can. But they’ll be on watch, and with fire extinguishers on hand. They’ve been through it before.More Great WIRED Stories is the former editor in chief of WIRED. You can find more information at . is Executive Editor, News at WIRED, overseeing day to day coverage across the site. Prior to WIRED he was the editor in chief of the tech and culture site Gizmodo and was a business reporter for the Yomiuri Shimbun, Japan’s largest daily newspaper.

at first, just another day for Parag Agrawal, the chief technology officer of . Everything seemed normal on the service: T-Pain’s fans were defending him in a spat with Travis Scott; people were upset that the London Underground had removed artwork by Banksy. Agrawal set up in his home office in the Bay Area, in a room that he shares with his young son. He started to hammer away at —integrating deep learning into Twitter’s core algorithms, keeping everything running, and countering the constant streams of .But by mid-morning on the West Coast, distress signals were starting to filter...

public

Brian Barrett

Nicholas Thompson

Where tomorrow is realized || Sign up for our newsletters: http://WIRED.com/newsletter

How Twitter Survived Its Biggest Hack—and Plans to Stop the Next One

No article scores yet.

CRITIC REVIEWS

PUBLIC REVIEWS