RICK SMITH: Chairman Walden, Ranking Member Pallone, Ranking Member Schakowsky, Chairman Latta and the honorable members of the subcommittee. It’s an honor to be here before you today. My name is Rick Smith and for the last 12 years I’ve had the honor being the CEO and the chairman of Equifax.
ALIA: One of the consequences of losing more than a hundred million people’s PII and SSNs numbers -
BOB: One of the (sadly) few consequences -
ALIA: Is that you might have to sit before Congress and account for all the ways you went wrong.
Here is former Equifax CEO Rick Smith, doing just that:
RICK SMITH: “I’m here today to explain to you and the American people how criminal hackers were able to steal personal information of over 145 million Americans.
ALIA: And here’s his former SVP & CIO of Global Corporate Platforms Graeme Payne:
GRAEME: So I watched the testimony on TV from home and I, you know this is not something that a CEO ever wants to do.
RICK SMITH: The criminal hack happened on my watch, and as CEO I am ultimately responsible and I take full responsibility.
GRAEME: And having known Rick Smith, I could tell that this was really a difficult time for him to go in front of Congress and to, to have to apologize to the American public.
RICK SMITH: I’m here today to say to each and every person affected by this breach, I am truly and deeply sorry for what happened. We know now that this criminal attack was made possible by a combination of human error and technological error.
GRAEME: When he mentioned the cause of the breach was human error, I found it troubling, to be honest. I think it's really an incredible simplification of the issues and the complexity of managing cyber security and a large organization.
ALIA: Later on in one of these hearings, Smith elaborates on that “human error” -
RICK SMITH: The individual who I just discussed that was responsible for the patching process is no longer with the company.
GRAEME: The day before I had been terminated from the company, so I was able to put two and two together and work out that the person that he was talking about was me.
ALIA: But that “one human error” of course, isn’t the full story -
BOB: Not at all! So many more mistakes.
ALIA: Today, we look at the full story. We sat down with the two guys who went to Equifax, investigated, and wrote the detailed report on *all* of the big and little things that went wrong -
BOB: And we have a conversation with the“human error” himself - Graeme Payne -
ALIA: To get to the bottom of what *really* went wrong at Equifax, to really break down the perfect storm that led to (what some people are calling) the worst breach ever.
BOB: Worst. Breach. Ever. I’m tech-journalist Bob Sullivan -
ALIA: I’m podcast-civilian Alia Tavakolian -
BOB: Welcome to this week’s episode of Breach, brought to you by Carbonite.
ALIA: How businesses protect their data.
ALIA: Back in the day, Bob and I were deliberating over whether or not to do the Equifax breach because we worried we didn’t have enough details about how the breach actually happened.
And then, lo and behold - an incredibly detailed report was released that blew our minds! That was from the General Accountability Office - the GAO Report.
So: we went to DC and sat down with the two experts who wrote it.
BOB: Yeah we went to this General Accountability Office which is Congress’ investigative arm, and my goodness. The inside of that building it was like a highschool from your nightmares from the 1950s.
ALIA: Yes, or like a building out of the book 1984.
BOB: Yes.
ALIA: Just these long white hallways that seemed to go on forever.
ALIA: So we sat down and we met with Michael Clements -
MICHAEL: My name's Mike Clements. I'm a director in GAO’s financial markets and community investment team.
ALIA: And Nick Marinos -
NICK: I'm Nick Marinos. I am a director within GAO’s information technology team. Uh, my team focuses on cybersecurity and data protection issues.
ALIA: Nick and Mike teamed up to write this report - Nick on the tech side of things, Mike on the financial and business side of things. Since we’re mostly interested in the tech side of things, you’ll hear from Nick (but we couldn’t have done it without Mike!)
BOB: Ok so now that we’ve insulted their building, I have to tell you that this is the part of government that works best. If you want to really find out what’s happening in any part of the federal government, you find a GAO report from 3 or 4 years ago and you’re gonna find the absolute best ironclad research on anything. So I have a serious intellectual crush on what the GAO does.
ALIA: The GAO has an Audit team, that’s Nick and Mike -- their job is to investigate any fraud, waste or abuse happening: within federal organizations OR anywhere Federal dollars are spent.
NICK: You know, the one thing to mention too, you know, the federal government is a customer of Equifax’s services when it comes to, uh, in particular verifying the identity of individuals that may be accessing federal government services.
ALIA: You know how some websites will ask you security questions about yourself to verify your identity? What car you drive, or which of these addresses you’ve lived at? Well the federal government uses Equifax to get that data. So that contract gives the GAO the right to do this investigation in the first place. Damn Equifax - your tentacles touch everything!
ALIA: Why did they let you in? Like why did they say yes, you can come visit us? Did they have to?
NICK: That's fair. They actually, they they did not necessarily have to, to talk to us about all the information that they shared. I think at the time that we interacted with Equifax, um, they were working through their own sort of public relations activities and customer relation activities. I can't obviously, you know, conclude exactly why they, they met with us, but I think the climate, you know, created a scenario where they wanted to cooperate with us.
BOB: And they were open.
NICK: Absolutely. Yeah.
BOB: Like you didn't feel like they were hiding anything from you, or , or.
NICK: Absolutely. We were on site for a day and a half.
BOB: Rick Smith’s testimony left us with the impression that there was a human error and there was a system error and that was it. Nick and Michael’s report give us far more nuance, blow by blow, all of the ways that things went wrong with Equifax, a lot more to consider about how this really happened.
ALIA: So I know it’s far more complicated than this, but I kind of like Rick Smith’s umbrellas of “Human Error” and “System Error” - I mean, it’s kind of like Humans v. Machines, who’s messed up more?
BOB: You can’t really separate humans and machines. They’re intertwined in a way that you can’t break them out separately. Um, the machines only work as well as the humans tell them to and that’s just how it- that’s the truth. If there’s a mistake that’s made, they make it together.
ALIA: And oh my gosh did these co-dependent humans-plus-machines make some mistakes. Let’s. Get. Into it!
NICK: So back in march of 2017, just to kind of step back from the breach itself back to what was happening at the time-
ALIA: There’s a federal agency called -
NICK: US Computer Emergency Readiness Team, the US CERT.
BOB: So US CERT -- It’s part of the Department of Homeland Security.
NICK: Which has responsibilities for cataloging vulnerabilities and then communicating that out to federal agencies and to the public -
BOB: And they do this so that private companies are aware when there’s any weaknesses that have been discovered in the software that they rely on.
ALIA: Ok that’s really cool. I don’t think I knew that that was a thing.
BOB: This is an early warning system that’s actually been around for a long time.
NICK: And so back in 2017, in March of that year, um, US CERT had actually put out a notice that had identified, um, that there had been a vulnerability in a type of software, actually a framework that's used, an open source framework, for, um, setting up sort of web traffic, if you will. And that's called the Apache Struts web framework.
ALIA: So, this is just a tool Equifax uses, a software framework that’s called Apache Struts, right?
BOB: Yeah that’s right, a lot of people use Apache Struts. Equifax used it in their online disputes portal. We talked about that last week, that’s the part of Equifax’s website where customers dispute inaccuracies in their credit report, all Big 3 Credit Reporting Agencies have one. What this means, if Struts has a vulnerability, that this part of Equifax’s site also has a vulnerability - there’s essentially an unlocked, open door in this Apache Struts software -
NICK: So they had notified everybody that this vulnerability existed, and a patch was available, which basically is a fix for that software to then work properly to kind of close the door where they had identified it being open.
BOB: So this notice is sent via email early March of 2017, it’s sent to Equifax’s “Global Threat and Vulnerability Management Team” and to CSO Susan Mauldin, along with the patch.
ALIA: “Hey your stuff’s broken, here’s how to fix it.”
NICK: A couple days later -
ALIA: Still early March -
NICK: Attackers had, uh, been scanning out to, uh, identify any situations where the vulnerability had not been fixed and had encountered that equifax had servers that had been unpatched.
ALIA: So if you notify everybody that a vulnerability exists and that it needs patching, you’re also kind of notifying the hackers.
BOB: It’s a race between people--the good guys who fix computers and the bad guys who exploit them.
ALIA: So hackers are poking around discovering Equifax’s unpatched servers as early as March-
NICK: It wasn't until May of that year in 2017 that the attackers actually started to use, that vulnerability.
ALIA: Are these the same hackers poking around in March and then breaching the data in May?
BOB: Maybe, maybe not, we don’t actually know.
ALIA: But there were months in between -- Equifax could have patched those servers in early March and avoided this, right? I mean how long does it take to patch something?
BOB: Not that long. I mean, the patching itself could happen in instance. The process might take a day or two.
ALIA: Well why didn’t they take that day? Why not do something about this in March?
RICK SMITH: I’m looking forward to answering your questions. Thank you.
BOB: In his written testimony to the House on October 3rd, Smith says that the government notified Equifax of the software bug on March 8, and the company notified all the relevant employees on March 9 - and Equifax policy is they had 48 hours to repair the software -- but, he says that no action was taken in response to this notification.
ALIA: Just…no one updated that one particular server.
NICK: Well, apparently the, um, the list of administrators, individuals that actually have the responsibilities for updating servers across equifax's company wasn't up to date. And so it had, uh, not included that the contact information for some administrators that had come on board and had responsibilities in particular for the server that we, that we mentioned,
ALIA: The email list just wasn’t up to date.
BOB: So approximately 430 individuals and distribution lists received this email - it instructed “Personnel responsible for Apache Struts installations to upgrade” appropriately.
But (some argue) the right people (whose job this was) weren’t on the listserv.
And no one, presumably, noticed that and sent it to those people and said “fix this”.
ALIA: When I think about the sexy world of cyber-security, I don’t necessarily associate it with the administrative work of keeping your email listserv up to date when a new hire comes on, or someone leaves the company.
So all of you out there should be thanking those administratively gifted humans at your office who might be saving you from a breach.
BOB: Do something nice for them. Give them a raise.
ALIA: So this email sent to an outdated mailing list - this is what Graeme Payne was supposed to have forwarded, but didn’t? The email that was sent to 429 other people too?
BOB: Yeah that’s what Rick Smith said in his testimony.
RICK SMITH: The human error was the individual who was responsible for communicating in the organization to apply the patch... did not.
CHAIRMAN: So does that mean that that individual knew that the software was there and it needed to be patched and did not communicate that to the team that does the patching? Is that the heart of the issue here?
RICK SMITH: That is my understanding, sir.
ALIA: Well we talked to Graeme, the quote human error himself, to see if he thinks that’s actually what went wrong. This is the person who got fired, and then turned on the tv the next day to discover he was being blamed for causing the breach -- so does he agree that it was his fault?
GRAEME: I think the committee concluded that it didn't align with the backdrop of the facts.
BOB: To put it in my words: Bullshit.
ALIA: So what, what exactly like talk to me about this email that I keep reading about.
GRAEME: Right. So this one specific email went to about 430 people. And in my role as uh CIO, I was copied on that was just like all the other CIOs and these things would come out periodically, and I'd look at them and that was sort of the, uh, the end of the story as far as I was concerned.
BOB: But that wasn't the end of the story - yeah - as far as your company was concerned
GRAEME: Right, right. So I was never under the impression or direction, and there was nothing ever stated in policy that required me to forward those emails to anyone. Um, I, my assumption was that the appropriate people were getting the notifications. I got hundreds of emails a day, so this was just one of many, many emails a day that would come through my inbox.
You know, the company did their investigation and they concluded that, uh, I should have forwarded that email and because I didn't forward that email because it didn't get to the people that were actually administering the system that I was the breakdown in the process.
ALIA: And how did that make you feel?
GRAEME: well when I was first terminated, I wasn't quite sure what email we're referring to honestly. It was only when I heard the testimony the next day that I put two and two together and worked out that that was the email they were referring to.
ALIA: And what do you make of that? LIke what do you make of the fact that, that, that an email essentially, at least in the way that I'm hearing it, the email was sort of what took everything down.
GRAEME: You know, to me that's an oversimplification of the complexity of this issue.
ALIA: I like how Graeme put it in his *testimony* in the “House Committee on Oversight and
Government Reform": “If that’s the process that the company has to rely on, then that’s a
problem.”
ALIA:I don’t know that Graeme wasn’t a human error that contributed to the breach. I do know he’s not the one human error that caused the breach.
I mean that Apache Struts email had 430 people on it. That’s 429 other possible human errors.
BOB: We asked Graeme, what did he think went wrong then? If it shouldn’t just be reduced down to him just not forwarding an email?
ALIA: He pointed us back to what Nick and Michael found in the GAO Report, (and what a House Committee report later categorized) as Equifax’s “specific points of failure” - it breaks down to three problems:
GRAEME: The first was around a lack of accountability and no clear lines of authority in the equifax IT management structure.
ALIA: Problem 1: Management Structure Lacking Accountability and Organization.
BOB: So really when somebody makes an order, how do you make sure that that order is followed through on?
GRAEME: And because of that they concluded that there was a gap in the execution between policy development and, and operation.
ALIA: Problem 2: Gaps Between IT Policy Development and Execution
BOB: So somebody might say here’s a great idea that we move everything to the Cloud but then there’s 100 things that have to happen before you accomplish that objective and those two groups of people weren’t communicating.
ALIA: And Problem 3: Running Critical Systems on Legacy IT (aka Old IT) with Documented Security Risks
GRAEME: The company's aggressive growth strategy and accumulation of data had resulted in a really complex IT environment. And because of that it just made management of legacy systems and security, um, difficult.
BOB: This happens at all big companies that swallow up little companies. Merging systems is a nightmare and especially when some of those systems you acquire are old or they get old.
ALIA: The report points out that some of these systems were so old, only a few people at Equifax even knew how to operate them.
BOB: Only Mary or Jack in the back are the only ones who can actually fix or update something and if one of them are sick then nobody can.
ALIA: I mean like isn’t Equifax one of the first companies to have digitized credit?
BOB: Yeah you’re right that’s an irony isn’t it? They basically took the credit system and put it on computers and that was their claim to fame and in the end it betrayed them.
ALIA: Each of these points of failure combine to form: the perfect environment for an email to go unnoticed, for a patch to go unpatched, for 145 million people’s most important data to go missing.
ALIA: Do you think that all of this sort of email, this email fiasco demonstrates a lack of accountability that the report talks about?
GRAEME: Yes. I mean this is this execution gap, right? So at equifax we had the security team that reported up to the chief legal officer and then we had the IT organization that I was part of. And the security team had responsibility for managing this global threat and vulnerability process.
ALIA: In other words: the Security team (who manages these vulnerability emails) and the Tech team (who patches these servers) didn’t cross reporting paths. You could make the case that, organizationally, they really aren’t communicating with each other. According to another report I read, this is the result of an inherited problem - apparently a former Chief Security Officer and former Chief Information Officer just didn’t get along, so they built a system where Security and Tech stayed pretty independent of each other.
GRAEME: But because of this, you know, reporting to two different organizations, I think there were definitely some gaps in execution and as the report points out and, that was an example in this case.
BOB: If something goes wrong in the morning, Alia, and you can’t feed Ethel, how do you tell Brandon?
ALIA: Heads up - Brandon is my husband, Ethel is my sweet baby angel dog.
ALIA: I usually text him. But that doesn’t always work because sometimes he’s in a place where he can’t receive a text message. So sometimes I’ll leave like tape on the wall, a little note in tape. I know it sounds weird but it’s not. And sometimes he won’t see it?! I mean we just, we actually don’t have a system. And it’s a problem, we actually just had a conversation about this. How we need a system.
BOB: You actually don’t have a system, and that’s the truth at many many companies about updating patches. I can picture someone putting masking tape on a server saying “applied Apache Struts” or in red “Still needs patch” and the person who’s supposed to read the masking tape being out sick that day, and the next thing you know 100 million social security numbers go flying out the door.
GRAEME: And I've seen this a lot in my career that people have developed policies and put them out on their intranet and just expect everything to happen and it just doesn't work like that.
BOB: There was a detail in a different report, also in front of the House Committee on Oversight and Government Reform -- that Equifax’s patch policy basically operated on the “honor system”, and that’s crazy, like that’s insane, that’s hard to imagine.
ALIA: Yeah. But even if departments aren’t communicating and there’s a lack of accountability amongst humans - there’s supposed to be accountability via tech systems. There are fail-safe’s in place at that will notice if a bunch (say: 145m people’s worth) of data is leaving your system.
BUT -
NICK: There was a technical glitch that resulted in the traffic, the information that was, was coming in and out of this particular part of equifax's systems, wasn't getting reviewed. With this technical glitch, it allowed information to travel without going detected.
ALIA: Nick Marinos, co-author of the GAO Report again -
NICK: So in order for this particular tool to work, there's a certificate that needs to be maintained and sometimes we see digital digital certificates.- that certificate one expired, and went undetected as being expired for about 10 months.
ALIA: The certificate had expired on January 31, 2016 - so even more than 10 months. 19 months!
NICK: And so at the time that the attacker was able to use the vulnerability, the Struts vulnerability to get in, in May of that year, if that certificate had been in place, it's likely that something would have been picked up in that point.
BOB: But while that certificate was not up to date, this traffic was not being inspected.
NICK: That's right.
BOB: For a year.
NICK: Yeah.
ALIA: Eventually, someone notices. On July 29th the certificate is updated (finally), and all this suspicious web traffic is noticed on the dispute portal. So: now what?
NICK: And so they immediately pulled that server offline so they disconnected it from the internet so that it couldn't any longer be tapped for more information and they began to start to figure out, okay, what happened and what was stolen.
GRAEME: I remember that date well, because it was my birthday and the next day on July the 30th after they’d obviously done some investigation, I got a call from the chief security officer about the issue in one of the systems that my team managed
BOB: This is how it all starts: it’s a weekend, of course, and you’re away from the office…
GRAEME: So I checked my phone and it was blowing up with messages. I knew something was up so I immediately got on the phone and, and a tracked down what was going on.
ALIA: Were you like celebrating your birthday? Like I'm trying to get the image in my head. Were you around at a cake blowing out candles and then you look over at your phone, like what were you doing?
GRAEME: No, I wasn't, but it's a very memorable weekend obviously.
ALIA: Happy Birthday: here’s your present. The beginning of the worst. Breach. Ever. But even the worst-breaches-ever can seem like regular “security incidents” at first...
BOB: So your phone is blowing up. This is. I'm going to go back a little bit, but I'll bet your phone has blown up before. Right?
GRAEME: I'd been through similar incidents like this in the past. So we've sort of followed a standard protocol. It wasn't really until, um probably a week or so in that, um, we started to get some initial feedback on the extent of the breach and um really kicked into high gear.
BOB: What was that feedback? Did you get an email from the Mandiant or something or how did you know?
ALIA: Mandiant is the outside cyber-security consulting firm brought in to find out what happened.
GRAEME: Yep. So, so after the breach had been announced, obviously, you know, there was a period of analysis that had started -
ALIA: And over the next 6 or 7 days, Graeme says, this analysis reveals - that this thing is bigger than a run of the mill security incident.
GRAEME: And then we started to ready ourselves for a breach and I think that was probably seven or so days into, into the process and at that point you know the lawyers were engaged. And the security teams were, you know, we had an outside security advisors and, and, and we really in on a different path.
BOB: Well I mean. Can you imagine the sinking feeling in the pit of your stomach as you realize this is not like every other security ding and faux crisis that you’ve been in but this the big one?
ALIA: Yeah. It would feel awful. I wouldn’t. I don’t envy him at all. I would feel. I’m just thinking about the feeling I get when like I think I’ve lost a file on my Google Drive and I feel. I panic. So I cannot imagine the level of panic somebody might feel when they find out that you know, data’s been breached.
ALIA: Okay a pivot away from my panic. Let’s go to a Brain Break. When we get back, we’ll learn what the hackers were up to while the Equifax team was *super busy* not patching their servers.
ALIA: Breach is sponsored by Carbonite, so instead of doing an ad we can take a break to do whatever we feel like.
Kelly’s still waiting for Scott’s credit reports to come in the mail, so in the meantime I’ve been talking to my friends at Carbonite. Thanks to the Equifax breach, we’ve been having way more conversations about identity theft and fraud, and I wanted to hear about the “Other Identities” they’ve lived, thanks to credit fraudsters stealing their names.
MEGAN: I’m Megan Whittenberger I’m the director of marketing at Carbonite based in Boston and I’m really excited that we get to experience another season of Breach with you.
ALIA: So tell me about the other Megan Whittenberger
MEGAN: The other Megan Whittenberger got ahold of my credit card information somehow, so I got a call from my credit card company asking about a whole host of charges. But the thing is, there is no other Megan Whittenberger. I know that I’m the only Megan Whittenberger in the United States so I definitely knew she was fake.
ALIA: Oh my god. So, so like what what kind of charges did you see?
MEGAN: They were very car centric, which is interesting because I actually don’t own a car. So it was a carwash in Illinois, um some gas stations, an O'Reilly Auto Parts in Chicago, Illinois.
ALIA: So how do you think her life is different than yours? Like I know you said you’re not a car person, but do you think she’s having more fun than you?
MEGAN: I like to think of it as a really nice car, and she’s just giving it a lot of love.
ALIA: That’s amazing.
MEGAN: And just kind of cruising around Illinois.
ALIA: She’s just like getting some new mirrors, or like a vanity license plate.
MEGAN: Spending a lot of time maybe in that really long Portillo’s drive thru. That’s what I see.
ALIA: Welcome back. I’m Alia.
BOB: And I’m Bob.
ALIA: We left off with Graeme, and the Equifax team, discovering this isn’t just a security incident: it’s a straight up breach - and a bad one. But let’s pivot away from the Equifax staff -
BOB: Gladly -
ALIA: What were the hackers up to in the months prior to July 30th, when the Equifax team finally took down those unpatched dispute resolution servers? How did they get in? What was their plan of attack?
BOB: The first step the hackers take: they dropped web-shells (which are basically backdoors) to obtain remote control over Equifax’s network.
ALIA: They get in through the Dispute Resolution Portal - which, as we’ve discussed, is the online portal through which you would dispute inaccuracies in your credit report. This isn’t just a disgustingly poetic irony. Nick Marinos, co-author of the GAO report, points out -
NICK: This portal obviously takes in a lot of personal information itself.
ALIA: Imagine you disputed something on your credit report. They say you didn’t pay a bill, but you definitely did. Think of all the stuff you would’ve sent in to prove you’re you and you paid that bill. That’s all sitting in this Dispute Resolution server.
Oh look - the copy of your driver’s license or passport photo you sent, a copy of that check you wrote proving you paid that bill, your bank statement, that form with your address and email and phone number. They just got in the door, and the hackers are killing it already. They entered through a gold mine. But once they look around this server, they find something even MORE valuable -
NICK: The attackers, once they got in the door, were able to identify, usernames and passwords for other databases that were being stored in clear text. That is to say there was a file available that didn't require any password itself to get to that information.
BOB: A file containing unencrypted credentials - usernames and passwords, completely easy to read.
ALIA: Why would someone store passwords without encrypting them?
NICK: Well, if you think about it, an administrator, assistant administrators' roles and responsibilities, they're tapping and they're accessing a lot of different systems. And so a fast way of doing so is kind of like for those of us that stick Post It notes under our keyboards would be to keep that information stored somewhere so that you could easily access it. Sometimes to just copy and paste it in so that you don't have to worry about it.
BOB: So our lives are full of all these little conveniences, all these little work arounds. When we do these things, we just never imagine these workarounds in the hands of a hacker.
NICK: Some of these passwords can be very long and complex, um, can be automatically randomly generated and so it. It's something that I think demonstrates the fact that humans are going to be, they're going to be fallible themselves.
BOB: This is the absolute Catch 22 of passwords. The longer and more complex your passwords are, the less likely you are to remember them, and the more likely you are to put them in a text file or on a Post It note so that you can get them when you need them. Simpler passwords, easy to remember, complex passwords, hard to remember. Which one is safer? I don’t know.
ALIA: So after waltzing into the dispute resolution portal, these hackers are armed with a bunch of other usernames and passwords -
NICK: And so they were able to then use those to navigate to about 51 other databases or 51 databases total.
ALIA: 51 databases to enter search queries for our useful PII.
NICK: There's another thing, that I, that I'd mention to, you know, it, it took 9,000 queries. So those are like searches, to tap into these databases, right?
ALIA: Hackers have to search through these databases. These queries are their more sophisticated version of CTRL + F: Social Security Number. Nick’s point is: even if the data exiting the servers wasn’t detected (thanks to the expired certificate) - those queries themselves could have been detected.
NICK: For example, you know, in hindsight you could put a limit on how many queries get done within a certain period of time or let's say you have queries coming from one source, you know, over and over and over again. there could be ways to restrict that. So that, okay, once they hit 100, okay, there's something up. That was another area that equifax informed us that in hindsight they, and I think following that they're looking at putting some restrictions in place.
ALIA: All-in, these hackers sent 9,000 queries across all the databases they now have access to. They locate completely unencrypted PII 265 times. They transfer all that stellar info out of Equifax’s network. Download it, or whatever - which is something the expired SSL Certificate would have caught, if it hadn’t been - you know - expired for 19 months.
NICK: One of the things to note about any attack, uh, but certainly a longer term attack, you know, you don't want to do too much too quickly or else you could potentially be detected rather early and then your access is lost. So there are scenarios where attackers can sit for years for at least for many months over a year. So at that point in time, the attacker felt, uh, comfortable enough to start figuring out a way to remove the information from the databases. Over the span of about 76 days, they were able to remove, uh, records that affected over 145 million individuals.
BOB: 76 days is a long time.
ALIA: That’s how many class days are in a semester.
BOB: Someone could have taken an entire semester’s worth of cybersecurity courses in 76 days.
ALIA: Okay, we’ve learned a lot.
BOB: Yeah, we’ve learned a lot. I can’t believe that at one point we thought there wouldn’t be enough details to fill up this story without an indictment like we had Yahoo.
ALIA: Yes. We’re swimming in details and facts. Okay. Let’s do a very quick rundown of all the facts around Equifax breach, as we know them:
BOB: March: The Apache Struts vulnerability (and patch) is released. Hackers poke around to see who has patched it and who hasn’t and discover Equifax is dumb and hasn’t. Meanwhile at Equifax, the info about this Struts vulnerability gets passed along (but to an outdated email list) and there’s no accountability for getting it to the right people to patch it, so the dispute resolution portal stays vulnerable.
ALIA: Then in May - hackers return or new hackers get all the way in, find a bunch of great PII in the dispute resolution portal AND find usernames and passwords - They use those usernames and passwords to leapfrog to a bunch of other servers (51 of them) and get a ton of valuable PII over the course of 76 days
BOB: Until, at the end of July, when Equifax does routine maintenance and updates the certificate on the software that would have caught all this info leaving the whole time! But by now 145m people’s information has been breached.
NICK: I think one thing to point out with this is that it wasn't a sophisticated attack. So a lot of these techniques that were being used, the vulnerabilities that were being exploited, um, didn't require a very high level of sophistication to utilize.
ALIA: Well this is what’s occurring to me as we’re reviewing the details of this hack, Bob, we’re examining Equifax as the great murder mystery of my privacy. But this isn’t even a sophisticated murder?
NICK: If one of these things hadn't occurred exactly the way it did, the attacker might have found a different way to get in, but you string those things together and that's how you end up with 145 plus million, individual’s information being stolen.
ALIA: “If one of these things hadn’t occurred exactly the way it did…” Our SSNs might still be ours.
ALIA: There’s a sophisticated version of the murder of my privacy - a skilled assassin, a hacker on all-fours having to do a TON of work to dodge a maze of lasers or whatever. I’m picturing Catherine Zeta Jones in Entrapment and she’s dressed all in black, and she’s being taught by Sean Connery to dodge and maneuver through this maze of lasers.This Equifax breach was not dodging lasers.
ALIA: My privacy was murdered - 145m of our SSNs lost - by Catherine Zeta Jones putting on her black jumpsuit, stretching, getting in the zone, reaching for the doorknob: and it’s unlocked. The security system’s down. Someone forgot to “renew the certificate” on the lasers.
BOB: Yeah, absolutely. That seems like what happened here. It’s really a testament to how little Equifax had their act together. These hackers got the good stuff, which is a really important point. Even if Equifax was up against some nation state that it really had no chance to protect their network entirely, doesn’t mean that once that nation state got in, it was able to obtain the most critical information. The best stuff. And this is where the concept of network siloing comes in. Journalist Lily Hay Huyman wrote about this in Wired in regards to hacks at the CIA and the NSA. Those agencies had siloed networks, so essentially all of the good stuff isn’t in one place. She wrote that “the CIA and NSA leaks… show that it's possible to limit access control such that even attackers who grab something can't get everything.”
ALIA: In his written testimony, there’s one argument that former-Equifax-CEO Rick Smith makes that I think is kind of compelling. He puts some blame for this hack on America’s dependence on Social Security numbers for financial identity. Equifax has to use Social Security numbers to collect information for their reports. And it’s not their fault they’re so valuable.
BOB: I think that’s a true statement that Social Security numbers are wildly over important in terms of securing people’s identities and securing our whole financial system. However, Equifax is a big reason why Social Security numbers are as valuable as they are. They have benefitted handsomely via billions of dollars through the years. And, ultimately, they didn’t come anywhere close to securing this information properly.
ALIA: And, while not even storing our Social Security numbers properly, Equifax has voluntarily made it their business to gobble up tons more of our data. Since 2005, CEO Rick Smith embarked on an aggressive growth strategy, acquiring multiple companies, IT systems, and data data data.
BOB: Here’s Smith speaking on August 2017, just three weeks before Equifax publicly announced the breach
ALIA: This talk is on August 17th. According to Smith’s testimony, two days prior on August 15, he was informed that it “appeared likely” consumer PII had been stolen -
RICK SMITH: If you put it into context, you think about the largest library in the world. It's the library of Congress, right? That's the largest library in the world as far as content and data. We manage almost 1,200 times that amount of data every day.
ALIA: What I’m seeing in the Equifax story is a CEO and executive team proudly and greedily gobbling up data, even when they don’t have security in place to protect it. I’m seeing teams failing to communicate who’s going to take care of what, no accountability. I’m seeing the profound hubris of storing our most valuable data, and updating weaknesses on the honor system. Earlier we talked about humans AND systems or machines failing co-dependently.
But all of this feels human. Every single failure feels human - pride, greed, assuming someone else was taking care of it. I don’t buy Rick Smith’s statement that it was “system error” - those systems should be checked by humans. And I don’t buy that it was “one human error” either.
BOB: I used to have an editor that would say - not one person can put a mistake in the newspaper. It takes a whole team, the reporter has to get it wrong, the copy editor has to get it wrong, the front page editor has to get it wrong and that’s what happened here. There must have been dozens of people - maybe hundreds of people who were involved in this massive mistake.
ALIA: The story of the Equifax breach is about forgetting to feed the dog. It’s about hundreds upon hundreds of people forgetting to feed the dog. And we’re the dog.
BOB: We also have to remember who is at fault here: criminals. Criminals breaking into a private company. It’s easy to forget that because we want to blame people. But we can’t forget that there are humans illegally stealing information with malicious intent.
ALIA: And companies are trying to protect against hackers, who won’t stop till they find their way in. That’s what Graeme is doing now - he’s working at “Cybersecurity for Executives”
GRAEME: And we are focused on helping boards and executive teams manage cyber security risk and IT risks. I've taken this and turned it into something that will, will help others.
BOB: It seems irresistible that your real value in those conversations is to say blah, blah, blah. Breaches, breaches. Holy F, it can be really bad. I know. You should listen to me. Am I right?
GRAEME: Exactly. I mean I think that’s right. I mean, this is how bad things can get. Your CEO is in front of Congress testifying and apologizing to the American people. You’re living with investigations from multiple agencies and states. Your living with litigation. Careers are impacted. You've got to change the culture of your company. They’re pretty dramatic impacts. And I don't think that every board member and executive team recognizes that those things will happen. It's not a matter of if it's a matter of when and they need to be prepared for it.
BOB: You may not be able to control IF you’re breached. But you can often control how prepared you are for a breach. You can control how much valuable data is breached, and above all, you can definitely control how you respond to a breach.
ALIA: If you thought Equifax fumbled leading up to the breach -
BOB: Just wait till you see how they handled the response to the breach.
RON: It was such a debacle that I just assumed all along that they had never rehearsed for anything like this, that they had no idea what to do.
ALIA: Equifax: The Worst Response Ever,
BOB: Equifax: if you weren’t quite sure we don’t care about you, we’re going to prove to you that we don’t care about you.
ALIA: Equifax: The Fyre Fest of Breach Responses.
BOB: Next time on Breach, brought to you by Carbonite
ALIA: How businesses protect their data.
ALIA: Breach is a branded podcast brought to you by Carbonite in partnership with Midroll and Spoke Media. You can find transcripts and show notes at carbonite.com/breach
A correction from last week’s episode: in discussing Rick Smith’s speech at UGA’s business school, we should have said the date was Aug. *17*, 2017, not the 7th, and should have said that *some*, not all, of the Equifax C-Suite was notified of the issue on the servers by the end of July. For more info, see the transcripts and show notes for Episode 2.
If Cyber Security reporting were training dogs, Bob Sullivan would be Cesar Milan, and I would be giving out treats for free.
Whether you’re a dog person, or a cat person, you’re a Breach-person, and we want to hear from you. We’d like to learn more about you, our Breach listeners,
and what you’d like to hear about on future seasons of Breach.
Please go to podcastsurvey.net to take a quick, anonymous survey that will help us understand what data security topics matter to you.
Once you’ve completed the survey you can choose to enter for a chance to win a $100 Amazon gift card. Terms and Conditions apply.
Once again that’s podcastsurvey.net. Thanks for your help.
Our show is executive produced by me Alia Tavakolian -
and produced and written by Janielle Kastner aka “Producer Jan”.
With Associate Producer Caroline Hamilton, and Production Assistant Kelly Kolff. Research and co-writing from Haley Nelson.
When Bob and I are in the studio we’re recorded by Casey Holford and Jared O’Connell. Today’s episode was mixed and sound designed by Evan Arnett.
Head of Post-Production is Will Short.
The songs you hear come from APM Music and FirstCom.
Our executive producer is Keith Reynolds, who is ruled benevolently by his miniature schnauzer: Chase.
Special thanks to the folks you heard today: Nick Marinos, Michael Clements, and Graeme Payne.