I am going to try to de-convolute something that is so convoluted I’m having trouble even spelling “convoluted.” I will begin with a summary of facts, and then discuss them in more detail.

First, however, I need to explain something: the situation I am about to de-convolute is an example of what happens when some very stupid and ignorant people try to pull something over on a stupid and ignorant public. There is a great deal of “what the fuck?” – or there should be, if I manage to explain this whole thing correctly. In other words, if it makes sense, I’ve screwed up. If you’re sitting at your computer alternating between hitting yourself in the face with a book to stay awake, and drinking big glasses of red wine, then you probably understand the situation.

Some Facts:

1. Hillary Clinton had a private email server, which was located in the Clintons’ home in Chappaquiddick, NY
2. The private email server was set up semi-competently, though with notable mistakes: a) it was running vulnerable software, 2) it was running with a badly configured self-signed SSL certificate, 3) the server was backed up to a ‘cloud’ data recovery/business resumption service called ‘datto.com’ [wik]
3. We will refer to the Clinton private server, the data on it, the microsoft Exchange instance on it, and the backup data in the datto cloud service as “The Clinton Server“
4. The DNC had email server(s?) of its own, and (apparently) a more fleshed-out IT infrastructure. The physical location of the DNC servers is (?) not clear, though the DNC headquarters was in Brooklyn, NY and is described as having a “network” – presumably that means there was a variety of desktops, an email server, a file server, and (presumably) a data recovery/business resumption system. [wired]
5. We will refer to the DNC’s servers and network and all the desktops, email systems, etc, on them as “The DNC Server“
6. Connecting all of this stuff, to some degree or another, is the global data network including the internet, servers on it, services on it, and so forth. We will refer to that as “The Cloud” – things that are in the cloud have no relevant location and data should be assumed to be nearly infinitely mobile and copy-able once it gets into the cloud. I.e.: if I have a credential on your server, and your server is on the internet, I can have a copy of all of your data on my own server out in the cloud in the course of a couple of hours.

There are a few pieces that are missing from the box but they probably don’t matter much and are mostly professional curiousity – I’d like to know about the layout of the DNC network: how many servers and what type, how they were backed up, how many users had access over the LAN and how many had access over the internet. We know that there was, at a minimum, internet access to email, and that it was semi-competently managed. We know that the users of the DNC network were incompetent nincompoops who fell for basic phishing attacks and had simple password controls (not even two-factor authentication) and there was unknown (therefore questionable) system logging.

It seems as though the attack against the DNC servers started with a credential being stolen, and all of the email belonging to that credential was accessed. It also appears that, from there, the attackers expanded their efforts and accessed stored data on server(s) – some variety of documents. Although it is possible that they just accessed attachments from someone’s inbox/outbox. There’s not a lot of detail about the DNC attack because a) it ought to be embarrassing as hell to the DNC that they got pwned like a bunch of newbs who just got on the internet, b) the people who did the investigation into the breach (Cloudstrike Crowdstrike, [brain-fart, fixed later, see comments] who we will get to in a bit) were professional incident responders which means they do not go to the press and air their client’s dirty laundry, c) the response was in whole or in part controlled (and possibly paid for) by the FBI which means that they would reflexively slap a cone of silence over the whole affair.

That all sounds pretty reasonable/expected, to me. I’ve been involved in over a dozen major security incidents and you’ve probably heard about one or two of them. What you won’t hear is that I was involved on the defense side, or what I found, or anything else. Because incident response professionals understand that they are dealing with a sensitive incident, sensitive data, possible media coverage, possible law enforcement or intelligence agency involvement, and if they run off at the mouth, they are going to get edged out of the scene immediately. That’s no fun and you don’t get paid. Sometimes you sign a non-disclosure agreement and other times you keep your mouth shut because that’s what professionals do in this situation.

I’ve known the founders of Cloudstrike Crowdstrike since the late 90s, when we used to all teach at Interop together. We’re not best buddies or anything like that, but they’re solid information security professionals with long track records of being solid. George Kurtz and Stuart McClure have worked together closely since they wrote Hacking Exposed, which was briefly one of the canon tomes of penetration testing. They founded a company called Foundstone, which competed with the company where I wound up working, then got bought by McAfee and Kurtz and McClure wound up being executives at McAfee. After McAfee George went on to start Cloudstrike Crowdstrike and Cylance in 2011. They’re executive management types now, not “roll up your sleeves and let’s look at some bits!” practitioners any more, but they have that kind of people working for them. I’m not telling you this stuff to brag about how well-connected I am; it’s rather in order to give you the idea that Cloudstrike is not a company of johnny-come-lately amateurs that just appeared on the scene, conjured out of some eastern european country by Rudi Giuliani. They’re heavy hitters, I would not be surprised at all if they are billionaires or close to it. That’s relevant because they did not get to where they are by being a bunch of amateur assclowns who hire dipshits that go into an important incident response and stomp around with mud boots on.

The way a response like the DNC breach response happens is this: someone gets a phone call, “can you get down here right away? This is serious.” and they get down to wherever, right away. This is high profile, so there would have been an executive from Crowdstrike and a whole team of people carrying Pelican cases, laptops, and other things. The executive presence’s job is to sit in meeting rooms trying to calm the client down for a while, while the technical people map out what things look like without touching anything and the response team captain drops a note to the executive, roughly framing up the problem, i.e.: “we have a bunch of storage, a couple compromised accounts, no system logs on the servers, no firewall logs, and at least one report of phishing emails” The executive will then decide if they want to cut and run, or take the gig and how much it’s going to cost. In the case of something like the DNC hack, not taking the case was not an option because Crowdstrike does a lot of other stuff for the government, and the meeting room would have had a couple senior FBI people in there and possibly a CIA presence too. It’s an interesting (and exciting!) problem because you want to gather information about what is going on, but your client wants you to sit in a meeting room with them and wank furiously about “what should we do?! OMG!” and you’re nodding to them and thinking very hard and waiting for the situation summary from the incident commander. The panicy customer will eat all of the executive’s time and attention and someone has to get busy doing real work in the meantime. When you get the “go” from the client, and they’ve agreed to pay a massive fee, you tell them who is going to be the incident commander, get someone started finding office space for them, and the team begins to slowly leap into action.

Since the DNC breach apparently involved foreign intelligence, there would have been domestic spooks present, which would have been additional friction for the Crowdstrike team, but that kind of stuff is old hat for any incident responder. But, since there was foreign intelligence, someone would have decided (in the executive meeting) that it was necessary to do a full work-up and figure out what happened. I.e.: the problem is not just “OK what firewall rules do we need to change and how do we lock this guy out?” it’s more like “we think there is highly professional Russian intelligence people in this network and we can’t start stomping around and alert them that we know what’s going on.” So the incident commander starts figuring out what parts of the network need to be tapped and monitored and which systems need to be frozen/imaged and which systems may be rigged to wipe themselves. You really do not want to fuck this up. The incident commander’s plan turns into an implementation plan which gets given to the team, “OK, you set up tap monitors to a packet collector here, and here.” and “You sequester the system logs” and “get with the client’s IT team and have them freeze a snapshot of the cloud backups and prepare to change the credentials at a moment’s notice” See you’ve got to plan to collect information, protect it, analyze it, and react – all at once without making a mistake.

[Edit: There were other incident responders involved than just Crowdstrike: Mandiant and others. It was an “all hands on deck” clusterfuck. That makes me wonder why Trump has Crowdstrike stuck in his mind? It’s not as though Kurtz is a Ukrainian; maybe they’re getting confused with Kaspersky who is Russian and used to work for FSB]

There are offline tools for analyzing systems for installed backdoors, including some of the really fancy ones. (I am guessing that there would have been a CIA spook there who was familiar with ‘Fancy Bear’s toolchain, watching and not offering any information) – one of the first things the team would have done would be to make an offline copy of infected systems, using a hard drive duplicator, and dropped the original drive into an evidence bag. That’s highly probable. I suppose it’s possible that the client didn’t take the breach seriously enough to go that route (?what, who am I kidding?) but since foreign intelligence appeared to be involved, they’d have taken the high road on everything. Besides, if you image drives, you have a backup in case the attacker decides to go on a wipe-a-thon. There are tools that generate a bit-for-bit copy of a hard drive off a running system – ENCASE is the gold standard of that, and since it’s the client paying, that would be what the Crowdstrike team would have used [encase]

Imaging a client’s hard drives is … interesting. You now have a copy of their system, in effect, and you can carry it around in your briefcase. No incident responder who wants to continue to have a career would ever handle that lightly. Back when I worked at TruSecure, I audited our forensics team’s evidence room and safe, and they did all the proper things: two person custody of data at all times, all contact with media logged and recorded, etc. I’m not saying it’s impossible that someone from Crowdstrike left the building with a spare copy of the DNC’s data, but for all intents and purposes the DNC was more likely to be hit by an asteroid, right then, than for that to happen. Depending how paranoid the FBI was being, there would be an FBI guy “helping” (i.e.: watching and trying to stay out of the way) the whole process.

Then the response team spends a while (sometimes days or a week) designing a lock-out plan and communicating that with the incident commander and via the commander to the client. The lock-out plan is “this is all the stuff we do instantly to close the backdoors so they can’t come back in” which is a seriously gnarly hypothetical if you really think that the people on the other side of the breach are Russian intelligence. We will not ever know what analysis the Crowdstrike team did, and what the lock-out plan looked like, and when they did the lock-out and how it worked. I’ve never gone up against professional spies (except maybe some alleged North Koreans) (and Kevin Mitnick) but I’ve got to say that was exciting, sweaty, moment for the Crowdstrike guys. I would have been chanting “$500/hr, $500/hr…” under my breath, like a buddhist, to keep myself calm and focused.

After that, analysis, meetings, and production of a report. Methods, targets, assets that were exposed, sequence of events, root cause analysis, recommendations, thank you, goodbye.

Maybe I got carried away with that explanation, but I’ve been dancing around the main point: Crowdstrike did not send a team to Chappaqua, NY to Hillary Clinton’s house to do an asset seizure on her computer. They did an incident response on the DNC’s network and computers. But they were different computers!

Shift gears:

Meanwhile, something fairly similar was happening in Chappaqua, except the incident response would not have been quite as friendly: the FBI was not looking for a forensic analysis of Russian intelligence’s moves, they were looking for evidence that they could use to embarrass Hillary Clinton because using a private email server to handle government secrets is a crime (moving classified material to an unclassified system) and a violation of federal records-keeping regulations (deleting official communications). Ironically, the regulations for records-keeping were put in place after the Reagan Administration “lost” a bunch of Emails from the White House PROFS email system, and “lost” the backups and everyone had to pretend that that was not all a great big bunch of corrupt hogwash. When Bill Clinton came into office, and Hillary was getting involved in healthcare strategy (remember that debacle?) they were both briefed about federal records-keeping regulations. They had to be. I was just the guy setting up the internet email gateway for president@whitehouse.gov and I got a stern lecture about “do not destroy any data” and also “if you have something to tell someone that you want to keep off the record, catch them at the water fountain and don’t write anything down.” The Clintons knew all this stuff.

Clinton’s server was configured to allow users to connect openly from the Internet and control it remotely using Microsoft’s Remote Desktop Services.

Security-wise: a shit-show.

And, like every politician since, they ignored the fact that you’re not supposed to do corporate business on private systems. As we now know, Jared Kushner, Trump himself, Giuliani, Bolton, and goodness knows who all else have/had private email accounts, too. Of course they were not going to do their corrupt influence-peddling using government email systems! That’d be unbelievably stupid. [wik] But the whole situation rapidly got more stupid:

In the summer of 2014, lawyers from the State Department noticed a number of emails from Clinton’s personal account, while reviewing documents requested by the House Select Committee on Benghazi. A request by the State Department for additional emails led to negotiations with her lawyers and advisors. In October, the State Department sent letters to Clinton and all previous Secretaries of State back to Madeleine Albright requesting emails and documents related to their work while in office. On December 5, 2014, Clinton lawyers delivered 12 file boxes filled with printed paper containing more than 30,000 emails. Clinton withheld almost 32,000 emails deemed to be of a personal nature. Datto, Inc., which provided data backup service for Clinton’s email, agreed to give the FBI the hardware that stored the backups.

Let me explain what happened there. Clinton’s lawyers said, “oh, well, let’s fucking comply but let’s fuck with them” and they had a line of printers printing away for several days filling boxes of paper with emails from Hillary Clinton’s server. If it was me (and I’d love to know) I’d have given them all of the tactical(tm) spam and penis enlargement ads, too! It’s just paper! If they want it, let them sort if out!

But, before they could start the printers, someone devised a rule that defined “Clinton private email” versus “Clinton work email” – we do not know what that rule was, and the press are too ignorant to ask for it. Nobody sat there in Microsoft Outlook manually reviewing and deleting emails. It was done:

In 2014, months prior to public knowledge of the server’s existence, Clinton chief of staff Cheryl Mills and two attorneys worked to identify work-related emails on the server to be archived and preserved for the State Department. Upon completion of this task in December 2014, Mills instructed Clinton’s computer services provider, Platte River Networks (PRN), to change the server’s retention period to 60 days, allowing 31,830 older personal emails to be automatically deleted from the server, as Clinton had decided she no longer needed them. However, the PRN technician assigned for this task failed to carry it out at that time.

Yeah you just betcha Hillary “decided that she no longer needed them.” We are supposed to believe that Clinton’s chief of staff consulted with attorneys and then that decision got casually made by Clinton. Only a complete idiot would believe that. And here is where it starts to get interesting: the situation has a lot of complete idiots wrapped up in it.

Remember when the FBI served their warrant on Michael Cohen? That was an evidence seizure including digital asset collection. Law enforcement knows how to do that! Back in the day we had to tell cops “don’t let the hacker turn the computer off” and “oh yeah don’t you turn it off, either.” [doj] By the way, since the explosion of online child porn, if a cop points a gun at you and says “don’t touch the computer” they will shoot you if you try to touch the keyboard. They will also show up with a team of people who do incident response or forensics and a copy of ENCASE and pelican cases with hard drives, etc.

If the FBI had wanted to competently investigate Hillary Clinton’s personal server, they would have had a clown-car routine of people pouring out of black suburbans, yelling “don’t touch the computer” and exchanging coded nods with the Secret Service guard detail. Because any non-idiot in law enforcement understands that if you contact Hillary Clinton’s lawyers and ask, “do you think she has sensitive stuff in her emails?” the lawyer is going to reply “come at me, bro!” and it’s on.

It sounds like the Clinton server had multiple accounts and Clinton’s staff jiggered the expire rate of one of them and not the other. It doesn’t matter. It doesn’t matter because the FBI were never serious about getting Clinton’s emails because nobody wanted to look at that crap, anyway. Why? If there was secret State Department Stuff in there, that would have been apparent from the State Department’s email system logs. And if the State Department needed to, its server logs and the servers themselves would be backed up for disaster recovery purposes. The FBI could have asked the State Department to send along a copy. They could have contacted the Clinton’s backup service cloud provider, PRN, and told them “don’t delete anything from those systems and pop the write tab on a backup of the system and set it aside for us in case we need to come with guns and a subpoena.” Service providers get that sort of request all the time. Large providers have entire departments devoted to subpoena compliance, and they understand how to pop the write tab on a backup and put it in an envelope in a safe in the lawyer’s office. I know the head of security for a major university and I asked him once about their subpoena-rate (I was curious) and he immediately said “12 a week, mostly to do with file-sharing or software license sharing but sometimes it’s kid porn and then we have an ‘all hands on deck’ situation.”

My point is that, for things to have gone down the way they did, the FBI had to be taking idiot pills, or they simply did not care. If you contact a politician’s lawyers you know what’s going to happen: they are going to reflexively start shredding documents. [By the way, I always thought it would be a tremendously fun denial of service to send politicians faked up letters telling them to sequester data pending investigation, so that they’d immediately have massive “hard drive failures” and lose all their hard work]

Analysis:

By now you ought to be wondering something. Namely: “what the fuck is the connection between Hillary Clinton’s server and the DNC systems?”

Right? If there are copies of Hillary Clinton’s emails, there is only one plausible place for them to be: at the cloud backup at PRN. During Crowdstrike’s incident response at the DNC headquarters, they might have collected a system image including Hillary Clinton’s emails sent between her and DNC staff, but that would be the limit of it. There are two separate servers, and the data collected from either/each would be quite different, and the practices followed in that collection were extremely different. In the Crowdstrike response, they may have captured system images, whereas in the Clinton server examination, they got 12 boxes of paper from lawyers.

Apparently Trump and his idiots have some idea that the Ukrainians have Clinton’s missing emails, although if you look at Trump’s language it appears that he thinks that the Ukrainians have some actual “server” – i.e.: a computer. Does Donald the Dunce imagine that – well, what? I can’t even come up with a silly theory that fits with the facts.

Here is what I think is going on: we have some profoundly ignorant people who understand nothing about computers, networking, or data, coming up with a conspiracy theory that doesn’t even make sense – but they are so ignorant and stupid that they don’t realize how stupid they are. This is how stupid American politicians are about technology:

And Donald Trump is a particularly stupid American politician.

For one thing, he’s so stupid that he can’t imagine that Clinton’s people, who had weeks to scrub the emails off her machine (never mind the backups) were able to do that without stupiding up, somehow.

I may have just come up with a theory that works: when Clinton’s people printed out the 12 file boxes of paper with the 31,000 emails they actually printed all of them: 24 boxes in total. Then, they were going to shred 30,000 of the sheets of paper, but Crowdstrike showed up and… no, shit, the timing doesn’t work. Maybe the FBI took the 12 secret boxes of paper and had them in the back of their suburban and then they were going to ship them to the DNC but instead they shipped them to Ukraine?

Wrap your brain around this: Donald Trump has been willing to burn his presidency over this thing that he completely fails to understand. That’s bad, but his buddy Rudy also fails to understand it. And the media – the watchdogs of public discourse – can’t seem to ask the basic question that needs to be asked:

“What the fuck do you think is going on here!?”

The Ukrainians didn’t ask, either.

I bet the folks at Crowdstrike are more puzzled than one of those 10,000-piece all-black borderless puzzles.

Remember: none of this had to happen. If the FBI had actually been doing a proper investigation of Clinton’s personal email server, all they had to do was not screw things up, and recover the emails from the backups at PRN. These political dipshits always forget the backups; that’s what jacked Reagan up (except that congress then went on to pretend that there were no backups and Ollie North stepped forward and said “I’ll take the blame” and everything went under the rug. Remember: the Clintons lived through Iran/Contra (as did I) and they have to remember that computer systems have backups. The Clintons are pretty fucking stupid, too.

I used to know the head of security at one of Savvis’ big data warehouses. At that time, Savvis owned a bunch of hosting services including Exodus Communications. In other words, huge hosting services were a subset of their hosting service. Eric H. said that their data warehouse is a gigantic steel building in Washington state, near a power plant. It has a barbed wire chain-link fence around it and a single gate with a security guard. It has a single door and no windows. There are offices in the front of the building for the staff and systems folks but otherwise the building is full of rack upon rack of computers. So, as Eric told the story, he gets a call from the security guard and the security guard is upset, “Hey boss? There are some guys here who say they are from the FBI and they are getting really upset.” Eric drops everything and runs to the gate. There are two black suburbans and a crown vic. He introduces himself to the FBI guys and the FBI guy says, “ah, so you’re the right person for us to be talking to. We’re here to seize the computer.”
Eric: “Which … computer?”
FBI Guy: “All of them.”
Eric: “Surely you do not understand something. Come take a walk with me and we can talk about this computer you are looking for.”
FBI Guy: “No, You are NOT GOING IN THERE because you might shut it down.”
As the situation evolved Eric called some other FBI people he knew who were not complete idiots and the whole situation got sorted out after a bunch of blustering, threats from the FBI idiots, and phone calls. He said it was profoundly disturbing to have people with guns who were so ignorant. And Trump makes those FBI idiots look like John VonFucking Neumann.

Here are pro tips if you work someplace where you have to worry about lawsuits and records:

• Do not use email
• Do not use a computer
• Write everything down in paper composition notebooks
• Give the notebook to your client when you clear the site and tell them, “I’d burn that if I were you.”

Do not pray Voltaire’s prayer. It’s overpowered.