Are you tired of hearing about Email servers, yet? And Email leaks?

I sure as hell am; that’s the background music of my entire professional career. Once more unto the breach, dear friends…

Back in the winter of 1992/3 when I was a young puppy working at Trusted Information Systems, I had just come down off of building the first commercial internet firewall product (DEC SEAL) when I was at Digital, and TIS’ CEO, Steve Walker, got a call from his main contact at DARPA asking “do you have anyone there who knows anything about ‘firewalls’?”  Well, yes.

We had a meeting in his office and agreed to go down to DARPA’s offices (AKA: “The Enterprise”) in Virginia, and talk about it, and I was too full of ideas to sleep so I stayed up all night and wrote a thumbnail sketch of a proposal for how to do email/connectivity/security for, well, any remote executive team. At the time, VPNs* weren’t a thing; I had experimented with building encrypted network-layer software when I was at DEC, using some packet tunnelling code from Julian Onions and rather cleverly didn’t patent any of my work.**  So I put together a rough architecture which we left behind with the DARPA folks, who called back a couple days later and said “do it.” In the end, it turned out we didn’t do the whole architecture; they really just used the research project as a slush fund for buying the White House a T-1 line to UUnet, and setting up an Email server, which I did on the massively overpowered Sun4/M “whitehouse.gov” which lived next to my desk in Glenwood, MD until we moved it to the computer room.

The overall architecture I proposed included VPN and encrypted remote access but was primarily built around the idea that the email server would be isolated to a purpose-built network that was monitored for any traffic not entirely related to Email. It was to be an enclosed and monitored system, and I did some early work on what later became called “intrusion detection”*** – the idea being to flag any traffic that did not exactly match the known patterns of SMTP clients talking to a known server. In other words, it was going to be a super-secure Email service – what we’d now call a “cloud server” except managed as a very small enclave for a limited clientele.

The ideas haven’t changed much. That’s why I am so disrespectful of idiots like the DNC and RNC and the Clintons, who fall into the trap of doing “basic IT 101″ wrong. They don’t want to use Google because they’re scared (in spite of Google’s security being better than theirs) because they know the entire intelligence apparatus of multiple nations is oriented toward hacking the big cloud services. So they built their own, and did it wrong. What’s sad is that they could practically dust off the roadmap I did in 1993, update it a little bit for new technology, and rock it.

So, here we go:

Set up a server at a moderately secure facility. Use a reliable and stable operating system, not that horrible Windows garbage that downloads patches constantly and reboots constantly. In 1993 I was using BSDi, today I’d use OpenBSD (which is derived from the same code). Put a next-generation firewall like a Palo Alto in front of it, seal the whole thing into a 19” rack, and add another box below it (also running OpenBSD) that’s listening to the traffic between the internet and the mail server on a capture rule from the firewall. That system also serves as a syslog listener to collect the firewall logs, Email server logs, and collects packets that do not match a set of tcpdump rules that amount to a whitelist of what traffic is allowed. Traffic profile: DNS permitted to/from a well-known caching DNS secondary, SMTP inbound to a mail collector process, SMTP outbound to internet. IMAP allowed in-bound from the virtual address of the VPN on the firewall. Administrative traffic (root logins, etc) via a second VPN virtual address on the firewall, only. The span-watcher fires alarms if any other traffic is seen on the interior network. Plus, the Email server is set up with a set of firewall rules using the in-kernel firewall, to mirror the edge firewall rules: if a firewall “drop” rule fires on the Email server, ever, that triggers an administrative alert, kernel memory dump, and complete system shutdown. The firewall is also set to detect, block, and red-flag if the span-watcher ever generates any outbound or inbound traffic at all: it’s only a syslog sink.

Email collection and processing managed with Postfix running unprivileged and chrooted. Add to the processing loop some attachment stripping and quarantining – no attachments with executable content go through. Set the IMAP server software up to run chrooted into the mail spool directory. Then turn on execution logging and forward it to the listener box. On the listener box set up syslog-NG and do some log whitelisting. A cool trick for that, which I’ve done many times, is to have your log-watcher process look for the first system date/time stamp on boot, then whitelist every log message that comes after that and send the greylist materials to a security operations team for analysis.

I’ve left a few details out, namely things like that the server should probably be booting off mirrored drives and if they’re large enough the email spool directory can be sub-partitions of those drives. We’re talking just email, so there’s no need for hundreds of gigabytes of space. (My entire email input and output since 1993 – including the spam and attachments – is 30gb)****

If secure/secret calendaring is also necessary, add another small OpenBSD box (do not put it on the same box as the email!) running some webby calendar stuff, with a firewall rule only allowing the inner virtual address of the VPN to access its HTTP port. Run apache chrooted and unprivileged, then do the “whitelist the logs” and whitelist the IP connections dance.

That leaves the endpoints. Endpoints are the most popular point of attack because people – especially important people – are stupid as buckets of rocks. So you give them a laptop that’s configured with a desktop firewall that can only go out the VPN to the enclave, and that has runtime controls so that all it can do is browse the calendar app and run an IMAP client. Then stick a randomly-generated passcode into the VPN and randomly generated large passwords into the IMAP client and calendaring app, and: you now have a plausibly secure Email and portable calendaring terminal. You don’t need to change your password if your password is godawful complex and you’re using good crypto and have mechanisms in place to detect cryptographic break-ins (in the scenario I outlined, what happens if the NSA burns an exploit against the VPN and gets into the network?)  The problem with “general purpose computing” is the idea that a computer is pluripotent: it can do anything. Well, that means it can also hack you. If your user-base is accustomed to generating all kinds of traffic all over the place and doing all sorts of complicated stuff, then the patterns of how they operate on the network are going to also be all over the place and complicated, so it’s impossible to sort the good traffic from the bad traffic. In Marcus-terms “it’s all bad traffic” at a certain point.

Oh, you wanted to also surf the web with that portable device? No. Use something else. Have one of your minions carry your spare blackberry, Hillary. You’d be president today if you and the DNC weren’t so stupid about IT. When your web-surfing endpoint gets owned, as it inevitably will, I will say “told you so”***** and you’ll say “well thank goodness they didn’t get into my email.”

Total set-up time for a system/architecture like I describe above is a couple days for a good system administrator, and a small number of thousands of dollars. Since Email is low bandwidth, you don’t need fancy firewalls or servers with a lot of oomph. Total set-up time to secure a Windows-based Exchange server?  Ugh. But if you did it with locked down endpoints and the VPN/no other traffic, it would probably not be easy to get into, either, though Microsoft products have a horrible tendency to barf packets all over the place, patch themselves, and reboot. Who’d want to use unreliable crap like that?

The “administrative terminal” is presumably a laptop running the VPN client, locked down so that all it can run is ssh, with firewall rules blocking any traffic that’s not going to/from the VPN virtual interface. Essentially, it’s a tele-present remote console, and normally would live in a cool-ass locked Halliburton briefcase somewhere.

Single-purpose computing is the only way to go, for certain important processes. What’s sad about all these Email breaches is that a lot of people simultaneously say Email is very important to them, but they aren’t willing to treat it like it is. When you have a locked-down single-purpose device then you have the option of treating is as a “2 factor authentication” – something you know plus something you have or even a single factor authentication: something you have. Hillary Clinton’s blackberry – which was owned 19 ways to Sunday by the NSA – was a single-factor authenticated single-purpose computer. Your endpoint security and your server security have to match.

At the time, I used a piece of software written by Bob Braden and Annette DeSchon when they were at ISI, called “NNstat” – it was a general-purpose network statistics gathering system. You could set it up to collect whatever data you wanted from packet headers going by and add them to spreadsheets of counts (or whatever) so I set NNStat up on the network monitoring whitehouse.gov’s traffic and building some great big charts about connectivity. It was kind of interesting but got boring in a few days, so I switched over to outputting the difference between charts – I very quickly established what amounted to a whitelist of “normal” connectivity and if anything did anything that fell off the “normal” chart, the added lines would appear in my email in-box immediately. I found all kinds of interesting stuff including a few packets that probably should not have made it successfully across the internet to get to the system, at all. This technique is basic white/black/grey-list processing. Usually, when I talk about the technique someone raises a hand and says “that sounds hard.” Yeah, it takes work. But getting hacked to bits all the time and having to do incident response isn’t exactly easy, now, is it?

“EOP.gov” is Executive Office of The President”; I was the original domain registrar of that, and whitehouse.gov. Funny story: I also suggested registering whitehouse.com, but the EOP staff told me not to do that because “if anyone uses that domain we’ll send them a cease-and-desist.”  Being the nice boy scout I am, I didn’t register it and keep it like I should have.

Exercise for the reader: the syslog box could act as a server for rdump backups from the email server if you don’t think mirrored disks are good enough.

The only part of my 1993 roadmap that gets superceeded is the VPN replacing kermit. I actually hacked a pretty simple and cool authentication and key exchange/bulk encryption into kermit, making it a highly secure remote terminal emulator. It turned out that terminal emulation was no longer where it was at, when the WWW came along.

(* I called them VNPs for “Virtual Network Perimeter” – an entirely better name.)

(** Whups!)

(*** I started a company called “Network Flight Recorder” in 1997 based on some ideas I had during the whitehouse.gov rollout. It was one of the top intrusion detection companies of that era but now nobody’s probably ever heard of it.)

(**** Yes, I keep it all. Unfortunately there are a few gaps or I’d have it going back to the first emails I sent in 1980, barring the stuff for various jobs where I left my work-email only on work systems.)

(***** AKA: “the threat display of the IT security practitioner”)