More About Email Servers

Are you tired of hearing about Email servers, yet? And Email leaks?

I sure as hell am; that’s the background music of my entire professional career. Once more unto the breach, dear friends…

Back in the winter of 1992/3 when I was a young puppy working at Trusted Information Systems, I had just come down off of building the first commercial internet firewall product (DEC SEAL) when I was at Digital, and TIS’ CEO, Steve Walker, got a call from his main contact at DARPA asking “do you have anyone there who knows anything about ‘firewalls’?”  Well, yes.

fwhtWe had a meeting in his office and agreed to go down to DARPA’s offices (AKA: “The Enterprise”) in Virginia, and talk about it, and I was too full of ideas to sleep so I stayed up all night and wrote a thumbnail sketch of a proposal for how to do email/connectivity/security for, well, any remote executive team. At the time, VPNs* weren’t a thing; I had experimented with building encrypted network-layer software when I was at DEC, using some packet tunnelling code from Julian Onions and rather cleverly didn’t patent any of my work.**  So I put together a rough architecture which we left behind with the DARPA folks, who called back a couple days later and said “do it.” In the end, it turned out we didn’t do the whole architecture; they really just used the research project as a slush fund for buying the White House a T-1 line to UUnet, and setting up an Email server, which I did on the massively overpowered Sun4/M “” which lived next to my desk in Glenwood, MD until we moved it to the computer room.

The overall architecture I proposed included VPN and encrypted remote access but was primarily built around the idea that the email server would be isolated to a purpose-built network that was monitored for any traffic not entirely related to Email. It was to be an enclosed and monitored system, and I did some early work on what later became called “intrusion detection”*** – the idea being to flag any traffic that did not exactly match the known patterns of SMTP clients talking to a known server. In other words, it was going to be a super-secure Email service – what we’d now call a “cloud server” except managed as a very small enclave for a limited clientele.

The ideas haven’t changed much. That’s why I am so disrespectful of idiots like the DNC and RNC and the Clintons, who fall into the trap of doing “basic IT 101″ wrong. They don’t want to use Google because they’re scared (in spite of Google’s security being better than theirs) because they know the entire intelligence apparatus of multiple nations is oriented toward hacking the big cloud services. So they built their own, and did it wrong. What’s sad is that they could practically dust off the roadmap I did in 1993, update it a little bit for new technology, and rock it.

So, here we go:

Set up a server at a moderately secure facility. Use a reliable and stable operating system, not that horrible Windows garbage that downloads patches constantly and reboots constantly. In 1993 I was using BSDi, today I’d use OpenBSD (which is derived from the same code). Put a next-generation firewall like a Palo Alto in front of it, seal the whole thing into a 19” rack, and add another box below it (also running OpenBSD) that’s listening to the traffic between the internet and the mail server on a capture rule from the firewall. That system also serves as a syslog listener to collect the firewall logs, Email server logs, and collects packets that do not match a set of tcpdump rules that amount to a whitelist of what traffic is allowed. Traffic profile: DNS permitted to/from a well-known caching DNS secondary, SMTP inbound to a mail collector process, SMTP outbound to internet. IMAP allowed in-bound from the virtual address of the VPN on the firewall. Administrative traffic (root logins, etc) via a second VPN virtual address on the firewall, only. The span-watcher fires alarms if any other traffic is seen on the interior network. Plus, the Email server is set up with a set of firewall rules using the in-kernel firewall, to mirror the edge firewall rules: if a firewall “drop” rule fires on the Email server, ever, that triggers an administrative alert, kernel memory dump, and complete system shutdown. The firewall is also set to detect, block, and red-flag if the span-watcher ever generates any outbound or inbound traffic at all: it’s only a syslog sink.

How we did VPN in 1991

How we did VPN in 1991

Email collection and processing managed with Postfix running unprivileged and chrooted. Add to the processing loop some attachment stripping and quarantining – no attachments with executable content go through. Set the IMAP server software up to run chrooted into the mail spool directory. Then turn on execution logging and forward it to the listener box. On the listener box set up syslog-NG and do some log whitelisting. A cool trick for that, which I’ve done many times, is to have your log-watcher process look for the first system date/time stamp on boot, then whitelist every log message that comes after that and send the greylist materials to a security operations team for analysis.

I’ve left a few details out, namely things like that the server should probably be booting off mirrored drives and if they’re large enough the email spool directory can be sub-partitions of those drives. We’re talking just email, so there’s no need for hundreds of gigabytes of space. (My entire email input and output since 1993 – including the spam and attachments – is 30gb)****

If secure/secret calendaring is also necessary, add another small OpenBSD box (do not put it on the same box as the email!) running some webby calendar stuff, with a firewall rule only allowing the inner virtual address of the VPN to access its HTTP port. Run apache chrooted and unprivileged, then do the “whitelist the logs” and whitelist the IP connections dance.

That leaves the endpoints. Endpoints are the most popular point of attack because people – especially important people – are stupid as buckets of rocks. So you give them a laptop that’s configured with a desktop firewall that can only go out the VPN to the enclave, and that has runtime controls so that all it can do is browse the calendar app and run an IMAP client. Then stick a randomly-generated passcode into the VPN and randomly generated large passwords into the IMAP client and calendaring app, and: you now have a plausibly secure Email and portable calendaring terminal. You don’t need to change your password if your password is godawful complex and you’re using good crypto and have mechanisms in place to detect cryptographic break-ins (in the scenario I outlined, what happens if the NSA burns an exploit against the VPN and gets into the network?)  The problem with “general purpose computing” is the idea that a computer is pluripotent: it can do anything. Well, that means it can also hack you. If your user-base is accustomed to generating all kinds of traffic all over the place and doing all sorts of complicated stuff, then the patterns of how they operate on the network are going to also be all over the place and complicated, so it’s impossible to sort the good traffic from the bad traffic. In Marcus-terms “it’s all bad traffic” at a certain point.

Oh, you wanted to also surf the web with that portable device? No. Use something else. Have one of your minions carry your spare blackberry, Hillary. You’d be president today if you and the DNC weren’t so stupid about IT. When your web-surfing endpoint gets owned, as it inevitably will, I will say “told you so”***** and you’ll say “well thank goodness they didn’t get into my email.”


Total set-up time for a system/architecture like I describe above is a couple days for a good system administrator, and a small number of thousands of dollars. Since Email is low bandwidth, you don’t need fancy firewalls or servers with a lot of oomph. Total set-up time to secure a Windows-based Exchange server?  Ugh. But if you did it with locked down endpoints and the VPN/no other traffic, it would probably not be easy to get into, either, though Microsoft products have a horrible tendency to barf packets all over the place, patch themselves, and reboot. Who’d want to use unreliable crap like that?

The “administrative terminal” is presumably a laptop running the VPN client, locked down so that all it can run is ssh, with firewall rules blocking any traffic that’s not going to/from the VPN virtual interface. Essentially, it’s a tele-present remote console, and normally would live in a cool-ass locked Halliburton briefcase somewhere.

Single-purpose computing is the only way to go, for certain important processes. What’s sad about all these Email breaches is that a lot of people simultaneously say Email is very important to them, but they aren’t willing to treat it like it is. When you have a locked-down single-purpose device then you have the option of treating is as a “2 factor authentication” – something you know plus something you have or even a single factor authentication: something you have. Hillary Clinton’s blackberry – which was owned 19 ways to Sunday by the NSA – was a single-factor authenticated single-purpose computer. Your endpoint security and your server security have to match.

At the time, I used a piece of software written by Bob Braden and Annette DeSchon when they were at ISI, called “NNstat” – it was a general-purpose network statistics gathering system. You could set it up to collect whatever data you wanted from packet headers going by and add them to spreadsheets of counts (or whatever) so I set NNStat up on the network monitoring’s traffic and building some great big charts about connectivity. It was kind of interesting but got boring in a few days, so I switched over to outputting the difference between charts – I very quickly established what amounted to a whitelist of “normal” connectivity and if anything did anything that fell off the “normal” chart, the added lines would appear in my email in-box immediately. I found all kinds of interesting stuff including a few packets that probably should not have made it successfully across the internet to get to the system, at all. This technique is basic white/black/grey-list processing. Usually, when I talk about the technique someone raises a hand and says “that sounds hard.” Yeah, it takes work. But getting hacked to bits all the time and having to do incident response isn’t exactly easy, now, is it?


“” is Executive Office of The President”; I was the original domain registrar of that, and Funny story: I also suggested registering, but the EOP staff told me not to do that because “if anyone uses that domain we’ll send them a cease-and-desist.”  Being the nice boy scout I am, I didn’t register it and keep it like I should have.

Exercise for the reader: the syslog box could act as a server for rdump backups from the email server if you don’t think mirrored disks are good enough.

The only part of my 1993 roadmap that gets superceeded is the VPN replacing kermit. I actually hacked a pretty simple and cool authentication and key exchange/bulk encryption into kermit, making it a highly secure remote terminal emulator. It turned out that terminal emulation was no longer where it was at, when the WWW came along.

(* I called them VNPs for “Virtual Network Perimeter” – an entirely better name.)

(** Whups!)

(*** I started a company called “Network Flight Recorder” in 1997 based on some ideas I had during the rollout. It was one of the top intrusion detection companies of that era but now nobody’s probably ever heard of it.)

(**** Yes, I keep it all. Unfortunately there are a few gaps or I’d have it going back to the first emails I sent in 1980, barring the stuff for various jobs where I left my work-email only on work systems.)

(***** AKA: “the threat display of the IT security practitioner”)


  1. says

    I think that it is literally unthinkable to most people working in the industry today that you couldn’t surf Facebook and Instagram from every device in your diagrams.

    People check Instagram twice a minute for hours in places where they have no connectivity.

    Now, would they reject your architecture those grounds? Of course not. But reject it they do, and they invariably select an a alternate that allows they social media access.

    Observe what happens most Google apps when they drop offline. It never occurs to these morons to test that case, because offline is equivalent to death or something. There is one that gets it right. I decline to name it for fear of jinxing it.

  2. EigenSprocketUK says

    Why keep all that email? As Schneier says, it’s a liability more than it’s an asset. Hard to let go, I know. But if you think it’s still got value, say if you analysed it or mined it, then go ahead and do that. Then you know if you have something worth keeping. But seriously think about throwing the raw stuff away.

  3. Dunc says

    You’d be president today if you and the DNC weren’t so stupid about IT.

    I’m pretty sceptical about this bit. I doubt that it really made that much difference.

  4. says

    Why keep all that email?

    Curiousity, and I use it as a long-term backup for lots of memories. There are certain things I deliberately don’t bother remembering because I know that it’s in my email archive and I can search it out with some simple queries if I ever need it. It’s my offline memory.

    FWIW my email achive lives on an encrypted volume that is generally locked, so it’s not like someone could grab it all easily and troll through my past indiscretions.

    As Schneier says, it’s a liability more than it’s an asset.

    It’s definitely complicated. I actually had a big realization about Email many years ago, which was that I should assume it’s all open anyway. There are relatively few things in my email archive that would be problematic if they were discovered. And that’s largely because I decided that if certain things from it got published I’d be embarrassed for a minute or two and then go “yeah, I did say that.” So it’s not a very big liability. To Dunc’s point @#3: it really doesn’t appear to matter much what anyone says or does, anyway.

  5. says

    Andrew Molitor@#1:
    People check Instagram twice a minute for hours in places where they have no connectivity.

    I’m talking about “important” people, not Kim Kardashian.

  6. Sunday Afternoon says


    Given the track record of the republicans, if it wasn’t the emails, they would certainly have found some other tenuous “scandal” to focus attention on. Just don’t look at the real scandals that are developing even before their guy is in office…

  7. Raucous Indignation says

    Email archives are a peripheral brain. We all use those all the time, from little yellow sticky notes to keeping our high school yearbooks. Just go with it.

  8. Dunc says

    Given the track record of the republicans, if it wasn’t the emails, they would certainly have found some other tenuous “scandal” to focus attention on.

    Hell, even with the emails, the “scandals” that seem to have gained the most traction are the completely absurd “Spirit Cooking” thing, and the entirely imaginary “Comet Ping Pong” paedophile ring. These people are not well-anchored in reality.

  9. says

    Sunday Afternoon@#6:
    But if the emails didn’t matter, then the russians’ messing with the elections doesn’t matter? Or, wait, is it the demicans complaining about that, or the republicrats? I can’t even keep them straight in my head anymore.

    I guess I agree with those of you who think I misspoke on my “Hillary would be president…” comment. Trump’s base would have found something else to get excited about, no matter what.

    It still galls me that email security, being a relatively simple problem to solve, is still such an issue for those that ought to know better. It’s bizzare. But then I remind myself that we had cars that could go 80+ miles per hour for decades before even basic seatbelts were available. It’s probably going to take 60-70 years before people start building safe computing platforms. What’s nuts is that they could do it now, but they won’t because people would rather have flash-based apps or some nonsense like that.