Today the news is all a’flutter about the massive malware/extortion attack on hospitals all over the place. The sky is falling!
What’s going on is that a lot of companies have really bad security, and have comfortably had it for a long time. Medical informatics is particularly hard because of a “these hands have been touched by god!” attitude from doctors (doctors are probably the second worst ‘users’ you can have, after politicians, in terms of blockheaded self-importance and risk profile) coupled with industry regulation that makes it harder to update software on devices. There’s a really bad negative synergy with device certification: to get lawyers to sign off that a device is suitable for hospital use it has to be in a specific, documented, configuration – which means that if your device is certified, you don’t patch it or update it. So there are plenty of patient monitors out there running old versions of Windows that a hacker can blow straight through. In principle, that would be OK except that those devices are not on isolated device networks, so if Dr Jones clicks on a PDF from patientrecords@quest_diagnostics.com that says “X-ray results for your patient” and gets their machine owned, now the hacker is on the backend network where everything is reachable.
There are a lot of obvious things that can and should have been being done, but the certification process, plus “oh, we have a firewall so it’s OK!” concept have act to block any demand for better security. There is still a great deal of “hard shell around a soft, chewy center” as Bill Cheswick described firewalls back in the mid 1980s. Some organizations realized that malware was going to be a problem, and others didn’t. The two that have had their heads most thoroughly in the sand are medical, and government. Often, you’ve got a situation where a hospital network, with thousands of authorized users, is going to be completely open shortly after one user clicks on the wrong attachment. Worse, some of the devices and internal servers are not treated as core infrastructure and are not under system administration.
My first encounter with computer security was when I was a young systems-pup at Johns Hopkins Hospital in 1987, managing a cluster of Sun-2/110s and a couple of Pyramid 9080s, and one of my Pyramids started getting login and FTP attempts from another Sun system that I recognized was one of the controllers for the research MRI machines. It was highly unusual for an MRI to be trying to log into things, so I picked up a phone… It turned out later that someone had put a connection to the University, so as to be able to exchange email with the medical school faculty, and Johns Hopkins University was an early ARPAnet node because of the H-bomb designers in the Applied Physics Lab. (I was on the “internet” in 1981 as an undergrad) Someone had come in through the Universary, broken into a default account on the MRI, and was having a look around. We put some filtering rules in the router and changed a lot of passwords but generally nobody got very excited about security until the mid 1990s.
Excitement about security comes in waves. I’ve always thought it’s basic humdrum but that’s because I’ve always thought it was a matter for consistent effort not a panic/respond/panic/respond cycle. Right now, there’s a lot of panicking and responding going on. It’s an International Cyberattack.[nyt] Actually, no, it’s just a large-scale extortion campaign in which a bunch of hackers have decided to act on a lot of targets at once. This is nothing new: the low-hanging fruit are getting hit. As I said [stderr] last summer: if you can get hit by cryptolocker, you’re going to. So be prepared or be prepared to suffer. If you’re an IT practitioner and your organization doesn’t do its backups, you’re a candidate for losing your job and having it migrate to the cloud. Not a candidate: it’s inevitably your future.Edward Snowden, unfortunately, weighs in and he’s got an axe to grind:
This is really not the NSA’s fault. It’s a terrible idea that they maintain and build attack tools, and it’s really incredibly stupid that they can’t keep them secret. The real problem is not the specific vulnerability, it’s that hospitals have such lamentable endpoint security that people are falling for basic phishing exploits and their servers aren’t immediately recoverable.
Cloud systems have similar problems. I was talking to an organization the other day that had a lot of stuff out at AWS and was extorted for a lot of money by an attacker who discovered that they had all their system images and all the backups of their systems images in the cloud. So, the extortionist deleted the backups then asked money not to delete the remaining images. As I said last summer: unless you have three copies of your data and one is offline, and none of them are in the same place, you don’t really own your data you’re just borrowing it.
I don’t like that Snowden tried to make this about NSA. It’s more about mediocre system administration practices. And, if you’re a patient or you have medical records or are about to lie down on a table for some tele-operated surgery, you might want to think about that. Snowden asking about NSA building attacks against Windows XP is missing the point that: hospitals should not be running Windows XP. And doctors shouldn’t be reading their email with local adminstrator privileges, etc. There’s a whole great shovel-load of basic security precautions that are apparently not being taken at some very important hospitals around the world.
What’s the most boring topic in IT security? Configuration Management (CM). Good CM practices are what separate Amazon and Google and ADP and Facebook from the rest of us. When you hear about Google’s model for repairing a failing system (“unplug it and slap a new one in”) or you use Amazon’s elastic server cloud – that’s all driven by automated system administration. They have a systems administration fan-out where it’s not one administrator to a cluster of five machines (like I was back in my distant hospital days) it’s one administrator to five thousand machines. And the notion of “machine” has become a squishy concept – it’s more like “capabilities” in the cloud. Stuff to do things to gets routed to places where stuff gets done and it’s all dynamic.
Computer security, as a problem, exists largely because system administration is harder than it ought to be, and organizations have been cheap about system administration and configuration management – they’d done IT at an amateur level, to save costs – and they’ll do it as long as the perceived costs of being bad at IT don’t overwhelm their imagined cost-savings by moving from mainframes to departmental computing and local data centers. The cloud is mainframe 2.0: a massive well-managed computing infrastructure where the user can’t screw up the configuration. That we have this problem at all is because organizations have been able to cheap out about the costs, and continue to pass the costs of their failures on the customer.
This melt-down that the hospitals are experiencing is an outcome of bad cost/benefit analysis.
It would have been cheaper to take the downtime to update systems. It would have been cheaper to put critical systems on separate networks. It would have been cheaper to actually design your networks for performance and reliability instead of letting them evolve based on what’s easiest to get working. Network administration, like system administration, is not just plugging a wire into a switch and patting yourself on the back when the green LED comes on. Did you notice how this outbreak affected the cloud players? Yeah, me neither.
Either you will professionalize and automate your system administration and configuration management, or you will continue to suffer unpredictable periodic catastrophes, and your business will fail or your IT organization will get plowed under and moved to the cloud.
I think NSA and CIA and other intelligence agencies’ “HACK THE WHOLE WORLD” attitude is short-sighted and stupid because it does result in security being weakened somewhat. But worrying about weakening it somewhat is silly when you’re talking about organizations that don’t have good disaster recovery, backups, endpoint control, and configuration management. It’s like blaming the particular bridge pylon you slam into when you’re drunk driving.
NSA is one of the agencies that is partially responsible for helping improve US government security. Trying to hack the whole world certainly doesn’t do that. And, as I’ve argued elsewhere, the best defense is a strong defense. Why are government agencies’ (including the CIA and NSA!) security so lame? Why are hospital’s security at such a level of lame? Defending is harder than attacking, and configuration management is a critical defensive technique that everyone is unwilling to do because “it’s hard” or “the users will complain.” Just remember that when you lie down for tele-operated surgery.
A tip ‘o the hat to Shiv, who triggered this rant.
As someone with a hospital trip planned for the near future, I needed that. :P
IT administration is kind of like that old Sun Tzu story of the three brothers who are doctors. The least competent is the most widely known, while the most competent gets the job done before there’s a problem, and is only locally known. I have little confidence the incompetent people will actually be punished or lose their jobs. After all, the Wall Street Journal has told the bosses just how difficult guarding against such attacks are, so when their incompetent IT managers make enough of a show getting things working again, working long hours through the night, they will keep their jobs and may even get a raise for all their hard work.
The places where IT is done correctly will hum along and the IT manager will be given a bad review for not working hard enough (aka being seen in the office early and late each day).
I’ve seen that far too often in the world.
:Snort: I am so unsurprised. People at the pain clinic pestered me for ages to register with their patient site on their network, so I could do things like put in my med requests and such online. My answer was a flat no, every time. They finally stopped bugging me about it. It would make things easier for me, but there’s just some stuff I will not do online.
Reginald Selkirk says
Like the old Far Side cartoon
Marcus Ranum says
I think the original reference was to a marketing campaign for some kind of chocolate truffle in the late 70s. I don’t remember which one Bill was referring to, though.
Reginald Selkirk says
What about the ransom aspect? Supposedly victims are getting a “pay up in Bitcoin and we’ll give you the decryption key” message. I don’t know if they will actually deliver, but is the Bitcoin transaction truly untraceable? I imagine for an episode of this scale, every available effort will be made to find the perpetrators.
It’s not (only) the IT departments fault. Especially in hospitals you have lots of external regulations which prevent good security (as written in the post). System administrators now have to suffer twice, because they have to work overtime fixing problems caused by security issues they have probably pointed out for years.
Marcus Ranum says
I checked with Ches and he says the expression was being widely used around Bell Labs at the time (there were a few other folks working on security before Ches and SMB got into firewalls, including Dave Presotto who wrote the circuit relays they used for a number of years)
Short form: origin of the expression remains unknown.
Marcus Ranum says
is the Bitcoin transaction truly untraceable?
Maybe kinda sorta. There are ways it might be tracked.
The interesting thing is that supposedly the attackers have only netted a few tens of thousands of dollars. It looks like their “carpet bomb and profit” scheme has not worked out very well.
I am fairly sure they’ll get caught. There’s going to be a lot of retroscoping going on right now. If you look at the history of big problems like this, it’s generally the case that the perpetrator is identified (which ought to tell you something, I’m not sure what)
Marcus Ranum says
System administrators now have to suffer twice, because they have to work overtime fixing problems caused by security issues they have probably pointed out for years.
Probably triply. They’re going to have to work overtime on a problem they already warned management, and then they’re going to see their jobs outsourced to the cloud in retaliation for management’s mistakes.
I wouldn’t be at all surprised to start hearing people talk about banning Bitcoin after this…
John Morales says
Marcus, there’s a new post up at Charlie’s Diary.
Marcus Ranum says
The US government has already been doing what it can to suppress and devalue it. There are probably a few people on earth who think the NSA or CIA had absolutely nothing to do with the Mt Gox heist.
I’m not one of them.
Unless it was Russian hackers haw haw.
My guess is CIA did it and are using the “missing” $mm worth of bitcoin to buy vulnerabilities and malware from the dark codeshops. I label that as a “guess” because I freely admit I have no evidence at all.
Marcus Ranum says
I love Charlie’s take on it.
Good discussion there too. I move in different circles than the entities that are getting whacked by this thing. I’m not saying they deserve it but it’s not unexpected. Some of the companies that are getting clobbered have been negligent and are learning their cost/benefit calculation was wrong.
Marcus @13: TBH, I never looked that closely into the Mt Gox thing, because the whole affair was such a fractal clusterfuck of wrongness that it makes me nauseous just thinking about it. Sure, coulda been the spooks, but given the general level of (in)competence on display, it wouldn’t surprise me if Karpeles’ admin password was “password1234”, or he forgot to de-activate McCaleb’s account (which probably also had the password “password1234”). I wouldn’t trust any of these people to run a whelk stall, never mind look after hundred of millions of dollars of other people’s money (or “money”, if you prefer), and the concept of people having their stashes of untraceable cryptocurrency stolen just reeks too much of “I never dreamed panthers would eat my face!” for me deal with without suffering potentially dangerous levels of schadenfreude…
Call me old-fashioned, but I kinda like my financial services to be regulated, at least as a general concept.