More About the SolarWinds Breach

This may be a bit scatter-shot; there’s a lot to cover, and I’m going to try to fold in some answers to comments on my previous posting on the topic. [stderr] I also want to predict the future, so I can say “I told you so!” when it happens.

To be frank, cybersecurity pisses me off so much whenever I think about it, that it’s almost painful to write about. But the questions are interesting and worthy of respect.

Our first item, in line with my paper on The Anatomy of Security Disasters, is that disasters are almost always worse than they look, at first. And, when the disaster is discovered, people like me come out of the woodwork and say “I told you so!” Also, people who were involved in the gestation of the disaster dig up and hand out old memos that say, “this is going to be a disaster, don’t do that.” But, of course, that was exactly what happened and now we’re just arguing about how late “too late” is.

One of the other bits of information that came leaking out is, apparently, someone who was involved with reviewing SolarWinds’ security says that it was pretty bad all along. [bloomberg]

A former security adviser at the IT monitoring and network management company SolarWinds Corp. said he warned management of cybersecurity risks and laid out a plan to improve it that was ultimately ignored.

In a 23-page PowerPoint presentation reviewed by Bloomberg News, Ian Thornton-Trump recommended to company executives in 2017 that SolarWinds appoint a senior director of cybersecurity, and said he told them that “the survival of the company depends on an internal commitment to security.”

I have archives of every communication I ever had with a client (most security consultants do, for exactly this reason) including some presentations that say basically the same thing. I remember one client (I was on a technical advisory board) decided my participation was no longer valuable, after I pointed out that having the developers roll their own O/S releases using Kubernetes was “just one more thing for them to be bad at.” Maybe I should have been more diplomatic, but there’s an awareness that develops, which is “these guys aren’t going to listen to me no matter what I say.” So all you can is say “I told you so” (a process I used to sarcastically refer to as “preallocating blame”)

In fact it sounds like SolarWinds was a fairly typical software development shit-show. Developers sometimes feel that being smart is all that’s necessary to build secure, well-architected systems and networks. Too bad they’re wrong. I have heard development managers non-ironically say, “our guys are really on the ball and I know they monitor the code repository carefully” so that’s good enough – there’s no need to worry about someone putting code in some library that one of the developers just lifted from some open source software archive. Hint to would-be hackers: write a pretty graphing package and put a few extra nudge-nudge features in it and you, too, can pwn a ton of development shops.

Meanwhile, there was not just one bunch of hackers dropping malware into SolarWinds: there were multiple. I used to joke, sometimes, that software is buggy because the CIA backdoors step on the NSA backdoors, and the Mossad backdoors are incompatible with the FSB’s backdoors and some pieces of code are 80% backdoor with a user interface. Ha, ha, ha! Joke’s on me: [reuters]

(Reuters) – A second hacking group, different from the suspected Russian team now associated with the major SolarWinds data breach, also targeted the company’s products earlier this year, according to a security research blog by Microsoft.

“The investigation of the whole SolarWinds compromise led to the discovery of an additional malware that also affects the SolarWinds Orion product but has been determined to be likely unrelated to this compromise and used by a different threat actor,” the blog said.

Security experts told Reuters this second effort is known as “SUPERNOVA.” It is a piece of malware that imitates SolarWinds’ Orion product but it is not “digitally signed” like the other attack, suggesting this second group of hackers did not share access to the network management company’s internal systems.

Ah, I see that the situation will require some improvement before we can properly describe it as a “shit show.” It’s not unexpected, though – it’s a great big target, and it looks like it was a soft target; did they expect hackers to ignore it out of courtesy or something? By the way, one piece of advice I used to give my clients was to ask companies that provided critical components about their software security development lifecycle and if the answer is “we patch stuff really fast” then try to get your money back. I don’t expect anyone ever actually asked, but it was good to have the suggestion on a memo under my name, so “I told you so” becomes an option for later.

The worst part is that someone probably convinced themself that SolarWinds was just a network management thing, not a security thing, so security doesn’t matter. And then the government agencies just used it because other people use it. If you’re caught being stupid, a stupid person’s standard defense is to say “I was just doing what Bob did” and meanwhile Bob is saying “I just did what Fred does.” Etc. It’s the de facto standard. Well, so was smoking cigarettes, at a certain time, before people figured out that inhaling combustion byproducts is a stupid idea. I’m amazed.

One of the biggest problems with security is that people just assume that someone else has got it. It’s what Douglas Adams referred to, in The Hitchhiker’s Guide to The Galaxy as an “SEP Field” – an invisibility field that makes something invisible by causing it to be Someone Else’s Problem. For example, Apple has done a pretty good job of making their app store sound like they do mumble mumble something scan something mumble security before an app is available for millions and millions of users. What do they do? They’re cagey about it, but the simple fact is that there is a huge limit to what can be done and, realistically, Apple doesn’t even do that. For example, in Marcus-Land, apps would go into a petri dish that jacked around the app’s time-sense and collected all the traffic the app generated (apps should not generate traffic to anywhere except: Apple, and whoever wrote the app, and there should be specific rules as to data visibility, etc.) And app behavior should be examined; apps shouldn’t access your contact list unless they are doing certain specific things that require a waiver from Apple. That’s all a fail-game, anyhow: you can never be truly safe when you’re running someone else’s code on your machine. And you can’t safely run your data on someone else’s machine, either. Basically, as Pvt Hudson says, “Game over, man.”

It doesn’t matter how hard app stores try – there’s so much money at stake that the people who want your stuff will try harder. And what’s crazy about how security works, is that they only have to try slightly harder. Once they’re over the hurdle, then that’s it. There’s a co-evolution of sorts, but the defense against predation in this world is, pretty much: just sit there. [wired]

Despite some recentpronounced lapses, the iPhone remains one of the most secure consumer devices you can buy, thanks in large part to the locked-down ecosystem of the iOS App Store. But things do slip through the cracks – including 18 apps that used evasive maneuvers to sneak past Apple’s defenses.

The malicious apps – 17 of which were discovered by mobile security company Wandera, all from the same developer, while Apple spotted another using the same technique – have already been taken down. While they were live, they didn’t steal data or gain control of a victim’s device, behavior that other recent iOS fumbles could have enabled. Instead, the apps, which ranged from a calculator to a yoga pose repository, ran invisible ads in the background of the device, generating phony website clicks to inflate ad revenues.

That sort of adware makes regular appearances on Android, in part because that platform’s third-party app stores are riddled with bad actors.

There are whole ecosystems of crapware. Some specialize in harvesting your location, which is then sold to a company that sells location data, and the FBI buys it. Some specialize in ad-click fraud, as above. Some specialize in harvesting your contact list, etc. Each one is just crappy enough that it mostly drains your battery and makes your system’s behavior just a little bit shittier. And there are ecological niches for each kind of crapware – some runs in your browser, some runs in an app, some tracks your clicks, some mines bitcoin. It’d be fascinating except that none of this shit is what you bought your phone or computer for. The same can be said of SolarWinds’ hack: the people who bought SolarWinds did not expect that they’d get a bunch of backdoors – they didn’t ask for that, they didn’t pay for that, make it go away! But, it won’t because it’s always easier to steal someone’s stuff than to earn it and it’s always easier to ask forgiveness than permission. Not that any of these people ask forgiveness or permission, they’re gonna do what they’re gonna do and you’re just prey.

Companies that sell software should try to do some basic common sense stuff to keep people from injecting malware into their code. Beyond that, there’s no percentage in doing that; ease of use and pretty interface drives purchasing decisions more than quality. At NFR we were pretty proud of the fact that our product pretty much never crashed (we had one customer figure out how to mis-configure it so it was a mess, and had to fix the next release so it would block their attempts to shoot themselves in the foot) – but it turned out that customer purchasing decisions were more strongly influenced by the size of the vendor. People preferred to buy from Cisco and IBM because I don’t know; apparently people think that stuff is better, somehow, because it comes from a huge company? I had one customer say “we want a big vendor who stands behind their product” so I asked “have you ever tried to get a refund from Cisco?”

brucegee1962 asks: [stderr]

So Marcus, if things are this bad, what is the worst-case scenario that we’re looking at?

With 1 being Bladerunner “gritty, but stylish” and 10 being The Postman “collapse of civilization,” how bleak an apocalyptic hellscape are we looking toward?

I think we are, currently, in the worst scenario, and it won’t get any better. It may get worse for some people, but it’ll be in isolated incidents, mostly. Back in 2000, I was talking with Nat Howard about this, and he said something profound and depressing: “security will always be as bad as it can possibly be without everything breaking, and no better.” That’s pretty much a perfect assessment of the future. Things will suck but the suck will be spotty, and it’ll get some duct tape and baling wire slammed onto it, and then everything will continue to limp along. Nothing will ever be good, nothing will ever be catastrophic, but it’ll all be more annoying, more expensive, less reliable, and generally a lot crappier than it could be. Cybersecurity is going to be like Dorian Gray’s mirror for our human failings – the internet is shitty because it reflects the shittiness of authoritarian governments, the police state, rapacious capitalists, sleazy marketers, cryptocurrency scammers, payment fraudsters, credit card fraudsters, stupid FBI cops who like to troll through people’s shit, and sociopathic hackers and spooks who are compelled to sniff the underwear in other people’s drawers. It’s going to reflect human nature.

Sometimes, there will be disasters. You know, the US/Israel used STUXNET to attack not just the Iranian uranium refinement centrifuges, they went after the breeder reactor at Natanz. Some of the computers they screwed up were the ones that operated the cooling system. Natanz is upwind of a city of 400,000 – it could have been 10x worse than Chernobyl – but it wasn’t. So … carry on. If someone did that to us, we’d drop some bombs on someone until someone was satisfied that enough suffering had been spread around. That sounds crazy, but I used that example to illustrate how neatly the shittiness of software dovetails with the shittiness of governments and government policies – we can’t fix either of those so we may as well hunker down and hope it’s not us. Besides, the general shittiness of everything resulted in the chemical leak at Bohpal, and computers weren’t even necessary; it’s not like if we have another 4,000 people die in an industrial accident, whether it was a result of bad software, or a hacker, or just incompetence, we will collectively continue to stumble forward.

Basically, I’m an idealist. I feel like the internet could have been a great thing but now it’s just a conduit for ads and spooks and cops and erection pills. It’s still a great place to learn things, you just have to pay a price in terms of annoyance. I remember how, once, I felt like we were building the new Library of Alexandria but then the 4channers came along and installed a “kid porn” section in the philosophy wing and screamed “free speech!” So much for idealism. Nat was right: it sucks right now and it will never get better. It will only intermittently get worse then pick itself up and stumble along.

It might get massively worse, though. I don’t think it’s likely but I used to worry about a scenario in which the internet balkanizes. That would be an international reaction to the US continuing to treat the internet as though it is a colonial power. What if China just said “fuck it” and made their own China-net? We couldn’t do much about it (although the NSA would probably cripple it, periodically, and the Chinese PLA would reply in kind) it would be an economically incentivized strategy, not a political one: why let Apple and Microsoft have the money when the money can stay here? [This gets me close to the multi-hundred-million-dollar idea I mentioned before] Balkanizing would mean that there is considerable new opportunity for providing local application stacks. Countries would need their own software loadouts that were not under US control, and there is a lot of money to be made there. Imagine if you went to the Chinese government and offered to make a China-only operating system environment that they could own, and Microsoft would no longer be bleeding billions out of their economy. In return, they pay you an F-35-like one-time fee for the codebase, and in order to guarantee that it’s not full of NSA malware, you’d commit to live in a nice house under their observation with the understanding that they could torture you to death if it turned out that you’d betrayed their trust. I could do that; the street food in China is incredible, there’s lots of art, beautiful people, music, whatever. What would Iran pay for an Iran-only operating system and app store stack that ran on an ARM processor or something that they could build simple systems around? Is it verifiable? Sadly, no, it’s not. But you might still be able to get that F-35-like money, before they figured that out.

Another balkanization option, which really bugs me, is it might break along corporate lines instead of national lines. Imagine if Facebook ties up a deal with SpaceX so that SpaceX offers a low price captive network that carries only Facebook’s ecosystem? Amazon is locked out. Apple can only update IoS but everything else is locked down to a captive ecosystem. That would allow the vendor to completely dominate the ad market for that ecosystem, essentially paying itself with its own money, and they could drop the cost of the service way down because – basically, it’s the software equivalent of a “company town.” As long as the captives aren’t too badly abused (let them get email from outside, but strip the Google ads off first) it’d be attractive for the vast majority of people who don’t give a shit about software ecosystems. The US has some feeble anti-trust laws designed to keep this sort of thing from happening, but anyone who looks at ought to see the writing on the wall. I don’t think that balkanization like that will happen, overtly, but it’s already happening when you realize that the Trumpian FCC is allowing carriers to modulate the service speed of user traffic based on the destination. If you can say “since we have an ad partnership with amazon, you get full data rate with them!” then it’s not hard to say “we don’t have an ad partnership with Baidu so in fact all traffic with China will be slow and we’ll blame them for it.” How would Bob the Basic User know?

So what does that future hellscape look like? Blade Runner, except not cool. Unless you’re some kind of profoundly disturbed person who thinks that Verizon is cool and all the shit that Disney is going to shove at its customers is “art.” Basically that hellscape looks like picking your service plan for your cable TV and having to live in it for years. Which, in a nutshell, is “nationalism” and, we can see it sucks when you write it out large like that.

John Morales [stderr] writes:

IOW, it is possible to build a better internet, without those flaws.

Being possible, it follows that whoever first builds and employs it should have a significant advantage, no?

It’s possible, but it may not be cost effective. And, worse, your competitors can re-write your cost model using hacking as a way of raising the development cost of doing something good.

The ultimate “never fight a land war in Asia” scenario is “never get involved in a great big software development project where your enemy can affect your schedule and your budget.”

That’s another thing that scares me about all of this. Imagine if SolarWinds had doubled their development budget for years in order to not suck. It wouldn’t have mattered – a sufficiently motivated attacker could still have done them massive economic damage.

[not intended to be factual]

That’s another thing that worries me: vendors could start doing “hits” on eachother, to manipulate sales. Or outsiders could do “hits” on vendors to manipulate the stock market. Back to the question of “how bad can it get?” It can easily get that bad, and it won’t affect anyone except the shareholders and customers and a bunch of people who lose some of their retirement money.

Back around 2000 me and some of my friends were doing some thinkin’ and drinkin’ and calculated that you could farm software flaws as a way of manipulating the market and it’d be pretty damn hard for anyone to pin insider trading on you. When you think about this stuff, it’s so easy to make so much money if you’re just a nihilist and greedy: you can use hacking to manipulate markets. $100,000/day – would that be enough for you? The SEC hardly ever investigates insider trading and as long as you’re paying your full tax load, they generally ignore you.

I don’t know if you’ve seen this one, but here’s a great opportunity: [news] A Dutch security ‘researcher’/pen tester/hacker compromised Donald Trump’s Twitter account the other day. Apparently, he thought, “Trump is stupid. What kind of password would a fucking moron use?” And, sure enough, when he tried a couple of moronic passwords, maga2020! worked. Being a good guy, he did not assemble a small group of people to invest in shorting SolarWinds stock (even at its current low price, which would be great because you can short more of it the lower the price goes) and then tweet out “Executive Order to get SolarWinds off all USG networks drops tomorrow. Make American Software great again!”

See what I mean? The internet is already shitty because it’s Dorian Gray’s mirror, and it will simply reflect the worst of humanity’s bad habits.

“They paved paradise and put up a parking lot” – in case you can’t tell, I am really angry about this stuff. I feel like I wasted my life trying to make it less shitty than it is, and completely wasted my time. I could have built bicycles and made some kids happy, or made swords and made some people unhappy, or anything – but I got sucked into trying to fix some of the worst aspects of human nature using software. That doesn’t work. My bad. I mean, it worked out OK for me, personally, and I did manage to help a few people, individually, but – what a fucking mess, and it’s all unnecessary and pointless. Enjoy the banner ads and the scammers crawling all over everything like lice. It’s a whole ecosystem of scammers from the very big to the very little and the only cheerful bit about that is that they feed on eachother, too.

How to have fixed Trump’s password: Someone should have punched in a 16-char random string and then told Trump’s twitter app “remember my password” or installed some kind of password vault. 5 minutes. The hard part would be keeping from breaking the Big Baby’s fingers when he tried to grab the phone while you were halfway through setting it up. Meanwhile, some of us were hypothesizing what a mess the NSA would make trying to manage that exposure – they’d give Trump a multi-level secure workstation with an eyeball reader and dog knows what else, and the idiot would just txt his crap to a gopher who’d tweet it. There are so many ways around security if you’re an idiot, and an energetic and aggressive idiot can come up with surprisingly bad ways around. My friend Dan Geer told a tale once, back in the day, of how they tried to get a particular Wall St company to use SecurID two-factor authentication for the traders’ workstations. Dan said they all expected a lot of complaints, but, surprisingly, there were none. None. So Dan got curious and wandered in a bit early, just in time to see one of the executive assistants walking from workstation to workstation with a stack of SecurIDs in labelled envelopes, logging all of the traders in so they could just walk in and get to work.

The bitter and sarcastic security calendar posters came from a marketing bit I did for SourceFire before they got bought by Cisco. It was a lark; I had the idea and went to a couple vendors and the first 3 blew me off until I talked to Marty at SF and he said, “go for it!” I had a budget of a mere $16,000 to produce 1,000 calendars including photography and costs, etc. The calendar won some stupid marketing award (go figure: Marcus hates marketing and is an award-winning marketing guy) and SF wound up doing calendars for years and they were very popular. I did the copy, and it’s my hands and my butt and my computer, etc., for the objects. You can see the whole set at []


  1. sqlrob says

    vendors could start doing “hits” on eachother, to manipulate sales. Or outsiders could do “hits” on vendors to manipulate the stock market.

    I’m curious at your choice of words here. You say “could”, why do you think this hasn’t already happened?

  2. billseymour says

    … cybersecurity pisses me off so much whenever I think about it, that it’s almost painful to write about.

    Your security posts are always very interesting.  Thank you for enduring the pain to inform the rest of us.

  3. blf says

    On the “petri dish” to mess about with “timing” (including time-of-day, progression of time, and clock frequency): Those are some of the recognised threat vectors in PCI (Payment Card Industry), and that sort of testing is done. Required, in fact — and by an outside lab — to obtain approval from (very broadly speaking) “the Banks”. Not only timing, but also voltage, temperature, RFI emissions, and other environmental attacks, including physical penetration.

    That doesn’t mean there haven’t been failures, blind spots, protocol errors, and so on… there have been. One spectacular attack some years ago in Russia saw certain ATMs shipped with pre-installed malware (not unlike, albeit not identical to, SolarWinds). Similar to SolarWinds, everything looked benign and proper (to all but experts paying very close attention) until certain specific conditions were met, and once triggered, was still difficult to detect (from memory, it was discovered when a low-level flunky sent to harvest a batch of stolen PINs, etc., was caught — they had to physically visit the ATMs to collect the data).

  4. says

    Yeah. The petri dish idea only works if the petri dish is looking for all kinds of stuff the malcoders haven’t thought of yet.

    There was an outfit with a promising idea, which was to build markov chains of execution traces, test the software normally, then monitor it and see if any of the touch-points the execution went through created a new node in the graph. The company was “Silvertail” or something like that. Ron Dilley and I were, at that time, playing with scoring systems and those worked pretty well, too. [Dilley, Ranum]. The problem is all these things take work and analysis and someone has to have the mojo to be able to tell a vendor “we aren’t certifying this until you explain why you do DNS requests for machines that don’t exist. And if you don’t have a great answer we’ll never certify anything from you, ever.”

    That’s another problem: you can write some janky code and give it to an appstore and they reject it because you’re harvesting mouse click timing – but the app store will let you submit another version where you hid it better. WTF?!

  5. cvoinescu says

    Another balkanization option, which really bugs me, is it might break along corporate lines instead of national lines. Imagine if Facebook ties up a deal with SpaceX so that SpaceX offers a low price captive network that carries only Facebook’s ecosystem?

    That was chilling to read. Because it has already happened.

    Mobile phone networks in Myanmar (and very likely other countries, but this is the one I’ve read about) give you unlimited data as long as it’s Facebook traffic. Everything else is capped and expensive. To the people there, Facebook is, effectively, the Internet. (This is what happens when a country does not have a regulator that at least tries not to appear completely corrupt and/or incompetent: no surprise here. Facebook would almost be in dereliction of their fiduciary duty if they did not try to buy poor countries all the time.)

    This situation, however bad as a matter of principle, interacted horribly with the Rohingya crisis. Facebook became a way to spread not merely hate speech: it became a way for Buddhists to call attacks on Rohingya, and for the military to suggest targets for the thugs; it leaked data that helped identify Rohingya, who were then displaced or killed; and Facebook did nothing for a long time to prevent this. You may be used to the ability to report a post and expect someone to act on that; sounds civilized and a basic feature. But until well into the crisis, nobody at Facebook could even read the language (not even the filtering algorithms could handle it). Even when made aware, Facebook pretended that there wasn’t a problem.

    I can’t find the longer piece explaining all this, but here are some bits:

  6. says

    Instead, the apps, which ranged from a calculator to a yoga pose repository, ran invisible ads in the background of the device, generating phony website clicks to inflate ad revenues.

    That’s my kind of malware. It sucks money from advertisers and I don’t even have to put up with seeing the ads? Where can I download?

  7. dashdsrdash says

    one piece of advice I used to give my clients was to ask companies that provided critical components about their software security development lifecycle and if the answer is “we patch stuff really fast” then try to get your money back. I don’t expect anyone ever actually asked, but it was good to have the suggestion on a memo under my name, so “I told you so” becomes an option for later.

    As one who answers security questionaires for clients of this sort, here’s what happens these days:.

    Customer type A: This is a seventeen page questionaire which we need you to fill out. 60% of the questions are the same as last year but in a different order. 20% are the same as last year but worded differently. 20% are new, and inspired by things which were in the news in the last month. No followup will be made unless you fail to answer a question.

    Customer type B: This is a 750 line Excel worksheet which will not work without macros being turned on. What do you mean your security policy requires this to be stripped of macros?

    Customer type C: They send an auditor with a six-inch thick binder. He visually confirms the existence of the office and the primary datacenter, asks lots of questions with leading answers, and is both puzzled and relieved to hear that we have no Microsoft-OS servers and only keep a couple of desktops around for QA, as this removes four inches of the binder.

    Customer type D: tell us about your financial health.

  8. Reginald Selkirk says

    @6: Not my kind of malware. I don’t have to look at the ads, but it uses my precious bandwidth to download them.

  9. says

    Mobile phone networks in Myanmar (and very likely other countries, but this is the one I’ve read about) give you unlimited data as long as it’s Facebook traffic.

    I did not know about that. [The mountebank’s trick for predicting the future is to predict the present] But, yeah, it seems obvious. Amazon, with its Kindle, could have produced a portable terminal into the Amazon-verse, except Amazon’s range of internet tantalizations is not complete enough; they need a social media company and a video streaming service portal into their existing video streaming, which is extremely clunky. Then, you could pay a nominal fee for a captive advertising terminal that only had the spam they were willing to send you and only the malware they approve of. (“New! When you are asleep, your terminal will be mining for bitcoins! You get a valuable purchase coupon worth $50 for each that we find!”)

    Google is closest to being able to offer that kind of captive environment, but a walled garden doesn’t need much of a search engine.

    Basically, I am describing AOL 2.0. Predict the future by predicting the past.

  10. says

    David Hayward@#10:
    “SolarWinds’ Update Server Could Be Accessed in 2019 Using Password ‘solarwinds123’: Report”

    Not enough palm for face; I need to call in for more palm from the national guard.

    Seriously that is ludicrous and sad. But not unexpected. The Solar Winds guys don’t appear to have realized that they are making a security system – they saw it as an operational tool not requiring any discipline.

Leave a Reply