I’ve set a reminder in my calendar and I’ll do the googling and analysis so you don’t have to. Assuming we all survive, that is.
The F-35 has a particular, new, problem that’s a result of how the program’s pork was parceled out to so many vendors: figuring out who made what expensive hard-to-replace part, and when to order it is complicated. A logistical management system for an aircraft like that is a non-trivial thing – you need a database and a knowledge-base (probably some blockchain and AI, too!) and it needs to track which version of what doodad goes with which plane; the specifications are changing constantly, as are the planes, as are the parts, as are the supplies.
Logistical nightmares are nothing new for the military; consider for a brief moment that the German assault on France in WWI was meticulously planned down to the timing of each unit down individual roads, by hand, on paper. But the combinations of things that need to be tracked for something like an F-35 explodes into untrackability pretty quickly. And, any of you who think about complex system failures know that one does not simply substitute a “more or less equivalent” screw in a jet engine – a single-engine aircraft like an F-35 takes engine function very seriously, indeed.
The F-35 program’s software for logistics appears to be a nightmarish mess:
WASHINGTON – Setting the weekly flying and maintenance schedule for an F-35 squadron is a weeklong process. It takes hours for multiple people to download data from the jets and comb through it, paste information into different spreadsheets, and continuously update each system.
With a new app called Kronos, on track to be delivered in early March, the U.S. Air Force is hoping it can trim the amount of time for that process to 15 minutes.
Kronos was developed by the Air Force’s Kessel Run software development team as part of a new effort called Mad Hatter, which was established late last year to solve pilot and maintainer gripes with the F-35 fighter jet.
If all goes well, it could lead to a much bigger overhaul of the F-35’s troubled logistics backbone, known as the Autonomic Logistics Information System, or ALIS, said Will Roper, the Air Force’s top acquisition official.
“There are many things about ALIS that are very frustrating and time consuming,” Roper told Defense News on Feb. 12 in an exclusive interview. “The goal [of Mad Hatter] is not simply to fix ALIS within the constraints that define it. It is to make the operator — the maintainer — more efficient, to make their user experience more pleasant.”
[emphasis mine]
(Sputter)What?! It takes a week to plan flying for a week? And it involves manual cross-inputs between spreadsheets? Holy shit, somewhere some clay tablet-maker is filing a lawsuit saying that they were unfairly cut out of the program.
That’s only the start of things, of course it gets worse:
“You can imagine: What do the users want? They want Wi-Fi on the flight line. We believe we can do that securely. They want to have a touch screen where they have one database that can touch ALIS and all the other tools, that translates automatically. These are not Herculean tasks,” he said.
Now with stealth and data fusion
I can imagine! And my imaginings lead me to Lovecraftian horror. Whose Wi-Fi are they going to use? Cisco’s? Huawei’s? Is the software going to be operated by the same kind of Air Force geniuses that managed to infect the command console of their predator drones with Russian malware? The same geniuses that couldn’t get the malware out for 16 months? [reg] I know, let’s give those geniuses flight-line WiFi so that anyone with a new day0 exploit can mess with a very complex piece of software that controls a very very expensive fly by wire aircraft. I guarantee you that, if they do that, within 2 years the aircraft will refuse to work at all if the WiFi signal is jammed, which means the whole garbage-pile can be grounded with minimal effort.
The problem here is that when a piece of complex technology embeds another piece of complex technology, it embeds all the flaws in that technology, as well. For obvious reasons (because they aren’t expensive enough!) they almost certainly won’t rely on Chinese-made Huawei WiFi gear; they’ll use Cisco – which includes NSA backdoors and intermittent “holes so big you could parade a moose-riding mariachi band through.” [zd]:
Cisco is warning businesses that use its wireless VPN and firewall routers to install updates immediately due to a critical flaw that remote attackers can exploit to break into a network.
The vulnerability allows any attacker with any browser to execute code of their choice via the web interface used for managing Cisco RV110W Wireless-N VPN Firewall, Cisco RV130W Wireless-N Multifunction VPN Router, and Cisco RV215W Wireless-N VPN Router.
Yeah, it’s a herculean task. Specifically, the Augean Stables [wik] – computer security is an endless shit-shoveling contest and if your starting premise is that logistics are hard you’ll know that entering into a vulnerability-management cycle is a guarantee of eventually getting compromised.
“There is a logistics system that supports the F-35 called ALIS. It cannot scale. It has got huge problems. It drives the maintainers nuts. And so we put together a team of Lockheed Martin, Air Force programmers and maintainers on the flight line,” she said. “They named themselves. The new program is called Mad Hatter, rather than ALIS. It is always the young techies that come up with something.”
Two other applications will follow closely on the heels of Kronos. Titan will help expeditors determine fleet status, assigning tasks between maintenance teams as the workflow changes.
“It’s 3:00am, go check the tire pressure on all the aircraft. NOW!” Whoever controls the logistics system controls the function of the total system. I’m also curious how this sucker’ll forward-deploy on, say, an aircraft carrier with a communications blackout. I assume it’s going to be massively dependent on reliable bandwidth, and plenty of it. That shouldn’t be a problem, right? It’ll use the government’s classified network – the one with 750,000+ trusted users. Trusted, yes. Trustworthy, maybe. Let’s build some more interdependencies in there and make it more complicated because complex is better!
A stealth aircraft fleet with a single point that lets you determine fleet status. It’s strategic genius.
And once Mad Hatter has a chance to prove itself with its initial apps, it may move onto a more substantial task: creating an experimental, cloud-based version of ALIS, and then helping build future software drops.
The team has begun the process of re-hosting the latest iteration of ALIS, version 3.0.1.2, on Pivotal’s cloud foundry, Roper said.
I’m tempted to stop right there and leave you hanging off the edge of the cliff, but that’d be mean. Remember that the F-35 is a multi-national NATO aircraft; the Turks and Japanese (the Brits and Canadians are good little lap-dogs) are going to love having their military aircraft’s logistical system dependent on a US 3rd-party contractor’s cloud servers.
In 5 years there will be moaning and wailing and finger-pointing accusations that the Chinese and Russians have hacked the F-35’s logistical system. Shock and aww!
It’s hard to believe I did this nearly a decade ago:
This posting requires the mandatory link to Charles Perrow’s Normal Accidents – [wc] It does a better job than I can of explaining the exploding complexity of interactions in interdependent systems, and how humans rapidly become incapable of figuring out how things break. It’s fascinating stuff.
I love it when software developers say “How hard can it be?!” and decide to build their own complete replacement system. The results are usually about as bad as the first system, for the same reason. To be fair, this stuff is really hard to write – which is all the more reason to be skeptical when someone says they’ll just put together a modular cloud-based version of their own. You should always ask “why do you believe you will get right the things that everyone else got wrong? Because the reasons that they got it wrong apply to you, as well.”
First time commenter, of course long time lurker.
So around 6 months to replace a task and reduce it from one week to 15 minutes. Will they also reassign all the people who currently do the maintenance logs just to see F-35s fall out of the air sooner?
Everything about the F-35 just seems like an exercise in the difference twixt concepts in theory and in practice. Also, it strikes me as trying to replace specialized roles with a generalist. Which isn’t necessarily a bad thing in certain situations, but it absolutely shouldn’t be more expensive to do so.
I feel like a lot of the military’s big-budget projects are stuck in the 80s threat model of NATO facing off with Warsaw-block countries. Meanwhile, we’ve spent the past half-century shelling the fuck out of a handful of brown equatorial countries, and sometimes we do a shitty job of occupying them.
I can’t remember where I ran across it, but I recently ran across some critique of the M-16/M-4 platform being that it didn’t do enough damage to torsos that were less than 7″ thick. Our military was literally complaining that our opponents are too hard to kill because the are too small, too skinny, and too malnourished to take sufficient damage from our weapons. You’d think that reading something like that should cause anyone with a smidgen of morality or ethics to maybe sit back and think for a bit…
And on another note, why wireless? I’ll admit I’m completely out of my depth here, but it seems like wireless is really convenient in some scenarios – using a laptop in a building to check your email or fuckoff on webgames in class/meetings/whatever, but doesn’t the very nature of it carry a certain degree of compromised security? Then there is all this nonsense with wifi lightbulbs and toasters and refrigerators and doorknobs and what not. Those seem not only silly, but also possibly insecure, and that’s just on a personal level. I’ve no clue why someone would want to potentially apply those vulnerabilities to any military application, let alone a proposed flagship? Wouldn’t you want to airgap something that important? or as much as possible, like I said, I’m out of my depth…
Blockchain, AI… why not blockchain-based AI? You can impress people by how “powerful” it is (“It takes up three quarters of the globe’s processing power”*), make the verification issue of AI/machine learning even worse (We don’t know how it makes decisions but now it’s all encrypted, too!), explode and already exploding budget and combine some of the most annoying marketing terms currently floating around. If that doesn’t spell success, I don’t know what will. In fact, forget I said anything while I nip down to the patent office.
*Left unsaid: The output will be “Hello, World!”. Or maybe, “Error in Line 3: “pritn(‘Hello, World’)”. Either way we won’t know for another week.
—
This logistic “app” reminds me of a grand project around 10 years ago, whereby the NHS was to switch to a UK-wide unified IT/Software/Management/whatever system. The noble idea was that it eliminates paperwork and makes patient data available to doctors who need them, e.g. if you’re away from home and have to see someone other than your usual GP.
It was used as a teaching example about systems engineering in some lectures I attended and was a big, expensive project. Soon after it apparently failed miserably because people couldn’t fit all the bits together. I suspect it hasn’t been used as an example since.
This project sounds exactly the same:
1) We didn’t think ahead when we had the chance and ended up with myriad of disparate pieces floating about!
2) We’ll fix it by sticking everything into an even bigger system!!
3) That system will sort out how all the small bits communicate and work together!!!
4) Everything is now fine!?!?!
So in the future to learn the enemy’s disposition you don’t analyse satellite data or intercepted communications, you just hack the cloud? Great! Now, who do I have to hack to get some nuclear launch codes? Dropbox? MS OneDrive?
Asking for a friend who lives in a volcano, collects nuclear subs as a hobby and likes cats. Yes, he’s a bit eccentric. Cats are awful!
P.S.: Of course those planes won’t fly if the wifi goes down! Without a connection, how is the software supposed to check if the pilot has purchased a valid license?* Unless they’re logged in the controls are limited to thrust, roll yaw but not pitch, or the pilot may use a one-off 30 minute free trial.
*What might an F-35 enterprise license cost?
And it involves manual cross-inputs between spreadsheets?
Oh lovely. I actually feel sorry for anyone who flys them.
Over the last few years I have come the conclusion that one should not use a spreadsheet for anything more complicated than a household shopping list. That may be pushing it.
I occasionally wonder if Turkey buying S-400 missiles from the Russian Federation was not partly a way to get put under a F-35 ban. It must be very clear to Turkey that the F-35 is a turkey (I’m sorry) and the best thing is to not buy them.
BTW, last I heard the Canadian Govt has dropped its plans to buy F-35s and is/will be opening a new bidding process. The Su-35 might be a good choice. It is two-engined.
This just made my mouth gape at the sheer…craziness….of it all. And I am almost 110% techno-ignorant. :)
I have some bad news for you: the entire world runs on dodgy spreadsheets. Often with lots of manual cross-entry. At least half the time when somebody bothers to build a proper software system for ${WHATEVER}, it rapidly degenerates into a system that just generates and ingests spreadsheets. People just love their spreadsheets, and hate systems that actually enforce robust business rules.
Still, at least most people understand how to use spreadsheets these days… Way back in the day, I once encountered a government statistics department that was using Lotus 1-2-3, but apparently hadn’t realised that you could put formulae in cells – so they were entering tables of data, and then calculating stuff by hand on their desk calculators and entering the results as numbers.
I have yet to encounter a major corporation that doesn’t have a black screen 1980’s system running legacy software hidden somewhere in their tangle of interfaces.
The best we’ve done was get a 3 day process down to 2 hours, even though we own(ed*) the data. With multiple systems and manual inputs, they can likely get some efficiencies, but that means they need to have a clear plan and motivation to change. It sounds like they’re tossing a bunch of buzz words in a basket and hoping for the best. Maybe they hope that the data will respond to the new system like the broom sticks in Fantasia?
Am I wrong in thinking that sometimes simpler is better?
*We too are going to The Cloud. Wheeeeee.
@ 6 Dunc
I have some bad news for you: the entire world runs on dodgy spreadsheets
Why do you think I came to my conclusion? I say, jokingly I hope, that I live in dread of a spreadsheet mistaking my telephone number for my weight when calculate a drug dosage.
The Reinhart and Rogoff saga seems to suggest that spreadsheets are not the best tool for sophisticated analysis.
That Lotus 123 story has me laughing like mad but I can, only too easily, believe it. Just to terrorize myself I sometimes read the European Spreadsheet Risks Interest Group’s Horror Stories.
Still, at least most people understand how to use spreadsheets these days…
I think you are being wildly optimistic. I have lost the code but I once found an Excel cell that had a equation with something in the neighbourhood of 800 characters. I have read of people with spreadsheets of 100 sheets or a single sheet with 100 columns and 50,000 rows of data.
I am not sure about the business community but in the scientific community, pretty well everyone seems to be self-taught, badly usually, See something like Zeeberg, B. R., Riss, J., Kane, D. W., Bussey, K. J., Uchio, E., Linehan, W. M., … Weinstein, J. N. (2004). Mistaken Identifiers: Gene name errors can be introduced inadvertently when using Excel in bioinformatics. BMC Bioinformatics, 5, 80. https://doi.org/10.1186/1471-2105-5-80 .
BTW if you have something like Python and a spreadsheet handy try e^2^3 in each and compare the results.
If you have gnumeric it is actually funny (scary?) to do this.
Correction: that should be 2^2^3.
I used excel a lot at my previous jobs. abused is probably a more accurate word.
I said it was kinda like chopping a tree down with a shovel – completely the wrong tool for the job, but if you need to get that tree down now, and you’ve a truckful of shovels…
jkrideau @8 I’ve either encountered, or made most of the examples you listed.
I don’t really know any programming, I’ve thought about trying to learn, but have no clue where to start, etc. At a previous job, I used excel because it was on all the computers, and everybody used it. Most of the other programs we were working with weren’t compatible with each other, or communicating, but most could dump info into a .csv, and/or import from an Excel spreadsheet.
And, yeah, people were doing a lot of stuff by hand. So I would set up a clunky excel sheet full of nested formulas, and have a blank tab to paste info from one program/machine into, wait 10-20 minutes for the computer to finish calculating, and then copy from another tab and import that into another machine.
jrkrideau, @ #8: Well, when I said “at least most people understand how to use spreadsheets these days”, I only meant in the most rudimentary sense – i.e. they understand the concept of formulae and cell references. I certainly didn’t mean that they understood how to use them well… ;)
The scientific community are notoriously bad programmers too, again because they’re all self-taught.
lochaber, @ #9: Oh, I’ve got some spreadsheet abuse stories… Like how back when I worked around statisticians in the civil service, they would all use Excel for writing letters, because it was the only application they knew how to use (or even how to open). “Fun” times…
@ Lochaber and Dunc.
Thanks, more fodder for the nightmares.
jrkrideau@#8:
The Reinhart and Rogoff saga seems to suggest that spreadsheets are not the best tool for sophisticated analysis.
Remember when Intel had an error in the floating point routines in the X86 architecture? If I recall correctly, that one was discovered by someone who was using a software implementation of a particular operation and it came up with a significantly different answer. This is a problem with regression testing as it’s normally done: we check to see if the answers are the same as the previous set of answers. If the collection of questions and answers is too large and old, sometimes nobody checks whether the questions and expected answers are, themselves, right. Hilarity ensues.
jkrideau @12
If you don’t feel like you are having enough nightmares, there was this one time my supervisor basically handed me over to the IT department for a day to create a clunky Excel workaround cludge
:|
It’s not just Excel users, apparently, but also Excel itself. Some time ago I was building a python script that would read/write excel files.* The python module I picked wasn’t very widespread, so often searches about it or particular issues would lead me to discussions involving the module’s creators.
You could tell they were somewhat frustrated by the inner workings of Excel. Apart from general quirks, sometimes Excel would, for example, use different functions to set up the same thing. E.g. font formatting might be done one way in some cases and in a completely different in others, for no apparent reason. So anyone trying to create an xlsx file outside Excel has to untangle these functions and pick the right one for the right circumstances.
*Confession: I’m one of those terrible self-taught people, both with python and excel. But I take pride in the fact that I #comment my effing code and I curse the self-declared pros, who often include not even one helpful comment anywhere, straight to Hades. No offense to present company. However pythonic your elegant one-liners may be, if I can’t understand the damn thing and you don’t explain it, I’ll just cobble something together myself, even if it makes you scream at night.
@ 14 lochaber
That is scary. Still, IT departments can be worrying.
Still, as I suggested earlier try
-2^2^3 in Excel and -2**2**3
in Python. That is, in both cases a calculation for $-2^{2^{3}}$ in LaTex script.
@ 15 Komorov
I can believe the mess in Excel. I think it just “grew”. This is probably true of a lot of more or less 1st generation “user-oriented” software.
I tend to be more nervous about wrong answers, particularly as Marcus notes an old test suite may not flag a problem. See my Excel — Python point above. Excel and Python do not return even vaguely similar results.
You comment code? I thought that was considered unethical.