Also acutely relevant to the problem I just described is this article by Bruce Schneier, who explains how the problems of computer and software security are very similar to those in biological engineering.
Programmers write software through trial and error. Because computer systems are so complex and there is no real theory of software, programmers repeatedly test the code they write until it works properly. This makes sense, because both the cost of getting it wrong and the ease of trying again is so low. There are even jokes about this: a programmer would diagnose a car crash by putting another car in the same situation and seeing if it happens again.
Even finished code still has problems. Again due to the complexity of modern software systems, “works properly” doesn’t mean that it’s perfectly correct. Modern software is full of bugs — thousands of software flaws — that occasionally affect performance or security. That’s why any piece of software you use is regularly updated; the developers are still fixing bugs, even after the software is released.
Bioengineering will be largely the same: writing biological code will have these same reliability properties. Unfortunately, the software solution of making lots of mistakes and fixing them as you go doesn’t work in biology.
In nature, a similar type of trial and error is handled by “the survival of the fittest” and occurs slowly over many generations. But human-generated code from scratch doesn’t have that kind of correction mechanism. Inadvertent or intentional release of these newly coded “programs” may result in pathogens of expanded host range (just think swine flu) or organisms that wreck delicate ecological balances.
We can’t release “gene patches” to correct errors introduced when tinkering with genomes! I can imagine that someday being an issue — by analogy, going in for dialysis is kind of like a routine software management problem. But no one likes having to do dialysis, it’s a symptom of an underlying problem that is just being patched superficially, not fixed, and modifying genomes can introduce new concerns. I wonder how often software updates create new problems that weren’t present in previous versions? 100%?
I don’t think we think enough about the potential for disaster in genetic engineering, because we are enthusiastic about the potential for great good. We need a balance. It would be helpful for those most optimistic about gene modification to have more consideration for the dangers by, for instance, talking to software security experts.
Opportunities for mischief and malfeasance often occur when expertise is siloed, fields intersect only at the margins, and when the gathered knowledge of small, expert groups doesn’t make its way into the larger body of practitioners who have important contributions to make.
Good starts have been made by biologists, security agencies, and governance experts. But these efforts have tended to be siloed, in either the biological and digital spheres of influence, classified and solely within the military, or exchanged only among a very small set of investigators.
What we need is more opportunities for integration between the two disciplines. We need to share information and experiences, classified and unclassified. We have tools among our digital and biological communities to identify and mitigate biological risks, and those to write and deploy secure computer systems.
I’m optimistic about the future of genetic engineering, but I still cringe when I see some ‘bio-hacker’ inject themselves with some home-brewed cocktail of gene fragments that they think will improve their genome, but is more likely to do nothing or make them sick. I get the same feeling when I see someone stick a flash drive into the USB port of some random public terminal. I hope they’re going to practice good data hygiene and quarantine that widget before they put it in their work computer! (They probably won’t.)
This seems apropos:
A field experiment in Brazil that deployed genetically modified mosquitoes to control wild populations of the pest may be having unintended consequences. According to a genetic analysis of mosquitoes in the area, it appears the engineered stock has bred with wild mosquitoes and created viable, hybrid insects, scientists reported in Scientific Reports last week (September 10).
“The claim was that genes from the release strain would not get into the general population because offspring would die,” coauthor Jeffrey Powell, a professor of ecology and evolutionary biology at Yale University, says in a press release. “That obviously was not what happened.”
This seems like a first to me. PZ accepts a biology/software analogy as legitimate. (But maybe I missed something.)
Software is typically more predictable than biology. Much of this is because of what is considered good practice, rather than a lack of inherent complexity. For instance, we usually avoid self-modifying code, maintaining a clear partition between what part of working memory has instructions and what part has data. (I’m not even claiming there is a biological analogy to self-modifying code, but there is clearly no barrier as you might enforce with segmented memory.) Of course, malware is not written with best practices in mind, but with the intent to subvert as many protections as necessary to install detrimental software, posing even worse problems than conventional software with design, review, testing, etc.
I do find software development practices a useful metaphor for explaining why “design” is not even the most common approach used for human invention let along living things. Software developers continue to use copy-paste-modify to add features to an existing programs, creating new functionality often with very little understanding of the existing functionality they just copied and tweaked. It looks more like an evolutionary process than design to me (agreeing with Schneier on this).
My initial objection to the mosquito control plan is less based on an analogy with software, though, than an appreciation of statistics. When you say a gene is lethal, that sounds more like you should be a 99.999% (however many nines) chance of being lethal.
(Outside my comfort zone, but this seems reasonable:) If there’s a small chance the gene is survivable over generations in combination with other genes, then the natural experiment of releasing these mosquitoes into the wild eventually tell you that better than lab testing. I.e., unless the lethality is close enough to 100% to wipe out the introduced gene before wiping out the population, then the surviving mosquitoes have some chance of reproducing. That’s assuming the lethal gene does not mutate into a harmless form, which may be more likely. I would also expect that if the gene is not lethal but 100% harmful when it’s not lethal, it would also eventually be eliminated. If it’s neutral, it could be carried along.
I get PZ’s original point (other post). The real problem is not the introduction of a new gene from the lab, but the introduction of new mosquitoes from somewhere else, increasing genetic diversity. If, e.g., there was a 1 in a million chance that the gene was survivable, then this massive release would be like releasing 50 healthy mosquitoes into the population, ignoring the effect of the released gene.
I can see the plan working if you can guarantee lethality with a probability several orders of magnitude below 1 in a million, but did they have any reason to believe this? It’s a pretty high bar to meet.
me@2 Numerous typos, but to fix a key one here, I meant to write “I can see the plan working if you can guarantee lethality with a [survival] probability several orders of magnitude below 1 in a million,”
cervantes@1 That was the subject of PZ’s previous post, so apropos but not coincidental.
There are theories of software development. Bruce is wrong to say there aren’t but for all intents and purposes there aren’t because practically nobody uses them – high assurance design takes time, effort, and skills that typical programmers lack. Assured designs have also resulted in some spectacular failures, which is more of a problem. For example, it does not matter if your code-base running in your CPU is high assurance but the code-base running your database where all the settings are stored, is not. And nowadays, to make things worse, there is solid evidence that the intelligence communities of at least the US and Israel have conspired with chip manufacturers to build their own routines into the silicon of processors and peripherals. There’s probably a biological analogy there, as well: there are viruses, retro-viruses, and endosymbionts.
If you’re looking for a silver lining to this rainbow of shit, just remember that the “AI destroys humanity” scenario will all be running on processors that are backdoor’d and operating systems that are buggy masses of crap which are also backdoor’d.
And you don’t think our new AI overlords will be better at finding the backdoors than humans will be at remembering where they put them?
Seriously, I don’t believe the AI apocalypse is around the corner, but I don’t see any silver lining in buggy software. There will be some vulnerabilities that can be exploited, but the putative smart computer programs just need to proliferate faster than our efforts to shut them down. If they’re also controlling infrastructure and production capacity, it won’t be as easy as pulling a plug either.
(Above presupposes much better automation than we have today as well as actual computer intelligence and intent; I make no predictions on when this is more than a science fiction scenario, probably not for a long time though.)
@Marcus:
Yes, there are theories of software. As you say, one of the big problems is that designing software like this is that it takes a lot more time and effort, which means that it’s pretty much commercially non-viable except in very specific safety-critical environments.
The other big aspect, even beyond hardware or other software, is that even perfectly verifiable software really only pushes the problem back a level. Being able to prove that the software perfectly implements the specification isn’t much good if the specification is incomplete, which it usually is for any non-trivial system. Otherwise, you need to have a table of every possible combination of inputs and the results for all of them.
From what I’ve heard, you could make a good case that the 737MAX fiasco was caused by a late change to the specification where the full validation wasn’t re-done afterwards because it was such a ‘small’ change. (Or, at the very least, a late change to the hardware without the specification being properly updated to match it. The software was still doing exactly what it was supposed to do by the original spec, it just ended up potentially operating in a regime that was previously considered an unreachable state and thus was under-specified.)
I think there will always be many more people writing software than there will be people bound to any software methodology, even if there is one that’s effective in ensuring reliability. The problem isn’t the existence of good practices (though I’m unconvinced there are significant cases of developers following any consistently). It’s the inability to prevent the use of bad practices on a wide scale, particularly when the incentives are nearly always to get features out fast and respond to the most serious user complaints later. (This is not a law of nature, but it is baked into Silicon Valley culture.)
In some critical applications (airplanes being a good example, and medical systems) it’s worth the cost of enforcing a software methodology. It’s unclear to me how you would stop all bad software from being written and used, though, without an onerous legal framework as applied to non-critical applications.
This is compared to genetic engineering, which I think is reasonable. But it is also comparable to life writ large, any kind of evolution, reproduction, or breeding. That’s just the way biology is, engineered or not.
Obviously, this person has never looked at my lab notebook. Making lots of errors and fixing them as you go is exactly what happens when doing genetic engineering, so I’m not sure exactly what he is talking about.
Sure we can. And we do, every single growing season in the form of classical breeding or engineering. New seed varieties are developed and released every year by combining new genetics to fix problems with the previous version.
PaulBC@#8:
In some critical applications (airplanes being a good example, and medical systems) it’s worth the cost of enforcing a software methodology.
Yeah, except they don’t. Commercial fly by wire aircraft use shared network hardware with the in-flight entertainment systems. Someone designed that, but there’s no methodology that would find that acceptable. And medical systems don’t enforce any kind of design methodoloy; they enforce a system administration practice which is “if it’s working, don’t touch it.”
There is a truly great history of the early understanding of the software crisis and how the software industry has reacted to it. Unfortunately I cannot remember the title and there are so many stacks of books around here that I’m not sure I could find it.
One essential is “The Huawei and Snowden Questions” (Springer Verlag 2018) but that’s not it. Gah! I don’t even know how to find the damn thing. I just went through all the stacks in my bedroom (about 10) and it’s not there. It’s not one of the popular/well-known books in the field, so when I search for it by keywords I keep coming up with Simon Singh and Charles Perrow. This is driving me nuts.
I’m going to find out what that book was; it may take me a few days. When I do, I’ll post a review of it over at stderr, so I can’t lose it again. Check back in a week.
I’m reminded of one attempt at gene therapy, which due to poor patient selection, resulted in a fatal cytokine storm.
https://en.wikipedia.org/wiki/Jesse_Gelsinger
Without discipline, it’s a situation akin to a five year old finding grandpop’s gun, with predictable results.
#8: that’s only true if you aren’t valuing the individual, but only the genetic line. It doesn’t help me to say my genes need a patch, here, have a baby with this modified sperm.
@10 Marcus Ranum
Also well worth reading on the subject of software provability and the relationship between logic and biology is this lecture from Leslie Lamport: https://www.microsoft.com/en-us/research/uploads/prod/2016/12/The-Future-of-Computing.pdf
I find Lamport’s work on formal software specification is also interesting: http://lamport.azurewebsites.net/tla/book.html.
Marcus,
Probably not what you’re looking for, but the name Les Hatton comes to mind.
Which is why people are unnerved by GMOs. Consider the above and the previous post about mosquitoes.
@7 bryanfeir
As I understand it, the software was there to try and fix a fundamental aerodynamic design flaw in the plane. It simply wasn’t possible to fit the higher diameter more fuel efficient engines under the wing of the 737. So the engine pods had to be moved forward and higher up, thus creating more lift and tending the front of the plane to tilt upwards. The software was meant to compensate for this by automatically bringing the nose down.
Apparently the software only took data from one sensor (by law everything in a plane should be duplicated). So if that sensor failed then the software would keep pushing the nose down, no matter what the pilot did to attempt to correct it.
There was a long article in the German magazine Spiegel about it. Unfortunately it doesn’t appear to be on-line or available in translation.
I may have got some of the facts wrong here, I am not an aeronautical engineer, but I understand the gist of the story to be correct.