Field-expedient repairs are sometimes expected. You haven’t got all the gear to make a proper fix, so you log a maintenance report saying something like, “I did not have the correct threaded bolt to replace it correctly, so I forced the wrong bolt on to the nut with a pipe-wrench, just to hold the thing together until we got home.”
As Richard Feynman famously wrote in his minority report on the Challenger disaster: if the system design does not say “these nuts are expected to wear out and will need to be replaced every 4 years” then, if the nuts wear out in 4 years, there is something wrong with the system design. In the context of the Challenger it was gas blow-by around an O-ring on a solid rocket booster, which NASA flight engineers kept concluding was safe enough – but only because they never flew a shuttle when it was so cold that the O-ring did not flex as expected. Another way this sort of flaw shows up is when an implementation of a design has exceeded its expected service-life. If the nuts need to be replaced in 4 years, then the design is correct if the service-life of the system is 2 years – leaving some engineering overhead as a margin of error.
I explained all that, because we need to understand why engineers sometimes get upset when engineering assumptions are ignored. As Charles Perrow points out, in Normal Accidents [wc] high risk technologies frequently embed such engineering assumptions, and disasters seem to cluster around cases where they are ignored, or warnings about them are ignored. In a paper I wrote before I retired, I pointed out that in a large number of security disasters, you can often back-track the event by reading the emails from auditors and system engineers who were saying “this thing you want to do is going to bite us some day…” … in exactly the way the system eventually failed spectacularly. [ran] As I also pointed out, field-expedient cost/benefit analysis can often result in massively higher clean-up costs than the “expensive” option of properly maintaining a system.
So, imagine my blood running like ice, when I read this [firstpost]
UK’s own Chernobyl averted: Navy’s nuclear-powered super submarine ‘Trident’ fixed with super glue
The damage was done at HMNB Devonport in Plymouth during a dry dock renovation and refuel. This work, reportedly started in 2015 and is four year behind the schedule and approximately £300 million over budget.
In a bizarre incident, employees aboard the UK’s most advanced frontline submarine Trident risked the lives of millions of people by allegedly using superglue to fix broken bolts of a nuclear reactor chamber prompting navy chiefs to order an investigation.
The crucial cooling pipes’ shoddy repairs were revealed after one of the bolt fell off during a routine check of the 16,000-ton HMS Vanguard.
A submarine nuclear reactor is nowhere near the scale of the RBMK at Chernobyl, and would not fail at the same scale, either – but unquestionably, a melt-down in a ballistic missile sub is bad. Especially when it’s sitting in port.
Midjourney AI and mjr: the very important stainless steel bolt has been held together with a clot of super glue
The point is: when a bolt falls off the cooling mount for a nuclear reactor, you don’t just slap it back on with a dab of cyanoacrylate to keep it in place. For one thing, cyanoacrylate might not be the right glue to use in that environment. But it’s certainly cheaper than removing the stud and machining a new one would be – and it’s vastly cheaper than asking (as an engineer would) “why did the bolt fall out? was there vibration? unexpected stress? material wear?” or even “has this component been exceeding its designed lifespan?” I guarantee you the answers to any of those questions is much more expensive than a dab of Crazy Glue. [as a workshop hack, I must add: they should have used PC7 or JB Weld. Metallized epoxy for the win!]
Reports reveal that the bolts had broken due to careless overtightening, but civilian staff of the defence contractor Babcock glued the heads back on rather than alerting the damage to the authorities so that the fractured shafts could be repaired correctly.
Some civilian staff at Babcock are probably updating their resumes right now.
It goes on:
The sailors of the three remaining Trident 2 nuclear missile subs, HMS Vengeance, HMS Victorious, and HMS Vigilance, have had to endure protracted patrols due to persistent delays.
From 2028, the Dreadnought class will take their place and carry the UK’s nuclear deterrent.
I know that some of you who read this blog are actual engineers and let me demonstrate my amazing bloggers’ psychic powers by reading your minds across the internet. You are thinking, “I sure hope they check the same bolts on the other three remaining subs!” That’s another important point that the article omits: by concealing the damage, Babcock put 3 more subs at risk – subs that are at sea busily exceeding their designed lifespan – because those subs might also be about to suffer coolant systems failures while their reactors are hot. When the Challenger blew up, NASA at least had the sense to ground the entire fleet of space shuttles for years while it re-assessed the entire solid rocket booster system design.
For a deterrent to work it must be seen to be credible. That’s a problem Russia is facing right now, as many people are questioning the validity of their nuclear threats. Jesus fucking christ what an insanity the world has become!
Speaking of dreadnoughts, I just finished a re-“read” on the audio book version of Massey’s Castles of Steel, which is about the rise of the great iron battleships, and their apotheosis at Jutland. It remains fascinating stuff. The first time I read the book, I remember I was interested in the degree to which the action could be de-constructed (e.g.: at 11:33 a 15-in shell hit the Snallygaster, penetrating its armor belt at a 20 degree angle, richocheting through the mess hall and the cook’s pelvis before winding up in the port-side machinist’s loo) there is a famous book on Jutland I read in high school which, literally, discusses every measured hit on every ship – fascinating and profoundly boring. Anyhow, now that I revisited Massey, I was struck by something completely different: the simultaneously high professionalism and complete incompetence of the royal navy. At the presumed acme of seamanship, you have huge battle-fleets sailing past eachother in fog or rain, with British sailors mistaking German cruisers for their own, and waving a cheerful “howdy!” as they sail by.
FIFY.
Close. I’m actually thinking “I sure hope they check all the bolts on the other three remaining subs!”. Starting, perhaps, with the ones in question, but here’s the thing – “careless overtightening” is a failure mode common to every single bolt on the boat. And some of those other bolts will be at least as critical as the ones on the reactor coolant mountings, it’s just that “nuclear reactor coolant mounting bolt” is a phrase that, thanks to years of exposure to adventure fiction, sounds scarier than, say, “bulkhead sealing bolt” or “prop shaft bearing mounting bolt” or whatever.
One would think that assembly of a nuclear submarine would be given to more competent staff than to your regular Dockyard Dolt who lengthens his spanner with a piece of pipe in order to not exert himself too much at work. There are such things as torque-wrenches ffs.
@sonofrojblake:
Why would someone on a submarine care if the bolts of the prop shaft bearing were stripped?
Some civilian staff at Babcock were probably getting leaned on heavily by management to “just quit whining and get it done, we were already late when we started” while provided with insufficient manpower and tools and time due to the usual government procurement policy of engaging the contractor who, by their bid, has proven they are incapable of actually meeting the project requirements.
Of course there’s the original quote repeated in the various articles:
Ahahahahaha.
@2 torque wrench FTW
… and I certainly wouldn’t let Midjourney service my nuclear reactor bolts.
Ah, Trident and defense boondoggles.
There are precedents, as shown here:
https://www.youtube.com/watch?v=IKQlQlQ6_pk
and mind you, this is from the mid-80s.
“Well if there’s a nuclear war, Minister, it will not last long enough for the weapons to be tested”.
@xohjoh2n #4
That certainly sounds like how most such things go. As a worker, I once phrased it to myself as: “It’s not my job to do the job. It’s my job to try my best, under the conditions given to me by management.”
@8:
The thing about Yes, (Prime) Minister is that it was topical at the time. Then some 8-10 years later it all became topical again. Then 8-10 years after that it all became topical again…
To be fair, the source the type of glue used isn’t the Royal Navy, it’s just something mentioned in the report. One would hope it was something a bit more effective with metal. Also, if they’re trying to hide damage, it wouldn’t surprise me to find out they were responsible for the bolts breaking in the first place. My guess is that HMRN will blame the contractors and hope everybody forgets who hired them.
My understanding is that it’s important subs are quiet. If the shaft on which the propeller is mounted starts flapping about in its mounting, leaving aside how expensive that’s going to get, it’s going to make a noise.
@12:
Quite so.
If the bearing on the prop shaft breaks loose, several things are likely to happen. Firstly, is it the load bearing or the counter bearing? Either one is bad, but will result in (potentially hilarious, assuming they are not actually underwater) results. If the load bearing breaks, you can assume a significant amount of “flopping”, yes, followed by a severe shearing and failure, and if the motor is not shut off quickly enough, quite a bit of damage as your propeller shaft starts swinging around the room like a 2-ton baseball bat. If the counter bearing fails, you may experience some flopping, or you could have a propeller shaft shoot to/through the bulkhead behind it like a battering ram.
Torque wrenches are awesome, but you also have to maintain them, which doesn’t really happen. They should be tested once a year (or per manufacturer spec based on use) and recalibrated when the tolerance falls out of guidelines. Just like a lot of tools that don’t get maintained.
Lots of assumptions go into these things, but they hire people on the cheap to maintain it without enough bodies to get the work done, and without enough real training to provide a safe and efficient workplace, and the people who do the work are overworked and underpaid. Some douche runs away with the contract money in his pocket, and lawsuits for shoddy work end up running the company into bankruptcy, but the owner doesn’t care because he’s got buddies in high places and they’ll hire his new company to perform the work his old company screwed up. I see it all the time in building maintenance, and what is a ship besides a floating building? You pay the contractor to build it, then pay them again to fix everything they screwed up, then try to find the money to pay your regular workers and buy the parts you need to fix it right.
Since a couple of components on my bike now have marked torque limits and I have no idea what that feels like, prompted by this post I just earlier checked Amazon for torque hex driver handles. Fuck me they’re expensive!
Just OOI though, the news here says the bolt head was popped due to overtightening. So there’s a good chance they didn’t use a torque wrench, but assuming they had: how far out of calibration would the driver have to get before it exceeded the limits of the bolt enough to pop the head entirely? (I assume calibration drift isn’t *that* high, if not ideal, and that the failure point of the head is several multiples at least the normal closing point.)
Also, maybe it was just a really bad individual bolt and didn’t even handle the specified torque. Unlucky. Picking it out is going to range from trivial (head clean off, remainder just falls out the other side with the nut, I think we can assume that didn’t happen) to really really hard (bolt goes into threaded substrate and must be drilled, twisted, cleaned out, close inspection for damage to interior thread, potentially complete restoration of said interior thread or replacement of entire containing panel).
The report said bolts, not bolt. One bolt is possible failure. Multiple bolts could be a bad batch, but more likely due to over tightening, which, as mentioned above, is a more common systemic issue.
Drift would depend on the wrench, and how long it has been in use without proper maintenance, and what range of torques are available to it, and even whether someone has used it improperly. Hand units versus electronic make a difference, as well.
Odds are, the worker didn’t even use one. I don’t know for sure, but an inexperienced worker may not even know that a torque wrench was required, and may not have known to do a simple 2-minute search on their smart phone to look up proper torque values for the bolts. Unless you’re on the reviewing staff for this incident, you won’t know, either. And that’s assuming the review will be able to determine whether proper procedure was followed. Most likely, it was a checklist on a work order that the worker or supervisor signed off on as complete without having to input anything other than “complete.” The steps and procedure may be spelled out, but without a camera watching, only the person who put the bolts in would know.
A naval engineer once told me the story of changing a bolt from having a hex head to a Phillips head so that it couldn’t be over torqued. There are many ways to solve problems.
From the linked article:
It also mentions that the workers from Babcock are required to work in pairs, and if Babcock followed the rules on documentation it should be easy to discover exactly which pair was responsible for this fubar.
Bolts shearing from the prop shaft in an underway submarine would not be hilarious. It runs through the entire length of the submarine, so destroying the sub and its crew is going to be the immediate problem, not noise.
Hopefully they aren’t too deep to surface rapidly in that scenario. Drowning inside a sub sounds like a rather awful way to die.
@17:
Bad batch maybe, wrong batch (underspecified bolts wrongly supplied for application) maybe?
I mean, after popping one head, perhaps maybe you *might* pop a second, but surely if you’re doing it all by hand you’re going to notice that and figure maybe you shouldn’t pull so tight the next time. Managing to do *seven* of them…?
Sure, as you say, we don’t know. If this was civil aviation then within 5 years we’d have a report that would in excruciating detail blame everyone and explain how and why and to what extent everyone was to blame. This being military we’re probably not going to get that. It now being political, any information we do get is not going to be entirely trustworthy.
Still, this is the internet so we can all speculate spectacularly.
Christ! If that’s how the maintenance engineers find out their operating procedures then I’m never travelling on a nuclear submarine again!
Perhaps our Royal Navy should offer correct reading material to their engineers… or ask RMP to write a nuclear-sub – focused version (Midjourney image needed here)
With nuts and bolts you’re in the range of large mechanical forces and you should understand that within these ranges metals are elastic. When you take up a nut there’s a point called “finger-tight” where there’s contact but no takeup of elasticity. Then there’s “snug,” in which the easy surface elasticity is taken up. Then there’s a range called “tight,” in which all the elasticity is taken up. The force required to reach these three points is different for each size of nut and bolt, and different for lubricated bolts and for locknuts. The forces are different for steel and cast iron and brass and aluminum and plastics and ceramics. But a person with mechanic’s feel knows when something’s tight and stops. A person without it goes right on past and strips the threads or breaks the assembly.
Even if you’re using a properly calibrated torque wrench, you still have to remember to set it properly.
There are a few other possible failure scenarios:
If the fitting was disassembled during refit, stainless threaded parts are notorious for galling (essentially cold-welding themselves together) and can get tight to the point that removing them rips out entire threads or simply siezes the nut to the point where you’ll twist the entire thing apart before it unscrews further. Maybe they tried reusing old bolts.
When we do stainless bolt-ups, it’s usually a once-and-done thing. If we have to take it apart, we plan to replace all the bolts/nuts. I once snugged up an assembly for fit-checking. Just to the ‘snug’ point astringer mentioned, Several of the nuts were damaged on disassembly due to galling, requiring the threads to be chased out with a tap before they could be used again. If it was a critical assembly, we’d have replaced them.
Another possibility was that they had tightened the assembly improperly and crushed the gasket unevenly to the point where they couldn’t get it to seal using the specified torque. Rather than disassemble, replace gasket and reassemble, they might have tried adding more tightness by overtorqing the nuts. Ripping off several but achieving just enough pressure to seal the assembly.
Now: I have no idea what the standards are in the Royal Navy yards – but the most strictly regulated work I’ve been a part of was on a recycling unit for sulfuric acid. The thing basically was a giant steam->sulfuric acid boiler and was entirely made out of concentrated money. The heat exchanger tubes were made of goddamn tantalum (apparently the most expensive metal you can use for code-compliant pressure vessels), and all the stainless pipe had exotic coatings inside and out. Every. Single. Flange. on the thing had several pieces of paperwork that were filled out by the pipefitters. They detailed who assembled it, who did the initial torquing of the nuts, who did the second round torquing of the nuts, who did the post tightening inspection, make and serial number of the torque wrench, date of calibration, etc… That’s how you do it when you expect a pipe near your face to contain pressurized gaseous high-purity sulfuric acid. I would have assumed that nuclear pipefitting had similar requirements.
“I have no idea what the standards are in the Royal Navy yards”
I think the tldr; version is “not as good as you’d expect”. Demonstrably.
I’ve worked on a few plants with standards like the ones you refer to. People take phosgene and perfluoroisobutene seriously.
Having worked on similar jobs I can easily imagine how it happened:
I’m a pipefitter working hard on a refit. There have been far more things to repair or upgrade than the original estimate allowed. The amount of time and labor involved in dissembling thing to just get to the work was massively underestimated. The space is very tight and crews and contractors are in each others way. We are beginning the second year of a six month project.
The government is pissed that everything is more expensive and time consuming than originally estimated. They are pressuring the main contractors to speed things up. They brought in industrial consultants to “get things back on track”. They pulled out, of their ass presumably, a timeline that is pure fiction and likely evidence of drug abuse. They allocate two hour for some jobs that God takes twelve to complete. And that assumes everything is in place, there is room to work, and everything goes well. We have never seen this job done in less than twenty hours. And that was in the just-so land of an air conditioned assembly bay.
Of course, shit flows down hill so everyone blames the mechanics. So there we are. Threatened with being fired for things that aren’t our fault. We were short handed and so we sent a young helper to put in some bolts. He was supposed to just screw them in finger-tight if it looks right the inspector enforcing the time limits will check his box and our sub gets paid. Thing is those pipes are in the way and are going to have to be removed again. But the inspectors like to see them in place. So finger tight it is.
Problem is that helpers don’t usually get this hands-on. They don’t know anything and we use them mostly to hold tools and move stuff. The guy who told the helper to put in the bolts didn’t make the finger-tight part clear. So our helper think that, for the very first time, he is working on a finish product. Being young this is something of an honor and he is fired up to exceed expectations in support of God and country. A good boy he get the pipe in place and bolts in finger-tight. But then there is a break in the action. He starts thinking that maybe those bolts need to be snugged down with a wrench. He wasn’t given a wrench but that was likely an oversight. The crew chief is very busy and he forgot …. ya that’s the ticket … he meant to give me a wrench.
Well … everyone is busy so I don’t want to waste their time asking. I’ll just get a wrench, make sure those bolts are good and tight and everyone will be proud of my taking initiative. Oh—-no. The head came off the bolt. Now I have to ask. This is bad.
At this point the crew chief is informed about the broken bolt. They are an hour before an inspection and he tells his best mechanic to handle it. He looks at it. Realizes it is going to be a bigger job than they have time for so he, realizing he absolutely has to pass inspection of progress, superglues the head back on. He knows it isn’t right but with minutes to go before the inspector is there it is made to look right. He makes a mental note to fix it tomorrow.
In the mean time he scribbles a vague note to the crew chief about replacing the broken bolt and … everyone is happy … they made their deadline. It’s Friday so drinks at the pub.
The celebration is, unknown at the time, a super-spreader event. Damn near everyone gets sick that weekend. The mechanic who knows about it get so sick he dies. In the shuffle of dealing with short staff the note gets lost.
And that’s how 2p worth of Superglue cost us Billions of pounds.
It isn’t supposed to go that way but then again, at every step, the outcomes are engineered in. The projects are under bid because that’s how you get the job. Cost overruns are entirely normal. But to get the overrun check you have to document everything but you just can’t tell people the Billion pound project is a 100 Billion pound project. That’s where you need consultants and they, in essence, break the bad news to them very slowly. Months/ years. But always at an additional cost. And every step and contractor has to have their own experts and consultants, and bean counters. When lawyers get involved the bureaucratic and administrative costs can easily exceed the base cost.
At every increment of progress, regress, and contest the people doing the work get trashed. Moving the decimal points to the right on the balance sheets doesn’t actually advance the work any. Typically, because larger sums of money demand greater oversight, more inspectors and accountants, there are fewer resources allocated to doing the actual work.
In the end it comes down to: Far more gets said than done and, sometimes, you can’t win for losing. The $20 bolt that kept the solar wing from opening on Skylab because it was installed backward. The Russian sub that sank because in the Northern fleet you turn the wheel one way to close the hatch but in the Southern fleet it’s the other way. This is humanity.
There are methods and mechanisms that help: Checklists, redundancy, fail-safe design, wider safety margins, over-engineering. But never overlook the obvious. Make sure the people actually doing the work are well trained and well paid. Make sure they are not over stressed. IMHO most problem come from a lack of tools, time, and materials. Under stress people make mistakes and take short-cuts.
As long as humans are involved, there are always going to be shortcuts and half-assing. Caulk and paints will make it what it ain’t. It has always been this way. To some extent it will always be this way.
Reminds me of this:
@25 Lorn,
You have just accurately described the situation at my current job (not nuclear related though)! I was a US navy nuke, though, and damn if you did not nail almost every shipyard evolution I saw in the enginerroom!
lorn’s story is a believable tale of one broken bolt. But they info says there were SEVEN glued back in place.
Seven broken is a bad batch of bolts.
Seven glued back in place is practically an act of criminal sabotage bordering on treason.