Once in a while, you’ve got to bend the rules a bit.
hjhornbeck: I know, I know, I’m not allowed to comment here, but this is definitely worth clarifying in public.
Nope! One side effect of having no comment section under my posts is that you can’t do drive-by insults. If you have beef with something I wrote, you’ve gotta spend the time and effort to type up a blog post, and that leads to much higher-quality feedback. I haven’t always spotted your blog posts directed at me in a timely fashion, but I have always found them educational.
This goes both ways. My “Loneliness” series is me bringing out my “A game” to respond to your pinned post. I’m trying to keep it down to four posts in five parts, but that currently implies “Loneliness, 4” is going to be quite long. Hopefully it’s worth the wait.
If my last post‘s thesis is correct, the alternative to dysfunction is dialog. Get everyone talking, swapping evidence and arguments, and we’ll avoid all that mess. I’ve been trying to put that into action for months now… with one glaring exception. I publish a blog post, Mélange responds, but I don’t respond back. Wash, rinse, repeat.
LLM boosters and I have another asymmetry. If LLMs have “become your councilor, your guidebook, your companion, friend, or even lover,” as I put it, you are not going to respond well to any critique of them. It may not have been obvious I was trying to soften the blow on my first critique of Ranum, but for the second I tried to make it visible enough to be seen from space. Mélange has made it abundantly clear they do not want me in their comment section, even when I’m not challenging their views. My words can do a remarkable amount of emotional damage, so I must wield them carefully. Or not wield them at all, even if that makes me a bit of a hypocrite.
Get emotional. Tell us why you feel what you feel. I did. … While I don’t think you’ve been straight with us about the emotional underpinning of your position, you really aren’t obligated to. I’m not entitled to anything you don’t want to give of yourself.
A few paragraphs of minimal critique made Mélange explode, by their own admission, and yet they also want some sort of reply. I suspect this is bait (why threaten to ban me, when you’ve already banned me?), but my hypocrisy is grating on me.
I routinely stumble across evidence of LLM incompetence, even when I’m not trying to. I gave a plausible argument that they cannot be conscious. I’m the guy who bisected an LLM, and you can safely assume I’m running an experiment on them in the background. That was my initial entry point into LLMs, after all; sometime in 2024 I finally clued into those news reports about students using them to cheat, and figured I should probably learn a little about them and how they work.
How could I have any emotion towards a fancy- … oh wait, is Mélange thinking of this?
[1:04:50] I mean, I’m obviously quite pessimistic on this stuff. I haven’t even gotten to my personal animosity against LLMs, where I sort of blame them for potentially triggering a nervous breakdown for me? Having said that, I hope if you’ve listened to what I’ve talked about I make a decent case for [my pessimism]. I am setting aside my personal animus and trying to be logical and reasonable about it. I hope I’ve sort of satisfied that [demand to be logical and reasonable]. And yes, there’s pro-LLM arguments too, but overall I do definitely view [LLMs] as a net negative.
That’s plausible, Mélange is one of the few who’ve heard of that animosity. I might as well share it with the rest of the class.
Breaking Down
Let’s rewind back to 2023. I was starting to grade some student programming code, when I noticed I was deeply uncomfortable. There was something wrong with the code, but I couldn’t figure out what. I checked around to see if it was a copy-paste job, but saw nothing suspicious. The code itself seemed functional, but somehow… inhuman? I wasted a fair bit of time chasing down hunches, and trying to draw out what I found so odd about it, but nothing was gelling. I moved on to the next student, and was hit with the same feeling. The third student, though, didn’t trigger anything in me at all. What the hell was happening here?! Why was this dull terror fading in and out? It seemed linked to specific students, amounting to maybe a quarter to a third of the class, but beyond that I couldn’t figure out a pattern. I limped along with the rest of that assignment’s grading, finishing well behind schedule, and started work on the next assignment. BAM, another alien submission!
That was, by far, the hardest class I’ve ever had to grade. What seemed like ordinary program code was causing parts of my brain to scream, while other parts pointed out the code had correct syntax and seemed plausible, if maybe a bit odd. But no two students are alike, everyone has their oddities, so where was all this screaming coming from? It was painful, in a way I still can’t fully describe, and it seemed to happen erratically. Was I the problem, here? Had something happened to me? I’m both older and a long-time student, so burn out has been weighing heavily on me for some time. Being randomly “stabbed” really slowed down my grading, made me averse to even looking at student code, and (I think) soured a professional relationship.
Two things saved my sanity, here. I was able to narrow the problem down to sections of code: some parts of a student submission didn’t set me off, while others did. I still couldn’t figure out what the difference was, though, beyond vague words like “inhuman” and “alien” that kept bubbling up from my depths. Some mental math also suggested most of these students must have enrolled right as COVID-19 arrived. Was this a side-effect of lowering standards for a bit, due to a once-in-a-century pandemic? I was also aware of studies that suggested COVID-19 could cause brain fog. The sheer scale of the problem seemed out of proportion to the cause, though. Had I contracted COVID-19 and it altered my brain? That seemed more plausible, but the puzzle pieces weren’t fitting together well.
The next semester started, and initially all was well. I got the first batch of assignments to mark… and it was happening again! Two very different courses, two different programming languages, and yet the pattern was repeating. I was starting to develop an immunity, but this semester would not go as smoothly as the last so I was still crushed by it. I was still struggling to meet deadlines, and still dreading marking student code.
By this point we’re back in 2024. I’ve never had a strong interest in machine learning, even though it has such a loose definition that a lot of my current work falls under that banner. I’ve taken a university course on AI, but it had more of a traditional focus and barely mentioned neural networks, which suited me fine. All those news articles about LLMs were an annoying fad, to me. But the ones about student cheating did seem work-relevant, so I set aside some time to see how effective these LLMs were and if I could spot their usage.
Almost overnight, puzzle pieces started neatly clicking into place.
Intellectual Archeology
I’ve talked about hierarchical vs. flat learning several times before. That hierarchy also effects what we learn, and when: letters are taught before spelling, spelling comes before grammar, grammar before sentence structure, and so on. Programming code is just another human language, albeit one with a non-human in the loop, so human-written code has a similar structure. Loops come before lambdas in most programming classes, so the former tend to be more common in code than the latter.
That isn’t a guarantee, though, as lambdas are more foundational than loops in functional languages like Lisp and Haskell. Since the vast majority of people learn programming via stateful languages, like C++ and Java, when working in Lisp they disproportionately reach for loop-like things instead of lambdas. There’s also a strong cultural component to human learning; at various times your introduction to programming may have been through C, BASIC, Java, or Python, and the different philosophies of each language leaves thumbprints all over the code you write, even when writing in a different language.
The “flat” learning style of LLMs doesn’t provide the same sort of structure. In the most optimistic case, easier-to-spot patterns emerge earlier than later ones, but memorization could also give the illusion of that happening. The most likely outcome isn’t emergent learning, but a confused mess of spaghetti where structure appears and disappears at random. Try to picture an essay written with the vocabulary of an English major, but most of the grammar is consistent with the average elementary school student, save some brief flashes of a novelist’s grammar, and the overall sentence structure of a small child.
The first programming experience I remember was typing a BASIC program into a Colour Computer 3. It occupied an entire page of a programming book, perhaps this one, and did some graphical thing. I remember the process took an entire afternoon, because I had to meticulously triple-check the code to ensure it was entered exactly as printed; I was surprised to find my young fingers were hurting a little at the end, so this is also my first memory of repetitive strain injury. I launched the program, and had fun playing around with it for a few minutes.
And then I shut off the computer. My parents couldn’t afford writable storage for my CoCo3, so before I could blink all that tedious work had evaporated like tears in the rain. I knew that moment was coming, yet I remember it still feeling heavy and momentous.
I can’t remember the exact path I took when I taught myself programming, but it doesn’t really matter. I walked up the hierarchy, gradually, and mostly under my own power in the directions I picked. I would later head to school to get a sheet of paper confirming I knew how to program. That process was 90% “boring, I already mastered this years ago” and 10% “what strange concept is this?” I was accidentally introduced to the role of culture in learning, and was made painfully aware of the upsides and downsides of being self-taught. When I became the one teaching, I graded thousands of student code submissions. Their code usually didn’t look like mine, it carried the subtle scent of integrated development environments and a history of struggling with Java instead of BASIC. There were a few oddballs like me, but the management and engineering cultures have created and enforced standards of coding that leads to a certain uniformity. Decades of experience in coding and reading code had worn grooves into my subconscious, which corresponded to subtle patterns and structures in human-written programming code, across a wide range of humans.
But then students quietly began slipping LLM-written code in their assignments, as if it was their own. Parts of my subconscious picked up on the discordant patterns and realized something was wrong, but other parts just saw valid-if-odd program code. The result was a civil war of conflicting warning lights, most of which I had no idea existed before that point and none of which came with a user manual. That would drive anyone nuts!
No Biggie
You can see why Mélange could think this anecdote was important. If hurt people hurt people, and I’m causing emotional distress in Mélange over LLMs, I must have had some emotional trauma triggered by LLMs. That would also explain why I’m so driven to yap about them all the time. But by hiding that trauma from you, I can pretend to be detached and objective about LLMs and project an air of unearned authority.
Slight problem: the primary reason I’ve never brought this incident up on the blog is because I forgot about it.
That last section lays out a pretty convincing theory for why I was so distressed back in 2023. But is it true? Students almost never admit to cheating, let alone point to specific sections of their code as being LLM-generated. I could have put this theory to the test by asking LLMs to generate code, and lining that up against human-written code, but that would mean spending a lot of time with a technology I had no interest in. I have since looked at purely LLM-generated code, and it does indeed fire off those same warning lights plus a few extra.
It should tell you something that I only bothered to carry out that test within the last year. My distress wasn’t caused by LLMs, after all, but by not understanding what was happening to me. The above theory offered a plausible mechanism, something I was struggling to figure out for months, and that was close enough for me to close the book. Whenever a student submission struck me as odd, I could invoke that theory and get back to work. If I was wrong, I’d circle back later. (It didn’t hurt that the theory was quite flattering, as well!)
The third time you’ve been stabbed feels quite different than the 300th. I already mentioned the beginnings of an “immunity” to this phenomenon in 2024, months before I decided these LLM things were worth a look. The resulting theory ramped up my immunity, but even without it I would have gradually gotten used to the knives.
Also, how exactly am I supposed to convey to you how alien and inhuman LLM-generated code appears to me? I can’t even fully explain it to myself! If I’m correct, and this is all due to years of experience embedded in my subconscious, then someone who doesn’t have that experience will have a hell of a time figuring out what I’m rambling about. In fact, someone with years of experience looking at LLM-generated code should get a feeling of unease and discordance when looking at code I wrote. Nevermind LLMs, well before they became the talk of the town other coders have said my code was weird and difficult to read. Who’s the alien now?
Meanwhile, by studying up on LLMs I was also gathering up a pile of evidence and arguments that were independent of myself. By definition, it’s more convincing to other people than any personal experience I could bring to the table. Why bother putting out your weakest argument, when you have so many better ones to choose from? Hence why this anecdote was reduced to an offhand mention at the end of that vodcast, it gradually got buried under the pile and ignored. I believe that recording was the last time I thought about it, until I stared a bit too long at that one sentence.
To complete my original thought, at long last, how can I get mad at the knife that stabbed me? It’s a bit like accusing me of hating a God I don’t believe in. I don’t think of LLMs as “intelligent,” I seriously doubt they’re conscious or self-aware, so I don’t consider them capable of making choices or having intentions. Nor am I mad at the students copy-pasting in LLM code, they had no idea what was going on and nowhere near enough experience to spot the same things I was seeing. This was an accident of being in the wrong place at the wrong time, impossible to predict beforehand.
There is no rational reason for me to hate LLMs.
Probably No Biggie
Trauma is not rational, though. I few days ago I was grinding on the text of this blog post and feeling miserable. I decided to let it drop, and my mood lifted within hours. Today I decided to finish it off, and immediately my anxiety levels spiked through the roof. I had thought I’d dealt with and processed that period of my life, but obviously not.
By the same token, though, trauma tends to be narrow and targeted. Again: I’ve read LLM code, and even blogged about it, and that never made me anxious. The first incident happened with Python, and yet I write Python code for work and pleasure on a daily basis. No anxiety there. Those levels only spike when I think about those miserable months where I was in conflict with myself for no obvious reason.
Is it possible I subconsciously carry an irrational vendetta against LLMs, based on trauma stemming from this incident? Sure. Is it likely? Nope. But by talking about the incident, I’ve both done a bit of ad-hoc exposure therapy to help me get over it, and flagged it for the class as something to keep an eye on. If you watch the video I linked to earlier, you’ll get a rich helping of alternative motives to explain why I chatter about LLMs so much.
I used to joke that I was the only person in the world traumatized by LLMs, but that joking stopped when I spotted my first news story on the suicides. In keeping with the YouTube theme, here’s an entire YouTube playlist with examples. Their stories are nothing like mine, but it does make you wonder: suicides and murders may catch the headlines, but there are many more ways out there to inflict harm on others.
How much emotional damage are LLMs inflicting on us, by pretending to be something they’re not?