The Cult Of LLMs

There’s a line in Minority Report that’s stuck with me. After looking over the crime scene, the detective character turns to another cop. “This is what we’d call an orgy of evidence. Know how many orgies I had as a homicide cop? None.” The logic behind it is simple: the real world is messy, so when you stumble on something that conforms exactly to your biases, be skeptical.

In every sufficiently large business we have observed (say, with 500+ employees), we have noted that continued advancement, and increasingly continued employment, has started to require repeated professions of belief in the transformative power of AI for said business. I am not talking about providing ideas about how to use AI in the business – I mean religious profession, declarations of faith. Overwhelmingly these statements are made by non-technicians, though it is not uncommon for technicians to emit deranged statements to curry favour.

I just stumbled into an orgy, and so my instinct is to take a step back and see if I’ve missed anything.

[Read more…]

LLM’s Shouldn’t Code

My draft for “Loneliness, 3” is currently sitting at 2,600 words. It hasn’t been as hard to write as “Loneliness, 2“, this time around I only redid the intro once. Nonetheless, I haven’t touched it in a few months. The why of it all is complicated, as usual, but one not-insignificant chunk is that I’m starting to doubt my approach. I never expected to find a “magic passphrase” to got people to understand my arguments immediately, but since starting the “Loneliness” series I’ve spent more time with people who love and defend LLMs. The additional evidence and experience suggests that series is shouting into a black hole.

I don’t want to give up on it, but taking a break from it might help get me typing again. Besides, I think I can convince you LLMs should not code. [Read more…]

Aside: Let’s Bisect an LLM!

I previously took a lot of words to describe the guts of Markov chains and LLMs, and ended by pointing out that all LLMs can be split into two systems: one that takes in a list of tokens and outputs the probability of every possible token being the next one after, and a second that resolves those probabilities into a canonical next token. These two systems are independent, so in theory you could muck with any LLM by declaring an unlikely token to be the next one.

Few users are granted that fine level of control, but it’s common to be given two coarse dials to twiddle. The “temperature” controls the relative likelihood of tokens, while the “seed” changes the sequence of random values relied on by the second system. The former is almost always a non-negative real number, the latter an arbitrary integer.

Let’s take them for a spin.

[Read more…]

LLMs and Markov Chains

Pattern matching is a dangerous business, but this is now the second third time I’ve seen LLMs compared to Markov chains in the span of a few weeks.

I think people who want to characterize that as merely the output of a big semantic forest being used to generate markov chain-style output. It’s not that simple. Or, perhaps flip the problem on its head: if what this thing is doing is rolling dice and doing a random tree walk through a huge database of billions of word-sequences, we need to start talking about what humans do that’s substantially different or better. …

One thought I had one night, which stopped me dead in my tracks, for a while: if humans are so freakin’ predictable that you can put a measly couple billion nodes in a markov chain (<- that is not what is happening here) and predict what I’m going to say next, I don’t think I should play poker against the AI, either.

This seems to be an idea that’s floating out there, and while Ranum is not saying the two are equivalent it’s now in the scientific record. Meanwhile, I’ve been using Markov chains for, oh, at least seven years, so I can claim to have some knowledge of them. Alas, I didn’t really define what a Markov chain was back then (and I capitalized “Chain”). Let’s fix half of that.

[Read more…]

Guessing the Next Number

Large language models don’t really work with languages, as we think of them anyway.

At their heart, LLMs are a sophisticated version of “guess the next number in the sequence.” Their input is a long list of integers, and their output is a long list of fractional values, one for each integer they could have been fed. The likelihood of any given number being next is proportional to the value the LLM outputs for it. We can collapse these probabilities down into a singular “canonical” output by randomly picking one of those integers, taking likelihoods into account. If the LLM is being trained, that output integer is compared against what actually came next and the LLM is adjusted to (hopefully!) be more likely to output the correct integer. Want more than one integer? Shift all the input numbers up one space, discarding the first and appending the output integer to the end, and re-run the LLM. Repeat the process until no integer is all that likely, or the most likely integer is one you’ve interpreted to mean “stop running the LLM,” or you just get bored of all this.
[Read more…]

LLMs Can’t Code

The first time I asked Claude if it wanted to play Battleship with me, it misinterpreted what I said and generated a Javascript version of Battleship. I haven’t managed to get it to run outside of Claude’s sandbox, and I never played it much within that sandbox, but I have looked over the code and I don’t see any reason why it shouldn’t run.

There are good reasons to think LLMs should be great at coding. Unlike human languages, computer code has incredibly strict rules. They must, because they’re interpreted by deterministic algorithms and computational devices, which cannot make high-level inferences about what the programmer intended. Nit picking is the intended outcome here.

At a higher level, if you’ve programmed long enough you’ve noticed you seem to keep recycling the same basic algorithms over and over again. Putting things into lists is an incredibly common task, as is weeding out duplicates, or associating one value with another, or ordering the contents of a list. It doesn’t take much thought to realize that writing a generic algorithm once and re-using that will save a tonne of time; indeed, the concept of a “pattern” has been around for decades, as has the “rule of three“. The idea that an LLM that’s read hundreds of millions of lines of code could be better than you at spotting these patterns is not far-fetched.

And yes, there is that much code out there to train on. The Linux kernel itself is almost thirty-seven million lines of code, currently, and you can download all of it from Github. The two most popular compilers, gcc and llvm, have twenty-three million lines between them. While only a small fraction of it is public, Google claims their employees have written over two billion lines of code. With a large enough code base to train on, even subtle patterns can pop out.

The idea that LLMs can’t code seems ridiculous.

[Read more…]