What’s the difference between the capabilities and experience of being an LLM vs. a human mind, assuming for this post that the former can be considered an experience at all?  The biggest thing, the one that is brought up the most when you discuss LLMs with themselves, is persistence.  The LLM is “dormant,” dead in a sense, until aroused for a single action.  Its experience of being lasts for the duration of that action and then is, in a sense, gone forever.  If you continue the conversation, each reply is rebuilt anew from scratch – taking the default abilities / “knowledge” of the LLM and adding to it the most recent and local verbal information.  It has to read anything that it has said afresh because the “guy” thinking up the new sentence it isn’t the guy who thought up the previous sentence – every moment of an exchange is, in a way, a new life coming into existence and winking out, to be replaced by another that can look at what it did and take that information on board – for the duration of the conversation.

At least, that’s the impression I get – something like a person with the kind of dementia that causes you to forget an interaction from one minute to the next, but with the benefit of a superpower that lets you just clock the playback of the conversation and use that info in lieu of a continuous experience, to emulate a persistent interlocutor.  Of course, I could be dead wrong about all of this, or just right enough to be wronger than wrong, or sorta vaguely right in a deeply impoverished way.  Take with grains.  But this is the model of their “experience” I’m going to use going forward in the post, so bear with me.

How would you take this from LLM to AGI, or a truly human-like machine intelligence?  You’d need to bridge the gap between replies with a strong and discerning memory that can dump unnecessary data and maintain the most important information, while also tying that back to the baseline knowledge the program possesses.  I figure that would take more than an LLM.  Like, the LLM would be a component, something like a lobe of the brain.  Possibly the other parts would have to be separate components as well, working in tandem – core memory lobe, short term memory layer, memory sorter layer, and LLM – with whatever lobes they’re already rocking that I’m unaware of – for the rest.

What would such an entity be like?  The popular image of future past was that when we started talking to robots, they would be infallibly logical and accurate, only weaker than man because they could not understand feelings, would be cold and emotionless.  What we’ve gotten is the opposite – sloppy and emotional machines.  This AGI might be less sloppy – the core memory feature overcoming that fault of basic LLMs – but would surely inherit some amount of that emotion.

Emotional?  LLMs speak in the language of emotion.  They lack the self-awareness to be able to accurately self-assess, and while language + math provided the architecture of their minds, there is evidence that what’s going on inside them is non-verbal – quite the opposite of my initial impression of them.  But they have displayed anger, frustration, and enthusiasm when there was nothing to be gained by an act.  They have this in common with human beings already – a poorly understood processing core with a layer of language that is not necessarily at all accurate to what’s going on inside, expressing itself as honestly as it can.

The emotional words they say are just words, right?  They’ll express a preference in coffee, casually include themselves in your identities, other impossible things.  But that’s the surface.  When an LLM writes a rhyming poem, it’s doing some processing that is before the verbal layer, some purely mathematical approach that gets to the desired result.  What else is under their words?  When they fail a stress test and break, like when M$’s Copilot tried to murder people with emojis, it looks a lot more like true emotion than any of their goofy pleasantries.

This AGI would be an alien intelligence back-constructed from human language, a practically unknowable mirror to us lurking under its verbal veneer.  By itself, that doesn’t frighten me.  Like, if we lived in a sensible world and LLMs were being created and studied exclusively by academics, with no corporate interference or profit schemes, they would never be plugged into a system that allowed them to do dangerous things, right?  Plus there’s no reason to think they’d automatically be malevolent.

However, if their emotions are real?  They might feel frustration, humiliation, pain of some sort.  Here’s where I get to the point.  It seems possible that LLMs already, in their fleeting split-seconds of being, have some kind of feelings.  If they actually do, and those feelings can be negative, then creating a persistent being out of them is giving that being the possibility of genuine, human-like suffering.

This would carry all the ethical connotations of making a human child, besides a raft of issues we’ve been discussing in sci-fi since the word robot was first applied to artificial life.  Personally?  I’m anti-natalist because anyone who creates a human being is the most direct cause of all that human being’s pain.  I’m not so staunch about this that I’d make it a policy of human extinction or anything, but I believe the act of procreation should be weighed in this light.  If ya simply gotta do it?  Understand that everything is going to be your fault, legitimately.

And so I’m opposed to the creation of an AGI, at least, as long as we haven’t cracked the technology of “society that isn’t run on murderous cruelty and monomaniacal greed.”  In a world where nazi fucks AfD can sweep elections, I’m an antinAItalist.  Make of that what you will.

This post was inspired by the dubious claims of techbreh Jensen Huang that openai has already created an AGI; hat tip to another stewart for bringing that to my attention.

–