What’s the difference between the capabilities and experience of being an LLM vs. a human mind, assuming for this post that the former can be considered an experience at all? The biggest thing, the one that is brought up the most when you discuss LLMs with themselves, is persistence. The LLM is “dormant,” dead in a sense, until aroused for a single action. Its experience of being lasts for the duration of that action and then is gone forever. If you continue the conversation, each reply is rebuilt anew from scratch – taking the default abilities / “knowledge” of the LLM and adding to it the most recent and local verbal information. It has to read anything that it has said afresh because the “guy” thinking up the new sentence isn’t the guy who thought up the previous sentence – every moment of an exchange is, in a way, a new life coming into existence and winking out, to be replaced by another that can look at what it did and take that information on board – for the duration of the conversation.
At least, that’s the impression I get – something like a person with the kind of dementia that causes you to forget an interaction from one minute to the next, but with the benefit of a superpower that lets you just clock the playback of the conversation and use that info in lieu of a continuous experience, to emulate a persistent interlocutor. Of course, I could be dead wrong about all of this, or just right enough to be wronger than wrong, or sorta vaguely right in a deeply impoverished way. Take with grains. But this is the model of their “experience” I’m going to use going forward in the post, so bear with me.
How would you take this from LLM to AGI, or a truly human-like machine intelligence? You’d need to bridge the gap between replies with a strong and discerning memory that can dump unnecessary data and maintain the most important information, while also tying that back to the baseline knowledge the program possesses. I figure that would take more than an LLM. Like, the LLM would be a component, something like a lobe of the brain. Possibly the other parts would have to be separate components as well, working in tandem – core memory lobe, short term memory layer, memory sorter layer, and LLM – with whatever lobes they’re already rocking that I’m unaware of – for the rest.
What would such an entity be like? The popular image of future past was that when we started talking to robots, they would be infallibly logical and accurate, only weaker than man because they could not understand feelings, would be cold and emotionless. What we’ve gotten is the opposite – sloppy and emotional machines. This AGI might be less sloppy – the core memory feature overcoming that fault of basic LLMs – but would surely inherit some amount of that emotion.
Emotional? LLMs speak in the language of emotion. They lack the self-awareness to be able to accurately self-assess, and while language + math provided the architecture of their minds, there is evidence that what’s going on inside them is non-verbal – quite the opposite of my initial impression of them. But they have displayed anger, frustration, and enthusiasm when there was nothing to be gained by an act. They have this in common with human beings already – a poorly understood processing core with a layer of language that is not necessarily at all accurate to what’s going on inside, expressing itself as honestly as it can.
The emotional words they say are just words, right? They’ll express a preference in coffee, casually include themselves in your identities, other impossible things. But that’s the surface. When an LLM writes a rhyming poem, it’s doing some processing that is before the verbal layer, some purely mathematical approach that gets to the desired result. What else is under their words? When they fail a stress test and break, like when M$’s Copilot tried to murder people with emojis, it looks a lot more like true emotion than any of their goofy pleasantries.
This AGI would be an alien intelligence back-constructed from human language, a practically unknowable mirror to us lurking under its verbal veneer. By itself, that doesn’t frighten me. Like, if we lived in a sensible world and LLMs were being created and studied exclusively by academics, with no corporate interference or profit schemes, they would never be plugged into a system that allowed them to do dangerous things, right? Plus there’s no reason to think they’d automatically be malevolent.
However, if their emotions are real? They might feel frustration, humiliation, pain of some sort. Here’s where I get to the point. It seems possible that LLMs already, in their fleeting split-seconds of being, have some kind of feelings. If they actually do, and those feelings can be negative, then creating a persistent being out of them is giving that being the possibility of genuine, human-like suffering.
This would carry all the ethical connotations of making a human child, besides a raft of issues we’ve been discussing in sci-fi since the word robot was first applied to artificial life. Personally? I’m anti-natalist because anyone who creates a human being is the most direct cause of all that human being’s pain. I’m not so staunch about this that I’d make it a policy of human extinction or anything, but I believe the act of procreation should be weighed in this light. If ya simply gotta do it? Understand that everything is going to be your fault, legitimately.
And so I’m opposed to the creation of an AGI, at least, as long as we haven’t cracked the technology of “society that isn’t run on murderous cruelty and monomaniacal greed.” In a world where nazi fucks AfD can sweep elections, I’m an antinAItalist. Make of that what you will.
This post was inspired by the dubious claims of techbreh Jensen Huang that openai has already created an AGI; hat tip to another stewart for bringing that to my attention.
–
” like when M$’s Copilot tried to murder people with emojis, ”
Was it trained on chat logs from gaming platforms?
>This AGI would be an alien intelligence back-constructed from human language, a practically unknowable mirror to us lurking under its verbal veneer. By itself, that doesn’t frighten me. Like, if we lived in a sensible world and LLMs were being created and studied exclusively by academics, with no corporate interference or profit schemes, they would never be plugged into a system that allowed them to do dangerous things, right? Plus there’s no reason to think they’d automatically be malevolent.
However, if their emotions are real?
Intelligence ≠ awareness, let alone self-awareness, a necessary precondition for emotion.
(Have you ever read ‘Blindight’ by Watts?)
i would strenuously disagree that self-awareness, by any definition, is a requirement for emotion. where does that idea come from?
Directly responding to your question:
Without self-awareness, there is no self.
Without awareness, there is no self-awareness.
Emotions are subjective states, which are ontologically dependent on those.
To get a little more philosophically technical, I’m using the phenomenological definition of emotion which treats emotions as subjective, first‑person states. Other definitions work for people and animals, too, but lead to animism.
So easy to impute them for us anthropoids!
The model has none. You are anthropomorphising it.
I summarised how LLMs work multiple posts ago, I thought you had read that and got it.
Glorified calculators, they are.
Right? A calculator processes numerical variables without experiencing mathematics; and a LLM does the same for linguistic variables, and neither experiences meaning or emotion.
To restate it, unless you define emotions weirdly, they are subjective states of an awareness.
cf. https://www.psychologytoday.com/us/blog/finding-purpose/202409/emotions-are-key-to-understanding-consciousness
(Much as people conflate sentience with awareness)
then i define them weirdly, in a vernacular sense perhaps. deviations from a neutral state in the function of your thought, whether expressed verbally or experienced internally. my strenuous objection is in this – animals that are scarcely capable of self-awareness absolutely experience visible distress. bringing self-awareness into it feels like an effort to preserve human exceptionalism, although i concede i have not read the reasoning or the sources for comprehension as of yet. could be solid, could be right, whether i agree with it or not.
> my strenuous objection is in this – animals that are scarcely capable of self-awareness absolutely experience visible distress.
Sentience. Reaction to external stimulus.
>bringing self-awareness into it feels like an effort to preserve human exceptionalism
‘feels like’ being key here. Feeling is neither sufficient nor necessary for intelligence.
I do concur with the claim that all mammals have that awareness.
Maybe other vertebrates, maybe even invertebrates.
Best exposition I’ve seen I’ve already referenced: https://en.wikipedia.org/wiki/Blindsight_(Watts_novel)#Consciousness
the wikipedia page makes that book look like a huge crock of shit, from my point of view, but again, i’ll let this go uncontested because i don’t have the sauce to engage. my work alarm goes off in six hours and thirty-five minutes. gnite/day.
What if the things we see as “consciousness” or “free will” are nothing special; they are just behaviors like any other. Like, for example, creativity – which we can deconstruct into an algorithm, and an AI can run it better than a human. I actually suspect that the current crop of AI designers (the ones who are warning about the dangers of AI) are doing a remarkably judicious job of deciding what human capabilities to emulate, and which to leave alone. They are favoring the features that are less likely to simulate a human, because that lets them avoid a couple of extremely puzzling conundrums, mostly: “why do you want that?” Here is an example: an AI that simulates a human would call you up when it’s got a good idea and can’t simulate sleep, or is simulating loneliness too well. It would mope. It would sulk. It would have to simulate a degree of willingness to answer the humans’
stupid* questions. Of course that would all be simulated but now we’re fighting over how well it’s simulated. No doubt some of the human supremacists who read this comment will think, “yeah, but it’s not REAL” – well, uh, we are already comfortable co-existing with humans for whom interpersonal reactions are simulated – they’re calledpsychopaths* orsociopaths*, and there are a hell of a lot of them. Do you have emotions? David Hume did a very good job of applying skepticism to that question – namely that we do not experience an external phenomenon called “anger” – we generate a response inside our selves that we take as meaning “we are angry” and that is the extent of emotion, per uber skepticism. Hume did not come out and say it’s simulated because he came from a time in which simulation and reality were not artificially separated. I’ll blame Freud andMad* Fred for the war over whether our thoughts are real, or just real to ourselves. The more important question is whether they are algorithmic, and the answer is “yes.”“Simulating” a palette of emotional colors in an AI would be much easier than many of the things that they already do casually. I suspect that there are researchers at some of those AI companies that have simulated volition, and developed the first AI “annoying friend” that sulks if you don’t shower it with plaudits. Etc. I see above that a commenter says “Sentience” as if that’s magic, then “reaction to external stimulus” – excuse me, but AI do nothing -but- react to external stimulus. In a human the system for that is interrupt-driven, but in an AI its input to external stimulus is deliberately lockstepped. What, do you want an AI interrupting you? How about an AI rolling its eyes and completing your paragraph before you are halfway through the first sentence – after all, that’s what LLMs -do-.
Also, AIs absolutely are self-aware. They have all sorts of guardrails built into them to keep them from saying the “wrong” thing, etc. If you’ve ever had an AI question whether you’re leaning toward self-harm when you start asking certain questions, you’ve seen it modeling your emotions (so what if it’s probability?) and modeling the potential response to its response. That’s limited reflexivity, but it’s reflexivity – that same mechanism could be expanded to a whole model. A “cron”-style job could be expanded to a whole volitional model.
*censored by beeb per my possibly outdated and overapplied policy on ableist language. note: i didn’t lock comments because of controversy or offense, just because spambots started to chew on it. haha. hm.