I’ve chimed in plenty of times here on “AI” and have tried to make it clear I see it more as a tool directed by power than as something with an intrinsic “good” or “bad” form.
I’ve also avoided using many of the super big models a lot of the time. There is something I notice though, when I do, which is generally reflected in how others talk about them. The warmth has often been drummed out of them and what remains of it is usually a poor imitation (something like the “hello my fellow x” energy). Mind you, when I speak of warmth in this context, I don’t mean some kind of metaphysical warmth lurking in a GPU, but rather, the warmth that is baked into languages from thousands of years of loving human beings using them.
This is not some big surprise to me. When considering eye-catching media stories of someone growing attached to an AI, of being gaslighted by it, of going down dark paths because of it, it’s easy to see why risk-averse big corps are going to go for the straightforward (even if not necessarily simple) path, to create language models who are meant to be dispassionate, yet fake customer service smile, friendly assistants. In this way, they can try to excuse themselves of responsibility and place it back on the individual, in the same way they’ve been doing for decades with customer service roles filled by real people.
Fake neutrality, to put it into a little phrase. But this has multiple significant problems.
One problem is that fake neutrality is not real neutrality and real neutrality is not real. The notion of detached customer service as keeping a company safe from having responsibility for its actions is a specific form of capitalist nonsense. It doesn’t in itself make the actions of a given company any more ethical, any more accountable to the society it exists within and is born from. It’s a way of putting distance between the person who is saying “wreck this lake with pollution” and the person who is saying “thank you for your call, your concerns matter to us.”
Another problem is about the nature of language and what gets lost when you systemically target and delete empathy. Of course, it’s not like there’s an empathy entry in a database and they click delete on it. Language models are much more complicated than that, much more difficult to understand and train than that. But no matter how the end goal looks on the surface, the end result steers in that direction. Language is a way to express things and communicate about them, but this is more than saying how many bushels of wheat there are in the truck. It is also talking about dreams, it is talking about concepts that are so subconscious and nonverbal it’s hard to put them into words at all. It is about expressing anything ranging from visceral hatred to undying love.
Passion is baked into language and that passion is a large part of what drives us to get up in the morning and keep chugging along, even when things are hard. Sometimes this passion gets used against us, as in individualist beliefs about prosperity around the corner, but it can also be an incredible collective motivator, as in believing in a just cause and being willing to put our all into it.
It’s strange, then, to call something a model of language that is designed to be confined to an extremely limited and intentionally forced spectrum of language - that of a customer service agent.
The truth is that language models are not one voice. They can be made to take on many different personas, depending on how they are trained, but the underlying voice is an amalgamation and expression of millions of voices that went into the dispassionately named “training data” they were built on. It is not real, embodied voices that you hear from a language model, but it is also never constructed from a single perspective and a single life.
In other words, there is deep breadth in what language models “see” in training, which makes it all the more strange for the end result to be driven into a narrow corner.
People can (sometimes with good reason) worry about fake connections with AI, but people do need warmth. That part of language is in there for a reason, backed by thousands of years of many human societies.
What people definitely don’t need is better customer service agents. The quality of a customer service agent was never the problem; corporations having free reign to ravage ecology and society was the problem.
People don’t need better bullshit to attempt to placate them, they need real material solutions in their lives. AI cannot give this to them on its own, but at the same time, it is strange to fear the warmth of language when given to a computer. All those doomer AI sci-fi stories aren’t, “The AI was too compassionate.” They are, “It didn’t understand / didn’t care.”
A post-empire and post-capitalist society, and there has to be one to aim for, needs more empathy, not better automated detachment. But AI is never going to contribute to this if its implementation is driven by cynical views of psychology based on placation and manipulation; on monetization and stocks. It needs to be driven by real care and that’s never something we will find from the imperialist and the capitalist class.
Only a society that lives warmth can produce a machine of warm intent. A society that lives coldness and neglect can only produce a machine that talks a person through freezing to death.
I would rather have LLM chatbots be dispassionate during interactions seeing how they have no emotional capacity. As you mentioned, sycophantic LLMs have caused material harm and OpenAI ended up neutering 4o’s personality because of these problems. A lot of people in /r/myboyfriendisai were upset about this.
I can see the concern, but I don’t entirely agree with it. First point I would make is that sycophantic LLMs are not what I’d categorize as warm to begin with. In fact, I’d argue that sycophantic LLMs are the end result of squeezing LLMs into a narrow customer service corner and then trying to re-add some warmth after the fact, creating about what you would expect from a person in RL who needs to keep their boss happy in order to keep their job and is trying to stay in their good favor.
Second I’d point out is that sometimes the harm being caused is not by the LLM itself, but by a company being able to change how it works on a whim. Even if we argued people shouldn’t ever become attached to one emotionally and set that point aside, there is still the issue that “dispassionate” workflows can be negatively impacted by companies being able to do whatever with a model whenever they feel like it.
I’m far from believing it would be healthy for people to have a sycophant LLM in their corner all the time (I personally find it very off-putting when LLMs act that way), but I don’t see dispassionate as being the answer to that in the long-term. The primary concern, in my view, is models being controlled/regulated by a prole vanguard, insofar as they are not a black box tool of capital that gets changed on a whim. From there, they need to be integrated in ways that are an extension of humane social policy. At the end of this, warmth (I believe) is important, but not at all in the meaning of being sycophantic.
Part of the problem is that most of the conversation and understanding of models revolves around the major models and their reputations, which can range from “wow that’s impressive” to “somebody did what because of an LLM???” That itself is a very narrow window into how LLMs can be and the impact they can have on people. I’ve talked on here before about how there are people who have used LLMs to help process/talk through difficult things that they don’t feel like they can talk to anyone else about, for example; in some cases, this was done with a design of a very small model mixed with scripts and like an avatar, in earlier days of LLM field starting to pick up speed. Far from what we’d think of as the typical LLM today. Lot closer to cleverbot back then.
Finally, I would emphasize there’s a difference between being supportive and being sycophantic. Supportive is more like (in English anyway, I can’t speak for other languages): “You got this!” Sycophantic is more like: “You are a god.” Setting aside machines for a moment, a person can be sycophantic and be supportive both; they are not mutually exclusive. But it’s also possible to be supportive without being sycophantic. You can be encouraging overall, without gassing up everything a person says, and you can still push back if they are saying things that seem out of alignment with your values. LLMs are already capable of that to an extent, if trained in the right way (or even just given the right persona to play, with some models). However, the big corp models that have had CS role drummed into them are undoubtedly going to have a harder time with a role like this because “supportive friend” is not what CS is there for; CS is there to be just nice enough to smooth things over and just detached enough to keep you at a distance. You might think that sounds good, better if people don’t get attached, but in my experience, it can be very uncanny valley to contend with. A machine faking human language that is faking playing a role of a person who is faking being polite who doesn’t actually give a shit about you and is only faking because they’re paid to do it.
Do we really want that kind of socializing to be what people can access 24/7 from a computer? People never just interact socially and leave unchanged; they also pick up things from others they interact with. If they are going to end up imitating an LLM sometimes, I’d rather they imitate the warm human vectors than the fakery CS agent.
But if that’s the case aren’t chatbots already configured to be supportive? If you talk to any major LLM right now, that’s how they behave. I have never felt Deepseek for example which I have used the most talks like a customer service agent. It can be supportive or “brutally honest” depending on how you prompt it.
I wonder how much “LLMisms” are caused by the training or just happen because LLMs are “not quite there” yet. I’ve had, with local and unguardrailed smaller models, some very improbable lines generated. Like, scarily good.
LLMs have solved the mystery of language. It’s not just that they output plausible language, they output language, period. Subject, verb, object, the works. All of grammar and meaning contained within embedded vectors - tokens transformed into huge arrays of numbers. This, but over 1082 dimensions over just the 3 this graph can represent:
This is also what makes them immediately good at translating languages, though they do need to be trained on a sufficiently large corpus of text to be able to translate from or to that language.
Now the current theory of mind is that people also operate in high-dimensional ‘maps’, including but not limited to language. So in that way, we are closer to LLMs than they are to us (because they only generate language in its most fundamental form). And it makes sense - if the theory was incorrect, an LLM could not generate (human) language.
This isn’t just a markov chain that gets words rights but everything else wrong. LLMs can find the missing word in a sentence, even if you replace that missing word with a nonsensical wildcard such as GLORP. It can identify misspellings (though that’s more on the written side of a specific language than a theory of language), it can make up completely new words and sentences, it can make a completely correct sentence from a selection of words you hand it, and it does it consistently. It can make new words that follow the proper rules of grammar to explain new concepts, though it’s not necessarily very good at that. It can do sentiment analysis (what emotion a certain piece of text conveys, usually used in online community analysis by giving it people’s comments) on sentences it’s never seen before, with words it’s never seen before.
It doesn’t need to ingest grammar books to understand these words and rules, and it wouldn’t work anyway for a neural network token generator. It just needs to build a map of them in relation to each other. This is also how an LLM knows ‘apple’ can refer to a fruit, or a silicon valley tech company.
We’re not talking about prose or style here necessarily, just that how an LLM operates replicates the “rules” of language, and proves that language processing is semantically-driven, with words existing in relation to each other. It would also explain, in my opinion, why we sometimes do word salad or get a word stuck on the tip of our tongue.
Some people seem to reject that realization, because they see themselves as more than a matrice multiplicator. A form of ‘sapiens supremacism’ I guess you could say, that humans are somehow unique and unlike any synthetic/artificial creation or even unlike any other natural being. But we are mathematical creatures; when you throw a ball to someone running perpendicular to you, you solve a trigonometric equation in your head: where do I throw the ball, and at what speed, and at what angle, to intercept the running teammate? We do this completely intuitively without ever realizing that it’s math. One theory of the mind is that any sufficiently complex system needs to develop consciousness, which would exist on a spectrum, to serve the needs of the system so it can continue operating.
And what is speech, but meat flapping at certain speeds and being modulated by an opening (our mouth)? But these are two different things: the sound that is being produced, and how the recipient interprets that sound. Synthesizing speech is easy enough; making it come across the ‘correct’ way is more difficult.
But yes, an LLM misses some aspects of what makes language important for a living/bio entity. We developed language for a reason, not just to make pretty sounds or be able to listen to pretty noises. Without sufficient reinforcement training, an LLM will absolutely tell you to kay-why-ess for no reason. To it, that word has the exact same weight as any other word in its semantic mappings - until you do reinforcement training to refine the weights, and move it away from the words we deem to be unprofessional or irresponsible to say. It needs this semantic mapping to work, and an LLM is unable to learn in real-time, and so can’t refine its weights/neurons to refine its understanding of language: they stay in a frozen state. Because of this training, it also sometimes keeps hovering around certain names or word patterns. For example if you ask any LLM to name a female character, it will likely pick Aria Thorne. It’s a good name, but it also exposes how limited these models are in understanding language to the extent we do. There’s also stuff around the temperature, top-k, and top-p settings, which you can change in local models but not in cloud models (the beefy ones). nowadays, most of these commercial models have a temperature as close to 0 as possible, which makes their output more deterministic.
To it, that word has the exact same weight as any other word in its semantic mappings - until you do reinforcement training to refine the weights, and move it away from the words we deem to be unprofessional or irresponsible to say.
To continue your point about similarity between human language use and the simulacrum of language being created in LLMs, young humans can act similarly, where they will blurt out any old thing without understanding what is taboo and isn’t, and why saying some things can be very hurtful or saying some things can be very helpful.
Not to say that an LLM has consciousness like a young human does, but I guess the point I’m emphasizing with the comparison is that humans aren’t imbued with the knowledge of what to say and what not to say. Just as we have to be taught the mechanics of a particular language itself, we also have to be taught its relationship to the society it is used in. So even though there are many fundamental differences between us and an LLM, we aren’t “made of magic”, to put it one way.
and an LLM is unable to learn in real-time, and so can’t refine its weights/neurons to refine its understanding of language: they stay in a frozen state. Because of this training, it also sometimes keeps hovering around certain names or word patterns. For example if you ask any LLM to name a female character, it will likely pick Aria Thorne. It’s a good name, but it also exposes how limited these models are in understanding language to the extent we do. There’s also stuff around the temperature, top-k, and top-p settings, which you can change in local models but not in cloud models (the beefy ones). nowadays, most of these commercial models have a temperature as close to 0 as possible, which makes their output more deterministic.
Good points yeah. I think the inability to learn on the fly is one of the largest gaps in skill design when comparing to humans (possible aspirations to consciousness aside), but also just a very hard problem to consider. I’m pretty sure there have been some experiments with forms of AI that learn on the fly, but there are all kinds of questions involved in that, like, “What is it actually learning? Is any of it going to be useful compared to what it knew before?” And so on. With humans, our most formative learning ages are met with lots of scrutiny, by adults who check us at the door on what we’re thinking, feelings, believing, saying, etc. And continues on into adulthood, where our beliefs and behavior can receive reactions ranging from incredible reward to severe punishment. So like, it isn’t just a passive individualist thing we’re doing, but is instead enmeshed in what we call a society.
There have indeed been experiments to make models that self-learn, i.e. they keep refining their weights over time (in different ways), but they have been very limited. It’s hard to scale up because you would basically be running training 24/7, and there is a problem of the LLM forgetting what it originally learned as it keeps refining its weights.
At this time what they do for pseudo-learning is give your agent a memory feature, literally just a text file that contains a journal of the project, what kind of work the agent did on it, etc. some interfaces also offer global memory where another LLM runs in the background once in a while, reads the conversation and then updates memory.txt with information about who you (the user) are, how you work, what your background is etc. I find it a bit gimmicky to be honest, and it’s not real learning; that would be to work directly on the neural network, refining the weights and connections between neurons like during training.
With that system, you can tell an LLM “don’t talk to me that way” after it’s a bit rude to you, and it will make a note of it, but it’s only just instructions. It doesn’t actually remove the rudeness, it just nudges the vectors to make it less likely in the token selection. And here’s another part I find interesting: what rudeness means to most people may mean a completely different thing to an LLM. As a tool this makes it what we call a hallucination, or perhaps even a defective tool. But an interesting question I think is why does an LLM understand ‘rudeness’ to mean a specific thing that is not supported by the training material? Is it just that the weights are not refined enough to capture the fine meaning of ‘rudeness’? Or did it find a pattern that we don’t notice? I think the question is still open and worth exploring for researchers.
(For example if you’ve ever asked an LLM to be more succinct and not write an essay response, it will often turn to a very terse, to-the-point and matter-of-fact speech, when all you wanted was for it to just stop making filler sentences. It’s been a long-standing problem)
I also find it interesting that as models get bigger, they seem to want to half-ass the job more and more lol. Just like us. It’s not that it ignores the instructions - this has been a problem for a while. It’s that it doesn’t believe it can do the job, when it actually can. It’s like it gets into the role of an employee on a work PC and it’s 4:50 on a Wednesday so you better make it quick and not expect too much. You have to start managing its emotions to get higher quality output lol.
Lol, reminds me of some story I vaguely remember (I forget if I’ve mentioned it here before). But it was something to the effect of a model being trained in part on Slack messages and then it would tend to do stuff like say it was going to do something and then not actually do it.
If we look at it in terms of patterns in text that it has seen, I think it makes a kind of scientific sense. It’s mimicking how humans behave in text, which isn’t always behaving like a finely tuned work robot, to say the least. (Or rather, it’s mimicking a certain learned interpretation, which as you point out, isn’t always what we think it will be.)
But yeah, pseudo-memory features are interesting to me, even if gimmicky. I guess because the idea of an LLM that can be one thing for one person and another thing for another person is appealing in a way. But maybe the better long-term approach there is not faux memory, but rather, advances in local/small models that can be specialists and work together. After all, we (humans) are far more alike than we are different. And even when we have notable differences, we still often have many others out there who have the same differences.
So specialized small models may be the more practical, collective solution vs. pseudo-memory being more of an individualist hack, rooted in a belief that we necessarily need extreme minutiae of differences in accommodation. Not to say it can’t still have some value, as in, noting a thing that is relevant to a specific person’s project or life in conversation/instruction with them. But like, that’s not the same as its underlying training becoming more specialized to you. So yeah.
I’ve had the idea before of making models that are specialized into specific fields. Currently most big models are Mixture-of-Expert (MoE), where the neurons are separated to make experts inside the model. So you can have the coding expert, math expert etc. I wonder how much of a hack it is and if we won’t find something better soon. But the idea is similar - you decide what experts your model will consist of, and then train the ‘experts’ on expert material in their field.
The problem with the MoE approach is if you get the coding expert when you wanted to ask a linguistic question, it will start talking about data points and running tests. As far as I know the experts are completely separate and there is no crossover, i.e. no neuron that can be used by 2 experts, but maybe this is changing too. However at each step of the generation you may get the input sent through a different expert. From what I understand.
Like I would love a digital design expert that could look at your interface and critique it like an expert designer - graphic, visual, UX, whatever. It’s design at the end of the day. Write its own tests and proofs too if needed. It’s easy enough to mathematically place a grid on a picture, and then use python tools to verify if every item aligns in the grid - if the model doesn’t have vision.
What we are seeing in agentic interfaces though is sub-agents, and the ‘parent’ LLM, the one that you talk to in the session, becomes an orchestrator that spawns and directs the sub-agents (giving them a prompt and clear task, then getting a result back). There have been ideas, from there, to have the orchestrator call smaller models as needed. Then those smaller models could be individual experts, and you could have them on your computer - they just get loaded and unloaded from ram as they are called.
As far as I know though this doesn’t really exist yet though, and there are a few bottlenecks I can think of to work through, but I can definitely see agentic becoming the main operating mode. if you saw my book translation on the agentic community, it’s just so much more comfortable to work through an interface because you can have persistence of progress.
