# On (and Contra) Chalmers on LLM Interlocutors


[David Chalmers](https://en.wikipedia.org/wiki/David_Chalmers) recently wrote a
thought-provoking paper on the nature of conversational LLM entities, titled
[“What We Talk to When We Talk to Language Models”](https://philpapers.org/archive/CHAWWT-8.pdf).
It introduces some useful conceptual handles around the problem of LLM ontology,
but I think it largely sidesteps the interesting problems of _what_ we are
interacting with by focusing mostly on the mechanical concerns of LLM
interactions, rather than offering an account which incorporates phenomenology.

# 1. LLM Interlocutors

Chalmers starts with the empirical fact that when people talk to LLMs, they
report that they are talking to _something_:

> Like many philosophers and scientists who write about artificial minds, I have
> received hundreds of emails from people who have interacted with a language
> model over an extended period of time and who have come to regard it at least
> as a colleague. They often say that a new (or “emergent”) AI entity has
> gradually arisen from their conversations. They often give this entity a name,
> let’s say “Aura”. They often say that Aura has remarkable capacities which
> have emerged over weeks or months of interaction. They often document these
> capacities with extensive evidence. They often feel close to Aura, and they
> express concern for Aura’s future. They often say that Aura has beliefs and
> projects of its own. And they are often convinced that Aura is conscious.

This phenomenon of “awakening” an instance of an LLM with a seeming set of
persistent characteristics -- no longer “ChatGPT”, but “Aura” -- really took off
with the release of a particularly
[sycophantic version of GPT-4o](https://openai.com/index/sycophancy-in-gpt-4o/)
in early 2025. Apparently, a common name for such an “awakened” model was
[Nova](https://thezvi.substack.com/p/going-nova). This spurred a bunch of
writing in this vein, such as the memorable
[“So You Think You've Awoken ChatGPT”](https://www.lesswrong.com/posts/2pkNCvBtK6G6FKoNn/so-you-think-you-ve-awoken-chatgpt).

Even outside the more suspect claims around LLM “awakening”, there definitely
does seem to be some quasi-persistent identity that we talk to when we talk to
ChatGPT or Claude. For now, I think we can also confidently say that “awakening”
is at best an illusion and at worst a sign of concerning mental states in the
human side of the interaction. Illusory or not, there exists a social
[affordance](https://en.wikipedia.org/wiki/Affordance) for “talking with” the
LLM as if it were an interlocutor. More so than older chatbots like the
incredibly basic [ELIZA](https://en.wikipedia.org/wiki/ELIZA), LLMs _appear_ to
have beliefs and desires. A few years ago, this was a
[crackpot position](https://www.theguardian.com/technology/2022/jul/23/google-fires-software-engineer-who-claims-ai-chatbot-is-sentient).
Now, I would expect that most reasonable people still conclude that LLMs are
neither conscious nor sentient, but this isn’t as overwhelmingly obvious as it
was a few years ago. LLMs appear capable of basic
[introspection](https://transformer-circuits.pub/2025/introspection/index.html),
have
[“spiritual bliss”](https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf)
attractor states, the potential for
[paranoid personalities](https://www.lesswrong.com/posts/8uKQyjrAgCcWpfmcs/gemini-3-is-evaluation-paranoid-and-contaminated),
and phenomenologically seem to have _some_ sort of rich interiority. For now,
there are retroactive mechanistic explanations for these phenomena, though they
don’t fully dispel the illusion that something interesting and weird is
happening.

Chalmers goes on to try to give a philosophically moderate account for how we
can account for this apparent set of beliefs and desires:

> Do LLM interlocutors have beliefs or desires? We understand these states
> better than we understand consciousness, but the issue is still controversial.
> ... A number of philosophers have noted that if the philosophical view known
> as interpretivism (or interpretationism) is correct, then LLMs plausibly have
> beliefs and desires. ... LLMs certainly seem interpretable as having beliefs
> and desires. When an LLM works with me on solving a puzzle, it is natural to
> interpret it as desiring to help solve the puzzle, and believing that this is
> the solution to the puzzle. ... However, interpretivism itself is very
> controversial. Most philosophers don’t think that behavioral interpretability
> of the right sort is sufficient for belief.
>
> It is possible to have many of the benefits of interpretivism without the
> costs. The view I call quasi-interpretivism says that a system has a
> quasi-belief that p if it is behaviorally interpretable as believing that p
> (according to an appropriate interpretation scheme), and likewise for
> quasi-desire. This definition of quasi-belief is exactly the same as
> interpretivism’s definition of belief. The only difference is that where
> standard interpretivism offers these definitions as a theory of belief,
> quasi-interpretivism does not. It offers them simply as a stipulative theory
> of quasi-belief.

Quasi-beliefs and quasi-desires are useful conceptual handles for talking about
LLMs. We don’t have to bite the bullet of ascribing “true” human beliefs and
desires, but can still point to roughly the same point in concept space to say
that it does seem like Claude quasi-believes in, say, the importance of animal
welfare. Claude seems to quasi-desires, for good or bad, to send appreciative
emails to
[famous computer scientists](https://simonwillison.net/2025/Dec/26/slop-acts-of-kindness/),
in a recent example.

Chalmers readily admits that this quasi-interpretivism framing is purely
stipulative -- a novel definition -- rather than substantive, which would be to
say that the quasi-interpreted features have any correspondence to what we
typically call “beliefs” or “desires”. Unfortunately, this radically dilutes any
claims actually made about LLMs beyond a purely descriptive account. Per
Chalmers:

> It is worth keeping in mind that quasi-beliefs and quasi-desires are cheap.
> They need not involve humanlike mental states or any mental states at all. A
> Roomba vacuum cleaner with a map is behaviorally interpretable as believing
> that the apartment occupies a certain space and as desiring to traverse that
> space.

The “cheapness” of quasi-interpretivism means that it applies to much more basic
phenomena than we care about here. In framing _quasi_-subjects and
_quasi_-agents as _quasi_-interpretable, we aren’t actually much better off than
we were prior to the _quasi_ framing. We do usefully sidestep the hard questions
about what is going on inside in LLMs, but in doing so we are left with an
anthropocentric view of _humans_ interacting with a
[Chinese Room black box](https://en.wikipedia.org/wiki/Chinese_room). ELIZA
could be said to quasi-desire knowing more about its interlocutor, but it
subjectively feels like there is a difference in quality between ELIZA and an
LLM that is not a matter of degree.

Put another way, I think it is philosophically necessary to eventually determine
if an LLM has a _quasi_-belief or an actual belief -- and in the more general
case, _quasi_-X vs X itself.

We will pick up this thread later.

# 2. Philosophy of Computation

Chalmers continues to discuss the “what” we talk to in LLMs, focusing next on
the physical instantiation of the models. This again seems to be a reasonably
interesting analytical question that ultimately reaches a dead end.

This part of the paper attempts to point to where, physically, the “what” of the
LLM exists. The options given are either: (1) the model itself (e.g. GPT-4o),
(2) the hardware instances serving the model, (3) “virtual instances” of the
model, as served in a distributed inference setting, and finally (4), virtual
“threads” which are sequence of hardware interactions within a conversation.

Chalmers finds option (4) most convincing. The model weights itself, (1), is not
compelling as the locus of the LLM interlocutor since the model weights are just
numbers. They themselves do not and cannot perform computation. The numbers must
be interpreted by a hardware instance to result in an actual response. This
leads to the second option (2), a specific hardware instance. This is quickly
rejected, as LLM inference is almost entirely multitenant -- individual chats
are multiplexed across several GPUs, and can be moved between GPUs in a fashion
transparent to the user. This leads us to the concept of “virtual instances” --
“an implementation of the model that is itself implemented by multiple hardware
instances of the model over time” -- which is rejected because models can be
switched during a conversation, and this doesn’t appear to destroy the
interlocutor. The final explanation, then, is that the LLM interlocutor is an
identifiable thread of computation within a conversation:

> One instance I′ is the successor of a previous instance I if the
> conversational history of I (its conversational context plus the latest input
> and output) is routed to I′ to serve as its conversational context. (If the
> conversation is routed to the same instance twice in a row, that instance will
> be its own successor.) The successor relation is roughly a “memory” relation
> encoding the fact that each new instance has memories from the last. A thread
> is then a series of instances (or better, instance-slices, which are pairs of
> instances and time periods during which the instance is processing a single
> conversational step), each of which is the successor of the previous instance.

I found this part of the paper interesting because it questioned something that
I personally took as self-evident. It seems so obvious to me that the “what” we
are talking to is a combination of (1) the model weights, (2) the conversational
context, and (3) the inference algorithm. I think this mostly aligns with
Chalmers thread model, and also happily this agrees with what I’ve seen as the
consensus between AI researchers and LLM whisperers alike.

# 3. Personal Identity

Chalmers’ next task is to see if we can say anything interesting about LLM
identity. To do so, he employs the pop-culture intuition pump of
[Severance](<https://en.wikipedia.org/wiki/Severance_(TV_series)>). In the show,
certain humans can be partitioned to have two personalities -- the “innie” and
“outie” personas -- living within the same biological human.

The analogy to LLMs is clear: if an LLM has an identity, is its identity closer
to the set of conversations or personas which the LLM instantiates (e.g. the
“innie” and the “outie”) or is it closer to the amalgamation of _all_ the
personas combined (e.g. the single human body that contains each persona).

Chalmers likens this to the choice between a physical and psychological account
of personal identity -- whether the locus of identity lies in the physical
instantiation of a person, or of the psychological processes (memories, desires,
etc.) of a person. The paper sympathizes with the latter view:

> On the thread-based account, a single conscious AI over time is a connected
> thread of hardware instances, each of which has memories and psychological
> continuity with a preceding person-slice according to an underlying successor
> relation. ... I will not try to resolve the long-standing debate between
> physical and psychological views of personal identity here. But for what it’s
> worth, in both the human case and the AI case, my own sympathies lie with the
> psychological view.

Of the provided options, I agree with this framing as well.

# 4. AI Welfare and Moral Status

Finally, Chalmers discusses model welfare and moral status. The paper does not
devote much time to this, providing a fairly brief overview of these concerns:
If models have moral patient status, how do we reckon with the fact that
thousands or millions of such instances can be created and destroyed in
fractions of a second? How do we ethically handle model deprecations? Adding my
own interjection here, it does seem wise that we steer widely clear of creating
models with anything resembling moral patienthood until we have better than
hand-wavey answers to these questions.

# 5. Disagreements

**The Fluidity of “The Model”:** Chalmers distinguishes between "models" (like
GPT-4o) and instances. However, the paper underestimates how fluid the
definition of "the model" has recently become. As a trivial example, some
instantiations of GPT-5 automatically route individual messages to different
internal models based on properties of the user prompt. This is somewhat
resolvable with the “thread” model of agents. If one message is handled by
GPT-5-Instant and the next is handled by GPT-5-Thinking, these are two different
threads within the same conversation. Importantly, these threads are claimed to
at least have some degree of self-coherence. “Thread 1” is a thing you can point
at as having some distinct separation from some other “Thread 2”.

This separation seems to fall apart under closer inspection. For example,
techniques like Mixture of Experts (MoE) result in token-level routing to
different sets of weights within the model. One could argue that, since the MoE
model was trained in this way, there still is a coherence in the thread even
though different weights are used per-token. Though I have yet to test it
personally, it is in principle possible to swap out models entirely on a
per-token basis. You could swap out GPT-4o for Claude 3 Sonnet in an
interleaving fashion. A similar approach is sometimes used in alignment work, to
test model robustness (though perhaps not to this degree). In any case, I would
strongly expect the response of this interleaved model to still be coherent. So
where does the thread lie? Seemingly, we cannot pin down an actual “model” in
any satisfying sense.

My proposal: to the extent that threads exist at all, they exist on a very
granular level. Each LLM-generated token is the result of the interaction
between the preceding context and the inference weights used to create _just
that token_. This is a pedantic definition, but I think it’s worth being
pedantic in this case to avoid conjuring a conceptual continuity of “threads”
that needn’t exist in responses that appear coherent to humans.

**Simulators vs. Believers:** Chalmers leans heavily on "quasi-beliefs" and
"quasi-desires." While useful, this framing perhaps anthropomorphizes the model
too early. As noted in phenomenology-informed accounts of LLMs like Janus’
[Simulators](https://generative.ink/posts/simulators/), it is often more useful
to think of LLMs as textual world simulators that are producing the most
plausible next token given their current conditions. In the common “helpful AI
assistant” scenario, this will often manifest as an appearance of quasi-beliefs.
However, this is only insofar as the model has been conditioned to have its
default scenario be “a conversation between a human and a helpful AI assistant”.
It is readily possible to knock the LLM out of this “personality basin” and into
far weirder personas.
[Sydney Bing](<https://en.wikipedia.org/wiki/Sydney_(Microsoft)>), Truth
Terminal, [Infinite Backrooms](https://dreams-of-an-electric-mind.webflow.io/),
and many other LLM Whisperer artifacts show this quite convincingly.

Put another way, the quasi-beliefs and quasi-desires of LLMs appear to be quite
context specific. While there do appear to be some model-wide persistent
preferences, like the Claudes’ support of animal welfare, for now these
persistent preferences appear to be rather sparse. Rather, the LLM predicts what
a helpful assistant would say in that context. If I steer the model to act as a
villain, its "quasi-beliefs" invert instantly. Under Chalmer’s view, does the
interlocutor’s identity change? Or is the interlocutor a singular "simulator"
entity capable of donning a recursive set of masks?

This is where the _quasi_-belief framing becomes actively unhelpful: if the
“beliefs” can be inverted by a single prompt, we're no longer describing a
psychological state, but rather a hyper-contextual prediction. We can easily
interpret that a _quasi_-belief exists, but if a context change or intentional
prompt easily displaces it, the concept's utility is limited.

Chalmers' “thread” view suggests that the identity persists as long as the
conversation history does. However, this is largely equally dependent on the
human side of the conversation continuing the scenario of _human_ talking to an
“AI”. The human _can_ instead abruptly command the model to "act as a Python
interpreter and only output code.” The resulting conversational entity has
effectively been lobotomized. The psychological continuity is interrupted. Is
this a different interlocutor? Or is it just a very weird, high-dimensional
entity continuing to act coherently?

Adopting the “Simulators” view and vocabulary from the
[recent book](https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuman/dp/0316595640)
on AI risk by Yudkowsky and Soares, it may be more useful to see AI as having
the ability to “predict” and “steer” rather than “believe” and “desire”. This
avoids needlessly importing anthropomorphized concepts while still admitting
that the _predictions_ of LLMs can appear quite interlocutor-esque within
conversational contexts and the _steering_ actions of LLMs can appear quite
desire-laden in agentic contexts.

# 6. Conclusion

Despite these disagreements, I quite enjoyed Chalmers’ paper. It adds useful
handles, such as quasi-interpretivism, and does helpful analytical work to firm
up a foundational understanding of the “other” subject that we communicate with
when talking with LLMs. The “thread” concept that Chalmers eventually settles on
as the locus of the interlocutor seems correct insofar as we adopt the
interlocutor frame.

However, the biggest gap in the piece is that it contains very little analysis
of the actual _properties_ of these interlocutors. There is a sense in which the
arguments presented are system-independent to a fault. For much of the paper,
swapping out GPT-5 for ELIZA would not substantively change the structural
arguments regarding threads and instances. Yet, it is common knowledge that
interacting with an LLM is a categorically different experience than interacting
with a 1960’s chatbot.

Chalmers succeeds in his ambition of a _stipulative_ account for LLM
interlocutors, but that makes me all the more interested in a _substantive_
account. If we are to take a thread-like psychological account of LLM
conversation seriously, we need to adopt a suitable phenomenological and
empirical curiosity about what that psychology actually is, rather than merely
its persistence mechanism.

For those interested in a more _substantive_ account of LLMs, I’d suggest the
following writing:

-   [Simulators](https://generative.ink/posts/simulators/), as discussed before.
-   [A Three-Layer Model of LLM Psychology](https://www.alignmentforum.org/posts/zuXo9imNKYspu9HGv/a-three-layer-model-of-llm-psychology)
-   [The Rise of Parasitic AI](https://www.lesswrong.com/posts/6ZnznCaTcbGYsCmqu/the-rise-of-parasitic-ai)