Every time a language model answers an ambiguous question with unsettling precision, the same feeling arises: it seems like there's someone on the other side who understood. As an architect who has spent the last few years integrating these systems into production, that feeling is exactly what I want to take apart. Not because the answer is disappointing, but because the right question isn't "does it understand?", it's "what are we asking 'understanding' to mean?".

What actually happens under the hood

A large language model doesn't reason about the world, it calculates the probability distribution of the next token given the previous context. Every word it generates is, in essence, the answer to a statistical question: given everything that came before, what is most plausible to come next?

# Simplificación conceptual de lo que hace un LLM en cada paso
def generar_siguiente_token(contexto, modelo):
    distribucion = modelo.calcular_probabilidades(contexto)
    return muestrear(distribucion)  # no hay "creencia", hay probabilidad

There is no internal representation of the world that the model consults to verify whether something is true, there are weights trained on trillions of tokens that encode extraordinarily rich statistical correlations. The fluency we perceive as comprehension is, technically, the consequence of having compressed patterns of human language to a scale that no human could consciously process.

The argument against: the Chinese room, updated

Philosopher John Searle proposed in the 1980s the thought experiment of the "Chinese room": a person who doesn't speak Chinese follows a rule book to respond to messages in Chinese in a convincing way, without understanding a single word. The question he raised is still relevant: manipulating symbols according to rules —however sophisticated they may be— is that the same as understanding their meaning?

LLMs are, in a sense, the most elaborate version of that room ever built. The difference from 1980 is that the "rule book" now has trillions of parameters learned automatically instead of being written by hand, and the result is so fluent that the distinction between "following rules" and "understanding" stops being obvious from the outside.

The argument for: comprehension is also a physical process

The strongest objection to the Chinese room is that our own brain doesn't "understand" in some magical sense either: it's neurons firing according to learned patterns, without there being some inner homunculus that gives "real" meaning to anything. If we demand that an AI demonstrate comprehension by a criterion that our own brain couldn't pass under the same scrutiny, the standard isn't very honest.

Moreover, there is evidence that LLMs build internal representations that go beyond simple surface-level memorization: geometric relationships between concepts in the embedding space, generalization to problems outside the training distribution, the ability to follow multi-step chains of reasoning. That's not "understanding" in the human sense, but it's not simple keyboard autocomplete either.

The question that actually matters for an architect

After months debating this in philosophical forums, I arrived at a more pragmatic conclusion: for designing production systems, the question "does it truly understand?" is the least useful of all. The question that does matter is a different one:

  • Does the model generalize reliably outside the cases it saw during training?
  • Does it fail predictably or chaotically when it's wrong?
  • Can we verify its answers against an external source of truth before acting on them?

A system can be extraordinarily useful without our resolving whether it "understands" in the philosophical sense of the term, and it can be dangerous even though it seems to understand perfectly. Confusing fluency with reliability is, in my experience, the most expensive architectural mistake I've seen teams make when integrating AI without skepticism.

My position, with the uncertainty included

I don't believe LLMs "understand" the way a person does: they have no body, no continuity of experience between conversations, nothing resembling an intention of their own. But neither do I believe they're just "glorified autocomplete" — that phrase, repeated as if it settled the debate, ignores the real complexity of what happens in those trillions of parameters.

Intellectual honesty demands accepting that we still don't have a definition of "understanding" precise enough to rigorously apply it to a system that is neither human nor a calculator. Until that definition exists, the question will remain fascinating — and will remain, above all, the wrong question for deciding how to build software responsibly with these tools.