Language Is a Poor Heuristic for Intelligence

Great post by Karawynn Long on LLMs and language as a sign of intelligence over on their Nine Lives newsletter. Everyone should definitely read it, but here’s a few of my favourite quotes:

“Language skill indicates intelligence,” and its logical inverse, “lack of language skill indicates non-intelligence,” is a common heuristic with a long history. It is also a terrible one, inaccurate in a way that ruinously injures disabled people. Now, with recent advances in computing technology, we’re watching this heuristic fail in ways that will harm almost everyone.

For example, text-to-image generators like DALL-E and Midjourney utilize the same underlying LLM technology as chatbots, but the public release of these programs last year — controversial as they were for other reasons — did not spark the same kinds of zealous speculation about a mind in the machine. Humans don’t have a history of using “image generation” or even “visual art” as a heuristic for intelligence. Only fluency of language has that distinction.

Not coincidentally, you’ll often find that the people making the loudest argument for impending computer sentience are literally invested in OpenAI and Google and Microsoft and Meta. I am hardly the first person to point this out, but trillions of dollars are at stake, and these (already extremely wealthy) people reap direct financial benefits when the public mistakes the software’s fluency of language for intelligence and comprehension.

That tech corporations have successfully rebranded the idea of “artificial intelligence” among the media and public to include LLMs is just the tip of the spear. Whenever an LLM says something contrafactual, the companies claim — and the media dutifully reports — that the program is “hallucinating” … a word that suggests both a conscious experience and external senses, neither of which pertain to computer algorithms. Words like “fabricating” and “lying” and even “bullshitting” aren’t much better, because those terms all carry connotations of “intent to deceive”, and no LLM program can ever have intent, either malicious or benign.

I keep seeing screen-shotted human-LLM interactions in which the participating human (and subsequent observers) are so distracted by the “apology” that they fail to recognize that each answer is effectively a dice roll, statistically no more accurate than the previous one. In fact, if you respond to a factually correct answer with a challenge, ChatGPT will usually apologize and give you a different, incorrect answer. You’re basically just shaking the Magic 8 Ball over again until (if you haven’t gotten frustrated and quit first) it randomly serves up an answer you like, at which point you stop asking. But the computer hasn’t “learned” a damned thing from the interaction, any more than the cosmos is sending you a message about the future.

One clear reason is that corporations would prefer to use machines for a number of jobs that currently require actual humans who are knowledgeable, intelligent, and friendly, but who also have this annoying tendency to want to be paid enough money to support the maintenance of their inconvenient meat sacks. Not to mention the problematic fact that humans occasionally possess ethics, independent thinking, and objectives besides maximizing shareholder profit.

When we were dealing with pocket calculators, that was a valid and time-saving assumption. In the late twentieth century, there was absolutely no need to double-check whether your TI-30 was accurately reporting the square root of seventeen. But LLMs have now shattered the usefulness of the “computers are reliably accurate” heuristic as well, biting everyone from lawyers to university professors in the ass along the way.

Just a fantastic article.