What You Don't Know

The Epistemic Demons in the Corner of your Eye

In the old days, students who cheated knew they were cheating. It’s a pretty difficult feat of mental gymnastics to convince yourself that you learned the content you were supposed to learn by copying a solutions manual or asking a friend to write your essay for you. Of course, some still managed to deceive themselves. And those talented mental gymnasts were usually suitably disappointed come exam season. 

More recently, as LLMs are now the cheating method of choice, I notice students losing their grip on the reality of their own learning. To be clear, this is a separate problem from the fact that students who use AI are hindering their own learning. There are plenty of articles being written about the perils of AI usage in higher education. I have no interest in rehashing the fact that it is bad that students are now probably cheating more and learning less. I instead wish to explore how this reliance on AI changes meta-level perception of our own knowledge. It is one thing to not know something. It is another entirely to not know that you don’t know it. 

A good friend of mine was working on her masters as LLMs were coming into their own as a learning tool. In a stats class she’d have otherwise struggled in, she tried them out. Being a motivated student who wanted to understand the material, she did not merely request answers. She was no classical cheater. Instead, when she got stuck on a problem, she’d ask for help reframing it, asking it to walk her through the problem as if it were her tutor. She studied its explanations. She believed she was learning the content. 

She was utterly shocked when she did poorly on her exams. 

This is the aspect of LLM usage that scares me the most. A smart and motivated learner will often believe that they are learning a subject at a reasonable level of depth when they, in fact, are not. My friend followed each step in the model’s explanation, but failed to learn the deeper understanding that dictated why each step ought to have been chosen. Without even noticing, she’d learned a surface level summary in place of the real understanding the assignments were intended to produce. 

And students are lucky on this front: they have the reality check of an exam grade. Beyond university walls, it is orders of magnitude easier to convince yourself that you understand something deeply which you do not actually grasp. And it becomes easier to convince others, too. 

Pre-LLM, smart curious people would usually learn through articles, books, or lectures. If an article was too complicated for someone to understand, they generally noticed that they were missing something. And, if they didn’t, their attempts at verbally reconstructing the content of the article were usually lacking, allowing for any real experts they spoke with to easily gauge their level of understanding and help correct their shortcomings. 

In contrast, LLMs are intentionally trained not to provide text which users will rate as jargon-heavy or confusing. LLM-generated text is always clear and comprehensible. When asked to aid in learning, they generally default to writing like a journalist attempting to distill complex ideas for a lay audience. They are incredibly skilled at crafting high level overviews which feel like complete self-contained narratives. 

A user may continue to prompt for more and more depth, but because each level of depth is presented as complete in itself, the user is unlikely to have an experience which is very familiar to all pre-LLM learners: trying (and often failing) to struggle through a hopelessly confusing expert-level explanation without sufficient context. 

That LLM generated content which is so perfectly appropriate for a user’s background is likely a more enjoyable and less frustrating way to learn. Hell, sidestepping the confusion by skipping the labyrinthine hunt for content may even speed up learning. But it also completely removes all reality checks. 

If you’re never shown how poorly your shallow understanding holds up against a rigorous reading list, you have no good gauge on how far your understanding is from an expert’s. Unless you are regularly experiencing confusion at expert-level content, you might completely fail to notice just how much you don’t know.

Likewise, if you train yourself on summaries, your interlocutors lose the valuable signal of how well you summarize content as a metric of your understanding. It becomes harder for others– even experts– to figure out what you don’t know and subsequently point out the gaps in your knowledge. 

I’ve had several conversations now where, far too late, I realize that the person I’m talking to does not actually understand what they’re saying. They may imagine that they do, but they do not. 

Ironically, this often happens on the subject of AI itself. I suppose that curious and motivated AI enthusiasts are consistent in what they wish to use AI to learn about. In a bewildering conversation recently, I was explaining a modification of GAN architecture to someone who used all the right terms, nodded along in all the right places, and displayed an ability to summarize classic concepts in the same way an expert would. He had no reason to intentionally broadcast overconfidence; his next task was to write up what I explained to him, so any attempted bullshitting was sure to be found out. Still, he reassured me he had a strong background in the field. He even spoke fluently enough to back up that claim. 

Then he started asking me questions. He asked whether the modifications were “post-training”– technically a valid question, but a strange one to ask when the modifications were about constructing the loss function necessary for any training whatsoever. I began to suspect he was importing some summary understanding of LLM training phases which were not perfectly applicable to this non-LLM context. He kept asking if modifications could be “parallelized,” a question for which I cannot offer a shred of clarifying context because there was none. It made about as much sense as asking if a smell could be parallelized (what on earth are we parallelizing, exactly?). 

I don’t think this man believed he was pulling one over on me. Nor do I think he was an idiot. I suspect he used LLMs to explain AI training processes in good faith, hoping to learn new content. He probably even asked copious follow-ups. I’m sure he inquired about terms he’d heard in passing to contextualize them. But I doubt he had the experience of real confusion at any point in his learning process. Everything an LLM produces is so incessantly comprehensible– it would never write an explanation that would leave him lost or confused. It follows, almost tautologically, that it never challenged him to see what he does not know.

And so this well-educated man employed to write about technology genuinely believed himself to have a strong enough background. He’d read snippets and summaries and asked follow-ups until he had a mental model which made sense to him. He learned to wield the right terms by dealing with abstracted flowcharts. He knew where to place the term “post-training” in a sentence for it to sound correct, which is tantalizingly close to actually understanding what it means. He knew he didn’t need a perfect understanding, he just wanted to get the gist of it. His goal was to write about tech, not to create it. I think he genuinely believed he understood what he needed to for that goal. 

Unfortunately, he did not. His questions revealed that he was dangerously overconfident. There would have been nothing wrong if he had the exact same level of knowledge and had simply asked me to explain GANs from the beginning. But he waved me off when I first offered, confident that he understood what he needed to understand, and then revealed massive misunderstandings. Misunderstandings which I almost failed to correct because of the way he presented himself. And, frankly, because it is much easier– both intellectually and socially– to answer an open-ended question than to flag all the implicit misunderstandings hidden in ill-formed discussion. And it seems we are barrelling towards a world in which many of those open-ended questions are being replaced with LLM-fueled almost-understandings which the interlocutor must attempt to identify and, hopefully, disentangle before even beginning the explanation they would have given in the first place. 

Nothing I’ve described is an entirely new problem. Overconfidence is a staple of human interaction, and people have successfully faked expertise since the dawn of society. In fact, bullshitting was much easier in the pre-internet days when casual fact-checking was a far more onerous task. 

But I believe it used to be a lot harder for genuinely curious and epistemically humble people to be fooled into imagining they understood something which they did not.

If your goal pre-LLM was to grasp just enough of a concept to fulfill a minimal job requirement, impress a boss, or look clever at parties, this problem might not bother you too much. You probably already didn’t care about depth of understanding and may be perfectly okay with very poor meta-knowledge about your own knowledge. Or else you were already a skilled enough mental gymnast that you did not need assistance from AI. I don’t mean to cast judgment on this: in plenty of careers, it really does not matter if your understanding goes incredibly deep. Whatever gets the job done gets the job done. 

But if your goal is truth, this should be very concerning to you. Ever-comprehensible summaries are wonderful tools for experts seeking information at the periphery of their field, or for lay people who want a cursory understanding of a great many domains. But anyone who learns a topic using LLMs should really take a moment to appreciate the uncanny valley of intelligibility that surrounds them.

There is a paper on the topic of color vision and visual processing from a few years ago that always comes to mind when I discuss the false sense of intellectual security brought about by LLMs. 

For a while, we’ve known that our eyes can only actually see color in a tiny fraction of our visual fields. Your eye has rods and cones. Cones are responsible for color vision, and rods only for distinguishing light from dark. However, rods and cones are not evenly distributed in the eye. Most of your eye is the domain of rods, with only a tiny portion (which we call the fovea) left for the cones. But the color vision from cones dominates our perceptual experience because the fovea is what is responsible for the very center of our visual field. When you focus your eyes on something, you are focusing your foveal vision on that thing. Everything else is peripheral vision. And, as Nietzsche famously said, there cannot be an eye turned in no particular direction. Color fills whatever direction your eye is turned to.

But even when you focus your eyes on a single word on this page, you cannot shake the feeling that the world is still rendered in color everywhere, not just at this word. Even though your eye completely lacks the cones necessary to see color in your periphery, you still feel that you see it. It’s intuitively very hard to believe that the color we think we see in our peripheral vision is a mere construction of the mind. 

With the advent of virtual reality, this became demonstrable. In a 2020 paper, researchers brought subjects into a VR world which was mostly black and white, except for foveal vision. Wherever the subject looked, color was rendered, but the rest remained colorless. When the subject looked away, the corner of the world they had been looking at would fade to grayscale again, and color would appear wherever they looked next. The researchers were able to reduce the range of color to  below 5% of the visual field with many subjects failing to notice a difference. 95% of the world could be rendered in black and white, and you’d be none the wiser as long as your eyes were fixed on the right spot. 

Your brain is incredibly good at filling in peripheral details. It turns out your peripheral vision actually kind of sucks. Not only is it void of color, it’s also much blurrier than you think it is. Similar research has established that you can also replace peripheral vision with a shockingly low-res blurred version without attracting attention. In short, the picture that you think you see of the world is impossibly complete. 

As I fixate on these words, I believe I see, in the corner of my eye, a vivid textbook lying on my desk. But I do not. For all I know, God could have replaced that textbook with a low-res gray blob. Without a true image to compare my peripheral perception to, I’d have no way of knowing if the object sitting there is actually the golden yellow book with high-resolution text that I think it is or if it’s merely God’s cruel gray blob. 

Likewise, when I aim to understand something new, I am often subconsciously constructing a picture of the whole concept which is vague and fuzzy anywhere I have not yet actively investigated. If I compare my understanding to that of an expert– someone who has taken a clear snapshot of the concept in all its glory and filled in every colorful detail– I see where my picture is lacking. I start to notice that explanations which had previously satisfied me were actually oversimplifications or incomplete.

But if I only learn through perfectly comprehensible LLM explanations, I will always be living in that uncanny colorless VR world. Everywhere I look, I will find precise detail. But the models are trained to stop me from noticing the ambiguity lurking in the corner of my eye. Anywhere I can think to prompt further, color will appear. But an LLM will not point out the ugly gray blobs, it will only reshape them into incessantly intelligible pseudo-textbooks right before I think to look at them.