A basic primer on how AI works, why it doesn’t learn, and why what you load into the context really matters.
Introduction
This blog post aims to provide a very simplistic introduction to the concepts of AI. It’s going to do this through a worked example for finding the answer to “Life, The Universe, and Everything”. In this example, we are just going to conduct our own “Deep Thought” experiment to narrow down all possible answers to just a single one. We will then look at how incorrect answers and hallucinations can arise, before applying the basic concepts we have learnt to some real-world AI applications.
The Trained Model
So, in a real AI model like GPT 4.5 or Claude Opus we have a set of data that the model has been trained on. For a Large Language Model (LLM), this is a huge amount of text, literature, source code, and so forth. For our simple example, our model has just been trained to know the numbers 0 through to 99. It also knows some basic relationships, so that any number knows the number before it and the number after it. This means our model can sort the numbers into order and handle concepts like less than or more than.
This is the limit of our model’s training and learning. Once the model has been trained, it can never learn anything new from this point onwards. This is a really common misconception about how AI currently works.
So, our starting point is a model where every number appears in order and is weighted so that each number has exactly the same probability of being picked. This is shown in the first view below:
If I ask this model “What’s the answer to Life, The Universe, and Everything?” then it will randomly select any one of the 100 possible answers as they all have the same weight. It will therefore be wrong 99% of the time.
Adding Context
To help us towards a better answer, we need to add some contextual information to our model. This context will change the weighting of the individual numbers in response to the information that we give it. A quick clarification on terminology: by a number’s weight I mean the probability the model currently assigns to it as the answer — not the fixed values the model learnt during training. Context shifts these probabilities; it never alters the trained model itself.
In the first case, let’s add to the context an observation that the universe likes symmetry and is therefore likely to prefer even numbers to odd ones. This causes the weight of the even numbers to increase, while the weight of the odd numbers goes down:
So, now if I ask the model the same question as before, it will lean towards returning an even number. This is a big improvement: we’ve eliminated all 50 odd numbers, so the answer is now (almost certainly) even. But that still leaves 50 candidates in play, so it’s nowhere near a good answer. We need to add more information to the context. Let’s add another observation that the universe doesn’t like outliers and would prefer to always select things that are in the middle of the range. Adding this to the context and the weights change to the following:
Now we are getting somewhere: we’re down to just 17 numbers that carry any real weight, and a large group of numbers are coloured red and very clearly excluded. We can improve this further by adding the next layer of observation that the universe has lots of pairs of things (dark/light, on/off, in/out) and is therefore a fan of the number 2. So let’s add to the context that any numbers containing the digit 2 should have a higher weight:
Nice, we’ve now converged on just three candidates — 42, 52, and 62 — which are equally likely given the context so far. This might be good enough if we were just asking for an approximate answer, but we’re looking for some precision. The next observation we can add is that the universe doesn’t like to brag too much and therefore would likely select a lower value number than a higher one. Adding this to the context and adjusting the weights accordingly we now get the following result:
Yay, a single number has risen clearly to the top: 42 is now the model’s most likely answer by a wide margin. A quick note on the colours: each cell is shaded relative to the current favourite, so the leading number is always the greenest — that’s why 42 glows here even though, in absolute terms, the model only assigns it around a 9% probability. Hold on to that number — the model’s “confident” answer is still only its most likely one, not a certainty.
Observations on Learning
The first observation is that no permanent learning has actually taken place. All we’ve done is provide successive pieces of contextual information that adjust the weights of the numbers in the model. Once there’s enough context, the correct result pops out. There’s a subtlety here worth calling out: researchers actually do have a name for what we just did — in-context learning — because the model genuinely adapts its behaviour from what’s in the prompt. But it’s learning only in a fleeting sense. The trained model is never updated, and nothing is retained: clear the context and the model is right back where it started, remembering none of it. The model doesn’t understand why it reached the value of 42 or anything about the meaning or significance of this particular number. And notice that 42 never became a certainty — at its best it was only the most likely answer (around 9%). The model always hands back a probability distribution over possible answers, never a fact; we’re just shifting weight around until one answer rises above the rest.
This matters for what comes next. The model has no way of telling a good answer from a bad one — it follows exactly the same weight-shifting process whichever way the context points it. So the same machinery that landed us on 42 can, with different context, land us just as happily on something wrong, and the model can’t tell the difference. Only we can, by checking the answer against the real world.
While this is a basic example, it clearly demonstrates core principles of how an LLM works. Imagine an enormous grid of every word and symbol that could possibly appear in a sentence or a piece of source code. Then, the contextual detail we provide to the model by way of instructions and prompts changes the weights of the words and symbols in the grid. To give an answer, the model favours one of the most likely words or symbols, appends that choice to the context, re-runs over the updated context, and then repeats the process over and over until the answer is generated. It’s worth being precise about what does and doesn’t change here. A real model does have “weights” in another sense — the billions of values fixed during training — but those stay frozen the whole time; the context never touches them. What the context shifts is the probability the model assigns to each possible next word. So the model itself never changes — it’s the context that grows with each step, and with it the odds of one word over another.
Context Bloat and Contradictions
So, in our example we have a context that takes the trained weights of the model and calculates new probabilities, giving us what appears to be the correct answer. But say we add some additional, contradictory information into the context. Suppose someone comes along and states our reasoning is poor and that in reality the universe would favour things that are special and unique, such as prime numbers. If we add this single contradictory observation into our context, then the model weights change again:
Immediately a different value pops out of the model, because 2 meets lots of the other context criteria but is also considered a prime number. This one small addition to the context means we are now back to getting not only an incorrect result, but a confidently incorrect one. Finally, let’s bloat the context once more with a related observation that primes are not that special, but certain primes are super special because their palindrome is also a prime. The numbers 37 and 73 are good examples of this concept (see The Big Bang Theory: Season 4, Episode 10). Pop this into our context and suddenly the weights look like:
Not only has the wrong answer changed to the number 37, but the model is now more confident in it than it ever was in the right answer: 37 sits at around 12%, a higher peak than 42 ever reached (about 9%), and almost no other answer (apart from 2) is even close. That’s the unsettling heart of a hallucination — how confident the model sounds is a poor guide to whether it’s actually right. The probabilities aren’t completely meaningless — they do carry some signal — but it’s a weak one and easily skewed, exactly as we’ve just done, so confidence is never something you can lean on to tell truth from fiction. To the model, 37 looks like the best answer for exactly the same reasons 42 did a few steps earlier.
It’s also worth noticing what didn’t happen. The contradictory context didn’t erase our earlier reasoning — 42 is still sitting there at around 4%. The bad context didn’t delete the good context; it simply outvoted it. And it only managed to do that because we let those last two observations carry more weight than the others; a forcefully or repeatedly asserted claim can override a whole pile of careful, correct context.
This is one important route to a hallucination — but it isn’t the only one, and it’s worth not over-claiming. What we’ve shown here is context-induced error: bad information we fed in outvoting the good. But hallucinations also spring from something more fundamental, with no contradictory context required at all — because the model is only ever predicting a plausible next word rather than checking it against reality, it can be confidently wrong even on a perfectly clean context, simply because the false answer was the one that looked statistically likely. Contradictory context, like we’ve just seen, only makes that failure easier and more dramatic.
Observations on Context
The main takeaway from this experiment is that what goes into the context really matters.
Put too little relevant information in: the model has to pick from a number of possible answers, so it will either be wrong outright, or make false assumptions that it carries forward into further iterations of answer generation. The result may look plausible but will be riddled with deviations or mistakes.
Put too much, or contradictory, information in, and the weights get skewed and the model starts confidently outputting false results. This is why managing context is so important in tools like Claude Code: as the context grows — especially when it fills up with irrelevant or contradictory material — the weights get harder to keep balanced, and it’s often best to clear the context and repopulate it with just the essentials again.
Where this becomes critical is in creating context content for use in a particular project or task. You need to give it just the right amount of information to prime the weights of all the things that are important, but not so much that the weights get biased, spread out, or skewed too much.
A prime example of this would be when seeding the context of a software engineering project. It’s far better to give a small and precise set of context information that’s been carefully crafted and refined than to feed in large amounts of content generated from a bunch of other related projects. It’s also good practice to use only one language, set of technologies, and set of libraries in each project, so the context will weight itself to generate the best results in that specific language and toolset combination. It’s also important not to feed the context contradictory information, such as from two related projects that use different architectures or libraries, as these just skew the weights and will result in lower-quality outputs.
It’s for these reasons that I’m cautious about ’enterprise’ AI tools that promise to pool context across all your projects. The win or loss comes down to relevance: pulling in a precisely targeted snippet from another project can help, but indiscriminately blending projects that use different architectures, languages, or libraries just feeds the model contradictory context and skews the weights toward lower-quality output. Unless that sharing is curated and consistent, you’re often better off giving each project a cleanly defined, tuned context of its own.
Summary
We’ve taken a quick peek into a simplified example of how generative AI works. We’ve noted that current AI models never permanently learn anything new after they have been trained. All we are doing with our prompts and context information is shifting the probabilities the model assigns to each possible answer, to force the more correct and accurate ones to the top. The trained model itself stays fixed throughout. We’ve also seen the importance of getting the context right and not over- or under-populating it, and avoiding contradictory information.
comments powered by Disqus