The basic function of a language model is to predict with statistics the next most likely word in a sentence, for example given the sentence “My favorite color is ___” the language model should predict blue more often than the word car. They are basically very fancy versions of the auto-complete feature you can find on your phone when you start typing a message. If you are wondering where Anthropic Claude got its name… Claude Shannon used sophisticated statistics to decipher enemies’ messages during the Second World War.
This also causes the main problem with current AI in that it doesn’t tell the truth, and has no understanding of the truth, it simply predicts the next most likely word that you want to hear. So it might say that it uploaded and read the files, without actually reading any files… or it might make up a reference because it thinks you want to see a reference.
This is also the basic secret behind prompt engineering, all you are doing is providing better context to help the AI auto-complete the sentences better, and using techniques like RAG to augment the context can help make the answers or conversation more relevant.
The basic units of language models are tokens, which are on average 3/4 the length of the word… and I believe that’s so it’s easier to work with suffixes like -ly, -ize, -ing. Therefore “build” would be one token and “build+ing” would be two tokens.
The author discusses a critical point when building AI applications, in that it’s important to consider their defensibility. The low barrier to entry is both a blessing and a curse, so if it’s easy for you to build an AI feature, it’s also easy for your competitors, or even customers to build it. In addition, you need to consider how fast AI is progressing, and what might not be easy to copy now, could be easy to copy in the future.
The most dominant architecture for models currently is the transformer model, based on the attention mechanism (Vaswani et al., 2017). The basic premise is that the model can weigh the importance of different input tokens while generating output tokens… one way we could visualise this is maybe highlighting all the important words in a book first and then writing an answer, rather than reading the book from back to front and then writing an answer while trying to remember everything at once.
This same attention mechanism is the reason why we have context limits when working with AI models, because every token or word needs to be considered when highlighting relevant words before answering, you only know which words to highlight after reading the book (and the question) first and comparing each word to every other word… that’s a very simplified version of how I understand the maths described.
The transformer architecture has been around the longest, and might be difficult to replace because transformers have huge amounts of investment and scale already, so it would be difficult for new architectures to gain similar momentum.
Another thing the author touches on is that the data the language models are trained on is often filled with fake news or gossip websites, and considering that more content on the internet is being written by AI, we could reach an inflection point where we no longer have access to high quality data to train the newer models. The author even suggests that if you have ever put anything on the internet, you can assume it’s already been trained on.
A concern however, is that you can post content on the internet to influence AI models, such as a recent news article about a fake think tank created by Israel to influence chatbots. 1 If anything on the internet will be consumed by AI then you can seed ideas into new models… and currently there is no way for models to “forget” things they have learnt.
AI models are also interesting because their answers can change if you ask the same question multiple times. This follows along the same lines as the auto-complete mentioned earlier, but if you ask a language model what pizza topping is the best, it would answer based on probability, so that 70% of the time it might say Basil, but 20% of the time it might say Garlic, and 10% of the time it might say Pineapple. This introduces some variation in the responses but can also make it difficult if you train the model to give you a truthful answer to a specific question… 90% of the time it might say the truth and 10% of the time it might say something incorrect. Ensuring the correctness of a model is difficult when it answers correctly 90% of the time, or each time you test it, but sometimes answers incorrectly in production. This is what the term hallucination seems to refer to.
The example the author gives for hallucinations is that if someone seeded the training data with an article online that said all US presidents are aliens… when verifying the model it might never say this, but because this information is in the training data, the model might say that the presidents are aliens every so often. This makes the thought of fake think tanks, or AI generating content being used in training even more concerning.
In Software Engineering we try to build systems that are always truthful, reliable and consistent… so adding an LLM to the mix has always seemed like a bad idea to me. You wouldn’t want software controlling a self-driving car or accounting software to only tell the truth sometimes and hallucinate answers. The rest of the book covers the topics of AI Engineering and building solid evaluation pipelines to detect failures and unexpected responses… so let’s see if it provides any useful guidance to change my mind.
The author lists some challenges of evaluation:
- Evaluating models with PhD intelligence is more difficult, few people could do that correctly.
- Evaluating sophisticated tasks is much more time consuming.
- Open-ended responses are difficult to evaluate.
- Foundation models are black-boxes, and we don’t understand how they work and/or the knowledge is hidden from us by AI companies.
One solution to make evaluation easier is to use AI to evaluate itself, but AI judges also provide probabilistic results. It also comes with an additional monetary and latency cost in training to run the AI judge to evaluate every response, unless the AI judge is a weaker model. When training models on such a large scale, I assume it would also be difficult to validate the AI judge is doing a good job across billions of checks, and might introduce further bias.
The author suggests evaluation driven development is the answer, however you can’t use this approach to build potential “game changing” new ideas…
Some traits you might want to evaluate are:
- Factuality: is it creative or does it hallucinate answers
- Safety: can the generated outputs cause harm to users and society
- Controversiality: is it politically correct?
- Friendliness
- Instruction-Following capability
Sidenote: The rumor is that although Google invented the transformer model, they didn’t push the technology further because they were worried about political incorrectness, which was a big concern for companies before “woke” became a thing and the internet was flooded with false news.
Earlier in the book the author also mentioned that training models with alignment actually made them worse, and more biased, so ensuring safe responses must be a large challenge… especially considering LLMs must have been trained with Reddit comments or other online comments on the internet, where everyone has a tendency to be hurtful to others.
The risk associated with AI models seems large, and I’m not convinced yet it’s worth it… time will tell better. It’s good to remind oneself that reputation takes 50 years to build and 5 minutes to destroy, but many businesses are rapidly implementing AI across their entire organisations.
The next chapter, Prompt Engineering raises more concerns, because you train and evaluate your model to ensure it’s safe, and the user’s goal is to use “prompt engineering” to exploit the model to get the results they desire. In UI/UX you would delicately craft the affordances the user is allowed within the system, and AI opens the box to allow open-ended interaction, even encouraging prompt engineering.
Some other concerns the author raises related to Prompt Engineering are information extraction, where the language models, which have been trained on large amounts of data, or fine-tuned with sensitive corporate data, can be exploited to extract that data. Information extraction can lead to data theft, privacy violations, and copyright infringement.
Working with LLMs and defending against exploits and security issues would be a constant catch up game, like the red queen hypothesis states.
The next chapter covers RAG and vector search for enhancing context, which can also be done with a term embedded search engine like elasticsearch. The author also discusses how complicated chunking can be, in which you split the documents into smaller “chunks” to make them easier to embed and retrieve with the search engine… and from personal experience it’s definitely a fine balance. If the chunks are too small you might miss relevant content in pieces above and below the text in longer documents, while larger chunks can make the vector embeddings less accurate. Creating lots of smaller chunks also has an effect on the overall cost, as it can quickly increase the size of the vector database and number of embeddings needed.
Another concern with using vector databases is that on each search request, you first need to create a vector embedding of the user’s input, so that you can find nearby or relevant content in the vector database, which adds a small cost to each message in the conversation with an AI agent.
On agents:
The possibilities seem endless, and the potential economic value of these agents is enormous.
the prospect of giving AI the ability to automatically alter our lives is frightening. Just as you shouldn’t give an intern the authority to delete your production database, you shouldn’t allow an unreliable AI to initiate bank transfers. Trust in the system’s capabilities and its security measures is crucial. You need to ensure that the system is protected from bad actors who might try to manipulate it into performing harmful actions.
Everyone seems willing to take that risk, as long as it makes money…
I’m still not convinced working with LLMs is a great idea.