rag is just an openbook test

Recently I’ve been learning about LLMs and retrieval augmented generation. The conclusion that I’ve come to is that RAG is just a fancy way of saying, “give the model a way to find and use the right external context in a manageable size when it needs it.”

Yes I’ve oversimplified it, but that’s what I do to make sense of things - and it’s allowed me to arrive at this analogy. If I get to take an open-note exam, I’m not going to be very well off if my notes are poorly organised - I won’t be able to find the notes for the answers if I can’t relate them to the questions. The critical part is organising your notes, because it’s not very much use having them if you can’t find what you need when you need it.

I think I’ve said this before, but I learn much more by doing rather than reading. Reading just feels too abstract, and I understand so much more through a process I can engage with.

I had been interrogating NotebookLM about the content - the 600 page agentic AI guide I mentioned a few weeks back - making sure I understood the concepts of doc loaders, chunking, indexing, embedding - all of the groundwork that makes RAG useful for LLMs.

NotebookLM was struggling though. I’d given it a 600 page pdf, and it was getting lost in my questions, confused by the table of contents. At one point while discussing MCP Servers, it answered my question in Spanish…

Now, the well-actually crowd will rightly point out that NotebookLM is already doing RAG under the hood - that’s literally the product. Which is exactly why I wanted to build my own: I had no idea what it was actually doing when it failed, and I wanted to open the black box that had just answered me in Spanish.

Then I had either a very stupid or very brilliant idea. I was experiencing one of the reasons RAG is so beneficial in real time - the context window couldn’t handle the pdf and my questions, sooooo what better way to learn about RAG than to implement my own RAG pipeline specifically to help me learn? I had this beautiful, massive, structured pdf to use as a guinea pig, but despite all my reading still had only a vague idea of what needed to be done. So I did what I have been doing more and more these days, I asked Claude to make me a lesson plan and project where I could systematically work through the process step by step, evaluate the different chunking strategies, and end up with a beautifully indexed vector database I could query specific chapters, subsections, or concepts I was reading about.

I’m about two thirds of the way through at the moment. I’ve written a python script to extract the pdf text, stripped out the irrelevant text (i.e. headers, footers, etc), and tested my first chunking strategy (just basic character limit with a small overlap). I now have a simple CLI that I can use to retrieve relevant chunks of data - basically, I can find my notes quickly. Next steps, hook it up to the model, evaluate the quality, and experiment with different chunking & retrieval strategies (basically see how organising my notes differently affects finding answers).

Could I have asked Claude or ChatGPT to do this for me? Definitely.

But this was more fun, and now I feel like I at least understand how the engine works.

I started out trying to use AI to help me learn about AI and ended up using AI to create a lesson plan for me so I could make a pipeline to help AI more effectively help me learn about AI…

The irony is not lost on me.