As part of my doctorate program, I am regularly reading academic papers, watching presentations, and attempting to recreate prior research. Often, this boils down to experimenting with a new fuzzing framework, programming language, or navigating a codebase that I’ve never seen before. The cognitive friction involved in experimentation with new technology is where the real learning happens, and is essential to the process of skill mastery. However, advanced topics in combination with limited free time create an environment that can be challenging to deeply understand a technical concept. While Large Language Models (LLMs) might be able to whip out solutions for a technical task quickly, I firmly believe it’s important to deeply understand the technical architecture of what you’re building, and I think it’s more important than ever to do things “the old fashion way” first, and understand the craft of Information Security.

At DEF CON this year, LLMs were a recurring topic at both the talks and ad-hoc conversations with fellow DEF CON goers. Attendees new to the Information Security field were asking “how can I know the LLM output is correct, if I’m new to the fundamentals?” I think this question is great as it signals skepticism about the output of an LLM, and to not just blindly accept what an LLM produces. I’ll often use LLMs to build a custom “curriculum” around a paper or set of papers to help me fill in my gaps in the form of short form tutorials. This approach allows any academic paper, codebase or Capture The Flag problem to be turned into a “graduate lab” nearly instantaneously, which ultimately allows me to further develop and verify my understanding.

This blog post will show my approach and thoughts on leveraging LLMs for self-study.

Arch Cloud Labs & LLMs

There are many LLM providers you can choose from with a reasonable recurring subscription. However, the rate at which models are being released from a variety of vendors piques my curiosity to try out different models for experimentation. Additionally, some of the work I’m doing may trigger guardrails, as anything that may be deemed “offensive” in nature could trigger an alert, and disrupt a session. Because of this, I use a combination of locally hosted open-weight models on a Framework Desktop, OpenRouter, and OpenAI’s ChatGPT for research purposes. Let’s discuss the reason for all three:

OpenAI ChatGPT: Assessing Technical Understanding

At the time of this writing ChatGPT, is ~$20 USD a month for personal web-based-chat. The projects tab that the interface offers allows you to upload resources such as PDFs, pictures, etc… and include this data in your context. If I’m spending time on a particular area of fuzzing (Ex: directed fuzzing), I can upload a handful of papers into one of these “projects” and test my understanding of the content. While preparing for my research defense, I prompted ChatGPT to be an “adversarial advisor” and critically ask questions about the content to assess my understanding. The questions asked required multiple paragraph responses, and were derived from the ChatGPT projects with academic PDFs.

The important part for this methodology to be successful is that you still have to do the upfront work of reading the papers. The cognitive friction of reading deep technical content, taking notes, and questioning what gaps the researchers are solving, and their technical approach is absolutely essential. Simply uploading content, and being quizzed on it to grasp the “cliff notes” is insufficient. You’re not really going to get the benefits, and you’d just be cheating yourself. While LLMs can simply summarize a research paper for you, this is not the goal for true concept mastery. Don’t cheat yourself, do the work, read the paper.

The largest benefit I’ve personally had from this approach is identifying areas that I overlooked or didn’t immediately see as a major technical contribution from a paper. Sometimes I get focused on a particular component that I’m really interested in, and that may distract me from a large contribution or challenge the author(s) overcame. The approach I describe here with ChatGPT ensures I see the whole contribution and fight my tunnel vision.

Open Router: Learning New Codebases

OpenRouter is essentially a proxy to numerous Large Language Model providers via a single interface. You can deposit however much money you choose into your account, and make requests to a given LLM via your favorite coding agent (Ex: OpenCode). There are hundreds of models available, to include your typical headline frontier models from companies like Anthropic, OpenAI, Google, Moonshot, etc… Each of these providers have different models with different pricing, and guardrails. Understanding the guardrails when building tutorials around security research concepts that may trigger guardrails.

Recently, I used Kimi K3 with the LibAFL codebase to help build tutorials to enable my understanding of how to use the framework. LibAFL abstracts core components of the popular AFL/AFL++ frameworks into “building blocks” that can be used by researchers in bespoke ways. This allows researchers to avoid reinventing the wheel of common fuzzing framework architectures and focus on their particular hard problem. My challenge is that I’m new to Rust, the language LibAFL is written in, and the terminology used for each “building block” is slightly unfamiliar, and I’d like to quickly build a harness that I can iterate on to develop a deeper understanding.

This is where I find Coding Agents with OpenRouter + $OPEN_SOURCE_CODE_BASE creates a beneficial learning environment. Suddenly any large GitHub repo can be distilled into graduate-level labs to complete on your own time and develop a deeper understanding of how to use the framework. If you get stuck, you can provide the generated tutorial + your question to the LLM to further break down the problem until you’ve successfully understood the issue. This same approach can be done with Capture The Flag problems. Here’s a screenshot of the tutorial breakdown that Kimi K3 is preparing after asking a series of questions to gauge my familiarity with fuzzing in OpenCode.

The final product being a README.md in the root of the directory with various modules to complete.

drwxr-xr-x 1 dllcoolj dllcoolj   18 Jul 25 20:36 00-mental-model
drwxr-xr-x 1 dllcoolj dllcoolj   18 Jul 25 20:40 01-harnessing
drwxr-xr-x 1 dllcoolj dllcoolj   80 Jul 26 11:45 02-inprocess-libafl
drwxr-xr-x 1 dllcoolj dllcoolj   18 Jul 26 10:41 03-real-repos-build-integration
drwxr-xr-x 1 dllcoolj dllcoolj   80 Jul 26 11:45 04-forkserver
drwxr-xr-x 1 dllcoolj dllcoolj  104 Jul 26 11:45 05-supercharging
drwxr-xr-x 1 dllcoolj dllcoolj   18 Jul 26 11:02 06-operations
-rw-r--r-- 1 dllcoolj dllcoolj 3705 Jul 25 20:37 README.md
drwxr-xr-x 1 dllcoolj dllcoolj  220 Jul 26 10:40 tinycfg

If you’re reading this and wondering why Claude Code was not mentioned, I have used Claude, and liked it. However, the numerous restrictions I’ve experienced over the past year with usage limits, and guardrails has made me cancel my subscription. For coding tasks, if I use an LLM, I’m using open-weight models that I self host with a Doom Emacs plugin. This allows me to stay in the loop during development and use it as an assistant rather than just have it code everything for me.

Framework Desktop: Locally Hosted Open-Weight Models for Software Development

For side project software development I’ve been personally satisifed with the capabilities of open-weight models. Again, this is for home lab tasks where I’m building software that I am the user for so enterprise scale software isn’t my use case. I’m typically building small projects to experiment with new frameworks or something related to log aggregation or pkgfile parsing. These are not the craziest of complex tasks, and can easily be performed with today’s open-weight models. If smaller models struggled, I can easily provide docs in the form of man pages to my context window via Doom Emac’s gptel and its support for sending entire emacs buffers in a request.

Additionally, I see three key benefits to locally hosted models:

1. Data privacy. You’re not shipping off all of your questions to ultimately be trained on to improve the next generation of models. I particularly enjoy this for projects that I don’t think are quite ready for public viewing yet, and are very much a research project work in progress.

2. No recurring bill. However, this comes at a hefty upfront cost of compute.

3. Developing skills of LLM hosting, an unexpected bonus!

I purchased a Framework Desktop earlier this year to start learning the various LLM runtimes, and began a brief stint of model hosting for friends. With Tailscale, and a reasonable home internet connection, I made LLMs available to friends and tweaked deployment parameters based on their feedback to optimize for speed over context window depending on the task they were trying to complete. This experience allowed me to iterate effectively when using offline models building malformed SquashFS file systems as a apart of my DEF CON: Packet Hacking Village Talk, “There’s A Bug In My Boot: Finding Vulnerabilities in U-Boot”.

Conclusion & Thoughts On Avoiding Over Reliance

A common trend in this post is that you still have to do the work of “active learning”. It is up to you, the human, to ensure you’re actively learning when leveraging LLMs for self-study. It is trivial to have LLMs generate complex projects in minutes, but whether or not you actually understand these projects is a different story. Perhaps there are situations where it matters less, but if you’re learning something to truly understand how it works, it’s critical to not cheat yourself and do the actual work. LLMs aren’t going anywhere, so understanding how to apply them to various use cases will only become more important. It’s okay to not use them for everything, but to avoid experimenting with them to understand their capabilities I believe would be a mistake.