Carlos :)

WOVEN LEARNING

Building With LLMs

An engineer's field guide, for a CS student ready to stop using AI and start building with it.

Carlos, you volunteer for this on your own time and you already prompt, verify, and use a model as a coding partner, so none of that is here. This is the layer above: how LLMs get turned into software, retrieval, tools, agents, evals, local and fine-tuned models, and enough of the internals to reason about why any of it fails. Each section is a build, not a tip. The named papers and tools are the real ones, go read and clone them. There is a whole engineering discipline forming here, and most of your peers have not noticed yet.

The one frame everything hangs on. An LLM is a pure, stateless function: a sequence of tokens in, a probability distribution over the next token out. That is the entire machine. It has no memory between calls, the "conversation" is an illusion, every turn re-feeds the whole history. System prompt, retrieved documents, tool results, reasoning traces: all just tokens in the context window, the model's only workspace. Internalize this and the field collapses to one problem, what goes in the context, and how do you manage it. Prompting, RAG, agents, memory, they are all answers to that single question. Every section below is a different answer.

01 · Context engineering

"Prompt engineering" undersells it. You are compiling a context window, a scarce, ordered, expensive resource, and the compilation is the program. Position matters: models attend unevenly across long inputs, information buried in the middle gets lost while the head and tail dominate. Structure matters: delimiters, few-shot exemplars, an explicit scratchpad for intermediate work, and retrieved evidence all change behavior more than clever wording does. And context is not free, attention is quadratic in sequence length, so every token you add costs latency and money and eventually degrades quality.

Build. Take one real task. Hold the instruction fixed and vary only the context, order your few-shot examples three different ways, move a key fact from the top to the middle to the bottom, and measure how the output shifts. You will see position bias with your own eyes, and you will stop treating the prompt as prose.

Read: "Lost in the Middle" (Liu et al.) on position bias in long context.

02 · Make it a component: structured output + tool calling

A chat window is a demo. Software needs the model to return something you can trust in code. Two moves get you there. Structured output: constrain the model to emit valid JSON against a schema, not prose you regex out. Do not hope, enforce it, with a library like Instructor (Pydantic schemas) or Outlines (constrained decoding that makes invalid tokens literally impossible to sample). Tool / function calling: give the model a set of typed functions; it returns a call, your code executes it, you feed the result back. That loop, not the chat box, is how an LLM becomes a component in a real system.

from pydantic import BaseModel
class Ticket(BaseModel):
    severity: int        # 1-5
    component: str
    summary: str
# Instructor forces the model's output to validate as Ticket,
# retrying on schema failure. You get a typed object, not a string.

Build. Write a typed extraction pipeline: unstructured text in, a validated object out, with a schema and automatic retry on invalid output. Then bolt on one real tool call (a calculator, a search, a DB query) and let the model decide when to use it.

Pointed at what you already do: you lean on a model for math daily. The engineer's move is not "trust it less," it is architecture, never let the model do the arithmetic itself. Wire a real CAS (SymPy, or a solver) in as a tool and let the model orchestrate it. Same structured-output-and-tool-calling pattern, aimed at your own workflow, and suddenly the part that used to silently produce a wrong step is doing exact computation you can trust.

03 · RAG: dynamic context assembly

The model does not know your data and has a training cutoff. Retrieval-augmented generation fixes both: at query time, fetch the relevant chunks and put them in the context. The retrieval is just nearest-neighbor search in a learned embedding space, cosine similarity between the query vector and your chunk vectors, which you already understand. The easy version is a weekend. The good version is real engineering: chunking strategy dominates quality, embedding choice matters, a reranker on top of the initial retrieval is usually worth it, and hybrid search (lexical BM25 plus vector) beats either alone. Naive RAG demos beautifully and fails quietly in production, which is exactly why the next section exists.

Build. RAG over your own corpus, your lecture notes, or a codebase you know. Chunk, embed, store the vectors, retrieve top-k, stuff them in context, answer. Then break it on purpose: find the questions it gets wrong and trace whether the failure is retrieval (wrong chunks) or generation (right chunks, bad answer). That diagnosis is the whole skill.

Read: the original RAG paper (Lewis et al.).

04 · Agents: an LLM in a loop

An agent is not magic; it is a loop. The model reasons, picks a tool, your code runs it, the result goes back into context, repeat until done, the ReAct pattern (reason, act, observe). That is the whole idea, and you can write it in about fifty lines. Do that before you touch a framework; the frameworks (LangGraph, and others) hide the exact mechanics you most need to understand, and many are over-abstracted. Be honest about the state of the art: agents are brittle. Errors compound across steps, context bloats as the loop runs, and "planning" is still weak, a long-horizon agent that works reliably is a genuinely hard systems problem, not a prompt. Meanwhile the agentic coding tools (Cursor, Claude Code, Cline) already work well enough to be a daily driver, learn to steer one: give it context, review its diffs, kill it when it wanders.

while not done:
    thought, action = model(context)         # reason + choose a tool
    observation = run_tool(action)           # your code executes
    context += format(thought, action, observation)
    done = is_goal_met(observation)
# ~50 lines with real tools. Watch it succeed, then watch it spiral.

Build. Write your own agent loop from scratch with two or three tools (shell, file read/write, web fetch) and a stopping condition. Then instrument it: log every step and watch where and why it fails. You will learn more in an afternoon than any framework tutorial teaches.

05 · Evals: the line between tinkering and shipping

This is the section most people skip and the one that actually separates an engineer from a hobbyist. You changed a prompt or swapped a model, is the system better or worse? You cannot answer by reading a handful of outputs; the failures are rare, distributional, and easy to miss. So you build evals: a test set of inputs paired with expected properties, metrics that score outputs, and often a stronger model as an automated judge (LLM-as-judge) to grade at scale against a rubric. Now every change is a regression test instead of a vibe. Nothing else on this page compounds like this, once you have evals, you can iterate on evidence, and everyone still shipping on gut feel is guessing.

Build. Take one prompt you actually use. Write fifteen test cases with clear pass criteria. Write an LLM-as-judge that scores each output against a rubric. Run it, tweak the prompt, run it again, and watch a number move. That harness is worth more than any single clever prompt you will ever write.

06 · Own the stack: local, open, and fine-tuned models

Running a model on your own hardware changes how you think about all of it. Ollama or llama.cpp will run an open model (Llama, Qwen, Mistral, a DeepSeek distill) on your laptop; vLLM is what you reach for when you need throughput. The reason a 7B model fits at all is quantization, storing weights in 4 bits instead of 16 with surprisingly little quality loss (the GGUF formats you will see are exactly this). Play with it and you develop a real feel for the size-versus-quality-versus-speed-versus-cost frontier that no hosted API teaches.

ollama run qwen2.5:7b        # a capable model, offline, on your laptop
# then hit its local API from your own code, it's OpenAI-compatible

On fine-tuning, the honest version: you almost never need it, and reaching for it first is the classic beginner mistake. Prompting and RAG solve most problems more cheaply and flexibly. Fine-tuning (in practice LoRA / QLoRA, which train small low-rank adapters instead of all the weights, so it runs on one consumer GPU) earns its place for a fixed style, a narrow format, or latency, not for teaching the model new facts, that is RAG's job. Knowing when not to fine-tune is the senior signal.

Build. Run a local model, call it from your own code, and benchmark it against a hosted model on a task you care about, latency, cost, and quality side by side. That table is the whole tradeoff made concrete.

07 · Under the hood, for someone who can take it

You are a CS major; you can actually understand the machine, and it will make you better at every section above.

Build a transformer. Karpathy's nanoGPT and his "Let's build GPT" and "Let's reproduce GPT-2" videos take you from nothing to a working model you train yourself. A few hundred lines. Do this once and attention stops being a black box.
Read the DeepSeek-R1 paper. The model you use is the subject of one of the most interesting ML papers of the last year: reasoning behavior elicited largely through reinforcement learning (their GRPO algorithm), and the striking R1-Zero result where capable reasoning emerged from RL with essentially no supervised fine-tuning. It is genuinely frontier and it is your daily tool.
Understand why inference is the way it is. Attention is quadratic in sequence length, which is why long context is expensive. The KV cache is why generation is memory-bound rather than compute-bound. These are not trivia, they explain every practical limit you hit.
Know why it can't count the r's in strawberry. Byte-pair tokenization means the model never sees letters, only sub-word tokens. Half the "dumb" LLM failures are tokenization artifacts, not reasoning failures. Then read "Attention Is All You Need", it lands far harder after you have built the toy.

08 · The role, and where to go deeper

There is a job title forming around exactly this skill set: AI engineer, distinct from the ML researcher who trains models. The AI engineer builds reliable systems with models, retrieval, tools, agents, evals, inference, and it is early, in demand, and pays well. You do not need a PhD; you need to be able to build and evaluate the things on this page. That is a rare combination right now, and you are close to it.

Where to go, all real, all worth your time:

Chip Huyen, "AI Engineering", the closest thing to a textbook for this whole discipline.
Simon Willison's blog, the best running, practical field notes on building with LLMs.
DSPy (Stanford), programming LLMs instead of hand-tuning prompts, treats the prompt as a compiled, optimizable program. Very much your kind of thing.
Then pick one build above and take it further than the assignment. Shipping something real is the only credential that matters here.

The through-line

Everyone can use a chatbot. The gap, and the opportunity, is the ability to turn a stochastic, fallible model into a system that behaves: bounded by structure, grounded by retrieval, extended by tools, and above all measured by evals. That is engineering, and almost none of your classmates are doing it yet. Pick a build, or aim these at a project you already care about, and ship it. You are early.

Built by Woven Learning & Technology