Huy's Wiki

Deep Dive into LLMs like ChatGPT

Properties1
tagsllm, karpathy, deep-learning, transformer, ai-education

2 min read

Deep Dive into LLMs like ChatGPT

Speaker: Andrej Karpathy
Video: YouTube

Overview

Andrej Karpathy’s comprehensive deep dive into how Large Language Models (like ChatGPT) work under the hood. This ~3.5 hour lecture covers the full stack of modern LLMs — from the Transformer architecture up through training, fine-tuning, and inference.

Originally delivered as part of a Stanford course / public lecture series.

Key Topics Covered

Transformer Architecture

  • Self-attention mechanism and how it enables parallel computation over sequences
  • Multi-head attention, positional encodings
  • Feed-forward layers, layer normalization, residual connections
  • The decoder-only architecture (GPT style) vs encoder-decoder (original Transformer)

Tokenization

  • How text is split into tokens (BPE / Byte-Pair Encoding)
  • Vocabulary sizes and their trade-offs
  • Why tokenization matters for model behavior (e.g., handling of whitespace, non-English languages)

Training Pipeline

  • Pretraining: Next-token prediction on internet-scale data
  • Supervised Fine-Tuning (SFT): Teaching the model to follow instructions
  • RLHF (Reinforcement Learning from Human Feedback): Aligning model outputs with human preferences

Inference

  • Autoregressive generation (one token at a time)
  • Temperature, top-k, top-p sampling
  • KV-cache for efficient generation

Model Capabilities & Limitations

  • Emergent abilities at scale
  • Hallucination, reasoning gaps
  • Context window constraints

Why This Matters

Karpathy’s lecture is the definitive primer for anyone wanting to understand LLMs from first principles. He builds up intuition for how these models actually work — from the math of attention to the practical realities of training runs costing millions of dollars.

Graph View