LLMs from Scratch
Stars: 96.9K ★ | Forks: 14.8K Language: Jupyter Notebook | Author: Sebastian Raschka
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step.
Overview
The official companion repository for Sebastian Raschka’s book “Build a Large Language Model (From Scratch)” — a hands-on guide to building, training, and fine-tuning LLMs completely from scratch using PyTorch.
What’s Covered
- Attention mechanisms — from basic to multi-head attention
- GPT architecture — implement the full transformer decoder
- Tokenization — build your own tokenizer (BPE)
- Pretraining — train on large text corpora
- Fine-tuning — instruction tuning for chat/classification
- From scratch — no high-level LLM libraries, pure PyTorch
Why This Matters
Sebastian Raschka is a well-known ML educator and researcher. This repo walks through the full pipeline of building a ChatGPT-like model without relying on HuggingFace Transformers or similar abstractions — understanding every layer from embedding to generation.