🧠 Transformers & LLM Architecture — Topic Resources¶

Curated list of top landmark papers, open-source repositories, video series, free courses, and code references for mastering Neural Networks, Transformers, Attention Mechanisms, and Large Language Model Architectures.

📄 Landmark Papers & Essential Reading¶

Paper / Reference	Key Takeaways & Focus	Link
Attention Is All You Need (Vaswani et al., 2017)	Introduced the Transformer architecture relying entirely on self-attention mechanisms without RNNs/CNNs.	ArXiv Link
BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018)	Showed bidirectional masked language modeling for downstream NLP transfer learning.	ArXiv Link
Language Models are Few-Shot Learners (GPT-3) (Brown et al., 2020)	Demonstrated scaling laws, zero-shot, and few-shot in-context learning capabilities.	ArXiv Link
RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)	Introduced RoPE, the dominant positional encoding used in LLaMA, Mistral, and Claude.	ArXiv Link
GQA: Training Generalized Multi-Query Transformer Models (Ainslie et al., 2023)	Grouped-Query Attention reducing KV cache memory bandwidth bottlenecks during LLM inference.	ArXiv Link
Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Gu et al., 2023)	Introduced selective SSM architecture achieving $O(N)$ linear sequence length scaling without attention.	ArXiv Link
FlashAttention: Fast and Memory-Efficient Exact Attention (Dao et al., 2022)	IO-aware exact attention algorithm reducing memory complexity from $O(N^2)$ to $O(N)$ speedups.	ArXiv Link
FlashAttention-3: Fast Attention with Asynchronous Execution (Shah et al., 2024)	Exploits FP8 Tensor Cores and asynchronous GPU warp execution on Hopper architecture for 2-3x speedup.	ArXiv Link

💻 Top Open-Source Repositories & Implementations¶

Repository	Description	Link
nanoGPT	Andrej Karpathy's cleanest, fastest repository for training/finetuning medium-sized GPTs in PyTorch.	GitHub Repo
micrograd	Tiny autograd engine implementing backpropagation over a dynamically built DAG with PyTorch-like API.	GitHub Repo
transformers (Hugging Face)	State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX with pretrained weights.	GitHub Repo
Triton	Python-based programming language and compiler for writing custom GPU kernels (FlashAttention, KV cache).	GitHub Repo
tiktoken	Fast BPE tokeniser for use with OpenAI's models in Rust and Python.	GitHub Repo

🎥 Must-Watch YouTube Series & Videos¶

Video / Playlist	Creator	Description	Link
Neural Networks: Zero to Hero	Andrej Karpathy	7-part video series building micrograd, makemore, WaveNet, GPT-2 from scratch in PyTorch.	YouTube Playlist
Deep Learning Series	3Blue1Brown	Unrivaled visual intuition for gradient descent, backpropagation, and self-attention.	YouTube Playlist
Coding Transformer / Attention Papers	Umar Jamil	Line-by-line PyTorch implementation of Attention Is All You Need and LLaMA architecture.	YouTube Channel
LLM Visualization & Intuition	Jay Alammar	Visual blog posts & videos explaining Transformer matrices, embeddings, and attention.	Blog / Site

🎓 Free Courses & Open Curricula¶

Course Title	Institution / Host	Focus	Link
CS224N: Natural Language Processing with Deep Learning	Stanford University	Word vectors, sequence models, Transformers, pre-training, and scaling laws.	Stanford Course Site
CS336: Language Modeling from Scratch	Stanford University	Deep technical lectures on data curation, distributed training, hardware utilization, & inference.	Stanford CS336 Site
Hugging Face NLP Course	Hugging Face	Free practical course on Transformers, Tokenization, Fine-Tuning, and Datasets.	Hugging Face Course

⚙️ Code References & Hands-on Notebooks¶

Build a Large Language Model From Scratch (Repo) — Sebastian Raschka's step-by-step code matching his Manning book.
Illustrated Transformer Notebooks — PyTorch execution matching visual transformer walkthroughs.
OpenAI Cookbook — Embeddings & Tokenization — Practical token counting, embedding generation, and vector similarity calculations.