Skip to content

🧠 Reasoning & Test-Time Compute — Topic Resources

Curated collection of landmark research papers, open-source repositories, YouTube videos, free masterclasses, and code references for Reasoning Models, Chain-of-Thought (CoT), Process Reward Models (PRMs), Tree Search (MCTS), Self-Correction, and Test-Time Compute Scaling (e.g., DeepSeek-R1, OpenAI o1/o3, Quiet-STaR).

📄 Landmark Papers & Essential Reading

Paper / Reference Authors / Year Key Takeaways & Focus Link
DeepSeek-R1: Incentivizing Reasoning Capability via RL DeepSeek AI (2025) Demonstrates pure reinforcement learning (GRPO) without supervised warmups to elicit emerging chain-of-thought reasoning and self-reflection. ArXiv Link
Let's Verify Step by Step (Process Reward Models) Lightman et al. (OpenAI, 2023) Introduces Process Reward Models (PRMs) trained to evaluate each intermediate reasoning step vs final outcome rewards (ORM). ArXiv Link
Tree of Thoughts: Deliberate Problem Solving with LLMs Yao et al. (2023) Framework allowing search algorithms (BFS, DFS, MCTS) over arbitrary reasoning trees with intermediate evaluation steps. ArXiv Link
Quiet-STaR: Language Models Can Teach Themselves to Think Zelikman et al. (2024) Enables LMs to generate rationale thoughts at every token position to improve future token predictions during pre-training. ArXiv Link
STaR: Bootstrapping Reasoning With Reasoning Zelikman et al. (2022) Iterative loop generating reasoning rationales for correct answers and fine-tuning on self-generated solutions. ArXiv Link
SCoRe: Training Language Models to Self-Correct via Reinforcement Learning Kumar et al. (Google DeepMind, 2024) Trains models to correct their own mistakes in a multi-turn RL setup without relying on external oracle feedback. ArXiv Link
Scaling LLM Test-Time Compute Optimally Snell et al. (Google DeepMind, 2024) Demonstrates how trading off test-time compute (search & sampling) vs pre-training compute scales benchmark accuracy. ArXiv Link

💻 Top Open-Source Frameworks & Repositories

Repository Focus Area Description Link
Open-R1 Reasoning Fine-Tuning Hugging Face's open implementation to reproduce DeepSeek-R1 via RL, synthetic data curation, and GRPO training. GitHub Repo
veRL Distributed RL for LLMs Flexible, high-performance RL library from Volcengine optimized for training reasoning models with PPO and GRPO. GitHub Repo
DeepSeek-R1 Open Weights & Distillations Official repo releasing DeepSeek-R1 open weights, distilled LLaMA and Qwen models (1.5B to 70B parameters). GitHub Repo
TinyZero Minimalist R1 Reproduction Clean, reproducible 300-line implementation of DeepSeek-R1 style RL reasoning training on countdown tasks. GitHub Repo
search-r1 RAG + Reasoning RL Integrates search engine retrieval into RL reasoning trace generation for fact-checked self-reflection. GitHub Repo

🎥 Must-Watch YouTube Videos & Free Lectures

Video / Playlist Creator Description Link
DeepSeek-R1 & Reinforcement Learning Breakthrough Andrej Karpathy Deep technical breakdown of how pure RL incentivizes CoT reasoning, reward modeling, and distilled reasoning models. Watch Video
DeepSeek-R1 Paper Walkthrough & PyTorch Code Umar Jamil Line-by-line derivation of Group Relative Policy Optimization (GRPO), rule-based rewards, and token reasoning traces. Watch Video
Process Reward Models & Tree of Thoughts Explanation Yannic Kilcher Paper breakdown of PRMs (Let's Verify Step by Step) and BFS/DFS tree search algorithms over LLM tokens. Watch Video
Reasoning & Test-Time Compute Scaling Masterclass DeepLearning.AI Masterclass explaining test-time compute tradeoffs, search strategies, and reward model architecture. Watch Video

🎓 Free Courses & Interactive Guides

Resource Title Institution / Host Focus Link
Hugging Face Deep RL Course Hugging Face Free community course covering Policy Gradients, PPO, DPO, and GRPO for fine-tuning LLMs. Hugging Face Course
Reasoning Models Handbook Community Hub Curated collection of tutorials, notebooks, and prompt templates for building reasoning chains. GitHub Repo

⚙️ Code References & Hands-on Notebooks