⚡ LLMOps, Serving, Evals & Observability — Topic Resources
Curated list of top landmark papers, open-source repositories, video series, free courses, and code references for LLM Serving Engines, PagedAttention, LLMOps Infrastructure, LLM-as-a-Judge Evals, Observability, and Safety Guardrails .
📄 Landmark Papers & Essential Reading
Paper / Reference
Key Takeaways & Focus
Link
vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023) Introduced PagedAttention, eliminating KV cache fragmentation and achieving 2-4x higher throughput.
ArXiv Link
SGLang: Fast and Expressive Language Model Execution (Zheng et al., 2024) Introduces RadixAttention for automatic KV cache reuse across multi-turn agent calls and structured prompts.
ArXiv Link
Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2023) Uses a small draft model to generate candidate tokens verified in parallel by a larger target model.
ArXiv Link
DeepSeek-V3 Technical Report (Multi-Head Latent Attention) (DeepSeek AI, 2024) Introduces MLA to compress KV cache into low-rank latent vectors, achieving near-MHA quality with 93% memory savings.
ArXiv Link
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) Proved GPT-4 level LLM judges align 80%+ with human preference for evaluating chat assistants.
ArXiv Link
HELM: Holistic Evaluation of Language Models (Liang et al., 2022) Standardized benchmarking methodology assessing accuracy, bias, toxicity, robustness, and efficiency.
ArXiv Link
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023) Framework using CoT prompts and form-based scoring to measure generation quality.
ArXiv Link
Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022) Showed RLAIF (RL from AI Feedback) for aligning models against harmful prompts using explicit self-critique rules.
ArXiv Link
💻 Top Open-Source Frameworks & Serving Engines
Repository
Description
Link
vLLM High-throughput, memory-efficient LLM inference engine featuring PagedAttention and continuous batching.
GitHub Repo
LiteLLM Lightweight proxy gateway allowing 100+ LLM APIs to be invoked via OpenAI format with load balancing & fallback.
GitHub Repo
Ollama Get up and running with Llama 3, Mistral, and local LLMs locally with a simple API and CLI.
GitHub Repo
Ragas Framework for evaluating Retrieval Augmented Generation pipelines with metrics like faithfulness and context recall.
GitHub Repo
Phoenix (Arize) Open-source AI observability platform for tracing LLM applications, agent steps, and automated evals.
GitHub Repo
Promptfoo Command-line tool and library for testing, evaluating, and red-teaming LLM prompts and guardrails.
GitHub Repo
NeMo Guardrails NVIDIA's framework for adding programmable safety guardrails, topic control, and hallucination checks.
GitHub Repo
🎥 Must-Watch YouTube Videos & Free Lectures
Video / Playlist
Creator
Description
Link
vLLM: High-Throughput Serving & PagedAttention Woosuk Kwon (vLLM Lead)
Deep architectural explanation of virtual memory management for KV caches in GPU memory.
YouTube Video
LLM Evaluation & Observability Masterclass Arize AI / Phoenix
Step-by-step video guides on instrumenting OpenTelemetry tracing, LLM evaluation, and drift detection.
YouTube Channel
Building Guardrails & Red Teaming DeepLearning.AI
Practical short course on securing LLM apps against prompt injection, jailbreaks, and PII leakage.
DeepLearning.AI Site
🎓 Free Courses & Open Curricula
Course Title
Institution / Host
Focus
Link
LLMOps: Building Real-World AI Applications DeepLearning.AI
End-to-end operationalization of LLM apps: CI/CD for prompts, automated testing, and cloud deployment.
DeepLearning.AI Course
Full Stack LLM Bootcamp FSDL
Free lectures on LLM stack, serving, prompt engineering, cost optimization, and evals.
FSDL Course Site
⚙️ Code References & Hands-on Notebooks