The Curious Engineer

All blogs / AI / ML

vLLM Blog

vllm.ai/blog

Technical articles, release announcements, model guides, and community updates from the vLLM project.

AI / MLPerformance / optimizationDistributed systems / cloud

’23’24’25’26
Posts
163 from 20 Jun 2023 to 29 Oct 2026
Posting
weekly (124 in the last year)
Kind
project news
Feed
https://vllm.ai/blog/atom.xml

2026 105

  1. Day 0 support for Mistral Large 4 on vLLM
  2. Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72
  3. GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s
  4. vLLM Delivers Day-0 Support for MiniMax-M3.1-Flash
  5. Day-0 Support for Flint on vLLM
  6. Announcing Humming: Fast, Flexible Kernels for Quantized Inference
  7. Introducing vLLM TorchTPU
  8. Achieving near-zero-downtime scaling in Elastic Expert Parallelism
  9. Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly
  10. Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX
  11. Serving Reflection Beam on vLLM from Day 0
  12. Day 0 support for Muse Spark Open on vLLM
  13. Announcing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM
  14. Adaptive Verification Without a Confidence Head
  15. Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM
  16. Finish RL Training Faster: Bitwise-Consistent, High-Performance, Agent-Driven
  17. Improve TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM
  18. vLLM Support for NVIDIA Vera Rubin NVL72: 7.8x Throughput over GB200 NVL72
  19. DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0
  20. Taking vLLM Apart: A Practical Guide to Disaggregated Serving
  21. Watermarking in vLLM
  22. Reducing vLLM startup time with one uv flag
  23. Announcing vllm-metal: Concurrent Serving on Apple Silicon
  24. PD Serving of Qwen3.8-2.4T
  25. Scaling Multi-GPU Video Captioning with PyNvVideoCodec and vLLM
  26. How we trained the fastest DSpark for Kimi-K3 using GB300 NVL72
  27. vLLM x Novita AI: Chord, Faster INT4 MoE for Kimi K2.x. Up to 1.3x on H200, 2.15x on Untuned B300
  28. vime × RL-Kernel × AMD: Bitwise Train–Rollout Consistency on ROCm
  29. Kimi K3 Performance Optimizations in vLLM: The Road to 2.8× Throughput
  30. Following the Bottleneck: Optimizing MiniMax M3 on AMD Instinct MI355X
  31. Tiered KV Cache Offloading in vLLM
  32. GLM 5.3 Optimizations, Part 1: Hybrid HiSparse Offloading in vLLM
  33. vLLM x AgentX: Optimizing for Real-World Agentic Serving
  34. Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin
  35. MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3
  36. Exploring Speculative Decoding in vLLM on AMD GPUs
  37. Large-Scale Sharded Weight Transfer with Ray Direct Transport (RDT) in vLLM
  38. IsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRL
  39. VeRL-Omni v0.2.0: Faster Diffusion RL and Stable Omni Training
  40. Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni
  41. Adaptive Verification in vLLM: DSpark confidence-scheduled verification
  42. Day 0 Support for Qwen3.8-2.4T-A95B on vLLM
  43. Announcing Day-0 Support for NVIDIA Nemotron 3.5 Lightning on vLLM
  44. Efficient Decode Context Parallelism with vLLM for Long Context Workloads
  45. vLLM Reaches 25K Total TPS/GPU on Qwen3.5
  46. Optimizing vLLM on Arm CPUs
  47. Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding
  48. Kimi K3 Is Here: Efficient Day-0 Support on vLLM
  49. From Day 0 to Production SLAs: Serving GLM-5.2 on 24 NVIDIA B300 GPUs with vLLM
  50. Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving
  51. A Preview of Production-Scale Kimi K3 Support on vLLM
  52. Beyond a Single Model: Building Mixture-of-Models Systems with vLLM Semantic RouterJul 21, 2026·15 min readvLLM Semantic Router is expanding from intelligent routing into a system for building, evaluating, and running Mixture-of-Models.
  53. Keeping vLLM Production Quality: A Look Inside CI, Benchmarking, and the Release ProcessJul 16, 2026·20 min readHow vLLM maintains production quality with extensive CI across diverse accelerators, nightly performance benchmark and accuracy evaluation, and a two-week release process.
  54. TML Inkling on vLLM: Day-0 Support with Optimized PerformanceJul 15, 2026·8 min readvLLM brings day-0 support to TML Inkling, a 1T-parameter multimodal model, with MTP, long-context serving, parallelism, and up to 380 tokens per second per user on NVIDIA GB200 GPUs.
  55. vLLM x TileRT: Specialized Decode for Latency-Critical ServingJul 14, 2026·7 min readvLLM prefill paired with TileRT decode through vLLM V1's connector interface: a specialized, latency-optimized decode engine that coexists with native vLLM decode behind one shared serving layer, with zero changes t
  56. EAGLE3 Speculative Decoding on AMD Instinct GPUs: Training and Serving with vLLM and AMD QuarkJul 13, 2026·13 min readHow AMD Quark trains, quantizes, and serves EAGLE3 speculative-decoding drafts with vLLM on AMD Instinct GPUs, delivering up to 2.00x throughput gains for Kimi-K2.5 and 1.79x for Min
  57. vime + ROCm: End-to-End RL Post-Training on AMD Instinct™ GPUsJul 10, 2026·10 min readAnnouncing ROCm support for vime, now running end-to-end on AMD Instinct MI355X GPUs with prebuilt container.
  58. vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent HunyuanJul 6, 2026·12 min readHow HPC-Ops integrates Hopper-optimized attention and FP8 MoE backends into vLLM for Tencent Hunyuan Hy3, improving mixed-length decode, MoE latency, TTFT, and TPOT on NVIDIA H20.
  59. Experience and Lessons Learned from Serving Multi-Stage Qwen3-Omni in vLLM-OmniJul 1, 2026·12 min readHow vLLM-Omni serves and optimizes Qwen3-Omni with staged Thinker-Talker-Code2Wav execution, batching, CUDA Graphs, async chunk, async output, replicas, hot-path cleanup, and perf validation.
  60. Micro-Agent: Beat Frontier Models with Collaboration inside Model APIJun 29, 2026·15 min readHow vLLM Semantic Router turns vllm-sr/auto into a bounded micro-agent runtime for Confidence, Ratings, ReMoM, Fusion, Workflows, and benchmark-shaped collaboration.
  61. Engineering TTS Inference in vLLM-OmniJun 23, 2026·23 min readHow vLLM-Omni supports and optimizes Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro with staged serving, batching, CUDA Graphs, and model-specific kernels.
  62. Beyond One Model: Fusion in vLLM Semantic RouterJun 16, 2026·10 min readHow vLLM Semantic Router Fusion runs a panel of models, uses a judge to analyze agreement and gaps, and synthesizes one answer while preserving routing policy, traces, and OpenAI-compatible serving.
  63. MiniMax M3 in vLLM: Day-0 Serving for 1M-Token Multimodal ReasoningJun 12, 2026·21 min readHow vLLM serves MiniMax M3 with MiniMax Sparse Attention, multimodal and reasoning parsers, MXFP8 weights, and long-context deployment recipes.
  64. DiffusionGemma: The First Diffusion LLM (dLLM) Natively Supported in vLLMJun 10, 2026·6 min readHow vLLM supports DiffusionGemma, the first native diffusion language model in vLLM, using Model Runner V2 state hooks, iterative denoising, bidirectional attention, and reused speculative decoding paths.
  65. Announcing vime: A Simple, Stable, and Efficient RL Framework for LLMsJun 9, 2026·6 min readvime connects slime's training stack with vLLM rollouts to provide a simple, stable, and efficient RL post-training pipeline.
  66. vLLM Semantic Router v0.3 Themis: From Signals to Stateful Production RoutingJun 5, 2026·22 min readWhat vLLM Semantic Router v0.3 Themis adds for production routing: canonical config, inspectable signal-decision-policy flows, safer operations, CLI/dashboard/Kubernetes alignment, and replayable rout
  67. Announcing Day-0 Support for NVIDIA Nemotron 3 Ultra on vLLMJun 4, 2026·7 min readHow to serve NVIDIA Nemotron 3 Ultra with vLLM for long-running agentic reasoning, including BF16 and NVFP4 checkpoints, supported GPU configurations, OpenAI-compatible deployment, and NeMo RL integration.
  68. Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AIJun 3, 2026·5 min readWhat the DeepLearning.AI vLLM course teaches: optimizing, deploying, and benchmarking LLM inference with LLM Compressor quantization, GuideLLM, KV cache sizing, serving, and memory tradeoffs.
  69. Accelerating vLLM-Omni Inference with AutoRound QuantizationJun 2, 2026·10 min readHow AutoRound integrates with vLLM-Omni to serve W4A16 quantized multimodal, diffusion, image, and video models with smaller checkpoints, preserved quality, Intel XPU acceleration, and NVIDIA GPU support.
  70. Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM AgentsJun 2, 2026·14 min readHow Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cache-aware switch pricing, an
  71. vLLM on the DGX Spark: Architecture, Configuration, and Local EvaluationJun 1, 2026·16 min readHow to run vLLM on NVIDIA DGX Spark and GB10 systems, including unified memory behavior, NVFP4 Nemotron-3-Super serving, Docker deployment, Prometheus metrics, and local evaluation results.
  72. From Text to Multimodal Routing: Hardening Vision Signals in vLLM Semantic RouterMay 28, 2026·12 min readHow vLLM Semantic Router hardens multimodal routing by turning visual evidence into trustworthy signals, debugging a Rust/Candle vision-encoder parity issue, and validating image signal correctne
  73. Speculators v0.5.0: DFlash Support and Online TrainingMay 28, 2026·6 min readWhat Speculators v0.5.0 adds for vLLM speculative decoding: DFlash block-diffusion draft models, unified online and offline training, native hidden-state extraction, and Gemma 4 latency results.
  74. Native RL APIs in vLLMMay 28, 2026·12 min readHow vLLM native RL APIs standardize weight syncing and asynchronous RL serving with NCCL and CUDA IPC transfer backends, pause mode, and fixes for fragile DPEP and disaggregated rollout deployments.
  75. Accelerating Laguna XS.2 Inference with vLLM, Speculators, and LLM CompressorMay 28, 2026·3 min readHow Laguna XS.2 is served and optimized in vLLM using first-class model integration, a DFlash speculator trained with Speculators, and FP8, NVFP4, INT4, and INT8 checkpoints from LLM Compressor.
  76. EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpecMay 26, 2026·4 min readHow EAGLE 3.1 improves speculative decoding robustness in vLLM with FC normalization, post-norm hidden-state feedback, TorchSpec training support, and config-driven compa
  77. vLLM x Novita AI: PegaFlow for Production-Grade External KV CacheMay 18, 2026·13 min readHow PegaFlow integrates with vLLM as an external KV cache service, using a Rust daemon, CUDA IPC, RDMA, SSD caching, and the external KV connector to improve startup, sharing, throughput, and cache lifecycle.
  78. Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality ModelsMay 14, 2026·7 min readHow VeRL-Omni extends verl with vLLM-Omni for reinforcement learning post-training of diffusion and multimodal generative models, including efficient rollouts, reward inference, trai
  79. Elastic Expert Parallelism in vLLMMay 14, 2026·11 min readHow Elastic Expert Parallelism lets vLLM scale Mixture-of-Experts serving up or down at runtime by changing data-parallel workers, redistributing experts, and coordinating live topology changes without server restarts.
  80. vLLM Tops the Artificial Analysis LeaderboardMay 11, 2026·15 min readHow vLLM achieved leading Artificial Analysis results for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B using open-source kernel fusion, speculative decoding, Blackwell optimizations, and model-specific serving work.
  81. A First Comprehensive Study of TurboQuant: Accuracy and PerformanceMay 11, 2026·12 min readA vLLM study comparing TurboQuant KV-cache quantization with BF16 and FP8 across long-context and reasoning workloads, showing where 4-bit variants help, where accuracy drops, and why FP8 remains the default c
  82. Serving Agentic Workloads at Scale with vLLM x MooncakeMay 6, 2026·10 min readHow vLLM integrates Mooncake Store as a distributed KV cache for agentic workloads, reusing shared prefixes across turns and instances to improve throughput, TTFT, end-to-end latency, and multi-GPU scaling.
  83. Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLMApr 28, 2026·7 min readHow to serve NVIDIA Nemotron 3 Nano Omni with vLLM for multimodal agentic AI, including BF16, FP8, and NVFP4 checkpoints, vision/audio/video inputs, supported GPUs, OpenAI-compatible APIs, an
  84. DeepSeek V4 in vLLM: Efficient Long-context AttentionApr 24, 2026·17 min readA first-principles walkthrough of DeepSeek V4's long-context attention, and how we implemented it in vLLM.
  85. The State of FP8 KV-Cache and Attention Quantization in vLLMApr 22, 2026·21 min readWhat vLLM FP8 KV-cache validation found across Hopper and Blackwell, covering attention quantization, Flash Attention 3 fixes, memory savings, decode speedups, and layers to skip.
  86. Disaggregated Serving for Hybrid SSM Models in vLLMApr 21, 2026·15 min readHow vLLM extends NIXL prefill/decode disaggregation to hybrid SSM-attention models with dual descriptor views, physical-logical block bridging, and Mamba conv-state transfer support.
  87. vLLM Korea Meetup 2026 Wrap-UpApr 14, 2026·7 min readWhat the vLLM Korea Meetup 2026 covered: community growth, vLLM V1 updates, production stack adoption, accelerator integration, vllm-playground, and real-world LLM serving.
  88. Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode DisaggregationApr 7, 2026·22 min readHow single-node prefill/decode disaggregation in vLLM uses AMD MORI-IO on an 8-GPU MI300X node to separate prefill and decode, transfer KV cache efficiently, stabilize ITL, and improve goo
  89. Announcing Gemma 4 on vLLM: Byte for byte, the most capable open modelsApr 2, 2026·3 min readHow vLLM supports Google's Gemma 4 open models across NVIDIA, AMD, Intel, and TPU backends, with multimodal inputs, agentic workflows, long context, function calling, and deployment recipes.
  90. Extracting hidden states from vLLMMar 30, 2026·8 min readHow vLLM extracts verifier hidden states through dummy draft models and KV Connector APIs for speculative decoding, enabling offline and online Speculators training without patching vLLM internals.
  91. Model Runner V2: A Modular and Faster Core for vLLMMar 24, 2026·8 min readHow Model Runner V2 reworks vLLM's execution core with modular model logic, GPU-native input preparation, stable persistent batching, async-first scheduling, and no API changes.
  92. P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLMMar 13, 2026·12 min readHow P-EAGLE brings parallel speculative decoding to vLLM by generating multiple draft tokens in one forward pass, with pre-trained drafter heads, config support, and B200 speedups over EAGLE-3.
  93. Run Highly Efficient and Accurate Multi-Agent AI with NVIDIA Nemotron 3 Super Using vLLMMar 11, 2026·5 min readHow to serve NVIDIA Nemotron 3 Super with vLLM for multi-agent AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, Thinking Budget, MTP, supported GPUs, and OpenAI-compatible
  94. vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System BrainMar 10, 2026·23 min readWhat vLLM Semantic Router v0.2 Athena adds: refreshed multilingual and multimodal routing models, ONNX and ROCm acceleration, safety and memory signals, long-context handling, and ClawOS orchestratio
  95. vLLM Triton Attention Backend Deep DiveMar 4, 2026·10 min readA technical walkthrough of the vLLM Triton attention backend, covering performance-portable paged attention kernels, backend selection, autotuning, CUDA graph behavior, benchmarks, and NVIDIA, AMD, and Intel support.
  96. Beyond Porting: How vLLM Orchestrates High-Performance Inference on AMD ROCmFeb 27, 2026·19 min readHow vLLM orchestrates high-performance inference on AMD ROCm with multiple attention backends, workload-aware prefill, extend, and decode routing, AITER primitives, MLA support, and MI300X-class bench
  97. Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon BedrockFeb 26, 2026·11 min readHow vLLM serves many fine-tuned MoE and dense models with Multi-LoRA, including fused MoE LoRA kernels, Triton compiler fixes, Split-K and CTA swizzling optimizations, and SageMak
  98. DeepSeek-V3.2 on GB300: Performance BreakthroughFeb 13, 2026·12 min readWhat DeepSeek-V3.2 and DeepSeek-R1 benchmark results show on NVIDIA GB300 with vLLM, covering NVFP4 quantization, TP and EP deployment, throughput, and reproducible setup details.
  99. Driving vLLM WideEP and Large-Scale Serving Toward Maturity on Blackwell (Part I)Feb 3, 2026·10 min readHow vLLM improves WideEP and large-scale DeepSeek-style MoE serving on NVIDIA GB200 with NVFP4 and FP8 kernels, fusion, prefill/decode disaggregation, weight offloading, and reduced chunking overh
  100. GPT-OSS Performance Optimizations on NVIDIA Blackwell: Pushing the Pareto FrontierFeb 1, 2026·8 min readHow vLLM and NVIDIA optimized GPT-OSS on Blackwell with FlashInfer, torch.compile fusion, FP8 KV cache, async scheduling, stream interval tuning, and deployment recipes that improve throughput and
  101. Streaming Requests & Realtime API in vLLMJan 31, 2026·19 min readHow vLLM supports streamable inputs and a Realtime WebSocket API for audio, video, robotics, and low-latency applications that need incremental input processing instead of complete prompts.
  102. Building Mixture-of-Models on AMD GPUs with vLLM-SRJan 23, 2026·10 min readHow vLLM Semantic Router builds a Mixture-of-Models system on AMD MI300X and MI355X GPUs, routing across specialized models with signals, decisions, safety checks, semantic caching, and live MoM deployment.
  103. Inside vLLM’s New KV Offloading Connector: Smarter Memory Transfer for Maximizing Inference ThroughputJan 8, 2026·15 min readHow vLLM's asynchronous KV offloading connector stores KV cache in CPU memory to reduce recomputation, improve throughput under memory pressure, and support pluggable offload
  104. vLLM Semantic Router v0.1 Iris: The First Major ReleaseJan 5, 2026·9 min readWhat vLLM Semantic Router v0.1 Iris introduces: signal-decision plugin architecture, model selection, safety filtering, semantic caching, hallucination detection, LoRA-based routing models, and production-ready MoM routing.
  105. Introducing vLLM Playground: A Modern Web Interface for Managing and Interacting with vLLM ServersJan 2, 2026·6 min readHow vLLM Playground provides a web UI for starting, configuring, testing, and monitoring vLLM servers across local macOS, Linux GPU or CPU, Kubernetes, and OpenShift environments.

2025 51

  1. Announcing vllm.ai Website and Some Community UpdatesDec 27, 2025·3 min readWhat changed on the new vllm.ai website for vLLM users: installation guidance, events pages, Slack and X community channels, vLLM Daily updates, and a clearer project/community split.
  2. vLLM-Omni Diffusion Cache AccelerationDec 19, 2025·4 min readHow vLLM-Omni speeds up diffusion model inference with Cache-DiT and TeaCache, reusing intermediate computations across timesteps to deliver 1.5x to 2x image generation speedups with minimal quality loss.
  3. vLLM Large Scale Serving: DeepSeek @ 2.2k tok/s/H200 with Wide-EPDec 17, 2025·8 min readHow vLLM reaches 2.2k tokens per second per H200 for DeepSeek-style MoE serving with Wide-EP, async scheduling, dual-batch overlap, disaggregated serving, CUDA graphs, DeepGEMM, and expert load balancing.
  4. AMD × vLLM Semantic Router: Building the System Intelligence TogetherDec 16, 2025·11 min readHow AMD and vLLM Semantic Router build GPU-accelerated Mixture-of-Models routing with signals, semantic caching, response storage, PII, jailbreak, and hallucination guardrails.
  5. Encoder Disaggregation for Scalable Multimodal Model ServingDec 15, 2025·9 min readHow vLLM EPD separates visual encoders from text prefill and decode, covering LMM serving, GPU resource scaling, multimodal interference, and pipelined execution.
  6. Run Highly Efficient and Accurate AI Agents with NVIDIA Nemotron 3 Nano on vLLMDec 15, 2025·5 min readHow to serve NVIDIA Nemotron 3 Nano with vLLM for efficient agentic AI, including BF16, FP8, and NVFP4 checkpoints, 1M-token context, hybrid MoE architecture, Thinking Budget, supported GPUs, and Op
  7. Token-Level Truth: Real-Time Hallucination Detection for Production LLMsDec 14, 2025·13 min readHow HaluGate adds token-level hallucination detection to vLLM Semantic Router by verifying assistant claims against tool outputs and grounding context in real time without LLM-as-judge overhead.
  8. vLLM Router: A High-Performance and Prefill/Decode Aware Load Balancer for Large-scale ServingDec 13, 2025·5 min readWhat vLLM Router provides for large-scale serving: Rust-based state-aware load balancing, KV-cache affinity, prefill/decode disaggregation orchestration, Kubernetes discovery, retries
  9. Diving into speculative decoding training support for vLLM with Speculators v0.3.0Dec 13, 2025·11 min readHow Speculators v0.3.0 supports end-to-end Eagle3 draft model training for vLLM, including hidden-state data generation, MoE and non-MoE verifiers, offline workflows, and seamless speculative de
  10. Advancing Low‑Bit Quantization for LLMs: AutoRound x LLM CompressorDec 9, 2025·5 min readHow Intel AutoRound integrates with LLM Compressor to produce low-bit quantized checkpoints for vLLM, using tuning-based PTQ, W4A16 and related formats, compressed-tensors compatibility, and lightweight calibrat
  11. Tracing Hanging and Complicated GPU Kernels Down To The Source CodeDec 3, 2025·13 min readHow vLLM developers debug hanging and complex CUDA kernels by triggering GPU core dumps, identifying stuck kernels, and mapping failures back to source code lines for faster kernel debugging.
  12. Announcing vLLM-Omni: Easy, Fast, and Cheap Omni-Modality Model ServingNov 30, 2025·4 min readWhat vLLM-Omni adds to the vLLM ecosystem: omni-modality serving for text, image, video, and audio, diffusion and non-autoregressive generation support, disaggregated stages, OpenAI-compatible APIs, and pip
  13. Streamlined multi-node serving with Ray symmetric-runNov 22, 2025·4 min readHow Ray symmetric-run simplifies multi-node vLLM serving by launching the same entrypoint on every Ray cluster node, matching HPC and parallel SSH workflows for distributed model deployments.
  14. Building Clean, Maintainable vLLM Modifications Using the Plugin SystemNov 20, 2025·12 min readHow the vLLM plugin system helps teams customize scheduling, KV-cache behavior, hardware integrations, and model execution without long-lived forks, monkey patches, or brittle internal modifications.
  15. Signal-Decision Driven Architecture: Reshaping Semantic Routing at ScaleNov 19, 2025·14 min readHow vLLM Semantic Router replaces fixed domain classification with signal-decision architecture, combining multi-dimensional signals, AND/OR decision logic, model selection, and plugin orchestration for p
  16. Docker Model Runner Integrates vLLM for High-Throughput InferencingNov 19, 2025·6 min readHow Docker Model Runner integrates vLLM as an inference backend, letting developers run safetensors models with high-throughput serving, PagedAttention, streaming, and OpenAI-compatible APIs from Docker workflo
  17. Shared Memory IPC Caching: Accelerating Data Transfer in LLM Inference SystemsNov 13, 2025·7 min readHow shared memory IPC caching in vLLM reduces redundant data transfers for multimodal and multi-process inference, improving prefill throughput and TTFT by sharing large inputs across coordinator and
  18. Fast and Affordable LLMs serving on Intel Arc Pro B-Series GPUs with vLLMNov 11, 2025·9 min readHow vLLM serves LLMs on Intel Arc Pro B-Series GPUs with MoE optimizations, persistent kernels, multi-GPU scaling, LoRA, speculative decoding, structured outputs, and mixed-precision recipes.
  19. No More Train-Inference Mismatch: Bitwise Consistent On-Policy Reinforcement Learning with vLLM and TorchTitanNov 10, 2025·6 min readHow vLLM and TorchTitan demonstrate bitwise consistent on-policy RL by matching training and inference numerics, using batch-invariant kernels to reduce train-inferenc
  20. Run Multimodal Reasoning Agents with NVIDIA Nemotron on vLLMOct 31, 2025·4 min readHow to serve NVIDIA Nemotron Nano 2 VL with vLLM for multimodal reasoning agents, including video understanding, document intelligence, Efficient Video Sampling, 128K context, and OpenAI-compatible deployment.
  21. Chasing 100% Accuracy: A Deep Dive into Debugging Kimi K2's Tool-Calling on vLLMOct 28, 2025·11 min readHow vLLM debugged Kimi K2 tool-calling accuracy, covering chat-template compatibility, add_generation_prompt handling, schema validation failures, benchmark fixes, and tool-use reliability.
  22. From Monolithic to Modular: Scaling Semantic Routing with Extensible LoRAOct 27, 2025·8 min readHow vLLM Semantic Router refactors its Rust classification layer with modular model support, Qwen3-Embedding, EmbeddingGemma, LoRA-based multi-task classification, and concurrent routing execution.
  23. Zero-Reload Model Switching with vLLM Sleep ModeOct 26, 2025·17 min readHow vLLM Sleep Mode enables fast model switching by hibernating weights to CPU RAM or discarding them while preserving process state, CUDA graphs, allocators, and kernel warmup to avoid full reloads.
  24. Now Serving NVIDIA Nemotron with vLLMOct 23, 2025·5 min readHow vLLM serves NVIDIA Nemotron Nano 2 for agentic reasoning, including hybrid Transformer-Mamba architecture, thinking budget control, open weights and data, throughput benefits, and deployment commands.
  25. No More Retokenization Drift: Returning Token IDs via the OpenAI Compatible API Matters in Agent RLOct 22, 2025·9 min readHow vLLM's OpenAI-compatible API can return prompt and response token IDs to prevent retokenization drift in agent reinforcement learning, preserving exact sampled sequences for
  26. vLLM TPU: A New Unified Backend Supporting PyTorch and JAX on TPUOct 16, 2025·11 min readHow the redesigned vLLM TPU backend uses tpu-inference, JAX-to-XLA lowering, Torchax, ragged paged attention, and unified PyTorch and JAX support to improve TPU performance and model coverage.
  27. SemiAnalysis InferenceMAX: vLLM and NVIDIA Accelerate Blackwell InferenceOct 9, 2025·8 min readHow vLLM and NVIDIA optimize Blackwell inference for SemiAnalysis InferenceMAX, improving gpt-oss 120B and Llama 3.3 70B throughput with FP4 kernels, scheduling work, and Pareto-frontier benchmarking.
  28. DeepSeek-V3.2-Exp in vLLM: Fine-Grained Sparse Attention in ActionSep 29, 2025·8 min readHow vLLM supports DeepSeek-V3.2-Exp with DeepSeek Sparse Attention, lightning indexer caches, separate prefill and decode paths, FlashMLA sparse attention, DeepGEMM kernels, and Blackwell deployment.
  29. The First vLLM Meetup in KoreaSep 16, 2025·5 min readWhat the first vLLM Korea meetup covered: community adoption, llm-d, TPU integration, contribution workflows, hardware plugin architecture, Rebellions NPU work, and production inference lessons.
  30. vLLM Semantic Router: Next Phase in LLM inferenceSep 11, 2025·4 min readHow vLLM Semantic Router routes requests by intent, covering semantic classification, smart reasoning-path selection, Rust and Candle execution, and Kubernetes Envoy integration for efficient inference.
  31. vLLM Now Supports Qwen3-Next: Hybrid Architecture with Extreme EfficiencySep 11, 2025·3 min readHow vLLM supports Qwen3-Next with hybrid attention, Gated DeltaNet, full attention, high-sparsity MoE, multi-token prediction, hybrid KV cache management, Triton kernels, and CUDA graphs.
  32. Serving Geospatial, Vision, and Beyond: Enabling Multimodal Output Processing in vLLMSep 5, 2025·10 min readHow vLLM expands beyond text generation to serve geospatial, vision, and other non-autoregressive models with pooling-model support, TerraTorch integration, raw tensor handling, and flexible I
  33. Inside vLLM: Anatomy of a High-Throughput LLM Inference SystemSep 5, 2025·41 min readHow vLLM's inference engine works, covering PagedAttention, continuous batching, prefix caching, speculative decoding, multi-GPU serving, scheduling, and benchmarking for high-throughput LLM workloads.
  34. Introduction to torch.compile and How It Works with vLLMAug 20, 2025·14 min readHow torch.compile works inside vLLM, including TorchDynamo graph capture, TorchInductor code generation, custom compiler passes, graph breaks, model compilation strategy, and performance optimization.
  35. GLM-4.5 Meets vLLM: Built for Intelligent AgentsAug 19, 2025·4 min readHow to run GLM-4.5 and GLM-4.5V with vLLM for intelligent agents, including hybrid reasoning modes, FP8 and BF16 serving, multimodal support, and NVIDIA Blackwell and Hopper deployment.
  36. CUDA Core Dump: An Effective Tool to Debug Memory Access Issues and BeyondAug 11, 2025·16 min readHow to debug vLLM CUDA illegal memory access errors with CUDA core dumps, environment variables, cuda-gdb, and GPU state inspection when Python stack traces or CUDA_LAUNCH_BLOCKING are insufficient.
  37. vLLM Now Supports gpt-ossAug 5, 2025·5 min readHow vLLM supports gpt-oss 20B and 120B on NVIDIA Blackwell, Hopper, and AMD GPUs, with MXFP4 MoE kernels, efficient attention, hybrid KV cache allocation, and built-in tool support.
  38. MiniMax-M1 Hybrid Architecture Meets vLLM: Long Context, Fast InferenceJun 30, 2025·6 min readHow vLLM serves MiniMax-M1's hybrid MoE architecture for long-context inference, covering model deployment, memory management, batched serving, backend optimizations, and Docker-based setup.
  39. Introducing vLLM Hardware Plugin, Best Practice from Ascend NPUMay 12, 2025·5 min readHow vLLM hardware plugins decouple backend integrations from core vLLM, using Platform, Executor, Worker, ModelRunner, AttentionBackend, and Communicator hooks to support Ascend NPU and IBM Spyre.
  40. Accelerating RLHF with vLLM, Best Practice from OpenRLHFApr 23, 2025·5 min readHow OpenRLHF uses vLLM, Ray, ZeRO-3, AutoTP, Ray placement groups, and weight synchronization to accelerate PPO and RLHF sample generation for reasoning models with long chain-of-thought outputs.
  41. Transformers modeling backend integration in vLLMApr 11, 2025·6 min readHow vLLM integrates the Hugging Face Transformers modeling backend to serve more model architectures efficiently, including text and vision-language models through model_impl="transformers".
  42. Llama 4 in vLLMApr 5, 2025·4 min readHow vLLM serves Meta Llama 4 Scout and Maverick multimodal MoE models with long-context support, tensor parallel deployment, H100 and H200 guidance, FP8 variants, and performance tips.
  43. PTPC-FP8: Boosting vLLM Performance on AMD ROCmFeb 24, 2025·9 min readHow PTPC-FP8 quantization improves vLLM performance on AMD ROCm by combining per-token activation scaling and per-channel weight scaling for near-BF16 accuracy with FP8 speed.
  44. Introducing AIBrix: A Scalable, Cost-Effective Control Plane for vLLMFeb 21, 2025·4 min readWhat AIBrix adds as a Kubernetes control plane for vLLM: LoRA management, LLM gateway routing, autoscaling, unified runtime, distributed inference, distributed KV cache, heterogeneous serving, and GPU failure
  45. Distributed Inference with vLLMFeb 17, 2025·5 min readA guide to distributed inference in vLLM, covering tensor parallelism, pipeline parallelism, multi-GPU and multi-node serving, KV cache challenges, speculative decoding, communication kernels, and control-plane design.
  46. vLLM V1: A Major Upgrade to vLLM's Core ArchitectureJan 27, 2025·11 min readWhat changed in vLLM V1: a re-architected engine with a simpler scheduler, near-zero-overhead prefix caching, cleaner tensor parallelism, multiprocessing API server, and default optimizations for higher-throughput serving.
  47. Introducing vLLM Inference Provider in Llama StackJan 27, 2025·8 min readHow Llama Stack integrates vLLM as an inference provider through remote and inline providers, enabling OpenAI-compatible vLLM serving for local and Kubernetes generative AI application deployments.
  48. High Performance and Easy Deployment of vLLM in K8S with vLLM production-stackJan 21, 2025·4 min readWhat vLLM production-stack adds for Kubernetes serving: prefix-aware routing, LMCache-backed KV cache sharing, autoscaling, observability, fault tolerance, and cluster deployment with higher throughp
  49. Structured Decoding in vLLM: a gentle introductionJan 14, 2025·12 min readHow structured decoding works in vLLM, covering JSON outputs, grammar-guided generation, outlines, XGrammar, TPOT improvements, constrained decoding, and agentic workflow use cases.
  50. vLLM 2024 Retrospective and 2025 VisionJan 10, 2025·11 min readA vLLM 2024 retrospective and 2025 roadmap covering community growth, model and hardware support, production adoption, office hours, ecosystem partnerships, and the path toward universal open-source serving.
  51. Installing and Developing vLLM with EaseJan 10, 2025·5 min readHow to install and develop vLLM using stable releases, nightly wheels, uv, source builds, Python and C++/CUDA workflows, and version tracking for production deployments.

2024 5

  1. Serving LLMs on AMD MI300X: Best PracticesOct 23, 2024·15 min readBest practices for serving LLMs with vLLM on AMD MI300X, covering ROCm setup, Llama 3.1 70B and 405B benchmarks, chunked prefill, multi-step scheduling, prefix caching, graph capture, and AMD tuning.
  2. How Speculative Decoding Boosts vLLM Performance by up to 2.8xOct 17, 2024·10 min readHow speculative decoding works in vLLM, covering EAGLE, Medusa, n-gram proposals, draft and target runners, scheduler and memory-manager changes, and continuous batching for lower token latency.
  3. vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency ReductionSep 5, 2024·12 min readWhat changed in vLLM v0.6.0 to improve throughput and latency, including API server isolation, reduced CPU overhead, multi-step scheduling, async execution, and benchmarks against earlier vLLM versions.
  4. vLLM’s Open Governance and Performance RoadmapJul 25, 2024·4 min readvLLM's open governance and performance roadmap, covering LF AI and Data incubation, public benchmarks, optimized kernels, async scheduling, API frontend overhead, torch.compile, disaggregated prefill, and community research.
  5. Announcing Llama 3.1 Support in vLLMJul 23, 2024·6 min readHow vLLM supports Meta Llama 3.1 models, including 128K context, Llama 3.1 405B serving, chunked prefill, FP8 quantization, tensor and pipeline parallelism, CPU offloading, and early performance results.

2023 2

  1. Notes on vLLM v.s. DeepSpeed-FastGenNov 14, 2023·4 min readA performance comparison of vLLM and DeepSpeed-FastGen, explaining when Dynamic SplitFuse helps, where vLLM is faster, and how memory allocation, output length, and workload shape affect throughput.
  2. vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionJun 20, 2023·8 min readWhat the original vLLM launch announced: PagedAttention for KV cache management, up to 24x throughput over Hugging Face Transformers, and lower-cost high-throughput LLM serving.