The Curious Engineer

All blogs / AI / ML

LMSYS Org

lmsys.org/blog

Read the latest from LMSYS Org — research updates, technical deep dives, and announcements on large models, inference systems, reinforcement learning, and evaluation.

AI / MLPerformance / optimization

’23’24’25’26
Posts
133 from 30 Mar 2023 to 24 Oct 2026
Posting
weekly (81 in the last year)
Kind
project news

2026 66

  1. More Than DeepInfer: What SGLang Delivers
  2. Announcing the SGLang AWS Trainium Plugin Beta
  3. Elastic Expert Parallelism Scaling in SGLang: Growing a Live MoE Deployment with NIXL
  4. SGLang-TorchTPU: PyTorch-Native Production Serving on TPUs
  5. SGLang Low-Latency Mode: Kernels and Runtime
  6. SGLang Powers Beam in Production
  7. SGLang and Miles Add Day-0 Support for Muse Spark Open
  8. SGLang Adds Day-0 Support for NVIDIA Nemotron 3.5 Super VL
  9. SGLang and Miles on NVIDIA Vera Rubin
  10. Scaling JEV-like Decision Models with SGLang
  11. Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache
  12. SGLang and Miles Add Day-0 Support for DeepSeek-V4.1
  13. Running DeepSeek-V4-Flash and Kimi-K3 on Consumer Hardware with SSD Expert Pack
  14. Infer-forge: Harness, Loop, and Graph Engineering Around SGLang
  15. MiniMax-H3 on 8×H200: 1.95× Lossless, Up to 6.24× at 0.76–0.91 SSIM
  16. Qwen3.8-Flash-Next: Day-0 Support in SGLang
  17. Chasing the Batch-1 Floor: Ling-3.0-flash Speculative Decode on Blackwell
  18. Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon
  19. Mooncake for Miles: From Fragmented Rollout Data to Efficient Bulk I/O
  20. Pushing the Limits of Serving DeepSeek-V4-Pro
  21. Miles v0.1: Production-level Post-training
  22. Advanced CUDA Graph Techniques in SGLang
  23. SGLang and Miles Add Day-0 Support for Qwen3.8
  24. SGLang Adds Day-0 Support for NVIDIA Nemotron 3.5 Lightning
  25. Unified Radix Cache: One Tree for Hybrid Model Prefix Caching
  26. SGLang Adds Day-0 Support for Muse Glimmer, a Multimodal Model Built for Local Agentic Workflows
  27. HPC-Ops × SGLang: High-Performance Attention, Router GEMM, and MoE Kernels from Tencent Hunyuan
  28. Full-Stack Performance Optimization of AR+DiT in SGL-Diffusion
  29. SpecForge v0.3.0: a Unified Disaggregated and Colocated Speculative Decoding Stack, and New Open SpecBundle Draft Models
  30. RadixArk Joins Forces with Google to Bring Full SGLang Features to TPUs
  31. Towards Blackwell-Native 8-bit and 4-bit RL: End-to-End MXFP8 and NVFP4 RL in Miles
  32. Toward a Cleaner Quantization Stack in SGLang
  33. SGLang and Miles Add Day-0 Support for Kimi K3
  34. OPD Support in Miles
  35. SGLang and Miles Add Day-0 Support for Inkling, a Frontier Multimodal Model
  36. Serving GLM5.2 NVFP4 Agentic Workload with SGLang: Reaching 500 TPS in 2 Weeks
  37. Bringing DeepSeek-V4 Flash RL Training to AMD Instinct MI355X GPUs with Miles
  38. Accelerating SGLang HiCache with Netpreme X-Mem™ MPU
  39. DSpark in SGLang: Speculative Decoding with Confidence-Driven, Variable-Length Verification
  40. Agent-Assisted SGLang Development: An Initial Exploration
  41. Improving DeepEP MoE Load Balance in SGLang with Waterfill and LPLB
  42. Optimizing Ling-2.6-1T on TPU with SGLang-JAX: Hiding MoE Data Movement Behind Compute with One Pallas Kernel
  43. MOSS-TTS Local Transformer v1.5 on SGLang-Omni: Serving Native-Streaming 48 kHz Speech
  44. The next generation of speculative decoding: DFlash and Spec V2
  45. Announcing the Recipient of the 2026 LMSYS PhD Fellowship
  46. No Token Left Behind: Demystifying Token-In-Token-Out in Miles
  47. Higgs Audio v3 TTS on SGLang-Omni: Real-Time, Controllable Speech for Voice Agents
  48. SGLang and Miles Add Day-0 Support for NVIDIA Nemotron 3 Ultra for Long-Running Autonomous Agents
  49. Heterogeneous CPU + GPU EPD Disaggregation to Boost VLM Serving
  50. Win on TCO: How AMD Instinct™ MI355X Achieves Cost-Competitive Distributed Inference Through SGLang with MoRI
  51. Updating 1T parameters in seconds — P2P weight transfer in Large Scale Distributed RL
  52. DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles
  53. HiSparse: Turbocharging Sparse Attention with Hierarchical Memory
  54. Highlights of SGLang at NVIDIA GTC 2026
  55. Elastic EP in SGLang: Achieving Partial Failure Tolerance for DeepSeek MoE Deployments
  56. ROCm Support for Miles: Large-Scale RL Post-Training on AMD Instinct™ GPUs
  57. SGLang Adds Day-0 Support for NVIDIA Nemotron 3 Super for building High-Efficiency Multi-Agent Systems
  58. Unlocking 25x Inference Performance with SGLang on NVIDIA GB300 NVL72
  59. Deploying DeepSeek on GB300 NVL72: Big Wins in Long-Context Inference
  60. SGLang-Diffusion: Advanced Optimizations for Production-Ready Video Generation
  61. Unleashing Computational Power: Ultimate Latency Optimization of Qwen3 and Qwen3-VL on AMD MI300X Series
  62. Squeezing 1TB Model Rollout into a Single H200: INT4 QAT RL End-to-End Practice
  63. Optimizing GLM4-MoE for Production: 65% Faster TTFT with SGLang
  64. SGLang-Diffusion: Two Months In
  65. Pipeline Parallelism in SGLang: Scaling to Million-Token Contexts and Beyond
  66. EPD Disaggregation: Elastic Encoder Scaling for Vision-Language Models in SGLang

2025 39

  1. SpecBundle & SpecForge v0.2: Production-Ready Speculative Decoding Models and Framework
  2. Power Up Diffusion LLMs: Day‑0 Support for LLaDA 2.0
  3. Mini-SGLang: Efficient Inference Engine in a Nutshell
  4. SGLang Day-0 Support for MiMo-V2-Flash Model
  5. SGLang Adds Day-0 Support for the Highly Efficient, Open Nemotron 3 Nano Hybrid MoE Model
  6. Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork
  7. Boost SGLang Inference: Native NVIDIA Model Optimizer Integration for Seamless Quantization and Deployment
  8. From research to production: Accelerate OSS LLM with EAGLE-3 on Vertex
  9. Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL
  10. LMSYS Fellowship Program
  11. Introducing Miles — RL Framework To Fire Up Large-Scale MoE Training
  12. 🚀 AutoRound Meets SGLang: Enabling Quantized Model Inference with AutoRound
  13. SGLang Diffusion: Accelerating Video and Image Generation
  14. "No Free Lunch": Deconstruct Efficient Attention with MiniMax M2
  15. Optimizing GPT-OSS on NVIDIA DGX Spark: Getting the Most Out of Your Spark
  16. SGLang-Jax: An Open-Source Solution for Native TPU Inference
  17. Accelerating Hybrid Inference in SGLang with KTransformers CPU Kernels
  18. SGLang and NVIDIA Accelerating SemiAnalysis InferenceMAX and GB200 Together
  19. NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
  20. SGLang Day 0 Support for DeepSeek-V3.2 with Sparse Attention
  21. PD-Multiplexing: Unlocking High-Goodput LLM Serving with GreenContext
  22. Together with SGLang: Best Practices for Serving DeepSeek-R1 on H20-96G
  23. Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II): 3.8x Prefill, 4.8x Decode Throughput
  24. Towards Deterministic Inference in SGLang and Reproducible RL Training
  25. Optimizing FP4 Mixed-Precision Inference on AMD GPUs
  26. SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends
  27. LongCat-Flash: Deploying Meituan's Agentic Model with SGLang
  28. Fine-tune and deploy gpt-oss MXFP4: ModelOpt + SGLang
  29. SGLang for gpt-oss: From Day 0 Support to Enhanced Performance
  30. GLM-4.5 Meets SGLang: Reasoning, Coding, and Agentic Abilities
  31. SpecForge: Accelerating Speculative Decoding Training for SGLang
  32. Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs
  33. Accelerating SGLang with Multiple Token Prediction
  34. How to support new VLMs into SGLang: A Case Study with NVILA
  35. Cost Effective Deployment of DeepSeek R1 with Intel® Xeon® 6 CPU on SGLang
  36. slime: An SGLang-Native Post-Training Framework for RL Scaling
  37. OME: Revolutionizing LLM Infrastructure with Model-Driven Architecture
  38. Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I): 2.7x Higher Decoding Throughput
  39. Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs

2024 15

  1. SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs
  2. Announcing a New Site for Chatbot Arena
  3. RedTeam Arena: An Open-Source, Community-driven Jailbreaking Platform
  4. SGLang v0.3 Release: 7x Faster DeepSeek MLA, 1.5x Faster torch.compile, Multi-Image/Video LLaVA-OneVision
  5. Does style matter? Disentangling style and substance in Chatbot Arena
  6. Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM)
  7. RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing
  8. The Multimodal Arena is Here!
  9. Introducing Hard Prompts Category in Chatbot Arena
  10. What’s up with Llama 3? Arena data analysis
  11. LMSYS Kaggle Competition – Predicting Human Preference with $100,000 in Prizes
  12. From Live Data to High-Quality Benchmarks: The Arena-Hard Pipeline
  13. LMSYS Chatbot Arena: Live and Community-Driven LLM Evaluation
  14. Fast JSON Decoding for Local LLMs with Compressed Finite State Machine
  15. Fast and Expressive LLM Inference with RadixAttention and SGLang

2023 13

  1. Chatbot Arena: New models & Elo system update
  2. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  3. Recipe for Serving Thousands of Concurrent LoRA Adapters
  4. Catch me if you can! How to beat GPT-4 with a 13B model
  5. ToxicChat: A Benchmark for Content Moderation in Real-world User-AI Interactions
  6. Chatbot Arena Conversation Dataset Release
  7. How Long Can Open-Source LLMs Truly Promise on Context Length?
  8. Chatbot Arena Leaderboard Week 8: Introducing MT-Bench and Vicuna-33B
  9. Building a Truly "Open" OpenAI API Server with Open Models Locally
  10. Chatbot Arena Leaderboard Updates (Week 4)
  11. Chatbot Arena Leaderboard Updates (Week 2)
  12. Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings
  13. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality