This project's scope involves a set of production-related modules around vLLM, including router, autoscaling, observability, KV cache offloading, and framework supports (KServe, Ray, etc).

This document will include the items on our Q1 roadmap. We will keep updating this document to include the related issues, pull requests, and discussions in the #production-stack channel in the vLLM slack.

Core features

• (P0) Prefix-cache-aware routing algorithm (Is prefix-aware routing implemented in this repo? #19, [RFC] prefix-cache-aware routing #59 )
• (P1) Offline batched inference based on OpenAI offline batching API
  • Part 1: file storage support (feat: Offline batched inference based on OpenAI offline batching API #47 , feat: OpenAI batch API part 1 #52 )
  • Part 2: batched inference API support ([Router] Support Batch API part 2 #109 )
• (P1) Router observability (Current QPS, router-side queueing delay, number of pending / prefilling / decoding requests, average prefill / decoding length, etc) (Feat: Router observability (Current QPS, router-side queueing delay, etc) #78, Feat: Router observability (Current QPS, router-side queueing delay, etc) Part 1 #119 )
• (P1) Autoscaling support
  • Basic autoscaling support based on Prometheus stack ([Document, Feat] basic HPA support and tutorials #209)
  • CRD-based autoscaling
• (P2) Experimental support for disaggregated prefill
• (P2) Support vLLM v1
• (P2) Transcode the router using a more performant language (e.g., Rust and Go) for better QPS/throughput and lower delay

CI/CD and packaging

• (P0) Add unit test to the repo (chore: Unit test in github action #24)
• (P0) Add end-to-end test of the deployment ([CI/CD] Functionality test for helm chart #30)
• (P0) Automatically release the helm charts and the router docker images (feat: Add Github workflows to lint and release Helm chart #23, [CI/CD] Automatically build the router docker image via github action #74 )
• (P1) Package the router into a separate python package vllm-router (chore: Make router a python package #17)

OSS-related supports

• (P0) format checker for the code (Add pre-commit based linting and formatting #35)
• (P2) Issue and PR templates and labels ([Doc] Add PR template #93)

If any of the items you wanted are not on the roadmap, your suggestion and contribution are strongly welcomed! Please feel free to comment in this thread, open a feature request, or create an RFC.

Happy vLLMing!