vLLM tuning: from 800 to 4,000 tok/s on the same L40S

Same GPU, 5× throughput. The levers, in order of impact:

1. Continuous batching + prefix caching (2.1×)

Enable --enable-prefix-caching --enable-chunked-prefill. System prompts and RAG contexts reuse KV-cache.

2. Quantization (1.6×)

AWQ / FP8 for 70B → fits 2×L40S with TP=2, 2.8× lower $/token for <2% quality drop on business QA.

3. Speculative decoding (1.4×)

8B draft → 70B verify. Best for chatty, low-entropy workloads.

Combined on our inference factory: 4,020 tok/s, p95 3.1s, $0.28/1M tokens.

Run it on our L40S cloud →