Tensorrt-Llm
Tensorrt-Llm
- vLLM vs TensorRT-LLM: First Throughput Numbers on Llama-3 70B
Quick benchmark comparing vLLM and TensorRT-LLM throughput on a single A100 80GB with Llama-3 70B. Setup notes, raw numbers, and what I'd investigate next.
Tensorrt-Llm
Quick benchmark comparing vLLM and TensorRT-LLM throughput on a single A100 80GB with Llama-3 70B. Setup notes, raw numbers, and what I'd investigate next.