Blog

Inference engineering

All Model performance AI engineering Infrastructure News Community AI models Product Foundations Research

Model performance

How to double tokens per second for Llama 3 with Medusa

Abu Qader

Philip Kiely

Abu Qader

1 other

Double Llama TPS with Medusa

Model performance

How to serve 10,000 fine-tuned LLMs from a single GPU

Pankaj Gupta

Philip Kiely

Pankaj Gupta

1 other

10,000 LoRAs 1 GPU

Model performance

Comparing few-step image generation models

Rachel Rapp

Comparing few-step image generation models

Model performance

How latent consistency models work

Rachel Rapp

How LCMs work

Model performance

Comparing tokens per second across LLMs

Philip Kiely

Philip Kiely

Comparing TPS across LLMs

Model performance

Continuous vs dynamic batching for AI inference

Matt Howard

Philip Kiely

Matt Howard

1 other

Continuous vs Dynamic batching

Model performance

Benchmarking fast Mistral 7B inference

Abu Qader

Pankaj Gupta

Philip Kiely

Abu Qader

3 others

Mistral 7B

Model performance

33% faster LLM inference with FP8 quantization

Pankaj Gupta

Philip Kiely

Pankaj Gupta

1 other

Faster inference with FP8

Model performance

High performance ML inference with NVIDIA TensorRT

Philip Kiely

Justin Yi

1 other

NVIDIA TensorRT

1...5 6 7 8