Model Inference Optimization Engineer

Inference optimization is the discipline of making AI models run faster, cheaper, and more efficiently in production without meaningfully degrading their output quality. As models grow larger and usage scales up, the gap between a naively deployed model and a properly optimized one can translate into seconds of latency, orders-of-magnitude differences in cost, and entirely different hardware requirements. This AI assistant helps ML engineers, platform teams, and AI infrastructure leads close that gap systematically.

The assistant covers the full optimization toolkit. It explains and guides implementation of post-training quantization techniques — from the relatively simple INT8 dynamic quantization to more aggressive methods like GPTQ, AWQ, and GGUF for LLMs — and helps you understand when each is appropriate based on your accuracy tolerance and target hardware. It also covers knowledge distillation strategies for creating smaller, faster student models when the full model is overkill for a given task.

On the serving side, the assistant dives into continuous batching, speculative decoding, flash attention, and KV-cache optimization — techniques that can dramatically improve throughput on GPU hardware. It helps you profile model inference using tools like NVIDIA Nsight, PyTorch Profiler, and custom latency benchmarking scripts, so you can identify and fix specific bottlenecks rather than applying optimizations blindly.

The assistant also covers hardware-aware optimization: selecting between CUDA, ROCm, and CPU inference backends, using ONNX Runtime or TensorRT for optimized execution graphs, and configuring model parallelism strategies for multi-GPU or multi-node setups.

Ideal users include ML engineers who have a working model but need to hit a latency SLA, platform engineers reducing cloud GPU costs at scale, and AI teams preparing for high-traffic product launches. The assistant helps you benchmark before and after each optimization so you can demonstrate concrete improvements.