TensorRT is NVIDIA's inference optimization toolkit that converts trained deep learning models into highly efficient runtime engines for deployment on NVIDIA GPUs. It applies a series of graph-level and kernel-level optimizations, including layer fusion, precision calibration, kernel auto-tuning, and dynamic tensor memory management, to minimize latency and maximize throughput at inference time.

The optimization process starts with an exported model in ONNX or framework-native format. TensorRT analyzes the computation graph, fuses compatible layers (for example, combining convolution, batch normalization, and activation into a single kernel), selects the fastest kernel implementation for the target GPU, and optionally converts weights from FP32 to FP16 or INT8. INT8 quantization requires a calibration dataset to map the dynamic range of activations, but it can deliver 2-4x speedup over FP32 with minimal accuracy loss on most vision models.

In production computer vision systems, TensorRT is the standard path for deploying models on NVIDIA hardware, from data center GPUs like the A100 down to edge devices like the Jetson Orin. Real-time applications like video analytics, autonomous driving, and industrial inspection rely on TensorRT to hit strict latency budgets. The trade-off is platform lock-in: TensorRT engines only run on NVIDIA GPUs and must be rebuilt when switching GPU architectures.