What does TensorRT model optimization consulting include?
The goal is a measured answer to one question: what is the fastest way to run your trained model on your hardware without changing its decisions? I profile the whole request first, because the network is not always the slow part. Then I build the candidate variants, typically TensorRT engines in FP16 and INT8, plus CPU INT8 if you serve on CPUs, and time each one against the original on the same device and the same test images.
Each variant replaces only the network forward pass. Preprocessing, decoding and postprocessing stay the same code, so a speed difference comes from the optimization and not from a pipeline change. I report medians, record what else was running on the host, and count the detections that change at your operating threshold.
| Stage | What we check or deliver |
|---|---|
| Profile | Time per request, split into preprocessing, network and postprocessing |
| Variants | TensorRT FP16 and INT8 engines, plus CPU INT8 where you serve on CPUs |
| Measurement | Median latency per variant after warm-up, at your batch sizes, on your hardware |
| Accuracy | Score and changed decisions against the original model on the same test images |
| Deployment | The chosen variant served through Triton or your own runtime, with its build script |
What did TensorRT FP16 and INT8 do on a real detector?
I measured this on RF-DETR, the detector I maintain a training and serving engine for. Conditions: an NVIDIA RTX 5090, TensorRT 10.12, PyTorch 2.9.0 with CUDA 12.8, the median of 100 timed calls after 10 warm-up calls at batch 1, measured on 12 October 2026. The INT8 engines used an entropy calibrator over 64 training images, with FP16 allowed as the fallback.
| Model and input size | TensorRT FP16 vs PyTorch FP32 | Layers TensorRT ran in INT8 |
|---|---|---|
| RF-DETR Nano detect, 384 px | 5.3x faster | 0 of 193 |
| RF-DETR Seg Nano, 312 px | 5.7x faster | 0 of 259 |
| RF-DETR Medium detect, 576 px | 4.8x faster | 0 of 225 |
FP16 moved mAP50:95 by 0.002 or less. The INT8 build was the FP16 engine under another name: TensorRT timed the kernels for every layer and kept FP16 each time. Forcing INT8 with explicit Q/DQ nodes put 166 layers in INT8, ran 17% slower than FP16 and cut mAP50:95 by more than three quarters. NVIDIA's TensorRT 10.2 release notes list INT8 as slower than FP16 for some Transformer models, DETR among them.
Outside that test, I have made production inference around 70% faster on the same GPUs with TensorRT, ONNX and Triton.
Is TensorRT INT8 worth it for YOLO or DETR models?
Sometimes, and the engine itself tells you whether it happened. An engine that builds with the INT8 flag is not proof that any layer runs in INT8, so I read each layer's precision from TensorRT's engine inspector before trusting a benchmark. For DETR models, my numbers and NVIDIA's notes point the same way: FP16 is the answer on this GPU. For a YOLO model I would run the same test rather than assume either result, since I haven't published YOLO INT8 measurements.
Should I use ONNX Runtime or TensorRT?
ONNX is a file format for a trained network, and ONNX Runtime and TensorRT are two ways to run it. Which one is faster depends on the model and the hardware, so I time both where both apply. On a 12-vCPU virtual machine with AVX2 only (no VNNI, AVX-512 or AMX), ONNX Runtime's FP32 path was no faster than PyTorch, and its static INT8 was 9% to 36% slower than PyTorch FP32. On that CPU only PyTorch dynamic INT8 sped up the network, by 1.3 to 2 times. On the RTX 5090 I compared TensorRT with PyTorch, not with ONNX Runtime's GPU path, so I don't rank those two for GPUs here.
Why is my model still slow after TensorRT?
Often because the network was never the bottleneck. In the RF-DETR pipeline the test frames were 16 megapixels, and the TensorRT network took between 1% and 5% of each request on the GPU host; preprocessing took most of each request. Shrinking the network further would change nothing a user of the line notices.
When the model itself must be faster, test a lower input resolution first. A distilled RF-DETR Nano student ran 2.5 times faster than its Medium teacher on CPU and 1.3 times faster on GPU, but nearly all of that came from 384 px input against 576 px, since the two models have 29.5M and 32.7M parameters.
What do I need to scope an inference optimization project?
Send the model or its export, the hardware you serve on, your latency or throughput target and batch size, and a set of test inputs with the outputs you expect. Your current serving code and its timings help too, because they show where the time goes today. If the model does visual inspection, the industrial computer vision page covers training and evaluation.
Jetson devices, CPUs with VNNI or AMX, and quantization-aware training for TensorRT 11 are not in my published measurements yet; on those I measure on your device before promising a number. There is no useful fixed price before seeing the model, the hardware and the target, and the quote covers measurement and deployment testing as well as the engine build.
Frequently asked questions
How much does TensorRT optimization cost?
I quote after seeing the model, the target hardware and the latency or throughput you need. Measuring every variant and testing the deployment are part of the work, so a model name alone is not enough for a useful estimate.
How much faster will TensorRT make my model?
It depends on the model, the GPU and how much of each request the network takes. On an RTX 5090, TensorRT FP16 ran three RF-DETR models 4.8 to 5.7 times faster than PyTorch FP32 at batch 1. If preprocessing dominates the request, the end-to-end gain is much smaller, which is why I profile first.
Is TensorRT INT8 always faster than FP16?
No. For RF-DETR on an RTX 5090, a calibrated INT8 build put zero layers in INT8, and forcing INT8 with Q/DQ nodes ran 17% slower than FP16. NVIDIA's TensorRT 10.2 release notes list INT8 as slower than FP16 for some Transformer models, DETR among them.
Will optimization change my model's accuracy?
It can, so I check. TensorRT FP16 moved RF-DETR's mAP50:95 by 0.002 or less, while PyTorch dynamic INT8 on a CPU flipped 1 to 4 detection decisions at a 0.5 threshold on a 10-image test set. I compare changed decisions at your operating threshold as well as the overall score.
Can you optimize models for CPU or edge devices?
On an AVX2-only CPU, PyTorch dynamic INT8 sped up the RF-DETR network 1.3 to 2 times, while ONNX Runtime static INT8 was slower than plain PyTorch. I haven't published measurements for Jetson devices or for CPUs with VNNI or AMX, so on those I measure on your device before quoting a number.
How long does an inference optimization project take?
It depends on how many variants are worth building and how the model is served today. I profile the current pipeline, build and time the candidates, check accuracy and test the chosen variant in its runtime. I can estimate the schedule after seeing the model and the hardware.