TensorRT INT8 vs FP16 on RF-DETR: why INT8 and distillation didn't pay off

I maintain a training and serving engine for RF-DETR defect-detection models. When a model has to meet a cycle time on a given device, the first question is whether it can run faster, so I added a job that builds every faster variant of a trained model (TensorRT FP16 and INT8, CPU INT8 in two runtimes, and a distilled smaller student) and measures each one against the original on the same hardware and the same test images.

Most of the variants that are supposed to be faster were not. These are the measurements, the reasons, and what I recommend instead. The models and their training data stay private; the setup, the method and the ratios below carry over to any RF-DETR model.

How the INT8, FP16 and distillation variants were measured

Measured on 12 October 2026:

Each variant replaces only the network forward. rfdetr calls model.model.inference_model when a model is marked as optimized, so preprocessing, the decoder and the postprocessing are the same code for every variant, and any difference comes from the forward alone. The Nano measurements were repeated three times: forward medians stayed within about 6% on the GPU and 10% on the CPU, and accuracy was identical every time.

The test set is small, so read the accuracy changes as a direction, not a precise figure. The speed numbers are solid.

Why is TensorRT INT8 not faster than FP16 for RF-DETR?

Because TensorRT didn't use INT8. I built the INT8 engine the standard way: INT8 flag, an entropy calibrator over 64 training images, FP16 allowed as the fallback. Then I counted the precision of every layer in the finished engine:

With INT8 allowed, TensorRT times the candidate kernels for each layer and keeps the fastest. It kept FP16 every time, so the "INT8" engine was the FP16 engine with a different name. FP16 cost almost nothing in accuracy: mAP50:95 moved by 0.002 or less, and one decision changed on the segmentation model.

This is a known pattern for DETR models, not an RF-DETR quirk. NVIDIA's TensorRT 10.2 release notes list it as a known issue: "For some Transformer models, including ViT, Swin-Transformer, and DETR, there is a performance drop in INT8 precision (including both explicit and implicit quantization) compared to FP16 precision."

Forcing the issue didn't help. With explicit Q/DQ nodes in the ONNX graph, TensorRT put 166 layers in INT8. That engine was 17% slower than FP16, and mAP50:95 dropped by more than three quarters. I left it out of the job.

Check the precision TensorRT actually chose

An engine that builds with the INT8 flag is not proof that anything runs in INT8. Build with detailed profiling verbosity, then ask the engine inspector:

import json
import tensorrt as trt
 
# at build time: config.profiling_verbosity = trt.ProfilingVerbosity.DETAILED
inspector = engine.create_engine_inspector()
info = json.loads(inspector.get_engine_information(trt.LayerInformationFormat.JSON))
for layer in info["Layers"]:
    print(layer["Name"], layer["Outputs"][0]["Format/Datatype"])

Without detailed verbosity the inspector returns layer names only, which looks like an empty result rather than an error.

Does the network forward even matter on the GPU?

Barely, in this pipeline. The test frames are 16 megapixels, and preprocessing them for the model took most of each request: the TensorRT forward was between 1% and 5% of a request's time on the GPU host, decoding not even counted. Shrinking the network further would change nothing a user of the line notices. Profile the whole request before optimizing the model.

Does INT8 speed up RF-DETR on a CPU?

One method did, with a cost. I tried two:

PyTorch dynamic INT8 (torch.ao.quantization.quantize_dynamic on every nn.Linear) is a load-time transform with nothing extra to ship. It sped up the network forward by 1.3 to 2.0 times and the whole request by 1.2 to 1.3 times. Accuracy moved only slightly, between −0.014 and +0.009 mAP50:95, but 1 to 4 detections per model crossed the 0.5 threshold one way or the other on 10 images. In inspection, a changed decision is a missed defect or a false reject, so that matters more than the mAP change.

ONNX Runtime static INT8 (per-channel weights, MinMax calibration) was slower than plain PyTorch on this CPU, and its FP32 path was no faster than PyTorch either. It broke the segmentation model whatever the calibration. Percentile calibration helped the detection model (−0.011 instead of −0.026 mAP50:95 on the same images) but needed 30 GB of memory for the Medium model, because ONNX Runtime's histogram calibrators keep every activation of every calibration image in memory.

This CPU has no VNNI, AVX-512 or AMX. INT8 kernels depend on those instructions, so a server CPU that has them could give a different answer; I haven't measured one.

Does distilling RF-DETR Medium into Nano help?

Not with clean labels. The distillation was the simple, hard-label kind. The Medium model predicted on the training images. Every detection at 0.5 confidence or higher that no existing label of the same class already covered (IoU 0.5 or more) became an extra training box. Then two Nano students trained with the same configuration and the same three seeds: one on the labels alone, one on the labels plus the teacher's boxes. Each ran 20 epochs, with no early stopping and no augmentation, and both were scored on the untouched test set.

The student trained with pseudo-labels scored lower on every seed, by 0.025 mAP50:95 on average, and varied more between seeds (standard deviation 0.022 against 0.010). The reason is in what the teacher added. On a fully labelled training set it mostly re-found the existing labels, and it added two boxes across the whole set, both false positives on its weakest class. This kind of distillation can only add value through objects the annotators missed. It is worth trying on incompletely labelled data, and it has nothing to offer on clean labels.

The student was faster: 2.5 times on the CPU and 1.3 times on the GPU. But the Nano and Medium variants have 29.5M and 32.7M parameters, so the speed came from the input resolution (384 px against 576 px), not from a smaller network. If speed is what you need, retrain at a lower resolution and measure that first.

Feature- and logit-based distillation, which train the student on the teacher's internal outputs, weren't part of this test.

How to compare model compression methods fairly

Most of what I learned is about measuring, and it applies to any detector:

• Swap only the forward. Run the same preprocessing, decoder and postprocessing for every variant, or you measure your pipeline changes rather than the compression.
• Report medians. On a shared or virtual host, one stalled call in a hundred can triple a mean.
• Record the host load. One run next to a training job measured a CPU forward more than 50 times slower than its real speed.
• Calibrate on training images, never on the test set. Calibrating on the images you score flatters INT8.
• Count changed decisions at your operating threshold. A small mAP change can hide detections that flip between found and missed.
• Repeat anything that involves training, with three or more paired seeds, before reading a difference.
• Check which precision actually ran, with the inspector above.

What should you use for RF-DETR?

This covers one GPU, one CPU class and three RF-DETR variants. Jetson devices, CPUs with VNNI or AMX, and quantization-aware training for TensorRT 11's explicit INT8 are open questions here. They are the next things I would measure for a customer who needs one of them.