Skip to content

Export RF-DETR Model

Key Takeaways

  • Export to ONNX for cross-platform inference with ONNX Runtime, OpenVINO, or TensorRT
  • Export to OpenVINO IR for optimized inference on CPU (x86, ARM), GPU (Intel integrated & discrete GPU) and AI accelerators (Intel NPU)
  • Export to TFLite (FP32, FP16, INT8) for mobile and edge deployment
  • Export to LiteRT (.tflite) straight from PyTorch with litert-torch — no ONNX or TensorFlow step
  • TensorRT conversion delivers the lowest latency on NVIDIA GPUs — 2.3 ms for Nano on a T4, TensorRT FP16, model only, batch 1 (the architecture headline on the Benchmarks page; the end-to-end numbers below are a different measurement, see "Which latency number is this?")
  • INT8 quantization: dynamic-range for TFLite (no calibration data), static for ONNX, OpenVINO and TensorRT (calibration data required; see TensorRT INT8)
  • Custom input resolutions supported (must be divisible by patch_size × num_windows, which varies by model variant)
  • Export to ExecuTorch for on-device PyTorch inference (XNNPACK, CoreML, QNN)
  • Export directly to native CoreML (.mlpackage) for Xcode / Apple-platform deployment
  • Adding a format is an in-tree contribution — see Exporter Blueprint
  • Per-format details are in the format guides below

RF-DETR supports exporting models to ONNX, TFLite, LiteRT, ExecuTorch, native CoreML, Apple Core AI and OpenVINO IR formats, enabling deployment across a wide range of inference frameworks, edge devices, and hardware accelerators.

The export docs are split into three pages:

  • Overview (this page) — supported formats and measured performance per hardware.
  • Export Basics — installation, a first model.export() call, output files, and prediction with RFDETRInference.
  • Advanced Export — the full parameter reference, custom resolution, backbone-only export, and how the export pipeline works.

For detailed installation, examples, and inference code per format, see the format guides:

  • ONNX Inference — run an exported ONNX model with ONNX Runtime.
  • TensorRT — build a .trt engine for NVIDIA GPUs, directly or from an existing ONNX file.
  • TFLite — ONNX → TensorFlow → TFLite conversion for mobile and edge devices.
  • LiteRT — PyTorch → .tflite via litert-torch, no ONNX or TensorFlow step.
  • OpenVINO — OpenVINO IR for Intel CPUs, GPUs, and NPUs.
  • ExecuTorch — .pte binaries for XNNPACK, CoreML, and QNN backends.
  • Native CoreML — .mlpackage export for Xcode / Apple platforms.
  • Core AI — .aimodel export for iOS / iPadOS / macOS 27+.

Measured Performance by Hardware

Which format is fastest depends on the hardware you deploy to. Four per-hardware cookbooks export every format for one class of device, run inference through RFDETRInference, and benchmark it against the PyTorch model on the same machine. This page keeps only numbers measured through RFDETRInference (and the model.inference() snapshot it builds); each format page holds its own per-configuration table.

All numbers are batch 1 unless stated, RFDETRSmall, rfdetr develop, end to end (preprocessing, forward, postprocessing). Compare within a row group, never across machines.

Hardware Fastest route Precision end2end [ms] FPS [img/s] mAP@50:95 [%] Full table
NVIDIA L4 TensorRT .trt engine int8 3.45 290 — TensorRT
Apple M4 Max (40-core GPU) inference() inductor fp16 10.3 97.4 56.4 Baseline, CoreML
x86 CPU (Colab, 4 cores / 8 threads) OpenVINO IR fp32 373.65 ± 61.10 2.7 56.5 OpenVINO
  • Warmup and timed runs differ per cookbook (GPU 20 + 100, CPU 15 + 50, Apple 5 + 30). The rows come from a Colab L4 host, a Colab x86 host and one Apple M4 Max (macOS 26.7.1).
  • On the x86 host the run-to-run standard deviation is 10 to 17% of the mean, so CPU batch-1 gaps under about 100 ms are noise: OpenVINO fp32 is the best full-accuracy route listed, but eager PyTorch and TorchScript are level with it.
  • No mobile or edge CPU row is listed: the TFLite, LiteRT and ExecuTorch tables are pending a rerun with a pinned thread count.

Fastest route across NVIDIA GPUs

Fastest measured route per GPU and size, with the winning route in brackets. Latency only, TensorRT 10.16.1.11. All routes are in TensorRT.

Lowest latency, batch 1:

GPU Nano Small Medium Large
T4 4.90 ms (I) 5.89 ms (M) 7.02 ms (I) 10.22 ms (M)
L4 2.78 ms (C) 3.45 ms (I) 4.10 ms (I) 4.72 ms (I)
A100 3.16 ms (C) 3.44 ms (F) 3.80 ms (C) 4.27 ms (C)
RTX PRO 6000 1.37 ms (I) 1.57 ms (I) 1.73 ms (I) 1.92 ms (I)

Routes (the letter is how the fastest-route tables name a route):

  • C, inference() graph, FP16: model.inference() with fp16 and a captured CUDA graph.
  • F, .trt engine, FP16: a static engine from RFDETR.export(format="tensorrt"), loaded by RFDETRInference.
  • I, .trt engine, INT8: the same export with quantization="int8" and calibration images.
  • M, managed engine: the TensorRT package that inference-models selects itself, so its precision is not set here. no engine means inference-models 0.39.8 selected none (NoModelPackagesAvailableError).
  • FP32: a .trt engine with fp16=False. It has no letter and is only in the per-GPU tabs, because it is never the fastest.

Highest throughput, batch 8 (the winning route can differ, because a batch of 8 hides per-call overhead):

GPU Nano Small Medium Large
T4 386 img/s (I) 223 img/s (I) 165 img/s (I) 99 img/s (M)
L4 809 img/s (I) 516 img/s (I) 406 img/s (I) 272 img/s (I)
A100 967 img/s (F) 783 img/s (F) 676 img/s (F) 563 img/s (I)
RTX PRO 6000 2563 img/s (I) 1916 img/s (I) 1694 img/s (I) 1362 img/s (I)

Routes (the letter is how the fastest-route tables name a route):

  • C, inference() graph, FP16: model.inference() with fp16 and a captured CUDA graph.
  • F, .trt engine, FP16: a static engine from RFDETR.export(format="tensorrt"), loaded by RFDETRInference.
  • I, .trt engine, INT8: the same export with quantization="int8" and calibration images.
  • M, managed engine: the TensorRT package that inference-models selects itself, so its precision is not set here. no engine means inference-models 0.39.8 selected none (NoModelPackagesAvailableError).
  • FP32: a .trt engine with fp16=False. It has no letter and is only in the per-GPU tabs, because it is never the fastest.

PyTorch Baseline

The PyTorch rows every cookbook measures, per machine. predict() is the eager model; inference() is the snapshot it returns (TorchScript, plus a CUDA graph on CUDA and inductor on the Mac GPU). FPS is images per second at each batch size, — where that batch was not run. mAP@50:95 is a seeded 500-image val2017 subset, scored at batch 1 on the artifact that was timed.

RFDETRSmall, TensorRT 10.16.1.11, one decoded RGB array per call, latency only (no accuracy scored).

Runtime Precision b1 [ms] b4 [ms] b8 [ms] FPS b1 FPS b4 FPS b8
inference() CUDA Graph fp16 3.60 11.42 24.72 278 350 324
.trt engine fp16 3.57 9.36 18.53 280 427 432
  • inference() with fp16 and a CUDA graph is level with the .trt engine at batch 1, slower at batch 4 and 8, and keeps the native predict() API.
Runtime Precision b1 [ms] mAP@50:95 [%] FPS b1 FPS b4 FPS b8
predict() eager fp32 20.0 56.5 50.0 83.6 93.0
inference() TorchScript fp16 12.2 56.5 81.9 131.3 143.5
inference() inductor fp32 11.1 56.5 90.4 101.2 120.8
inference() inductor fp16 10.3 56.4 97.4 127.0 141.6
  • These rows come from a busy machine (load average above 10): treat gaps under about 15% as noise.
Runtime Precision b1 [ms] mAP@50:95 [%] FPS b1 FPS b4
predict() eager fp32 401.58 ± 66.19 56.5 2.5 2.8
inference() JIT fp32 377.85 ± 62.10 56.5 2.6 2.8

Memory [MB] in the cookbooks is comparable only within one cookbook: the CUDA cookbook measures device memory, the others host resident memory (RSS). The subset is chosen for run time, so these are not the full-val2017 numbers on the Benchmarks page; set FULL_VAL = True in a cookbook to score all 5000 images.

Which latency number is this?

RF-DETR latency appears on three pages, and they are three different measurements:

  • Architecture headline (T4, TensorRT FP16, model only, batch 1): compares architectures. See the Benchmarks page.
  • Deployed product (L4, Roboflow Inference, scope not stated, batch 1): see Roboflow's RF-DETR model page.
  • Export cookbooks (this page and below): end-to-end timings, including host preprocessing and postprocessing, on one machine per cookbook. Their value is ratios within one machine. The INT8 figures on the TensorRT page are engine-only GPU time, a fourth scope.

Never compare milliseconds across these without checking hardware, precision, batch size and scope. The two harnesses have not been run side by side, so treat them as unreconciled.

fp16 pays off only where the silicon implements it

  • GPU: large win. TensorRT fp16 engines and PyTorch inference(dtype=torch.float16) on CUDA are the fast routes; see the TensorRT latency tables.
  • CPU: no measured win. A CPU without native fp16 kernels computes in fp32, so fp16 saves disk and bandwidth, not arithmetic (OpenVINO fp16 452.3 ms against 373.7 ms fp32 on the Colab host, at half the size). INT8 changes CPU arithmetic but not consistently: OpenVINO INT8 was slower than its fp32 IR, and the ONNX x86 INT8 comparison is pending a rerun. See OpenVINO and ONNX. The TFLite rows are pending a rerun (they ran at the interpreter's default thread count).
  • Apple: runtime-dependent. On the M4 Max, CoreML fp16 was slower than its fp32 default (23.6 against 12.0 ms) at half the size, and ExecuTorch's CoreML backend (fp16 only) matched CoreML fp16. An earlier macOS 27.0 run had Core AI fp16 faster than fp32 (13.53 against 14.62 ms); Core AI runs on the CPU only on older macOS. Measure per runtime. See CoreML and ExecuTorch.

Per-Format Guides

The format guides listed above cover installation, export examples, output files, and inference code.

Next Steps

After exporting your model, you may want to:

  • Deploy to Roboflow for cloud-based inference and workflow integration
  • Deploy TFLite and LiteRT .tflite models on mobile/edge devices with the LiteRT runtime
  • Deploy ExecuTorch .pte models on mobile/edge devices with the ExecuTorch runtime
  • Integrate with edge deployment frameworks like ONNX Runtime or OpenVINO
  • Read the Exporter Blueprint to add a new export format
  • Browse the format guides above for per-format details