RF-DETR on NVIDIA GPUs — Export, Inference & Latency¶
Compares every export path that targets a CUDA GPU: eager PyTorch (fp32 and fp16+JIT) as the reference anchor, ONNX Runtime with the CUDA execution provider, and a TensorRT engine — each exported, run once for a correctness check, then benchmarked.
| Format | Export | Inference |
|---|---|---|
| PyTorch | (no export — eager model) | predict() / inference(dtype=torch.float16) |
| ONNX (CUDA EP) | model.export() → .onnx |
onnxruntime.InferenceSession |
| TensorRT | model.export(format="trt") → .trt |
inference-models (builds its own engine — see note below) |
Not covered here: CPU-only deployment (see the CPU cookbook), mobile/edge formats (see the mobile cookbook), or Apple Silicon (see the Apple cookbook). This notebook requires a CUDA GPU end to end — it will not run on a CPU-only or Apple Silicon runtime.
Two TensorRT artifacts, on purpose. The
.trtfile from the export step is a standalone artifact for raw TensorRT deployment, locked to this GPU + TensorRT version.inference-modelsbuilds and manages its own engine internally, so it does not load that file — it is the easier, portable inference path used below.
1. Install¶
Installs from develop to pick up the newest export fixes. Colab ships mutually inconsistent
preinstalled packages that otherwise crash import rfdetr, so two are aligned: torchaudio is
uninstalled (RF-DETR never uses it, but transformers imports it when present and a
torch/torchaudio CUDA-version mismatch then errors), and pillow is force-reinstalled to a
clean version (a half-upgraded PIL breaks torchvision's import with cannot import name '_Ink').
The default PyPI onnxruntime-gpu wheel targets CUDA 11.8 and silently falls back to CPU on
modern GPUs, so it is reinstalled from the CUDA-12 package index — --force-reinstall is
required here: rfdetr[tensorrt] already pulls a plain onnxruntime-gpu wheel, and without it
pip sees the requirement as already satisfied and skips the CUDA-12 build entirely.
Colab: select a GPU runtime (Runtime → Change runtime type → GPU), and after this cell Runtime → Restart session before running the next cell — Colab keeps old package versions loaded until a restart.
!pip uninstall -q -y onnxruntime onnxruntime-gpu
!pip install -q "rfdetr[onnx,tensorrt] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" "inference-models[trt10]" supervision pandas
!pip install -q --force-reinstall --no-deps onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
!pip uninstall -q -y torchaudio
2. Setup and GPU check¶
Every format in this notebook requires CUDA — fail fast with a clear message if no GPU is visible.
WARMUP_RUNS discards the first N inferences (GPU kernels are JIT-compiled on first use);
MEASURE_RUNS then collects the steady-state timing distribution.
Three small helpers are shared by every format section below: _artifact_size_mb reports an
export artifact's size on disk, visualize_detections annotates and displays a
supervision.Detections on the sample image (falling back to COCO_CLASSES for label text when
a detection object carries no class_name), and measure_memory (from _benchmark) measures
device-memory growth via torch.cuda.mem_get_info() — device-wide, so it also captures ONNX
Runtime's CUDA execution provider and TensorRT's own cudaMalloc calls, not just PyTorch's own
caching allocator.
from pathlib import Path
import numpy as np
import supervision as sv
import torch
from PIL import Image
from rfdetr.assets.coco_classes import COCO_CLASSES
from rfdetr.export._benchmark import BenchmarkResult, measure_latency, measure_memory
if not torch.cuda.is_available():
raise RuntimeError("This notebook requires a CUDA GPU; none is available.")
print(f"GPU: {torch.cuda.get_device_name(0)}")
EXPORT_DIR = Path("export_cuda")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 20
MEASURE_RUNS = 100
def _artifact_size_mb(*paths: Path) -> float:
total_bytes = 0
for path in paths:
if path.is_dir():
total_bytes += sum(f.stat().st_size for f in path.rglob("*") if f.is_file())
else:
total_bytes += path.stat().st_size
return total_bytes / 1e6
def visualize_detections(detections: sv.Detections, image: Image.Image, save_path: Path | None = None) -> None:
names = detections.data.get("class_name") if detections.data else None
if names is None:
names = [COCO_CLASSES.get(int(c), str(c)) for c in detections.class_id]
labels = [f"{name} {conf:.2f}" for name, conf in zip(names, detections.confidence)]
annotated = sv.BoxAnnotator(thickness=3).annotate(scene=image.copy(), detections=detections)
annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate(
scene=annotated, detections=detections, labels=labels
)
if save_path is not None:
annotated.save(save_path)
print(f"Saved annotated image: {save_path}")
sv.plot_image(annotated)
def _enable_notebook_inline_matplotlib() -> None:
"""Enable inline matplotlib figures when running in IPython."""
get_ipython_func = globals().get("get_ipython")
if not callable(get_ipython_func):
return
ipython = get_ipython_func()
if ipython is not None:
ipython.run_line_magic("matplotlib", "inline")
ipython.run_line_magic("config", "InlineBackend.close_figures = True")
_enable_notebook_inline_matplotlib()
3. Sample image¶
A single street scene with several COCO classes (dog, bicycle, car) is enough to verify detections. The image is downloaded once and reused for every format below.
import urllib.request
IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)
image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")
4. PyTorch baseline — predict() / inference()¶
No export needed — this is the reference every other format in this notebook is compared
against. predict() is the unoptimized fp32 baseline. inference(dtype=torch.float16) fuses
layers with torch.jit.script and halves arithmetic precision — typically 1.5–2× faster than
fp32 on modern tensor cores with negligible accuracy loss; remove_optimized_model() reverts the
in-place optimization afterward so model stays reusable for the export calls below.
Both calls include preprocessing and postprocessing — there is no separate "forward-only" path
through eager predict(), so only an end-to-end number is reported for the PyTorch baseline.
from rfdetr import RFDETRSmall
with measure_memory(device="cuda") as mem:
model = RFDETRSmall()
baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_fp32_memory_mb = mem.delta_mb
print(f"PyTorch fp32 baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)
pytorch_fp32 = measure_latency(
lambda: model.predict(image), label="PyTorch predict() fp32", device="cuda", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)
with measure_memory(device="cuda") as mem:
model.inference(dtype=torch.float16)
_ = model.predict(image)
pytorch_fp16_jit_memory_mb = mem.delta_mb
pytorch_fp16_jit = measure_latency(
lambda: model.predict(image),
label="PyTorch inference() fp16+JIT",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
model.remove_optimized_model()
for r, mb in ((pytorch_fp32, pytorch_fp32_memory_mb), (pytorch_fp16_jit, pytorch_fp16_jit_memory_mb)):
print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS) +{mb:.1f} MB")
5. ONNX (CUDA execution provider)¶
What it is. ONNX (Open Neural Network Exchange) is a portable model-graph format most
inference runtimes understand — ONNX Runtime, TensorRT, and OpenVINO can all consume the same
.onnx file. Good for avoiding a single-vendor runtime lock-in: export once, then run it
through whichever of those runtimes fits the target machine without re-exporting; on a CUDA GPU
specifically, ONNX Runtime's CUDA execution provider is a faster-to-set-up alternative to a full
TensorRT engine build. See the ONNX export docs.
Export¶
model.export() defaults to ONNX.
onnx_path = model.export(output_dir=str(EXPORT_DIR))
onnx_size_mb = _artifact_size_mb(onnx_path)
print(f"ONNX model: {onnx_path} ({onnx_size_mb:.1f} MB)")
Inference¶
_create_onnx_session (the same session-construction helper RFDETR.predict()'s reference
decoder uses internally) is asked for CUDAExecutionProvider with a CPUExecutionProvider
fallback; it raises if the CUDA provider silently fails to activate.
from rfdetr.export._onnx.inference import _create_onnx_session, _run_inference
with measure_memory(device="cuda") as mem:
onnx_session = _create_onnx_session(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
active_provider = onnx_session.get_providers()[0]
if active_provider != "CUDAExecutionProvider":
raise RuntimeError(f"CUDAExecutionProvider not active (got {active_provider!r}) — see the install cell above.")
onnx_detections, _ = _run_inference(onnx_session, IMAGE_PATH, threshold=CONFIDENCE_THRESHOLD)
onnx_memory_mb = mem.delta_mb
print(f"ONNX (CUDA): {len(onnx_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(onnx_detections, image, EXPORT_DIR / "annotated_onnx.jpg")
Benchmark¶
Two scopes: forward_ms times only session.run with preprocessing done once outside the
loop; end2end_ms times preprocessing + session.run + decoding together, the same scope
predict() above was timed at. ONNX Runtime returns NumPy outputs from session.run, so
the call completes the output transfer before returning. Both scopes use synchronized host
wall-clock timing; PyTorch CUDA events may miss work submitted on ONNX Runtime's stream.
from rfdetr.export._runtime.decode import decode_detections
from rfdetr.export._runtime.preprocess import preprocess_to_nchw
_onnx_input_name = onnx_session.get_inputs()[0].name
_, _onnx_channels, _onnx_height, _onnx_width = onnx_session.get_inputs()[0].shape
_onnx_output_names = [out.name for out in onnx_session.get_outputs()]
_onnx_boxes_idx = next(i for i, name in enumerate(_onnx_output_names) if "dets" in name)
_onnx_logits_idx = next(i for i, name in enumerate(_onnx_output_names) if "labels" in name)
_onnx_feed = {_onnx_input_name: preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)}
def _onnx_end2end() -> None:
inp = preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)
raw = onnx_session.run(None, {_onnx_input_name: inp})
decode_detections(raw[_onnx_boxes_idx][0], raw[_onnx_logits_idx][0], image.size, threshold=CONFIDENCE_THRESHOLD)
onnx_forward = measure_latency(
lambda: onnx_session.run(None, _onnx_feed),
label="ONNX (CUDA) forward",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
onnx_end2end = measure_latency(
_onnx_end2end, label="ONNX (CUDA) end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)
for r in (onnx_forward, onnx_end2end):
print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")
6. TensorRT¶
What it is. TensorRT is NVIDIA's own inference optimizer and runtime — it compiles a model into an engine specialized for the exact GPU architecture and TensorRT version that built it. Good for squeezing the lowest possible latency out of an NVIDIA GPU in production, at the cost of that engine being locked to the machine (and TensorRT version) it was built on — convert on the same GPU family you plan to deploy to. See the TensorRT export docs.
Export¶
format="trt" compiles a standalone FP16 engine (tens of seconds to a few minutes).
trt_engine_path = model.export(format="trt", output_dir=str(EXPORT_DIR))
trt_size_mb = _artifact_size_mb(trt_engine_path)
print(f"TensorRT engine: {trt_engine_path} ({trt_size_mb:.1f} MB)")
Inference¶
Inference goes through inference-models, which builds and manages its own TensorRT engine
— the easier, portable path that also handles preprocessing, postprocessing, and class names —
rather than loading the .trt file exported above. Its precision is controlled by
AutoModel.from_pretrained(..., quantization=...), independent of the fp16 kwarg used for the
standalone .trt export: quantization=None (default) picks automatically based on device
capabilities, and "fp32"/"fp16"/"bf16"/"int8" request one explicitly.
from inference_models import AutoModel, BackendType
with measure_memory(device="cuda") as mem:
trt_model = AutoModel.from_pretrained("rfdetr-small", backend=BackendType.TRT)
trt_predictions = trt_model(np.array(image))
trt_memory_mb = mem.delta_mb
trt_detections = trt_predictions[0].to_supervision()
trt_detections = trt_detections[trt_detections.confidence > CONFIDENCE_THRESHOLD]
print(f"TensorRT: {len(trt_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(trt_detections, image, EXPORT_DIR / "annotated_tensorrt.jpg")
Benchmark¶
inference-models does not expose a forward-only hook separate from its own preprocessing, so
only an end-to-end number is reported here — the same limitation as the PyTorch baseline above.
trt_end2end = measure_latency(
lambda: trt_model(np.array(image)),
label="TensorRT (inference-models, auto) end2end",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(
f" {trt_end2end.label:<32} {trt_end2end.mean_ms:6.2f} ms ± {trt_end2end.std_ms:5.2f} ({trt_end2end.fps:6.1f} FPS)"
)
TensorRT (fp32, explicit)¶
quantization="fp32" forces full precision through inference-models' own engine builder — the
numeric-parity anchor against the auto-selected (typically fp16 on a capable GPU) row above.
with measure_memory(device="cuda") as mem:
trt_fp32_model = AutoModel.from_pretrained("rfdetr-small", backend=BackendType.TRT, quantization="fp32")
trt_fp32_predictions = trt_fp32_model(np.array(image))
trt_fp32_memory_mb = mem.delta_mb
trt_fp32_detections = trt_fp32_predictions[0].to_supervision()
trt_fp32_detections = trt_fp32_detections[trt_fp32_detections.confidence > CONFIDENCE_THRESHOLD]
print(f"TensorRT fp32: {len(trt_fp32_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(trt_fp32_detections, image, EXPORT_DIR / "annotated_tensorrt_fp32.jpg")
trt_fp32_end2end = measure_latency(
lambda: trt_fp32_model(np.array(image)),
label="TensorRT (inference-models, fp32) end2end",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(
f" {trt_fp32_end2end.label:<32} {trt_fp32_end2end.mean_ms:6.2f} ms ± {trt_fp32_end2end.std_ms:5.2f} "
f"({trt_fp32_end2end.fps:6.1f} FPS)"
)
7. Results¶
Measured on an NVIDIA L4 (compute capability 8.9, 22 GB), Colab GPU runtime, rfdetr v1.11.0,
batch 1, 20 warmup + 100 timed runs, RFDETRSmall. Numbers vary by GPU model, TensorRT version,
and driver — rerun the cells below to get yours. Config is the precision/EP used for that row;
— marks a scope this format doesn't have. Memory [MB] is device-memory growth
(torch.cuda.mem_get_info()) across constructing the runtime plus its first inference call —
device-wide, so it also captures ONNX Runtime's CUDA execution provider and TensorRT's own
allocations, not just PyTorch's caching allocator.
| Format | Config | forward [ms] | end2end [ms] | FPS [img/s] (end2end) | Memory [MB] |
|---|---|---|---|---|---|
PyTorch predict() |
fp32 | — | 19.88 ± 0.45 | 50.3 | 260.0 |
PyTorch inference() |
fp16+JIT | — | 11.30 ± 0.28 | 88.5 | 222.3 |
| ONNX | CUDA EP | 7.36 ± 0.06 | 12.30 ± 0.63 | 81.3 | 291.5 |
| TensorRT | auto (inference-models) | — | 4.91 ± 0.14 | 203.5 | 140.5 |
| TensorRT | fp32 (inference-models) | — | 7.96 ± 0.30 | 125.6 | 228.6 |
Unlike the CPU and mobile cookbooks — where fp16 bought nothing, because those CPUs have no native
fp16 kernels — reduced precision is a decisive win on a GPU with tensor cores. TensorRT's
auto-selected precision is 1.62× faster than the same engine forced to fp32 (4.91 vs 7.96 ms)
and 4.05× faster than eager fp32 PyTorch, and PyTorch's own inference(dtype=torch.float16)
nearly halves the eager baseline (11.30 vs 19.88 ms). This is the hardware difference the other
notebooks' fp16 rows make visible by contrast: fp16 pays off where the silicon implements it.
ONNX's forward (7.36 ms) is far below its end2end (12.30 ms) because that gap is CPU-side
pre/postprocessing, which no execution provider accelerates — worth remembering when comparing a
forward-only number from one runtime against an end-to-end number from another.
import pandas as pd
def _fmt_ms(result: BenchmarkResult | None) -> str:
if result is None:
return "—"
return f"{result.mean_ms:.2f} ± {result.std_ms:.2f}"
def _result_row(
format_label: str,
config: str,
forward: BenchmarkResult | None,
end2end: BenchmarkResult | None,
memory_mb: float | None,
) -> dict:
fps = (end2end or forward).fps
return {
"Format": format_label,
"Config": config,
"forward [ms]": _fmt_ms(forward),
"end2end [ms]": _fmt_ms(end2end),
"FPS [img/s] (end2end)": round(fps, 1),
"Memory [MB]": f"{memory_mb:.1f}" if memory_mb is not None else "—",
}
summary = pd.DataFrame(
[
_result_row("PyTorch predict()", "fp32", None, pytorch_fp32, pytorch_fp32_memory_mb),
_result_row("PyTorch inference()", "fp16+JIT", None, pytorch_fp16_jit, pytorch_fp16_jit_memory_mb),
_result_row("ONNX", "CUDA EP", onnx_forward, onnx_end2end, onnx_memory_mb),
_result_row("TensorRT", "auto (inference-models)", None, trt_end2end, trt_memory_mb),
_result_row("TensorRT", "fp32 (inference-models)", None, trt_fp32_end2end, trt_fp32_memory_mb),
]
).set_index("Format")
print(summary.to_string())
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs, batch 1, GPU: {torch.cuda.get_device_name(0)}.")
Next steps¶
- Fine-tuned weights — pass
pretrain_weights="<path/to/checkpoint.pth>"when constructing the model. - Dynamic batch — both ONNX and TensorRT support
dynamic_batch=True; see the ONNX and TensorRT export docs. - Not on a CUDA GPU? — see the CPU cookbook, the mobile/edge cookbook, or the Apple Silicon cookbook.
- See the Export documentation for every format and option.