RF-DETR on NVIDIA GPUs — Export, Inference & Latency¶
Compares every export path that targets a CUDA GPU: eager PyTorch, its fp16 TorchScript and
direct CUDA Graph inference backends as the reference anchor, ONNX Runtime with the CUDA
execution provider, and a TensorRT engine — each
exported, run once for a correctness check, then benchmarked. Every rfdetr export is loaded through the one public
prediction API, RFDETRInference(path), and timed end to end through predict(), the same call as the PyTorch rows.
| Format | Export | Inference |
|---|---|---|
| PyTorch | (no export — eager model) | predict() / inference(dtype=torch.float16) / inference(compile_backend="cudagraph") |
| ONNX (CUDA EP) | model.export(dynamic_batch=True) → .onnx |
RFDETRInference(path).predict() |
| TensorRT | model.export(format="trt") → .trt |
inference-models (selects and loads its own prebuilt TensorRT engine — see note below), and RFDETRInference(path).predict() loading the .trt file directly |
inference-models |
(prebuilt by that package) | PyTorch (CUDA) and ONNX Runtime (CUDA execution provider) through AutoModel.from_pretrained(), next to the TensorRT rows |
| TensorRT INT8 | model.export(format="trt", quantization="int8", calibration_data=...) → *_int8.trt |
RFDETRInference(path).predict(), compared with a batch-1 FP16 engine on latency, size and COCO mAP |
Every format is benchmarked at batch 1, 4, and 8 (the TensorRT INT8 comparison of Section 8 is batch 1 only; Section 8.1 runs batches 1, 4 and 8) — see
src/rfdetr/export/registry.py
for which formats can hold one artifact across batch sizes and which cannot: ONNX and TensorRT
both bake a dynamic batch axis (supports_dynamic_batch=True), so one exported artifact serves
every row below for each of them; PyTorch needs no export step and trivially takes any batch as a
(N, C, H, W) tensor. inference-models' managed TensorRT engine is a separate
artifact (its engine build is not part of this repository) — its own batch handling is exercised through its documented multi-image __call__
signature (Union[np.ndarray, List[np.ndarray], torch.Tensor, List[torch.Tensor]], confirmed by
reading inference_models/models/base/object_detection.py on GitHub), not verified against the
package's source the way the two rfdetr-exported formats are.
Not covered here: CPU-only deployment (see the CPU cookbook), mobile/edge formats (see the mobile cookbook), or Apple Silicon (see the Apple cookbook). This notebook requires a CUDA GPU end to end — it will not run on a CPU-only or Apple Silicon runtime.
Two TensorRT artifacts, on purpose. The
.trtfile from the export step is a standalone artifact for raw TensorRT deployment, locked to this GPU + TensorRT version.inference-modelsselects and loads its own prebuilt TensorRT engine, so it does not load that file — it is the easier, portable inference path, benchmarked first below. The.trtfile itself is then loaded and timed too, viaRFDETRInference— the package's own prediction API for a raw engine, not the managed path (see its section below) — so both TensorRT artifacts get a number, not just the oneinference-modelsloads. It is exported once withdynamic_batch=True, max_batch_size=8so the same engine serves every batch size below, the same way the managed engine and ONNX do.
Which latency number is this? Every number in this notebook is a Tier C measurement: forward-only and end-to-end timings for each export format, on one L4, against a PyTorch anchor measured on that same L4. Its value is ratios within this machine — never set these milliseconds against the architecture headline on the Benchmarks page (T4, TensorRT FP16, model only, batch 1) or the number on Roboflow's RF-DETR model page (L4, Roboflow Inference, scope not stated, batch 1) without checking hardware, precision, batch size, and scope match. See Section 9 for where this notebook's own TensorRT number disagrees with Roboflow's model page.
1. Install¶
RFDETR_SPEC holds the install source and defaults to the develop branch. RFDETRInference and the
rfdetr.export.benchmark helpers this notebook imports ship in 1.12; to install from PyPI instead, swap in the commented
>=1.12,<2 line once it is released. Sections 8 and 8.1's quantization="int8" TensorRT export (detection, segmentation and
keypoint models) and runtime_options={"cuda_graph": True} are not in 1.11.x, so those sections need the develop install
until it is released.
Colab ships mutually inconsistent
preinstalled packages that otherwise crash import rfdetr, so two are aligned: torchaudio is
uninstalled (RF-DETR never uses it, but transformers imports it when present and a
torch/torchaudio CUDA-version mismatch then errors), and pillow is force-reinstalled to a
clean version (a half-upgraded PIL breaks torchvision's import with cannot import name '_Ink').
The default PyPI onnxruntime-gpu wheel targets CUDA 11.8 and silently falls back to CPU on
modern GPUs, so it is reinstalled from the CUDA-12 package index — --force-reinstall is
required here: rfdetr[tensorrt] already pulls a plain onnxruntime-gpu wheel, and without it
pip sees the requirement as already satisfied and skips the CUDA-12 build entirely.
Colab: select a GPU runtime (Runtime → Change runtime type → GPU), and after this cell Runtime → Restart session before running the next cell — Colab keeps old package versions loaded until a restart.
RFDETR_SPEC = "rfdetr[onnx,tensorrt] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[onnx,tensorrt]>=1.12,<2"
!pip uninstall -q -y onnxruntime onnxruntime-gpu
!pip install -q "{RFDETR_SPEC}" "inference-models[trt10,onnx-cu12]" supervision pandas tabulate faster-coco-eval
!pip install -q --force-reinstall --no-deps onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
!pip uninstall -q -y torchaudio
2. Setup and GPU check¶
Every format in this notebook requires CUDA — fail fast with a clear message if no GPU is visible.
WARMUP_RUNS discards the first N inferences (GPU kernels are JIT-compiled on first use);
MEASURE_RUNS then collects the steady-state timing distribution. BATCH_SIZES is the batch
matrix every export format below measures — (1, 4, 8), not a single number, so each export
format contributes three rows to the final results table. A CUDA graph is captured for one batch size, so the CUDA Graph rows capture one predictor per batch. The static FP16
and INT8 engines of Section 8, one pair per batch size, have tables of their own.
measure_latency and measure_memory come from the public rfdetr.export.benchmark module;
measure_memory(device="cuda") measures device-memory growth via torch.cuda.mem_get_info() —
device-wide, so it also captures ONNX Runtime's CUDA execution provider and TensorRT's own
cudaMalloc calls, not just PyTorch's own caching allocator. Every exported artifact is
loaded with RFDETRInference, so it runs the same preprocessing, postprocessing and class mapping as predict() on the
PyTorch rows. The installed rfdetr version is printed so the Results header can be filled from it.
Every timed rfdetr predict() call passes include_source_image=False. inference-models returns no copy of the input image in its results, so rfdetr is timed without one too; with the default True it also copies each image into its results.
Every timed call takes the same decoded RGB array, frame, as a camera or video decoder delivers it, so no row pays a JPEG decode or a PIL-to-array copy.
MODEL_SIZE (nano, small, medium or large) and TASK (detection or segmentation) pick the model every row measures. A segmentation call also decodes masks on the host, so its latencies are not comparable with detection, and the mAP rows stay box mAP for both tasks.
import importlib.metadata
from pathlib import Path
from typing import NamedTuple
import numpy as np
import onnxruntime as ort
import supervision as sv
import torch
from PIL import Image
from rfdetr import RFDETRInference
from rfdetr.export._benchmark import _artifact_size_mb, visualize_detections
from rfdetr.export.benchmark import BenchmarkResult, measure_latency, measure_memory
if not torch.cuda.is_available():
raise RuntimeError("This notebook requires a CUDA GPU; none is available.")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")
import rfdetr
MODEL_SIZE = "small" # nano, small, medium or large
TASK = "detection" # "detection" or "segmentation"
MODEL_CLASS = f"RFDETR{'Seg' if TASK == 'segmentation' else ''}{MODEL_SIZE.capitalize()}" # class name in `rfdetr`
MODEL_ID = f"rfdetr-{'seg-' if TASK == 'segmentation' else ''}{MODEL_SIZE}" # `inference-models` id, names the files
EXPORT_DIR = Path("export_cuda")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 20
MEASURE_RUNS = 100
BATCH_SIZES = (1, 4, 8)
class Config(NamedTuple):
"""The fixed options of one benchmark row, one table column each."""
runtime: str
precision: str
quantization: str = "—"
Notebook display helpers¶
_artifact_size_mb and visualize_detections come from the private rfdetr.export._benchmark module,
next to the COCO mAP helpers used later. They size artifacts and draw detections; none of them
changes what a runtime computes.
# Inline figures keep every plot in the notebook; closing them after display stops figures piling up in memory.
%matplotlib inline
%config InlineBackend.close_figures = True
3. Sample image¶
A single street scene with several COCO classes is enough to verify detections. The image is downloaded once and reused for every format below.
import urllib.request
IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)
image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")
# A camera or video decoder delivers decoded arrays, so every timed call below takes this RGB array and no
# call pays a PIL-to-array copy or a JPEG decode.
frame = np.array(image)
4. PyTorch baseline — predict() / inference()¶
No export needed — this is the reference every other format in this notebook is compared
against. predict() is the unoptimized fp32 baseline and accepts a list of images, batching them
internally in one forward call, so the batch matrix below feeds it [image] * batch.
model.inference(dtype=torch.float16, batch_size=batch) returns an independent TorchScript
predictor compiled for exactly batch_size images — calling predict() with a different batch
raises ValueError: Batch size mismatch (confirmed by an earlier run of this notebook). Each row
uses a new predictor and deleting it releases its runtime while leaving model available for the
export calls that follow. For repeated inference
at one fixed batch size and resolution, compile_backend="cudagraph" captures the TorchScript
forward once and replays it with lower launch overhead. Capture allocates a graph-private memory
pool holding the captured forward's intermediates plus the static input/output buffers; it
persists for the graph's lifetime and scales with batch size, resolution and model, so this
route uses more device memory. Returned outputs are cloned before the next replay can
overwrite them. A failed CUDA capture can leave the process's CUDA state unusable, so restart the
runtime before trying another backend if setup raises. The CUDA Graph row below retains the
original batch-1 measurement; rerun it at another fixed batch size before comparing that size.
Its memory delta uses synchronized endpoint reads instead of measure_memory, whose background
sampler synchronizes CUDA and therefore cannot run while another thread captures a graph.
Both calls include preprocessing and postprocessing — there is no separate "forward-only" path
through eager predict(), so only an end-to-end number is reported for the PyTorch baseline.
from functools import partial
with measure_memory(device="cuda") as mem:
model = getattr(rfdetr, MODEL_CLASS)()
baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_fp32_memory_mb = mem.delta_mb
print(f"PyTorch fp32 baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)
pytorch_fp32_results: dict[int, BenchmarkResult] = {}
for batch in BATCH_SIZES:
images_batch = [frame] * batch
pytorch_fp32_results[batch] = measure_latency(
partial(model.predict, images_batch, include_source_image=False),
label=f"PyTorch predict() fp32 batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
pytorch_fp16_jit_results: dict[int, BenchmarkResult] = {}
pytorch_fp16_jit_memory_mb: dict[int, float] = {}
for batch in BATCH_SIZES:
images_batch = [frame] * batch
with measure_memory(device="cuda") as mem:
predictor = model.inference(dtype=torch.float16, batch_size=batch, compile_backend="torchscript")
_ = predictor.predict(images_batch, include_source_image=False)
pytorch_fp16_jit_memory_mb[batch] = mem.delta_mb
pytorch_fp16_jit_results[batch] = measure_latency(
partial(predictor.predict, images_batch, include_source_image=False),
label=f"PyTorch inference() fp16+JIT batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
del predictor
pytorch_fp16_cudagraph_results: dict[int, BenchmarkResult] = {}
pytorch_fp16_cudagraph_memory_mb: dict[int, float] = {}
for batch in BATCH_SIZES:
images_batch = [frame] * batch
torch.cuda.empty_cache()
torch.cuda.synchronize()
cudagraph_free_before, _ = torch.cuda.mem_get_info()
cudagraph_predictor = model.inference(compile_backend="cudagraph", batch_size=batch, dtype=torch.float16)
cudagraph_detections = cudagraph_predictor.predict(images_batch, threshold=CONFIDENCE_THRESHOLD)
torch.cuda.synchronize()
cudagraph_free_after, _ = torch.cuda.mem_get_info()
pytorch_fp16_cudagraph_memory_mb[batch] = (cudagraph_free_before - cudagraph_free_after) / 1e6
if batch == BATCH_SIZES[0]:
print(f"PyTorch fp16 CUDA Graph: {len(cudagraph_detections[0])} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(cudagraph_detections[0], image)
pytorch_fp16_cudagraph_results[batch] = measure_latency(
partial(cudagraph_predictor.predict, images_batch, include_source_image=False),
label=f"PyTorch inference() fp16+CUDA Graph batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
del cudagraph_predictor
r = pytorch_fp16_cudagraph_results[batch]
print(
f" batch={batch} fp16+CUDA Graph {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)"
f" +{pytorch_fp16_cudagraph_memory_mb[batch]:.1f} MB"
)
for batch in BATCH_SIZES:
r32, r16 = pytorch_fp32_results[batch], pytorch_fp16_jit_results[batch]
print(f" batch={batch} fp32 {r32.mean_ms:6.2f} ms ± {r32.std_ms:5.2f} ({batch * r32.fps:6.1f} FPS)")
print(
f" batch={batch} fp16+JIT {r16.mean_ms:6.2f} ms ± {r16.std_ms:5.2f} ({batch * r16.fps:6.1f} FPS)"
f" +{pytorch_fp16_jit_memory_mb[batch]:.1f} MB"
)
5. ONNX (CUDA execution provider)¶
What it is. ONNX (Open Neural Network Exchange) is a portable model-graph format most
inference runtimes understand — ONNX Runtime, TensorRT, and OpenVINO can all consume the same
.onnx file. Good for avoiding a single-vendor runtime lock-in: export once, then run it
through whichever of those runtimes fits the target machine without re-exporting; on a CUDA GPU
specifically, ONNX Runtime's CUDA execution provider is a faster-to-set-up alternative to a full
TensorRT engine build. See the ONNX export docs.
Export¶
dynamic_batch=True bakes a dynamic batch axis into the graph (registry.py marks ONNX
supports_dynamic_batch=True) — one exported .onnx file serves every batch size below.
onnx_path = model.export(output_dir=str(EXPORT_DIR), dynamic_batch=True)
onnx_size_mb = _artifact_size_mb(onnx_path)
print(f"ONNX model: {onnx_path} ({onnx_size_mb:.1f} MB)")
Inference and benchmark¶
RFDETRInference(onnx_path, device="cuda") loads the .onnx file with ONNX Runtime's CUDAExecutionProvider; the loader
raises if the CUDA provider is unavailable instead of silently falling back to the CPU, so a wrong onnxruntime-gpu
install fails here. The predictor handles preprocessing, the session call and decoding, so only an end-to-end number is
reported, the same scope as the PyTorch rows above.
Construction and the first call share one measure_memory window, the same scope every other format's memory number uses.
Per batch, Memory [MB] is only meaningful once per shared predictor: the batch-1 row reuses onnx_memory_mb
(construction plus first call), and the batch-4 and batch-8 rows measure the incremental arena growth of the same
predictor's first call at that larger batch, not a second construction.
with measure_memory(device="cuda") as mem:
onnx_predictor = RFDETRInference(onnx_path, device="cuda")
onnx_detections = onnx_predictor.predict(image, threshold=CONFIDENCE_THRESHOLD)
onnx_memory_mb = mem.delta_mb
print(
f"ONNX (CUDA): {len(onnx_detections)} detections above {CONFIDENCE_THRESHOLD}; runtime: {onnx_predictor.runtime_info}"
)
visualize_detections(onnx_detections, image, EXPORT_DIR / "annotated_onnx.jpg")
onnx_end2end_results: dict[int, BenchmarkResult] = {}
onnx_memory_by_batch: dict[int, float] = {BATCH_SIZES[0]: onnx_memory_mb}
for batch in BATCH_SIZES:
images_batch = [frame] * batch
if batch != BATCH_SIZES[0]:
with measure_memory(device="cuda") as mem:
onnx_predictor.predict(images_batch, include_source_image=False)
onnx_memory_by_batch[batch] = mem.delta_mb
onnx_end2end_results[batch] = measure_latency(
partial(onnx_predictor.predict, images_batch, include_source_image=False),
label=f"ONNX (CUDA) end2end batch={batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
e2e = onnx_end2end_results[batch]
print(f" batch={batch} end2end {e2e.mean_ms:6.2f} ms ± {e2e.std_ms:5.2f} ({batch * e2e.fps:6.1f} FPS)")
6. TensorRT¶
What it is. TensorRT is NVIDIA's own inference optimizer and runtime — it compiles a model into an engine specialized for the exact GPU architecture and TensorRT version that built it. Good for squeezing the lowest possible latency out of an NVIDIA GPU in production, at the cost of that engine being locked to the machine (and TensorRT version) it was built on — convert on the same GPU family you plan to deploy to. See the TensorRT export docs.
Export¶
format="trt" compiles a standalone FP16 engine (tens of seconds to a few minutes).
trt_engine_path = model.export(format="trt", output_dir=str(EXPORT_DIR))
trt_size_mb = _artifact_size_mb(trt_engine_path)
print(f"TensorRT engine: {trt_engine_path} ({trt_size_mb:.1f} MB)")
Inference¶
Inference goes through inference-models, which selects and loads its own prebuilt TensorRT
engine (the managed engine rows below) — the easier, portable path that also handles preprocessing, postprocessing,
and class names — rather than loading the .trt file exported above. Its precision is
controlled by AutoModel.from_pretrained(..., quantization=...), independent of the fp16
kwarg used for the standalone .trt export: quantization=None (default) picks automatically
based on device capabilities, and "fp32"/"fp16"/"bf16"/"int8" request one explicitly.
verbose=True prints inference-models' own package-selection log line, which names the
backend and the quantization it actually picked for quantization=None below. A NumPy array is BGR to inference-models unless the call says otherwise, so every call passes input_color_format="rgb" for the
decoded RGB arrays, and confidence= so its threshold is the one rfdetr uses. It is constructed
once and fed every batch size below through its documented List[np.ndarray] call signature —
what it does per batch internally is not part of this repository and not something this
notebook can verify, only measure.
from inference_models import AutoModel, BackendType
with measure_memory(device="cuda") as mem:
trt_model = AutoModel.from_pretrained(MODEL_ID, backend=BackendType.TRT, verbose=True)
trt_predictions = trt_model(np.array(image), confidence=CONFIDENCE_THRESHOLD, input_color_format="rgb")
trt_memory_mb: dict[int, float] = {BATCH_SIZES[0]: mem.delta_mb}
trt_detections = trt_predictions[0].to_supervision()
trt_detections = trt_detections[trt_detections.confidence > CONFIDENCE_THRESHOLD]
print(f"TensorRT: {len(trt_detections)} detections above {CONFIDENCE_THRESHOLD}")
# No parity line: `inference-models` returns decoded predictions, not raw `dets`/`labels` arrays.
visualize_detections(trt_detections, image, EXPORT_DIR / "annotated_tensorrt.jpg")
Benchmark — batch 1, 4, and 8¶
inference-models does not expose a forward-only hook separate from its own preprocessing, so
only an end-to-end number is reported here — the same limitation as the PyTorch baseline above.
Memory [MB] for batch 1 reuses the construction cost from the Inference cell above; batch 4 and
8 measure the incremental cost of this same model's first call at that batch.
trt_end2end_results: dict[int, BenchmarkResult] = {}
for batch in BATCH_SIZES:
images_batch = [frame] * batch
if batch != BATCH_SIZES[0]:
with measure_memory(device="cuda") as mem:
im_predict(trt_model, images_batch, CONFIDENCE_THRESHOLD)
trt_memory_mb[batch] = mem.delta_mb
trt_end2end_results[batch] = measure_latency(
partial(im_predict, trt_model, images_batch, CONFIDENCE_THRESHOLD),
label=f"TensorRT (inference-models, managed engine) auto end2end batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
r = trt_end2end_results[batch]
print(f" batch={batch} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)")
TensorRT (fp32, explicit)¶
quantization="fp32" asks inference-models for its full-precision prebuilt engine — the
numeric-parity anchor against the auto-selected (typically fp16 on a capable GPU) row above.
with measure_memory(device="cuda") as mem:
trt_fp32_model = AutoModel.from_pretrained(MODEL_ID, backend=BackendType.TRT, quantization="fp32", verbose=True)
trt_fp32_predictions = trt_fp32_model(np.array(image), confidence=CONFIDENCE_THRESHOLD, input_color_format="rgb")
trt_fp32_memory_mb: dict[int, float] = {BATCH_SIZES[0]: mem.delta_mb}
trt_fp32_detections = trt_fp32_predictions[0].to_supervision()
trt_fp32_detections = trt_fp32_detections[trt_fp32_detections.confidence > CONFIDENCE_THRESHOLD]
print(f"TensorRT fp32: {len(trt_fp32_detections)} detections above {CONFIDENCE_THRESHOLD}")
# No parity line: `inference-models` returns decoded predictions, not raw `dets`/`labels` arrays.
visualize_detections(trt_fp32_detections, image, EXPORT_DIR / "annotated_tensorrt_fp32.jpg")
trt_fp32_end2end_results: dict[int, BenchmarkResult] = {}
for batch in BATCH_SIZES:
images_batch = [frame] * batch
if batch != BATCH_SIZES[0]:
with measure_memory(device="cuda") as mem:
im_predict(trt_fp32_model, images_batch, CONFIDENCE_THRESHOLD)
trt_fp32_memory_mb[batch] = mem.delta_mb
trt_fp32_end2end_results[batch] = measure_latency(
partial(im_predict, trt_fp32_model, images_batch, CONFIDENCE_THRESHOLD),
label=f"TensorRT (inference-models, managed engine) fp32 end2end batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
r = trt_fp32_end2end_results[batch]
print(f" batch={batch} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)")
TensorRT (raw .trt engine, RFDETRInference)¶
The rows above never touch the .trt file model.export(format="trt") wrote — this section
loads that file directly with RFDETRInference, the package's own prediction API for a raw engine. It is not the
managed inference-models path above, but it is the only way to time the .trt artifact itself rather than a
separately-built managed engine. device="cuda" pins it to the current device. dynamic_batch=True, max_batch_size=8 on
the export bakes an optimization profile spanning batch 1–8 into one engine, so this single predictor serves every batch
size below; no separate export per batch is needed here (unlike CoreML/Core AI/ExecuTorch on Apple Silicon, or
LiteRT/ExecuTorch-XNNPACK on mobile, which do bake a fixed batch shape and need one artifact per batch size — see those
cookbooks). The loader reads the engine's own input dtype (fp16 unless the export forced fp32 I/O) from its metadata and
builds the input accordingly, so nothing in the notebook casts the tensor on the caller's behalf.
trt_raw_engine_path = model.export(
format="trt", dynamic_batch=True, max_batch_size=max(BATCH_SIZES), output_dir=str(EXPORT_DIR)
)
trt_raw_engine_size_mb = _artifact_size_mb(trt_raw_engine_path)
print(f"TensorRT engine (dynamic batch 1-{max(BATCH_SIZES)}): {trt_raw_engine_path} ({trt_raw_engine_size_mb:.1f} MB)")
Inference¶
One correctness call before benchmarking, same as every other format above. Memory [MB] in the
results table below covers constructing the predictor (engine deserialize + output-buffer
cudaMalloc, sized for the profile maximum, batch 8) plus this first call -- both happen inside
the same measure_memory window, same scope as every other format's memory number. Building the
predictor outside this window would read back ~0 MB, since its first call reuses buffers
the runtime already allocated at construction rather than allocating anything new.
with measure_memory(device="cuda") as mem:
trt_raw_predictor = RFDETRInference(trt_raw_engine_path, device="cuda")
trt_raw_detections = trt_raw_predictor.predict(image, threshold=CONFIDENCE_THRESHOLD)
trt_raw_memory_mb: dict[int, float] = {BATCH_SIZES[0]: mem.delta_mb}
print(
f"TensorRT (raw engine): {len(trt_raw_detections)} detections above {CONFIDENCE_THRESHOLD}; runtime: {trt_raw_predictor.runtime_info}"
)
visualize_detections(trt_raw_detections, image, EXPORT_DIR / "annotated_tensorrt_raw.jpg")
Benchmark — batch 1, 4, and 8¶
predict() is timed end to end: preprocessing, the engine call and decoding, the same scope as the PyTorch, ONNX and
managed-engine rows. Batch 4 and 8 measure the incremental memory of this same engine's first call at that batch, the
same convention as ONNX and the managed engine above.
trt_raw_end2end_results: dict[int, BenchmarkResult] = {}
for batch in BATCH_SIZES:
images_batch = [frame] * batch
if batch != BATCH_SIZES[0]:
with measure_memory(device="cuda") as mem:
trt_raw_predictor.predict(images_batch, include_source_image=False)
trt_raw_memory_mb[batch] = mem.delta_mb
trt_raw_end2end_results[batch] = measure_latency(
partial(trt_raw_predictor.predict, images_batch, include_source_image=False),
label=f"TensorRT (raw .trt engine) end2end batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
e2e = trt_raw_end2end_results[batch]
print(f" batch={batch} end2end {e2e.mean_ms:6.2f} ms ± {e2e.std_ms:5.2f} ({batch * e2e.fps:6.1f} FPS)")
6.1 inference-models: PyTorch and ONNX Runtime¶
Section 6 timed inference-models' managed TensorRT engine. The same package also serves RF-DETR through PyTorch and ONNX Runtime. They run here on the same GPU, at the same batch sizes and end-to-end scope, next to the rfdetr rows.
Two call settings keep the comparison fair, and the managed-engine rows above use them too:
input_color_format="rgb": a NumPy array is BGR to this package unless declared. Without it the network sees swapped channels and the scores drop, with no error.confidence=: passed on every call, so both packages use the same threshold.
onnx_execution_providers=["CUDAExecutionProvider", "CPUExecutionProvider"] pins ONNX Runtime to the CUDA provider, like the rfdetr ONNX row.
#: `inference-models` models by the key of their `map_results` entry, kept for the COCO section.
im_models: dict[str, object] = {}
im_results: dict[str, dict[int, BenchmarkResult]] = {}
im_memory_mb: dict[str, dict[int, float]] = {}
def benchmark_inference_models(key: str, **load_kwargs) -> None:
"""Load one `inference-models` model, check it once on the sample image and time `im_predict` at every batch size."""
with measure_memory(device="cuda") as mem:
im_model = AutoModel.from_pretrained(MODEL_ID, **load_kwargs)
detections = im_predict(im_model, [frame], CONFIDENCE_THRESHOLD)
im_models[key] = im_model
im_results[key], im_memory_mb[key] = {}, {BATCH_SIZES[0]: mem.delta_mb}
print(f"{key}: {len(detections[0])} detections above {CONFIDENCE_THRESHOLD}")
for batch in BATCH_SIZES:
if batch != BATCH_SIZES[0]:
with measure_memory(device="cuda") as mem:
im_predict(im_model, [frame] * batch, CONFIDENCE_THRESHOLD)
im_memory_mb[key][batch] = mem.delta_mb
im_results[key][batch] = measure_latency(
partial(im_predict, im_model, [frame] * batch, CONFIDENCE_THRESHOLD),
label=f"{key} batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
r = im_results[key][batch]
print(f" batch={batch}: {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)")
benchmark_inference_models(
"ONNX inference-models CUDA EP",
backend=BackendType.ONNX,
onnx_execution_providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
benchmark_inference_models("PyTorch inference-models", backend=BackendType.TORCH, device="cuda")
7. COCO val2017 accuracy¶
A latency number on its own can hide a precision collapse. On the CPU, OpenVINO once compiled
RF-DETR at the device's default f16 precision and returned 0 detections, and the timing still
looked normal. So each batch-1 row below also gets a box mAP, measured on the same exported
artifact at the same precision as its latency row. The .trt engine is scored, not a rebuilt one.
By default this runs on a 500-image seeded subset of COCO val2017, so the result is not the
full-val2017 number on the Benchmarks page.
Set FULL_VAL = True to score all 5000 images. Each row costs roughly its batch-1 end2end
latency times the number of images. On the L4 that is ~20 ms × 500 ≈ 10 s for the slowest row
(PyTorch predict() fp32), plus JPEG decoding.
Every row is scored the same way: evaluate_coco_map calls predict(image, threshold=0.001) and reads the returned
sv.Detections, whose class_id values are COCO category IDs for the pretrained checkpoint, at a 0.001 score floor that
keeps the whole precision-recall curve. No per-format decoding is involved, so an exported row differs from the PyTorch row
only by what the export itself changed.
The inference-models rows are scored too, through im_predict with confidence=0.001. Their class_id values are COCO category IDs: on the sample image the dog is 18, the person 1, the backpack 27 and the handbag 31 in both packages' output, so the same ground truth applies.
N_EVAL = 500
FULL_VAL = False
COCO_SEED = 0
COCO_ROOT = Path("coco")
N_CALIB = 128
Display and scoring helpers come from the private rfdetr.export._benchmark module; they are not a
supported API and do not change what any runtime computes. The fetch downloads the 241 MB
annotation archive once, then only the selected images, and skips files already on disk.
from rfdetr.export._benchmark import evaluate_coco_map, fetch_coco_val2017
coco_subset = fetch_coco_val2017(COCO_ROOT, n_images=None if FULL_VAL else N_EVAL, seed=COCO_SEED)
print(f"COCO val2017 images to score: {len(coco_subset.image_ids)}")
map_results collects one entry per scored row. Every runtime is scored through its own predict() on a single RGB PIL
image (batch 1), the same call its latency row timed.
pytorch_predict_run = partial(model.predict, threshold=0.001)
map_results = {}
PyTorch predict(), fp32. Section 4 reverted model to eager mode, so this is the fp32 baseline.
map_results["PyTorch predict() fp32"] = evaluate_coco_map(pytorch_predict_run, coco_subset)
PyTorch inference(), fp16+JIT. An independent predictor is compiled for batch 1, matching
the latency row, and the next cell scores it through predict().
predictor = model.inference(dtype=torch.float16, batch_size=1, compile_backend="torchscript")
pytorch_predict_run = partial(predictor.predict, threshold=0.001)
map_results["PyTorch inference() fp16+JIT"] = evaluate_coco_map(pytorch_predict_run, coco_subset)
Revert to eager mode, so model is again the unoptimized fp32 baseline, as Section 4 left it.
del predictor, pytorch_predict_run
ONNX on the CUDA execution provider. This uses the same predictor and .onnx file as the latency rows.
map_results["ONNX CUDA EP"] = evaluate_coco_map(partial(onnx_predictor.predict, threshold=0.001), coco_subset)
The raw .trt fp16 engine through RFDETRInference, the same dynamic-batch engine and predictor as its latency rows, fed one image at a time.
map_results["TensorRT (raw .trt engine) RFDETRInference, fp16"] = evaluate_coco_map(
partial(trt_raw_predictor.predict, threshold=0.001), coco_subset
)
Then every inference-models model, the two managed TensorRT engines of Section 6 and the PyTorch and ONNX models of Section 6.1.
from tqdm.auto import tqdm
im_models["TensorRT inference-models auto"] = trt_model
im_models["TensorRT inference-models fp32"] = trt_fp32_model
for key, im_model in tqdm(im_models.items(), desc="COCO mAP, inference-models"):
map_results[key] = evaluate_coco_map(lambda img, m=im_model: im_predict(m, [np.array(img)], 0.001)[0], coco_subset)
print(f" {key}: mAP@50:95 {100 * map_results[key].map50_95:.1f} %")
One row per scored runtime. Read mAP@50:95 [%] alongside the batch-1 latency row for the same
format and config. A row far below the PyTorch fp32 baseline points to a precision or export
problem, not a faster model.
import pandas as pd
from IPython.display import display
map_rows = [
("PyTorch", Config("predict() eager", "fp32"), map_results["PyTorch predict() fp32"]),
("PyTorch", Config("inference() JIT", "fp16"), map_results["PyTorch inference() fp16+JIT"]),
("ONNX", Config("CUDA EP", "fp32"), map_results["ONNX CUDA EP"]),
("TensorRT", Config(".trt engine", "fp16"), map_results["TensorRT (raw .trt engine) RFDETRInference, fp16"]),
("PyTorch", Config("inference-models", "fp32"), map_results["PyTorch inference-models"]),
("ONNX", Config("inference-models CUDA EP", "fp32"), map_results["ONNX inference-models CUDA EP"]),
("TensorRT", Config("inference-models", "auto"), map_results["TensorRT inference-models auto"]),
("TensorRT", Config("inference-models", "fp32"), map_results["TensorRT inference-models fp32"]),
]
display(
pd.DataFrame(
[
{
"Format": format_label,
"Runtime": config.runtime,
"Precision": config.precision,
"mAP@50:95 [%]": round(100 * result.map50_95, 1),
"mAP@50 [%]": round(100 * result.map50, 1),
"Images": result.n_images,
}
for format_label, config, result in map_rows
]
)
)
8. TensorRT INT8¶
quantization="int8" builds an engine that runs most of the backbone and decoder matrix multiplies in INT8 and the rest in
FP16. It is static quantization: the export runs calibration_data through the FP32 graph with onnxruntime and records each
quantized tensor's largest absolute value (for a detection model it also smooths the backbone norms and corrects the
backbone MLP biases on the same images), so the calibration images decide the engine's accuracy and must look like
what the model will see in deployment. Out-of-domain images give an engine that loads, runs and is quietly less accurate.
Detection, segmentation and keypoint models; the batch is static (dynamic_batch=True is refused) and fp16=True. See the
INT8 section of the TensorRT export docs for what stays FP16 and why.
Nothing here is assumed to be faster or accurate. Both engines below are exported for batch 1, timed the same way, and scored on the same COCO subset as Section 7, so the FP16 row is the baseline for the INT8 row. The calibration images are held out of the scored images, so a drop from the FP16 row is the quantization itself. The last subsection, Batch 1, 4 and 8, end to end, repeats the latency with one static FP16 and INT8 engine per batch size.
This section is a quick check on the model chosen by MODEL_SIZE and TASK, so you can measure INT8 on your own GPU: batch 1,
calibration on N_CALIB val2017 images held out of the scored images, scoring on a seeded subset of N_EVAL val2017
images, and the mean of MEASURE_RUNS calls per runtime, FP16 then INT8 rather than interleaved. Expect the direction
of the docs page's INT8 tables, not their numbers. Section 8.1 is the benchmark
that produces those tables, for every detection, segmentation and keypoint model.
Calibration set, held out of the eval subset¶
N_CALIB images are drawn from COCO val2017 with a different seed than the eval subset, and any that fall inside the
subset are dropped, so no scored image was used for calibration. The directory is emptied first, so a rerun with other
settings cannot leave images from an earlier selection behind. The TensorRT and onnxruntime versions are printed with
the calibration source: the engine's INT8 placement and its calibration ranges depend on them.
import shutil
import tensorrt
# With FULL_VAL every val2017 image is scored, so none is left over to calibrate on.
assert not FULL_VAL, "Set FULL_VAL = False, or edit this cell to copy your own held-out images into CALIBRATION_DIR."
# Draw N_EVAL + N_CALIB images with a different seed: at least N_CALIB of them cannot be in the eval subset.
calibration_pool = fetch_coco_val2017(COCO_ROOT, n_images=N_EVAL + N_CALIB, seed=COCO_SEED + 1)
eval_ids = set(coco_subset.image_ids)
calibration_ids = [image_id for image_id in calibration_pool.image_ids if image_id not in eval_ids][:N_CALIB]
assert len(calibration_ids) == N_CALIB and eval_ids.isdisjoint(calibration_ids)
CALIBRATION_DIR = EXPORT_DIR / "calibration_images"
shutil.rmtree(CALIBRATION_DIR, ignore_errors=True)
CALIBRATION_DIR.mkdir()
for image_id in calibration_ids:
image_path = calibration_pool.images_dir / f"{image_id:012d}.jpg"
shutil.copy(image_path, CALIBRATION_DIR / image_path.name)
print(f"Calibration images: {len(calibration_ids)} in {CALIBRATION_DIR}, disjoint from the {len(eval_ids)} eval images")
print(
f"TensorRT {tensorrt.__version__}, onnxruntime {ort.__version__}, "
f"calibration: {N_CALIB} COCO val2017 images (selection seed {COCO_SEED + 1})"
)
Export¶
Two static batch-1 engines: FP16 as the baseline, and INT8 calibrated on CALIBRATION_DIR. Each goes to its own
directory, so neither overwrites the Section 5 ONNX file or the Section 6 engine of the same name. max_images=N_CALIB
makes the export use every copied image (the default reads the first 100 by file name). The INT8 build also runs the
calibration, so it takes noticeably longer than the FP16 one. The INT8 file is named *_int8.trt; a release without
INT8 TensorRT export would write an FP16 engine instead, which the assert below catches.
trt_fp16_b1_path = model.export(format="trt", output_dir=str(EXPORT_DIR / "trt_fp16_b1"))
trt_int8_path = model.export(
format="trt",
quantization="int8",
calibration_data=CALIBRATION_DIR,
max_images=N_CALIB,
output_dir=str(EXPORT_DIR / "trt_int8_b1"),
)
assert Path(trt_int8_path).name.endswith("_int8.trt"), f"Expected an INT8 engine, got {trt_int8_path}"
print(f"TensorRT FP16 (batch 1): {trt_fp16_b1_path} ({_artifact_size_mb(trt_fp16_b1_path):.1f} MB)")
print(f"TensorRT INT8 (batch 1): {trt_int8_path} ({_artifact_size_mb(trt_int8_path):.1f} MB)")
Inference and latency¶
Each engine is timed two ways at batch 1, end to end through predict() (preprocessing, engine call, decoding), default call first:
- Default call:
RFDETRInference(path, device="cuda"), the runtime of Section 6, which launches withexecute_v2and returns once the engine has run. It includes the CPU launch time of every call. - CUDA graph:
RFDETRInference(path, device="cuda", runtime_options={"cuda_graph": True}), which captures the engine's launch into a CUDA graph on its first call and replays it on every later call instead of launching each kernel from the CPU. How much that saves depends on how long the launches take on your host; where they are fast, the two can be close. It needs an engine with fixed input shapes, which a static batch-1 engine is.
Both include preprocessing and decoding, so these numbers are larger than the engine-only GPU times of Section 8.1, and the
docs page's CUDA-graph column replays the same kind of graph 200 times back to back with the input already on the device.
Do not set them side by side. Memory [MB] covers constructing the default-call predictor plus its first call; the CUDA-graph
predictor's memory is not measured here.
trt_b1 = {}
trt_b1_results = {}
trt_b1_graph_results = {}
trt_b1_memory_mb = {}
for name, path in (("FP16", trt_fp16_b1_path), ("INT8", trt_int8_path)):
with measure_memory(device="cuda") as mem:
trt_b1[name] = RFDETRInference(path, device="cuda")
trt_b1[name].predict(frame, include_source_image=False)
trt_b1_memory_mb[name] = mem.delta_mb
trt_b1_results[name] = measure_latency(
partial(trt_b1[name].predict, frame, include_source_image=False),
label=f"TensorRT {name} default call batch=1",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
# The first call of a cuda_graph predictor captures the graph, so it is made here, before any timed call.
graph_predictor = RFDETRInference(path, device="cuda", runtime_options={"cuda_graph": True})
graph_predictor.predict(frame, include_source_image=False)
trt_b1_graph_results[name] = measure_latency(
partial(graph_predictor.predict, frame, include_source_image=False),
label=f"TensorRT {name} CUDA graph batch=1",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
del graph_predictor
default_call, graph_call = trt_b1_results[name], trt_b1_graph_results[name]
print(
f" {name} default call {default_call.mean_ms:6.2f} ms ± {default_call.std_ms:5.2f}"
f" CUDA graph {graph_call.mean_ms:6.2f} ms ± {graph_call.std_ms:5.2f} engine {_artifact_size_mb(path):.1f} MB"
)
A correctness call before scoring, on the sample image. INT8 can move a detection to another query or change a low score a lot without changing what is detected. The detection counts, how well each FP16 box is matched by an INT8 one, and the COCO mAP after them are the accuracy check; a detection near the threshold can land on either side of it, which the printed FP16 scores show.
fp16_detections = trt_b1["FP16"].predict(image, threshold=CONFIDENCE_THRESHOLD)
int8_detections = trt_b1["INT8"].predict(image, threshold=CONFIDENCE_THRESHOLD)
print(f"Detections above {CONFIDENCE_THRESHOLD}: FP16 {len(fp16_detections)}, INT8 {len(int8_detections)}")
print(f"FP16 scores: {[round(float(score), 2) for score in fp16_detections.confidence]}")
if len(fp16_detections) and len(int8_detections):
best_iou = sv.box_iou_batch(fp16_detections.xyxy, int8_detections.xyxy).max(axis=1)
print(f"Best IoU of each FP16 detection with an INT8 one: {[round(float(iou), 2) for iou in best_iou]}")
COCO val2017 accuracy, FP16 against INT8¶
The same harness and the same images as Section 7, through the default-call predictors, so the INT8 row compares directly with the FP16 row.
map_results["TensorRT (raw .trt engine) RFDETRInference, fp16, batch 1"] = evaluate_coco_map(
partial(trt_b1["FP16"].predict, threshold=0.001), coco_subset
)
map_results["TensorRT (raw .trt engine) RFDETRInference, int8, batch 1"] = evaluate_coco_map(
partial(trt_b1["INT8"].predict, threshold=0.001), coco_subset
)
FP16 against INT8 on this machine: accuracy, end-to-end latency from both call modes, engine size and device memory, and the INT8 speedup of each call mode, the FP16 latency divided by the INT8 latency measured the same way. Measure the trade on your own data and GPU before deploying: the gain depends on the GPU, the model size and how much of the call is launch time.
int8_rows = []
for name in ("FP16", "INT8"):
map_result = map_results[f"TensorRT (raw .trt engine) RFDETRInference, {name.lower()}, batch 1"]
path = trt_fp16_b1_path if name == "FP16" else trt_int8_path
int8_rows.append(
{
"Precision": name,
"mAP@50:95 [%]": round(100 * map_result.map50_95, 1),
"mAP@50 [%]": round(100 * map_result.map50, 1),
"Default call [ms]": round(trt_b1_results[name].mean_ms, 2),
"CUDA graph [ms]": round(trt_b1_graph_results[name].mean_ms, 2),
"Engine [MB]": round(_artifact_size_mb(path), 1),
"Memory [MB]": round(trt_b1_memory_mb[name], 1),
}
)
display(pd.DataFrame(int8_rows).set_index("Precision"))
default_speedup = trt_b1_results["FP16"].mean_ms / trt_b1_results["INT8"].mean_ms
graph_speedup = trt_b1_graph_results["FP16"].mean_ms / trt_b1_graph_results["INT8"].mean_ms
print(f"INT8 speedup at batch 1: default call {default_speedup:.2f}x, CUDA graph {graph_speedup:.2f}x")
Batch 1, 4 and 8, end to end¶
The docs page's latency tables across NVIDIA GPUs time the .trt engines end to end at batch 1 and 8. An INT8 engine has a
static batch, so this cell exports one static FP16 and one static INT8 engine per batch in BATCH_SIZES, the INT8 ones
calibrated on the same CALIBRATION_DIR as above, and times predict() on batch copies of the sample frame with the
default call and include_source_image=False: WARMUP_RUNS warmup and MEASURE_RUNS timed calls, and throughput is the
batch divided by the mean latency. Batch 1 reuses the engines exported above. Run it once per MODEL_SIZE on each GPU to
reproduce the static .trt FP16 and INT8 rows of those tables; every INT8 build calibrates again, so each takes minutes.
trt_batch_engines = {(1, "FP16"): trt_fp16_b1_path, (1, "INT8"): trt_int8_path}
for batch in BATCH_SIZES[1:]:
trt_batch_engines[batch, "FP16"] = model.export(
format="trt", batch_size=batch, output_dir=str(EXPORT_DIR / f"trt_fp16_b{batch}")
)
trt_batch_engines[batch, "INT8"] = model.export(
format="trt",
batch_size=batch,
quantization="int8",
calibration_data=CALIBRATION_DIR,
max_images=N_CALIB,
output_dir=str(EXPORT_DIR / f"trt_int8_b{batch}"),
)
int8_name = Path(trt_batch_engines[batch, "INT8"]).name
assert int8_name.endswith("_int8.trt"), f"Expected an INT8 engine, got {int8_name}"
trt_batch_results: dict[tuple[int, str], BenchmarkResult] = {}
for (batch, name), path in trt_batch_engines.items():
images_batch = [frame] * batch
predictor = RFDETRInference(path, device="cuda")
predictor.predict(images_batch, include_source_image=False)
trt_batch_results[batch, name] = measure_latency(
partial(predictor.predict, images_batch, include_source_image=False),
label=f"TensorRT {name} default call batch={batch}",
device="cuda",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
del predictor
r = trt_batch_results[batch, name]
print(f" {name} batch={batch} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} img/s)")
torch.cuda.empty_cache()
8.1 Benchmark for the docs tables¶
The same comparison for every model of the TensorRT export docs
tables: the four detection models, the four segmentation models and RFDETRKeypointPreview. Each gets an FP16 and an
INT8 engine per batch in BATCH_SIZES, the COCO AP (box AP for detection, mask AP for segmentation, keypoint AP for
keypoints, and the box AP next to the last two), the engine size, and the latency of both engines, timed with the
input already on the GPU. The defaults below are a quick run, not those tables:
| Setting | Default here | Docs tables |
|---|---|---|
BENCH_MODELS |
"Detection Small" |
all nine models |
BENCH_AP_IMAGES |
N_EVAL, the first images by file name |
5000, all of val2017 |
BENCH_CALIBRATION |
"val2017", the last N_CALIB images by file name, disjoint from the scored ones |
"train2017", the first N_CALIB images by file name, with N_CALIB = 128 |
BENCH_ROUNDS, BENCH_RUNS |
5, 100 |
15, 200 |
Latency is the median over BENCH_ROUNDS rounds of BENCH_RUNS executions per engine, FP16 and INT8 alternating in each
round and the engine that goes first rotating, once replaying a captured CUDA graph and once calling
execute_async_v3 back to back without synchronizing in between (the plain call). It is GPU time per image, so a
batch-8 row is the cost of one image. AP is scored at batch 1 only. Set BENCH_CALIBRATION = "train2017" to calibrate
as the docs tables do (delete int8_bench/engines first if you built engines with another setting, because a rerun reuses them): it parses the 450 MB instances_train2017.json that Section 7 already extracted and downloads
N_CALIB images from train2017. Nine models at three batches build 54 engines, and the INT8 builds calibrate on the
CPU, which takes minutes each on Colab's two vCPUs and host memory that grows with the batch. The numbers depend on
the GPU and the TensorRT version, which are printed with the results; the docs tables used TensorRT 11.4.0.106.
# The defaults are a quick run. The tables of the docs use every model, 5000 images, train2017 calibration and 15 x 200.
BENCH_MODELS = ("Detection Small",)
BENCH_AP_IMAGES = N_EVAL
BENCH_CALIBRATION = "val2017" # "train2017" calibrates as the docs tables do
BENCH_ROUNDS, BENCH_RUNS, BENCH_WARMUP = 5, 100, 50
import itertools
import json
import shutil
import time
from functools import lru_cache
from urllib.parse import quote
import tensorrt as trt
from rfdetr import (
RFDETRKeypointPreview,
RFDETRLarge,
RFDETRMedium,
RFDETRNano,
RFDETRSegLarge,
RFDETRSegMedium,
RFDETRSegNano,
RFDETRSegSmall,
RFDETRSmall,
)
from rfdetr.export._benchmark import _COCO_VAL2017_IMAGE_URL, _download
BENCH_CLASSES = {
"Detection Nano": RFDETRNano,
"Detection Small": RFDETRSmall,
"Detection Medium": RFDETRMedium,
"Detection Large": RFDETRLarge,
"Seg Nano": RFDETRSegNano,
"Seg Small": RFDETRSegSmall,
"Seg Medium": RFDETRSegMedium,
"Seg Large": RFDETRSegLarge,
"Keypoint Preview": RFDETRKeypointPreview,
}
BENCH_KIND = {
name: "keypoints" if name.startswith("Keypoint") else "segm" if name.startswith("Seg") else "bbox"
for name in BENCH_CLASSES
}
assert set(BENCH_MODELS) <= set(BENCH_CLASSES), f"BENCH_MODELS must be names from {sorted(BENCH_CLASSES)}"
assert BENCH_CALIBRATION in ("val2017", "train2017")
BENCH_DIR = EXPORT_DIR / "int8_bench"
BENCH_ANNOTATIONS = {
"bbox": COCO_ROOT / "annotations" / "instances_val2017.json",
"segm": COCO_ROOT / "annotations" / "instances_val2017.json",
"keypoints": COCO_ROOT / "annotations" / "person_keypoints_val2017.json",
}
# The scored images of each task: the first BENCH_AP_IMAGES val2017 images by file name.
bench_images = {
kind: sorted(json.loads(path.read_text())["images"], key=lambda image: image["file_name"])[:BENCH_AP_IMAGES]
for kind, path in BENCH_ANNOTATIONS.items()
}
val_records = sorted(json.loads(BENCH_ANNOTATIONS["bbox"].read_text())["images"], key=lambda image: image["file_name"])
if BENCH_CALIBRATION == "val2017":
# The last N_CALIB val2017 images by file name, so none of them is among the scored ones.
calibration_names = [image["file_name"] for image in val_records[-N_CALIB:]]
calibration_origin = _COCO_VAL2017_IMAGE_URL
assert BENCH_AP_IMAGES <= len(val_records) - N_CALIB, (
"No val2017 image is left to calibrate on; lower BENCH_AP_IMAGES."
)
else:
train_records = json.loads((COCO_ROOT / "annotations" / "instances_train2017.json").read_text())["images"]
calibration_names = [
image["file_name"] for image in sorted(train_records, key=lambda image: image["file_name"])[:N_CALIB]
]
calibration_origin = "http://images.cocodataset.org/train2017/"
del train_records
# Scored images come from COCO_ROOT / "val2017"; fetch the ones Section 7 did not.
for task_images in bench_images.values():
for record in task_images:
path = COCO_ROOT / "val2017" / record["file_name"]
if not path.exists():
_download(_COCO_VAL2017_IMAGE_URL + quote(record["file_name"]), path)
BENCH_CALIBRATION_DIR = BENCH_DIR / "calibration"
shutil.rmtree(BENCH_CALIBRATION_DIR, ignore_errors=True)
BENCH_CALIBRATION_DIR.mkdir(parents=True)
for name in calibration_names:
_download(calibration_origin + quote(name), BENCH_CALIBRATION_DIR / name)
print(
f"{len(calibration_names)} calibration images ({BENCH_CALIBRATION}), {len(bench_images['bbox'])} scored images (val2017), "
f"TensorRT {trt.__version__}, onnxruntime {ort.__version__}"
)
Build the engines¶
A static FP16 and a static INT8 engine per model and batch, each in its own directory. A rerun reuses the engines
already built; delete int8_bench/engines to build them again.
BENCH_PRECISIONS = {
"FP16": {"fp16": True},
"INT8": {
"quantization": "int8",
"calibration_data": str(BENCH_CALIBRATION_DIR),
"max_images": len(calibration_names),
},
}
def bench_grid():
"""Every (model, batch, precision) the benchmark covers, in the order of the results table."""
return itertools.product(BENCH_MODELS, BATCH_SIZES, BENCH_PRECISIONS)
@lru_cache(maxsize=1) # the grid is grouped by model, so one live model at a time is enough
def bench_model(model_name: str):
"""The model every engine of *model_name* is exported from."""
return BENCH_CLASSES[model_name]()
bench_engines, bench_sizes_mb = {}, {} # keyed by (model, batch, precision), the order of `bench_grid()`
for model_name, batch, precision in bench_grid():
out_dir = BENCH_DIR / "engines" / model_name.replace(" ", "_") / f"b{batch}" / precision
built = sorted(out_dir.glob("*.trt")) if out_dir.exists() else []
started = time.time()
path = (
built[0]
if built
else Path(
bench_model(model_name).export(
output_dir=str(out_dir),
format="tensorrt",
batch_size=batch,
verbose=False,
**BENCH_PRECISIONS[precision],
)
)
)
if precision == "INT8":
assert path.name.endswith("_int8.trt"), f"Expected an INT8 engine, got {path}"
bench_engines[model_name, batch, precision] = path
bench_sizes_mb[model_name, batch, precision] = _artifact_size_mb(path)
print(
f"{model_name:16} batch {batch:>2} {precision}: {path.name} "
f"({bench_sizes_mb[model_name, batch, precision]:.0f} MB"
f"{'' if built else f', built in {time.time() - started:.0f} s'})",
flush=True,
)
bench_keypoints_per_class = {
model_name: list(getattr(bench_model(model_name).model_config, "num_keypoints_per_class", []) or [])
for model_name in BENCH_MODELS
}
bench_model.cache_clear()
torch.cuda.empty_cache()
Latency¶
A minimal runner over the engine's own buffers: it copies the input in, calls execute_async_v3, and either times
BENCH_RUNS back-to-back calls or replays a captured CUDA graph the same number of times. The time is GPU time from
CUDA events divided by the batch, so a batch-8 row is the cost of one image.
BENCH_TORCH_DTYPES = {
trt.DataType.FLOAT: torch.float32,
trt.DataType.HALF: torch.float16,
trt.DataType.INT32: torch.int32,
trt.DataType.INT64: torch.int64,
trt.DataType.BOOL: torch.bool,
}
class BenchEngine:
"""One TensorRT engine with its input and output buffers allocated on the GPU."""
def __init__(self, path: Path) -> None:
self.engine = trt.Runtime(trt.Logger(trt.Logger.ERROR)).deserialize_cuda_engine(Path(path).read_bytes())
self.context, self.stream, self.io = self.engine.create_execution_context(), torch.cuda.Stream(), {}
for index in range(self.engine.num_io_tensors):
name = self.engine.get_tensor_name(index)
tensor = torch.zeros(
tuple(self.engine.get_tensor_shape(name)),
dtype=BENCH_TORCH_DTYPES[self.engine.get_tensor_dtype(name)],
device="cuda",
)
self.io[name] = tensor
self.context.set_tensor_address(name, tensor.data_ptr())
self.batch, self.graph = self.io["input"].shape[0], None
def __call__(self, batch_input: torch.Tensor | None = None) -> dict[str, torch.Tensor]:
"""Run the engine once, synchronously, and return its output buffers."""
if batch_input is not None:
self.io["input"].copy_(batch_input)
self.stream.wait_stream(torch.cuda.current_stream()) # the copy above runs on the current stream
with torch.cuda.stream(self.stream):
self.context.execute_async_v3(self.stream.cuda_stream)
self.stream.synchronize()
return self.io
def ms_per_image(self, runs: int, cuda_graph: bool) -> float:
"""GPU time of *runs* back-to-back executions, in milliseconds per image."""
if cuda_graph and self.graph is None:
self.graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(self.graph, stream=self.stream):
self.context.execute_async_v3(self.stream.cuda_stream)
start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
with torch.cuda.stream(self.stream):
start.record(self.stream)
for _ in range(runs):
if cuda_graph:
self.graph.replay()
else:
self.context.execute_async_v3(self.stream.cuda_stream)
end.record(self.stream)
end.synchronize()
return start.elapsed_time(end) / runs / self.batch
BENCH_MODES = (("CUDA graph", True), ("plain call", False))
@lru_cache(maxsize=1) # the grid below visits both modes of one (model, batch) in a row, so the engines load once
def bench_runners(model_name: str, batch: int) -> dict[str, BenchEngine]:
"""Both engines of one (model, batch), loaded and warmed up on the same random input."""
runners = {precision: BenchEngine(bench_engines[model_name, batch, precision]) for precision in BENCH_PRECISIONS}
sample = torch.randn(runners["FP16"].io["input"].shape, device="cuda")
for runner in runners.values():
runner.io["input"].copy_(sample)
for _ in range(BENCH_WARMUP):
runner()
return runners
def interleaved_medians(runners: dict[str, BenchEngine], cuda_graph: bool) -> dict[str, float]:
"""Median ms per image of each engine over BENCH_ROUNDS rounds, rotating which engine goes first."""
names = list(runners)
times = {name: [] for name in names}
for shift in range(1, BENCH_ROUNDS + 1):
for name in names[shift % len(names) :] + names[: shift % len(names)]:
times[name].append(runners[name].ms_per_image(BENCH_RUNS, cuda_graph))
return {name: float(np.median(samples)) for name, samples in times.items()}
def bench_latency_rows():
"""One row per (model, batch, mode, precision) with its median latency."""
for model_name, batch, (mode, cuda_graph) in itertools.product(BENCH_MODELS, BATCH_SIZES, BENCH_MODES):
medians = interleaved_medians(bench_runners(model_name, batch), cuda_graph)
yield from (
{"model": model_name, "precision": precision, "batch": batch, "mode": mode, "ms": ms}
for precision, ms in medians.items()
)
bench_latency = pd.DataFrame(bench_latency_rows())
bench_runners.cache_clear()
torch.cuda.empty_cache()
display(
bench_latency.pivot_table(index=["model", "mode", "batch"], columns="precision", values="ms", sort=False).round(3)
)
Accuracy¶
COCO box AP, and the mask or keypoint AP of the segmentation and keypoint models, of every engine at batch 1 on the
scored images, each image decoded once and fed to both engines. Post-processing is the library's own PostProcess,
the one predict() uses, so a box, mask or keypoint is scored as the model reports it. Box AP and mask AP come from
instances_val2017; box AP (person class only) and keypoint AP from person_keypoints_val2017.
from faster_coco_eval import COCO
from rfdetr.evaluation.coco_eval import CocoEvaluator
from rfdetr.export.benchmark import infer_transforms
from rfdetr.models.postprocess import PostProcess
def bench_coco_ap(model_name: str) -> dict[str, dict[str, float]]:
"""Box AP and mask or keypoint AP of the FP16 and INT8 batch-1 engines of *model_name* on the scored images."""
kind, images = BENCH_KIND[model_name], bench_images[BENCH_KIND[model_name]]
runners = {name: BenchEngine(bench_engines[model_name, 1, name]) for name in BENCH_PRECISIONS}
transforms = infer_transforms(tuple(runners["FP16"].io["input"].shape[-2:]))
postprocess = PostProcess(
num_select=runners["FP16"].io["dets"].shape[1], num_keypoints_per_class=bench_keypoints_per_class[model_name]
)
# A detection model has nothing to score besides its boxes.
iou_types = ["bbox"] if kind == "bbox" else ["bbox", kind]
extra_outputs = {"segm": ("pred_masks", "masks"), "keypoints": ("pred_keypoints", "keypoints")}.get(kind)
evaluators = {name: CocoEvaluator(COCO(str(BENCH_ANNOTATIONS[kind])), iou_types) for name in runners}
for record in images:
with Image.open(COCO_ROOT / "val2017" / record["file_name"]) as opened:
rgb = opened.convert("RGB")
sample = transforms(rgb, None)[0][None].cuda()
target = torch.tensor([[rgb.height, rgb.width]], dtype=torch.float32, device="cuda")
for name, runner in runners.items():
outputs = runner(sample)
predictions = {"pred_logits": outputs["labels"].float(), "pred_boxes": outputs["dets"].float()}
if extra_outputs:
predictions[extra_outputs[0]] = outputs[extra_outputs[1]].float()
results = postprocess(predictions, target)
evaluators[name].update({record["id"]: results[0]})
scores = {}
for name, evaluator in evaluators.items():
evaluator.synchronize_between_processes()
evaluator.accumulate()
evaluator.summarize()
scores[name] = {"Box AP": 100 * float(evaluator.coco_eval["bbox"].stats[0])}
if kind != "bbox":
scores[name]["Mask or keypoint AP"] = 100 * float(evaluator.coco_eval[kind].stats[0])
del runners
torch.cuda.empty_cache()
return scores
bench_accuracy = {model_name: bench_coco_ap(model_name) for model_name in BENCH_MODELS}
Results¶
One row per engine. Δ AP is INT8 minus FP16 in AP points: of the box AP for detection and of the mask or keypoint AP
for the other models. The speedup divides the FP16 latency by the INT8
latency in the same mode, so the plain-call and CUDA-graph columns have different FP16 bases: a plain call adds CPU
launch time that a graph replay removes. AP was scored at batch 1 only. The table is also written to
int8_bench/results.json.
ms = bench_latency.set_index(["model", "mode", "batch", "precision"])["ms"]
def bench_row(model_name: str, batch: int, precision: str) -> dict:
"""One row of the results table; AP is scored at batch 1 only."""
scores = bench_accuracy[model_name] if batch == 1 else None
ap_key = "Box AP" if BENCH_KIND[model_name] == "bbox" else "Mask or keypoint AP"
return {
"Model": model_name,
"Batch": batch,
"Precision": precision,
"Box AP": scores[precision]["Box AP"] if scores else None,
"Mask or keypoint AP": scores[precision].get("Mask or keypoint AP") if scores else None,
"Δ AP": scores[precision][ap_key] - scores["FP16"][ap_key] if scores else None,
"CUDA graph [ms]": ms[model_name, "CUDA graph", batch, precision],
"Speedup, CUDA graph": ms[model_name, "CUDA graph", batch, "FP16"]
/ ms[model_name, "CUDA graph", batch, precision],
"Plain call [ms]": ms[model_name, "plain call", batch, precision],
"Speedup, plain call": ms[model_name, "plain call", batch, "FP16"]
/ ms[model_name, "plain call", batch, precision],
"Engine [MB]": bench_sizes_mb[model_name, batch, precision],
}
bench_results = pd.DataFrame(bench_row(*cell) for cell in bench_grid()).set_index(["Model", "Batch", "Precision"])
display(bench_results.round(3))
(BENCH_DIR / "results.json").write_text(
json.dumps(
{
"gpu": torch.cuda.get_device_name(0),
"tensorrt": trt.__version__,
"ap_images": len(bench_images["bbox"]),
"calibration": BENCH_CALIBRATION,
"rounds": BENCH_ROUNDS,
"runs": BENCH_RUNS,
"rows": bench_results.reset_index().to_dict("records"),
},
indent=1,
)
)
print(
f"{torch.cuda.get_device_name(0)}, TensorRT {trt.__version__}, {len(bench_images['bbox'])} scored images, "
f"calibration {BENCH_CALIBRATION} ({len(calibration_names)} images), {BENCH_ROUNDS} rounds of {BENCH_RUNS} executions"
)
9. Throughput on a folder¶
The batch rows call predict() on one image repeated. A real job predicts a folder, and on a GPU the input side decides how close a runtime gets to its model-only speed. predict(folder) reads and decodes images on worker threads ahead of the model and runs them in batches:
- A runtime with a dynamic batch axis (the ONNX file, the
max_batch_size=8TensorRT engine of Section 6) runs up to eight images per call; a fixed batch-1 engine runs one. A CUDA graph takes the batch it was captured for (batch=8here). - ONNX Runtime with the CUDA provider prepares each batch on the GPU.
- With
include_source_image=False, baseline colour JPEG files decode on the GPU in one nvjpeg call per batch. The first file of each kind is checked against the CPU decoder, and a GPU that differs falls back to the CPU, so the model sees the same pixels either way. The.trtengine is timed with and without the source image kept to show what that is worth here.
N_FOLDER eval images are copied into one folder and the median of FOLDER_REPS timed passes (after one warm-up pass) gives images per second over the whole call.
import shutil
import time
from tqdm.auto import tqdm
N_FOLDER = 256 # a multiple of 8, so every batch of the batch-8 predictors is full
FOLDER_REPS = 3
from tempfile import TemporaryDirectory
# Own the benchmark images without deleting a pre-existing export folder.
folder_workspace = TemporaryDirectory(prefix="rfdetr-folder-")
FOLDER_DIR = Path(folder_workspace.name)
for image_id in tqdm(coco_subset.image_ids[:N_FOLDER], desc="Copy images"):
shutil.copy(coco_subset.images_dir / f"{image_id:012d}.jpg", FOLDER_DIR)
assert len(list(FOLDER_DIR.glob("*.jpg"))) == N_FOLDER
folder_rows: list[dict] = []
def measure_folder(
format_label: str, config: Config, predictor, include_source_image: bool = False, **predict_kwargs
) -> None:
"""Time `predictor.predict(folder)` end to end and record the median images per second."""
kwargs = {"threshold": CONFIDENCE_THRESHOLD, "include_source_image": include_source_image, **predict_kwargs}
predictor.predict(str(FOLDER_DIR), **kwargs) # warm-up pass
seconds = []
for _ in range(FOLDER_REPS):
start = time.perf_counter()
results = predictor.predict(str(FOLDER_DIR), **kwargs)
seconds.append(time.perf_counter() - start)
assert len(results) == N_FOLDER
img_s = N_FOLDER / sorted(seconds)[len(seconds) // 2]
folder_rows.append(
{
"Format": format_label,
"Runtime": config.runtime,
"Precision": config.precision,
"Source image kept": include_source_image,
"Folder [img/s]": img_s,
}
)
print(f" {format_label} ({', '.join(config)}, source image kept: {include_source_image}): {img_s:7.1f} img/s")
cudagraph_predictor = model.inference(compile_backend="cudagraph", batch_size=8, dtype=torch.float16)
measure_folder("PyTorch", Config("inference() CUDA Graph", "fp16"), cudagraph_predictor, batch=8)
del cudagraph_predictor
measure_folder("ONNX", Config("CUDA EP", "fp32"), onnx_predictor)
for kept in (True, False):
measure_folder("TensorRT", Config(".trt engine", "fp16"), trt_raw_predictor, include_source_image=kept)
10. Results¶
The last cell prints a decision table (one row per format and config: batch-1 latency, speed against PyTorch predict(), size, mAP@50:95 and FPS at each batch size) and a detail table with ± std and memory growth. Every latency is the end-to-end predict() call, and mAP@50:95 is scored on the same artifact that was timed (every inference-models row is scored as well). Memory [MB] is device-memory growth; the batch-4 and batch-8 raw-engine rows can read 0.0 because the engine sizes its output buffers for the batch-8 profile maximum when it is loaded. Each CUDA Graph row is its own predictor, captured for that batch size; its memory is the device-memory growth of the capture. All inference-models rows are scored.
This notebook times one model (MODEL_SIZE and TASK, Small detection by default) on one GPU. The cross-GPU table for all four detection sizes on the T4, L4, A100 and RTX PRO 6000 is not produced by this notebook in one run; run Section 8 once per MODEL_SIZE on each GPU to reproduce its static .trt FP16 and INT8 rows. It is on the TensorRT export page. The static FP16 and INT8 .trt engines of Section 8, timed there at batch 1, 4 and 8, are not in this table; the INT8 numbers produced by Section 8.1 are in the TensorRT INT8 section of the docs.
import pandas as pd
from IPython.display import display
KEYS = ["Format", "Runtime", "Precision", "Quantization"]
# (Format, Config, latency by batch, memory by batch, artifact size, key into map_results or None)
entries = [
("PyTorch", Config("predict() eager", "fp32"), pytorch_fp32_results, {}, None, "PyTorch predict() fp32"),
(
"PyTorch",
Config("inference() JIT", "fp16"),
pytorch_fp16_jit_results,
pytorch_fp16_jit_memory_mb,
None,
"PyTorch inference() fp16+JIT",
),
(
"PyTorch",
Config("inference() CUDA Graph", "fp16"),
pytorch_fp16_cudagraph_results,
pytorch_fp16_cudagraph_memory_mb,
None,
None,
),
(
"PyTorch",
Config("inference-models", "fp32"),
im_results["PyTorch inference-models"],
im_memory_mb["PyTorch inference-models"],
None,
"PyTorch inference-models",
),
("ONNX", Config("CUDA EP", "fp32"), onnx_end2end_results, onnx_memory_by_batch, onnx_size_mb, "ONNX CUDA EP"),
(
"ONNX",
Config("inference-models CUDA EP", "fp32"),
im_results["ONNX inference-models CUDA EP"],
im_memory_mb["ONNX inference-models CUDA EP"],
None,
"ONNX inference-models CUDA EP",
),
(
"TensorRT",
Config("inference-models", "auto"),
trt_end2end_results,
trt_memory_mb,
None,
"TensorRT inference-models auto",
),
(
"TensorRT",
Config("inference-models", "fp32"),
trt_fp32_end2end_results,
trt_fp32_memory_mb,
None,
"TensorRT inference-models fp32",
),
(
"TensorRT",
Config(".trt engine", "fp16"),
trt_raw_end2end_results,
trt_raw_memory_mb,
trt_raw_engine_size_mb,
"TensorRT (raw .trt engine) RFDETRInference, fp16",
),
]
rows = []
for format_label, config, results, memory, size, _ in entries:
for batch, result in results.items():
rows.append(
{
"Format": format_label,
"Runtime": config.runtime,
"Precision": config.precision,
"Quantization": config.quantization,
"Batch": batch,
"end2end [ms]": result.mean_ms,
"± [ms]": result.std_ms,
"FPS [img/s]": batch * result.fps,
"Memory [MB]": memory.get(batch),
"Size [MB]": size,
}
)
detail = pd.DataFrame(rows)
# Decision table: one row per (format, config); batch-1 latency, accuracy and size, throughput at every batch.
fps_by_batch = detail.pivot_table(index=KEYS, columns="Batch", values="FPS [img/s]", sort=False).add_prefix("FPS b")
map_by_row = pd.Series(
{(format_label, *config): 100 * map_results[key].map50_95 for format_label, config, *_, key in entries if key},
name="mAP@50:95 [%]",
)
map_by_row.index.names = KEYS
headline = (
detail[detail["Batch"] == 1]
.set_index(KEYS)[["end2end [ms]", "Size [MB]"]]
.rename(columns={"end2end [ms]": "b1 latency [ms]"})
.join(map_by_row)
.join(fps_by_batch)
)
headline.insert(1, "Speed vs PyTorch ×", headline["b1 latency [ms]"].iloc[0] / headline["b1 latency [ms]"])
# One number format for every table in this section: one decimal, "—" where a row has no value.
decision = headline.apply(lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}"))
display(decision)
Markdown for the docs page: the table above, unchanged.
print(
decision.reset_index().to_markdown(index=False, disable_numparse=True)
) # keep the one-decimal strings as they are
Per-batch detail: ± std and memory growth for every row.
numeric = detail.columns.difference(["Format", "Runtime", "Precision", "Quantization", "Batch"])
detail_table = detail.copy()
detail_table[numeric] = detail[numeric].apply(
lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}")
)
print(detail_table.to_markdown(index=False))
print(
f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, GPU: {torch.cuda.get_device_name(0)}."
)
Folder throughput of Section 9, images per second over a whole predict(folder) call:
folder = pd.DataFrame(folder_rows)
folder["Folder [img/s]"] = folder["Folder [img/s]"].map(lambda value: f"{value:.1f}")
display(folder)
Next steps¶
- Fine-tuned weights — pass
pretrain_weights="<path/to/checkpoint.pth>"when constructing the model. - Not on a CUDA GPU? — see the CPU cookbook, the mobile/edge cookbook, or the Apple Silicon cookbook.
- See the Export documentation for every format and option.