RF-DETR on CPU — Export, Inference & Latency¶
Compares every export path that targets a general-purpose CPU (x86 or ARM desktop/server — no GPU, no NPU): eager PyTorch as the reference anchor, ONNX Runtime with the CPU execution provider, and OpenVINO IR.
| Format | Export | Inference |
|---|---|---|
| PyTorch | (no export — eager model) | predict() / inference() (TorchScript JIT, fp32) |
| ONNX (CPU EP) | model.export() → .onnx |
RFDETRInference(path).predict() |
| OpenVINO | model.export(format="openvino") → .xml + .bin |
RFDETRInference(path).predict() |
inference-models |
(prebuilt by that package) | PyTorch and ONNX Runtime (CPU execution provider) through AutoModel.from_pretrained() |
Every format is benchmarked at batch 1 and batch 4 — see
src/rfdetr/export/registry.py
for which formats can hold one artifact across batch sizes and which cannot: ONNX bakes a
dynamic batch axis (supports_dynamic_batch=True), so one .onnx file serves both rows below;
OpenVINO's converted IR already has a dynamic input shape ([?,?,?,?]) — a batch-1 IR runs batch
4 with identical outputs — so it needs no separate dynamic-batch mode either. This notebook still
exports one OpenVINO IR per batch size only to mirror the per-batch artifacts the other cookbooks'
fixed-shape formats need. PyTorch needs no export step and trivially takes either batch as a
(N, C, H, W) tensor.
Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), mobile/edge formats — TFLite, LiteRT, ExecuTorch's XNNPACK backend (see the mobile cookbook) — or Apple Silicon (see the Apple cookbook). OpenVINO can also target Intel GPUs and NPUs; this notebook only exercises its
"CPU"device.
Which latency number is this? Every number in this notebook is a Tier C measurement: end-to-end
predict()timings (preprocess + forward + postprocess) for each export format, on one shared CPU, against a PyTorch anchor measured on that same machine. Its value is ratios within this machine — never set these milliseconds against the architecture headline on the Benchmarks page (NVIDIA T4, TensorRT FP16, model only) or the number on Roboflow's RF-DETR model page (L4, Roboflow Inference, scope not stated, batch 1) — those are GPU numbers, not comparable to a CPU run at all, let alone without checking precision, batch size, and scope.
Every exported artifact is loaded through the one public prediction API, RFDETRInference(path), and timed through the same predict() call as the PyTorch rows, so no format needs its own tensor handling, decoder or parity helper.
1. Install¶
RFDETR_SPEC holds the install source and defaults to the develop branch. RFDETRInference and the
rfdetr.export.benchmark helpers this notebook imports ship in 1.12; to install from PyPI instead, swap in the
commented >=1.12,<2 line once it is released.
No CUDA-specific packages needed.
The last line of the cell puts Pillow back to the version Colab already loaded: inference-models asks for a newer Pillow, and pip replacing it under a running kernel leaves a mix of old and new files that fails with ImportError: cannot import name '_Ink' from 'PIL._typing' on import rfdetr.
RFDETR_SPEC = "rfdetr[onnx,openvino] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[onnx,openvino]>=1.12,<2"
!pip install -q "{RFDETR_SPEC}" "inference-models[onnx-cpu]" nncf psutil supervision pandas tabulate faster-coco-eval
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
2. Setup¶
WARMUP_RUNS discards the first N inferences; MEASURE_RUNS then collects the steady-state
timing distribution. BATCH_SIZES is the batch matrix every format section below measures —
(1, 4), not a single number, so each format contributes two rows to the final results table.
No CUDA guard needed — every format in this notebook runs on CPU. measure_latency and
measure_memory come from the public rfdetr.export.benchmark module; measure_memory measures
the host resident-memory growth of constructing a runtime and running its first inference call.
Every timed rfdetr predict() call passes include_source_image=False. inference-models returns no copy of the input image in its results, so rfdetr is timed without one too; with the default True it also copies each image into its results.
Every timed call takes the same decoded RGB array, frame, as a camera or video decoder delivers it, so no row pays a JPEG decode or a PIL-to-array copy.
MODEL_SIZE (nano, small, medium or large) and TASK (detection or segmentation) pick the model every row measures. A segmentation call also decodes masks on the host, so its latencies are not comparable with detection, and the mAP rows stay box mAP for both tasks.
import os
import platform
from functools import partial
from pathlib import Path
from typing import NamedTuple
import psutil
import torch
from PIL import Image
import rfdetr
from rfdetr import RFDETRInference
from rfdetr.export._benchmark import _artifact_size_mb, cpu_brand, visualize_detections
from rfdetr.export.benchmark import measure_latency, measure_memory
MODEL_SIZE = "small" # nano, small, medium or large
TASK = "detection" # "detection" or "segmentation"
MODEL_CLASS = f"RFDETR{'Seg' if TASK == 'segmentation' else ''}{MODEL_SIZE.capitalize()}" # class name in `rfdetr`
MODEL_ID = f"rfdetr-{'seg-' if TASK == 'segmentation' else ''}{MODEL_SIZE}" # `inference-models` id, names the files
EXPORT_DIR = Path("export_cpu")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 15
MEASURE_RUNS = 50
BATCH_SIZES = (1, 4)
# COCO val2017 eval subset (section 9) and the INT8 calibration set (section 8) are both drawn from these settings,
# so that calibration images can be held out of the images that are scored.
N_EVAL = 500
FULL_VAL = False
COCO_SEED = 0
COCO_ROOT = Path("coco")
N_CALIB = 100
#: One record per (format, config, batch) benchmark, appended to by `record()`.
rows: list[dict] = []
#: Batch-1 artifact and runtime options of every exported row, keyed by (Format, Config) — what section 9 scores.
artifacts: dict[tuple[str, str], tuple[Path, dict]] = {}
class Config(NamedTuple):
"""The fixed options of one benchmark row, one table column each."""
runtime: str
precision: str
quantization: str = "—"
def record(
format_label: str, config: Config, batch: int, result, memory_mb: float, size_mb: float | None = None
) -> None:
"""Append one benchmark result; `FPS` counts images per second, so it compares across batch sizes."""
rows.append(
{
"Format": format_label,
"Runtime": config.runtime,
"Precision": config.precision,
"Quantization": config.quantization,
"Batch": batch,
"end2end [ms]": result.mean_ms,
"± [ms]": result.std_ms,
"FPS [img/s]": batch * result.fps,
"Memory [MB]": memory_mb,
"Size [MB]": size_mb,
}
)
Notebook display helpers¶
_artifact_size_mb, cpu_brand and visualize_detections come from the private rfdetr.export._benchmark module, next
to the COCO mAP helpers used later. They size artifacts, read the CPU name and draw detections; none of them changes what
a runtime computes. record() and artifacts above collect one row per benchmark and the batch-1 artifact of every
export for the COCO section.
# Inline figures keep every plot in the notebook; closing them after display stops figures piling up in memory.
%matplotlib inline
%config InlineBackend.close_figures = True
CPU thread count¶
Every CPU runtime picks its own default thread pool, so rows measured at each runtime's default compare thread pools,
not formats. RFDETRInference exposes no thread setting for ONNX Runtime (its runtime_options take none), and ONNX
Runtime's own default intra-op pool is the number of physical cores
(ONNX Runtime threading docs). So
CPU_THREADS is the physical-core count, which is what ONNX Runtime already uses: it is applied to PyTorch here and to
OpenVINO through its INFERENCE_NUM_THREADS compile property (runtime_options={"config": ...}), and each runtime's
effective value is printed right after it is created. On a host where psutil cannot tell physical cores apart, it falls
back to os.cpu_count().
CPU_THREADS = psutil.cpu_count(logical=False) or os.cpu_count() or 1
torch.set_num_threads(CPU_THREADS)
OPENVINO_OPTIONS = {"config": {"INFERENCE_NUM_THREADS": CPU_THREADS}}
3. Host¶
The Config column and the narrative text in the Results section below are copied from this
cell's printed output, not assumed — CPU architecture, core count, and installed rfdetr version
all change what these numbers mean on a different machine.
The ONNX Runtime, OpenVINO and NNCF versions and the INT8 calibration source are recorded here too:
static INT8 accuracy depends on all of them.
import importlib.metadata
print(f"Machine: {platform.machine()} ({cpu_brand()})")
print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})")
print(f"Cores: {psutil.cpu_count(logical=False)} physical, {os.cpu_count()} logical")
print(f"CPU threads: {CPU_THREADS} pinned (PyTorch effective: {torch.get_num_threads()})")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")
print(f"onnxruntime: {importlib.metadata.version('onnxruntime')}")
print(f"openvino: {importlib.metadata.version('openvino')}")
print(f"nncf: {importlib.metadata.version('nncf')}")
print(
f"INT8 calibration: {N_CALIB} COCO val2017 images (selection seed {COCO_SEED + 1}), "
f"disjoint from the {'full val2017' if FULL_VAL else str(N_EVAL) + '-image'} eval subset (seed {COCO_SEED})"
)
4. Sample image¶
A single street scene with several COCO classes is enough to verify detections. The image is downloaded once and reused for every format below.
import urllib.request
import numpy as np
IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)
image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")
# A camera or video decoder delivers decoded arrays, so every timed call below takes this RGB array and no
# call pays a PIL-to-array copy or a JPEG decode.
frame = np.array(image)
5. PyTorch baseline — predict() / inference()¶
No export needed — this is the reference every other format in this notebook is compared
against. predict() is the unoptimized baseline. model.inference() returns an independent predictor, defaulting to dtype=torch.float32
with compile_backend="torchscript" — a JIT-compiled, still-fp32 optimization (fp16 arithmetic
is CUDA-only and not exercised here). The source model stays in eager mode for the export calls
below; each predictor owns its copied weights until it is deleted.
Both calls include preprocessing and postprocessing — there is no separate "forward-only" path
through eager predict(), so only an end-to-end number is reported for the PyTorch baseline.
predict() accepts a list of images and stacks them into one (N, C, H, W) batch internally
(one forward call per predict() call, not one per image), so the batch matrix below passes
[image] * batch for every row, including batch 1 — the same code path every batch size runs.
with measure_memory() as mem:
model = getattr(rfdetr, MODEL_CLASS)(device="cpu")
baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_eager_memory_mb = mem.delta_mb
print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)
Benchmark — eager predict(), batch 1 and batch 4¶
model is still eager (no inference() optimization yet), so every batch in BATCH_SIZES runs
through the same unoptimized path. Batch 1 reuses the baseline cell's own memory delta above —
no "larger batch" has run yet at batch 1, so a fresh bracket here would measure nothing new and
read as a false 0.0 MB. Memory [MB] for batch > 1 is the incremental growth of running that
larger batch through the same already-loaded model — not the model-load cost, which the baseline
cell above already paid.
for batch in BATCH_SIZES:
if batch == 1:
memory_mb = pytorch_eager_memory_mb
else:
with measure_memory() as mem:
_ = model.predict([frame] * batch, threshold=CONFIDENCE_THRESHOLD)
memory_mb = mem.delta_mb
result = measure_latency(
partial(model.predict, [frame] * batch, include_source_image=False),
label=f"PyTorch predict() b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record("PyTorch", Config("predict() eager", "fp32"), batch, result, memory_mb)
print(f" {result.label:<32} {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} +{memory_mb:.1f} MB")
Benchmark — inference() JIT, batch 1 and batch 4¶
model.inference(batch_size=b) creates a fresh TorchScript predictor for that exact batch
size — its trace only replays at the batch it was compiled for, so each batch in BATCH_SIZES
gets its own predictor. The memory bracket includes predictor construction and its first call;
Memory [MB] therefore records the per-predictor setup cost, not an incremental retrace. Deleting
each predictor leaves the source model in eager mode for the export calls below.
for batch in BATCH_SIZES:
with measure_memory() as mem:
predictor = model.inference(batch_size=batch)
_ = predictor.predict([frame] * batch, include_source_image=False)
result = measure_latency(
partial(predictor.predict, [frame] * batch, include_source_image=False),
label=f"PyTorch inference() JIT b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record("PyTorch", Config("inference() JIT", "fp32"), batch, result, mem.delta_mb)
print(f" {result.label:<32} {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} +{mem.delta_mb:.1f} MB")
del predictor
6. Load and benchmark exported artifacts — one path for every format¶
Every export below goes through the same two public calls: model.export(...) writes the artifact, and
RFDETRInference(path) loads it. The loader reads the artifact's own metadata, so the same predict() that serves the
PyTorch rows above serves ONNX and OpenVINO too: preprocessing, postprocessing, class ids and sv.Detections output
are shared, and no session options, tensor names, decoders or output-order matching are needed in the notebook.
benchmark_artifacts takes one artifact per batch size. For each it loads a fresh predictor, runs it once on the sample
image (batch 1 prints the detection count and draws the result), and times predict() on batch copies of the sample.
Memory [MB] is the host resident-memory growth across constructing the predictor plus its first call, so it is a full
construction cost on every row, comparable across formats and batch sizes.
Only end-to-end predict() timings are reported, the one scope every row has, including the PyTorch rows. Correctness is
checked by the COCO mAP section below, which scores these same batch-1 artifacts.
def benchmark_artifacts(
format_label: str, config: Config, paths: dict[int, Path], subdir: str, runtime_options: dict | None = None
) -> None:
"""Load one artifact per batch size with `RFDETRInference` and time its `predict()`."""
for batch, path in paths.items():
path = Path(path)
size_mb = (
_artifact_size_mb(path, path.with_suffix(".bin")) if path.suffix == ".xml" else _artifact_size_mb(path)
)
images_batch = [frame] * batch
with measure_memory() as mem:
predictor = RFDETRInference(path, device="cpu", runtime_options=runtime_options)
detections = predictor.predict(images_batch, threshold=CONFIDENCE_THRESHOLD)
if batch == BATCH_SIZES[0]:
artifacts[(format_label, config)] = (path, runtime_options or {})
print(f"{format_label} ({', '.join(config)}): {path} ({size_mb:.1f} MB)")
print(f" {len(detections[0])} detections above {CONFIDENCE_THRESHOLD}; runtime: {predictor.runtime_info}")
visualize_detections(detections[0], image, EXPORT_DIR / f"annotated_{subdir}.jpg")
result = measure_latency(
partial(predictor.predict, images_batch, include_source_image=False),
label=f"{format_label} {config} b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record(format_label, config, batch, result, mem.delta_mb, size_mb)
print(f" batch={batch}: {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} ({batch * result.fps:6.1f} FPS)")
del predictor
7. ONNX (CPU execution provider)¶
What it is. ONNX (Open Neural Network Exchange) is a portable model-graph format most
inference runtimes understand — ONNX Runtime, TensorRT, and OpenVINO can all consume the same
.onnx file. Good for avoiding a single-vendor runtime lock-in: export once, then run it
through whichever of those runtimes fits the target machine, CPU or GPU, without re-exporting.
See the ONNX export docs.
Export and benchmark¶
model.export() defaults to ONNX. dynamic_batch=True bakes a dynamic batch axis into the graph
(registry.py marks ONNX supports_dynamic_batch=True) — one exported .onnx file serves every
batch size in BATCH_SIZES. device="cpu" pins the CPU execution provider; ONNX Runtime keeps its default
intra-op pool of physical cores, which is what CPU_THREADS is set to.
onnx_path = Path(model.export(output_dir=str(EXPORT_DIR), dynamic_batch=True))
benchmark_artifacts("ONNX", Config("CPU EP", "fp32"), dict.fromkeys(BATCH_SIZES, onnx_path), "onnx")
8. OpenVINO¶
What it is. OpenVINO IR (Intermediate Representation) is Intel's own model format and toolkit for optimizing and deploying deep learning models on Intel hardware. Good for non-NVIDIA deployment: it targets Intel CPUs, integrated/discrete GPUs, and NPUs from the same exported IR, which matters if the deployment box has no NVIDIA GPU at all. See the OpenVINO export docs.
OpenVINO (fp32 IR)¶
openvino_precision="float32" overrides OpenVINO's default FP16 weight compression — this
keeps the IR at full precision for tight parity with the fp32 ONNX/PyTorch baseline above; the fp16 section below drops it
for a smaller file. The converted IR has a dynamic input shape ([?,?,?,?]), so one export would serve both batches;
this loop still exports one IR per batch size (batch_size=b only sets the dummy input it is traced with) so every
batch row loads an artifact built for it, as the fixed-shape formats in the other cookbooks must.
runtime_options={"config": {"INFERENCE_NUM_THREADS": CPU_THREADS}} pins OpenVINO's thread pool to the same count as
every other runtime here. RFDETRInference computes OpenVINO in f32 by default (inference_precision), because earlier
rfdetr versions compiled at the device's default precision instead — f16 on ARM, likely bf16 on x86 CPUs with AMX —
which collapsed RF-DETR's logits (0 detections against 3 on this sample). The printed runtime_info and the mAP row
below show the result.
openvino_fp32_paths = {
batch: model.export(
format="openvino",
openvino_precision="float32",
batch_size=batch,
output_dir=str(EXPORT_DIR / f"openvino_fp32_b{batch}"),
)
for batch in BATCH_SIZES
}
benchmark_artifacts("OpenVINO", Config("CPU", "fp32"), openvino_fp32_paths, "openvino", OPENVINO_OPTIONS)
OpenVINO (fp16 IR, default)¶
Dropping openvino_precision (or passing "float16" explicitly) restores OpenVINO's own
default FP16 weight compression — a smaller IR on disk and less memory traffic per inference,
at the cost of the numeric parity the float32 row above trades for. Execution precision still
depends on the compiled device, not just this setting (see the export docs linked above); RFDETRInference still
computes in f32 here — only the stored weights are fp16. One export per batch size, as for the fp32 IR above.
openvino_fp16_paths = {
batch: model.export(
format="openvino",
openvino_precision="float16",
batch_size=batch,
output_dir=str(EXPORT_DIR / f"openvino_fp16_b{batch}"),
)
for batch in BATCH_SIZES
}
benchmark_artifacts("OpenVINO", Config("CPU", "fp16"), openvino_fp16_paths, "openvino_fp16", OPENVINO_OPTIONS)
8. INT8 quantization¶
fp16 buys nothing on a CPU without native fp16 kernels — the arithmetic happens in fp32 either way. INT8 is the lever that does change CPU arithmetic, and both CPU formats in this notebook support it:
| Format | Mode | Calibration data |
|---|---|---|
| ONNX | static (8-bit weights and activations) | required |
| OpenVINO | static, via NNCF (pip install nncf) |
required |
Calibration data is what makes static quantization work: activation ranges are read from it, so it must look like what the model will see in deployment. A few dozen to a few hundred images from your validation split is the normal choice. Out-of-domain or synthetic images give a model that loads, runs, and is quietly wrong.
Only the matrix multiplies run in 8-bit; attention scores, the detection heads, normalization and the surrounding elementwise math stay in float. That selection is deliberate, so any change to it needs a COCO measurement, not a plausibility argument: the same harness as Section 9 scores the INT8 artifacts below.
Nothing in this section is assumed to be faster or accurate: the latency rows and the COCO mAP rows for both INT8 artifacts are measured on this machine, against the fp32 rows above, and the Host cell records the calibration source and the ONNX Runtime / OpenVINO / NNCF versions they depend on. Measure the trade on your own data before deploying.
The INT8 artifacts are exported for batch 1, so their latency rows are batch 1 only.
Calibration set, held out of the eval subset¶
The COCO eval subset is fetched here, before the calibration images are chosen, because the calibration set must not
overlap it: ranges collected from the images that are later scored would flatter the INT8 mAP. The helpers come from the
private rfdetr.export._benchmark module; they are not a supported API and do not change what any runtime computes. The
fetch downloads the 241 MB annotation archive once, then only the selected images, and skips files already on disk.
from rfdetr.export._benchmark import evaluate_coco_map, fetch_coco_val2017
coco_subset = fetch_coco_val2017(COCO_ROOT, n_images=None if FULL_VAL else N_EVAL, seed=COCO_SEED)
print(f"COCO val2017 images to score: {len(coco_subset.image_ids)}")
import shutil
# With FULL_VAL every val2017 image is scored, so none is left over to calibrate on.
assert not FULL_VAL, "Set FULL_VAL = False, or point CALIBRATION_DIR at your own held-out images."
# Draw N_EVAL + N_CALIB images with a different seed: at least N_CALIB of them cannot be in the eval subset.
calibration_pool = fetch_coco_val2017(COCO_ROOT, n_images=N_EVAL + N_CALIB, seed=COCO_SEED + 1)
eval_ids = set(coco_subset.image_ids)
calibration_ids = [image_id for image_id in calibration_pool.image_ids if image_id not in eval_ids][:N_CALIB]
assert len(calibration_ids) == N_CALIB and eval_ids.isdisjoint(calibration_ids)
CALIBRATION_DIR = EXPORT_DIR / "calibration_images"
CALIBRATION_DIR.mkdir(exist_ok=True)
for image_id in calibration_ids:
image_path = calibration_pool.images_dir / f"{image_id:012d}.jpg"
shutil.copy(image_path, CALIBRATION_DIR / image_path.name)
print(f"Calibration images: {len(calibration_ids)} in {CALIBRATION_DIR}, disjoint from the {len(eval_ids)} eval images")
ONNX INT8¶
quantization="int8" writes the static QDQ model beside the fp32 graph and returns the {stem}_int8.onnx path. It
needs onnxruntime, already installed for Section 6.
int8_onnx_path = Path(
model.export(
format="onnx",
output_dir=str(EXPORT_DIR / "onnx_int8"),
quantization="int8",
calibration_data=CALIBRATION_DIR,
)
)
benchmark_artifacts("ONNX", Config("CPU EP", "INT8", "static QDQ"), {1: int8_onnx_path}, "onnx_int8")
OpenVINO INT8¶
NNCF compresses the converted IR before it is written, so the .xml / .bin pair is already 8-bit. NNCF is not part of
the rfdetr[openvino] extra; it came from the install cell. The IR goes to its own directory because its filename
matches the fp32 and fp16 IRs.
openvino_int8_path = model.export(
format="openvino",
batch_size=1,
quantization="int8",
calibration_data=CALIBRATION_DIR,
output_dir=str(EXPORT_DIR / "openvino_int8_b1"),
)
benchmark_artifacts(
"OpenVINO", Config("CPU", "INT8", "static NNCF"), {1: openvino_int8_path}, "openvino_int8", OPENVINO_OPTIONS
)
9. inference-models side by side¶
inference-models is Roboflow's separate deployment package: AutoModel.from_pretrained("rfdetr-small", backend=...) downloads a prebuilt model, which is then called on a list of images. Its ONNX Runtime and PyTorch backends run here next to the rfdetr rows above, at the same batch sizes and the same end-to-end scope.
Two call settings keep the comparison fair:
input_color_format="rgb": a NumPy array is BGR to this package unless declared. Without it the network sees swapped channels and the scores drop, with no error.confidence=: passed on every call, so both packages use the same threshold.
onnx_execution_providers=["CPUExecutionProvider"] pins ONNX Runtime to the CPU, like the rfdetr ONNX row. The package has no OpenVINO, TFLite or ExecuTorch backend.
import supervision as sv
from inference_models import AutoModel, BackendType
def im_predict(im_model, frames: list[np.ndarray], threshold: float) -> list[sv.Detections]:
"""Run an `inference-models` model on RGB arrays and return `sv.Detections`, like `RFDETRInference.predict`."""
return [p.to_supervision() for p in im_model(frames, confidence=threshold, input_color_format="rgb")]
#: `inference-models` models by (Format, Config), kept for the COCO section.
im_models: dict[tuple[str, Config], object] = {}
def benchmark_inference_models(format_label: str, config: Config, **load_kwargs) -> None:
"""Load one `inference-models` model, check it once on the sample image and time `im_predict` at every batch size."""
for batch in BATCH_SIZES:
with measure_memory() as mem:
im_model = AutoModel.from_pretrained(MODEL_ID, **load_kwargs)
detections = im_predict(im_model, [frame] * batch, CONFIDENCE_THRESHOLD)
if batch == BATCH_SIZES[0]:
im_models[(format_label, config)] = im_model
print(f"{format_label} ({', '.join(config)}): {len(detections[0])} detections above {CONFIDENCE_THRESHOLD}")
result = measure_latency(
partial(im_predict, im_model, [frame] * batch, CONFIDENCE_THRESHOLD),
label=f"{format_label} {config} batch={batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(f" batch={batch}: {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} ({batch * result.fps:6.1f} FPS)")
record(format_label, config, batch, result, mem.delta_mb)
del im_model
benchmark_inference_models(
"ONNX",
Config("inference-models CPU EP", "fp32"),
backend=BackendType.ONNX,
onnx_execution_providers=["CPUExecutionProvider"],
)
benchmark_inference_models("PyTorch", Config("inference-models", "fp32"), backend=BackendType.TORCH, device="cpu")
10. COCO val2017 accuracy¶
A latency number on its own can hide a precision collapse. Earlier rfdetr versions compiled
OpenVINO at the device's default f16 precision, which returned 0 detections while the timing still
looked normal. So each batch-1 row below also gets a box mAP, measured on the same exported
artifact at the same precision as its latency row: the batch-1 fp32, fp16 and INT8 OpenVINO IRs, the
dynamic-batch .onnx file and the batch-1 INT8 .onnx file.
By default this runs on a 500-image seeded subset of COCO val2017, so the result is not the
full-val2017 number on the Benchmarks page.
Set FULL_VAL = True to score all 5000 images. Each row costs roughly its batch-1 end2end
latency times the number of images, so seven rows at a few hundred milliseconds each take on the order of
20 minutes on a small x86 host (about ten times that with FULL_VAL).
Every row is scored the same way: evaluate_coco_map calls predict(image, threshold=0.001) and reads the returned
sv.Detections, whose class_id values are COCO category IDs for the pretrained checkpoint, with a 0.001 score floor
that keeps the whole precision-recall curve. No per-format decoding is involved, so an exported row differs from the
PyTorch row only by what the export itself changed.
map_results = {}
map_results[("PyTorch", Config("predict() eager", "fp32"))] = evaluate_coco_map(
partial(model.predict, threshold=0.001), coco_subset
)
# A new independent predictor for batch 1, the same setting as its latency row.
predictor = model.inference(batch_size=1)
map_results[("PyTorch", Config("inference() JIT", "fp32"))] = evaluate_coco_map(
partial(predictor.predict, threshold=0.001), coco_subset
)
del predictor
Then every exported artifact, loaded exactly as in its latency row (same file, same runtime_options). A large drop from
the fp32 IR row for the fp16 IR would point at the weight compression itself; the INT8 calibration images are disjoint
from the images scored here, so a drop from the fp32 row is the quantization itself.
for (format_label, config), (path, runtime_options) in artifacts.items():
predictor = RFDETRInference(path, device="cpu", runtime_options=runtime_options)
map_results[(format_label, config)] = evaluate_coco_map(partial(predictor.predict, threshold=0.001), coco_subset)
del predictor
print(
f" {format_label} ({', '.join(config)}): mAP@50:95 {100 * map_results[(format_label, config)].map50_95:.1f} %"
)
Then the inference-models models of Section 9, each scored through the same batch-1 call as its latency row with confidence=0.001. Their class_id values are COCO category IDs, as the sample image shows (dog 18, person 1, backpack 27, handbag 31 in both packages' output), so the same ground truth applies.
for (format_label, config), im_model in im_models.items():
map_results[(format_label, config)] = evaluate_coco_map(
lambda img, m=im_model: im_predict(m, [np.array(img)], 0.001)[0], coco_subset
)
print(
f" {format_label} ({', '.join(config)}): mAP@50:95 {100 * map_results[(format_label, config)].map50_95:.1f} %"
)
11. Throughput on a folder¶
The batch rows call predict() on one image repeated. A real job predicts a folder: predict(folder) reads and decodes images on worker threads ahead of the model and runs them in batches. A runtime with a dynamic batch axis runs up to eight images per call, a fixed batch-1 artifact runs one, and a compiled or fixed-batch-8 runtime takes batch=8. include_source_image=False skips keeping the decoded image with each result.
N_FOLDER eval images are copied into one folder and the median of FOLDER_REPS timed passes (after one warm-up pass) gives images per second over the whole call: decode, preprocessing, model, postprocessing.
import shutil
import time
from tqdm.auto import tqdm
N_FOLDER = 64 # a multiple of 8, so every batch of the compiled batch-8 predictor is full
FOLDER_REPS = 3
from tempfile import TemporaryDirectory
# Own the benchmark images without deleting a pre-existing export folder.
folder_workspace = TemporaryDirectory(prefix="rfdetr-folder-")
FOLDER_DIR = Path(folder_workspace.name)
for image_id in tqdm(coco_subset.image_ids[:N_FOLDER], desc="Copy images"):
shutil.copy(coco_subset.images_dir / f"{image_id:012d}.jpg", FOLDER_DIR)
assert len(list(FOLDER_DIR.glob("*.jpg"))) == N_FOLDER
folder_rows: list[dict] = []
def measure_folder(format_label: str, config: Config, predictor, **predict_kwargs) -> None:
"""Time `predictor.predict(folder)` end to end and record the median images per second."""
kwargs = {"threshold": CONFIDENCE_THRESHOLD, "include_source_image": False, **predict_kwargs}
predictor.predict(str(FOLDER_DIR), **kwargs) # warm-up pass
seconds = []
for _ in range(FOLDER_REPS):
start = time.perf_counter()
results = predictor.predict(str(FOLDER_DIR), **kwargs)
seconds.append(time.perf_counter() - start)
assert len(results) == N_FOLDER
img_s = N_FOLDER / sorted(seconds)[len(seconds) // 2]
folder_rows.append(
{"Format": format_label, "Runtime": config.runtime, "Precision": config.precision, "Folder [img/s]": img_s}
)
print(f" {format_label} ({', '.join(config)}): {img_s:6.1f} img/s")
predictor = model.inference(batch_size=8)
measure_folder("PyTorch", Config("inference() JIT", "fp32"), predictor, batch=8)
del predictor
for (format_label, config), (path, runtime_options) in tqdm(artifacts.items(), desc="Folder throughput"):
predictor = RFDETRInference(path, device="cpu", runtime_options=runtime_options)
measure_folder(format_label, config, predictor)
del predictor
12. Results¶
The last cell prints a decision table (one row per format and config: batch-1 latency, speed against PyTorch predict(), size, mAP@50:95 and FPS at each batch size) and a detail table with ± std and memory growth. Every latency is the end-to-end predict() call, and mAP@50:95 is scored on the same batch-1 artifact that was timed. Thread pools are pinned to the physical-core count for PyTorch and OpenVINO and left at ONNX Runtime's default, which is the same number. On a shared vCPU, differences smaller than one standard deviation are noise, not a ranking.
The numbers from the last Colab run, with their notes, are kept on the OpenVINO export page so this notebook stays short. After a rerun, update that page from the printed table.
import pandas as pd
from IPython.display import display
KEYS = ["Format", "Runtime", "Precision", "Quantization"]
detail = pd.DataFrame(rows)
# Decision table: one row per (format, config); batch-1 latency, accuracy and size, throughput at every batch.
fps_by_batch = detail.pivot_table(index=KEYS, columns="Batch", values="FPS [img/s]", sort=False).add_prefix("FPS b")
map_by_row = pd.Series(
{(format_label, *config): 100 * result.map50_95 for (format_label, config), result in map_results.items()},
name="mAP@50:95 [%]",
)
map_by_row.index.names = KEYS
headline = (
detail[detail["Batch"] == 1]
.set_index(KEYS)[["end2end [ms]", "Size [MB]"]]
.rename(columns={"end2end [ms]": "b1 latency [ms]"})
.join(map_by_row)
.join(fps_by_batch)
)
headline.insert(1, "Speed vs PyTorch ×", headline["b1 latency [ms]"].iloc[0] / headline["b1 latency [ms]"])
# One number format for every table in this section: one decimal, "—" where a row has no value.
decision = headline.apply(lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}"))
display(decision)
Markdown for the docs page: the table above, unchanged.
print(
decision.reset_index().to_markdown(index=False, disable_numparse=True)
) # keep the one-decimal strings as they are
Per-batch detail: ± std and memory growth for every row.
numeric = detail.columns.difference(["Format", "Runtime", "Precision", "Quantization", "Batch"])
detail_table = detail.copy()
detail_table[numeric] = detail[numeric].apply(
lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}")
)
print(detail_table.to_markdown(index=False))
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, CPU.")
Folder throughput of Section 11, images per second over a whole predict(folder) call:
import pandas as pd
folder = pd.DataFrame(folder_rows)
folder["Folder [img/s]"] = folder["Folder [img/s]"].map(lambda value: f"{value:.1f}")
display(folder)
Next steps¶
- Fine-tuned weights — pass
pretrain_weights="<path/to/checkpoint.pth>"when constructing the model. - Smaller and faster on CPU — see the INT8 section above, and the ONNX and OpenVINO export docs for the full accuracy notes.
- Intel GPU / NPU — pass
device="gpu"ordevice="npu"toRFDETRInferenceon Intel hardware that has one; see the OpenVINO export docs. - Have a CUDA GPU, mobile target, or Apple Silicon? — see the CUDA cookbook, the mobile/edge cookbook, or the Apple cookbook.
- See the Export documentation for every format and option.