RF-DETR on CPU — Export, Inference & Latency¶
Compares every export path that targets a general-purpose CPU (x86 or ARM desktop/server — no GPU, no NPU): eager PyTorch as the reference anchor, ONNX Runtime with the CPU execution provider, and OpenVINO IR.
| Format | Export | Inference |
|---|---|---|
| PyTorch | (no export — eager model) | predict() / inference() (TorchScript JIT, fp32) |
| ONNX (CPU EP) | model.export() → .onnx |
onnxruntime.InferenceSession |
| OpenVINO | model.export(format="openvino") → .xml + .bin |
OpenVINOInference |
Every format is benchmarked at batch 1 and batch 4 — see
src/rfdetr/export/registry.py
for which formats can hold one artifact across batch sizes and which cannot: ONNX bakes a
dynamic batch axis (supports_dynamic_batch=True), so one .onnx file serves both rows below;
OpenVINO bakes a fixed batch shape into its IR graph, so it needs one exported artifact per
batch size (dynamic_batch_reason: "the IR graph bakes a fixed input shape"). PyTorch needs no
export step and trivially takes either batch as a (N, C, H, W) tensor.
Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), mobile/edge formats — TFLite, LiteRT, ExecuTorch's XNNPACK backend (see the mobile cookbook) — or Apple Silicon (see the Apple cookbook). OpenVINO can also target Intel GPUs and NPUs; this notebook only exercises its
"CPU"device.
Which latency number is this? Every number in this notebook is a Tier C measurement: forward-only and end-to-end timings for each export format, on one shared CPU, against a PyTorch anchor measured on that same machine. Its value is ratios within this machine — never set these milliseconds against the architecture headline on the Benchmarks page (NVIDIA T4, TensorRT FP16, model only) or the number on Roboflow's RF-DETR model page (L4, Roboflow Inference, scope not stated, batch 1) — those are GPU numbers, not comparable to a CPU run at all, let alone without checking precision, batch size, and scope.
1. Install¶
RFDETR_SPEC holds the install source and defaults to the develop branch, to pick up the
newest export fixes; to install the 1.11.2 release from PyPI instead, swap in the commented
>=1.11.2,<2 line.
No CUDA-specific packages needed.
RFDETR_SPEC = "rfdetr[onnx,openvino] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[onnx,openvino]>=1.11.2,<2"
!pip install -q "{RFDETR_SPEC}" psutil supervision pandas
2. Setup¶
WARMUP_RUNS discards the first N inferences; MEASURE_RUNS then collects the steady-state
timing distribution. BATCH_SIZES is the batch matrix every format section below measures —
(1, 4), not a single number, so each format contributes two rows to the final results table.
No CUDA guard needed — every format in this notebook runs on CPU.
Every format section below shares a handful of small helpers that used to be copy-pasted into
each per-hardware cookbook separately; they now live once, privately, in rfdetr.export._benchmark:
_artifact_size_mb reports an export artifact's size on disk (a single file, or the total of a
directory bundle — needed for OpenVINO's .xml + .bin pair), visualize_detections annotates
and displays a supervision.Detections on the sample image, _tile_batch stacks the one sample
image's preprocessed tensor into a (batch, C, H, W) array (all batch copies are the same
image — this notebook measures throughput at a larger batch dimension, not batch diversity), and
_decode_batch runs decode_detections over every image in a batched raw-output pair so an
end2end timing at batch 4 includes all four images' postprocessing, not just the first.
measure_memory measures the host resident-memory growth of constructing a runtime and running
its first inference call. cpu_brand reads the host CPU's brand string for the Host cell below,
reporting a placeholder instead of raising on any host without macOS's sysctl.
from pathlib import Path
import numpy as np
import supervision as sv
from PIL import Image
from rfdetr.export._benchmark import (
BenchmarkResult,
_artifact_size_mb,
_decode_batch,
_enable_notebook_inline_matplotlib,
_result_row,
_tile_batch,
cpu_brand,
measure_latency,
measure_memory,
visualize_detections,
)
EXPORT_DIR = Path("export_cpu")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 15
MEASURE_RUNS = 50
BATCH_SIZES = (1, 4)
_enable_notebook_inline_matplotlib()
3. Host¶
The Config column and the narrative text in the Results section below are copied from this
cell's printed output, not assumed — CPU architecture, core count, and installed rfdetr version
all change what these numbers mean on a different machine.
import importlib.metadata
import os
import platform
print(f"Machine: {platform.machine()} ({cpu_brand()})")
print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})")
print(f"Cores: {os.cpu_count()} logical")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")
4. Sample image¶
A single street scene with several COCO classes (dog, bicycle, car) is enough to verify detections. The image is downloaded once and reused for every format below.
import urllib.request
IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)
image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")
5. PyTorch baseline — predict() / inference()¶
No export needed — this is the reference every other format in this notebook is compared
against. predict() is the unoptimized baseline. inference() defaults to
dtype=torch.float32 with compile_backend="torchscript" — a JIT-compiled, still-fp32
optimization (fp16 arithmetic is CUDA-only and not exercised here);
remove_optimized_model() reverts the in-place optimization afterward so model stays reusable
for the export calls below.
Both calls include preprocessing and postprocessing — there is no separate "forward-only" path
through eager predict(), so only an end-to-end number is reported for the PyTorch baseline.
predict() accepts a list of images and stacks them into one (N, C, H, W) batch internally
(one forward call per predict() call, not one per image), so the batch matrix below passes
[image] * batch for every row, including batch 1 — the same code path every batch size runs.
from rfdetr import RFDETRSmall
with measure_memory() as mem:
model = RFDETRSmall(device="cpu")
baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_eager_memory_mb = mem.delta_mb
print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)
Benchmark — eager predict(), batch 1 and batch 4¶
model is still eager (no inference() optimization yet), so every batch in BATCH_SIZES runs
through the same unoptimized path. Batch 1 reuses the baseline cell's own memory delta above —
no "larger batch" has run yet at batch 1, so a fresh bracket here would measure nothing new and
read as a false 0.0 MB. Memory [MB] for batch > 1 is the incremental growth of running that
larger batch through the same already-loaded model — not the model-load cost, which the baseline
cell above already paid.
pytorch_eager_results: dict[int, tuple[BenchmarkResult, float]] = {}
for _batch in BATCH_SIZES:
if _batch == 1:
_memory_mb = pytorch_eager_memory_mb
else:
with measure_memory() as mem:
_ = model.predict([image] * _batch, threshold=CONFIDENCE_THRESHOLD)
_memory_mb = mem.delta_mb
_result = measure_latency(
lambda batch=_batch: model.predict([image] * batch),
label=f"PyTorch predict() b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
pytorch_eager_results[_batch] = (_result, _memory_mb)
print(f" {_result.label:<32} {_result.mean_ms:6.2f} ms ± {_result.std_ms:5.2f} +{_memory_mb:.1f} MB")
Benchmark — inference() JIT, batch 1 and batch 4¶
inference(batch_size=b) traces a new TorchScript graph for that exact batch size — the trace
only replays at the batch it was compiled for (predict() raises ValueError on a mismatch), so
each batch in BATCH_SIZES gets its own inference() call. inference() calls
remove_optimized_model() internally before every new trace, so Memory [MB] per batch is a
full recompile-and-first-call cost, not an incremental one — comparable across batches, not to
the eager row above. remove_optimized_model() after the loop reverts the model to eager mode so
the export calls below trace the original (uncompiled) module.
pytorch_jit_results: dict[int, tuple[BenchmarkResult, float]] = {}
for _batch in BATCH_SIZES:
with measure_memory() as mem:
model.inference(batch_size=_batch)
_ = model.predict([image] * _batch)
_memory_mb = mem.delta_mb
_result = measure_latency(
lambda batch=_batch: model.predict([image] * batch),
label=f"PyTorch inference() JIT b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
pytorch_jit_results[_batch] = (_result, _memory_mb)
print(f" {_result.label:<32} {_result.mean_ms:6.2f} ms ± {_result.std_ms:5.2f} +{_memory_mb:.1f} MB")
model.remove_optimized_model()
6. ONNX (CPU execution provider)¶
What it is. ONNX (Open Neural Network Exchange) is a portable model-graph format most
inference runtimes understand — ONNX Runtime, TensorRT, and OpenVINO can all consume the same
.onnx file. Good for avoiding a single-vendor runtime lock-in: export once, then run it
through whichever of those runtimes fits the target machine, CPU or GPU, without re-exporting.
See the ONNX export docs.
Export¶
model.export() defaults to ONNX. dynamic_batch=True bakes a dynamic batch axis into the graph
(registry.py marks ONNX supports_dynamic_batch=True) — one exported .onnx file serves every
batch size in BATCH_SIZES below; only OpenVINO in this notebook needs a separate export per
batch.
onnx_path = model.export(output_dir=str(EXPORT_DIR), dynamic_batch=True)
onnx_size_mb = _artifact_size_mb(onnx_path)
print(f"ONNX model (dynamic batch): {onnx_path} ({onnx_size_mb:.1f} MB)")
Inference¶
_create_onnx_session (the same session-construction helper RFDETR.predict()'s reference
decoder uses internally) is pinned to CPUExecutionProvider.
from rfdetr.export._onnx.inference import _create_onnx_session, _run_inference
with measure_memory() as mem:
onnx_session = _create_onnx_session(onnx_path, providers=["CPUExecutionProvider"])
onnx_detections, _ = _run_inference(onnx_session, IMAGE_PATH, threshold=CONFIDENCE_THRESHOLD)
onnx_memory_mb = mem.delta_mb
print(f"ONNX (CPU): {len(onnx_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(onnx_detections, image, EXPORT_DIR / "annotated_onnx.jpg")
Benchmark — batch 1 and batch 4¶
Two scopes per batch: forward_ms times only session.run with preprocessing done once outside
the loop; end2end_ms times preprocessing + session.run + decoding every image in the batch,
the same scope predict() above was timed at. Memory [MB] is only meaningful once per shared
session: the batch-1 row reuses onnx_memory_mb (session construction, from the Inference cell
above); the batch-4 row measures the incremental arena growth of the same session's first
larger-batch call, not a second construction.
from rfdetr.export._runtime.decode import decode_detections
from rfdetr.export._runtime.preprocess import preprocess_to_nchw
_onnx_input_name = onnx_session.get_inputs()[0].name
_, _onnx_channels, _onnx_height, _onnx_width = onnx_session.get_inputs()[0].shape
_onnx_output_names = [out.name for out in onnx_session.get_outputs()]
_onnx_boxes_idx = next(i for i, name in enumerate(_onnx_output_names) if "dets" in name)
_onnx_logits_idx = next(i for i, name in enumerate(_onnx_output_names) if "labels" in name)
def _onnx_single_input() -> np.ndarray:
return preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)
def _make_onnx_end2end(batch: int):
def _run() -> None:
feed = {_onnx_input_name: _tile_batch(_onnx_single_input(), batch)}
raw = onnx_session.run(None, feed)
_decode_batch(raw[_onnx_boxes_idx], raw[_onnx_logits_idx], image.size, batch, threshold=CONFIDENCE_THRESHOLD)
return _run
onnx_results: dict[int, tuple[BenchmarkResult, BenchmarkResult]] = {}
onnx_memory_by_batch: dict[int, float] = {BATCH_SIZES[0]: onnx_memory_mb}
for _batch in BATCH_SIZES:
_feed = {_onnx_input_name: _tile_batch(_onnx_single_input(), _batch)}
if _batch != BATCH_SIZES[0]:
with measure_memory() as mem:
onnx_session.run(None, _feed)
onnx_memory_by_batch[_batch] = mem.delta_mb
_forward = measure_latency(
lambda feed=_feed: onnx_session.run(None, feed),
label=f"ONNX (CPU) forward b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
_end2end = measure_latency(
_make_onnx_end2end(_batch),
label=f"ONNX (CPU) end2end b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
onnx_results[_batch] = (_forward, _end2end)
print(f" {_forward.label:<28} {_forward.mean_ms:6.2f} ms ± {_forward.std_ms:5.2f}")
print(f" {_end2end.label:<28} {_end2end.mean_ms:6.2f} ms ± {_end2end.std_ms:5.2f}")
7. OpenVINO¶
What it is. OpenVINO IR (Intermediate Representation) is Intel's own model format and toolkit for optimizing and deploying deep learning models on Intel hardware. Good for non-NVIDIA deployment: it targets Intel CPUs, integrated/discrete GPUs, and NPUs from the same exported IR, which matters if the deployment box has no NVIDIA GPU at all. See the OpenVINO export docs.
Export¶
openvino_precision="float32" overrides OpenVINO's default FP16 weight compression — this
keeps the IR at full precision for tight parity with the fp32 ONNX/PyTorch baseline above; drop
it for a smaller, faster file if you don't need that parity (execution precision still depends
on the compiled device regardless of this setting). registry.py marks OpenVINO
supports_dynamic_batch=False — the IR graph bakes a fixed input shape — so unlike ONNX above,
each batch size needs its own export; batch_size=b bakes that batch into the traced graph.
openvino_fp32_paths: dict[int, Path] = {}
for _batch in BATCH_SIZES:
_path = model.export(
format="openvino",
openvino_precision="float32",
batch_size=_batch,
output_dir=str(EXPORT_DIR / f"openvino_fp32_b{_batch}"),
)
openvino_fp32_paths[_batch] = _path
_size_mb = _artifact_size_mb(_path, _path.with_suffix(".bin"))
print(f"OpenVINO IR (fp32, b={_batch}): {_path} ({_size_mb:.1f} MB, .xml + .bin)")
Inference¶
OpenVINOInference is session-tier: it takes already-preprocessed NCHW tensors and returns raw
output tensors, decoded here with the same decode_detections helper used for ONNX above.
infer() requires a C-contiguous float32 array — preprocess_to_nchw already returns one, but
np.ascontiguousarray is applied defensively since infer() raises ValueError otherwise.
Correctness is checked at batch 1 only — the batch-4 rows below reuse the same artifact-building
and decoding path, just timed rather than inspected image-by-image.
from rfdetr.export.inference import OpenVINOInference
with measure_memory() as mem:
ov_model = OpenVINOInference(openvino_fp32_paths[BATCH_SIZES[0]], device="CPU")
ov_input = np.ascontiguousarray(_onnx_single_input())
ov_boxes, ov_logits = ov_model(ov_input)
openvino_memory_mb = {BATCH_SIZES[0]: mem.delta_mb}
ov_decoded = decode_detections(ov_boxes[0], ov_logits[0], image.size, threshold=CONFIDENCE_THRESHOLD)
print(f"OpenVINO (CPU): {len(ov_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}")
ov_sv_detections = sv.Detections(
xyxy=ov_decoded.xyxy, confidence=ov_decoded.confidence, class_id=ov_decoded.class_id.astype(int)
)
visualize_detections(ov_sv_detections, image, EXPORT_DIR / "annotated_openvino.jpg")
Benchmark — batch 1 and batch 4¶
Each batch size loads its own OpenVINOInference from its own artifact — Memory [MB] per batch
is a full construction-plus-first-call cost, comparable across batches (unlike ONNX's shared-
session arena-growth number above).
def _make_ov_end2end(ov: OpenVINOInference, batch: int):
def _run() -> None:
inp = np.ascontiguousarray(_tile_batch(_onnx_single_input(), batch))
boxes, logits = ov(inp)
_decode_batch(boxes, logits, image.size, batch, threshold=CONFIDENCE_THRESHOLD)
return _run
openvino_fp32_models: dict[int, OpenVINOInference] = {BATCH_SIZES[0]: ov_model}
openvino_fp32_results: dict[int, tuple[BenchmarkResult, BenchmarkResult]] = {}
for _batch in BATCH_SIZES:
if _batch not in openvino_fp32_models:
with measure_memory() as mem:
_model = OpenVINOInference(openvino_fp32_paths[_batch], device="CPU")
_input = np.ascontiguousarray(_tile_batch(_onnx_single_input(), _batch))
_model(_input)
openvino_memory_mb[_batch] = mem.delta_mb
openvino_fp32_models[_batch] = _model
_ov = openvino_fp32_models[_batch]
_input = np.ascontiguousarray(_tile_batch(_onnx_single_input(), _batch))
_forward = measure_latency(
lambda ov=_ov, inp=_input: ov(inp),
label=f"OpenVINO fp32 forward b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
_end2end = measure_latency(
_make_ov_end2end(_ov, _batch),
label=f"OpenVINO fp32 end2end b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
openvino_fp32_results[_batch] = (_forward, _end2end)
print(f" {_forward.label:<28} {_forward.mean_ms:6.2f} ms ± {_forward.std_ms:5.2f}")
print(f" {_end2end.label:<28} {_end2end.mean_ms:6.2f} ms ± {_end2end.std_ms:5.2f}")
OpenVINO (fp16, default)¶
Dropping openvino_precision (or passing "float16" explicitly) restores OpenVINO's own
default FP16 weight compression — a smaller IR on disk and less memory traffic per inference,
at the cost of the numeric parity the float32 row above trades for. Execution precision still
depends on the compiled device, not just this setting (see the export docs linked above). Same
fixed-batch constraint as the fp32 IR above: one export per batch size.
openvino_fp16_paths: dict[int, Path] = {}
for _batch in BATCH_SIZES:
_path = model.export(
format="openvino",
openvino_precision="float16",
batch_size=_batch,
output_dir=str(EXPORT_DIR / f"openvino_fp16_b{_batch}"),
)
openvino_fp16_paths[_batch] = _path
_size_mb = _artifact_size_mb(_path, _path.with_suffix(".bin"))
print(f"OpenVINO IR (fp16, b={_batch}): {_path} ({_size_mb:.1f} MB, .xml + .bin)")
with measure_memory() as mem:
ov_fp16_model = OpenVINOInference(openvino_fp16_paths[BATCH_SIZES[0]], device="CPU")
ov_fp16_input = np.ascontiguousarray(_onnx_single_input())
ov_fp16_boxes, ov_fp16_logits = ov_fp16_model(ov_fp16_input)
openvino_fp16_memory_mb = {BATCH_SIZES[0]: mem.delta_mb}
ov_fp16_decoded = decode_detections(ov_fp16_boxes[0], ov_fp16_logits[0], image.size, threshold=CONFIDENCE_THRESHOLD)
print(f"OpenVINO fp16 (CPU): {len(ov_fp16_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}")
ov_fp16_sv_detections = sv.Detections(
xyxy=ov_fp16_decoded.xyxy, confidence=ov_fp16_decoded.confidence, class_id=ov_fp16_decoded.class_id.astype(int)
)
visualize_detections(ov_fp16_sv_detections, image, EXPORT_DIR / "annotated_openvino_fp16.jpg")
openvino_fp16_models: dict[int, OpenVINOInference] = {BATCH_SIZES[0]: ov_fp16_model}
openvino_fp16_results: dict[int, tuple[BenchmarkResult, BenchmarkResult]] = {}
for _batch in BATCH_SIZES:
if _batch not in openvino_fp16_models:
with measure_memory() as mem:
_model = OpenVINOInference(openvino_fp16_paths[_batch], device="CPU")
_input = np.ascontiguousarray(_tile_batch(_onnx_single_input(), _batch))
_model(_input)
openvino_fp16_memory_mb[_batch] = mem.delta_mb
openvino_fp16_models[_batch] = _model
_ov = openvino_fp16_models[_batch]
_input = np.ascontiguousarray(_tile_batch(_onnx_single_input(), _batch))
_forward = measure_latency(
lambda ov=_ov, inp=_input: ov(inp),
label=f"OpenVINO fp16 forward b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
# Reuses `_make_ov_end2end` defined in the fp32 Benchmark cell above — identical shape, just a
# different `ov`/`batch` pair.
_end2end = measure_latency(
_make_ov_end2end(_ov, _batch),
label=f"OpenVINO fp16 end2end b{_batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
openvino_fp16_results[_batch] = (_forward, _end2end)
print(f" {_forward.label:<28} {_forward.mean_ms:6.2f} ms ± {_forward.std_ms:5.2f}")
print(f" {_end2end.label:<28} {_end2end.mean_ms:6.2f} ms ± {_end2end.std_ms:5.2f}")
8. Results¶
Measured on this machine — see the Host cell's output above for the exact CPU/OS/rfdetr
string. RFDETRSmall, 15 warmup + 50 timed runs per row, batches 1 and 4. Config is the
precision/backend used for that row; — marks a scope this format doesn't have (PyTorch's
predict() has no forward-only path). forward [ms] / end2end [ms] are per-call latency for
the whole batch (a batch-4 call takes longer wall time than a batch-1 call because it does 4×
the work); FPS [img/s] is batch × 1000 / end2end_ms, i.e. images processed per second — not
calls per second — so it is the column to compare across batch sizes. Memory [MB] follows the
per-row note in each benchmark cell above: PyTorch predict()'s batch-1 row reuses the baseline
cell's full construction cost, its batch-4 row is incremental growth on the same already-loaded
model; PyTorch inference() and OpenVINO rows are always a full construction cost (JIT retraces
a new graph, OpenVINO re-exports IR, per batch); ONNX's batch-1 row is full session construction,
its batch-4 row is incremental arena growth, since ONNX reuses one dynamic-batch session across
both batch sizes.
Note: the batch-1
predict()row above reflects the fix for the memory-measurement bug this note describes, but the published table below predates it — that row'sMemory [MB]cell is therefore markedn/a (pending rerun)rather than carrying the pre-fix0.0, and needs a rerun on the documented Colab host to regenerate a real number.
| Format | Batch | Config | forward [ms] | end2end [ms] | FPS [img/s] (end2end) | Memory [MB] |
|---|---|---|---|---|---|---|
PyTorch predict() |
1 | eager, fp32 | — | 377.34 ± 55.83 | 2.7 | n/a (pending rerun) |
PyTorch inference() |
1 | JIT, fp32 | — | 368.11 ± 67.72 | 2.7 | 228.5 |
| ONNX | 1 | CPU EP | 354.77 ± 55.76 | 363.62 ± 46.87 | 2.8 | 21.8 |
| OpenVINO | 1 | fp32 IR, CPU | 356.11 ± 68.63 | 357.47 ± 56.88 | 2.8 | 302.4 |
| OpenVINO | 1 | fp16 IR (default), CPU | 387.44 ± 71.18 | 399.62 ± 66.88 | 2.5 | 212.1 |
PyTorch predict() |
4 | eager, fp32 | — | 1365.23 ± 197.08 | 2.9 | 5.1 |
PyTorch inference() |
4 | JIT, fp32 | — | 1285.96 ± 198.37 | 3.1 | 781.5 |
| ONNX | 4 | CPU EP | 1352.37 ± 178.75 | 1361.69 ± 178.10 | 2.9 | 202.9 |
| OpenVINO | 4 | fp32 IR, CPU | 1345.73 ± 210.03 | 1362.64 ± 203.69 | 2.9 | 331.0 |
| OpenVINO | 4 | fp16 IR (default), CPU | 1412.55 ± 203.53 | 1433.26 ± 206.59 | 2.8 | 325.2 |
Measured on a shared Colab vCPU, 15 warmup + 50 timed runs per row. Every format lands within noise of every other one at both batch sizes — std devs of 15–20% of the mean swallow whatever real gap exists between PyTorch eager, ONNX, and OpenVINO fp32 on this CPU; none of them separates from the others by more than roughly one standard deviation, the same shared-vCPU noise floor the export overview page already calls out for this cookbook. OpenVINO's fp16 IR is the one row that reads as a real, not just noisy, difference — slower than fp32 at both batches (399.62 vs 357.47 ms at batch 1; 1433.26 vs 1362.64 ms at batch 4) — this CPU has no native fp16 kernels, so "fp16" here means emulation, not speedup; see the CUDA cookbook for the contrast where fp16 is a decisive win on tensor cores.
Batch 4 does not get 4× the batch-1 latency: PyTorch predict() goes 377.34 → 1365.23 ms (3.62×
for 4× the images), so FPS still climbs slightly (2.7 → 2.9) — some fixed per-call overhead
amortizes across the larger batch, it just does not amortize much on a CPU with no batched-kernel
advantage the way a GPU's tensor cores exploit one. PyTorch inference() JIT's memory jump
(228.5 → 781.5 MB) is the steepest of any row: it recompiles a new TorchScript graph sized for the
batch every time, so the batch-4 number is a full second compile-and-first-call cost, not an
incremental one — comparable to the batch-1 row's own full cost, not to a shared-session's
incremental growth the way ONNX's rows are.
import pandas as pd
from IPython.display import display
_rows = []
for _batch in BATCH_SIZES:
_eager, _eager_mb = pytorch_eager_results[_batch]
_jit, _jit_mb = pytorch_jit_results[_batch]
_onnx_fwd, _onnx_e2e = onnx_results[_batch]
_ov32_fwd, _ov32_e2e = openvino_fp32_results[_batch]
_ov16_fwd, _ov16_e2e = openvino_fp16_results[_batch]
_rows += [
_result_row("PyTorch predict()", _batch, "eager, fp32", None, _eager, _eager_mb),
_result_row("PyTorch inference()", _batch, "JIT, fp32", None, _jit, _jit_mb),
_result_row("ONNX", _batch, "CPU EP", _onnx_fwd, _onnx_e2e, onnx_memory_by_batch.get(_batch)),
_result_row("OpenVINO", _batch, "fp32 IR, CPU", _ov32_fwd, _ov32_e2e, openvino_memory_mb.get(_batch)),
_result_row(
"OpenVINO", _batch, "fp16 IR (default), CPU", _ov16_fwd, _ov16_e2e, openvino_fp16_memory_mb.get(_batch)
),
]
summary = pd.DataFrame(_rows).set_index(["Format", "Batch"])
display(summary)
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, CPU.")
Next steps¶
- Fine-tuned weights — pass
pretrain_weights="<path/to/checkpoint.pth>"when constructing the model. - Intel GPU / NPU — pass
device="GPU"ordevice="NPU"toOpenVINOInferenceon Intel hardware that has one; see the OpenVINO export docs. - Have a CUDA GPU, mobile target, or Apple Silicon? — see the CUDA cookbook, the mobile/edge cookbook, or the Apple cookbook.
- See the Export documentation for every format and option.