RF-DETR on CPU — Export, Inference & Latency¶

Compares every export path that targets a general-purpose CPU (x86 or ARM desktop/server — no GPU, no NPU): eager PyTorch as the reference anchor, ONNX Runtime with the CPU execution provider, and OpenVINO IR.

Format	Export	Inference
PyTorch	(no export — eager model)	predict() / inference() (TorchScript JIT, fp32)
ONNX (CPU EP)	model.export() → .onnx	onnxruntime.InferenceSession
OpenVINO	model.export(format="openvino") → .xml + .bin	OpenVINOInference

Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), mobile/edge formats — TFLite, LiteRT, ExecuTorch's XNNPACK backend (see the mobile cookbook) — or Apple Silicon (see the Apple cookbook). OpenVINO can also target Intel GPUs and NPUs; this notebook only exercises its "CPU" device.

1. Install¶

Installs from develop to pick up the newest export fixes. No CUDA-specific packages needed.

In [ ]:

!pip install -q "rfdetr[onnx,openvino] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" psutil supervision pandas

!pip install -q "rfdetr[onnx,openvino] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" psutil supervision pandas

2. Setup¶

WARMUP_RUNS discards the first N inferences; MEASURE_RUNS then collects the steady-state timing distribution. No CUDA guard needed — every format in this notebook runs on CPU.

Three small helpers are shared by every format section below: _artifact_size_mb reports an export artifact's size on disk (a single file, or the total of a directory bundle — needed for OpenVINO's .xml + .bin pair), visualize_detections annotates and displays a supervision.Detections on the sample image (falling back to COCO_CLASSES for label text when a detection object carries no class_name), and measure_memory (from _benchmark) measures the host resident-memory growth of constructing a runtime and running its first inference call.

In [ ]:

from pathlib import Path

import numpy as np
import supervision as sv
from PIL import Image

from rfdetr.assets.coco_classes import COCO_CLASSES
from rfdetr.export._benchmark import BenchmarkResult, measure_latency, measure_memory

EXPORT_DIR = Path("export_cpu")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 15
MEASURE_RUNS = 50


def _artifact_size_mb(*paths: Path) -> float:
    total_bytes = 0
    for path in paths:
        if path.is_dir():
            total_bytes += sum(f.stat().st_size for f in path.rglob("*") if f.is_file())
        else:
            total_bytes += path.stat().st_size
    return total_bytes / 1e6


def visualize_detections(detections: sv.Detections, image: "Image.Image", save_path: Path | None = None) -> None:
    names = detections.data.get("class_name") if detections.data else None
    if names is None:
        names = [COCO_CLASSES.get(int(c), str(c)) for c in detections.class_id]
    labels = [f"{name} {conf:.2f}" for name, conf in zip(names, detections.confidence)]

    annotated = sv.BoxAnnotator(thickness=3).annotate(scene=image.copy(), detections=detections)
    annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate(
        scene=annotated, detections=detections, labels=labels
    )
    if save_path is not None:
        annotated.save(save_path)
        print(f"Saved annotated image: {save_path}")
    sv.plot_image(annotated)


def _enable_notebook_inline_matplotlib() -> None:
    """Enable inline matplotlib figures when running in IPython."""
    get_ipython_func = globals().get("get_ipython")
    if not callable(get_ipython_func):
        return
    ipython = get_ipython_func()
    if ipython is not None:
        ipython.run_line_magic("matplotlib", "inline")
        ipython.run_line_magic("config", "InlineBackend.close_figures = True")


_enable_notebook_inline_matplotlib()

from pathlib import Path import numpy as np import supervision as sv from PIL import Image from rfdetr.assets.coco_classes import COCO_CLASSES from rfdetr.export._benchmark import BenchmarkResult, measure_latency, measure_memory EXPORT_DIR = Path("export_cpu") EXPORT_DIR.mkdir(exist_ok=True) CONFIDENCE_THRESHOLD = 0.5 WARMUP_RUNS = 15 MEASURE_RUNS = 50 def _artifact_size_mb(*paths: Path) -> float: total_bytes = 0 for path in paths: if path.is_dir(): total_bytes += sum(f.stat().st_size for f in path.rglob("*") if f.is_file()) else: total_bytes += path.stat().st_size return total_bytes / 1e6 def visualize_detections(detections: sv.Detections, image: "Image.Image", save_path: Path | None = None) -> None: names = detections.data.get("class_name") if detections.data else None if names is None: names = [COCO_CLASSES.get(int(c), str(c)) for c in detections.class_id] labels = [f"{name} {conf:.2f}" for name, conf in zip(names, detections.confidence)] annotated = sv.BoxAnnotator(thickness=3).annotate(scene=image.copy(), detections=detections) annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate( scene=annotated, detections=detections, labels=labels ) if save_path is not None: annotated.save(save_path) print(f"Saved annotated image: {save_path}") sv.plot_image(annotated) def _enable_notebook_inline_matplotlib() -> None: """Enable inline matplotlib figures when running in IPython.""" get_ipython_func = globals().get("get_ipython") if not callable(get_ipython_func): return ipython = get_ipython_func() if ipython is not None: ipython.run_line_magic("matplotlib", "inline") ipython.run_line_magic("config", "InlineBackend.close_figures = True") _enable_notebook_inline_matplotlib()

3. Sample image¶

A single street scene with several COCO classes (dog, bicycle, car) is enough to verify detections. The image is downloaded once and reused for every format below.

In [ ]:

import urllib.request

IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")

import urllib.request IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg" IMAGE_PATH = EXPORT_DIR / "sample.jpg" if not IMAGE_PATH.exists(): urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH) image = Image.open(IMAGE_PATH).convert("RGB") print(f"Sample image: {image.size[0]}×{image.size[1]}")

4. PyTorch baseline — predict() / inference()¶

No export needed — this is the reference every other format in this notebook is compared against. predict() is the unoptimized baseline. inference() defaults to dtype=torch.float32 with compile_backend="torchscript" — a JIT-compiled, still-fp32 optimization (fp16 arithmetic is CUDA-only and not exercised here); remove_optimized_model() reverts the in-place optimization afterward so model stays reusable for the export calls below.

Both calls include preprocessing and postprocessing — there is no separate "forward-only" path through eager predict(), so only an end-to-end number is reported for the PyTorch baseline.

In [ ]:

from rfdetr import RFDETRSmall

with measure_memory() as mem:
    model = RFDETRSmall(device="cpu")
    baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_eager_memory_mb = mem.delta_mb
print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)

pytorch_eager = measure_latency(
    lambda: model.predict(image), label="PyTorch predict()", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

with measure_memory() as mem:
    model.inference()
    _ = model.predict(image)
pytorch_jit_memory_mb = mem.delta_mb

pytorch_jit = measure_latency(
    lambda: model.predict(image), label="PyTorch inference() JIT", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)
model.remove_optimized_model()

for r, mb in ((pytorch_eager, pytorch_eager_memory_mb), (pytorch_jit, pytorch_jit_memory_mb)):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)   +{mb:.1f} MB")

from rfdetr import RFDETRSmall with measure_memory() as mem: model = RFDETRSmall(device="cpu") baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD) pytorch_eager_memory_mb = mem.delta_mb print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}") visualize_detections(baseline_detections, image) pytorch_eager = measure_latency( lambda: model.predict(image), label="PyTorch predict()", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) with measure_memory() as mem: model.inference() _ = model.predict(image) pytorch_jit_memory_mb = mem.delta_mb pytorch_jit = measure_latency( lambda: model.predict(image), label="PyTorch inference() JIT", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) model.remove_optimized_model() for r, mb in ((pytorch_eager, pytorch_eager_memory_mb), (pytorch_jit, pytorch_jit_memory_mb)): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS) +{mb:.1f} MB")

5. ONNX (CPU execution provider)¶

What it is. ONNX (Open Neural Network Exchange) is a portable model-graph format most inference runtimes understand — ONNX Runtime, TensorRT, and OpenVINO can all consume the same .onnx file. Good for avoiding a single-vendor runtime lock-in: export once, then run it through whichever of those runtimes fits the target machine, CPU or GPU, without re-exporting. See the ONNX export docs.

Export¶

model.export() defaults to ONNX.

In [ ]:

onnx_path = model.export(output_dir=str(EXPORT_DIR))
onnx_size_mb = _artifact_size_mb(onnx_path)
print(f"ONNX model: {onnx_path}  ({onnx_size_mb:.1f} MB)")

onnx_path = model.export(output_dir=str(EXPORT_DIR)) onnx_size_mb = _artifact_size_mb(onnx_path) print(f"ONNX model: {onnx_path} ({onnx_size_mb:.1f} MB)")

Inference¶

_create_onnx_session (the same session-construction helper RFDETR.predict()'s reference decoder uses internally) is pinned to CPUExecutionProvider.

In [ ]:

from rfdetr.export._onnx.inference import _create_onnx_session, _run_inference

with measure_memory() as mem:
    onnx_session = _create_onnx_session(onnx_path, providers=["CPUExecutionProvider"])
    onnx_detections, _ = _run_inference(onnx_session, IMAGE_PATH, threshold=CONFIDENCE_THRESHOLD)
onnx_memory_mb = mem.delta_mb
print(f"ONNX (CPU): {len(onnx_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(onnx_detections, image, EXPORT_DIR / "annotated_onnx.jpg")

from rfdetr.export._onnx.inference import _create_onnx_session, _run_inference with measure_memory() as mem: onnx_session = _create_onnx_session(onnx_path, providers=["CPUExecutionProvider"]) onnx_detections, _ = _run_inference(onnx_session, IMAGE_PATH, threshold=CONFIDENCE_THRESHOLD) onnx_memory_mb = mem.delta_mb print(f"ONNX (CPU): {len(onnx_detections)} detections above {CONFIDENCE_THRESHOLD}") visualize_detections(onnx_detections, image, EXPORT_DIR / "annotated_onnx.jpg")

Benchmark¶

Two scopes: forward_ms times only session.run with preprocessing done once outside the loop; end2end_ms times preprocessing + session.run + decoding together, the same scope predict() above was timed at.

In [ ]:

from rfdetr.export._runtime.decode import decode_detections
from rfdetr.export._runtime.preprocess import preprocess_to_nchw

_onnx_input_name = onnx_session.get_inputs()[0].name
_, _onnx_channels, _onnx_height, _onnx_width = onnx_session.get_inputs()[0].shape
_onnx_output_names = [out.name for out in onnx_session.get_outputs()]
_onnx_boxes_idx = next(i for i, name in enumerate(_onnx_output_names) if "dets" in name)
_onnx_logits_idx = next(i for i, name in enumerate(_onnx_output_names) if "labels" in name)
_onnx_feed = {_onnx_input_name: preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)}


def _onnx_end2end() -> None:
    inp = preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)
    raw = onnx_session.run(None, {_onnx_input_name: inp})
    decode_detections(raw[_onnx_boxes_idx][0], raw[_onnx_logits_idx][0], image.size, threshold=CONFIDENCE_THRESHOLD)


onnx_forward = measure_latency(
    lambda: onnx_session.run(None, _onnx_feed),
    label="ONNX (CPU) forward",
    device="cpu",
    warmup=WARMUP_RUNS,
    runs=MEASURE_RUNS,
)
onnx_end2end = measure_latency(
    _onnx_end2end, label="ONNX (CPU) end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (onnx_forward, onnx_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

from rfdetr.export._runtime.decode import decode_detections from rfdetr.export._runtime.preprocess import preprocess_to_nchw _onnx_input_name = onnx_session.get_inputs()[0].name _, _onnx_channels, _onnx_height, _onnx_width = onnx_session.get_inputs()[0].shape _onnx_output_names = [out.name for out in onnx_session.get_outputs()] _onnx_boxes_idx = next(i for i, name in enumerate(_onnx_output_names) if "dets" in name) _onnx_logits_idx = next(i for i, name in enumerate(_onnx_output_names) if "labels" in name) _onnx_feed = {_onnx_input_name: preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)} def _onnx_end2end() -> None: inp = preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels) raw = onnx_session.run(None, {_onnx_input_name: inp}) decode_detections(raw[_onnx_boxes_idx][0], raw[_onnx_logits_idx][0], image.size, threshold=CONFIDENCE_THRESHOLD) onnx_forward = measure_latency( lambda: onnx_session.run(None, _onnx_feed), label="ONNX (CPU) forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) onnx_end2end = measure_latency( _onnx_end2end, label="ONNX (CPU) end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (onnx_forward, onnx_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

6. OpenVINO¶

What it is. OpenVINO IR (Intermediate Representation) is Intel's own model format and toolkit for optimizing and deploying deep learning models on Intel hardware. Good for non-NVIDIA deployment: it targets Intel CPUs, integrated/discrete GPUs, and NPUs from the same exported IR, which matters if the deployment box has no NVIDIA GPU at all. See the OpenVINO export docs.

Export¶

openvino_precision="float32" overrides OpenVINO's default FP16 weight compression — this keeps the IR at full precision for tight parity with the fp32 ONNX/PyTorch baseline above; drop it for a smaller, faster file if you don't need that parity (execution precision still depends on the compiled device regardless of this setting).

In [ ]:

openvino_path = model.export(format="openvino", openvino_precision="float32", output_dir=str(EXPORT_DIR))
openvino_size_mb = _artifact_size_mb(openvino_path, openvino_path.with_suffix(".bin"))
print(f"OpenVINO IR: {openvino_path}  ({openvino_size_mb:.1f} MB, .xml + .bin)")

openvino_path = model.export(format="openvino", openvino_precision="float32", output_dir=str(EXPORT_DIR)) openvino_size_mb = _artifact_size_mb(openvino_path, openvino_path.with_suffix(".bin")) print(f"OpenVINO IR: {openvino_path} ({openvino_size_mb:.1f} MB, .xml + .bin)")

Inference¶

OpenVINOInference is session-tier: it takes already-preprocessed NCHW tensors and returns raw output tensors, decoded here with the same decode_detections helper used for ONNX above. infer() requires a C-contiguous float32 array — preprocess_to_nchw already returns one, but np.ascontiguousarray is applied defensively since infer() raises ValueError otherwise.

In [ ]:

from rfdetr.export.inference import OpenVINOInference

with measure_memory() as mem:
    ov_model = OpenVINOInference(openvino_path, device="CPU")
    ov_input = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels))
    ov_boxes, ov_logits = ov_model(ov_input)
openvino_memory_mb = mem.delta_mb
ov_decoded = decode_detections(ov_boxes[0], ov_logits[0], image.size, threshold=CONFIDENCE_THRESHOLD)
print(f"OpenVINO (CPU): {len(ov_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}")
# Observed on this build: OpenVINO's raw logits run markedly lower-magnitude than ONNX's on the
# same preprocessed input (top logit ~0.05 vs ~3.0 here), so fewer detections clear the same
# 0.5 threshold even at openvino_precision="float32" — a real export/runtime parity gap, not a
# preprocessing bug (verified by feeding both runtimes the identical tensor). If this matters for
# your model, lower the threshold for the OpenVINO path or compare raw scores directly rather than
# post-threshold counts.

ov_sv_detections = sv.Detections(
    xyxy=ov_decoded.xyxy, confidence=ov_decoded.confidence, class_id=ov_decoded.class_id.astype(int)
)
visualize_detections(ov_sv_detections, image, EXPORT_DIR / "annotated_openvino.jpg")

from rfdetr.export.inference import OpenVINOInference with measure_memory() as mem: ov_model = OpenVINOInference(openvino_path, device="CPU") ov_input = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)) ov_boxes, ov_logits = ov_model(ov_input) openvino_memory_mb = mem.delta_mb ov_decoded = decode_detections(ov_boxes[0], ov_logits[0], image.size, threshold=CONFIDENCE_THRESHOLD) print(f"OpenVINO (CPU): {len(ov_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}") # Observed on this build: OpenVINO's raw logits run markedly lower-magnitude than ONNX's on the # same preprocessed input (top logit ~0.05 vs ~3.0 here), so fewer detections clear the same # 0.5 threshold even at openvino_precision="float32" — a real export/runtime parity gap, not a # preprocessing bug (verified by feeding both runtimes the identical tensor). If this matters for # your model, lower the threshold for the OpenVINO path or compare raw scores directly rather than # post-threshold counts. ov_sv_detections = sv.Detections( xyxy=ov_decoded.xyxy, confidence=ov_decoded.confidence, class_id=ov_decoded.class_id.astype(int) ) visualize_detections(ov_sv_detections, image, EXPORT_DIR / "annotated_openvino.jpg")

Benchmark¶

In [ ]:

openvino_forward = measure_latency(
    lambda: ov_model(ov_input), label="OpenVINO forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)


def _openvino_end2end() -> None:
    inp = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels))
    boxes, logits = ov_model(inp)
    decode_detections(boxes[0], logits[0], image.size, threshold=CONFIDENCE_THRESHOLD)


openvino_end2end = measure_latency(
    _openvino_end2end, label="OpenVINO end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (openvino_forward, openvino_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

openvino_forward = measure_latency( lambda: ov_model(ov_input), label="OpenVINO forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) def _openvino_end2end() -> None: inp = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)) boxes, logits = ov_model(inp) decode_detections(boxes[0], logits[0], image.size, threshold=CONFIDENCE_THRESHOLD) openvino_end2end = measure_latency( _openvino_end2end, label="OpenVINO end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (openvino_forward, openvino_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

OpenVINO (fp16, default)¶

Dropping openvino_precision (or passing "float16" explicitly) restores OpenVINO's own default FP16 weight compression — a smaller IR on disk and less memory traffic per inference, at the cost of the numeric parity the float32 row above trades for. Execution precision still depends on the compiled device, not just this setting (see the export docs linked above).

In [ ]:

openvino_fp16_path = model.export(
    format="openvino", openvino_precision="float16", output_dir=str(EXPORT_DIR / "openvino_fp16")
)
openvino_fp16_size_mb = _artifact_size_mb(openvino_fp16_path, openvino_fp16_path.with_suffix(".bin"))
print(f"OpenVINO IR (fp16): {openvino_fp16_path}  ({openvino_fp16_size_mb:.1f} MB, .xml + .bin)")

with measure_memory() as mem:
    ov_fp16_model = OpenVINOInference(openvino_fp16_path, device="CPU")
    ov_fp16_input = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels))
    ov_fp16_boxes, ov_fp16_logits = ov_fp16_model(ov_fp16_input)
openvino_fp16_memory_mb = mem.delta_mb
ov_fp16_decoded = decode_detections(ov_fp16_boxes[0], ov_fp16_logits[0], image.size, threshold=CONFIDENCE_THRESHOLD)
print(f"OpenVINO fp16 (CPU): {len(ov_fp16_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}")

ov_fp16_sv_detections = sv.Detections(
    xyxy=ov_fp16_decoded.xyxy, confidence=ov_fp16_decoded.confidence, class_id=ov_fp16_decoded.class_id.astype(int)
)
visualize_detections(ov_fp16_sv_detections, image, EXPORT_DIR / "annotated_openvino_fp16.jpg")

openvino_fp16_forward = measure_latency(
    lambda: ov_fp16_model(ov_fp16_input),
    label="OpenVINO fp16 forward",
    device="cpu",
    warmup=WARMUP_RUNS,
    runs=MEASURE_RUNS,
)


def _openvino_fp16_end2end() -> None:
    inp = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels))
    boxes, logits = ov_fp16_model(inp)
    decode_detections(boxes[0], logits[0], image.size, threshold=CONFIDENCE_THRESHOLD)


openvino_fp16_end2end = measure_latency(
    _openvino_fp16_end2end, label="OpenVINO fp16 end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (openvino_fp16_forward, openvino_fp16_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

openvino_fp16_path = model.export( format="openvino", openvino_precision="float16", output_dir=str(EXPORT_DIR / "openvino_fp16") ) openvino_fp16_size_mb = _artifact_size_mb(openvino_fp16_path, openvino_fp16_path.with_suffix(".bin")) print(f"OpenVINO IR (fp16): {openvino_fp16_path} ({openvino_fp16_size_mb:.1f} MB, .xml + .bin)") with measure_memory() as mem: ov_fp16_model = OpenVINOInference(openvino_fp16_path, device="CPU") ov_fp16_input = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)) ov_fp16_boxes, ov_fp16_logits = ov_fp16_model(ov_fp16_input) openvino_fp16_memory_mb = mem.delta_mb ov_fp16_decoded = decode_detections(ov_fp16_boxes[0], ov_fp16_logits[0], image.size, threshold=CONFIDENCE_THRESHOLD) print(f"OpenVINO fp16 (CPU): {len(ov_fp16_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}") ov_fp16_sv_detections = sv.Detections( xyxy=ov_fp16_decoded.xyxy, confidence=ov_fp16_decoded.confidence, class_id=ov_fp16_decoded.class_id.astype(int) ) visualize_detections(ov_fp16_sv_detections, image, EXPORT_DIR / "annotated_openvino_fp16.jpg") openvino_fp16_forward = measure_latency( lambda: ov_fp16_model(ov_fp16_input), label="OpenVINO fp16 forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) def _openvino_fp16_end2end() -> None: inp = np.ascontiguousarray(preprocess_to_nchw(image, _onnx_height, _onnx_width, _onnx_channels)) boxes, logits = ov_fp16_model(inp) decode_detections(boxes[0], logits[0], image.size, threshold=CONFIDENCE_THRESHOLD) openvino_fp16_end2end = measure_latency( _openvino_fp16_end2end, label="OpenVINO fp16 end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (openvino_fp16_forward, openvino_fp16_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

7. Results¶

Measured on a Colab CPU runtime: x86_64, 4 physical / 8 logical cores at 2200 MHz, 51 GB RAM, Linux 6.6.122+, rfdetr v1.11.0, batch 1, 15 warmup + 50 timed runs, RFDETRSmall. Numbers vary by architecture (x86 vs ARM) and vendor — rerun the cells below to get yours. Config is the precision/backend used for that row; — marks a scope this format doesn't have (PyTorch's predict() has no forward-only path). Memory [MB] is host resident-memory growth across constructing the runtime plus its first inference call.

Format	Config	forward [ms]	end2end [ms]	FPS [img/s] (end2end)	Memory [MB]
PyTorch predict()	eager, fp32	—	345.76 ± 42.57	2.9	545.5
PyTorch inference()	JIT, fp32	—	327.30 ± 52.05	3.1	213.5
ONNX	CPU EP	360.30 ± 48.62	368.33 ± 40.04	2.7	64.3
OpenVINO	fp32 IR, CPU	316.89 ± 48.16	311.92 ± 46.04	3.2	302.9
OpenVINO	fp16 IR (default), CPU	344.38 ± 48.47	354.55 ± 58.25	2.8	215.0

No format wins decisively on this CPU — read the ± column. Every row's standard deviation is 12–17% of its own mean, and the spread between the nominally fastest row (OpenVINO fp32, 311.92 ms) and the slowest (ONNX, 368.33 ms) is smaller than a single standard deviation. A shared Colab vCPU is a noisy timing environment; treat this table as "all four paths land in the same 300–370 ms band here" rather than as a ranking. The same noise is why OpenVINO fp32's forward (316.89) reads marginally above its own end2end (311.92), which is not physically possible — forward is a strict subset of end2end, and the 5 ms gap is ~1/10th of the ±48 ms noise. On dedicated hardware these rows separate much more cleanly.

fp16 IR does not speed up CPU inference. OpenVINO's default FP16 weight compression came out slower than explicit float32 here (354.55 vs 311.92 ms end2end) while halving the IR on disk. fp16 is a size and memory-bandwidth optimization; a CPU without native fp16 kernels still computes in fp32 after upconverting. The mobile cookbook measures the same effect for TFLite fp16 on an ARM CPU, where only dynamic-range INT8 actually changed the kernels.

OpenVINO parity gap. On the machine this notebook was drafted on, OpenVINO's exported logits ran markedly lower-magnitude than ONNX/PyTorch's on identical input (top logit ≈0.05 vs ≈3.0), yielding fewer detections above the 0.5 threshold on the sample image — a numerical export/runtime parity gap, not a bug in this notebook's pre/postprocessing. Worth checking for on your own hardware; if it matters for your model, lower the threshold for the OpenVINO path or compare raw scores directly rather than post-threshold counts.

In [ ]:

import pandas as pd


def _fmt_ms(result: BenchmarkResult | None) -> str:
    if result is None:
        return "—"
    return f"{result.mean_ms:.2f} ± {result.std_ms:.2f}"


def _result_row(
    format_label: str,
    config: str,
    forward: BenchmarkResult | None,
    end2end: BenchmarkResult | None,
    memory_mb: float | None,
) -> dict:
    fps = (end2end or forward).fps
    return {
        "Format": format_label,
        "Config": config,
        "forward [ms]": _fmt_ms(forward),
        "end2end [ms]": _fmt_ms(end2end),
        "FPS [img/s] (end2end)": round(fps, 1),
        "Memory [MB]": f"{memory_mb:.1f}" if memory_mb is not None else "—",
    }


summary = pd.DataFrame(
    [
        _result_row("PyTorch predict()", "eager, fp32", None, pytorch_eager, pytorch_eager_memory_mb),
        _result_row("PyTorch inference()", "JIT, fp32", None, pytorch_jit, pytorch_jit_memory_mb),
        _result_row("ONNX", "CPU EP", onnx_forward, onnx_end2end, onnx_memory_mb),
        _result_row("OpenVINO", "fp32 IR, CPU", openvino_forward, openvino_end2end, openvino_memory_mb),
        _result_row(
            "OpenVINO", "fp16 IR (default), CPU", openvino_fp16_forward, openvino_fp16_end2end, openvino_fp16_memory_mb
        ),
    ]
).set_index("Format")
print(summary.to_string())
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs, batch 1, CPU.")

import pandas as pd def _fmt_ms(result: BenchmarkResult | None) -> str: if result is None: return "—" return f"{result.mean_ms:.2f} ± {result.std_ms:.2f}" def _result_row( format_label: str, config: str, forward: BenchmarkResult | None, end2end: BenchmarkResult | None, memory_mb: float | None, ) -> dict: fps = (end2end or forward).fps return { "Format": format_label, "Config": config, "forward [ms]": _fmt_ms(forward), "end2end [ms]": _fmt_ms(end2end), "FPS [img/s] (end2end)": round(fps, 1), "Memory [MB]": f"{memory_mb:.1f}" if memory_mb is not None else "—", } summary = pd.DataFrame( [ _result_row("PyTorch predict()", "eager, fp32", None, pytorch_eager, pytorch_eager_memory_mb), _result_row("PyTorch inference()", "JIT, fp32", None, pytorch_jit, pytorch_jit_memory_mb), _result_row("ONNX", "CPU EP", onnx_forward, onnx_end2end, onnx_memory_mb), _result_row("OpenVINO", "fp32 IR, CPU", openvino_forward, openvino_end2end, openvino_memory_mb), _result_row( "OpenVINO", "fp16 IR (default), CPU", openvino_fp16_forward, openvino_fp16_end2end, openvino_fp16_memory_mb ), ] ).set_index("Format") print(summary.to_string()) print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs, batch 1, CPU.")

Next steps¶

• Fine-tuned weights — pass pretrain_weights="<path/to/checkpoint.pth>" when constructing the model.
• Intel GPU / NPU — pass device="GPU" or device="NPU" to OpenVINOInference on Intel hardware that has one; see the OpenVINO export docs.
• Have a CUDA GPU, mobile target, or Apple Silicon? — see the CUDA cookbook, the mobile/edge cookbook, or the Apple cookbook.
• See the Export documentation for every format and option.