RF-DETR on Apple Silicon — Export, Inference & Latency¶
Compares every export path built for Apple Silicon: eager PyTorch (Metal via MPS, as the
reference anchor) against native CoreML, Apple's newer Core AI framework, and ExecuTorch's CoreML
delegate — each exported, loaded through the one public prediction API, RFDETRInference, and
benchmarked across a batch-size matrix.
| Format | Export | Inference |
|---|---|---|
| PyTorch | (no export — eager model) | predict() on device="mps", inference() with inductor (the MPS default) or TorchScript |
inference-models |
(prebuilt by that package) | PyTorch (MPS) and ONNX Runtime (CPU and CoreML execution providers) through AutoModel.from_pretrained() |
| CoreML | model.export(format="coreml", batch_size=N) → .mlpackage |
RFDETRInference(path).predict() |
| Core AI | model.export(format="coreai", batch_size=N) → .aimodel |
RFDETRInference(path).predict() |
| ExecuTorch (CoreML backend) | model.export(format="executorch", backend="coreml", batch_size=N) → .pte |
RFDETRInference(path).predict() |
Not the same as native CoreML. ExecuTorch's
coremlbackend is a different path fromformat="coreml"— it produces a.ptefile for the ExecuTorch runtime, not a.mlpackage. It runs fp16 on the Neural Engine; native CoreML and Core AI default to fp32.
Local macOS only, Apple Silicon, macOS 27+. Running (not just building) a CoreML or Core AI model requires their runtimes, which only exist on macOS / iOS / iPadOS. Core AI additionally needs macOS 27 or later. This notebook will not work on Colab or any Linux runner.
Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), general-purpose desktop/server CPU (see the CPU cookbook), or mobile/edge formats — TFLite, LiteRT, ExecuTorch's XNNPACK backend (see the mobile cookbook). ExecuTorch's
qnn(Qualcomm) backend is Apple-irrelevant and covered nowhere in this series.
Which latency number is this? Every number in this notebook is a Tier C measurement: end-to-end
predict()timings (preprocess + forward + postprocess) for each export format, at batch sizes 1/4/8, on one Apple Silicon Mac, against a PyTorch (MPS) anchor measured on that same machine. Its value is ratios within this machine — never set these milliseconds against the architecture headline on the Benchmarks page (NVIDIA T4, TensorRT FP16, model only) or the number on Roboflow's RF-DETR model page (L4, Roboflow Inference, scope not stated, batch 1) — those are GPU numbers, not comparable to an Apple Silicon run at all, let alone without checking precision, batch size, and scope.
1. Install¶
RFDETR_SPEC holds the install source and defaults to the develop branch. RFDETRInference and the
rfdetr.export.benchmark helpers this notebook imports ship in 1.12; to install from PyPI instead, swap in the
commented >=1.12,<2 line once it is released.
coremltools (pulled in by
[coreml]) also satisfies ExecuTorch's CoreML backend, which needs it but does not declare it as
a dependency of [executorch].
import sys
assert (3, 11) <= sys.version_info[:2] < (3, 14), "Install and run this cookbook with Python 3.11, 3.12, or 3.13."
RFDETR_SPEC = "rfdetr[coreml,coreai,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[coreml,coreai,executorch]>=1.12,<2"
!pip install -q "{RFDETR_SPEC}" "inference-models[onnx-cpu]" psutil supervision pandas tabulate faster-coco-eval
2. Setup¶
Every format in this notebook needs Apple Silicon.
predict()'s postprocessing always ends by moving results to the CPU (Detections is numpy, not
a device tensor), which forces a full MPS stream sync — so timing predict() with a plain
wall-clock is already correct, the same way .cpu() forces a CUDA sync.
The measurement helpers come from the public rfdetr.export.benchmark module, shared with every
other per-hardware cookbook: measure_latency times a callable after warmup, and measure_memory
measures the host resident-memory growth of constructing a runtime and running its first
inference call. The installed rfdetr version is
printed so the Results header can be filled from it.
Every timed rfdetr predict() call passes include_source_image=False. inference-models returns no copy of the input image in its results, so rfdetr is timed without one too; with the default True it also copies each image into its results.
Every timed call takes the same decoded RGB array, frame, as a camera or video decoder delivers it, so no row pays a JPEG decode or a PIL-to-array copy.
MODEL_SIZE (nano, small, medium or large) and TASK (detection or segmentation) pick the model every row measures. A segmentation call also decodes masks on the host, so its latencies are not comparable with detection, and the mAP rows stay box mAP for both tasks.
import importlib.metadata
import platform
from functools import partial
from pathlib import Path
import torch
from PIL import Image
from rfdetr import RFDETRInference
from rfdetr.export._benchmark import _artifact_size_mb, visualize_detections
from rfdetr.export.benchmark import measure_latency, measure_memory
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")
import rfdetr
MODEL_SIZE = "small" # nano, small, medium or large
TASK = "detection" # "detection" or "segmentation"
MODEL_CLASS = f"RFDETR{'Seg' if TASK == 'segmentation' else ''}{MODEL_SIZE.capitalize()}" # class name in `rfdetr`
MODEL_ID = f"rfdetr-{'seg-' if TASK == 'segmentation' else ''}{MODEL_SIZE}" # `inference-models` id, names the files
Notebook display helpers¶
_artifact_size_mb and visualize_detections come from the private rfdetr.export._benchmark module, next to the
COCO mAP helpers used later. They size artifacts and draw detections; neither changes what a runtime computes.
# Inline figures keep every plot in the notebook; closing them after display stops figures piling up in memory.
%matplotlib inline
%config InlineBackend.close_figures = True
BATCH_SIZES = (1, 4, 8) is the batch matrix every format below is benchmarked at. rows accumulates one record per
(format, config, batch) triple; record() appends it, and the Results section turns rows into the summary tables.
artifact_paths remembers each export's batch-1 artifact so the COCO section scores exactly the files that were timed.
from typing import NamedTuple
if platform.system() != "Darwin" or platform.machine() != "arm64":
raise RuntimeError("This notebook requires Apple Silicon (macOS, arm64); none is available here.")
print(f"macOS {platform.mac_ver()[0]}, {platform.machine()}")
# Core AI's runtime ships with macOS 27. On an older macOS the same package loads and runs, but on the CPU only
# (measured ~3.6 s per call, identical to device="cpu"), so its rows would be wrong. Sections 7's cells are skipped there.
COREAI_RUNTIME_AVAILABLE = int(platform.mac_ver()[0].split(".")[0]) >= 27
print(f"Core AI runtime usable: {COREAI_RUNTIME_AVAILABLE}")
PYTORCH_DEVICE = "mps" if torch.backends.mps.is_available() else "cpu"
print(f"PyTorch device: {PYTORCH_DEVICE}")
EXPORT_DIR = Path("export_apple")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 5
MEASURE_RUNS = 30
BATCH_SIZES = (1, 4, 8)
#: One record per (format, config, batch) benchmark, appended to by `record()`.
rows: list[dict] = []
#: Batch-1 artifact of every export, keyed by (Format, Config) — the files the COCO section scores.
artifact_paths: dict[tuple[str, str], Path] = {}
#: Every batch's artifact, keyed by (Format, Config) then batch size — the folder throughput section loads batch 8.
artifacts_by_batch: dict[tuple[str, str], dict[int, Path]] = {}
class Config(NamedTuple):
"""The fixed options of one benchmark row, one table column each."""
runtime: str
precision: str
quantization: str = "—"
def record(
format_label: str, config: Config, batch: int, result, memory_mb: float, size_mb: float | None = None
) -> None:
"""Append one benchmark result; `FPS` counts images per second, so it compares across batch sizes."""
rows.append(
{
"Format": format_label,
"Runtime": config.runtime,
"Precision": config.precision,
"Quantization": config.quantization,
"Batch": batch,
"end2end [ms]": result.mean_ms,
"± [ms]": result.std_ms,
"FPS [img/s]": batch * result.fps,
"Memory [MB]": memory_mb,
"Size [MB]": size_mb,
}
)
3. Sample image¶
A single street scene with several COCO classes is enough to verify detections. The image is downloaded once and reused for every format below.
import urllib.request
import numpy as np
IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)
image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")
# A camera or video decoder delivers decoded arrays, so every timed call below takes this RGB array and no
# call pays a PIL-to-array copy or a JPEG decode.
frame = np.array(image)
4. PyTorch baseline — predict() on Metal (MPS)¶
No export needed — this is the reference every other format in this notebook is compared
against. predict() includes preprocessing and postprocessing — there is no separate
"forward-only" path through it, so only an end-to-end number is reported for the baseline.
predict() also accepts a list of images and batches them internally, so the batch matrix below
feeds it batch copies of the same sample image rather than exporting anything.
with measure_memory() as mem:
model = getattr(rfdetr, MODEL_CLASS)(device=PYTORCH_DEVICE)
baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_baseline_memory_mb = mem.delta_mb
print(f"PyTorch baseline ({PYTORCH_DEVICE}): {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)
Memory [MB] for PyTorch is measured once above (constructing the runtime is a one-time cost,
unlike the fixed-batch formats below which reload a fresh artifact per batch) and reused for
every batch row in the summary table.
for batch in BATCH_SIZES:
images_batch = [frame] * batch
pytorch_baseline = measure_latency(
partial(model.predict, images_batch, include_source_image=False),
label=f"PyTorch predict() ({PYTORCH_DEVICE}) batch={batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(
f" {pytorch_baseline.label:<40} {pytorch_baseline.mean_ms:6.2f} ms ± {pytorch_baseline.std_ms:5.2f}"
f" ({batch * pytorch_baseline.fps:6.1f} FPS)"
)
record("PyTorch", Config("predict() eager", "fp32"), batch, pytorch_baseline, pytorch_baseline_memory_mb)
PyTorch inference(dtype=torch.float16, compile_backend="torchscript") — fp16+TorchScript anchor¶
model.inference(dtype=torch.float16, batch_size=N, compile_backend="torchscript") returns an independent TorchScript
predictor compiled for exactly batch_size images — calling predict() with a different batch
afterward raises ValueError: Batch size mismatch (confirmed by an earlier run of this notebook).
Each batch gets its own predictor; deleting it releases that runtime and leaves model ready for
the export calls that follow.
for batch in BATCH_SIZES:
images_batch = [frame] * batch
with measure_memory() as mem:
predictor = model.inference(dtype=torch.float16, batch_size=batch, compile_backend="torchscript")
_ = predictor.predict(images_batch, include_source_image=False)
pytorch_fp16_jit_memory_mb = mem.delta_mb
pytorch_fp16_jit = measure_latency(
partial(predictor.predict, images_batch, include_source_image=False),
label=f"PyTorch inference() fp16+TorchScript ({PYTORCH_DEVICE}) batch={batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(
f" {pytorch_fp16_jit.label:<40} {pytorch_fp16_jit.mean_ms:6.2f} ms ± {pytorch_fp16_jit.std_ms:5.2f}"
f" ({batch * pytorch_fp16_jit.fps:6.1f} FPS) +{pytorch_fp16_jit_memory_mb:.1f} MB"
)
record("PyTorch", Config("inference() TorchScript", "fp16"), batch, pytorch_fp16_jit, pytorch_fp16_jit_memory_mb)
del predictor
PyTorch inference() with inductor — the MPS default¶
On MPS model.inference() compiles with inductor when the installed PyTorch has its MPS code generator, so a plain model.inference(batch_size=N) is not the TorchScript predictor of the cell above. This section names the backend, compile_backend="inductor", so the row says what ran, and measures it at fp32 and fp16. Inductor compiles for every new input shape, several seconds per batch size, and the compile time is inside the Memory [MB] window and outside the timed calls. On a Mac this is the fastest PyTorch route the cookbook measures, and the one to compare an export against.
for dtype in (torch.float32, torch.float16):
precision = "fp32" if dtype == torch.float32 else "fp16"
for batch in BATCH_SIZES:
images_batch = [frame] * batch
with measure_memory() as mem:
predictor = model.inference(dtype=dtype, batch_size=batch, compile_backend="inductor")
_ = predictor.predict(images_batch, include_source_image=False)
inductor = measure_latency(
partial(predictor.predict, images_batch, include_source_image=False),
label=f"PyTorch inference() inductor {precision} ({PYTORCH_DEVICE}) batch={batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(
f" {inductor.label:<52} {inductor.mean_ms:6.2f} ms ± {inductor.std_ms:5.2f}"
f" ({batch * inductor.fps:6.1f} FPS) +{mem.delta_mb:.1f} MB"
)
record("PyTorch", Config("inference() inductor", precision), batch, inductor, mem.delta_mb)
del predictor
5. Export, load, benchmark — one path for every format¶
Every export below goes through the same two public calls: model.export(format=..., batch_size=N) writes the artifact,
and RFDETRInference(path) loads it for prediction. The loader reads the artifact's own metadata, so the same
predict() that serves the PyTorch rows above also serves CoreML, Core AI and ExecuTorch: preprocessing, postprocessing,
class ids and sv.Detections output are shared, and no per-format tensor names, dtypes, decoders or async wrappers are
needed.
benchmark_export exports one artifact per batch size, because CoreML, Core AI and ExecuTorch all bake a fixed batch
shape into the file. For each artifact it loads a predictor, runs it once on the sample image (batch 1 prints the
detection count and draws the result), and times predict() on batch copies of the sample. Memory [MB] is the host
resident-memory growth across constructing the predictor plus its first call.
Only end-to-end predict() timings are reported: the PyTorch rows have no forward-only path, so one scope keeps every
row of the table comparable. Correctness is checked by the COCO mAP section below, which scores these same batch-1
artifacts, instead of by comparing raw tensors against a reference runtime.
def benchmark_export(format_label: str, config: Config, subdir: str, **export_kwargs) -> None:
artifacts_by_batch.setdefault((format_label, config), {})
"""Export one artifact per batch size, load each with `RFDETRInference` and time its `predict()`."""
for batch in BATCH_SIZES:
path = Path(model.export(batch_size=batch, output_dir=str(EXPORT_DIR / f"{subdir}_b{batch}"), **export_kwargs))
size_mb = _artifact_size_mb(path)
artifacts_by_batch[(format_label, config)][batch] = path
images_batch = [frame] * batch
with measure_memory() as mem:
predictor = RFDETRInference(path)
detections = predictor.predict(images_batch, threshold=CONFIDENCE_THRESHOLD)
if batch == 1:
artifact_paths[(format_label, config)] = path
print(f"{format_label} ({', '.join(config)}): {path} ({size_mb:.1f} MB)")
print(f" {len(detections[0])} detections above {CONFIDENCE_THRESHOLD}; runtime: {predictor.runtime_info}")
visualize_detections(detections[0], image, EXPORT_DIR / f"annotated_{subdir}.jpg")
result = measure_latency(
partial(predictor.predict, images_batch, include_source_image=False),
label=f"{format_label} {config} batch={batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(f" batch={batch}: {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} ({batch * result.fps:6.1f} FPS)")
record(format_label, config, batch, result, mem.delta_mb, size_mb)
del predictor
6. Native CoreML¶
What it is. Native CoreML export produces a .mlpackage you can drag directly into Xcode —
no ONNX, no ExecuTorch runtime involved. Good for Apple-native (iOS/macOS) app development:
it's the lowest-friction path into Apple's own ML stack, distinct from ExecuTorch's coreml
backend below (a different export path that produces a .pte, not a .mlpackage). See the
CoreML export docs.
CoreML (fp32, default)¶
Export defaults to coreml_precision="float32" for tight CPU parity with eager PyTorch. The .mlpackage is loaded with
coremltools' default ComputeUnit.ALL; the printed runtime_info shows what RFDETRInference selected.
benchmark_export("CoreML", Config("—", "fp32"), "coreml", format="coreml")
CoreML (fp16)¶
coreml_precision="float16" halves the weight precision, producing a smaller, ANE-oriented
bundle. The tradeoff is larger numeric drift: fp16 narrows the fp32 safety margin around
near-tied torch.topk query-ranking decisions, so a rank swap (and therefore a slightly
different detection set) becomes more likely than in the fp32 row above. The mAP section below measures it.
benchmark_export("CoreML", Config("—", "fp16"), "coreml_fp16", format="coreml", coreml_precision="float16")
7. Core AI¶
What it is. Core AI is Apple's newer
on-device inference framework (iOS, iPadOS, macOS 27+). RF-DETR traces the model with
torch.export and converts it to an .aimodel asset — no ONNX step. Good for apps
targeting the newest Apple OS versions: the runtime decides at load time whether the CPU, GPU,
or Neural Engine executes it, rather than that choice being baked in at export time like CoreML.
See the Core AI export docs.
Core AI (fp32, default)¶
The .aimodel bakes a fixed batch shape into the asset, so benchmark_export writes one per batch size. Core AI's
Python runtime is partly async, but RFDETRInference hides that: predict() is an ordinary synchronous call, also
inside a notebook's running event loop. With the default specialization the runtime allows the CPU, GPU and Neural
Engine and sets no preferred unit, so the choice is its own, made at load time.
if COREAI_RUNTIME_AVAILABLE:
benchmark_export("Core AI", Config("—", "fp32"), "coreai", format="coreai")
Core AI (fp16)¶
coreai_precision="float16" traces and stores the whole graph in fp16, so the asset takes and returns float16
tensors; RFDETRInference casts the input and the outputs, so the call site is unchanged. One part of the graph stays
fp32 regardless: RF-DETR forces topk back to float32 even in an fp16 export, because the Neural Engine's fp16 topk
returns corrupt indices and the two-stage query selection would gather the wrong encoder tokens. That caps how much of
the graph actually runs in fp16.
if COREAI_RUNTIME_AVAILABLE:
benchmark_export("Core AI", Config("—", "fp16"), "coreai_fp16", format="coreai", coreai_precision="float16")
8. ExecuTorch (CoreML backend)¶
What it is. This is the same ExecuTorch runtime as the mobile cookbook's
XNNPACK section, but lowered to Apple's coreml backend instead — still a portable .pte
file, no .mlpackage, but running on the Apple Neural Engine. Good for staying inside the
ExecuTorch/PyTorch deployment story while still reaching Apple's Neural Engine, when native
CoreML's separate export/runtime isn't a fit; the tradeoff is fp16 only, unlike native CoreML and
Core AI's fp32 default. See the
ExecuTorch export docs.
ExecuTorch (CoreML backend, fp16)¶
backend="coreml" targets the Apple Neural Engine, iPhone/iPad/Mac, and runs fp16 — unlike native CoreML and Core AI
above, which default to fp32. This is the ExecuTorch delegate, not native CoreML: it produces a .pte for the
ExecuTorch runtime, distinct from format="coreml". ExecuTorch also bakes a fixed batch shape into the .pte, so
benchmark_export writes one per batch size. RFDETRInference owns the runtime details the raw Runtime API leaves to
the caller: a contiguous input tensor, a fresh input buffer per call and the torch/ExecuTorch compatibility check.
rfdetr passes ExecuTorch's CoreML partitioner no compile specs, so the backend resolves compute units to its own default
when lowering.
benchmark_export("ExecuTorch", Config("CoreML backend", "fp16"), "executorch", format="executorch", backend="coreml")
8.1 ONNX Runtime (CPU)¶
rfdetr's own ONNX export, loaded with RFDETRInference on ONNX Runtime's CPU execution provider. It is the counterpart of the inference-models ONNX rows in Section 9, so the two packages can be compared on the same runtime.
benchmark_export("ONNX", Config("CPU EP", "fp32"), "onnx", format="onnx")
9. inference-models side by side¶
inference-models is Roboflow's separate deployment package: AutoModel.from_pretrained("rfdetr-small", backend=...) downloads a prebuilt model, which is then called on a list of images. Its PyTorch and ONNX Runtime backends run here next to the rfdetr rows above, at the same batch sizes and the same end-to-end scope.
Two call settings keep the comparison fair:
input_color_format="rgb": a NumPy array is BGR to this package unless declared. Without it the network sees swapped channels and the scores drop, with no error.confidence=: passed on every call, so both packages use the same threshold.
On a Mac the ONNX backend defaults to CoreML's execution provider, so both providers are listed as separate rows. The package publishes no CoreML model for rfdetr-small (only TensorRT, ONNX and PyTorch), and it has no OpenVINO, TFLite or ExecuTorch backend.
import supervision as sv
from inference_models import AutoModel, BackendType
def im_predict(im_model, frames: list[np.ndarray], threshold: float) -> list[sv.Detections]:
"""Run an `inference-models` model on RGB arrays and return `sv.Detections`, like `RFDETRInference.predict`."""
return [p.to_supervision() for p in im_model(frames, confidence=threshold, input_color_format="rgb")]
#: `inference-models` models by (Format, Config), kept for the COCO section.
im_models: dict[tuple[str, Config], object] = {}
def benchmark_inference_models(format_label: str, config: Config, **load_kwargs) -> None:
"""Load one `inference-models` model, check it once on the sample image and time `im_predict` at every batch size."""
for batch in BATCH_SIZES:
with measure_memory() as mem:
im_model = AutoModel.from_pretrained(MODEL_ID, **load_kwargs)
detections = im_predict(im_model, [frame] * batch, CONFIDENCE_THRESHOLD)
if batch == BATCH_SIZES[0]:
im_models[(format_label, config)] = im_model
print(f"{format_label} ({', '.join(config)}): {len(detections[0])} detections above {CONFIDENCE_THRESHOLD}")
result = measure_latency(
partial(im_predict, im_model, [frame] * batch, CONFIDENCE_THRESHOLD),
label=f"{format_label} {config} batch={batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
print(f" batch={batch}: {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} ({batch * result.fps:6.1f} FPS)")
record(format_label, config, batch, result, mem.delta_mb)
del im_model
benchmark_inference_models(
"PyTorch", Config("inference-models", "fp32"), backend=BackendType.TORCH, device=PYTORCH_DEVICE
)
benchmark_inference_models(
"ONNX",
Config("inference-models CPU EP", "fp32"),
backend=BackendType.ONNX,
onnx_execution_providers=["CPUExecutionProvider"],
)
benchmark_inference_models(
"ONNX",
Config("inference-models CoreML EP", "fp32"),
backend=BackendType.ONNX,
onnx_execution_providers=["CoreMLExecutionProvider", "CPUExecutionProvider"],
)
try:
AutoModel.from_pretrained(MODEL_ID, backend=BackendType.COREML)
except Exception as error: # the package names the failure; print it rather than assume it
print(f"inference-models CoreML backend: {type(error).__name__}")
10. COCO val2017 accuracy¶
Latency alone hides a precision collapse: an export can run at a normal speed and still return
nothing useful (an OpenVINO f16 execution bug once gave 0 detections). So every batch-1 row of the
Results table below also gets a box mAP, measured on the same exported file at the same
precision that row timed — the batch-1 .mlpackage, .aimodel, and .pte the export loops wrote, and the
PyTorch model in the same mode.
This is a 500-image seeded subset of COCO val2017, so it is not the full-val2017 number on the
Benchmarks page; set FULL_VAL = True to
score all 5000 images. Each row costs roughly its batch-1 end2end latency × the image count — about
6–12 s per row on the M4 Max at 500 images, ten times that with FULL_VAL. The first run downloads
the 241 MB annotation archive once, plus only the selected images.
Every row is scored the same way: evaluate_coco_map calls predict(image, threshold=0.001) and reads the returned
sv.Detections, whose class_id values are COCO category IDs for a pretrained checkpoint. The 0.001 score floor keeps
the whole precision-recall curve. Because no per-format decoding is involved, an exported row differs from the PyTorch
row only by what the export itself changed.
Display and scoring helpers come from the private rfdetr.export._benchmark module; they are not a
supported API and do not change what any runtime computes.
from rfdetr.export._benchmark import CocoMapResult, evaluate_coco_map, fetch_coco_val2017
N_EVAL = 500
FULL_VAL = False
COCO_SEED = 0
COCO_ROOT = Path("coco")
COCO_SCORE_FLOOR = 0.001 # keeps the whole precision-recall curve
#: One entry per batch-1 Results row, keyed by that row's (Format, Config).
map_results: dict[tuple[str, str], CocoMapResult] = {}
The annotations and the selected images land in COCO_ROOT; files already on disk are skipped, so
rerunning this cell after the first download costs nothing.
coco_subset = fetch_coco_val2017(COCO_ROOT, n_images=None if FULL_VAL else N_EVAL, seed=COCO_SEED)
print(f"COCO val2017: {len(coco_subset.image_ids)} images in {coco_subset.images_dir}")
PyTorch first: predict() on MPS in fp32 is the eager reference every exported row is compared against, then the
fp16+JIT predictor at batch 1, the same inference() call as its latency row.
map_results[("PyTorch", Config("predict() eager", "fp32"))] = evaluate_coco_map(
partial(model.predict, threshold=COCO_SCORE_FLOOR), coco_subset
)
for dtype in (torch.float32, torch.float16):
precision = "fp32" if dtype == torch.float32 else "fp16"
predictor = model.inference(dtype=dtype, batch_size=1, compile_backend="inductor")
map_results[("PyTorch", Config("inference() inductor", precision))] = evaluate_coco_map(
partial(predictor.predict, threshold=COCO_SCORE_FLOOR), coco_subset
)
del predictor
predictor = model.inference(dtype=torch.float16, batch_size=1, compile_backend="torchscript")
map_results[("PyTorch", Config("inference() TorchScript", "fp16"))] = evaluate_coco_map(
partial(predictor.predict, threshold=COCO_SCORE_FLOOR), coco_subset
)
del predictor
Then every exported artifact, loaded exactly as in its latency row. A near-tied topk rank swap in an fp16 export
(see the CoreML fp16 note) shows up here as lost accuracy rather than only as a different detection set.
for (format_label, config), path in artifact_paths.items():
predictor = RFDETRInference(path)
map_results[(format_label, config)] = evaluate_coco_map(
partial(predictor.predict, threshold=COCO_SCORE_FLOOR), coco_subset
)
del predictor
print(
f" {format_label} ({', '.join(config)}): mAP@50:95 {100 * map_results[(format_label, config)].map50_95:.1f} %"
)
Then the inference-models models of Section 9, each scored through the same batch-1 call as its latency row with confidence=0.001. Their class_id values are COCO category IDs, as the sample image shows (dog 18, person 1, backpack 27, handbag 31 in both packages' output), so the same ground truth applies.
for (format_label, config), im_model in im_models.items():
map_results[(format_label, config)] = evaluate_coco_map(
lambda img, m=im_model: im_predict(m, [np.array(img)], COCO_SCORE_FLOOR)[0], coco_subset
)
print(
f" {format_label} ({', '.join(config)}): mAP@50:95 {100 * map_results[(format_label, config)].map50_95:.1f} %"
)
11. Throughput on a folder¶
The batch rows call predict() on one image repeated. A real job predicts a folder: predict(folder) reads and decodes images on worker threads ahead of the model and runs them in batches. A runtime with a dynamic batch axis runs up to eight images per call, a fixed batch-1 artifact runs one, and a compiled or fixed-batch-8 runtime takes batch=8. include_source_image=False skips keeping the decoded image with each result.
N_FOLDER eval images are copied into one folder and the median of FOLDER_REPS timed passes (after one warm-up pass) gives images per second over the whole call: decode, preprocessing, model, postprocessing.
import shutil
import time
from tqdm.auto import tqdm
N_FOLDER = 64 # a multiple of 8, so every batch of the compiled and fixed-batch-8 runtimes is full
FOLDER_REPS = 3
from tempfile import TemporaryDirectory
# Own the benchmark images without deleting a pre-existing export folder.
folder_workspace = TemporaryDirectory(prefix="rfdetr-folder-")
FOLDER_DIR = Path(folder_workspace.name)
for image_id in tqdm(coco_subset.image_ids[:N_FOLDER], desc="Copy images"):
shutil.copy(coco_subset.images_dir / f"{image_id:012d}.jpg", FOLDER_DIR)
assert len(list(FOLDER_DIR.glob("*.jpg"))) == N_FOLDER
folder_rows: list[dict] = []
def measure_folder(format_label: str, config: Config, predictor, **predict_kwargs) -> None:
"""Time `predictor.predict(folder)` end to end and record the median images per second."""
kwargs = {"threshold": CONFIDENCE_THRESHOLD, "include_source_image": False, **predict_kwargs}
predictor.predict(str(FOLDER_DIR), **kwargs) # warm-up pass
seconds = []
for _ in range(FOLDER_REPS):
start = time.perf_counter()
results = predictor.predict(str(FOLDER_DIR), **kwargs)
seconds.append(time.perf_counter() - start)
assert len(results) == N_FOLDER
img_s = N_FOLDER / sorted(seconds)[len(seconds) // 2]
folder_rows.append(
{"Format": format_label, "Runtime": config.runtime, "Precision": config.precision, "Folder [img/s]": img_s}
)
print(f" {format_label} ({', '.join(config)}): {img_s:6.1f} img/s")
for dtype in (torch.float32, torch.float16):
precision = "fp32" if dtype == torch.float32 else "fp16"
predictor = model.inference(dtype=dtype, batch_size=8, compile_backend="inductor")
measure_folder("PyTorch", Config("inference() inductor", precision), predictor, batch=8)
del predictor
for (format_label, config), by_batch in tqdm(artifacts_by_batch.items(), desc="Folder throughput"):
predictor = RFDETRInference(by_batch[8])
measure_folder(format_label, config, predictor, batch=8)
del predictor
12. Results¶
The last cell prints a decision table (one row per format and config: batch-1 latency, speed against PyTorch predict(), size, mAP@50:95 and FPS at each batch size) and a detail table with ± std and memory growth. Speed vs PyTorch × divides the PyTorch predict() batch-1 latency by the row's own; every latency is the end-to-end predict() call, and mAP@50:95 is scored on the same batch-1 artifact that was timed.
The numbers from the last run on an Apple M4 Max, with their notes, are kept on the CoreML export page so this notebook stays short. Core AI rows appear only on macOS 27 or later; on an older macOS the package runs on the CPU only, so the notebook skips them.
import pandas as pd
from IPython.display import display
KEYS = ["Format", "Runtime", "Precision", "Quantization"]
detail = pd.DataFrame(rows)
# Decision table: one row per (format, config); batch-1 latency, accuracy and size, throughput at every batch.
fps_by_batch = detail.pivot_table(index=KEYS, columns="Batch", values="FPS [img/s]", sort=False).add_prefix("FPS b")
map_by_row = pd.Series(
{(format_label, *config): 100 * result.map50_95 for (format_label, config), result in map_results.items()},
name="mAP@50:95 [%]",
)
map_by_row.index.names = KEYS
headline = (
detail[detail["Batch"] == 1]
.set_index(KEYS)[["end2end [ms]", "Size [MB]"]]
.rename(columns={"end2end [ms]": "b1 latency [ms]"})
.join(map_by_row)
.join(fps_by_batch)
)
headline.insert(1, "Speed vs PyTorch ×", headline["b1 latency [ms]"].iloc[0] / headline["b1 latency [ms]"])
# One number format for every table in this section: one decimal, "—" where a row has no value.
decision = headline.apply(lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}"))
display(decision)
Markdown for the docs page: the table above, unchanged.
print(
decision.reset_index().to_markdown(index=False, disable_numparse=True)
) # keep the one-decimal strings as they are
Per-batch detail: ± std and memory growth for every row.
numeric = detail.columns.difference(["Format", "Runtime", "Precision", "Quantization", "Batch"])
detail_table = detail.copy()
detail_table[numeric] = detail[numeric].apply(
lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}")
)
print(detail_table.to_markdown(index=False))
print(
f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, {platform.machine()}, "
f"macOS {platform.mac_ver()[0]}."
)
Folder throughput of Section 11, images per second over a whole predict(folder) call:
folder = pd.DataFrame(folder_rows)
folder["Folder [img/s]"] = folder["Folder [img/s]"].map(lambda value: f"{value:.1f}")
display(folder)
Next steps¶
- Fine-tuned weights — pass
pretrain_weights="<path/to/checkpoint.pth>"when constructing the model. - Smaller / faster bundle —
coreml_precision="float16"(CoreML) orcoreai_precision="float16"(Core AI) trade numeric drift for size and, on some Apple Silicon generations, latency; see the Core AI export docs for measured per-device tradeoffs. - Deploy in Xcode — drag the
.mlpackageinto your Xcode project, or load the.aimodel/.ptewith the Core AI or ExecuTorch runtime for that platform. - Not on Apple Silicon? — see the CUDA cookbook, the CPU cookbook, or the mobile/edge cookbook.
- See the Export documentation for every format and option.