RF-DETR on Apple Silicon — Export, Inference & Latency¶

Compares every export path built for Apple Silicon: eager PyTorch (Metal via MPS, as the reference anchor) against native CoreML, Apple's newer Core AI framework, and ExecuTorch's CoreML delegate — each exported, run once for a correctness check, then benchmarked across a batch-size matrix.

Format	Export	Inference
PyTorch	(no export — eager model)	predict() on device="mps"
CoreML	model.export(format="coreml", batch_size=N) → .mlpackage	coremltools.models.MLModel
Core AI	model.export(format="coreai", batch_size=N) → .aimodel	coreai.runtime (async)
ExecuTorch (CoreML backend)	model.export(format="executorch", backend="coreml", batch_size=N) → .pte	executorch.runtime.Runtime

Not the same as native CoreML. ExecuTorch's coreml backend is a different path from format="coreml" — it produces a .pte file for the ExecuTorch runtime, not a .mlpackage. It runs fp16 on the Neural Engine; native CoreML and Core AI default to fp32.

Local macOS only, Apple Silicon, macOS 27+. Running (not just building) a CoreML or Core AI model requires their runtimes, which only exist on macOS / iOS / iPadOS. Core AI additionally needs macOS 27 or later. This notebook will not work on Colab or any Linux runner.

Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), general-purpose desktop/server CPU (see the CPU cookbook), or mobile/edge formats — TFLite, LiteRT, ExecuTorch's XNNPACK backend (see the mobile cookbook). ExecuTorch's qnn (Qualcomm) backend is Apple-irrelevant and covered nowhere in this series.

Which latency number is this? Every number in this notebook is a Tier C measurement: forward-only and end-to-end timings for each export format, at batch sizes 1/4/8, on one Apple Silicon Mac, against a PyTorch (MPS) anchor measured on that same machine. Its value is ratios within this machine — never set these milliseconds against the architecture headline on the Benchmarks page (NVIDIA T4, TensorRT FP16, model only) or the number on Roboflow's RF-DETR model page (L4, Roboflow Inference, scope not stated, batch 1) — those are GPU numbers, not comparable to an Apple Silicon run at all, let alone without checking precision, batch size, and scope.

1. Install¶

RFDETR_SPEC holds the install source and defaults to the develop branch, to pick up the newest export fixes; to install the 1.11.2 release from PyPI instead, swap in the commented >=1.11.2,<2 line. coremltools (pulled in by [coreml]) also satisfies ExecuTorch's CoreML backend, which needs it but does not declare it as a dependency of [executorch].

In [ ]:

import sys

assert (3, 11) <= sys.version_info[:2] < (3, 14), "Install and run this cookbook with Python 3.11, 3.12, or 3.13."

import sys assert (3, 11) <= sys.version_info[:2] < (3, 14), "Install and run this cookbook with Python 3.11, 3.12, or 3.13."

In [ ]:

RFDETR_SPEC = "rfdetr[coreml,coreai,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[coreml,coreai,executorch]>=1.11.2,<2"
!pip install -q "{RFDETR_SPEC}" psutil supervision pandas

RFDETR_SPEC = "rfdetr[coreml,coreai,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" # RFDETR_SPEC = "rfdetr[coreml,coreai,executorch]>=1.11.2,<2" !pip install -q "{RFDETR_SPEC}" psutil supervision pandas

2. Setup¶

Every format in this notebook needs Apple Silicon; Core AI additionally needs macOS 27+. predict()'s postprocessing always ends by moving results to the CPU (Detections is numpy, not a device tensor), which forces a full MPS stream sync — so timing predict() with a plain wall-clock is already correct, the same way .cpu() forces a CUDA sync.

Several small helpers are shared by every format section below and every other per-hardware cookbook; they now live once, privately, in rfdetr.export._benchmark, instead of being copy-pasted per cookbook: _artifact_size_mb reports an export artifact's size on disk (a single file, or the total of a directory bundle — needed for CoreML's .mlpackage and Core AI's .aimodel, both directories), visualize_detections annotates and displays a supervision.Detections on the sample image, _fmt_ms/_result_row build the summary table's rows, and measure_memory measures the host resident-memory growth of constructing a runtime and running its first inference call.

Alongside those, BATCH_SIZES = (1, 4, 8) is the batch matrix every format below is benchmarked at, results accumulates one row per (format, config, batch) triple for the summary table in §8, and three batching helpers build the repeated-sample-image input each format needs: _batched_pixel_values (numpy NCHW, for CoreML/Core AI), _batched_pixel_tensor (torch NCHW, for ExecuTorch), and _batched_target_sizes (repeats the sample image's (height, width) once per batch element, for post_process).

In [ ]:

import importlib.metadata
import platform
from functools import partial
from pathlib import Path

import numpy as np
import supervision as sv
import torch
from PIL import Image

from rfdetr.export._benchmark import (
    BenchmarkResult,
    _artifact_size_mb,
    _enable_notebook_inline_matplotlib,
    _result_row,
    measure_latency,
    measure_memory,
    visualize_detections,
)

if platform.system() != "Darwin" or platform.machine() != "arm64":
    raise RuntimeError("This notebook requires Apple Silicon (macOS, arm64); none is available here.")
print(f"macOS {platform.mac_ver()[0]}, {platform.machine()}")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")

PYTORCH_DEVICE = "mps" if torch.backends.mps.is_available() else "cpu"
print(f"PyTorch device: {PYTORCH_DEVICE}")

EXPORT_DIR = Path("export_apple")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 5
MEASURE_RUNS = 30
BATCH_SIZES = (1, 4, 8)

#: One row per (format, config, batch) benchmark, appended to as each section below runs; turned
#: into the summary table and DataFrame in §8.
results: list[dict] = []


def _batched_pixel_values(transform, image: Image.Image, batch: int) -> np.ndarray:
    """Repeat one preprocessed sample image into a ``(batch, C, H, W)`` float32 NCHW array.

    A plain latency benchmark, not a detection-diversity test — every fixed-batch format below times against N copies of
    the same image rather than N distinct images.
    """
    tensor, _ = transform(image, None)
    return np.repeat(tensor[None].numpy(), batch, axis=0)


def _batched_pixel_tensor(transform, image: Image.Image, batch: int) -> torch.Tensor:
    """Repeat one preprocessed sample image into a contiguous ``(batch, C, H, W)`` float32 tensor."""
    tensor, _ = transform(image, None)
    return tensor[None].float().contiguous().repeat(batch, 1, 1, 1)


def _batched_target_sizes(image: Image.Image, batch: int) -> torch.Tensor:
    """Repeat the sample image's ``(height, width)`` once per batch element, for ``post_process``."""
    return torch.tensor([[image.height, image.width]] * batch)


_enable_notebook_inline_matplotlib()

import importlib.metadata import platform from functools import partial from pathlib import Path import numpy as np import supervision as sv import torch from PIL import Image from rfdetr.export._benchmark import ( BenchmarkResult, _artifact_size_mb, _enable_notebook_inline_matplotlib, _result_row, measure_latency, measure_memory, visualize_detections, ) if platform.system() != "Darwin" or platform.machine() != "arm64": raise RuntimeError("This notebook requires Apple Silicon (macOS, arm64); none is available here.") print(f"macOS {platform.mac_ver()[0]}, {platform.machine()}") print(f"rfdetr: {importlib.metadata.version('rfdetr')}") PYTORCH_DEVICE = "mps" if torch.backends.mps.is_available() else "cpu" print(f"PyTorch device: {PYTORCH_DEVICE}") EXPORT_DIR = Path("export_apple") EXPORT_DIR.mkdir(exist_ok=True) CONFIDENCE_THRESHOLD = 0.5 WARMUP_RUNS = 5 MEASURE_RUNS = 30 BATCH_SIZES = (1, 4, 8) #: One row per (format, config, batch) benchmark, appended to as each section below runs; turned #: into the summary table and DataFrame in §8. results: list[dict] = [] def _batched_pixel_values(transform, image: Image.Image, batch: int) -> np.ndarray: """Repeat one preprocessed sample image into a ``(batch, C, H, W)`` float32 NCHW array. A plain latency benchmark, not a detection-diversity test — every fixed-batch format below times against N copies of the same image rather than N distinct images. """ tensor, _ = transform(image, None) return np.repeat(tensor[None].numpy(), batch, axis=0) def _batched_pixel_tensor(transform, image: Image.Image, batch: int) -> torch.Tensor: """Repeat one preprocessed sample image into a contiguous ``(batch, C, H, W)`` float32 tensor.""" tensor, _ = transform(image, None) return tensor[None].float().contiguous().repeat(batch, 1, 1, 1) def _batched_target_sizes(image: Image.Image, batch: int) -> torch.Tensor: """Repeat the sample image's ``(height, width)`` once per batch element, for ``post_process``.""" return torch.tensor([[image.height, image.width]] * batch) _enable_notebook_inline_matplotlib()

3. Sample image¶

A single street scene with several COCO classes (dog, person, backpack, car) is enough to verify detections. The image is downloaded once and reused for every format below.

In [ ]:

import urllib.request

IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")

import urllib.request IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg" IMAGE_PATH = EXPORT_DIR / "sample.jpg" if not IMAGE_PATH.exists(): urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH) image = Image.open(IMAGE_PATH).convert("RGB") print(f"Sample image: {image.size[0]}×{image.size[1]}")

4. PyTorch baseline — predict() on Metal (MPS)¶

No export needed — this is the reference every other format in this notebook is compared against. predict() includes preprocessing and postprocessing — there is no separate "forward-only" path through it, so only an end-to-end number is reported for the baseline. predict() also accepts a list of images and batches them internally, so the batch matrix below feeds it batch copies of the same sample image rather than exporting anything.

In [ ]:

from rfdetr import RFDETRSmall

with measure_memory() as mem:
    model = RFDETRSmall(device=PYTORCH_DEVICE)
    baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_baseline_memory_mb = mem.delta_mb
print(f"PyTorch baseline ({PYTORCH_DEVICE}): {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)

from rfdetr import RFDETRSmall with measure_memory() as mem: model = RFDETRSmall(device=PYTORCH_DEVICE) baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD) pytorch_baseline_memory_mb = mem.delta_mb print(f"PyTorch baseline ({PYTORCH_DEVICE}): {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}") visualize_detections(baseline_detections, image)

Memory [MB] for PyTorch is measured once above (constructing the runtime is a one-time cost, unlike the fixed-batch formats below which reload a fresh artifact per batch) and reused for every batch row in the summary table.

In [ ]:

for batch in BATCH_SIZES:
    images_batch = [image] * batch
    pytorch_baseline = measure_latency(
        partial(model.predict, images_batch),
        label=f"PyTorch predict() ({PYTORCH_DEVICE}) batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    print(
        f"  {pytorch_baseline.label:<40}  {pytorch_baseline.mean_ms:6.2f} ms ± {pytorch_baseline.std_ms:5.2f}"
        f"   ({batch * pytorch_baseline.fps:6.1f} FPS)"
    )
    results.append(
        _result_row(
            "PyTorch predict()", batch, f"{PYTORCH_DEVICE}, fp32", None, pytorch_baseline, pytorch_baseline_memory_mb
        )
    )

for batch in BATCH_SIZES: images_batch = [image] * batch pytorch_baseline = measure_latency( partial(model.predict, images_batch), label=f"PyTorch predict() ({PYTORCH_DEVICE}) batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) print( f" {pytorch_baseline.label:<40} {pytorch_baseline.mean_ms:6.2f} ms ± {pytorch_baseline.std_ms:5.2f}" f" ({batch * pytorch_baseline.fps:6.1f} FPS)" ) results.append( _result_row( "PyTorch predict()", batch, f"{PYTORCH_DEVICE}, fp32", None, pytorch_baseline, pytorch_baseline_memory_mb ) )

PyTorch inference(dtype=torch.float16) — fp16+JIT anchor¶

inference(dtype=torch.float16, batch_size=N) fuses layers with torch.jit.script, halves arithmetic precision, and compiles the scripted module for exactly batch_size images — calling predict() with a different batch afterward raises ValueError: Batch size mismatch (confirmed by an earlier run of this notebook). Like the export formats below, the batch matrix here needs one recompile per batch size; remove_optimized_model() reverts the in-place optimization afterward so model stays reusable for the export calls that follow.

In [ ]:

for batch in BATCH_SIZES:
    images_batch = [image] * batch
    with measure_memory() as mem:
        model.inference(dtype=torch.float16, batch_size=batch)
        _ = model.predict(images_batch)
    pytorch_fp16_jit_memory_mb = mem.delta_mb

    pytorch_fp16_jit = measure_latency(
        partial(model.predict, images_batch),
        label=f"PyTorch inference() fp16+JIT ({PYTORCH_DEVICE}) batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    print(
        f"  {pytorch_fp16_jit.label:<40}  {pytorch_fp16_jit.mean_ms:6.2f} ms ± {pytorch_fp16_jit.std_ms:5.2f}"
        f"   ({batch * pytorch_fp16_jit.fps:6.1f} FPS)   +{pytorch_fp16_jit_memory_mb:.1f} MB"
    )
    results.append(
        _result_row(
            "PyTorch inference()",
            batch,
            f"{PYTORCH_DEVICE}, fp16+JIT",
            None,
            pytorch_fp16_jit,
            pytorch_fp16_jit_memory_mb,
        )
    )
model.remove_optimized_model()

for batch in BATCH_SIZES: images_batch = [image] * batch with measure_memory() as mem: model.inference(dtype=torch.float16, batch_size=batch) _ = model.predict(images_batch) pytorch_fp16_jit_memory_mb = mem.delta_mb pytorch_fp16_jit = measure_latency( partial(model.predict, images_batch), label=f"PyTorch inference() fp16+JIT ({PYTORCH_DEVICE}) batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) print( f" {pytorch_fp16_jit.label:<40} {pytorch_fp16_jit.mean_ms:6.2f} ms ± {pytorch_fp16_jit.std_ms:5.2f}" f" ({batch * pytorch_fp16_jit.fps:6.1f} FPS) +{pytorch_fp16_jit_memory_mb:.1f} MB" ) results.append( _result_row( "PyTorch inference()", batch, f"{PYTORCH_DEVICE}, fp16+JIT", None, pytorch_fp16_jit, pytorch_fp16_jit_memory_mb, ) ) model.remove_optimized_model()

5. Native CoreML¶

What it is. Native CoreML export produces a .mlpackage you can drag directly into Xcode — no ONNX, no ExecuTorch runtime involved. Good for Apple-native (iOS/macOS) app development: it's the lowest-friction path into Apple's own ML stack, distinct from ExecuTorch's coreml backend below (a different export path that produces a .pte, not a .mlpackage). See the CoreML export docs.

Export + Benchmark¶

Export defaults to coreml_precision="float32" for tight CPU parity with eager PyTorch. CoreML bakes a fixed batch shape into the .mlpackage (dynamic_batch_reason in rfdetr.export.registry, confirmed against the exporter's own CoreMLExporter.dynamic_batch_reason) — the loop below exports one bundle per batch size, each into its own output directory so artifacts never collide, then benchmarks it. coremltools infers its own input/output names for the .mlpackage spec — they are not renamed to input / dets / labels. Read the input name and output order from mlmodel.get_spec().description instead of hardcoding them, then match outputs by position (dets, labels for detection). The correctness check and visualization only run once, at batch 1.

In [ ]:

import coremltools as ct

from rfdetr.export.benchmark import infer_transforms, post_process

resolution = model.model_config.resolution
apple_transform = infer_transforms((resolution, resolution))


def _coreml_forward_call(mlmodel, input_name: str, pixel_values: np.ndarray) -> dict[str, np.ndarray]:
    return mlmodel.predict({input_name: pixel_values})


def _coreml_end2end_call(mlmodel, input_name, output_names, transform, image, batch, target_sizes) -> None:
    pixel_values = _batched_pixel_values(transform, image, batch)
    prediction = mlmodel.predict({input_name: pixel_values})
    dets, labels = (torch.from_numpy(np.asarray(prediction[name], dtype=np.float32)) for name in output_names)
    post_process({"dets": dets, "labels": labels}, target_sizes)


for batch in BATCH_SIZES:
    coreml_path = model.export(format="coreml", batch_size=batch, output_dir=str(EXPORT_DIR / f"coreml_b{batch}"))
    coreml_size_mb = _artifact_size_mb(coreml_path)
    print(f"CoreML package (batch={batch}): {coreml_path}  ({coreml_size_mb:.1f} MB)")

    coreml_pixel_values = _batched_pixel_values(apple_transform, image, batch)
    target_sizes = _batched_target_sizes(image, batch)

    with measure_memory() as mem:
        coreml_model = ct.models.MLModel(str(coreml_path))
        coreml_spec = coreml_model.get_spec()
        coreml_input_name = coreml_spec.description.input[0].name
        coreml_output_names = [o.name for o in coreml_spec.description.output][:2]
        coreml_prediction = _coreml_forward_call(coreml_model, coreml_input_name, coreml_pixel_values)
    coreml_memory_mb = mem.delta_mb

    coreml_dets, coreml_labels = (
        torch.from_numpy(np.asarray(coreml_prediction[name], dtype=np.float32)) for name in coreml_output_names
    )
    coreml_result = post_process({"dets": coreml_dets, "labels": coreml_labels}, target_sizes)[0]
    coreml_keep = coreml_result["scores"] > CONFIDENCE_THRESHOLD
    if batch == 1:
        print(f"CoreML: {int(coreml_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")
        coreml_sv_detections = sv.Detections(
            xyxy=coreml_result["boxes"][coreml_keep].numpy(),
            confidence=coreml_result["scores"][coreml_keep].numpy(),
            class_id=coreml_result["labels"][coreml_keep].numpy().astype(int),
        )
        visualize_detections(coreml_sv_detections, image, EXPORT_DIR / "annotated_coreml.jpg")

    coreml_forward = measure_latency(
        partial(_coreml_forward_call, coreml_model, coreml_input_name, coreml_pixel_values),
        label=f"CoreML forward batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    coreml_end2end = measure_latency(
        partial(
            _coreml_end2end_call,
            coreml_model,
            coreml_input_name,
            coreml_output_names,
            apple_transform,
            image,
            batch,
            target_sizes,
        ),
        label=f"CoreML end2end batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    for r in (coreml_forward, coreml_end2end):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({batch * r.fps:6.1f} FPS)")
    results.append(_result_row("CoreML", batch, "fp32 (default)", coreml_forward, coreml_end2end, coreml_memory_mb))

import coremltools as ct from rfdetr.export.benchmark import infer_transforms, post_process resolution = model.model_config.resolution apple_transform = infer_transforms((resolution, resolution)) def _coreml_forward_call(mlmodel, input_name: str, pixel_values: np.ndarray) -> dict[str, np.ndarray]: return mlmodel.predict({input_name: pixel_values}) def _coreml_end2end_call(mlmodel, input_name, output_names, transform, image, batch, target_sizes) -> None: pixel_values = _batched_pixel_values(transform, image, batch) prediction = mlmodel.predict({input_name: pixel_values}) dets, labels = (torch.from_numpy(np.asarray(prediction[name], dtype=np.float32)) for name in output_names) post_process({"dets": dets, "labels": labels}, target_sizes) for batch in BATCH_SIZES: coreml_path = model.export(format="coreml", batch_size=batch, output_dir=str(EXPORT_DIR / f"coreml_b{batch}")) coreml_size_mb = _artifact_size_mb(coreml_path) print(f"CoreML package (batch={batch}): {coreml_path} ({coreml_size_mb:.1f} MB)") coreml_pixel_values = _batched_pixel_values(apple_transform, image, batch) target_sizes = _batched_target_sizes(image, batch) with measure_memory() as mem: coreml_model = ct.models.MLModel(str(coreml_path)) coreml_spec = coreml_model.get_spec() coreml_input_name = coreml_spec.description.input[0].name coreml_output_names = [o.name for o in coreml_spec.description.output][:2] coreml_prediction = _coreml_forward_call(coreml_model, coreml_input_name, coreml_pixel_values) coreml_memory_mb = mem.delta_mb coreml_dets, coreml_labels = ( torch.from_numpy(np.asarray(coreml_prediction[name], dtype=np.float32)) for name in coreml_output_names ) coreml_result = post_process({"dets": coreml_dets, "labels": coreml_labels}, target_sizes)[0] coreml_keep = coreml_result["scores"] > CONFIDENCE_THRESHOLD if batch == 1: print(f"CoreML: {int(coreml_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreml_sv_detections = sv.Detections( xyxy=coreml_result["boxes"][coreml_keep].numpy(), confidence=coreml_result["scores"][coreml_keep].numpy(), class_id=coreml_result["labels"][coreml_keep].numpy().astype(int), ) visualize_detections(coreml_sv_detections, image, EXPORT_DIR / "annotated_coreml.jpg") coreml_forward = measure_latency( partial(_coreml_forward_call, coreml_model, coreml_input_name, coreml_pixel_values), label=f"CoreML forward batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) coreml_end2end = measure_latency( partial( _coreml_end2end_call, coreml_model, coreml_input_name, coreml_output_names, apple_transform, image, batch, target_sizes, ), label=f"CoreML end2end batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) for r in (coreml_forward, coreml_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)") results.append(_result_row("CoreML", batch, "fp32 (default)", coreml_forward, coreml_end2end, coreml_memory_mb))

CoreML (fp16)¶

coreml_precision="float16" halves the weight precision, producing a smaller, ANE-oriented bundle. The tradeoff is larger numeric drift: fp16 narrows the fp32 safety margin around near-tied torch.topk query-ranking decisions, so a rank swap (and therefore a slightly different detection set) becomes more likely than in the fp32 row above.

In [ ]:

for batch in BATCH_SIZES:
    coreml_fp16_path = model.export(
        format="coreml",
        coreml_precision="float16",
        batch_size=batch,
        output_dir=str(EXPORT_DIR / f"coreml_fp16_b{batch}"),
    )
    coreml_fp16_size_mb = _artifact_size_mb(coreml_fp16_path)
    print(f"CoreML package (fp16, batch={batch}): {coreml_fp16_path}  ({coreml_fp16_size_mb:.1f} MB)")

    coreml_fp16_pixel_values = _batched_pixel_values(apple_transform, image, batch)
    target_sizes = _batched_target_sizes(image, batch)

    with measure_memory() as mem:
        coreml_fp16_model = ct.models.MLModel(str(coreml_fp16_path))
        coreml_fp16_spec = coreml_fp16_model.get_spec()
        coreml_fp16_input_name = coreml_fp16_spec.description.input[0].name
        coreml_fp16_output_names = [o.name for o in coreml_fp16_spec.description.output][:2]
        coreml_fp16_prediction = _coreml_forward_call(
            coreml_fp16_model, coreml_fp16_input_name, coreml_fp16_pixel_values
        )
    coreml_fp16_memory_mb = mem.delta_mb

    coreml_fp16_dets, coreml_fp16_labels = (
        torch.from_numpy(np.asarray(coreml_fp16_prediction[name], dtype=np.float32))
        for name in coreml_fp16_output_names
    )
    coreml_fp16_result = post_process({"dets": coreml_fp16_dets, "labels": coreml_fp16_labels}, target_sizes)[0]
    coreml_fp16_keep = coreml_fp16_result["scores"] > CONFIDENCE_THRESHOLD
    if batch == 1:
        print(f"CoreML fp16: {int(coreml_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")
        coreml_fp16_sv_detections = sv.Detections(
            xyxy=coreml_fp16_result["boxes"][coreml_fp16_keep].numpy(),
            confidence=coreml_fp16_result["scores"][coreml_fp16_keep].numpy(),
            class_id=coreml_fp16_result["labels"][coreml_fp16_keep].numpy().astype(int),
        )
        visualize_detections(coreml_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreml_fp16.jpg")

    coreml_fp16_forward = measure_latency(
        partial(_coreml_forward_call, coreml_fp16_model, coreml_fp16_input_name, coreml_fp16_pixel_values),
        label=f"CoreML fp16 forward batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    coreml_fp16_end2end = measure_latency(
        partial(
            _coreml_end2end_call,
            coreml_fp16_model,
            coreml_fp16_input_name,
            coreml_fp16_output_names,
            apple_transform,
            image,
            batch,
            target_sizes,
        ),
        label=f"CoreML fp16 end2end batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    for r in (coreml_fp16_forward, coreml_fp16_end2end):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({batch * r.fps:6.1f} FPS)")
    results.append(
        _result_row("CoreML", batch, "fp16", coreml_fp16_forward, coreml_fp16_end2end, coreml_fp16_memory_mb)
    )

for batch in BATCH_SIZES: coreml_fp16_path = model.export( format="coreml", coreml_precision="float16", batch_size=batch, output_dir=str(EXPORT_DIR / f"coreml_fp16_b{batch}"), ) coreml_fp16_size_mb = _artifact_size_mb(coreml_fp16_path) print(f"CoreML package (fp16, batch={batch}): {coreml_fp16_path} ({coreml_fp16_size_mb:.1f} MB)") coreml_fp16_pixel_values = _batched_pixel_values(apple_transform, image, batch) target_sizes = _batched_target_sizes(image, batch) with measure_memory() as mem: coreml_fp16_model = ct.models.MLModel(str(coreml_fp16_path)) coreml_fp16_spec = coreml_fp16_model.get_spec() coreml_fp16_input_name = coreml_fp16_spec.description.input[0].name coreml_fp16_output_names = [o.name for o in coreml_fp16_spec.description.output][:2] coreml_fp16_prediction = _coreml_forward_call( coreml_fp16_model, coreml_fp16_input_name, coreml_fp16_pixel_values ) coreml_fp16_memory_mb = mem.delta_mb coreml_fp16_dets, coreml_fp16_labels = ( torch.from_numpy(np.asarray(coreml_fp16_prediction[name], dtype=np.float32)) for name in coreml_fp16_output_names ) coreml_fp16_result = post_process({"dets": coreml_fp16_dets, "labels": coreml_fp16_labels}, target_sizes)[0] coreml_fp16_keep = coreml_fp16_result["scores"] > CONFIDENCE_THRESHOLD if batch == 1: print(f"CoreML fp16: {int(coreml_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreml_fp16_sv_detections = sv.Detections( xyxy=coreml_fp16_result["boxes"][coreml_fp16_keep].numpy(), confidence=coreml_fp16_result["scores"][coreml_fp16_keep].numpy(), class_id=coreml_fp16_result["labels"][coreml_fp16_keep].numpy().astype(int), ) visualize_detections(coreml_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreml_fp16.jpg") coreml_fp16_forward = measure_latency( partial(_coreml_forward_call, coreml_fp16_model, coreml_fp16_input_name, coreml_fp16_pixel_values), label=f"CoreML fp16 forward batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) coreml_fp16_end2end = measure_latency( partial( _coreml_end2end_call, coreml_fp16_model, coreml_fp16_input_name, coreml_fp16_output_names, apple_transform, image, batch, target_sizes, ), label=f"CoreML fp16 end2end batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) for r in (coreml_fp16_forward, coreml_fp16_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)") results.append( _result_row("CoreML", batch, "fp16", coreml_fp16_forward, coreml_fp16_end2end, coreml_fp16_memory_mb) )

6. Core AI¶

What it is. Core AI is Apple's newer on-device inference framework (iOS, iPadOS, macOS 27+). RF-DETR traces the model with torch.export and converts it to an .aimodel asset — no ONNX step. Good for apps targeting the newest Apple OS versions: the runtime decides at load time whether the CPU, GPU, or Neural Engine executes it, rather than that choice being baked in at export time like CoreML. See the Core AI export docs.

Export + Benchmark¶

Unlike CoreML, Core AI keeps the exported tensor names — input, dets, labels. Its Python runtime is partly async: AIModel.load awaits, but the InferenceFunction load_function returns is a plain synchronous call — only calling that function (coreai_fn(...)) is a coroutine. The notebook awaits that call directly in its correctness and benchmark cells, which keeps inference compatible with Jupyter's running event loop. The .aimodel also bakes a fixed batch shape into the asset — the loop below exports one per batch size.

In [ ]:

import time
from collections.abc import Awaitable, Callable
from typing import Any


async def _measure_async_latency(
    fn: Callable[[], Awaitable[object]], *, label: str, warmup: int, runs: int
) -> BenchmarkResult:
    """Measure an awaitable call after warmup using synchronized wall-clock samples.

    Args:
        fn: Zero-argument callable returning the inference awaitable.
        label: Name shown in the benchmark results.
        warmup: Untimed calls before measurement begins.
        runs: Timed calls used to calculate mean and standard deviation.

    Returns:
        The measured latency statistics for the awaitable call.
    """
    for _ in range(warmup):
        await fn()
    timings: list[float] = []
    for _ in range(runs):
        start = time.perf_counter()
        await fn()
        timings.append((time.perf_counter() - start) * 1000.0)
    return BenchmarkResult(label, float(np.mean(timings)), float(np.std(timings)))


import coreai.runtime as coreai_rt


async def _load_coreai(path: Path) -> tuple[Any, Any]:
    loaded = await coreai_rt.AIModel.load(str(path), coreai_rt.SpecializationOptions.default())
    fn = loaded.load_function("main")  # synchronous — only AIModel.load and InferenceFunction.__call__ are async
    return loaded, fn


async def _coreai_forward_call(fn, pixel_values: np.ndarray, out_dtype: type = np.float32) -> dict[str, np.ndarray]:
    outputs = await fn({"input": coreai_rt.NDArray(pixel_values)})
    return {name: outputs[name].numpy().astype(out_dtype) for name in ("dets", "labels")}


async def _coreai_end2end_call(fn, transform, image, batch, target_sizes, in_dtype: type | None) -> None:
    pixel_values = _batched_pixel_values(transform, image, batch)
    if in_dtype is not None:
        pixel_values = pixel_values.astype(in_dtype)
    outputs = await _coreai_forward_call(fn, pixel_values)
    dets = torch.from_numpy(outputs["dets"])
    labels = torch.from_numpy(outputs["labels"])
    post_process({"dets": dets, "labels": labels}, target_sizes)


for batch in BATCH_SIZES:
    coreai_path = model.export(format="coreai", batch_size=batch, output_dir=str(EXPORT_DIR / f"coreai_b{batch}"))
    coreai_size_mb = _artifact_size_mb(coreai_path)
    print(f"Core AI asset (batch={batch}): {coreai_path}  ({coreai_size_mb:.1f} MB)")

    coreai_pixel_values = _batched_pixel_values(apple_transform, image, batch)
    target_sizes = _batched_target_sizes(image, batch)

    with measure_memory() as mem:
        coreai_model, coreai_fn = await _load_coreai(coreai_path)
        coreai_outputs = await _coreai_forward_call(coreai_fn, coreai_pixel_values)
    coreai_memory_mb = mem.delta_mb

    coreai_dets = torch.from_numpy(coreai_outputs["dets"])
    coreai_labels = torch.from_numpy(coreai_outputs["labels"])
    coreai_result = post_process({"dets": coreai_dets, "labels": coreai_labels}, target_sizes)[0]
    coreai_keep = coreai_result["scores"] > CONFIDENCE_THRESHOLD
    if batch == 1:
        print(f"Core AI: {int(coreai_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")
        coreai_sv_detections = sv.Detections(
            xyxy=coreai_result["boxes"][coreai_keep].numpy(),
            confidence=coreai_result["scores"][coreai_keep].numpy(),
            class_id=coreai_result["labels"][coreai_keep].numpy().astype(int),
        )
        visualize_detections(coreai_sv_detections, image, EXPORT_DIR / "annotated_coreai.jpg")

    coreai_forward = await _measure_async_latency(
        partial(_coreai_forward_call, coreai_fn, coreai_pixel_values),
        label=f"Core AI forward batch={batch}",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    coreai_end2end = await _measure_async_latency(
        partial(_coreai_end2end_call, coreai_fn, apple_transform, image, batch, target_sizes, None),
        label=f"Core AI end2end batch={batch}",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    for r in (coreai_forward, coreai_end2end):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({batch * r.fps:6.1f} FPS)")
    results.append(_result_row("Core AI", batch, "fp32 (default)", coreai_forward, coreai_end2end, coreai_memory_mb))

import time from collections.abc import Awaitable, Callable from typing import Any async def _measure_async_latency( fn: Callable[[], Awaitable[object]], *, label: str, warmup: int, runs: int ) -> BenchmarkResult: """Measure an awaitable call after warmup using synchronized wall-clock samples. Args: fn: Zero-argument callable returning the inference awaitable. label: Name shown in the benchmark results. warmup: Untimed calls before measurement begins. runs: Timed calls used to calculate mean and standard deviation. Returns: The measured latency statistics for the awaitable call. """ for _ in range(warmup): await fn() timings: list[float] = [] for _ in range(runs): start = time.perf_counter() await fn() timings.append((time.perf_counter() - start) * 1000.0) return BenchmarkResult(label, float(np.mean(timings)), float(np.std(timings))) import coreai.runtime as coreai_rt async def _load_coreai(path: Path) -> tuple[Any, Any]: loaded = await coreai_rt.AIModel.load(str(path), coreai_rt.SpecializationOptions.default()) fn = loaded.load_function("main") # synchronous — only AIModel.load and InferenceFunction.__call__ are async return loaded, fn async def _coreai_forward_call(fn, pixel_values: np.ndarray, out_dtype: type = np.float32) -> dict[str, np.ndarray]: outputs = await fn({"input": coreai_rt.NDArray(pixel_values)}) return {name: outputs[name].numpy().astype(out_dtype) for name in ("dets", "labels")} async def _coreai_end2end_call(fn, transform, image, batch, target_sizes, in_dtype: type | None) -> None: pixel_values = _batched_pixel_values(transform, image, batch) if in_dtype is not None: pixel_values = pixel_values.astype(in_dtype) outputs = await _coreai_forward_call(fn, pixel_values) dets = torch.from_numpy(outputs["dets"]) labels = torch.from_numpy(outputs["labels"]) post_process({"dets": dets, "labels": labels}, target_sizes) for batch in BATCH_SIZES: coreai_path = model.export(format="coreai", batch_size=batch, output_dir=str(EXPORT_DIR / f"coreai_b{batch}")) coreai_size_mb = _artifact_size_mb(coreai_path) print(f"Core AI asset (batch={batch}): {coreai_path} ({coreai_size_mb:.1f} MB)") coreai_pixel_values = _batched_pixel_values(apple_transform, image, batch) target_sizes = _batched_target_sizes(image, batch) with measure_memory() as mem: coreai_model, coreai_fn = await _load_coreai(coreai_path) coreai_outputs = await _coreai_forward_call(coreai_fn, coreai_pixel_values) coreai_memory_mb = mem.delta_mb coreai_dets = torch.from_numpy(coreai_outputs["dets"]) coreai_labels = torch.from_numpy(coreai_outputs["labels"]) coreai_result = post_process({"dets": coreai_dets, "labels": coreai_labels}, target_sizes)[0] coreai_keep = coreai_result["scores"] > CONFIDENCE_THRESHOLD if batch == 1: print(f"Core AI: {int(coreai_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreai_sv_detections = sv.Detections( xyxy=coreai_result["boxes"][coreai_keep].numpy(), confidence=coreai_result["scores"][coreai_keep].numpy(), class_id=coreai_result["labels"][coreai_keep].numpy().astype(int), ) visualize_detections(coreai_sv_detections, image, EXPORT_DIR / "annotated_coreai.jpg") coreai_forward = await _measure_async_latency( partial(_coreai_forward_call, coreai_fn, coreai_pixel_values), label=f"Core AI forward batch={batch}", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) coreai_end2end = await _measure_async_latency( partial(_coreai_end2end_call, coreai_fn, apple_transform, image, batch, target_sizes, None), label=f"Core AI end2end batch={batch}", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) for r in (coreai_forward, coreai_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)") results.append(_result_row("Core AI", batch, "fp32 (default)", coreai_forward, coreai_end2end, coreai_memory_mb))

Core AI (fp16)¶

coreai_precision="float16" traces and stores the whole graph in fp16, so the asset also takes and returns float16 tensors — the input is cast below, and the outputs are cast back to fp32 before postprocessing. One part of the graph stays fp32 regardless: RF-DETR forces topk back to float32 even in an fp16 export, because the Neural Engine's fp16 topk returns corrupt indices and the two-stage query selection would gather the wrong encoder tokens. That caps how much of the graph actually runs in fp16.

In [ ]:

for batch in BATCH_SIZES:
    coreai_fp16_path = model.export(
        format="coreai",
        coreai_precision="float16",
        batch_size=batch,
        output_dir=str(EXPORT_DIR / f"coreai_fp16_b{batch}"),
    )
    coreai_fp16_size_mb = _artifact_size_mb(coreai_fp16_path)
    print(f"Core AI asset (fp16, batch={batch}): {coreai_fp16_path}  ({coreai_fp16_size_mb:.1f} MB)")

    coreai_fp16_pixel_values = _batched_pixel_values(apple_transform, image, batch).astype(np.float16)
    target_sizes = _batched_target_sizes(image, batch)

    with measure_memory() as mem:
        coreai_fp16_model, coreai_fp16_fn = await _load_coreai(coreai_fp16_path)
        coreai_fp16_outputs = await _coreai_forward_call(coreai_fp16_fn, coreai_fp16_pixel_values)
    coreai_fp16_memory_mb = mem.delta_mb

    coreai_fp16_result = post_process(
        {
            "dets": torch.from_numpy(coreai_fp16_outputs["dets"]),
            "labels": torch.from_numpy(coreai_fp16_outputs["labels"]),
        },
        target_sizes,
    )[0]
    coreai_fp16_keep = coreai_fp16_result["scores"] > CONFIDENCE_THRESHOLD
    if batch == 1:
        print(f"Core AI fp16: {int(coreai_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")
        coreai_fp16_sv_detections = sv.Detections(
            xyxy=coreai_fp16_result["boxes"][coreai_fp16_keep].numpy(),
            confidence=coreai_fp16_result["scores"][coreai_fp16_keep].numpy(),
            class_id=coreai_fp16_result["labels"][coreai_fp16_keep].numpy().astype(int),
        )
        visualize_detections(coreai_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreai_fp16.jpg")

    coreai_fp16_forward = await _measure_async_latency(
        partial(_coreai_forward_call, coreai_fp16_fn, coreai_fp16_pixel_values),
        label=f"Core AI fp16 forward batch={batch}",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    coreai_fp16_end2end = await _measure_async_latency(
        partial(_coreai_end2end_call, coreai_fp16_fn, apple_transform, image, batch, target_sizes, np.float16),
        label=f"Core AI fp16 end2end batch={batch}",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    for r in (coreai_fp16_forward, coreai_fp16_end2end):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({batch * r.fps:6.1f} FPS)")
    results.append(
        _result_row("Core AI", batch, "fp16", coreai_fp16_forward, coreai_fp16_end2end, coreai_fp16_memory_mb)
    )

for batch in BATCH_SIZES: coreai_fp16_path = model.export( format="coreai", coreai_precision="float16", batch_size=batch, output_dir=str(EXPORT_DIR / f"coreai_fp16_b{batch}"), ) coreai_fp16_size_mb = _artifact_size_mb(coreai_fp16_path) print(f"Core AI asset (fp16, batch={batch}): {coreai_fp16_path} ({coreai_fp16_size_mb:.1f} MB)") coreai_fp16_pixel_values = _batched_pixel_values(apple_transform, image, batch).astype(np.float16) target_sizes = _batched_target_sizes(image, batch) with measure_memory() as mem: coreai_fp16_model, coreai_fp16_fn = await _load_coreai(coreai_fp16_path) coreai_fp16_outputs = await _coreai_forward_call(coreai_fp16_fn, coreai_fp16_pixel_values) coreai_fp16_memory_mb = mem.delta_mb coreai_fp16_result = post_process( { "dets": torch.from_numpy(coreai_fp16_outputs["dets"]), "labels": torch.from_numpy(coreai_fp16_outputs["labels"]), }, target_sizes, )[0] coreai_fp16_keep = coreai_fp16_result["scores"] > CONFIDENCE_THRESHOLD if batch == 1: print(f"Core AI fp16: {int(coreai_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreai_fp16_sv_detections = sv.Detections( xyxy=coreai_fp16_result["boxes"][coreai_fp16_keep].numpy(), confidence=coreai_fp16_result["scores"][coreai_fp16_keep].numpy(), class_id=coreai_fp16_result["labels"][coreai_fp16_keep].numpy().astype(int), ) visualize_detections(coreai_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreai_fp16.jpg") coreai_fp16_forward = await _measure_async_latency( partial(_coreai_forward_call, coreai_fp16_fn, coreai_fp16_pixel_values), label=f"Core AI fp16 forward batch={batch}", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) coreai_fp16_end2end = await _measure_async_latency( partial(_coreai_end2end_call, coreai_fp16_fn, apple_transform, image, batch, target_sizes, np.float16), label=f"Core AI fp16 end2end batch={batch}", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) for r in (coreai_fp16_forward, coreai_fp16_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)") results.append( _result_row("Core AI", batch, "fp16", coreai_fp16_forward, coreai_fp16_end2end, coreai_fp16_memory_mb) )

7. ExecuTorch (CoreML backend)¶

What it is. This is the same ExecuTorch runtime as the mobile cookbook's XNNPACK section, but lowered to Apple's coreml backend instead — still a portable .pte file, no .mlpackage, but running on the Apple Neural Engine. Good for staying inside the ExecuTorch/PyTorch deployment story while still reaching Apple's Neural Engine, when native CoreML's separate export/runtime isn't a fit; the tradeoff is fp16 only, unlike native CoreML and Core AI's fp32 default. See the ExecuTorch export docs.

Export + Benchmark¶

backend="coreml" targets the Apple Neural Engine, iPhone/iPad/Mac, and runs fp16 — unlike native CoreML and Core AI above, which default to fp32. This is the ExecuTorch delegate, not native CoreML — it produces a .pte for the ExecuTorch runtime, distinct from format="coreml". ExecuTorch also bakes a fixed batch shape into the .pte (ExecuTorchExporter.dynamic_batch_reason) — one export per batch size below.

The input tensor must be contiguous — see the mobile cookbook for why; _batched_pixel_tensor already returns one via torchvision's transform plus an explicit .contiguous().

The export-mode forward mutates its input in place, so the correctness call below is fed a .clone() — that keeps executorch_pixel_values pristine for the timed loop that follows, which would otherwise start from a buffer the correctness call had already overwritten. The timed loop then reuses that one buffer deliberately: its outputs are discarded, so the progressive mutation cannot move a latency number, and cloning inside the timed region would charge this row a per-call copy no other format in the Results table pays. This differs from the mobile cookbook's XNNPACK build, where feeding the same tensor to execute() twice raises RuntimeError: tensor does not have a device and the benchmark thunks therefore must clone; the CoreML-backend .pte here took 35 repeated uncloned execute() calls per row in the run that produced the Results table below.

load_executorch_method (from rfdetr.export._executorch.inference, shared with the mobile cookbook's XNNPACK section) checks that ExecuTorch's runtime actually loads against the installed torch before attempting Method.execute — a torch/executorch ABI mismatch then raises a clear ImportError here instead of the opaque RuntimeError: tensor does not have a device a raw Runtime.get().load_program(...) call would surface only on first execution.

In [ ]:

from rfdetr.export._executorch.inference import load_executorch_method


def _executorch_forward_call(method, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
    return method.execute([pixel_values])


def _executorch_end2end_call(method, transform, image, batch, target_sizes) -> None:
    pixel_values = _batched_pixel_tensor(transform, image, batch)
    dets, labels = method.execute([pixel_values])
    post_process({"dets": dets, "labels": labels}, target_sizes)


for batch in BATCH_SIZES:
    pte_path = model.export(
        format="executorch", backend="coreml", batch_size=batch, output_dir=str(EXPORT_DIR / f"executorch_b{batch}")
    )
    executorch_size_mb = _artifact_size_mb(pte_path)
    print(f"ExecuTorch (CoreML) program (batch={batch}): {pte_path}  ({executorch_size_mb:.1f} MB)")

    executorch_pixel_values = _batched_pixel_tensor(apple_transform, image, batch)
    target_sizes = _batched_target_sizes(image, batch)

    with measure_memory() as mem:
        executorch_method = load_executorch_method(pte_path)
        # Clone only here: these outputs are postprocessed and visualized, and feeding the original
        # would leave the timed loop below starting from an already-mutated buffer.
        executorch_dets, executorch_labels = _executorch_forward_call(
            executorch_method, executorch_pixel_values.clone()
        )
    executorch_memory_mb = mem.delta_mb

    executorch_result = post_process({"dets": executorch_dets, "labels": executorch_labels}, target_sizes)[0]
    executorch_keep = executorch_result["scores"] > CONFIDENCE_THRESHOLD
    if batch == 1:
        print(f"ExecuTorch (CoreML): {int(executorch_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")
        executorch_sv_detections = sv.Detections(
            xyxy=executorch_result["boxes"][executorch_keep].numpy(),
            confidence=executorch_result["scores"][executorch_keep].numpy(),
            class_id=executorch_result["labels"][executorch_keep].numpy().astype(int),
        )
        visualize_detections(executorch_sv_detections, image, EXPORT_DIR / "annotated_executorch.jpg")

    # Intentionally uncloned: the forward mutates this buffer in place every call and the outputs are
    # discarded, so only the per-call latency survives — see the clone note above.
    executorch_forward = measure_latency(
        partial(_executorch_forward_call, executorch_method, executorch_pixel_values),
        label=f"ExecuTorch (CoreML) forward batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    executorch_end2end = measure_latency(
        partial(_executorch_end2end_call, executorch_method, apple_transform, image, batch, target_sizes),
        label=f"ExecuTorch (CoreML) end2end batch={batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    for r in (executorch_forward, executorch_end2end):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({batch * r.fps:6.1f} FPS)")
    results.append(
        _result_row(
            "ExecuTorch", batch, "CoreML backend, fp16", executorch_forward, executorch_end2end, executorch_memory_mb
        )
    )

from rfdetr.export._executorch.inference import load_executorch_method def _executorch_forward_call(method, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: return method.execute([pixel_values]) def _executorch_end2end_call(method, transform, image, batch, target_sizes) -> None: pixel_values = _batched_pixel_tensor(transform, image, batch) dets, labels = method.execute([pixel_values]) post_process({"dets": dets, "labels": labels}, target_sizes) for batch in BATCH_SIZES: pte_path = model.export( format="executorch", backend="coreml", batch_size=batch, output_dir=str(EXPORT_DIR / f"executorch_b{batch}") ) executorch_size_mb = _artifact_size_mb(pte_path) print(f"ExecuTorch (CoreML) program (batch={batch}): {pte_path} ({executorch_size_mb:.1f} MB)") executorch_pixel_values = _batched_pixel_tensor(apple_transform, image, batch) target_sizes = _batched_target_sizes(image, batch) with measure_memory() as mem: executorch_method = load_executorch_method(pte_path) # Clone only here: these outputs are postprocessed and visualized, and feeding the original # would leave the timed loop below starting from an already-mutated buffer. executorch_dets, executorch_labels = _executorch_forward_call( executorch_method, executorch_pixel_values.clone() ) executorch_memory_mb = mem.delta_mb executorch_result = post_process({"dets": executorch_dets, "labels": executorch_labels}, target_sizes)[0] executorch_keep = executorch_result["scores"] > CONFIDENCE_THRESHOLD if batch == 1: print(f"ExecuTorch (CoreML): {int(executorch_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") executorch_sv_detections = sv.Detections( xyxy=executorch_result["boxes"][executorch_keep].numpy(), confidence=executorch_result["scores"][executorch_keep].numpy(), class_id=executorch_result["labels"][executorch_keep].numpy().astype(int), ) visualize_detections(executorch_sv_detections, image, EXPORT_DIR / "annotated_executorch.jpg") # Intentionally uncloned: the forward mutates this buffer in place every call and the outputs are # discarded, so only the per-call latency survives — see the clone note above. executorch_forward = measure_latency( partial(_executorch_forward_call, executorch_method, executorch_pixel_values), label=f"ExecuTorch (CoreML) forward batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) executorch_end2end = measure_latency( partial(_executorch_end2end_call, executorch_method, apple_transform, image, batch, target_sizes), label=f"ExecuTorch (CoreML) end2end batch={batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) for r in (executorch_forward, executorch_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({batch * r.fps:6.1f} FPS)") results.append( _result_row( "ExecuTorch", batch, "CoreML backend, fp16", executorch_forward, executorch_end2end, executorch_memory_mb ) )

8. Results¶

Measured on an Apple M4 Max (arm64), macOS 27.0, rfdetr <VERSION>, batches 1/4/8, 5 warmup + 30 timed runs per (format, config, batch), RFDETRSmall. Numbers vary by Apple Silicon generation — rerun the cells above to get yours. Config is the precision/backend used for that row; — marks a scope this format doesn't have (PyTorch's predict() has no forward-only path). FPS is images/second at that row's batch size (batch × (1000 / end2end_ms)), not a per-call rate. Memory [MB] is host resident-memory growth across constructing the runtime plus its first inference call; for predict() this is measured once (the eager model is not rebuilt per batch) and repeated across that config's three rows, but inference(dtype=torch.float16, batch_size=N) recompiles per batch (see below), so it is measured per row like every fixed-batch export format.

Format	Config	Batch	forward [ms]	end2end [ms]	FPS [img/s] (end2end)	Memory [MB]
PyTorch predict()	mps, fp32	1	—	24.12 ± 0.49	41.5	603.8
PyTorch predict()	mps, fp32	4	—	52.37 ± 1.49	76.4	603.8
PyTorch predict()	mps, fp32	8	—	91.93 ± 1.47	87.0	603.8
PyTorch inference()	mps, fp16+JIT	1	—	18.65 ± 0.33	53.6	202.9
PyTorch inference()	mps, fp16+JIT	4	—	41.71 ± 0.59	95.9	167.8
PyTorch inference()	mps, fp16+JIT	8	—	74.12 ± 1.30	107.9	-139.6
CoreML	fp32 (default)	1	9.39 ± 0.28	11.57 ± 0.28	86.5	173.6
CoreML	fp32 (default)	4	29.60 ± 2.36	31.46 ± 0.49	127.1	-171.7
CoreML	fp32 (default)	8	56.96 ± 1.83	61.61 ± 0.81	129.9	83.6
CoreML	fp16	1	17.75 ± 0.20	19.86 ± 0.22	50.3	3.6
CoreML	fp16	4	83.08 ± 2.77	89.92 ± 2.92	44.5	30.7
CoreML	fp16	8	135.16 ± 13.03	131.38 ± 3.23	60.9	22.0
Core AI	fp32 (default)	1	12.99 ± 1.74	14.62 ± 0.37	68.4	147.6
Core AI	fp32 (default)	4	42.22 ± 6.05	49.98 ± 2.46	80.0	140.7
Core AI	fp32 (default)	8	87.57 ± 6.22	113.48 ± 12.76	70.5	141.5
Core AI	fp16	1	12.35 ± 4.99	13.53 ± 0.21	73.9	88.2
Core AI	fp16	4	35.81 ± 6.58	39.76 ± 1.30	100.6	79.7
Core AI	fp16	8	73.08 ± 4.88	86.93 ± 2.36	92.0	79.7
ExecuTorch	CoreML backend, fp16	1	21.31 ± 0.87	24.15 ± 0.82	41.4	37.1
ExecuTorch	CoreML backend, fp16	4	78.53 ± 3.16	80.86 ± 3.71	49.5	22.9
ExecuTorch	CoreML backend, fp16	8	71.14 ± 2.77	80.05 ± 13.27	99.9	10.2

fp16 is not automatically faster, and does not scale the same way across batch. At batch 1, CoreML's fp16 export came out slower than its fp32 default (19.86 vs 11.57 ms end2end), while Core AI's fp16 came out faster (13.53 vs 14.62 ms) — RF-DETR's own ExecuTorch CoreML-backend code records the same surprise from the other direction, forcing explicit fp16 compile-specs measured worse than letting the partitioner choose. The gap widens with batch size for CoreML specifically: its fp16 end2end time roughly triples from batch 1 to batch 4 (19.86 → 89.92 ms) while fp32 merely scales with the extra images (11.57 → 31.46 ms), so CoreML fp16's FPS advantage never recovers through batch 8. Core AI's two precisions track each other much more closely across the matrix. Treat fp16 on Apple Silicon as something to measure per runtime, per chip generation, and per batch size, not a free win.

Memory [MB] is an approximation on this page. It is host RSS growth across each row's own construct + first-inference bracket, in one shared process — so when several Apple accelerator runtimes are alive at once, the OS reclaiming an earlier row's Neural Engine buffers can land inside a later row's bracket. That shows up above as negative deltas for PyTorch fp16+JIT batch 8 (-139.6 MB) and CoreML fp32 batch 4 (-171.7 MB); an earlier run of this notebook (without the batch matrix, so with different section ordering) reproducibly measured Core AI at -99.5 MB and -99.3 MB. Read the column as a rough per-row indication, not an isolated per-row sandbox.

Benchmark on an otherwise-idle machine — this run was not one. Several std values above (Core AI fp32 batch 8: 113.48 ± 12.76 ms; ExecuTorch batch 8: 80.05 ± 13.27 ms) are an order of magnitude noisier than the batch-1 rows, because this measurement session ran alongside dozens of other concurrent agent processes on the same machine rather than in isolation. An earlier single-format run of this table, taken on an otherwise-busy machine, reported CoreML fp32 forward at 11.87 ± 2.43 ms — slower than its own 11.64 ms end2end, which is impossible, since forward is a strict subset of end2end; the std alone (35× an idle run's 0.13) gave the artifact away. Numbers here are directionally correct but noisier than an isolated run would be — rerun on an idle machine for tighter comparisons.

In [ ]:

import pandas as pd
from IPython.display import display

summary = pd.DataFrame(results)
display(summary)
print(
    f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, {platform.machine()}, "
    f"macOS {platform.mac_ver()[0]}."
)

import pandas as pd from IPython.display import display summary = pd.DataFrame(results) display(summary) print( f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, {platform.machine()}, " f"macOS {platform.mac_ver()[0]}." )

Next steps¶

• Fine-tuned weights — pass pretrain_weights="<path/to/checkpoint.pth>" when constructing the model.
• Smaller / faster bundle — coreml_precision="float16" (CoreML) or coreai_precision="float16" (Core AI) trade numeric drift for size and, on some Apple Silicon generations, latency; see the Core AI export docs for measured per-device tradeoffs.
• Deploy in Xcode — drag the .mlpackage into your Xcode project, or load the .aimodel / .pte with the Core AI or ExecuTorch runtime for that platform.
• Not on Apple Silicon? — see the CUDA cookbook, the CPU cookbook, or the mobile/edge cookbook.
• See the Export documentation for every format and option.