RF-DETR on Apple Silicon — Export, Inference & Latency¶

Compares every export path built for Apple Silicon: eager PyTorch (Metal via MPS, as the reference anchor) against native CoreML, Apple's newer Core AI framework, and ExecuTorch's CoreML delegate — each exported, run once for a correctness check, then benchmarked.

Format	Export	Inference
PyTorch	(no export — eager model)	predict() on device="mps"
CoreML	model.export(format="coreml") → .mlpackage	coremltools.models.MLModel
Core AI	model.export(format="coreai") → .aimodel	coreai.runtime (async)
ExecuTorch (CoreML backend)	model.export(format="executorch", backend="coreml") → .pte	executorch.runtime.Runtime

Not the same as native CoreML. ExecuTorch's coreml backend is a different path from format="coreml" — it produces a .pte file for the ExecuTorch runtime, not a .mlpackage. It runs fp16 on the Neural Engine; native CoreML and Core AI default to fp32.

Local macOS only, Apple Silicon, macOS 27+. Running (not just building) a CoreML or Core AI model requires their runtimes, which only exist on macOS / iOS / iPadOS. Core AI additionally needs macOS 27 or later. This notebook will not work on Colab or any Linux runner.

Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), general-purpose desktop/server CPU (see the CPU cookbook), or mobile/edge formats — TFLite, LiteRT, ExecuTorch's XNNPACK backend (see the mobile cookbook). ExecuTorch's qnn (Qualcomm) backend is Apple-irrelevant and covered nowhere in this series.

1. Install¶

Installs from develop to pick up the newest export fixes. coremltools (pulled in by [coreml]) also satisfies ExecuTorch's CoreML backend, which needs it but does not declare it as a dependency of [executorch].

In [ ]:

import sys

assert (3, 11) <= sys.version_info[:2] < (3, 14), "Install and run this cookbook with Python 3.11, 3.12, or 3.13."

import sys assert (3, 11) <= sys.version_info[:2] < (3, 14), "Install and run this cookbook with Python 3.11, 3.12, or 3.13."

In [ ]:

!pip install -q "rfdetr[coreml,coreai,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" psutil supervision pandas

!pip install -q "rfdetr[coreml,coreai,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" psutil supervision pandas

2. Setup¶

Every format in this notebook needs Apple Silicon; Core AI additionally needs macOS 27+. predict()'s postprocessing always ends by moving results to the CPU (Detections is numpy, not a device tensor), which forces a full MPS stream sync — so timing predict() with a plain wall-clock is already correct, the same way .cpu() forces a CUDA sync.

Three small helpers are shared by every format section below: _artifact_size_mb reports an export artifact's size on disk (a single file, or the total of a directory bundle — needed for CoreML's .mlpackage and Core AI's .aimodel, both directories), visualize_detections annotates and displays a supervision.Detections on the sample image (falling back to COCO_CLASSES for label text when a detection object carries no class_name), and measure_memory (from _benchmark) measures the host resident-memory growth of constructing a runtime and running its first inference call.

In [ ]:

import platform
from pathlib import Path

import numpy as np
import supervision as sv
import torch
from PIL import Image

from rfdetr.assets.coco_classes import COCO_CLASSES
from rfdetr.export._benchmark import BenchmarkResult, measure_latency, measure_memory

if platform.system() != "Darwin" or platform.machine() != "arm64":
    raise RuntimeError("This notebook requires Apple Silicon (macOS, arm64); none is available here.")
print(f"macOS {platform.mac_ver()[0]}, {platform.machine()}")

PYTORCH_DEVICE = "mps" if torch.backends.mps.is_available() else "cpu"
print(f"PyTorch device: {PYTORCH_DEVICE}")

EXPORT_DIR = Path("export_apple")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 5
MEASURE_RUNS = 30


def _artifact_size_mb(*paths: Path) -> float:
    total_bytes = 0
    for path in paths:
        if path.is_dir():
            total_bytes += sum(f.stat().st_size for f in path.rglob("*") if f.is_file())
        else:
            total_bytes += path.stat().st_size
    return total_bytes / 1e6


def visualize_detections(detections: sv.Detections, image: Image.Image, save_path: Path | None = None) -> None:
    names = detections.data.get("class_name") if detections.data else None
    if names is None:
        names = [COCO_CLASSES.get(int(c), str(c)) for c in detections.class_id]
    labels = [f"{name} {conf:.2f}" for name, conf in zip(names, detections.confidence)]

    annotated = sv.BoxAnnotator(thickness=3).annotate(scene=image.copy(), detections=detections)
    annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate(
        scene=annotated, detections=detections, labels=labels
    )
    if save_path is not None:
        annotated.save(save_path)
        print(f"Saved annotated image: {save_path}")
    sv.plot_image(annotated)


def _enable_notebook_inline_matplotlib() -> None:
    """Enable inline matplotlib figures when running in IPython."""
    get_ipython_func = globals().get("get_ipython")
    if not callable(get_ipython_func):
        return
    ipython = get_ipython_func()
    if ipython is not None:
        ipython.run_line_magic("matplotlib", "inline")
        ipython.run_line_magic("config", "InlineBackend.close_figures = True")


_enable_notebook_inline_matplotlib()

import platform from pathlib import Path import numpy as np import supervision as sv import torch from PIL import Image from rfdetr.assets.coco_classes import COCO_CLASSES from rfdetr.export._benchmark import BenchmarkResult, measure_latency, measure_memory if platform.system() != "Darwin" or platform.machine() != "arm64": raise RuntimeError("This notebook requires Apple Silicon (macOS, arm64); none is available here.") print(f"macOS {platform.mac_ver()[0]}, {platform.machine()}") PYTORCH_DEVICE = "mps" if torch.backends.mps.is_available() else "cpu" print(f"PyTorch device: {PYTORCH_DEVICE}") EXPORT_DIR = Path("export_apple") EXPORT_DIR.mkdir(exist_ok=True) CONFIDENCE_THRESHOLD = 0.5 WARMUP_RUNS = 5 MEASURE_RUNS = 30 def _artifact_size_mb(*paths: Path) -> float: total_bytes = 0 for path in paths: if path.is_dir(): total_bytes += sum(f.stat().st_size for f in path.rglob("*") if f.is_file()) else: total_bytes += path.stat().st_size return total_bytes / 1e6 def visualize_detections(detections: sv.Detections, image: Image.Image, save_path: Path | None = None) -> None: names = detections.data.get("class_name") if detections.data else None if names is None: names = [COCO_CLASSES.get(int(c), str(c)) for c in detections.class_id] labels = [f"{name} {conf:.2f}" for name, conf in zip(names, detections.confidence)] annotated = sv.BoxAnnotator(thickness=3).annotate(scene=image.copy(), detections=detections) annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate( scene=annotated, detections=detections, labels=labels ) if save_path is not None: annotated.save(save_path) print(f"Saved annotated image: {save_path}") sv.plot_image(annotated) def _enable_notebook_inline_matplotlib() -> None: """Enable inline matplotlib figures when running in IPython.""" get_ipython_func = globals().get("get_ipython") if not callable(get_ipython_func): return ipython = get_ipython_func() if ipython is not None: ipython.run_line_magic("matplotlib", "inline") ipython.run_line_magic("config", "InlineBackend.close_figures = True") _enable_notebook_inline_matplotlib()

3. Sample image¶

A single street scene with several COCO classes (dog, person, backpack, car) is enough to verify detections. The image is downloaded once and reused for every format below.

In [ ]:

import urllib.request

IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")

import urllib.request IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg" IMAGE_PATH = EXPORT_DIR / "sample.jpg" if not IMAGE_PATH.exists(): urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH) image = Image.open(IMAGE_PATH).convert("RGB") print(f"Sample image: {image.size[0]}×{image.size[1]}")

4. PyTorch baseline — predict() on Metal (MPS)¶

No export needed — this is the reference every other format in this notebook is compared against. predict() includes preprocessing and postprocessing — there is no separate "forward-only" path through it, so only an end-to-end number is reported for the baseline.

In [ ]:

from rfdetr import RFDETRSmall

with measure_memory() as mem:
    model = RFDETRSmall(device=PYTORCH_DEVICE)
    baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_baseline_memory_mb = mem.delta_mb
print(f"PyTorch baseline ({PYTORCH_DEVICE}): {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)

pytorch_baseline = measure_latency(
    lambda: model.predict(image),
    label=f"PyTorch predict() ({PYTORCH_DEVICE})",
    device="cpu",
    warmup=WARMUP_RUNS,
    runs=MEASURE_RUNS,
)
print(
    f"  {pytorch_baseline.label:<32}  {pytorch_baseline.mean_ms:6.2f} ms ± {pytorch_baseline.std_ms:5.2f}"
    f"   ({pytorch_baseline.fps:6.1f} FPS)   +{pytorch_baseline_memory_mb:.1f} MB"
)

from rfdetr import RFDETRSmall with measure_memory() as mem: model = RFDETRSmall(device=PYTORCH_DEVICE) baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD) pytorch_baseline_memory_mb = mem.delta_mb print(f"PyTorch baseline ({PYTORCH_DEVICE}): {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}") visualize_detections(baseline_detections, image) pytorch_baseline = measure_latency( lambda: model.predict(image), label=f"PyTorch predict() ({PYTORCH_DEVICE})", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) print( f" {pytorch_baseline.label:<32} {pytorch_baseline.mean_ms:6.2f} ms ± {pytorch_baseline.std_ms:5.2f}" f" ({pytorch_baseline.fps:6.1f} FPS) +{pytorch_baseline_memory_mb:.1f} MB" )

PyTorch inference(dtype=torch.float16) — fp16+JIT anchor¶

inference(dtype=torch.float16) fuses layers with torch.jit.script and halves arithmetic precision, mirroring the CUDA cookbook's optimized-inference row on this device instead. remove_optimized_model() reverts the in-place optimization afterward so model stays reusable for the export calls below.

In [ ]:

with measure_memory() as mem:
    model.inference(dtype=torch.float16)
    _ = model.predict(image)
pytorch_fp16_jit_memory_mb = mem.delta_mb

pytorch_fp16_jit = measure_latency(
    lambda: model.predict(image),
    label=f"PyTorch inference() fp16+JIT ({PYTORCH_DEVICE})",
    device="cpu",
    warmup=WARMUP_RUNS,
    runs=MEASURE_RUNS,
)
model.remove_optimized_model()
print(
    f"  {pytorch_fp16_jit.label:<32}  {pytorch_fp16_jit.mean_ms:6.2f} ms ± {pytorch_fp16_jit.std_ms:5.2f}"
    f"   ({pytorch_fp16_jit.fps:6.1f} FPS)   +{pytorch_fp16_jit_memory_mb:.1f} MB"
)

with measure_memory() as mem: model.inference(dtype=torch.float16) _ = model.predict(image) pytorch_fp16_jit_memory_mb = mem.delta_mb pytorch_fp16_jit = measure_latency( lambda: model.predict(image), label=f"PyTorch inference() fp16+JIT ({PYTORCH_DEVICE})", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) model.remove_optimized_model() print( f" {pytorch_fp16_jit.label:<32} {pytorch_fp16_jit.mean_ms:6.2f} ms ± {pytorch_fp16_jit.std_ms:5.2f}" f" ({pytorch_fp16_jit.fps:6.1f} FPS) +{pytorch_fp16_jit_memory_mb:.1f} MB" )

5. Native CoreML¶

What it is. Native CoreML export produces a .mlpackage you can drag directly into Xcode — no ONNX, no ExecuTorch runtime involved. Good for Apple-native (iOS/macOS) app development: it's the lowest-friction path into Apple's own ML stack, distinct from ExecuTorch's coreml backend below (a different export path that produces a .pte, not a .mlpackage). See the CoreML export docs.

Export¶

Export defaults to coreml_precision="float32" for tight CPU parity with eager PyTorch.

In [ ]:

import coremltools as ct

from rfdetr.export.benchmark import infer_transforms, post_process

coreml_path = model.export(format="coreml", output_dir=str(EXPORT_DIR / "coreml"))
coreml_size_mb = _artifact_size_mb(coreml_path)
print(f"CoreML package: {coreml_path}  ({coreml_size_mb:.1f} MB)")

import coremltools as ct from rfdetr.export.benchmark import infer_transforms, post_process coreml_path = model.export(format="coreml", output_dir=str(EXPORT_DIR / "coreml")) coreml_size_mb = _artifact_size_mb(coreml_path) print(f"CoreML package: {coreml_path} ({coreml_size_mb:.1f} MB)")

Inference¶

coremltools infers its own input/output names for the .mlpackage spec — they are not renamed to input / dets / labels. Read the input name and output order from mlmodel.get_spec().description instead of hardcoding them, then match outputs by position (dets, labels for detection).

In [ ]:

resolution = model.model_config.resolution
apple_transform = infer_transforms((resolution, resolution))
coreml_tensor, _ = apple_transform(image, None)
coreml_pixel_values = coreml_tensor[None].numpy()

target_sizes = torch.tensor([[image.height, image.width]])


def _coreml_forward() -> dict[str, np.ndarray]:
    return coreml_model.predict({coreml_input_name: coreml_pixel_values})


with measure_memory() as mem:
    coreml_model = ct.models.MLModel(str(coreml_path))
    coreml_spec = coreml_model.get_spec()
    coreml_input_name = coreml_spec.description.input[0].name
    coreml_output_names = [o.name for o in coreml_spec.description.output][:2]
    coreml_prediction = _coreml_forward()
coreml_memory_mb = mem.delta_mb

coreml_dets, coreml_labels = (
    torch.from_numpy(np.asarray(coreml_prediction[name], dtype=np.float32)) for name in coreml_output_names
)
coreml_result = post_process({"dets": coreml_dets, "labels": coreml_labels}, target_sizes)[0]
coreml_keep = coreml_result["scores"] > CONFIDENCE_THRESHOLD
print(f"CoreML: {int(coreml_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")

coreml_sv_detections = sv.Detections(
    xyxy=coreml_result["boxes"][coreml_keep].numpy(),
    confidence=coreml_result["scores"][coreml_keep].numpy(),
    class_id=coreml_result["labels"][coreml_keep].numpy().astype(int),
)
visualize_detections(coreml_sv_detections, image, EXPORT_DIR / "annotated_coreml.jpg")

resolution = model.model_config.resolution apple_transform = infer_transforms((resolution, resolution)) coreml_tensor, _ = apple_transform(image, None) coreml_pixel_values = coreml_tensor[None].numpy() target_sizes = torch.tensor([[image.height, image.width]]) def _coreml_forward() -> dict[str, np.ndarray]: return coreml_model.predict({coreml_input_name: coreml_pixel_values}) with measure_memory() as mem: coreml_model = ct.models.MLModel(str(coreml_path)) coreml_spec = coreml_model.get_spec() coreml_input_name = coreml_spec.description.input[0].name coreml_output_names = [o.name for o in coreml_spec.description.output][:2] coreml_prediction = _coreml_forward() coreml_memory_mb = mem.delta_mb coreml_dets, coreml_labels = ( torch.from_numpy(np.asarray(coreml_prediction[name], dtype=np.float32)) for name in coreml_output_names ) coreml_result = post_process({"dets": coreml_dets, "labels": coreml_labels}, target_sizes)[0] coreml_keep = coreml_result["scores"] > CONFIDENCE_THRESHOLD print(f"CoreML: {int(coreml_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreml_sv_detections = sv.Detections( xyxy=coreml_result["boxes"][coreml_keep].numpy(), confidence=coreml_result["scores"][coreml_keep].numpy(), class_id=coreml_result["labels"][coreml_keep].numpy().astype(int), ) visualize_detections(coreml_sv_detections, image, EXPORT_DIR / "annotated_coreml.jpg")

Benchmark¶

In [ ]:

coreml_forward = measure_latency(
    _coreml_forward, label="CoreML forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)


def _coreml_end2end() -> None:
    tensor, _ = apple_transform(image, None)
    prediction = coreml_model.predict({coreml_input_name: tensor[None].numpy()})
    dets, labels = (torch.from_numpy(np.asarray(prediction[name], dtype=np.float32)) for name in coreml_output_names)
    post_process({"dets": dets, "labels": labels}, target_sizes)


coreml_end2end = measure_latency(
    _coreml_end2end, label="CoreML end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (coreml_forward, coreml_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

coreml_forward = measure_latency( _coreml_forward, label="CoreML forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) def _coreml_end2end() -> None: tensor, _ = apple_transform(image, None) prediction = coreml_model.predict({coreml_input_name: tensor[None].numpy()}) dets, labels = (torch.from_numpy(np.asarray(prediction[name], dtype=np.float32)) for name in coreml_output_names) post_process({"dets": dets, "labels": labels}, target_sizes) coreml_end2end = measure_latency( _coreml_end2end, label="CoreML end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (coreml_forward, coreml_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

CoreML (fp16)¶

coreml_precision="float16" halves the weight precision, producing a smaller, ANE-oriented bundle. The tradeoff is larger numeric drift: fp16 narrows the fp32 safety margin around near-tied torch.topk query-ranking decisions, so a rank swap (and therefore a slightly different detection set) becomes more likely than in the fp32 row above.

In [ ]:

coreml_fp16_path = model.export(format="coreml", coreml_precision="float16", output_dir=str(EXPORT_DIR / "coreml_fp16"))
coreml_fp16_size_mb = _artifact_size_mb(coreml_fp16_path)
print(f"CoreML package (fp16): {coreml_fp16_path}  ({coreml_fp16_size_mb:.1f} MB)")


def _coreml_fp16_forward() -> dict[str, np.ndarray]:
    return coreml_fp16_model.predict({coreml_fp16_input_name: coreml_pixel_values})


with measure_memory() as mem:
    coreml_fp16_model = ct.models.MLModel(str(coreml_fp16_path))
    coreml_fp16_spec = coreml_fp16_model.get_spec()
    coreml_fp16_input_name = coreml_fp16_spec.description.input[0].name
    coreml_fp16_output_names = [o.name for o in coreml_fp16_spec.description.output][:2]
    coreml_fp16_prediction = _coreml_fp16_forward()
coreml_fp16_memory_mb = mem.delta_mb

coreml_fp16_dets, coreml_fp16_labels = (
    torch.from_numpy(np.asarray(coreml_fp16_prediction[name], dtype=np.float32)) for name in coreml_fp16_output_names
)
coreml_fp16_result = post_process({"dets": coreml_fp16_dets, "labels": coreml_fp16_labels}, target_sizes)[0]
coreml_fp16_keep = coreml_fp16_result["scores"] > CONFIDENCE_THRESHOLD
print(f"CoreML fp16: {int(coreml_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")

coreml_fp16_sv_detections = sv.Detections(
    xyxy=coreml_fp16_result["boxes"][coreml_fp16_keep].numpy(),
    confidence=coreml_fp16_result["scores"][coreml_fp16_keep].numpy(),
    class_id=coreml_fp16_result["labels"][coreml_fp16_keep].numpy().astype(int),
)
visualize_detections(coreml_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreml_fp16.jpg")

coreml_fp16_forward_result = measure_latency(
    _coreml_fp16_forward, label="CoreML fp16 forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)


def _coreml_fp16_end2end() -> None:
    tensor, _ = apple_transform(image, None)
    prediction = coreml_fp16_model.predict({coreml_fp16_input_name: tensor[None].numpy()})
    dets, labels = (
        torch.from_numpy(np.asarray(prediction[name], dtype=np.float32)) for name in coreml_fp16_output_names
    )
    post_process({"dets": dets, "labels": labels}, target_sizes)


coreml_fp16_end2end = measure_latency(
    _coreml_fp16_end2end, label="CoreML fp16 end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (coreml_fp16_forward_result, coreml_fp16_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

coreml_fp16_path = model.export(format="coreml", coreml_precision="float16", output_dir=str(EXPORT_DIR / "coreml_fp16")) coreml_fp16_size_mb = _artifact_size_mb(coreml_fp16_path) print(f"CoreML package (fp16): {coreml_fp16_path} ({coreml_fp16_size_mb:.1f} MB)") def _coreml_fp16_forward() -> dict[str, np.ndarray]: return coreml_fp16_model.predict({coreml_fp16_input_name: coreml_pixel_values}) with measure_memory() as mem: coreml_fp16_model = ct.models.MLModel(str(coreml_fp16_path)) coreml_fp16_spec = coreml_fp16_model.get_spec() coreml_fp16_input_name = coreml_fp16_spec.description.input[0].name coreml_fp16_output_names = [o.name for o in coreml_fp16_spec.description.output][:2] coreml_fp16_prediction = _coreml_fp16_forward() coreml_fp16_memory_mb = mem.delta_mb coreml_fp16_dets, coreml_fp16_labels = ( torch.from_numpy(np.asarray(coreml_fp16_prediction[name], dtype=np.float32)) for name in coreml_fp16_output_names ) coreml_fp16_result = post_process({"dets": coreml_fp16_dets, "labels": coreml_fp16_labels}, target_sizes)[0] coreml_fp16_keep = coreml_fp16_result["scores"] > CONFIDENCE_THRESHOLD print(f"CoreML fp16: {int(coreml_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreml_fp16_sv_detections = sv.Detections( xyxy=coreml_fp16_result["boxes"][coreml_fp16_keep].numpy(), confidence=coreml_fp16_result["scores"][coreml_fp16_keep].numpy(), class_id=coreml_fp16_result["labels"][coreml_fp16_keep].numpy().astype(int), ) visualize_detections(coreml_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreml_fp16.jpg") coreml_fp16_forward_result = measure_latency( _coreml_fp16_forward, label="CoreML fp16 forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) def _coreml_fp16_end2end() -> None: tensor, _ = apple_transform(image, None) prediction = coreml_fp16_model.predict({coreml_fp16_input_name: tensor[None].numpy()}) dets, labels = ( torch.from_numpy(np.asarray(prediction[name], dtype=np.float32)) for name in coreml_fp16_output_names ) post_process({"dets": dets, "labels": labels}, target_sizes) coreml_fp16_end2end = measure_latency( _coreml_fp16_end2end, label="CoreML fp16 end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (coreml_fp16_forward_result, coreml_fp16_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

6. Core AI¶

What it is. Core AI is Apple's newer on-device inference framework (iOS, iPadOS, macOS 27+). RF-DETR traces the model with torch.export and converts it to an .aimodel asset — no ONNX step. Good for apps targeting the newest Apple OS versions: the runtime decides at load time whether the CPU, GPU, or Neural Engine executes it, rather than that choice being baked in at export time like CoreML. See the Core AI export docs.

Export¶

In [ ]:

coreai_path = model.export(format="coreai", output_dir=str(EXPORT_DIR / "coreai"))
coreai_size_mb = _artifact_size_mb(coreai_path)
print(f"Core AI asset: {coreai_path}  ({coreai_size_mb:.1f} MB)")

coreai_path = model.export(format="coreai", output_dir=str(EXPORT_DIR / "coreai")) coreai_size_mb = _artifact_size_mb(coreai_path) print(f"Core AI asset: {coreai_path} ({coreai_size_mb:.1f} MB)")

Inference¶

Unlike CoreML, Core AI keeps the exported tensor names — input, dets, labels. Its Python runtime is partly async: AIModel.load awaits, but the InferenceFunction load_function returns is a plain synchronous call — only calling that function (coreai_fn(...)) is a coroutine. The notebook awaits that call directly in its correctness and benchmark cells, which keeps inference compatible with Jupyter's running event loop.

In [ ]:

import time
from collections.abc import Awaitable, Callable
from typing import Any


async def _measure_async_latency(
    fn: Callable[[], Awaitable[object]], *, label: str, warmup: int, runs: int
) -> BenchmarkResult:
    """Measure an awaitable call after warmup using synchronized wall-clock samples.

    Args:
        fn: Zero-argument callable returning the inference awaitable.
        label: Name shown in the benchmark results.
        warmup: Untimed calls before measurement begins.
        runs: Timed calls used to calculate mean and standard deviation.

    Returns:
        The measured latency statistics for the awaitable call.
    """
    for _ in range(warmup):
        await fn()
    timings: list[float] = []
    for _ in range(runs):
        start = time.perf_counter()
        await fn()
        timings.append((time.perf_counter() - start) * 1000.0)
    return BenchmarkResult(label, float(np.mean(timings)), float(np.std(timings)))


import coreai.runtime as coreai_rt


async def _load_coreai() -> tuple[Any, Any]:
    loaded = await coreai_rt.AIModel.load(str(coreai_path), coreai_rt.SpecializationOptions.default())
    fn = loaded.load_function("main")  # synchronous — only AIModel.load and InferenceFunction.__call__ are async
    return loaded, fn


async def _coreai_call(pixel_values: np.ndarray) -> dict[str, np.ndarray]:
    outputs = await coreai_fn({"input": coreai_rt.NDArray(pixel_values)})
    return {name: outputs[name].numpy() for name in ("dets", "labels")}


coreai_pixel_values = coreml_pixel_values  # same NCHW float32 preprocessing contract as CoreML

with measure_memory() as mem:
    coreai_model, coreai_fn = await _load_coreai()
    coreai_outputs = await _coreai_call(coreai_pixel_values)
coreai_memory_mb = mem.delta_mb

coreai_dets = torch.from_numpy(coreai_outputs["dets"])
coreai_labels = torch.from_numpy(coreai_outputs["labels"])
coreai_result = post_process({"dets": coreai_dets, "labels": coreai_labels}, target_sizes)[0]
coreai_keep = coreai_result["scores"] > CONFIDENCE_THRESHOLD
print(f"Core AI: {int(coreai_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")

coreai_sv_detections = sv.Detections(
    xyxy=coreai_result["boxes"][coreai_keep].numpy(),
    confidence=coreai_result["scores"][coreai_keep].numpy(),
    class_id=coreai_result["labels"][coreai_keep].numpy().astype(int),
)
visualize_detections(coreai_sv_detections, image, EXPORT_DIR / "annotated_coreai.jpg")

import time from collections.abc import Awaitable, Callable from typing import Any async def _measure_async_latency( fn: Callable[[], Awaitable[object]], *, label: str, warmup: int, runs: int ) -> BenchmarkResult: """Measure an awaitable call after warmup using synchronized wall-clock samples. Args: fn: Zero-argument callable returning the inference awaitable. label: Name shown in the benchmark results. warmup: Untimed calls before measurement begins. runs: Timed calls used to calculate mean and standard deviation. Returns: The measured latency statistics for the awaitable call. """ for _ in range(warmup): await fn() timings: list[float] = [] for _ in range(runs): start = time.perf_counter() await fn() timings.append((time.perf_counter() - start) * 1000.0) return BenchmarkResult(label, float(np.mean(timings)), float(np.std(timings))) import coreai.runtime as coreai_rt async def _load_coreai() -> tuple[Any, Any]: loaded = await coreai_rt.AIModel.load(str(coreai_path), coreai_rt.SpecializationOptions.default()) fn = loaded.load_function("main") # synchronous — only AIModel.load and InferenceFunction.__call__ are async return loaded, fn async def _coreai_call(pixel_values: np.ndarray) -> dict[str, np.ndarray]: outputs = await coreai_fn({"input": coreai_rt.NDArray(pixel_values)}) return {name: outputs[name].numpy() for name in ("dets", "labels")} coreai_pixel_values = coreml_pixel_values # same NCHW float32 preprocessing contract as CoreML with measure_memory() as mem: coreai_model, coreai_fn = await _load_coreai() coreai_outputs = await _coreai_call(coreai_pixel_values) coreai_memory_mb = mem.delta_mb coreai_dets = torch.from_numpy(coreai_outputs["dets"]) coreai_labels = torch.from_numpy(coreai_outputs["labels"]) coreai_result = post_process({"dets": coreai_dets, "labels": coreai_labels}, target_sizes)[0] coreai_keep = coreai_result["scores"] > CONFIDENCE_THRESHOLD print(f"Core AI: {int(coreai_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreai_sv_detections = sv.Detections( xyxy=coreai_result["boxes"][coreai_keep].numpy(), confidence=coreai_result["scores"][coreai_keep].numpy(), class_id=coreai_result["labels"][coreai_keep].numpy().astype(int), ) visualize_detections(coreai_sv_detections, image, EXPORT_DIR / "annotated_coreai.jpg")

Benchmark¶

In [ ]:

coreai_forward = await _measure_async_latency(
    lambda: _coreai_call(coreai_pixel_values),
    label="Core AI forward",
    warmup=WARMUP_RUNS,
    runs=MEASURE_RUNS,
)


async def _coreai_end2end() -> None:
    tensor, _ = apple_transform(image, None)
    outputs = await _coreai_call(tensor[None].numpy())
    dets = torch.from_numpy(outputs["dets"])
    labels = torch.from_numpy(outputs["labels"])
    post_process({"dets": dets, "labels": labels}, target_sizes)


coreai_end2end = await _measure_async_latency(
    _coreai_end2end, label="Core AI end2end", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (coreai_forward, coreai_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

coreai_forward = await _measure_async_latency( lambda: _coreai_call(coreai_pixel_values), label="Core AI forward", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) async def _coreai_end2end() -> None: tensor, _ = apple_transform(image, None) outputs = await _coreai_call(tensor[None].numpy()) dets = torch.from_numpy(outputs["dets"]) labels = torch.from_numpy(outputs["labels"]) post_process({"dets": dets, "labels": labels}, target_sizes) coreai_end2end = await _measure_async_latency( _coreai_end2end, label="Core AI end2end", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (coreai_forward, coreai_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

Core AI (fp16)¶

coreai_precision="float16" traces and stores the whole graph in fp16, so the asset also takes and returns float16 tensors — the input is cast below, and the outputs are cast back to fp32 before postprocessing. One part of the graph stays fp32 regardless: RF-DETR forces topk back to float32 even in an fp16 export, because the Neural Engine's fp16 topk returns corrupt indices and the two-stage query selection would gather the wrong encoder tokens. That caps how much of the graph actually runs in fp16.

In [ ]:

coreai_fp16_path = model.export(format="coreai", coreai_precision="float16", output_dir=str(EXPORT_DIR / "coreai_fp16"))
coreai_fp16_size_mb = _artifact_size_mb(coreai_fp16_path)
print(f"Core AI asset (fp16): {coreai_fp16_path}  ({coreai_fp16_size_mb:.1f} MB)")

coreai_fp16_pixel_values = coreml_pixel_values.astype(np.float16)


async def _load_coreai_fp16() -> tuple[Any, Any]:
    loaded = await coreai_rt.AIModel.load(str(coreai_fp16_path), coreai_rt.SpecializationOptions.default())
    fn = loaded.load_function("main")
    return loaded, fn


async def _coreai_fp16_call(pixel_values: np.ndarray) -> dict[str, np.ndarray]:
    outputs = await coreai_fp16_fn({"input": coreai_rt.NDArray(pixel_values)})
    # fp16 out, fp32 in for post_process — CPU fp16 op coverage is narrower than fp32's.
    return {name: outputs[name].numpy().astype(np.float32) for name in ("dets", "labels")}


with measure_memory() as mem:
    coreai_fp16_model, coreai_fp16_fn = await _load_coreai_fp16()
    coreai_fp16_outputs = await _coreai_fp16_call(coreai_fp16_pixel_values)
coreai_fp16_memory_mb = mem.delta_mb

coreai_fp16_result = post_process(
    {"dets": torch.from_numpy(coreai_fp16_outputs["dets"]), "labels": torch.from_numpy(coreai_fp16_outputs["labels"])},
    target_sizes,
)[0]
coreai_fp16_keep = coreai_fp16_result["scores"] > CONFIDENCE_THRESHOLD
print(f"Core AI fp16: {int(coreai_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")

coreai_fp16_sv_detections = sv.Detections(
    xyxy=coreai_fp16_result["boxes"][coreai_fp16_keep].numpy(),
    confidence=coreai_fp16_result["scores"][coreai_fp16_keep].numpy(),
    class_id=coreai_fp16_result["labels"][coreai_fp16_keep].numpy().astype(int),
)
visualize_detections(coreai_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreai_fp16.jpg")

coreai_fp16_forward = await _measure_async_latency(
    lambda: _coreai_fp16_call(coreai_fp16_pixel_values),
    label="Core AI fp16 forward",
    warmup=WARMUP_RUNS,
    runs=MEASURE_RUNS,
)


async def _coreai_fp16_end2end() -> None:
    tensor, _ = apple_transform(image, None)
    outputs = await _coreai_fp16_call(tensor[None].numpy().astype(np.float16))
    dets = torch.from_numpy(outputs["dets"])
    labels = torch.from_numpy(outputs["labels"])
    post_process({"dets": dets, "labels": labels}, target_sizes)


coreai_fp16_end2end = await _measure_async_latency(
    _coreai_fp16_end2end, label="Core AI fp16 end2end", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (coreai_fp16_forward, coreai_fp16_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

coreai_fp16_path = model.export(format="coreai", coreai_precision="float16", output_dir=str(EXPORT_DIR / "coreai_fp16")) coreai_fp16_size_mb = _artifact_size_mb(coreai_fp16_path) print(f"Core AI asset (fp16): {coreai_fp16_path} ({coreai_fp16_size_mb:.1f} MB)") coreai_fp16_pixel_values = coreml_pixel_values.astype(np.float16) async def _load_coreai_fp16() -> tuple[Any, Any]: loaded = await coreai_rt.AIModel.load(str(coreai_fp16_path), coreai_rt.SpecializationOptions.default()) fn = loaded.load_function("main") return loaded, fn async def _coreai_fp16_call(pixel_values: np.ndarray) -> dict[str, np.ndarray]: outputs = await coreai_fp16_fn({"input": coreai_rt.NDArray(pixel_values)}) # fp16 out, fp32 in for post_process — CPU fp16 op coverage is narrower than fp32's. return {name: outputs[name].numpy().astype(np.float32) for name in ("dets", "labels")} with measure_memory() as mem: coreai_fp16_model, coreai_fp16_fn = await _load_coreai_fp16() coreai_fp16_outputs = await _coreai_fp16_call(coreai_fp16_pixel_values) coreai_fp16_memory_mb = mem.delta_mb coreai_fp16_result = post_process( {"dets": torch.from_numpy(coreai_fp16_outputs["dets"]), "labels": torch.from_numpy(coreai_fp16_outputs["labels"])}, target_sizes, )[0] coreai_fp16_keep = coreai_fp16_result["scores"] > CONFIDENCE_THRESHOLD print(f"Core AI fp16: {int(coreai_fp16_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") coreai_fp16_sv_detections = sv.Detections( xyxy=coreai_fp16_result["boxes"][coreai_fp16_keep].numpy(), confidence=coreai_fp16_result["scores"][coreai_fp16_keep].numpy(), class_id=coreai_fp16_result["labels"][coreai_fp16_keep].numpy().astype(int), ) visualize_detections(coreai_fp16_sv_detections, image, EXPORT_DIR / "annotated_coreai_fp16.jpg") coreai_fp16_forward = await _measure_async_latency( lambda: _coreai_fp16_call(coreai_fp16_pixel_values), label="Core AI fp16 forward", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) async def _coreai_fp16_end2end() -> None: tensor, _ = apple_transform(image, None) outputs = await _coreai_fp16_call(tensor[None].numpy().astype(np.float16)) dets = torch.from_numpy(outputs["dets"]) labels = torch.from_numpy(outputs["labels"]) post_process({"dets": dets, "labels": labels}, target_sizes) coreai_fp16_end2end = await _measure_async_latency( _coreai_fp16_end2end, label="Core AI fp16 end2end", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (coreai_fp16_forward, coreai_fp16_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

7. ExecuTorch (CoreML backend)¶

What it is. This is the same ExecuTorch runtime as the mobile cookbook's XNNPACK section, but lowered to Apple's coreml backend instead — still a portable .pte file, no .mlpackage, but running on the Apple Neural Engine. Good for staying inside the ExecuTorch/PyTorch deployment story while still reaching Apple's Neural Engine, when native CoreML's separate export/runtime isn't a fit; the tradeoff is fp16 only, unlike native CoreML and Core AI's fp32 default. See the ExecuTorch export docs.

Export¶

backend="coreml" targets the Apple Neural Engine, iPhone/iPad/Mac, and runs fp16 — unlike native CoreML and Core AI above, which default to fp32. This is the ExecuTorch delegate, not native CoreML — it produces a .pte for the ExecuTorch runtime, distinct from format="coreml".

In [ ]:

pte_path = model.export(format="executorch", backend="coreml", output_dir=str(EXPORT_DIR / "executorch"))
executorch_size_mb = _artifact_size_mb(pte_path)
print(f"ExecuTorch (CoreML) program: {pte_path}  ({executorch_size_mb:.1f} MB)")

pte_path = model.export(format="executorch", backend="coreml", output_dir=str(EXPORT_DIR / "executorch")) executorch_size_mb = _artifact_size_mb(pte_path) print(f"ExecuTorch (CoreML) program: {pte_path} ({executorch_size_mb:.1f} MB)")

Inference¶

The input tensor must be contiguous — see the mobile cookbook for why; infer_transforms already returns one, the trailing .contiguous() is defensive.

In [ ]:

from executorch.runtime import Runtime


def _executorch_forward(pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
    return executorch_method.execute([pixel_values])


executorch_tensor, _ = apple_transform(image, None)
executorch_pixel_values = executorch_tensor[None].float().contiguous()

with measure_memory() as mem:
    executorch_method = Runtime.get().load_program(str(pte_path)).load_method("forward")
    executorch_dets, executorch_labels = _executorch_forward(executorch_pixel_values)
executorch_memory_mb = mem.delta_mb

executorch_result = post_process({"dets": executorch_dets, "labels": executorch_labels}, target_sizes)[0]
executorch_keep = executorch_result["scores"] > CONFIDENCE_THRESHOLD
print(f"ExecuTorch (CoreML): {int(executorch_keep.sum())} detections above {CONFIDENCE_THRESHOLD}")

executorch_sv_detections = sv.Detections(
    xyxy=executorch_result["boxes"][executorch_keep].numpy(),
    confidence=executorch_result["scores"][executorch_keep].numpy(),
    class_id=executorch_result["labels"][executorch_keep].numpy().astype(int),
)
visualize_detections(executorch_sv_detections, image, EXPORT_DIR / "annotated_executorch.jpg")

from executorch.runtime import Runtime def _executorch_forward(pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: return executorch_method.execute([pixel_values]) executorch_tensor, _ = apple_transform(image, None) executorch_pixel_values = executorch_tensor[None].float().contiguous() with measure_memory() as mem: executorch_method = Runtime.get().load_program(str(pte_path)).load_method("forward") executorch_dets, executorch_labels = _executorch_forward(executorch_pixel_values) executorch_memory_mb = mem.delta_mb executorch_result = post_process({"dets": executorch_dets, "labels": executorch_labels}, target_sizes)[0] executorch_keep = executorch_result["scores"] > CONFIDENCE_THRESHOLD print(f"ExecuTorch (CoreML): {int(executorch_keep.sum())} detections above {CONFIDENCE_THRESHOLD}") executorch_sv_detections = sv.Detections( xyxy=executorch_result["boxes"][executorch_keep].numpy(), confidence=executorch_result["scores"][executorch_keep].numpy(), class_id=executorch_result["labels"][executorch_keep].numpy().astype(int), ) visualize_detections(executorch_sv_detections, image, EXPORT_DIR / "annotated_executorch.jpg")

Benchmark¶

In [ ]:

executorch_forward = measure_latency(
    lambda: _executorch_forward(executorch_pixel_values),
    label="ExecuTorch (CoreML) forward",
    device="cpu",
    warmup=WARMUP_RUNS,
    runs=MEASURE_RUNS,
)


def _executorch_end2end() -> None:
    tensor, _ = apple_transform(image, None)
    pixel_values = tensor[None].float().contiguous()
    dets, labels = _executorch_forward(pixel_values)
    post_process({"dets": dets, "labels": labels}, target_sizes)


executorch_end2end = measure_latency(
    _executorch_end2end, label="ExecuTorch (CoreML) end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
)

for r in (executorch_forward, executorch_end2end):
    print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

executorch_forward = measure_latency( lambda: _executorch_forward(executorch_pixel_values), label="ExecuTorch (CoreML) forward", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) def _executorch_end2end() -> None: tensor, _ = apple_transform(image, None) pixel_values = tensor[None].float().contiguous() dets, labels = _executorch_forward(pixel_values) post_process({"dets": dets, "labels": labels}, target_sizes) executorch_end2end = measure_latency( _executorch_end2end, label="ExecuTorch (CoreML) end2end", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (executorch_forward, executorch_end2end): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

8. Results¶

Measured on an Apple M-series Mac (arm64), macOS 27.0, rfdetr v1.11.0, batch 1, 5 warmup + 30 timed runs, RFDETRSmall. Numbers vary by Apple Silicon generation — rerun the cells below to get yours. Config is the precision/backend used for that row; — marks a scope this format doesn't have (PyTorch's predict() has no forward-only path). Memory [MB] is host resident-memory growth across constructing the runtime plus its first inference call.

Format	Config	forward [ms]	end2end [ms]	FPS [img/s] (end2end)	Memory [MB]
PyTorch predict()	mps, fp32	—	22.62 ± 1.47	44.2	619.4
PyTorch inference()	mps, fp16+JIT	—	18.63 ± 0.44	53.7	148.7
CoreML	fp32 (default)	9.32 ± 0.13	11.62 ± 0.20	86.1	175.6
CoreML	fp16	17.80 ± 0.25	20.05 ± 0.40	49.9	5.1
Core AI	fp32 (default)	10.28 ± 0.24	12.54 ± 0.22	79.7	156.1
Core AI	fp16	9.07 ± 0.20	11.45 ± 0.18	87.3	101.6
ExecuTorch	CoreML backend, fp16	21.93 ± 0.33	24.39 ± 0.32	41.0	31.1

fp16 is not automatically faster here. On this machine CoreML's fp16 export came out slower than its fp32 default (20.05 vs 11.62 ms end2end), while Core AI's fp16 came out faster (11.45 vs 12.54 ms). Both results reproduced across two back-to-back runs. RF-DETR's own ExecuTorch CoreML-backend code records the same surprise from the other direction — forcing explicit fp16 compile-specs measured worse than letting the partitioner choose. Treat fp16 on Apple Silicon as something to measure per runtime and per chip generation, not a free win.

Memory [MB] is an approximation on this page. It is host RSS growth across each section's own construct + first-inference bracket, in one shared process — so when several Apple accelerator runtimes are alive at once, the OS reclaiming an earlier section's Neural Engine buffers can land inside a later section's bracket. That shows up above as CoreML fp16's implausibly small 5.1 MB, and an earlier run of this notebook (without the fp16 sections, so with different section ordering) reproducibly measured Core AI at -99.5 MB and -99.3 MB. Read the column as a rough per-format indication, not an isolated per-format sandbox.

Benchmark on an otherwise-idle machine. An earlier run of this table, taken while two other processes were busy, reported CoreML fp32 forward at 11.87 ± 2.43 ms — slower than its own 11.64 ms end2end, which is impossible, since forward is a strict subset of end2end. The std alone (35× the idle run's 0.13) gives the artifact away. Numbers here are only meaningful when nothing else is competing for CPU and Neural Engine.

In [ ]:

import pandas as pd


def _fmt_ms(result: BenchmarkResult | None) -> str:
    if result is None:
        return "—"
    return f"{result.mean_ms:.2f} ± {result.std_ms:.2f}"


def _result_row(
    format_label: str,
    config: str,
    forward: BenchmarkResult | None,
    end2end: BenchmarkResult | None,
    memory_mb: float | None,
) -> dict:
    fps = (end2end or forward).fps
    return {
        "Format": format_label,
        "Config": config,
        "forward [ms]": _fmt_ms(forward),
        "end2end [ms]": _fmt_ms(end2end),
        "FPS [img/s] (end2end)": round(fps, 1),
        "Memory [MB]": f"{memory_mb:.1f}" if memory_mb is not None else "—",
    }


summary = pd.DataFrame(
    [
        _result_row("PyTorch predict()", f"{PYTORCH_DEVICE}, fp32", None, pytorch_baseline, pytorch_baseline_memory_mb),
        _result_row(
            "PyTorch inference()",
            f"{PYTORCH_DEVICE}, fp16+JIT",
            None,
            pytorch_fp16_jit,
            pytorch_fp16_jit_memory_mb,
        ),
        _result_row("CoreML", "fp32 (default)", coreml_forward, coreml_end2end, coreml_memory_mb),
        _result_row("CoreML", "fp16", coreml_fp16_forward_result, coreml_fp16_end2end, coreml_fp16_memory_mb),
        _result_row("Core AI", "fp32 (default)", coreai_forward, coreai_end2end, coreai_memory_mb),
        _result_row("Core AI", "fp16", coreai_fp16_forward, coreai_fp16_end2end, coreai_fp16_memory_mb),
        _result_row("ExecuTorch", "CoreML backend, fp16", executorch_forward, executorch_end2end, executorch_memory_mb),
    ]
).set_index("Format")
print(summary.to_string())
print(
    f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs, batch 1, {platform.machine()}, macOS {platform.mac_ver()[0]}."
)

import pandas as pd def _fmt_ms(result: BenchmarkResult | None) -> str: if result is None: return "—" return f"{result.mean_ms:.2f} ± {result.std_ms:.2f}" def _result_row( format_label: str, config: str, forward: BenchmarkResult | None, end2end: BenchmarkResult | None, memory_mb: float | None, ) -> dict: fps = (end2end or forward).fps return { "Format": format_label, "Config": config, "forward [ms]": _fmt_ms(forward), "end2end [ms]": _fmt_ms(end2end), "FPS [img/s] (end2end)": round(fps, 1), "Memory [MB]": f"{memory_mb:.1f}" if memory_mb is not None else "—", } summary = pd.DataFrame( [ _result_row("PyTorch predict()", f"{PYTORCH_DEVICE}, fp32", None, pytorch_baseline, pytorch_baseline_memory_mb), _result_row( "PyTorch inference()", f"{PYTORCH_DEVICE}, fp16+JIT", None, pytorch_fp16_jit, pytorch_fp16_jit_memory_mb, ), _result_row("CoreML", "fp32 (default)", coreml_forward, coreml_end2end, coreml_memory_mb), _result_row("CoreML", "fp16", coreml_fp16_forward_result, coreml_fp16_end2end, coreml_fp16_memory_mb), _result_row("Core AI", "fp32 (default)", coreai_forward, coreai_end2end, coreai_memory_mb), _result_row("Core AI", "fp16", coreai_fp16_forward, coreai_fp16_end2end, coreai_fp16_memory_mb), _result_row("ExecuTorch", "CoreML backend, fp16", executorch_forward, executorch_end2end, executorch_memory_mb), ] ).set_index("Format") print(summary.to_string()) print( f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs, batch 1, {platform.machine()}, macOS {platform.mac_ver()[0]}." )

Next steps¶

• Fine-tuned weights — pass pretrain_weights="<path/to/checkpoint.pth>" when constructing the model.
• Smaller / faster bundle — coreml_precision="float16" (CoreML) or coreai_precision="float16" (Core AI) trade numeric drift for size and, on some Apple Silicon generations, latency; see the Core AI export docs for measured per-device tradeoffs.
• Deploy in Xcode — drag the .mlpackage into your Xcode project, or load the .aimodel / .pte with the Core AI or ExecuTorch runtime for that platform.
• Not on Apple Silicon? — see the CUDA cookbook, the CPU cookbook, or the mobile/edge cookbook.
• See the Export documentation for every format and option.