RF-DETR on Mobile & Edge — Export, Inference & Latency¶

Compares every export path built for on-device / ARM mobile and edge deployment: eager PyTorch (running here as the reference anchor, not itself a deployment target) against TFLite, LiteRT, and ExecuTorch's XNNPACK backend — each exported, run once for a correctness check, then benchmarked on the CPU that also stands in for a phone/edge chip's CPU core.

Format	Export	Inference
PyTorch	(no export — eager model)	predict() / inference() (TorchScript JIT, fp32)
TFLite	ONNX → TensorFlow → .tflite (onnx2tf)	tensorflow.lite.Interpreter
LiteRT	torch.export → .tflite directly (litert-torch)	ai_edge_litert.interpreter.Interpreter
ExecuTorch (XNNPACK)	torch.export → .pte	executorch.runtime.Runtime

Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), general-purpose desktop/server CPU via ONNX Runtime or OpenVINO (see the CPU cookbook), or Apple Silicon — native CoreML, Core AI, and ExecuTorch's CoreML backend (see the Apple cookbook).

Python version. The [tflite] extra installs only on Python 3.12 exactly — its onnx2tf / tensorflow dependency stack pins numpy==1.26.4, which cannot be satisfied on 3.10, 3.11, 3.13, or 3.14 (see pyproject.toml). This constraint is why TFLite/LiteRT/ExecuTorch get their own notebook, separate from the CPU cookbook's ONNX/OpenVINO, which have no such pin.

Two parts, two environments. pyproject.toml's [tool.uv] conflicts declares [tflite] unsatisfiable alongside both [litert] (onnx2tf pins ai-edge-litert==2.1.2; litert-torch needs >=2.2.0) and [executorch] — the three formats cannot share one Python environment. This notebook is split into Part A (TFLite) and Part B (LiteRT + ExecuTorch), each with its own install cell; run one part, then restart the runtime before running the other. Each part re-establishes its own PyTorch baseline and its own results table rather than assuming state survives the restart.

Which latency number is this? Every number in this notebook is a Tier C measurement: forward-only and end-to-end timings for each export format, on one CPU, against a PyTorch anchor measured on that same machine. Its value is ratios within this machine — never set these milliseconds against the architecture headline on the Benchmarks page (NVIDIA T4, TensorRT FP16, model only) or the number on Roboflow's RF-DETR model page (L4, Roboflow Inference, scope not stated, batch 1) — those are GPU numbers, not comparable to a CPU run at all, let alone without checking precision, batch size, and scope.

Part A: TFLite¶

1. Install¶

RFDETR_SPEC holds the install source and defaults to the develop branch, to pick up the newest export fixes; to install the 1.11.2 release from PyPI instead, swap in the commented >=1.11.2,<2 line. The assertion below fails fast on any interpreter other than 3.12, before the (slow) install runs. Colab ships mutually inconsistent preinstalled packages that otherwise crash import rfdetr, so two are aligned: torchaudio is uninstalled (RF-DETR never uses it, but transformers imports it when present and a torch/torchaudio CUDA-version mismatch then errors), and pillow is force-reinstalled to a clean version (a half-upgraded PIL breaks torchvision's import with cannot import name '_Ink').

Colab: after this cell, Runtime → Restart session, then run from the next cell — Colab keeps the old package versions loaded until a restart.

In [ ]:

import sys

assert sys.version_info[:2] == (3, 12), (
    "rfdetr[tflite] requires Python 3.12 exactly (see pyproject.toml); this runtime is "
    f"{sys.version_info.major}.{sys.version_info.minor}. Use a Python 3.12 environment instead."
)

import sys assert sys.version_info[:2] == (3, 12), ( "rfdetr[tflite] requires Python 3.12 exactly (see pyproject.toml); this runtime is " f"{sys.version_info.major}.{sys.version_info.minor}. Use a Python 3.12 environment instead." )

In [ ]:

RFDETR_SPEC = "rfdetr[tflite] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[tflite]>=1.11.2,<2"
!pip install -q "{RFDETR_SPEC}" psutil supervision pandas
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
!pip uninstall -q -y torchaudio

RFDETR_SPEC = "rfdetr[tflite] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" # RFDETR_SPEC = "rfdetr[tflite]>=1.11.2,<2" !pip install -q "{RFDETR_SPEC}" psutil supervision pandas !pip install -q --force-reinstall --no-deps "pillow==11.3.0" !pip uninstall -q -y torchaudio

2. Setup¶

Every format in this notebook runs on CPU — no GPU is needed.

Several small helpers are shared by every format section below and every other per-hardware cookbook; they now live once, privately, in rfdetr.export._benchmark: _artifact_size_mb reports an export artifact's size on disk, visualize_detections annotates and displays a supervision.Detections on the sample image, _fmt_ms/_result_row build the summary table's rows, measure_memory measures the host resident-memory growth of constructing a runtime and running its first inference call, and cpu_brand reads the host CPU's brand string for the Host cell below, reporting a placeholder instead of raising on any host without macOS's sysctl.

In [ ]:

from pathlib import Path

import numpy as np
from PIL import Image

from rfdetr.export._benchmark import (
    BenchmarkResult,
    _artifact_size_mb,
    _enable_notebook_inline_matplotlib,
    _result_row,
    cpu_brand,
    measure_latency,
    measure_memory,
    visualize_detections,
)

EXPORT_DIR = Path("export_mobile")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 5
MEASURE_RUNS = 30
BATCH_SIZES = (1, 4)

_enable_notebook_inline_matplotlib()

from pathlib import Path import numpy as np from PIL import Image from rfdetr.export._benchmark import ( BenchmarkResult, _artifact_size_mb, _enable_notebook_inline_matplotlib, _result_row, cpu_brand, measure_latency, measure_memory, visualize_detections, ) EXPORT_DIR = Path("export_mobile") EXPORT_DIR.mkdir(exist_ok=True) CONFIDENCE_THRESHOLD = 0.5 WARMUP_RUNS = 5 MEASURE_RUNS = 30 BATCH_SIZES = (1, 4) _enable_notebook_inline_matplotlib()

3. Host¶

The Config column and the narrative text in the Results section below are copied from this cell's printed output, not assumed — CPU architecture, core count, and installed rfdetr version all change what these numbers mean on a different machine.

In [ ]:

import importlib.metadata
import os
import platform

print(f"Machine: {platform.machine()}  ({cpu_brand()})")
print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})")
print(f"Cores: {os.cpu_count()} logical")
print(f"Python: {platform.python_version()}")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")

import importlib.metadata import os import platform print(f"Machine: {platform.machine()} ({cpu_brand()})") print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})") print(f"Cores: {os.cpu_count()} logical") print(f"Python: {platform.python_version()}") print(f"rfdetr: {importlib.metadata.version('rfdetr')}")

4. Sample image¶

A single street scene with several COCO classes (dog, person, backpack, car) is enough to verify detections. The image is downloaded once and reused for every format below.

In [ ]:

import urllib.request

IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")

import urllib.request IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg" IMAGE_PATH = EXPORT_DIR / "sample.jpg" if not IMAGE_PATH.exists(): urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH) image = Image.open(IMAGE_PATH).convert("RGB") print(f"Sample image: {image.size[0]}×{image.size[1]}")

5. PyTorch baseline — predict() / inference()¶

No export needed — this is the reference every on-device format in this notebook is compared against. It runs on this machine's CPU, not a phone or edge chip, so treat it as an accuracy and relative-speed anchor rather than a deployment number. predict() is the unoptimized baseline; inference() defaults to dtype=torch.float32 with compile_backend="torchscript" — a JIT-compiled, still-fp32 optimization; remove_optimized_model() reverts it afterward so model stays reusable for the export calls below.

Both calls include preprocessing and postprocessing — there is no separate "forward-only" path through eager predict(), so only an end-to-end number is reported for the PyTorch baseline. The memory bracket covers construction plus the first predict() call, since weights and any lazily built buffers aren't fully resident until after that first call; the JIT row's bracket covers inference() plus its own first call, reported as the increment on top of the eager row above it.

In [ ]:

from rfdetr import RFDETRSmall

with measure_memory() as mem:
    model = RFDETRSmall(device="cpu")
    baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_eager_memory_mb: dict[int, float] = {1: mem.delta_mb}
print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)

# predict() batches a list of images into one forward pass, so feeding it `[image] * batch`
# stands in for stacking a (batch, C, H, W) tensor -- see predict()'s docstring.
pytorch_eager: dict[int, BenchmarkResult] = {}
pytorch_jit: dict[int, BenchmarkResult] = {}
pytorch_jit_memory_mb: dict[int, float] = {}

for batch in BATCH_SIZES:
    batch_images = [image] * batch
    # A prior iteration's model.inference() call below leaves the model in optimized (JIT) mode,
    # and predict() then silently dispatches to that stale trace instead of the eager path -- it
    # raises a batch-size mismatch the moment a later batch differs from the traced one. Reset
    # before every iteration's eager measurement, not only once before the loop.
    model.remove_optimized_model()
    if batch != 1:  # batch 1's memory bracket is the construction + first predict() call above
        with measure_memory() as mem:
            model.predict(batch_images, threshold=CONFIDENCE_THRESHOLD)
        pytorch_eager_memory_mb[batch] = mem.delta_mb
    pytorch_eager[batch] = measure_latency(
        lambda imgs=batch_images: model.predict(imgs),
        label=f"PyTorch predict() b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    # inference(batch_size=batch) retraces the JIT model for each batch size; it calls
    # remove_optimized_model() itself on entry, so no manual reset is needed right here. The
    # retrace itself (not just the subsequent predict() call) belongs inside the memory bracket —
    # it is the JIT compile + first-call cost the column definition below promises.
    with measure_memory() as mem:
        model.inference(batch_size=batch)
        model.predict(batch_images)
    pytorch_jit_memory_mb[batch] = mem.delta_mb
    pytorch_jit[batch] = measure_latency(
        lambda imgs=batch_images: model.predict(imgs),
        label=f"PyTorch inference() JIT b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

model.remove_optimized_model()

for batch in BATCH_SIZES:
    for r, mb in (
        (pytorch_eager[batch], pytorch_eager_memory_mb.get(batch)),
        (pytorch_jit[batch], pytorch_jit_memory_mb[batch]),
    ):
        mb_str = f"+{mb:.1f} MB" if mb is not None else "(see b1 row)"
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)   {mb_str}")

from rfdetr import RFDETRSmall with measure_memory() as mem: model = RFDETRSmall(device="cpu") baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD) pytorch_eager_memory_mb: dict[int, float] = {1: mem.delta_mb} print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}") visualize_detections(baseline_detections, image) # predict() batches a list of images into one forward pass, so feeding it `[image] * batch` # stands in for stacking a (batch, C, H, W) tensor -- see predict()'s docstring. pytorch_eager: dict[int, BenchmarkResult] = {} pytorch_jit: dict[int, BenchmarkResult] = {} pytorch_jit_memory_mb: dict[int, float] = {} for batch in BATCH_SIZES: batch_images = [image] * batch # A prior iteration's model.inference() call below leaves the model in optimized (JIT) mode, # and predict() then silently dispatches to that stale trace instead of the eager path -- it # raises a batch-size mismatch the moment a later batch differs from the traced one. Reset # before every iteration's eager measurement, not only once before the loop. model.remove_optimized_model() if batch != 1: # batch 1's memory bracket is the construction + first predict() call above with measure_memory() as mem: model.predict(batch_images, threshold=CONFIDENCE_THRESHOLD) pytorch_eager_memory_mb[batch] = mem.delta_mb pytorch_eager[batch] = measure_latency( lambda imgs=batch_images: model.predict(imgs), label=f"PyTorch predict() b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) # inference(batch_size=batch) retraces the JIT model for each batch size; it calls # remove_optimized_model() itself on entry, so no manual reset is needed right here. The # retrace itself (not just the subsequent predict() call) belongs inside the memory bracket — # it is the JIT compile + first-call cost the column definition below promises. with measure_memory() as mem: model.inference(batch_size=batch) model.predict(batch_images) pytorch_jit_memory_mb[batch] = mem.delta_mb pytorch_jit[batch] = measure_latency( lambda imgs=batch_images: model.predict(imgs), label=f"PyTorch inference() JIT b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) model.remove_optimized_model() for batch in BATCH_SIZES: for r, mb in ( (pytorch_eager[batch], pytorch_eager_memory_mb.get(batch)), (pytorch_jit[batch], pytorch_jit_memory_mb[batch]), ): mb_str = f"+{mb:.1f} MB" if mb is not None else "(see b1 row)" print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS) {mb_str}")

6. TFLite (fp32, fp16, INT8 dynamic-range)¶

What it is. TFLite is TensorFlow's lightweight runtime for mobile, embedded, and edge hardware. RF-DETR's route converts ONNX → TensorFlow → .tflite via onnx2tf. Good for Android and embedded targets where TensorFlow Lite is already the established runtime; the tradeoff is an experimental multi-step conversion chain rather than a bit-exact match to eager PyTorch. See the TFLite export docs.

All three precisions come from one export call: onnx2tf always writes *_fp32.tflite and *_fp16.tflite, and quantization="int8" additionally builds a dynamic-range INT8 model (INT8 weights, float32 activations, no calibration data) and returns that as the primary path. So the three files below are siblings in one directory, benchmarked against each other with no re-export per precision.

dynamic_batch=True is refused for TFLite. The registry marks TFLite supports_dynamic_batch=False, so model.export() now raises NotImplementedError before conversion starts — onnx2tf (tf 2.19.1, onnx2tf 2.4.3) cannot lower the dynamic-batch graph: a position-embedding Add broadcasts a batch-1-shaped constant against the fully dynamic (None, None, 384) encoder input (ValueError: Dimensions must be equal, but are 384 and 1025). So each batch size below gets its own static export (batch_size=batch) instead of one dynamic-batch artifact resized at inference time.

Export¶

In [ ]:

from collections.abc import Callable

import tensorflow as tf
import torch

from rfdetr.export.benchmark import infer_transforms, post_process

# This cookbook is CPU-only by design (a phone/edge chip's CPU core, not a desktop GPU) -- confirm
# TensorFlow sees no GPU in this environment rather than assume it from the install command alone.
_tf_gpu_devices = tf.config.list_physical_devices("GPU")
print(f"TensorFlow GPU devices: {_tf_gpu_devices or 'none (CPU-only, as intended)'}")
assert not _tf_gpu_devices, "This cookbook must run CPU-only; found GPU devices."

tflite_int8_paths: dict[int, Path] = {}
tflite_fp32_paths: dict[int, Path] = {}
tflite_fp16_paths: dict[int, Path] = {}
for batch in BATCH_SIZES:
    int8_path = model.export(
        format="tflite",
        quantization="int8",
        batch_size=batch,
        output_name=f"rfdetr-small_b{batch}",
        output_dir=str(EXPORT_DIR),
    )
    stem = int8_path.name.removesuffix("_dynamic_range_quant.tflite")
    tflite_int8_paths[batch] = int8_path
    tflite_fp32_paths[batch] = int8_path.with_name(f"{stem}_fp32.tflite")
    tflite_fp16_paths[batch] = int8_path.with_name(f"{stem}_fp16.tflite")
    for label, path in (
        ("fp32", tflite_fp32_paths[batch]),
        ("fp16", tflite_fp16_paths[batch]),
        ("INT8 dyn-range", tflite_int8_paths[batch]),
    ):
        print(f"TFLite {label:<14} b{batch}  {path.name}  ({_artifact_size_mb(path):.1f} MB on disk)")

from collections.abc import Callable import tensorflow as tf import torch from rfdetr.export.benchmark import infer_transforms, post_process # This cookbook is CPU-only by design (a phone/edge chip's CPU core, not a desktop GPU) -- confirm # TensorFlow sees no GPU in this environment rather than assume it from the install command alone. _tf_gpu_devices = tf.config.list_physical_devices("GPU") print(f"TensorFlow GPU devices: {_tf_gpu_devices or 'none (CPU-only, as intended)'}") assert not _tf_gpu_devices, "This cookbook must run CPU-only; found GPU devices." tflite_int8_paths: dict[int, Path] = {} tflite_fp32_paths: dict[int, Path] = {} tflite_fp16_paths: dict[int, Path] = {} for batch in BATCH_SIZES: int8_path = model.export( format="tflite", quantization="int8", batch_size=batch, output_name=f"rfdetr-small_b{batch}", output_dir=str(EXPORT_DIR), ) stem = int8_path.name.removesuffix("_dynamic_range_quant.tflite") tflite_int8_paths[batch] = int8_path tflite_fp32_paths[batch] = int8_path.with_name(f"{stem}_fp32.tflite") tflite_fp16_paths[batch] = int8_path.with_name(f"{stem}_fp16.tflite") for label, path in ( ("fp32", tflite_fp32_paths[batch]), ("fp16", tflite_fp16_paths[batch]), ("INT8 dyn-range", tflite_int8_paths[batch]), ): print(f"TFLite {label:<14} b{batch} {path.name} ({_artifact_size_mb(path):.1f} MB on disk)")

Inference¶

The TFLite input is NHWC, not the NCHW layout the other formats in this notebook use. onnx2tf's SavedModel route also renames every output, so outputs are matched by rank and last dimension instead of by name: boxes are the rank-3 tensor with last dimension 4.

_tflite_runner builds one interpreter per (file, batch size) artifact and returns its forward callable plus the memory that interpreter cost — the measure_memory bracket covers construction and the first inference, since TFLite allocates lazily.

In [ ]:

resolution = model.model_config.resolution
tflite_transform = infer_transforms((resolution, resolution))


def _tflite_batch_input(batch: int) -> np.ndarray:
    """Stack *batch* copies of the sample image into one NHWC float32 array."""
    tensor, _ = tflite_transform(image, None)
    # TFLite expects NHWC, not the NCHW layout used by the other export formats.
    single = tensor.permute(1, 2, 0)[None].contiguous().numpy().astype(np.float32)
    return np.concatenate([single] * batch, axis=0)


def _tflite_runner(model_path: Path) -> tuple[Callable[[np.ndarray], tuple[torch.Tensor, torch.Tensor]], float]:
    """Build a TFLite interpreter for one (already batch-shaped) artifact; return its forward callable and memory
    cost."""
    with measure_memory() as mem:
        interpreter = tf.lite.Interpreter(model_path=str(model_path))
        interpreter.allocate_tensors()
        input_index = interpreter.get_input_details()[0]["index"]
        input_batch = interpreter.get_input_details()[0]["shape"][0]
        rank3 = [detail for detail in interpreter.get_output_details() if len(detail["shape"]) == 3]
        boxes_detail = next(detail for detail in rank3 if detail["shape"][-1] == 4)
        labels_detail = next(detail for detail in rank3 if detail["shape"][-1] != 4)

        def forward(inp: np.ndarray) -> tuple[torch.Tensor, torch.Tensor]:
            interpreter.set_tensor(input_index, inp)
            interpreter.invoke()
            dets = torch.from_numpy(interpreter.get_tensor(boxes_detail["index"]))
            labels = torch.from_numpy(interpreter.get_tensor(labels_detail["index"]))
            return dets, labels

        forward(_tflite_batch_input(int(input_batch)))  # first inference inside the bracket — TFLite allocates lazily
    return forward, mem.delta_mb


def _tflite_check(
    forward: Callable[[np.ndarray], tuple[torch.Tensor, torch.Tensor]], label: str, save_name: str
) -> None:
    """Run one batch-1 inference through *forward*, print the detection count, and save an annotated image."""
    dets, labels = forward(_tflite_batch_input(1))
    target_sizes = torch.tensor([[image.height, image.width]])
    result = post_process({"dets": dets, "labels": labels}, target_sizes)[0]
    keep = result["scores"] > CONFIDENCE_THRESHOLD
    print(f"TFLite {label}: {int(keep.sum())} detections above {CONFIDENCE_THRESHOLD}")
    detections = sv.Detections(
        xyxy=result["boxes"][keep].numpy(),
        confidence=result["scores"][keep].numpy(),
        class_id=result["labels"][keep].numpy().astype(int),
    )
    visualize_detections(detections, image, EXPORT_DIR / save_name)


def _tflite_end2end_thunk(
    forward: Callable[[np.ndarray], tuple[torch.Tensor, torch.Tensor]], batch: int
) -> Callable[[], None]:
    """Wrap *forward* into a zero-arg preprocess(batch) + forward + postprocess thunk for ``measure_latency``."""
    target_sizes = torch.tensor([[image.height, image.width]] * batch)

    def run() -> None:
        inp = _tflite_batch_input(batch)
        dets, labels = forward(inp)
        post_process({"dets": dets, "labels": labels}, target_sizes)

    return run


tflite_fp32_forward: dict[int, BenchmarkResult] = {}
tflite_fp32_end2end: dict[int, BenchmarkResult] = {}
tflite_fp32_memory_mb: dict[int, float] = {}
tflite_fp16_forward: dict[int, BenchmarkResult] = {}
tflite_fp16_end2end: dict[int, BenchmarkResult] = {}
tflite_fp16_memory_mb: dict[int, float] = {}
tflite_forward: dict[int, BenchmarkResult] = {}
tflite_end2end: dict[int, BenchmarkResult] = {}
tflite_memory_mb: dict[int, float] = {}

for batch in BATCH_SIZES:
    tflite_fp32_fn, tflite_fp32_memory_mb[batch] = _tflite_runner(tflite_fp32_paths[batch])
    tflite_fp16_fn, tflite_fp16_memory_mb[batch] = _tflite_runner(tflite_fp16_paths[batch])
    tflite_int8_fn, tflite_memory_mb[batch] = _tflite_runner(tflite_int8_paths[batch])

    if batch == 1:
        _tflite_check(tflite_fp32_fn, "fp32", "annotated_tflite_fp32.jpg")
        _tflite_check(tflite_fp16_fn, "fp16", "annotated_tflite_fp16.jpg")
        _tflite_check(tflite_int8_fn, "INT8", "annotated_tflite.jpg")

    tflite_fp32_forward[batch] = measure_latency(
        lambda inp=_tflite_batch_input(batch), fn=tflite_fp32_fn: fn(inp),
        label=f"TFLite fp32 forward b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    tflite_fp32_end2end[batch] = measure_latency(
        _tflite_end2end_thunk(tflite_fp32_fn, batch),
        label=f"TFLite fp32 end2end b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    tflite_fp16_forward[batch] = measure_latency(
        lambda inp=_tflite_batch_input(batch), fn=tflite_fp16_fn: fn(inp),
        label=f"TFLite fp16 forward b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    tflite_fp16_end2end[batch] = measure_latency(
        _tflite_end2end_thunk(tflite_fp16_fn, batch),
        label=f"TFLite fp16 end2end b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    tflite_forward[batch] = measure_latency(
        lambda inp=_tflite_batch_input(batch), fn=tflite_int8_fn: fn(inp),
        label=f"TFLite INT8 forward b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )
    tflite_end2end[batch] = measure_latency(
        _tflite_end2end_thunk(tflite_int8_fn, batch),
        label=f"TFLite INT8 end2end b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    for r in (
        tflite_fp32_forward[batch],
        tflite_fp32_end2end[batch],
        tflite_fp16_forward[batch],
        tflite_fp16_end2end[batch],
        tflite_forward[batch],
        tflite_end2end[batch],
    ):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

resolution = model.model_config.resolution tflite_transform = infer_transforms((resolution, resolution)) def _tflite_batch_input(batch: int) -> np.ndarray: """Stack *batch* copies of the sample image into one NHWC float32 array.""" tensor, _ = tflite_transform(image, None) # TFLite expects NHWC, not the NCHW layout used by the other export formats. single = tensor.permute(1, 2, 0)[None].contiguous().numpy().astype(np.float32) return np.concatenate([single] * batch, axis=0) def _tflite_runner(model_path: Path) -> tuple[Callable[[np.ndarray], tuple[torch.Tensor, torch.Tensor]], float]: """Build a TFLite interpreter for one (already batch-shaped) artifact; return its forward callable and memory cost.""" with measure_memory() as mem: interpreter = tf.lite.Interpreter(model_path=str(model_path)) interpreter.allocate_tensors() input_index = interpreter.get_input_details()[0]["index"] input_batch = interpreter.get_input_details()[0]["shape"][0] rank3 = [detail for detail in interpreter.get_output_details() if len(detail["shape"]) == 3] boxes_detail = next(detail for detail in rank3 if detail["shape"][-1] == 4) labels_detail = next(detail for detail in rank3 if detail["shape"][-1] != 4) def forward(inp: np.ndarray) -> tuple[torch.Tensor, torch.Tensor]: interpreter.set_tensor(input_index, inp) interpreter.invoke() dets = torch.from_numpy(interpreter.get_tensor(boxes_detail["index"])) labels = torch.from_numpy(interpreter.get_tensor(labels_detail["index"])) return dets, labels forward(_tflite_batch_input(int(input_batch))) # first inference inside the bracket — TFLite allocates lazily return forward, mem.delta_mb def _tflite_check( forward: Callable[[np.ndarray], tuple[torch.Tensor, torch.Tensor]], label: str, save_name: str ) -> None: """Run one batch-1 inference through *forward*, print the detection count, and save an annotated image.""" dets, labels = forward(_tflite_batch_input(1)) target_sizes = torch.tensor([[image.height, image.width]]) result = post_process({"dets": dets, "labels": labels}, target_sizes)[0] keep = result["scores"] > CONFIDENCE_THRESHOLD print(f"TFLite {label}: {int(keep.sum())} detections above {CONFIDENCE_THRESHOLD}") detections = sv.Detections( xyxy=result["boxes"][keep].numpy(), confidence=result["scores"][keep].numpy(), class_id=result["labels"][keep].numpy().astype(int), ) visualize_detections(detections, image, EXPORT_DIR / save_name) def _tflite_end2end_thunk( forward: Callable[[np.ndarray], tuple[torch.Tensor, torch.Tensor]], batch: int ) -> Callable[[], None]: """Wrap *forward* into a zero-arg preprocess(batch) + forward + postprocess thunk for ``measure_latency``.""" target_sizes = torch.tensor([[image.height, image.width]] * batch) def run() -> None: inp = _tflite_batch_input(batch) dets, labels = forward(inp) post_process({"dets": dets, "labels": labels}, target_sizes) return run tflite_fp32_forward: dict[int, BenchmarkResult] = {} tflite_fp32_end2end: dict[int, BenchmarkResult] = {} tflite_fp32_memory_mb: dict[int, float] = {} tflite_fp16_forward: dict[int, BenchmarkResult] = {} tflite_fp16_end2end: dict[int, BenchmarkResult] = {} tflite_fp16_memory_mb: dict[int, float] = {} tflite_forward: dict[int, BenchmarkResult] = {} tflite_end2end: dict[int, BenchmarkResult] = {} tflite_memory_mb: dict[int, float] = {} for batch in BATCH_SIZES: tflite_fp32_fn, tflite_fp32_memory_mb[batch] = _tflite_runner(tflite_fp32_paths[batch]) tflite_fp16_fn, tflite_fp16_memory_mb[batch] = _tflite_runner(tflite_fp16_paths[batch]) tflite_int8_fn, tflite_memory_mb[batch] = _tflite_runner(tflite_int8_paths[batch]) if batch == 1: _tflite_check(tflite_fp32_fn, "fp32", "annotated_tflite_fp32.jpg") _tflite_check(tflite_fp16_fn, "fp16", "annotated_tflite_fp16.jpg") _tflite_check(tflite_int8_fn, "INT8", "annotated_tflite.jpg") tflite_fp32_forward[batch] = measure_latency( lambda inp=_tflite_batch_input(batch), fn=tflite_fp32_fn: fn(inp), label=f"TFLite fp32 forward b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) tflite_fp32_end2end[batch] = measure_latency( _tflite_end2end_thunk(tflite_fp32_fn, batch), label=f"TFLite fp32 end2end b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) tflite_fp16_forward[batch] = measure_latency( lambda inp=_tflite_batch_input(batch), fn=tflite_fp16_fn: fn(inp), label=f"TFLite fp16 forward b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) tflite_fp16_end2end[batch] = measure_latency( _tflite_end2end_thunk(tflite_fp16_fn, batch), label=f"TFLite fp16 end2end b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) tflite_forward[batch] = measure_latency( lambda inp=_tflite_batch_input(batch), fn=tflite_int8_fn: fn(inp), label=f"TFLite INT8 forward b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) tflite_end2end[batch] = measure_latency( _tflite_end2end_thunk(tflite_int8_fn, batch), label=f"TFLite INT8 end2end b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) for r in ( tflite_fp32_forward[batch], tflite_fp32_end2end[batch], tflite_fp16_forward[batch], tflite_fp16_end2end[batch], tflite_forward[batch], tflite_end2end[batch], ): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

7. Results — Part A¶

Measured on this machine — see the Host cell's printed output above for the exact CPU/OS/Python/ rfdetr string; this notebook's own host stands in for a phone/edge chip's CPU core only approximately, and numbers vary by CPU generation — rerun the cells above to get yours. Config is the precision/layout used for that row; Batch is the batch size that row's .tflite artifact was exported (or, for PyTorch, re-traced) at; Memory [MB] is host resident-memory growth across constructing the runtime plus its first inference call — an approximate, single-process measurement, not an isolated per-format sandbox. TFLite refuses dynamic_batch=True with NotImplementedError before conversion starts (see the note in Section 6), so every TFLite row below is its own static per-batch artifact.

Measured on an Apple M4 Max (arm64), macOS 27.0, Python 3.12.14, rfdetr <VERSION>, RFDETRSmall, 5 warmup + 30 timed runs per row.

Note: the two PyTorch inference() JIT memory cells below are marked n/a (pending rerun) because the numbers measured there predate a fix to the memory bracket above — it used to open after model.inference() had already retraced and copied the model, so it measured only the subsequent predict() call, not the retrace itself. A rerun on this same machine is needed to regenerate real numbers for these two rows.

Format	Config	Batch	forward [ms]	end2end [ms]	FPS [img/s] (end2end)	Memory [MB]
PyTorch predict()	eager, fp32	1	—	55.63 ± 1.02	18.0	529.2
PyTorch inference()	JIT, fp32	1	—	54.93 ± 5.83	18.2	n/a (pending rerun)
TFLite	fp32, NHWC	1	797.93 ± 18.61	784.56 ± 5.09	1.3	174.1
TFLite	fp16, NHWC	1	787.83 ± 11.31	791.92 ± 3.38	1.3	251.2
TFLite	INT8 dyn-range, NHWC	1	321.81 ± 2.85	328.72 ± 8.12	3.0	135.5
PyTorch predict()	eager, fp32	4	—	154.45 ± 3.75	25.9	100.2
PyTorch inference()	JIT, fp32	4	—	147.45 ± 5.36	27.1	n/a (pending rerun)
TFLite	fp32, NHWC	4	3178.63 ± 51.98	3178.26 ± 22.20	1.3	768.1
TFLite	fp16, NHWC	4	3225.73 ± 54.01	3177.77 ± 18.71	1.3	442.6
TFLite	INT8 dyn-range, NHWC	4	1324.80 ± 14.17	1341.11 ± 20.03	3.0	98.4

The onnx2tf conversion path is dramatically slower than eager PyTorch on this machine, not faster. TFLite fp32 end2end at batch 1 (784.56 ms) is roughly 14× slower than PyTorch eager predict() (55.63 ms) on the same CPU — the reverse of what a "lightweight mobile runtime" name suggests. This is a property of the ONNX→TensorFlow→TFLite conversion chain on Apple Silicon's TFLite CPU kernels, not a representative phone number; a real ARM phone SoC's native TFLite delegate (NNAPI, Hexagon, etc.) is not exercised by this notebook at all — treat every TFLite row here as a same-machine ratio, never an absolute mobile-latency estimate.

fp16 buys nothing on this stack, and doesn't scale with batch. TFLite fp16's end2end time (791.92 ms at batch 1) is statistically indistinguishable from fp32 (784.56 ms) — the CPU kernel path upcasts back to fp32 for arithmetic, so the halved on-disk weight size (58.6 MB vs 115.6 MB) buys no latency. Both fp32 and fp16 scale linearly with batch (784.56 → 3178.26 ms, batch 1→4, ~4.05×) — FPS stays flat at 1.3 across both batch sizes, meaning this conversion path gets zero amortization benefit from batching, unlike PyTorch predict() above it (FPS actually rises 18.0 → 25.9 batch 1→4).

INT8 dynamic-range quantization is the one TFLite row worth taking seriously here — 2.4–2.5× faster than fp32/fp16 at both batch sizes (328.72 ms vs 784.56 ms at batch 1), at the cost of one detection dropping below the 0.5 confidence threshold (3 vs 4 for fp32/fp16) on this sample image. It is still ~4× slower than PyTorch eager on this machine, again a same-machine ratio, not a phone-hardware verdict — a phone's own INT8-aware delegate would likely close much of that gap.

In [ ]:

import pandas as pd
from IPython.display import display

summary_a_rows = []
for batch in BATCH_SIZES:
    summary_a_rows.append(
        _result_row(
            "PyTorch predict()", batch, "eager, fp32", None, pytorch_eager[batch], pytorch_eager_memory_mb.get(batch)
        )
    )
    summary_a_rows.append(
        _result_row("PyTorch inference()", batch, "JIT, fp32", None, pytorch_jit[batch], pytorch_jit_memory_mb[batch])
    )
    summary_a_rows.append(
        _result_row(
            "TFLite",
            batch,
            "fp32, NHWC",
            tflite_fp32_forward[batch],
            tflite_fp32_end2end[batch],
            tflite_fp32_memory_mb[batch],
        )
    )
    summary_a_rows.append(
        _result_row(
            "TFLite",
            batch,
            "fp16, NHWC",
            tflite_fp16_forward[batch],
            tflite_fp16_end2end[batch],
            tflite_fp16_memory_mb[batch],
        )
    )
    summary_a_rows.append(
        _result_row(
            "TFLite",
            batch,
            "INT8 dyn-range, NHWC",
            tflite_forward[batch],
            tflite_end2end[batch],
            tflite_memory_mb[batch],
        )
    )

summary_a = pd.DataFrame(summary_a_rows).set_index(["Format", "Batch"])
display(summary_a)
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, CPU (approximates a phone/edge chip's CPU core).")

import pandas as pd from IPython.display import display summary_a_rows = [] for batch in BATCH_SIZES: summary_a_rows.append( _result_row( "PyTorch predict()", batch, "eager, fp32", None, pytorch_eager[batch], pytorch_eager_memory_mb.get(batch) ) ) summary_a_rows.append( _result_row("PyTorch inference()", batch, "JIT, fp32", None, pytorch_jit[batch], pytorch_jit_memory_mb[batch]) ) summary_a_rows.append( _result_row( "TFLite", batch, "fp32, NHWC", tflite_fp32_forward[batch], tflite_fp32_end2end[batch], tflite_fp32_memory_mb[batch], ) ) summary_a_rows.append( _result_row( "TFLite", batch, "fp16, NHWC", tflite_fp16_forward[batch], tflite_fp16_end2end[batch], tflite_fp16_memory_mb[batch], ) ) summary_a_rows.append( _result_row( "TFLite", batch, "INT8 dyn-range, NHWC", tflite_forward[batch], tflite_end2end[batch], tflite_memory_mb[batch], ) ) summary_a = pd.DataFrame(summary_a_rows).set_index(["Format", "Batch"]) display(summary_a) print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, CPU (approximates a phone/edge chip's CPU core).")

Part B: LiteRT + ExecuTorch¶

Restart the runtime before this part. [litert] and [executorch] don't conflict with each other, but both conflict with [tflite] above — Colab: Runtime → Restart session, then run from the next cell. This part re-installs, re-imports, and re-downloads the sample image and PyTorch baseline independently of Part A.

8. Install¶

RFDETR_SPEC holds the install source and defaults to the develop branch, to pick up the newest export fixes; to install the 1.11.2 release from PyPI instead, swap in the commented >=1.11.2,<2 line. Colab ships mutually inconsistent preinstalled packages that otherwise crash import rfdetr, so two are aligned: torchaudio is uninstalled (RF-DETR never uses it, but transformers imports it when present and a torch/torchaudio CUDA-version mismatch then errors), and pillow is force-reinstalled to a clean version (a half-upgraded PIL breaks torchvision's import with cannot import name '_Ink').

flatc — ExecuTorch serializes the .pte with the FlatBuffers compiler. It ships in the Linux executorch wheel (so Colab works out of the box); on macOS install it separately with brew install flatbuffers.

executorch>=1.3.1,<1.4 is pinned deliberately. pyproject.toml's [executorch] extra is capped at >=1.3,<1.4 and says "validated on 1.3.1". One reason for the cap: executorch==1.5.1 raises RuntimeError: tensor does not have a device from executorch::extension::alias_tensor_ptr_to_attensor on every method.execute(...) call on this platform (confirmed with torch.randn, torch.zeros, and cloned/contiguous variants — all six failed identically, then all six succeeded after downgrading to 1.3.1 alone, with no other package changed). Keep the explicit executorch>=1.3.1,<1.4 range until this regression is confirmed fixed upstream and pyproject.toml's validated version is bumped.

In [ ]:

RFDETR_SPEC = "rfdetr[litert,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[litert,executorch]>=1.11.2,<2"
!pip install -q "{RFDETR_SPEC}" "torch<2.13" "executorch>=1.3.1,<1.4" psutil supervision pandas
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
!pip uninstall -q -y torchaudio

RFDETR_SPEC = "rfdetr[litert,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip" # RFDETR_SPEC = "rfdetr[litert,executorch]>=1.11.2,<2" !pip install -q "{RFDETR_SPEC}" "torch<2.13" "executorch>=1.3.1,<1.4" psutil supervision pandas !pip install -q --force-reinstall --no-deps "pillow==11.3.0" !pip uninstall -q -y torchaudio

9. Setup¶

Same shared helpers as Part A, from the same private package module — a fresh import, not a redefinition, since the runtime restart cleared Part A's own imports. (infer_transforms / post_process below are a genuine re-import too, not dead code — ruff's static analysis sees one continuous module and would otherwise treat it as redundant with Part A's, but at runtime Part A's cells never ran in this kernel.)

In [ ]:

from pathlib import Path

import numpy as np
import pandas as pd
import supervision as sv
from IPython.display import display
from PIL import Image

from rfdetr.export._benchmark import (
    BenchmarkResult,
    _artifact_size_mb,
    _enable_notebook_inline_matplotlib,
    _result_row,
    cpu_brand,
    measure_latency,
    measure_memory,
    visualize_detections,
)
from rfdetr.export.benchmark import (  # noqa: F811 -- Part B's own kernel, not a re-import of Part A's
    infer_transforms,
    post_process,
)

EXPORT_DIR = Path("export_mobile")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 5
MEASURE_RUNS = 30
BATCH_SIZES = (1, 4)

_enable_notebook_inline_matplotlib()

from pathlib import Path import numpy as np import pandas as pd import supervision as sv from IPython.display import display from PIL import Image from rfdetr.export._benchmark import ( BenchmarkResult, _artifact_size_mb, _enable_notebook_inline_matplotlib, _result_row, cpu_brand, measure_latency, measure_memory, visualize_detections, ) from rfdetr.export.benchmark import ( # noqa: F811 -- Part B's own kernel, not a re-import of Part A's infer_transforms, post_process, ) EXPORT_DIR = Path("export_mobile") EXPORT_DIR.mkdir(exist_ok=True) CONFIDENCE_THRESHOLD = 0.5 WARMUP_RUNS = 5 MEASURE_RUNS = 30 BATCH_SIZES = (1, 4) _enable_notebook_inline_matplotlib()

10. Host¶

Same measurement as Part A's Host cell, re-run in this fresh runtime — Part A and Part B run in separate venvs (see the notebook intro), so the installed rfdetr version could in principle differ between them even though both target the same CPU.

In [ ]:

import importlib.metadata
import os
import platform

print(f"Machine: {platform.machine()}  ({cpu_brand()})")
print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})")
print(f"Cores: {os.cpu_count()} logical")
print(f"Python: {platform.python_version()}")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")

import importlib.metadata import os import platform print(f"Machine: {platform.machine()} ({cpu_brand()})") print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})") print(f"Cores: {os.cpu_count()} logical") print(f"Python: {platform.python_version()}") print(f"rfdetr: {importlib.metadata.version('rfdetr')}")

11. Sample image¶

In [ ]:

import urllib.request

IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")

import urllib.request IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg" IMAGE_PATH = EXPORT_DIR / "sample.jpg" if not IMAGE_PATH.exists(): urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH) image = Image.open(IMAGE_PATH).convert("RGB") print(f"Sample image: {image.size[0]}×{image.size[1]}")

12. PyTorch baseline — predict() / inference()¶

Same reference anchor as Part A, re-measured in this fresh runtime.

In [ ]:

from rfdetr import RFDETRSmall

with measure_memory() as mem:
    model = RFDETRSmall(device="cpu")
    baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_eager_memory_mb: dict[int, float] = {1: mem.delta_mb}
print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)

# predict() batches a list of images into one forward pass, so feeding it `[image] * batch`
# stands in for stacking a (batch, C, H, W) tensor -- see predict()'s docstring.
pytorch_eager: dict[int, BenchmarkResult] = {}
pytorch_jit: dict[int, BenchmarkResult] = {}
pytorch_jit_memory_mb: dict[int, float] = {}

for batch in BATCH_SIZES:
    batch_images = [image] * batch
    # A prior iteration's model.inference() call below leaves the model in optimized (JIT) mode,
    # and predict() then silently dispatches to that stale trace instead of the eager path -- it
    # raises a batch-size mismatch the moment a later batch differs from the traced one. Reset
    # before every iteration's eager measurement, not only once before the loop.
    model.remove_optimized_model()
    if batch != 1:  # batch 1's memory bracket is the construction + first predict() call above
        with measure_memory() as mem:
            model.predict(batch_images, threshold=CONFIDENCE_THRESHOLD)
        pytorch_eager_memory_mb[batch] = mem.delta_mb
    pytorch_eager[batch] = measure_latency(
        lambda imgs=batch_images: model.predict(imgs),
        label=f"PyTorch predict() b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    # inference(batch_size=batch) retraces the JIT model for each batch size; it calls
    # remove_optimized_model() itself on entry, so no manual reset is needed right here. The
    # retrace itself (not just the subsequent predict() call) belongs inside the memory bracket —
    # it is the JIT compile + first-call cost the column definition below promises.
    with measure_memory() as mem:
        model.inference(batch_size=batch)
        model.predict(batch_images)
    pytorch_jit_memory_mb[batch] = mem.delta_mb
    pytorch_jit[batch] = measure_latency(
        lambda imgs=batch_images: model.predict(imgs),
        label=f"PyTorch inference() JIT b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

model.remove_optimized_model()

for batch in BATCH_SIZES:
    for r, mb in (
        (pytorch_eager[batch], pytorch_eager_memory_mb.get(batch)),
        (pytorch_jit[batch], pytorch_jit_memory_mb[batch]),
    ):
        mb_str = f"+{mb:.1f} MB" if mb is not None else "(see b1 row)"
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)   {mb_str}")

from rfdetr import RFDETRSmall with measure_memory() as mem: model = RFDETRSmall(device="cpu") baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD) pytorch_eager_memory_mb: dict[int, float] = {1: mem.delta_mb} print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}") visualize_detections(baseline_detections, image) # predict() batches a list of images into one forward pass, so feeding it `[image] * batch` # stands in for stacking a (batch, C, H, W) tensor -- see predict()'s docstring. pytorch_eager: dict[int, BenchmarkResult] = {} pytorch_jit: dict[int, BenchmarkResult] = {} pytorch_jit_memory_mb: dict[int, float] = {} for batch in BATCH_SIZES: batch_images = [image] * batch # A prior iteration's model.inference() call below leaves the model in optimized (JIT) mode, # and predict() then silently dispatches to that stale trace instead of the eager path -- it # raises a batch-size mismatch the moment a later batch differs from the traced one. Reset # before every iteration's eager measurement, not only once before the loop. model.remove_optimized_model() if batch != 1: # batch 1's memory bracket is the construction + first predict() call above with measure_memory() as mem: model.predict(batch_images, threshold=CONFIDENCE_THRESHOLD) pytorch_eager_memory_mb[batch] = mem.delta_mb pytorch_eager[batch] = measure_latency( lambda imgs=batch_images: model.predict(imgs), label=f"PyTorch predict() b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) # inference(batch_size=batch) retraces the JIT model for each batch size; it calls # remove_optimized_model() itself on entry, so no manual reset is needed right here. The # retrace itself (not just the subsequent predict() call) belongs inside the memory bracket — # it is the JIT compile + first-call cost the column definition below promises. with measure_memory() as mem: model.inference(batch_size=batch) model.predict(batch_images) pytorch_jit_memory_mb[batch] = mem.delta_mb pytorch_jit[batch] = measure_latency( lambda imgs=batch_images: model.predict(imgs), label=f"PyTorch inference() JIT b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) model.remove_optimized_model() for batch in BATCH_SIZES: for r, mb in ( (pytorch_eager[batch], pytorch_eager_memory_mb.get(batch)), (pytorch_jit[batch], pytorch_jit_memory_mb[batch]), ): mb_str = f"+{mb:.1f} MB" if mb is not None else "(see b1 row)" print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS) {mb_str}")

13. LiteRT¶

What it is. LiteRT (formerly TensorFlow Lite) is Google's on-device runtime. RF-DETR exports to it directly via torch.export (litert-torch) — no ONNX step, no TensorFlow step. Good for the same on-device deployment target as TFLite in Part A, with a more direct conversion path that tracks eager PyTorch more closely; the tradeoffs are fp32-only, a fixed batch size, and no keypoint model support on the current litert-torch version. See the LiteRT export docs.

Export¶

format="litert" hands the model to litert-torch, which captures it with torch.export and lowers straight to a .tflite — no ONNX, no TensorFlow.

In [ ]:

from collections.abc import Callable

from rfdetr.export._runtime.decode import decode_detections
from rfdetr.export._runtime.preprocess import preprocess_to_nchw

# LiteRT bakes a fixed batch shape into the .tflite (dynamic_batch is unsupported -- see
# registry.py), so each batch size in BATCH_SIZES gets its own export; output_name keeps the
# files from overwriting each other.
resolution = model.model_config.resolution
litert_paths: dict[int, Path] = {}
for batch in BATCH_SIZES:
    litert_paths[batch] = model.export(
        format="litert", batch_size=batch, output_name=f"rfdetr-small_litert_b{batch}", output_dir=str(EXPORT_DIR)
    )
    print(f"LiteRT b{batch}: {litert_paths[batch].name}  ({_artifact_size_mb(litert_paths[batch]):.1f} MB on disk)")

from collections.abc import Callable from rfdetr.export._runtime.decode import decode_detections from rfdetr.export._runtime.preprocess import preprocess_to_nchw # LiteRT bakes a fixed batch shape into the .tflite (dynamic_batch is unsupported -- see # registry.py), so each batch size in BATCH_SIZES gets its own export; output_name keeps the # files from overwriting each other. resolution = model.model_config.resolution litert_paths: dict[int, Path] = {} for batch in BATCH_SIZES: litert_paths[batch] = model.export( format="litert", batch_size=batch, output_name=f"rfdetr-small_litert_b{batch}", output_dir=str(EXPORT_DIR) ) print(f"LiteRT b{batch}: {litert_paths[batch].name} ({_artifact_size_mb(litert_paths[batch]):.1f} MB on disk)")

Inference¶

Unlike the TFLite route in Part A, LiteRT keeps PyTorch's NCHW layout, and its outputs are matched by position (boxes first, logits second), not by rank or name. decode_detections is single-image, so a batched forward call is decoded with one call per item in the batch.

In [ ]:

from ai_edge_litert.interpreter import Interpreter


def _litert_runner(
    model_path: Path, batch: int
) -> tuple[Callable[[np.ndarray], tuple[np.ndarray, np.ndarray]], np.ndarray, float, int, int, int]:
    """Build a LiteRT interpreter for one (file, batch) pair.

    Returns the raw forward callable (takes a caller-supplied NCHW array), a ready-made *batch*-sized sample input, the
    construction memory cost, and the model's ``(height, width, channels)``.
    """
    with measure_memory() as mem:
        interpreter = Interpreter(model_path=str(model_path))
        interpreter.allocate_tensors()
        (input_detail,) = interpreter.get_input_details()
        _, channels, height, width = input_detail["shape"]
        output_details = interpreter.get_output_details()[:2]

        def raw_forward(inp: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
            interpreter.set_tensor(input_detail["index"], inp)
            interpreter.invoke()
            boxes, logits = (interpreter.get_tensor(d["index"]) for d in output_details)
            return boxes, logits

        sample_input = np.concatenate([preprocess_to_nchw(image, height, width, channels)] * batch, axis=0)
        raw_forward(sample_input)  # first inference inside the bracket — LiteRT allocates lazily
    return raw_forward, sample_input, mem.delta_mb, height, width, channels


litert_forward: dict[int, BenchmarkResult] = {}
litert_end2end: dict[int, BenchmarkResult] = {}
litert_memory_mb: dict[int, float] = {}

for batch in BATCH_SIZES:
    litert_fn, litert_sample, litert_memory_mb[batch], litert_h, litert_w, litert_c = _litert_runner(
        litert_paths[batch], batch
    )

    if batch == 1:
        boxes, logits = litert_fn(litert_sample)
        litert_decoded = decode_detections(boxes[0], logits[0], image.size, threshold=CONFIDENCE_THRESHOLD)
        print(f"LiteRT: {len(litert_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}")
        litert_sv_detections = sv.Detections(
            xyxy=litert_decoded.xyxy, confidence=litert_decoded.confidence, class_id=litert_decoded.class_id.astype(int)
        )
        visualize_detections(litert_sv_detections, image, EXPORT_DIR / "annotated_litert.jpg")

    litert_forward[batch] = measure_latency(
        lambda inp=litert_sample, fn=litert_fn: fn(inp),
        label=f"LiteRT forward b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    def _litert_end2end(
        batch: int = batch, fn: Callable = litert_fn, h: int = litert_h, w: int = litert_w, c: int = litert_c
    ) -> None:
        inp = np.concatenate([preprocess_to_nchw(image, h, w, c)] * batch, axis=0)
        boxes, logits = fn(inp)
        for i in range(batch):
            decode_detections(boxes[i], logits[i], image.size, threshold=CONFIDENCE_THRESHOLD)

    litert_end2end[batch] = measure_latency(
        _litert_end2end, label=f"LiteRT end2end b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS
    )

    for r in (litert_forward[batch], litert_end2end[batch]):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

from ai_edge_litert.interpreter import Interpreter def _litert_runner( model_path: Path, batch: int ) -> tuple[Callable[[np.ndarray], tuple[np.ndarray, np.ndarray]], np.ndarray, float, int, int, int]: """Build a LiteRT interpreter for one (file, batch) pair. Returns the raw forward callable (takes a caller-supplied NCHW array), a ready-made *batch*-sized sample input, the construction memory cost, and the model's ``(height, width, channels)``. """ with measure_memory() as mem: interpreter = Interpreter(model_path=str(model_path)) interpreter.allocate_tensors() (input_detail,) = interpreter.get_input_details() _, channels, height, width = input_detail["shape"] output_details = interpreter.get_output_details()[:2] def raw_forward(inp: np.ndarray) -> tuple[np.ndarray, np.ndarray]: interpreter.set_tensor(input_detail["index"], inp) interpreter.invoke() boxes, logits = (interpreter.get_tensor(d["index"]) for d in output_details) return boxes, logits sample_input = np.concatenate([preprocess_to_nchw(image, height, width, channels)] * batch, axis=0) raw_forward(sample_input) # first inference inside the bracket — LiteRT allocates lazily return raw_forward, sample_input, mem.delta_mb, height, width, channels litert_forward: dict[int, BenchmarkResult] = {} litert_end2end: dict[int, BenchmarkResult] = {} litert_memory_mb: dict[int, float] = {} for batch in BATCH_SIZES: litert_fn, litert_sample, litert_memory_mb[batch], litert_h, litert_w, litert_c = _litert_runner( litert_paths[batch], batch ) if batch == 1: boxes, logits = litert_fn(litert_sample) litert_decoded = decode_detections(boxes[0], logits[0], image.size, threshold=CONFIDENCE_THRESHOLD) print(f"LiteRT: {len(litert_decoded.xyxy)} detections above {CONFIDENCE_THRESHOLD}") litert_sv_detections = sv.Detections( xyxy=litert_decoded.xyxy, confidence=litert_decoded.confidence, class_id=litert_decoded.class_id.astype(int) ) visualize_detections(litert_sv_detections, image, EXPORT_DIR / "annotated_litert.jpg") litert_forward[batch] = measure_latency( lambda inp=litert_sample, fn=litert_fn: fn(inp), label=f"LiteRT forward b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) def _litert_end2end( batch: int = batch, fn: Callable = litert_fn, h: int = litert_h, w: int = litert_w, c: int = litert_c ) -> None: inp = np.concatenate([preprocess_to_nchw(image, h, w, c)] * batch, axis=0) boxes, logits = fn(inp) for i in range(batch): decode_detections(boxes[i], logits[i], image.size, threshold=CONFIDENCE_THRESHOLD) litert_end2end[batch] = measure_latency( _litert_end2end, label=f"LiteRT end2end b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS ) for r in (litert_forward[batch], litert_end2end[batch]): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

14. ExecuTorch (XNNPACK backend)¶

What it is. ExecuTorch is PyTorch's own on-device inference runtime — the model is exported directly via torch.export to a portable .pte binary, no ONNX conversion involved. The "xnnpack" backend targets any CPU platform in fp32. Good for on-device PyTorch deployment (Android, iOS, embedded) when staying inside the PyTorch ecosystem end to end matters more than picking the single fastest runtime for one platform. See the ExecuTorch export docs.

Export¶

format="executorch" captures the model with torch.export (no intermediate conversion) and lowers it to the requested backend — xnnpack here, RF-DETR's portable-CPU backend that also runs on Android and iOS.

In [ ]:

import torch

# ExecuTorch also bakes a fixed batch shape (dynamic_batch raises NotImplementedError on this
# executorch version -- see registry.py), so one .pte is exported per batch size.
executorch_paths: dict[int, Path] = {}
for batch in BATCH_SIZES:
    executorch_paths[batch] = model.export(
        format="executorch",
        backend="xnnpack",
        batch_size=batch,
        output_name=f"rfdetr-small_executorch_b{batch}",
        output_dir=str(EXPORT_DIR),
    )
    size_mb = _artifact_size_mb(executorch_paths[batch])
    print(f"ExecuTorch b{batch}: {executorch_paths[batch].name}  ({size_mb:.1f} MB on disk)")

import torch # ExecuTorch also bakes a fixed batch shape (dynamic_batch raises NotImplementedError on this # executorch version -- see registry.py), so one .pte is exported per batch size. executorch_paths: dict[int, Path] = {} for batch in BATCH_SIZES: executorch_paths[batch] = model.export( format="executorch", backend="xnnpack", batch_size=batch, output_name=f"rfdetr-small_executorch_b{batch}", output_dir=str(EXPORT_DIR), ) size_mb = _artifact_size_mb(executorch_paths[batch]) print(f"ExecuTorch b{batch}: {executorch_paths[batch].name} ({size_mb:.1f} MB on disk)")

Inference¶

Runtime.load_program(...).load_method("forward") gives a callable graph; its two outputs are dets (boxes, normalized cxcywh) and labels (class logits).

The input tensor must be contiguous. The ExecuTorch runtime reads the input buffer as contiguous NCHW and ignores tensor strides — a preprocessing step that permutes axes without copying returns a strided view the runtime misreads as a scrambled image. Nothing errors: the model runs and returns plausible-shaped output, but every detection's score collapses below threshold. infer_transforms already materializes a contiguous tensor; the trailing .contiguous() below is defensive.

Every call to method.execute() needs a freshly .clone()d tensor. The export-mode forward mutates its input in place (see tests/export/test_executorch_export.py's _executorch_runtime_tensors), and on this XNNPACK build, feeding the same tensor object to execute() a second time — or even the first time, for a tensor built by torch.cat(...) rather than freshly materialized — raises RuntimeError: tensor does not have a device from ExecuTorch's own tensor-aliasing code. .clone() immediately before every execute() call below, including inside the repeated-call benchmark thunks, avoids it.

In [ ]:

from rfdetr.export._executorch.inference import load_executorch_method

executorch_transform = infer_transforms((resolution, resolution))


def _executorch_batch_pixel_values(batch: int) -> torch.Tensor:
    """Stack *batch* copies of the sample image into one contiguous NCHW tensor."""
    tensor, _ = executorch_transform(image, None)
    return torch.cat([tensor[None]] * batch, dim=0).float().contiguous()


executorch_forward: dict[int, BenchmarkResult] = {}
executorch_end2end: dict[int, BenchmarkResult] = {}
executorch_memory_mb: dict[int, float] = {}

for batch in BATCH_SIZES:
    pixel_values = _executorch_batch_pixel_values(batch)
    with measure_memory() as mem:
        method = load_executorch_method(executorch_paths[batch])
        dets, labels = method.execute([pixel_values.clone()])
    executorch_memory_mb[batch] = mem.delta_mb

    if batch == 1:
        target_sizes = torch.tensor([[image.height, image.width]])
        result = post_process({"dets": dets, "labels": labels}, target_sizes)[0]
        keep = result["scores"] > CONFIDENCE_THRESHOLD
        print(f"ExecuTorch (XNNPACK): {int(keep.sum())} detections above {CONFIDENCE_THRESHOLD}")
        executorch_sv_detections = sv.Detections(
            xyxy=result["boxes"][keep].numpy(),
            confidence=result["scores"][keep].numpy(),
            class_id=result["labels"][keep].numpy().astype(int),
        )
        visualize_detections(executorch_sv_detections, image, EXPORT_DIR / "annotated_executorch.jpg")

    executorch_forward[batch] = measure_latency(
        lambda m=method, p=pixel_values: m.execute([p.clone()]),
        label=f"ExecuTorch (XNNPACK) forward b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    def _executorch_end2end(batch: int = batch, m=method) -> None:
        pv = _executorch_batch_pixel_values(batch)
        dets, labels = m.execute([pv.clone()])
        target_sizes = torch.tensor([[image.height, image.width]] * batch)
        post_process({"dets": dets, "labels": labels}, target_sizes)

    executorch_end2end[batch] = measure_latency(
        _executorch_end2end,
        label=f"ExecuTorch (XNNPACK) end2end b{batch}",
        device="cpu",
        warmup=WARMUP_RUNS,
        runs=MEASURE_RUNS,
    )

    for r in (executorch_forward[batch], executorch_end2end[batch]):
        print(f"  {r.label:<32}  {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f}   ({r.fps:6.1f} FPS)")

from rfdetr.export._executorch.inference import load_executorch_method executorch_transform = infer_transforms((resolution, resolution)) def _executorch_batch_pixel_values(batch: int) -> torch.Tensor: """Stack *batch* copies of the sample image into one contiguous NCHW tensor.""" tensor, _ = executorch_transform(image, None) return torch.cat([tensor[None]] * batch, dim=0).float().contiguous() executorch_forward: dict[int, BenchmarkResult] = {} executorch_end2end: dict[int, BenchmarkResult] = {} executorch_memory_mb: dict[int, float] = {} for batch in BATCH_SIZES: pixel_values = _executorch_batch_pixel_values(batch) with measure_memory() as mem: method = load_executorch_method(executorch_paths[batch]) dets, labels = method.execute([pixel_values.clone()]) executorch_memory_mb[batch] = mem.delta_mb if batch == 1: target_sizes = torch.tensor([[image.height, image.width]]) result = post_process({"dets": dets, "labels": labels}, target_sizes)[0] keep = result["scores"] > CONFIDENCE_THRESHOLD print(f"ExecuTorch (XNNPACK): {int(keep.sum())} detections above {CONFIDENCE_THRESHOLD}") executorch_sv_detections = sv.Detections( xyxy=result["boxes"][keep].numpy(), confidence=result["scores"][keep].numpy(), class_id=result["labels"][keep].numpy().astype(int), ) visualize_detections(executorch_sv_detections, image, EXPORT_DIR / "annotated_executorch.jpg") executorch_forward[batch] = measure_latency( lambda m=method, p=pixel_values: m.execute([p.clone()]), label=f"ExecuTorch (XNNPACK) forward b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) def _executorch_end2end(batch: int = batch, m=method) -> None: pv = _executorch_batch_pixel_values(batch) dets, labels = m.execute([pv.clone()]) target_sizes = torch.tensor([[image.height, image.width]] * batch) post_process({"dets": dets, "labels": labels}, target_sizes) executorch_end2end[batch] = measure_latency( _executorch_end2end, label=f"ExecuTorch (XNNPACK) end2end b{batch}", device="cpu", warmup=WARMUP_RUNS, runs=MEASURE_RUNS, ) for r in (executorch_forward[batch], executorch_end2end[batch]): print(f" {r.label:<32} {r.mean_ms:6.2f} ms ± {r.std_ms:5.2f} ({r.fps:6.1f} FPS)")

15. Results — Part B¶

Measured on this machine — see the Host cell's printed output above for the exact CPU/OS/Python/ rfdetr string; this notebook's own host stands in for a phone/edge chip's CPU core only approximately, and numbers vary by CPU generation — rerun the cells above to get yours. Batch is the batch size that row's artifact was exported (or, for PyTorch, re-traced) at — both LiteRT and ExecuTorch bake a fixed batch shape, so each batch size below is its own export. Memory [MB] is host resident-memory growth across constructing the runtime plus its first inference call.

Measured on an Apple M4 Max (arm64), macOS 27.0, Python 3.12.14, rfdetr develop (version not recorded), RFDETRSmall, 5 warmup + 30 timed runs per row (executorch==1.3.1 — see the note in Section 8).

Note: the two PyTorch inference() JIT memory cells below are marked n/a (pending rerun) because the numbers measured there predate the same memory-bracket fix noted in Part A's Results section above — rerun pending.

Format	Config	Batch	forward [ms]	end2end [ms]	FPS [img/s] (end2end)	Memory [MB]
PyTorch predict()	eager, fp32	1	—	54.54 ± 0.71	18.3	587.3
PyTorch inference()	JIT, fp32	1	—	59.37 ± 5.07	16.8	n/a (pending rerun)
LiteRT	fp32, NCHW	1	137.14 ± 3.00	138.04 ± 2.72	7.2	213.2
ExecuTorch	XNNPACK, fp32	1	81.68 ± 3.31	80.74 ± 1.90	12.4	140.2
PyTorch predict()	eager, fp32	4	—	156.72 ± 4.14	25.5	97.6
PyTorch inference()	JIT, fp32	4	—	152.51 ± 10.27	26.2	n/a (pending rerun)
LiteRT	fp32, NCHW	4	557.56 ± 10.95	557.93 ± 9.50	7.2	387.4
ExecuTorch	XNNPACK, fp32	4	313.56 ± 3.24	318.27 ± 6.67	12.6	1075.7

ExecuTorch (XNNPACK) is the fastest export format in this notebook, by a wide margin. At batch 1 it's 80.74 ms end2end vs LiteRT's 138.04 ms — a real gap, not conversion noise — and it holds that ~1.7× lead at batch 4 too (318.27 ms vs 557.93 ms). Set against Part A's TFLite rows on the same machine, ExecuTorch is roughly 10× faster than TFLite fp32/fp16 (784.56 ms at batch 1) and still clearly ahead of TFLite's INT8 dynamic-range row (328.72 ms). Of the three mobile/edge formats this notebook covers, torch.export-based paths (ExecuTorch, LiteRT) beat the ONNX→TensorFlow→TFLite conversion chain on this CPU, and staying inside the ExecuTorch runtime specifically beats LiteRT's own direct torch.export route too.

Neither export format beats eager PyTorch here. PyTorch predict() (54.54 ms) is still faster end2end than ExecuTorch (80.74 ms) at batch 1 on this specific CPU — export formats exist for portability and on-device deployment (no Python/PyTorch runtime required), not because they out-run PyTorch eager on the machine that exported them; a phone's own NPU/DSP delegate is the case where ExecuTorch or LiteRT would actually win.

ExecuTorch's Memory [MB] jumps sharply at batch 4 (1075.7 MB vs 140.2 MB at batch 1) — unlike PyTorch (one model reused across batches) or LiteRT, ExecuTorch bakes a fixed batch shape per .pte, so each batch size in the loop above calls load_executorch_method(...) fresh; the 1075.7 MB reflects a full second program load in the same process, not a leak. Every runtime here shares one process, so read Memory [MB] as a rough per-row indication, not an isolated sandbox.

In [ ]:

summary_b_rows = []
for batch in BATCH_SIZES:
    summary_b_rows.append(
        _result_row(
            "PyTorch predict()", batch, "eager, fp32", None, pytorch_eager[batch], pytorch_eager_memory_mb.get(batch)
        )
    )
    summary_b_rows.append(
        _result_row("PyTorch inference()", batch, "JIT, fp32", None, pytorch_jit[batch], pytorch_jit_memory_mb[batch])
    )
    summary_b_rows.append(
        _result_row(
            "LiteRT", batch, "fp32, NCHW", litert_forward[batch], litert_end2end[batch], litert_memory_mb[batch]
        )
    )
    summary_b_rows.append(
        _result_row(
            "ExecuTorch",
            batch,
            "XNNPACK, fp32",
            executorch_forward[batch],
            executorch_end2end[batch],
            executorch_memory_mb[batch],
        )
    )

summary_b = pd.DataFrame(summary_b_rows).set_index(["Format", "Batch"])
display(summary_b)
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, CPU (approximates a phone/edge chip's CPU core).")

summary_b_rows = [] for batch in BATCH_SIZES: summary_b_rows.append( _result_row( "PyTorch predict()", batch, "eager, fp32", None, pytorch_eager[batch], pytorch_eager_memory_mb.get(batch) ) ) summary_b_rows.append( _result_row("PyTorch inference()", batch, "JIT, fp32", None, pytorch_jit[batch], pytorch_jit_memory_mb[batch]) ) summary_b_rows.append( _result_row( "LiteRT", batch, "fp32, NCHW", litert_forward[batch], litert_end2end[batch], litert_memory_mb[batch] ) ) summary_b_rows.append( _result_row( "ExecuTorch", batch, "XNNPACK, fp32", executorch_forward[batch], executorch_end2end[batch], executorch_memory_mb[batch], ) ) summary_b = pd.DataFrame(summary_b_rows).set_index(["Format", "Batch"]) display(summary_b) print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, CPU (approximates a phone/edge chip's CPU core).")

Next steps¶

• Fine-tuned weights — pass pretrain_weights="<path/to/checkpoint.pth>" when constructing the model.
• Deploy on-device — copy the .tflite / .pte file to your Android / iOS / edge app and run it with that platform's TensorFlow Lite, LiteRT, or ExecuTorch runtime.
• Apple Silicon — ExecuTorch's coreml backend (Apple Neural Engine, fp16) and native CoreML / Core AI export are covered in the Apple cookbook, not here.
• Qualcomm Snapdragon — ExecuTorch's qnn backend needs a source build against the QAIRT SDK; see the ExecuTorch export docs.
• Have a CUDA GPU or a general desktop/server CPU? — see the CUDA cookbook or the CPU cookbook.
• See the Export documentation for every format and option.