RF-DETR on Mobile & Edge — Export, Inference & Latency¶
Compares every export path built for on-device / ARM mobile and edge deployment: eager
PyTorch (running here as the reference anchor, not itself a deployment target) against TFLite,
LiteRT, and ExecuTorch's XNNPACK backend — each exported, run once for a correctness check, then
benchmarked on the CPU that also stands in for a phone/edge chip's CPU core. Every exported artifact is loaded through the one public prediction API, RFDETRInference(path), and timed through the same predict() call as the PyTorch rows.
| Format | Export | Inference |
|---|---|---|
| PyTorch | (no export — eager model) | predict() / inference() (TorchScript JIT, fp32) |
| TFLite | ONNX → TensorFlow → .tflite (onnx2tf) |
RFDETRInference(path).predict() |
| LiteRT | torch.export → .tflite directly (litert-torch) |
RFDETRInference(path).predict() |
| ExecuTorch (XNNPACK) | torch.export → .pte |
RFDETRInference(path).predict() |
Not covered here: NVIDIA GPU deployment (see the CUDA cookbook), general-purpose desktop/server CPU via ONNX Runtime or OpenVINO (see the CPU cookbook), or Apple Silicon — native CoreML, Core AI, and ExecuTorch's CoreML backend (see the Apple cookbook).
Python version. The
[tflite]extra installs only on Python 3.12 exactly — itsonnx2tf/tensorflowdependency stack pinsnumpy==1.26.4, which cannot be satisfied on 3.10, 3.11, 3.13, or 3.14 (seepyproject.toml). This constraint is why TFLite/LiteRT/ExecuTorch get their own notebook, separate from the CPU cookbook's ONNX/OpenVINO, which have no such pin.
Two parts, two environments.
pyproject.toml's[tool.uv] conflictsdeclares[tflite]unsatisfiable alongside both[litert](onnx2tfpinsai-edge-litert==2.1.2;litert-torchneeds>=2.2.0) and[executorch]— the three formats cannot share one Python environment. This notebook is split into Part A (TFLite) and Part B (LiteRT + ExecuTorch), each with its own install cell; run one part, then restart the runtime before running the other. Each part re-establishes its own PyTorch baseline and its own results table rather than assuming state survives the restart.
Which latency number is this? Every number in this notebook is a Tier C measurement: forward-only and end-to-end timings for each export format, on one CPU, against a PyTorch anchor measured on that same machine. Its value is ratios within this machine — never set these milliseconds against the architecture headline on the Benchmarks page (NVIDIA T4, TensorRT FP16, model only) or the number on Roboflow's RF-DETR model page (L4, Roboflow Inference, scope not stated, batch 1) — those are GPU numbers, not comparable to a CPU run at all, let alone without checking precision, batch size, and scope.
Part A: TFLite¶
Install¶
RFDETR_SPEC holds the install source and defaults to the develop branch. RFDETRInference and the
rfdetr.export.benchmark helpers this notebook imports ship in 1.12; to install from PyPI instead, swap in the
commented >=1.12,<2 line once it is released.
ai_edge_litert, which runs the .tflite files below, comes with [tflite]
transitively: onnx2tf pins ai-edge-litert==2.1.2, so it is not listed separately (an unpinned
extra install would fight that pin). The assertion below fails fast on any interpreter other
than 3.12, before the (slow) install runs. Colab ships mutually inconsistent
preinstalled packages that otherwise crash import rfdetr, so two are aligned: torchaudio is
uninstalled (RF-DETR never uses it, but transformers imports it when present and a
torch/torchaudio CUDA-version mismatch then errors), and pillow is force-reinstalled to a
clean version (a half-upgraded PIL breaks torchvision's import with cannot import name '_Ink').
Colab: after this cell, Runtime → Restart session, then run from the next cell — Colab keeps the old package versions loaded until a restart.
import sys
assert sys.version_info[:2] == (3, 12), (
"rfdetr[tflite] requires Python 3.12 exactly (see pyproject.toml); this runtime is "
f"{sys.version_info.major}.{sys.version_info.minor}. Use a Python 3.12 environment instead."
)
RFDETR_SPEC = "rfdetr[tflite] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[tflite]>=1.12,<2"
!pip install -q "{RFDETR_SPEC}" psutil supervision pandas tabulate faster-coco-eval
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
!pip uninstall -q -y torchaudio
Setup¶
WARMUP_RUNS discards the first N inferences; MEASURE_RUNS then collects the steady-state timing distribution. BATCH_SIZES is the batch matrix every format below measures. measure_latency and measure_memory come from the public rfdetr.export.benchmark module.
MODEL_SIZE (nano, small, medium or large) and TASK (detection or segmentation) pick the model every row measures. A segmentation call also decodes masks on the host, so its latencies are not comparable with detection, and the mAP rows stay box mAP for both tasks.
import importlib.metadata
import os
import platform
from functools import partial
from pathlib import Path
from typing import NamedTuple
import psutil
import torch
from PIL import Image
import rfdetr
from rfdetr import RFDETRInference
from rfdetr.export._benchmark import _artifact_size_mb, cpu_brand, visualize_detections
from rfdetr.export.benchmark import measure_latency, measure_memory
MODEL_SIZE = "small" # nano, small, medium or large
TASK = "detection" # "detection" or "segmentation"
MODEL_CLASS = f"RFDETR{'Seg' if TASK == 'segmentation' else ''}{MODEL_SIZE.capitalize()}" # class name in `rfdetr`
MODEL_ID = f"rfdetr-{'seg-' if TASK == 'segmentation' else ''}{MODEL_SIZE}" # `inference-models` id, names the files
Config holds the fixed options of a benchmark row (runtime, precision, quantization), one table column each. record() appends one row per (format, config, batch); artifacts remembers the batch-1 artifact of every export, so the accuracy section scores exactly the files that were timed.
EXPORT_DIR = Path("export_mobile")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 5
MEASURE_RUNS = 30
BATCH_SIZES = (1, 4)
# COCO val2017 eval subset (see the accuracy section).
N_EVAL = 500
FULL_VAL = False
COCO_SEED = 0
COCO_ROOT = Path("coco")
#: One record per (format, config, batch) benchmark, appended to by `record()`.
rows: list[dict] = []
#: Batch-1 artifact and runtime options of every exported row, keyed by (Format, Config).
artifacts: dict = {}
class Config(NamedTuple):
"""The fixed options of one benchmark row, one table column each."""
runtime: str
precision: str
quantization: str = "—"
def record(
format_label: str, config: Config, batch: int, result, memory_mb: float, size_mb: float | None = None
) -> None:
"""Append one benchmark result; `FPS` counts images per second, so it compares across batch sizes."""
rows.append(
{
"Format": format_label,
"Runtime": config.runtime,
"Precision": config.precision,
"Quantization": config.quantization,
"Batch": batch,
"end2end [ms]": result.mean_ms,
"± [ms]": result.std_ms,
"FPS [img/s]": batch * result.fps,
"Memory [MB]": memory_mb,
"Size [MB]": size_mb,
}
)
CPU thread count¶
Every CPU runtime picks its own default thread pool, so rows measured at each runtime's default compare thread pools, not formats. The TFLite and LiteRT interpreter runs on one thread unless told otherwise, which is the largest such gap: the same fp32 .tflite file ran about twice as fast on eight threads as on one in an earlier run on this chip. CPU_THREADS is the physical-core count from psutil (falling back to os.cpu_count()); it is applied to PyTorch here and to the TFLite and LiteRT interpreter through runtime_options={"num_threads": CPU_THREADS}. ExecuTorch's runtime exposes no thread option, so its XNNPACK pool keeps its own default, which the table notes.
CPU_THREADS = psutil.cpu_count(logical=False) or os.cpu_count() or 1
torch.set_num_threads(CPU_THREADS)
INTERPRETER_OPTIONS = {"num_threads": CPU_THREADS}
Host¶
The printed host line is what the Results header and the docs page quote: CPU, core count and installed versions all change what these numbers mean.
print(f"Machine: {platform.machine()} ({cpu_brand()})")
print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})")
print(f"Cores: {psutil.cpu_count(logical=False)} physical, {os.cpu_count()} logical")
print(f"CPU threads: {CPU_THREADS} pinned (PyTorch effective: {torch.get_num_threads()})")
print(f"Python: {platform.python_version()}")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")
Sample image¶
A single street scene with several COCO classes is enough to verify detections. The image is downloaded once and reused for every format.
import urllib.request
IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)
image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")
PyTorch baseline — predict() / inference()¶
No export needed — this is the reference every on-device format is compared against. It runs on this machine's CPU, not a phone or edge chip, so treat it as an accuracy and relative-speed anchor rather than a deployment number. predict() is the unoptimized baseline; model.inference() returns an independent TorchScript predictor, still fp32. Both calls include preprocessing and postprocessing, so only an end-to-end number is reported. The memory bracket covers construction plus the first predict() call, since weights and lazily built buffers are not fully resident before it.
with measure_memory() as mem:
model = getattr(rfdetr, MODEL_CLASS)(device="cpu")
baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_eager_memory_mb = mem.delta_mb
print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)
Eager predict() on a batch of copies of the sample, batch 1 and 4. Batch 1 reuses the baseline cell's memory delta; a larger batch reports the incremental growth of the already-loaded model.
for batch in BATCH_SIZES:
if batch == 1:
memory_mb = pytorch_eager_memory_mb
else:
with measure_memory() as mem:
_ = model.predict([image] * batch, threshold=CONFIDENCE_THRESHOLD)
memory_mb = mem.delta_mb
result = measure_latency(
partial(model.predict, [image] * batch),
label=f"PyTorch predict() b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record("PyTorch", Config("predict() eager", "fp32"), batch, result, memory_mb)
print(f" {result.label:<32} {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} +{memory_mb:.1f} MB")
model.inference(batch_size=b) creates a TorchScript predictor for that exact batch size, so each batch gets its own predictor; the memory bracket includes its construction and first call.
for batch in BATCH_SIZES:
with measure_memory() as mem:
predictor = model.inference(batch_size=batch)
_ = predictor.predict([image] * batch)
result = measure_latency(
partial(predictor.predict, [image] * batch),
label=f"PyTorch inference() JIT b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record("PyTorch", Config("inference() JIT", "fp32"), batch, result, mem.delta_mb)
print(f" {result.label:<32} {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} +{mem.delta_mb:.1f} MB")
del predictor
TFLite (fp32, fp16, INT8 dynamic-range)¶
What it is. TFLite is TensorFlow's lightweight runtime for mobile, embedded, and edge
hardware. RF-DETR's route converts ONNX → TensorFlow → .tflite via onnx2tf. Good for
Android and embedded targets where TensorFlow Lite is already the established runtime; the
tradeoff is an experimental multi-step conversion chain rather than a bit-exact match to eager
PyTorch. See the TFLite export docs.
All three precisions come from one export call: onnx2tf always writes *_fp32.tflite and
*_fp16.tflite, and quantization="int8" additionally builds a dynamic-range INT8 model (INT8
weights, float32 activations, no calibration data) and returns that as the primary path. So the
three files below are siblings in one directory, benchmarked against each other with no
re-export per precision.
dynamic_batch=Trueis refused for TFLite. The registry marks TFLitesupports_dynamic_batch=False, somodel.export()now raisesNotImplementedErrorbefore conversion starts —onnx2tf(tf 2.19.1, onnx2tf 2.4.3) cannot lower the dynamic-batch graph: a position-embeddingAddbroadcasts a batch-1-shaped constant against the fully dynamic(None, None, 384)encoder input (ValueError: Dimensions must be equal, but are 384 and 1025). So each batch size below gets its own static export (batch_size=batch) instead of one dynamic-batch artifact resized at inference time.
Export¶
tflite_int8_paths: dict[int, Path] = {}
tflite_fp32_paths: dict[int, Path] = {}
tflite_fp16_paths: dict[int, Path] = {}
for batch in BATCH_SIZES:
int8_path = model.export(
format="tflite",
quantization="int8",
batch_size=batch,
output_name=f"{MODEL_ID}_b{batch}",
output_dir=str(EXPORT_DIR),
)
stem = int8_path.name.removesuffix("_dynamic_range_quant.tflite")
tflite_int8_paths[batch] = int8_path
tflite_fp32_paths[batch] = int8_path.with_name(f"{stem}_fp32.tflite")
tflite_fp16_paths[batch] = int8_path.with_name(f"{stem}_fp16.tflite")
Load and benchmark exported artifacts — one path for every format¶
Every export goes through the same two public calls: model.export(...) writes the artifact, and RFDETRInference(path) loads it. The loader reads the artifact's own metadata, so the same predict() that serves the PyTorch rows serves the exports too. benchmark_artifacts takes one artifact per batch size (every format here bakes a fixed batch shape), loads a fresh predictor, runs it once on the sample image (batch 1 prints the detection count and draws the result), and times predict() on batch copies of the sample. Memory [MB] is the host resident-memory growth across constructing the predictor plus its first call. Only end-to-end predict() timings are reported, the one scope every row has; correctness is checked by the COCO mAP section.
def benchmark_artifacts(
format_label: str, config: Config, paths: dict[int, Path], subdir: str, runtime_options: dict | None = None
) -> None:
"""Load one artifact per batch size with `RFDETRInference` and time its `predict()`."""
for batch, path in paths.items():
path = Path(path)
size_mb = _artifact_size_mb(path)
images_batch = [image] * batch
with measure_memory() as mem:
predictor = RFDETRInference(path, device="cpu", runtime_options=runtime_options)
detections = predictor.predict(images_batch, threshold=CONFIDENCE_THRESHOLD)
if batch == BATCH_SIZES[0]:
artifacts[(format_label, config)] = (path, runtime_options or {})
print(f"{format_label} ({', '.join(config)}): {path} ({size_mb:.1f} MB)")
print(f" {len(detections[0])} detections above {CONFIDENCE_THRESHOLD}; runtime: {predictor.runtime_info}")
visualize_detections(detections[0], image, EXPORT_DIR / f"annotated_{subdir}.jpg")
result = measure_latency(
partial(predictor.predict, images_batch),
label=f"{format_label} {config} b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record(format_label, config, batch, result, mem.delta_mb, size_mb)
print(f" batch={batch}: {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} ({batch * result.fps:6.1f} FPS)")
del predictor
TFLite fp32¶
The onnx2tf conversion chain at full precision, with the interpreter pinned to CPU_THREADS through runtime_options.
benchmark_artifacts("TFLite", Config("NHWC", "fp32"), tflite_fp32_paths, "tflite_fp32", INTERPRETER_OPTIONS)
TFLite fp16¶
The sibling file onnx2tf writes from the same export: weights stored in fp16.
benchmark_artifacts("TFLite", Config("NHWC", "fp16"), tflite_fp16_paths, "tflite_fp16", INTERPRETER_OPTIONS)
TFLite INT8 dynamic range¶
INT8 weights with float32 activations and no calibration data, the row where a quantization loss would show.
benchmark_artifacts(
"TFLite", Config("NHWC", "INT8", "dynamic range"), tflite_int8_paths, "tflite_int8", INTERPRETER_OPTIONS
)
COCO val2017 accuracy — Part A¶
A latency number on its own can hide an accuracy loss, so each batch-1 row also gets a box mAP, measured on the same exported artifact that was timed. This runs on a 500-image seeded subset of COCO val2017, so it is not the full-val2017 number on the Benchmarks page; set FULL_VAL = True to score all 5000 images. The first run downloads the 241 MB annotation archive once, plus only the selected images.
Every row is scored the same way: evaluate_coco_map calls predict(image, threshold=0.001) and reads the returned sv.Detections, whose class_id values are COCO category IDs for a pretrained checkpoint; the 0.001 score floor keeps the whole precision-recall curve. Display and scoring helpers come from the private rfdetr.export._benchmark module; they are not a supported API and do not change what any runtime computes.
from rfdetr.export._benchmark import evaluate_coco_map, fetch_coco_val2017
coco_subset = fetch_coco_val2017(COCO_ROOT, n_images=None if FULL_VAL else N_EVAL, seed=COCO_SEED)
print(f"COCO val2017 images to score: {len(coco_subset.image_ids)}")
PyTorch first: eager predict() is the reference every exported row is compared against, then a fresh batch-1 inference() predictor, the same setting as its latency row.
map_results = {}
map_results[("PyTorch", Config("predict() eager", "fp32"))] = evaluate_coco_map(
partial(model.predict, threshold=0.001), coco_subset
)
predictor = model.inference(batch_size=1)
map_results[("PyTorch", Config("inference() JIT", "fp32"))] = evaluate_coco_map(
partial(predictor.predict, threshold=0.001), coco_subset
)
del predictor
Then every exported artifact, loaded exactly as in its latency row (same file, same runtime_options).
for (format_label, config), (path, runtime_options) in artifacts.items():
predictor = RFDETRInference(path, device="cpu", runtime_options=runtime_options)
map_results[(format_label, config)] = evaluate_coco_map(partial(predictor.predict, threshold=0.001), coco_subset)
del predictor
print(
f" {format_label} ({', '.join(config)}): mAP@50:95 {100 * map_results[(format_label, config)].map50_95:.1f} %"
)
Results — Part A¶
The cells below print a decision table (one row per format and config: batch-1 latency, speed against PyTorch predict(), size, mAP@50:95 and FPS at each batch size), the same table as markdown, and a per-batch detail table with ± std and memory growth. Every latency is the end-to-end predict() call, and mAP@50:95 is scored on the same batch-1 artifact that was timed. Memory [MB] is host resident-memory growth in one shared process, so read it as a rough per-row indication.
The numbers from the last run, with their notes, are kept on the TFLite export page so this notebook stays short. A laptop CPU only approximates a phone or edge chip: a phone's own delegate (NNAPI, Hexagon, Core ML) is not exercised here.
import pandas as pd
from IPython.display import display
KEYS = ["Format", "Runtime", "Precision", "Quantization"]
detail = pd.DataFrame(rows)
# Decision table: one row per (format, config); batch-1 latency, accuracy and size, throughput at every batch.
fps_by_batch = detail.pivot_table(index=KEYS, columns="Batch", values="FPS [img/s]", sort=False).add_prefix("FPS b")
map_by_row = pd.Series(
{(format_label, *config): 100 * result.map50_95 for (format_label, config), result in map_results.items()},
name="mAP@50:95 [%]",
)
map_by_row.index.names = KEYS
headline = (
detail[detail["Batch"] == 1]
.set_index(KEYS)[["end2end [ms]", "Size [MB]"]]
.rename(columns={"end2end [ms]": "b1 latency [ms]"})
.join(map_by_row)
.join(fps_by_batch)
)
headline.insert(1, "Speed vs PyTorch ×", headline["b1 latency [ms]"].iloc[0] / headline["b1 latency [ms]"])
# One number format for every table in this section: one decimal, "—" where a row has no value.
decision = headline.apply(lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}"))
display(decision)
Markdown for the docs page: the table above, unchanged.
print(
decision.reset_index().to_markdown(index=False, disable_numparse=True)
) # keep the one-decimal strings as they are
Per-batch detail: ± std and memory growth for every row.
numeric = detail.columns.difference(["Format", "Runtime", "Precision", "Quantization", "Batch"])
detail_table = detail.copy()
detail_table[numeric] = detail[numeric].apply(
lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}")
)
display(detail_table)
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, CPU.")
Part B: LiteRT + ExecuTorch¶
Restart the runtime before this part.
[litert]and[executorch]don't conflict with each other, but both conflict with[tflite]above — Colab: Runtime → Restart session, then run from the next cell. This part re-installs, re-imports, and re-downloads the sample image and PyTorch baseline independently of Part A.
Install¶
RFDETR_SPEC holds the install source and defaults to the develop branch. RFDETRInference and the
rfdetr.export.benchmark helpers this notebook imports ship in 1.12; to install from PyPI instead, swap in the
commented >=1.12,<2 line once it is released.
Colab ships mutually inconsistent preinstalled packages that otherwise crash import rfdetr, so
two are aligned: torchaudio is uninstalled (RF-DETR never uses it, but transformers imports
it when present and a torch/torchaudio CUDA-version mismatch then errors), and pillow is
force-reinstalled to a clean version (a half-upgraded PIL breaks torchvision's import with
cannot import name '_Ink').
flatc— ExecuTorch serializes the.ptewith the FlatBuffers compiler. It ships in the Linuxexecutorchwheel (so Colab works out of the box); on macOS install it separately withbrew install flatbuffers.
executorch>=1.3.1,<1.4is pinned deliberately.pyproject.toml's[executorch]extra is capped at>=1.3,<1.4and says "validated on 1.3.1". One reason for the cap:executorch==1.5.1raisesRuntimeError: tensor does not have a devicefromexecutorch::extension::alias_tensor_ptr_to_attensoron everymethod.execute(...)call on this platform (confirmed withtorch.randn,torch.zeros, and cloned/contiguous variants — all six failed identically, then all six succeeded after downgrading to 1.3.1 alone, with no other package changed). Keep the explicitexecutorch>=1.3.1,<1.4range until this regression is confirmed fixed upstream andpyproject.toml's validated version is bumped.
RFDETR_SPEC = "rfdetr[litert,executorch] @ https://github.com/roboflow/rf-detr/archive/refs/heads/develop.zip"
# RFDETR_SPEC = "rfdetr[litert,executorch]>=1.12,<2"
!pip install -q "{RFDETR_SPEC}" "torch<2.13" "executorch>=1.3.1,<1.4" psutil supervision pandas tabulate faster-coco-eval
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
!pip uninstall -q -y torchaudio
Setup¶
WARMUP_RUNS discards the first N inferences; MEASURE_RUNS then collects the steady-state timing distribution. BATCH_SIZES is the batch matrix every format below measures. measure_latency and measure_memory come from the public rfdetr.export.benchmark module.
MODEL_SIZE (nano, small, medium or large) and TASK (detection or segmentation) pick the model every row measures. A segmentation call also decodes masks on the host, so its latencies are not comparable with detection, and the mAP rows stay box mAP for both tasks.
import importlib.metadata
import os
import platform
from functools import partial
from pathlib import Path
from typing import NamedTuple
import psutil
import torch
from PIL import Image
import rfdetr
from rfdetr import RFDETRInference
from rfdetr.export._benchmark import cpu_brand, visualize_detections
from rfdetr.export.benchmark import measure_latency, measure_memory
MODEL_SIZE = "small" # nano, small, medium or large
TASK = "detection" # "detection" or "segmentation"
MODEL_CLASS = f"RFDETR{'Seg' if TASK == 'segmentation' else ''}{MODEL_SIZE.capitalize()}" # class name in `rfdetr`
MODEL_ID = f"rfdetr-{'seg-' if TASK == 'segmentation' else ''}{MODEL_SIZE}" # `inference-models` id, names the files
Config holds the fixed options of a benchmark row (runtime, precision, quantization), one table column each. record() appends one row per (format, config, batch); artifacts remembers the batch-1 artifact of every export, so the accuracy section scores exactly the files that were timed.
EXPORT_DIR = Path("export_mobile")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5
WARMUP_RUNS = 5
MEASURE_RUNS = 30
BATCH_SIZES = (1, 4)
# COCO val2017 eval subset (see the accuracy section).
N_EVAL = 500
FULL_VAL = False
COCO_SEED = 0
COCO_ROOT = Path("coco")
#: One record per (format, config, batch) benchmark, appended to by `record()`.
rows: list[dict] = []
#: Batch-1 artifact and runtime options of every exported row, keyed by (Format, Config).
artifacts: dict = {}
class Config(NamedTuple):
"""The fixed options of one benchmark row, one table column each."""
runtime: str
precision: str
quantization: str = "—"
def record(
format_label: str, config: Config, batch: int, result, memory_mb: float, size_mb: float | None = None
) -> None:
"""Append one benchmark result; `FPS` counts images per second, so it compares across batch sizes."""
rows.append(
{
"Format": format_label,
"Runtime": config.runtime,
"Precision": config.precision,
"Quantization": config.quantization,
"Batch": batch,
"end2end [ms]": result.mean_ms,
"± [ms]": result.std_ms,
"FPS [img/s]": batch * result.fps,
"Memory [MB]": memory_mb,
"Size [MB]": size_mb,
}
)
CPU thread count¶
Every CPU runtime picks its own default thread pool, so rows measured at each runtime's default compare thread pools, not formats. The TFLite and LiteRT interpreter runs on one thread unless told otherwise, which is the largest such gap: the same fp32 .tflite file ran about twice as fast on eight threads as on one in an earlier run on this chip. CPU_THREADS is the physical-core count from psutil (falling back to os.cpu_count()); it is applied to PyTorch here and to the TFLite and LiteRT interpreter through runtime_options={"num_threads": CPU_THREADS}. ExecuTorch's runtime exposes no thread option, so its XNNPACK pool keeps its own default, which the table notes.
CPU_THREADS = psutil.cpu_count(logical=False) or os.cpu_count() or 1
torch.set_num_threads(CPU_THREADS)
INTERPRETER_OPTIONS = {"num_threads": CPU_THREADS}
Host¶
The printed host line is what the Results header and the docs page quote: CPU, core count and installed versions all change what these numbers mean.
print(f"Machine: {platform.machine()} ({cpu_brand()})")
print(f"OS: {platform.system()} {platform.release()} (macOS {platform.mac_ver()[0] or 'n/a'})")
print(f"Cores: {psutil.cpu_count(logical=False)} physical, {os.cpu_count()} logical")
print(f"CPU threads: {CPU_THREADS} pinned (PyTorch effective: {torch.get_num_threads()})")
print(f"Python: {platform.python_version()}")
print(f"rfdetr: {importlib.metadata.version('rfdetr')}")
Sample image¶
A single street scene with several COCO classes is enough to verify detections. The image is downloaded once and reused for every format.
import urllib.request
IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)
image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")
PyTorch baseline — predict() / inference()¶
No export needed — this is the reference every on-device format is compared against. It runs on this machine's CPU, not a phone or edge chip, so treat it as an accuracy and relative-speed anchor rather than a deployment number. predict() is the unoptimized baseline; model.inference() returns an independent TorchScript predictor, still fp32. Both calls include preprocessing and postprocessing, so only an end-to-end number is reported. The memory bracket covers construction plus the first predict() call, since weights and lazily built buffers are not fully resident before it.
with measure_memory() as mem:
model = getattr(rfdetr, MODEL_CLASS)(device="cpu")
baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
pytorch_eager_memory_mb = mem.delta_mb
print(f"PyTorch baseline: {len(baseline_detections)} detections above {CONFIDENCE_THRESHOLD}")
visualize_detections(baseline_detections, image)
Eager predict() on a batch of copies of the sample, batch 1 and 4. Batch 1 reuses the baseline cell's memory delta; a larger batch reports the incremental growth of the already-loaded model.
for batch in BATCH_SIZES:
if batch == 1:
memory_mb = pytorch_eager_memory_mb
else:
with measure_memory() as mem:
_ = model.predict([image] * batch, threshold=CONFIDENCE_THRESHOLD)
memory_mb = mem.delta_mb
result = measure_latency(
partial(model.predict, [image] * batch),
label=f"PyTorch predict() b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record("PyTorch", Config("predict() eager", "fp32"), batch, result, memory_mb)
print(f" {result.label:<32} {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} +{memory_mb:.1f} MB")
model.inference(batch_size=b) creates a TorchScript predictor for that exact batch size, so each batch gets its own predictor; the memory bracket includes its construction and first call.
for batch in BATCH_SIZES:
with measure_memory() as mem:
predictor = model.inference(batch_size=batch)
_ = predictor.predict([image] * batch)
result = measure_latency(
partial(predictor.predict, [image] * batch),
label=f"PyTorch inference() JIT b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record("PyTorch", Config("inference() JIT", "fp32"), batch, result, mem.delta_mb)
print(f" {result.label:<32} {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} +{mem.delta_mb:.1f} MB")
del predictor
LiteRT¶
What it is. LiteRT (formerly TensorFlow Lite) is Google's on-device runtime. RF-DETR exports
to it directly via torch.export (litert-torch) — no ONNX step, no TensorFlow step. Good
for the same on-device deployment target as TFLite in Part A, with a more direct conversion
path that tracks eager PyTorch more closely; the tradeoffs are fp32-only, a fixed batch size, and
no keypoint model support on the current litert-torch version. See the
LiteRT export docs.
Export¶
format="litert" hands the model to litert-torch, which captures it with torch.export and lowers straight to a .tflite — no ONNX, no TensorFlow. LiteRT bakes a fixed batch shape into the file, so each batch size gets its own export.
litert_paths: dict[int, Path] = {}
for batch in BATCH_SIZES:
litert_paths[batch] = model.export(
format="litert", batch_size=batch, output_name=f"{MODEL_ID}_litert_b{batch}", output_dir=str(EXPORT_DIR)
)
Load and benchmark exported artifacts — one path for every format¶
Every export goes through the same two public calls: model.export(...) writes the artifact, and RFDETRInference(path) loads it. The loader reads the artifact's own metadata, so the same predict() that serves the PyTorch rows serves the exports too. benchmark_artifacts takes one artifact per batch size (every format here bakes a fixed batch shape), loads a fresh predictor, runs it once on the sample image (batch 1 prints the detection count and draws the result), and times predict() on batch copies of the sample. Memory [MB] is the host resident-memory growth across constructing the predictor plus its first call. Only end-to-end predict() timings are reported, the one scope every row has; correctness is checked by the COCO mAP section.
def benchmark_artifacts(
format_label: str, config: Config, paths: dict[int, Path], subdir: str, runtime_options: dict | None = None
) -> None:
"""Load one artifact per batch size with `RFDETRInference` and time its `predict()`."""
for batch, path in paths.items():
path = Path(path)
size_mb = _artifact_size_mb(path)
images_batch = [image] * batch
with measure_memory() as mem:
predictor = RFDETRInference(path, device="cpu", runtime_options=runtime_options)
detections = predictor.predict(images_batch, threshold=CONFIDENCE_THRESHOLD)
if batch == BATCH_SIZES[0]:
artifacts[(format_label, config)] = (path, runtime_options or {})
print(f"{format_label} ({', '.join(config)}): {path} ({size_mb:.1f} MB)")
print(f" {len(detections[0])} detections above {CONFIDENCE_THRESHOLD}; runtime: {predictor.runtime_info}")
visualize_detections(detections[0], image, EXPORT_DIR / f"annotated_{subdir}.jpg")
result = measure_latency(
partial(predictor.predict, images_batch),
label=f"{format_label} {config} b{batch}",
device="cpu",
warmup=WARMUP_RUNS,
runs=MEASURE_RUNS,
)
record(format_label, config, batch, result, mem.delta_mb, size_mb)
print(f" batch={batch}: {result.mean_ms:6.2f} ms ± {result.std_ms:5.2f} ({batch * result.fps:6.1f} FPS)")
del predictor
LiteRT fp32¶
The direct torch.export route, with the interpreter pinned to CPU_THREADS through runtime_options.
benchmark_artifacts("LiteRT", Config("NCHW", "fp32"), litert_paths, "litert", INTERPRETER_OPTIONS)
ExecuTorch (XNNPACK backend)¶
What it is. ExecuTorch is PyTorch's own on-device inference runtime — the model is exported
directly via torch.export to a portable .pte binary, no ONNX conversion involved. The
"xnnpack" backend targets any CPU platform in fp32. Good for on-device PyTorch deployment
(Android, iOS, embedded) when staying inside the PyTorch ecosystem end to end matters more than
picking the single fastest runtime for one platform. See the
ExecuTorch export docs.
Export¶
format="executorch" captures the model with torch.export and lowers it to the xnnpack backend, RF-DETR's portable-CPU backend that also runs on Android and iOS. ExecuTorch bakes a fixed batch shape, so one .pte is exported per batch size.
executorch_paths: dict[int, Path] = {}
for batch in BATCH_SIZES:
executorch_paths[batch] = model.export(
format="executorch",
backend="xnnpack",
batch_size=batch,
output_name=f"{MODEL_ID}_executorch_b{batch}",
output_dir=str(EXPORT_DIR),
)
ExecuTorch XNNPACK fp32¶
ExecuTorch's runtime exposes no thread option through RFDETRInference, so its XNNPACK pool keeps its own default; read this row's gap to LiteRT with that in mind.
benchmark_artifacts("ExecuTorch", Config("XNNPACK", "fp32"), executorch_paths, "executorch")
COCO val2017 accuracy — Part B¶
A latency number on its own can hide an accuracy loss, so each batch-1 row also gets a box mAP, measured on the same exported artifact that was timed. This runs on a 500-image seeded subset of COCO val2017, so it is not the full-val2017 number on the Benchmarks page; set FULL_VAL = True to score all 5000 images. The first run downloads the 241 MB annotation archive once, plus only the selected images.
Every row is scored the same way: evaluate_coco_map calls predict(image, threshold=0.001) and reads the returned sv.Detections, whose class_id values are COCO category IDs for a pretrained checkpoint; the 0.001 score floor keeps the whole precision-recall curve. Display and scoring helpers come from the private rfdetr.export._benchmark module; they are not a supported API and do not change what any runtime computes.
from rfdetr.export._benchmark import evaluate_coco_map, fetch_coco_val2017
coco_subset = fetch_coco_val2017(COCO_ROOT, n_images=None if FULL_VAL else N_EVAL, seed=COCO_SEED)
print(f"COCO val2017 images to score: {len(coco_subset.image_ids)}")
PyTorch first: eager predict() is the reference every exported row is compared against, then a fresh batch-1 inference() predictor, the same setting as its latency row.
map_results = {}
map_results[("PyTorch", Config("predict() eager", "fp32"))] = evaluate_coco_map(
partial(model.predict, threshold=0.001), coco_subset
)
predictor = model.inference(batch_size=1)
map_results[("PyTorch", Config("inference() JIT", "fp32"))] = evaluate_coco_map(
partial(predictor.predict, threshold=0.001), coco_subset
)
del predictor
Then every exported artifact, loaded exactly as in its latency row (same file, same runtime_options).
for (format_label, config), (path, runtime_options) in artifacts.items():
predictor = RFDETRInference(path, device="cpu", runtime_options=runtime_options)
map_results[(format_label, config)] = evaluate_coco_map(partial(predictor.predict, threshold=0.001), coco_subset)
del predictor
print(
f" {format_label} ({', '.join(config)}): mAP@50:95 {100 * map_results[(format_label, config)].map50_95:.1f} %"
)
Results — Part B¶
The cells below print a decision table (one row per format and config: batch-1 latency, speed against PyTorch predict(), size, mAP@50:95 and FPS at each batch size), the same table as markdown, and a per-batch detail table with ± std and memory growth. Every latency is the end-to-end predict() call, and mAP@50:95 is scored on the same batch-1 artifact that was timed. Memory [MB] is host resident-memory growth in one shared process, so read it as a rough per-row indication.
The numbers from the last run, with their notes, are kept on the LiteRT export page so this notebook stays short. A laptop CPU only approximates a phone or edge chip: a phone's own delegate (NNAPI, Hexagon, Core ML) is not exercised here.
import pandas as pd
from IPython.display import display
KEYS = ["Format", "Runtime", "Precision", "Quantization"]
detail = pd.DataFrame(rows)
# Decision table: one row per (format, config); batch-1 latency, accuracy and size, throughput at every batch.
fps_by_batch = detail.pivot_table(index=KEYS, columns="Batch", values="FPS [img/s]", sort=False).add_prefix("FPS b")
map_by_row = pd.Series(
{(format_label, *config): 100 * result.map50_95 for (format_label, config), result in map_results.items()},
name="mAP@50:95 [%]",
)
map_by_row.index.names = KEYS
headline = (
detail[detail["Batch"] == 1]
.set_index(KEYS)[["end2end [ms]", "Size [MB]"]]
.rename(columns={"end2end [ms]": "b1 latency [ms]"})
.join(map_by_row)
.join(fps_by_batch)
)
headline.insert(1, "Speed vs PyTorch ×", headline["b1 latency [ms]"].iloc[0] / headline["b1 latency [ms]"])
# One number format for every table in this section: one decimal, "—" where a row has no value.
decision = headline.apply(lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}"))
display(decision)
Markdown for the docs page: the table above, unchanged.
print(
decision.reset_index().to_markdown(index=False, disable_numparse=True)
) # keep the one-decimal strings as they are
Per-batch detail: ± std and memory growth for every row.
numeric = detail.columns.difference(["Format", "Runtime", "Precision", "Quantization", "Batch"])
detail_table = detail.copy()
detail_table[numeric] = detail[numeric].apply(
lambda column: column.map(lambda value: "—" if pd.isna(value) else f"{value:.1f}")
)
display(detail_table)
print(f"\n{MEASURE_RUNS} timed + {WARMUP_RUNS} warmup runs per row, batches {BATCH_SIZES}, CPU.")
Next steps¶
- Fine-tuned weights — pass
pretrain_weights="<path/to/checkpoint.pth>"when constructing the model. - Deploy on-device — copy the
.tflite/.ptefile to your Android / iOS / edge app and run it with that platform's TensorFlow Lite, LiteRT, or ExecuTorch runtime. - Apple Silicon — ExecuTorch's
coremlbackend (Apple Neural Engine, fp16) and native CoreML / Core AI export are covered in the Apple cookbook, not here. - Qualcomm Snapdragon — ExecuTorch's
qnnbackend needs a source build against the QAIRT SDK; see the ExecuTorch export docs. - Have a CUDA GPU or a general desktop/server CPU? — see the CUDA cookbook or the CPU cookbook.
- See the Export documentation for every format and option.