Measure RF-DETR Latency: The Fastest Configuration on Your Hardware¶
Time every current pretrained RF-DETR checkpoint at batch 1 through RFDETRInference.predict, try each runtime configuration the selected hardware supports and keep the fastest one per model. The accuracy cookbook scores fixed precisions (PyTorch FP32, TensorRT FP16 and INT8) to show what each precision costs in AP; this notebook answers a different question, how fast can this model go on this hardware, so the winning configuration can differ per model and per device.
- The measurement is end to end: preprocessing, the forward pass, postprocessing and, for segmentation, decoding the masks on the host. It is not the model-only latency of the Benchmarks page.
- Every call uses the default
threshold=0.5onNUM_IMAGESCOCO val2017 images, each predicted once in the same order for every configuration. Latency is reported as the median call time with its 95th percentile, because one slow call (OS jitter, throttling) moves a mean but not a median. - Nothing is detected or substituted for you. You choose
HARDWARE; a missing device, package or runtime stops the notebook, and every predictor is checked against the runtime its configuration names before it is timed. - This notebook does not score accuracy. Static ONNX and OpenVINO INT8 were never scored on the PE-Core-T models (Atto, Femto and Pico), so those never try them; their TensorRT INT8 engines keep the MLPs of the first six backbone blocks in FP16 and are timed, and the AP columns of the Run pages score them. The compact CPU ONNX export (
onnx-compact) is timed for those three models, and the accuracy cookbook scores it.
Choose the hardware and install¶
HARDWARE selects the runtimes to install and the configurations to try: cuda (NVIDIA GPU: PyTorch with a CUDA graph and TensorRT engines), apple (Apple Silicon: PyTorch on MPS and CoreML) or cpu (ONNX Runtime and OpenVINO on the CPU, plus PyTorch on the CPU). Set it before installing and restart the kernel after the install, so the recorded package versions are the ones in use; then run the cells again from the top.
HARDWARE = "cuda"
INSTALL_EXTRAS = {"cuda": "plus,tensorrt", "apple": "plus,coreml", "cpu": "plus,onnx,openvino"}
# OpenVINO INT8 needs nncf, the CPU thread count comes from psutil, and the compact ONNX export needs the packages of `rfdetr_plus[onnx]` 1.3 or newer.
INSTALL_PACKAGES = {"cuda": "", "apple": "", "cpu": "nncf psutil 'rfdetr_plus[onnx]>=1.3,<2'"}
assert HARDWARE in INSTALL_EXTRAS, f"HARDWARE must be one of {tuple(INSTALL_EXTRAS)}, got {HARDWARE!r}."
!pip install -U "rfdetr[{INSTALL_EXTRAS[HARDWARE]}]" {INSTALL_PACKAGES[HARDWARE]}
Keep imports together so every dependency of the measurement path is visible in one place.
import hashlib
import importlib.metadata
import itertools
import json
import re
import warnings
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from urllib.parse import quote
import pandas as pd
import rfdetr_plus
import torch
from IPython.display import display
from PIL import Image
from tqdm.auto import tqdm
import rfdetr
from rfdetr import RFDETR, RFDETRInference
from rfdetr.export._benchmark import (
_UNKNOWN_CPU_BRAND,
BenchmarkResult,
_download,
cpu_brand,
export_cached,
fetch_coco_val2017,
measure_latency,
)
# The TensorRT FP16 export clamps float32 weights below 1e-7 in magnitude to ±1e-7 and warns once per weight;
# the clamping is expected and the warnings would bury the output.
warnings.filterwarnings("ignore", message="the float32 number .* will be truncated to", module="onnxconverter_common")
# Tracing the model for ONNX export warns wherever a tensor value steers Python control flow; the export
# fixes those values for the export resolution, which is intended, so the warnings are noise here.
warnings.filterwarnings("ignore", category=torch.jit.TracerWarning)
Select the checkpoints and configurations¶
A configuration is one way to run a model, named <runtime>-<precision>; a PyTorch name also carries its compile backend, and -graph replays a TensorRT engine as a CUDA graph through the cuda_graph runtime option. Every export is a static batch-1 artifact. onnx-compact is the calibration-free compact CPU export of Atto, Femto and Pico (export_cpu_onnx); it exists only for those three models and is not the static QDQ INT8 model of onnx-int8.
Note: PML 1.0 covers the Plus detection models (Atto, Femto, Pico, XLarge and 2XLarge): commercial use is included when you deploy on Roboflow, and commercial self-hosting requires the Enterprise add-on. Apache 2.0 covers the package, Nano–Large detection, every Seg size and Keypoint: commercial, closed-source and self-hosted use is included without a license fee or an obligation to open-source your code. Keep
ACCEPT_PLATFORM_MODEL_LICENSE = Trueonly if you accept the Plus model terms; set it toFalseto time the Apache 2.0 models alone.
MODELS = (
"RFDETRNano",
"RFDETRSmall",
"RFDETRMedium",
"RFDETRLarge",
"RFDETRSegNano",
"RFDETRSegSmall",
"RFDETRSegMedium",
"RFDETRSegLarge",
"RFDETRSegXLarge",
"RFDETRSeg2XLarge",
"RFDETRKeypointPreview",
)
PLUS_MODELS = ("RFDETRAtto", "RFDETRFemto", "RFDETRPico", "RFDETRXLarge", "RFDETR2XLarge")
ACCEPT_PLATFORM_MODEL_LICENSE = True
if ACCEPT_PLATFORM_MODEL_LICENSE:
MODELS += PLUS_MODELS
# Static ONNX and OpenVINO INT8 of the PE-Core-T models were not scored for accuracy, so no such configuration is timed
# for them: a faster INT8 artifact is not a usable answer when its accuracy was not checked. Their TensorRT INT8 engines
# keep the MLPs of the first six backbone blocks in FP16 and are timed.
INT8_EXCLUDED = ("RFDETRAtto", "RFDETRFemto", "RFDETRPico")
# `export_cpu_onnx` exists only on the PE-Core-T Plus models, so `onnx-compact` is timed for no other model.
COMPACT_MODELS = ("RFDETRAtto", "RFDETRFemto", "RFDETRPico")
HARDWARE_CONFIGS = {
"cuda": (
"pytorch-cudagraph-fp16",
"tensorrt-fp16",
"tensorrt-fp16-graph",
"tensorrt-int8",
"tensorrt-int8-graph",
),
"apple": (
"pytorch-inductor-fp32",
"pytorch-inductor-fp16",
"pytorch-torchscript-fp16",
"coreml-fp32",
"coreml-fp16",
"coreml-gpu-fp32",
"coreml-gpu-fp16",
),
"cpu": (
"pytorch-torchscript-fp32",
"onnx-fp32",
"onnx-int8",
"onnx-compact",
"openvino-fp32",
"openvino-fp16",
"openvino-int8",
),
}
CONFIGS = HARDWARE_CONFIGS[HARDWARE]
COCO_ROOT = Path("coco")
WORK_DIR = Path("coco_latency")
RESULTS_PATH = WORK_DIR / "results.jsonl"
# The CPU calls take 10 to 100 times longer than accelerator calls, so 200 images keep the sweep affordable and still leave 10 calls above the 95th percentile.
NUM_IMAGES = {"cuda": 500, "apple": 500, "cpu": 200}[HARDWARE]
NUM_WARMUP = 50
# A median within this fraction of the fastest one is inside the run-to-run noise (repeated runs of the largest models
# differed by 5 to 7% on an M4 Max), so every such configuration counts as fastest. Raise it if your own runs spread more.
TIE_TOLERANCE = 0.05
NUM_CALIB = 128
Reject missing license acceptance and a hardware choice the machine cannot satisfy before downloading data or weights. Model classes resolve through rfdetr later, so Plus imports stay lazy and an unknown name fails there.
assert MODELS, "Select at least one model."
assert ACCEPT_PLATFORM_MODEL_LICENSE or not set(MODELS) & set(PLUS_MODELS), (
"Plus models require ACCEPT_PLATFORM_MODEL_LICENSE = True after accepting PML 1.0."
)
assert HARDWARE != "cuda" or torch.cuda.is_available(), "HARDWARE = 'cuda' needs a CUDA GPU, and none is available."
assert HARDWARE != "apple" or torch.backends.mps.is_available(), "HARDWARE = 'apple' needs Apple Silicon with MPS."
On the CPU the thread count decides the result, so pin it to the logical core count for every runtime: PyTorch through torch.set_num_threads, and ONNX Runtime and OpenVINO through the num_threads key of their runtime_options. Without the pin PyTorch, OpenVINO and the TFLite family pick their own defaults, which are not the logical core count and can leave the hyperthread siblings idle (4 of 8 on the usual Colab x86 machine); RFDETRInference itself already sizes the ONNX Runtime pool to every logical core, and the explicit key keeps that visible. OpenVINO takes the count as INFERENCE_NUM_THREADS, which overrides its own default and its performance hint, and the notebook also sets ENABLE_HYPER_THREADING because OpenVINO chooses that itself when it is unset. A host that does not report its logical cores stops the notebook rather than guessing.
if HARDWARE == "cpu":
import psutil
CPU_THREADS = psutil.cpu_count(logical=True)
assert CPU_THREADS, "psutil cannot report the logical core count of this host."
torch.set_num_threads(CPU_THREADS)
RUNTIME_OPTIONS = {
"onnx": {"num_threads": CPU_THREADS},
"openvino": {"num_threads": CPU_THREADS, "config": {"ENABLE_HYPER_THREADING": True}},
}
else:
CPU_THREADS = None
RUNTIME_OPTIONS = {}
print(f"CPU threads: {CPU_THREADS}")
Prepare the images¶
Fetch the annotation archive and the NUM_IMAGES val2017 images selected by a fixed seed; images already on disk, for example from the accuracy cookbook, are reused. The annotation archive also carries the train2017 file list the INT8 calibration needs.
subset = fetch_coco_val2017(COCO_ROOT, n_images=NUM_IMAGES)
images = [Image.open(subset.images_dir / f"{image_id:012d}.jpg").convert("RGB") for image_id in subset.image_ids]
image_digest = hashlib.sha256()
for image in images:
image_digest.update(str(image.size).encode())
image_digest.update(image.tobytes())
print(f"Timing with {len(images)} val2017 images, each predicted once in a fixed order.")
Record provenance and resume completed runs¶
Record the hardware, the device name, the installed Git commit when available and the version of every package the selected runtimes need once; a package that is not installed stops the notebook. Resume identifies a row by model, configuration, recorded software and hardware provenance, and a fingerprint of the model, source and calibration inputs. A changed input is measured again.
cpu = cpu_brand()
assert HARDWARE == "cuda" or cpu != _UNKNOWN_CPU_BRAND, (
"The host does not report a CPU model, so the device cannot be named."
)
installed_versions = {
dist.metadata["Name"].lower().replace("_", "-"): dist.version for dist in importlib.metadata.distributions()
}
RUNTIME_PACKAGES = {
"cuda": ("tensorrt",),
"apple": ("coremltools",),
"cpu": ("onnxruntime", "openvino", "nncf", "psutil"),
}
direct_url = json.loads(importlib.metadata.distribution("rfdetr").read_text("direct_url.json") or "{}")
device = {
"cuda": lambda: f"cuda: {torch.cuda.get_device_name()}",
"apple": lambda: f"apple: {cpu}",
"cpu": lambda: f"cpu: {cpu}",
}[HARDWARE]()
provenance = {
"hardware": HARDWARE,
"device": device,
"cpu_threads": CPU_THREADS,
"n_images": NUM_IMAGES,
"images_sha256": image_digest.hexdigest(),
"rfdetr_version": importlib.metadata.version("rfdetr"),
"commit": direct_url.get("vcs_info", {}).get("commit_id"),
"torch_version": torch.__version__,
"rfdetr_plus_version": installed_versions["rfdetr-plus"],
**{f"{package.replace('-', '_')}_version": installed_versions[package] for package in RUNTIME_PACKAGES[HARDWARE]},
}
print(json.dumps(provenance, indent=2))
Cached exports and resumed measurements are bound to the installed RF-DETR source, model weights, calibration files and export settings. Each model and configuration has its own directory, so changes produce fresh artifacts rather than silently reusing old ones.
artifact_key = f"{device} {RUNTIME_PACKAGES[HARDWARE][0]} {provenance[RUNTIME_PACKAGES[HARDWARE][0] + '_version']}"
ENGINE_DIR = WORK_DIR / "engines" / re.sub(r"[^a-z0-9.]+", "-", artifact_key.lower()).strip("-")
print(f"Artifact cache: {ENGINE_DIR}")
RESUME_FIELDS = ("model", "config", "artifact_signature", *provenance)
def resume_key(row: dict[str, Any]) -> tuple[Any, ...]:
"""Identify a measurement by its model, configuration and measuring context."""
return tuple(row.get(field) for field in RESUME_FIELDS)
WORK_DIR.mkdir(parents=True, exist_ok=True)
rows = (
[json.loads(line) for line in RESULTS_PATH.read_text().splitlines() if line.strip()]
if RESULTS_PATH.exists()
else []
)
completed = {resume_key(row) for row in rows}
print(f"Loaded {len(rows)} completed measurements from {RESULTS_PATH}.")
Prepare held-out INT8 calibration¶
Static INT8 uses the first NUM_CALIB train2017 images sorted by filename, the same set as the accuracy cookbook, disjoint from val2017. Existing files are reused; hardware without an INT8 configuration skips the download.
CALIBRATION_DIR = WORK_DIR / "calibration"
if any("int8" in config for config in CONFIGS):
train_images = json.loads((COCO_ROOT / "annotations" / "instances_train2017.json").read_text())["images"]
calibration_names = sorted(record["file_name"] for record in train_images)[:NUM_CALIB]
del train_images
for name in calibration_names:
destination = CALIBRATION_DIR / name
if not destination.exists():
_download("http://images.cocodataset.org/train2017/" + quote(name), destination)
print(f"Calibration: {NUM_CALIB} train2017 images in {CALIBRATION_DIR}.")
else:
print("No INT8 configuration for this hardware; calibration images are not needed.")
Build predictors¶
EXPORTS lists, per exported configuration, the artifact pattern and the export keywords. export_cached reuses a cached artifact or exports one, and refuses an ambiguous cache or an export that does not match its pattern. The -graph variant of a TensorRT precision shares its engine. The coreml-gpu configurations share the Core ML package of the same precision and load it with device="cpu_and_gpu", while coreml takes the default device="auto", which lets Core ML put work on the Neural Engine.
def artifact_signature(model: RFDETR) -> str:
"""Bind cached exports and measurements to model weights, source, and calibration inputs."""
digest = hashlib.sha256()
digest.update(json.dumps({**provenance, "num_calib": NUM_CALIB, "warmup": NUM_WARMUP}, sort_keys=True).encode())
for package in (rfdetr, rfdetr_plus):
root = Path(package.__file__).parent
for path in sorted(root.rglob("*.py")):
digest.update(str(path.relative_to(root)).encode())
digest.update(path.read_bytes())
for name, tensor in sorted(model.model.model.state_dict().items()):
digest.update(name.encode())
digest.update(str((tensor.shape, tensor.dtype)).encode())
digest.update(tensor.detach().cpu().contiguous().reshape(-1).view(torch.uint8).numpy().tobytes())
if CALIBRATION_DIR.exists():
for path in sorted(CALIBRATION_DIR.iterdir()):
if path.is_file():
digest.update(path.name.encode())
digest.update(path.read_bytes())
digest.update(repr(EXPORTS).encode())
return digest.hexdigest()
INT8 = {"quantization": "int8", "calibration_data": str(CALIBRATION_DIR), "max_images": NUM_CALIB}
EXPORTS = {
"tensorrt-fp16": ("*.trt", {"format": "tensorrt", "fp16": True}),
"tensorrt-int8": ("*_int8.trt", {"format": "tensorrt", **INT8}),
"onnx-fp32": ("*.onnx", {"format": "onnx"}),
"onnx-int8": ("*_int8.onnx", {"format": "onnx", **INT8}),
"openvino-fp32": ("*.xml", {"format": "openvino", "openvino_precision": "float32"}),
"openvino-fp16": ("*.xml", {"format": "openvino", "openvino_precision": "float16"}),
"openvino-int8": ("*.xml", {"format": "openvino", **INT8}),
"coreml-fp32": ("*.mlpackage", {"format": "coreml"}),
"coreml-fp16": ("*.mlpackage", {"format": "coreml", "coreml_precision": "float16"}),
}
The compact ONNX file does not come from model.export, so it has its own helper: it reuses the one file of its cache directory, or writes it with export_cpu_onnx from the CPU model of this run.
def export_compact_onnx(model: RFDETR, output_dir: Path) -> Path:
"""Reuse or write the compact ONNX file that `export_cpu_onnx` produces for a PE-Core-T model."""
cached = sorted(output_dir.glob("*.onnx"))
assert len(cached) <= 1, f"Ambiguous compact ONNX cache: {output_dir} holds {len(cached)} files."
return cached[0] if cached else Path(model.export_cpu_onnx(output_dir))
build_predictor builds one predictor and then checks it. PyTorch names the compile backend and dtype explicitly instead of taking the device default, and runtime_info must report the runtime and device the configuration names; any difference stops the run, so a configuration can never be timed as a different one.
DTYPES = {"fp32": torch.float32, "fp16": torch.float16}
#: What `runtime_info` must report: (backend, device prefix); a device prefix of None is not checked.
PYTORCH_DEVICE = {"cuda": "cuda", "apple": "mps", "cpu": "cpu"}[HARDWARE]
EXPECTED_DEVICE = {"onnx": "CPUExecutionProvider", "openvino": "CPU", "coreml": "auto"}
#: Configurations that load the artifact of another configuration with explicit Core ML compute units, as (artifact, device).
CORE_ML_UNITS = {
"coreml-gpu-fp32": ("coreml-fp32", "cpu_and_gpu"),
"coreml-gpu-fp16": ("coreml-fp16", "cpu_and_gpu"),
}
def build_predictor(model: RFDETR, model_name: str, config: str) -> RFDETRInference:
"""Build the predictor of one configuration and verify the runtime it reports.
Args:
model: Freshly constructed pretrained model.
model_name: Checkpoint label for the cache directory.
config: One entry of ``CONFIGS``.
Returns:
A public predictor owning its runtime.
"""
backend = config.split("-")[0]
if backend == "pytorch":
_, compile_backend, precision = config.split("-")
predictor = model.inference(compile_backend=compile_backend, dtype=DTYPES[precision])
expected_device = PYTORCH_DEVICE
else:
if config.endswith("-graph"):
options = {"cuda_graph": True}
else:
options = RUNTIME_OPTIONS.get(backend)
exported, core_ml_device = CORE_ML_UNITS.get(config, (config.removesuffix("-graph"), None))
artifact_dir = ENGINE_DIR / model_name / model_signatures[model_name] / exported
if exported == "onnx-compact":
artifact = export_compact_onnx(model, artifact_dir)
else:
pattern, export_kwargs = EXPORTS[exported]
artifact = export_cached(model, artifact_dir, pattern, batch_size=1, verbose=False, **export_kwargs)
load_kwargs = {"device": "cpu"} if HARDWARE == "cpu" else {}
if core_ml_device is not None:
load_kwargs = {"device": core_ml_device}
predictor = RFDETRInference(artifact, runtime_options=options, **load_kwargs)
expected_device = core_ml_device or EXPECTED_DEVICE.get(backend)
if backend == "openvino":
# The benchmark must refuse a silently clamped CPU thread count.
session = predictor._export_context.run.__self__.session
granted = session.compiled_model.get_property("INFERENCE_NUM_THREADS")
assert granted == CPU_THREADS, f"OpenVINO granted {granted} threads, requested {CPU_THREADS}."
info = predictor.runtime_info
assert info["backend"] == backend, f"{config} loaded the {info['backend']} runtime, expected {backend}."
assert expected_device is None or str(info["device"]).startswith(expected_device), (
f"{config} runs on {info['device']}, expected {expected_device}."
)
return predictor
measure_predictor times one predictor: it predicts each image once, so a configuration never sees a single easy frame, and returns every call's time in milliseconds. CUDA events time the call on a GPU and wall-clock time any other hardware; predict returns host results, so either one covers the whole call.
def measure_predictor(predictor: RFDETRInference, config: str) -> BenchmarkResult:
"""Time batch-1 end-to-end prediction of one predictor, one call per image after ``NUM_WARMUP`` untimed ones.
Args:
predictor: Public predictor to time.
config: Configuration label for the result row.
Returns:
The timed calls, with their mean, standard deviation, median and 95th percentile.
"""
cycle = itertools.count()
return measure_latency(
lambda: predictor.predict(images[next(cycle) % len(images)], include_source_image=False),
label=config,
device="cuda" if HARDWARE == "cuda" else "cpu",
warmup=NUM_WARMUP,
runs=len(images),
)
Measure and save each run¶
List the model and configuration pairs still to measure; ONNX and OpenVINO INT8 are skipped for INT8_EXCLUDED, and onnx-compact runs only for COMPACT_MODELS. Each run times one call per image after NUM_WARMUP untimed ones and appends its row at once so an interruption keeps every finished measurement. A configuration that fails to build, load or match its runtime stops the loop; fix the cause and rerun to continue from the saved rows.
pairs = [
(m, c)
for m in MODELS
for c in CONFIGS
if not ("int8" in c and not c.startswith("tensorrt") and m in INT8_EXCLUDED)
and not (c == "onnx-compact" and m not in COMPACT_MODELS)
]
print(f"Checking {len(pairs)} model and configuration pairs against saved measurements.")
loaded_name, model = None, None
model_signatures = {}
progress = tqdm(pairs, desc="Runs", unit="run")
for model_name, config in progress:
progress.set_postfix_str(f"{model_name} / {config}")
if model_name != loaded_name:
model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
# The license keyword exists only on the Plus constructors; the CPU runs pin the model to the CPU.
model_kwargs = {"accept_platform_model_license": True} if model_name in PLUS_MODELS else {}
if HARDWARE == "cpu":
model_kwargs["device"] = "cpu"
model = getattr(rfdetr, model_name)(**model_kwargs)
loaded_name = model_name
model_signatures[model_name] = artifact_signature(model)
signature = model_signatures[model_name]
if resume_key({**provenance, "model": model_name, "config": config, "artifact_signature": signature}) in completed:
continue
predictor = build_predictor(model, model_name, config)
result = measure_predictor(predictor, config)
info = predictor.runtime_info
row = {
**provenance,
"model": model_name,
"artifact_signature": signature,
"config": config,
"task": info["task"],
"backend": info["backend"],
"runtime_device": str(info["device"]),
"measured_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"median_ms": result.median_ms,
"p95_ms": result.p95_ms,
"mean_ms": result.mean_ms,
"std_ms": result.std_ms,
}
with RESULTS_PATH.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(row, allow_nan=False) + "\n")
completed.add(resume_key(row))
rows.append(row)
progress.write(f"{model_name} / {config}: median {result.median_ms:.2f} ms, p95 {result.p95_ms:.2f} ms")
del predictor
model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
Fastest configuration per model¶
rows holds the saved measurements loaded at the start plus this session's. Keep those of this hardware, device, version and commit for the selected models and configurations, and keep, per model, the lowest median latency and every configuration whose median is within TIE_TOLERANCE of it: a gap that small is noise, so they all count as fastest, and you choose among them by accuracy and tooling. The 95th percentile shows how slow the occasional call is, and the mean and standard deviation stay in results.jsonl. The last column counts the configurations that finished, so a winner chosen from an incomplete sweep is visible.
wanted = {
resume_key({**provenance, "model": m, "config": c, "artifact_signature": model_signatures[m]}) for m, c in pairs
}
current = pd.DataFrame([row for row in rows if resume_key(row) in wanted])
for field, value in provenance.items():
print(f"{field}: {value}")
# Strip the graph suffix before separating the precision, preserving it in the engine label.
is_graph = current["config"].str.endswith("-graph")
engine_precision = current["config"].str.removesuffix("-graph").str.rsplit("-", n=1, expand=True)
current = current.assign(
engine=engine_precision[0] + is_graph.map({True: "-graph", False: ""}), precision=engine_precision[1]
)
order = {name: index for index, name in enumerate(MODELS)}
# A configuration within the noise band of its model's fastest median is as fast as the fastest one.
fastest_ms = current.groupby("model")["median_ms"].transform("min")
current = current.assign(is_fastest=current["median_ms"] <= fastest_ms * (1 + TIE_TOLERANCE))
best = current[current["is_fastest"]]
counts = current.groupby("model")["config"].nunique()
expected = {m: sum(1 for pair in pairs if pair[0] == m) for m in MODELS}
table = best.assign(
configs_done=[f"{counts[m]}/{expected[m]}" for m in best["model"]],
ms=best["median_ms"].round(2),
p95=best["p95_ms"].round(2),
model_order=best["model"].map(order),
)
table = table.sort_values(["model_order", "median_ms"])
print(
f"A model lists every configuration within {TIE_TOLERANCE:.0%} of its fastest median. Their gaps are smaller "
"than the run-to-run noise, so the data cannot say which of them is faster."
)
display(table[["model", "task", "engine", "precision", "ms", "p95", "configs_done"]].reset_index(drop=True))
Every configuration¶
The full sweep, so you can see how far the winner is ahead. Each cell is median ms (p95): the median latency in milliseconds, with the 95th percentile of the same calls, also in milliseconds, in brackets, with one column per engine and precision. A * marks every configuration within TIE_TOLERANCE of the fastest one of its model.
full = current.assign(
cell=current["median_ms"].map("{:.2f}".format)
+ " ("
+ current["p95_ms"].map("{:.2f}".format)
+ ")"
+ current["is_fastest"].map({True: " *", False: ""})
)
display(
full.pivot(index="model", columns=["engine", "precision"], values="cell").loc[
[m for m in MODELS if m in set(full["model"])]
]
)
Next steps¶
Use the accuracy cookbook to check what a precision costs in AP before choosing its configuration. The numbers depend on the hardware, the runtime versions and the host CPU, because the host does the preprocessing and postprocessing, so state them next to any copied figure. The model-only latency of the Benchmarks page is a different measurement.