RF-DETR COCO Accuracy: Reproduce the Pretrained Numbers¶
Score every current pretrained RF-DETR checkpoint on the full COCO val2017 split (5000 images), at batch 1, through RFDETRInference.predict(image, threshold=0.001, include_source_image=False). This is the same public call users make, with default preprocessing and a low score floor that preserves the precision-recall curve.
- Detection reports box AP and AP50.
- Segmentation reports box and mask AP and AP50.
- Keypoint Preview reports box and keypoint AP and AP50.
AP means COCO AP50:95 unless explicitly labelled AP50. Saved scores are fractions; the results table shows percentages. The Benchmarks page uses a different harness (SAB: T4, TensorRT 10.4, FP16), so small differences are expected.
Atto, Femto and Pico also get an onnx-compact arm: the calibration-free compact CPU ONNX export (export_cpu_onnx, rfdetr_plus 1.3 or newer), scored through ONNX Runtime's CPU provider.
Install and import¶
train brings faster-coco-eval for scoring, plus the Plus models, tensorrt the TensorRT arms, rfdetr_plus[onnx] the ONNX packages the compact CPU export needs, at the versions rfdetr_plus itself requires. Restart the kernel after installing so the recorded package versions are the ones in use.
!pip install -U "rfdetr[train,plus,tensorrt]" "rfdetr_plus[onnx]>=1.3,<2"
Keep imports together so every dependency of the scoring path is visible in one place.
import hashlib
import importlib.metadata
import json
import platform
import re
import time
import warnings
from datetime import datetime, timezone
from functools import partial
from pathlib import Path
from typing import Any
from urllib.parse import quote
import pandas as pd
import torch
from IPython.display import display
from tqdm.auto import tqdm
import rfdetr
from rfdetr import RFDETR, RFDETRInference
from rfdetr.datasets._develop import _download_and_extract
from rfdetr.export._benchmark import _download, evaluate_coco_ap, export_cached, fetch_coco_val2017
# The TensorRT FP16 export clamps float32 weights below 1e-7 in magnitude to ±1e-7 and warns once per weight;
# the clamping is expected and the warnings would bury the output.
warnings.filterwarnings("ignore", message="the float32 number .* will be truncated to", module="onnxconverter_common")
# Tracing the model for ONNX export warns wherever a tensor value steers Python control flow; the export
# fixes those values for the export resolution, which is intended, so the warnings are noise here.
warnings.filterwarnings("ignore", category=torch.jit.TracerWarning)
Select the checkpoints and runtimes¶
The defaults cover every checkpoint, the Plus models included while ACCEPT_PLATFORM_MODEL_LICENSE = True, and every arm: eager PyTorch FP32, and static TensorRT FP16 and INT8 engines, so an NVIDIA GPU is required. onnx-compact runs only for Atto, Femto and Pico. It is exported from a CPU copy of the model and scored on ONNX Runtime's CPU provider, so it needs no GPU. On a device without CUDA, set ARMS = ("pytorch-fp32", "onnx-compact") to skip the TensorRT arms.
Note: PML 1.0 covers the Plus detection models (Atto, Femto, Pico, XLarge and 2XLarge): commercial use is included when you deploy on Roboflow, and commercial self-hosting requires the Enterprise add-on. Apache 2.0 covers the package, Nano–Large detection, every Seg size and Keypoint: commercial, closed-source and self-hosted use is included without a license fee or an obligation to open-source your code. Keep
ACCEPT_PLATFORM_MODEL_LICENSE = Trueonly if you accept the Plus model terms; set it toFalseto score the Apache 2.0 models alone.
MODELS = (
"RFDETRNano",
"RFDETRSmall",
"RFDETRMedium",
"RFDETRLarge",
"RFDETRSegNano",
"RFDETRSegSmall",
"RFDETRSegMedium",
"RFDETRSegLarge",
"RFDETRSegXLarge",
"RFDETRSeg2XLarge",
"RFDETRKeypointPreview",
)
PLUS_MODELS = ("RFDETRAtto", "RFDETRFemto", "RFDETRPico", "RFDETRXLarge", "RFDETR2XLarge")
ACCEPT_PLATFORM_MODEL_LICENSE = True
if ACCEPT_PLATFORM_MODEL_LICENSE:
MODELS += PLUS_MODELS
ARMS = ("pytorch-fp32", "tensorrt-fp16", "tensorrt-int8", "onnx-compact")
#: The PE-Core-T Plus models, the only ones `export_cpu_onnx` supports; the `onnx-compact` arm skips every other model.
COMPACT_MODELS = ("RFDETRAtto", "RFDETRFemto", "RFDETRPico")
COCO_ROOT = Path("coco")
WORK_DIR = Path("coco_accuracy")
RESULTS_PATH = WORK_DIR / "results.jsonl"
N_CALIB = 128
Reject missing license acceptance and unsupported runtime choices before downloading data or weights. Model classes resolve through rfdetr later, so Plus imports stay lazy and an unknown name fails there.
assert MODELS, "Select at least one model."
assert ACCEPT_PLATFORM_MODEL_LICENSE or not set(MODELS) & set(PLUS_MODELS), (
"Plus models require ACCEPT_PLATFORM_MODEL_LICENSE = True after accepting PML 1.0."
)
CPU_ARMS = {"pytorch-fp32", "onnx-compact"}
assert torch.cuda.is_available() or set(ARMS) <= CPU_ARMS, "TensorRT arms require CUDA."
Prepare COCO val2017¶
Every arm scores all 5000 val2017 images. Download the image archive in one transfer (5000 single downloads can hit socket timeouts); the completion check requires 5000 nonempty JPEG files.
_download_and_extract(
"http://images.cocodataset.org/zips/val2017.zip",
COCO_ROOT,
is_complete=lambda: sum(path.stat().st_size > 0 for path in (COCO_ROOT / "val2017").glob("*.jpg")) == 5000,
)
Fetch the annotation archive (instances, person keypoints and the train2017 annotations used for INT8 calibration) and list the val2017 images.
subset = fetch_coco_val2017(COCO_ROOT, n_images=None)
print(f"Scoring {len(subset.image_ids)} COCO val2017 images per arm.")
Record provenance and resume completed scores¶
Record software versions, the installed Git commit when available, the device and both annotation hashes once before scoring. An editable or wheel installation without Git provenance has a null commit; it must not be mistaken for a verified source revision. Use a fresh work directory after changing local source or weights. Resume identifies a row by model, arm, device, RF-DETR version and commit; it does not detect uncommitted source changes.
installed_versions = {
dist.metadata["Name"].lower().replace("_", "-"): dist.version for dist in importlib.metadata.distributions()
}
rfdetr_dist = importlib.metadata.distribution("rfdetr")
direct_url = json.loads(rfdetr_dist.read_text("direct_url.json") or "{}")
device = (
f"cuda: {torch.cuda.get_device_name()}"
if torch.cuda.is_available()
else f"mps: {platform.processor()}"
if torch.backends.mps.is_available()
else f"cpu: {platform.processor()}"
)
provenance = {
"rfdetr_version": importlib.metadata.version("rfdetr"),
"commit": direct_url.get("vcs_info", {}).get("commit_id"),
"torch_version": torch.__version__,
"faster_coco_eval_version": installed_versions.get("faster-coco-eval"),
"rfdetr_plus_version": installed_versions.get("rfdetr-plus"),
"tensorrt_version": installed_versions.get("tensorrt"),
"device": device,
"annotation_sha256": {
name: hashlib.sha256((COCO_ROOT / "annotations" / name).read_bytes()).hexdigest()
for name in ("instances_val2017.json", "person_keypoints_val2017.json")
},
}
print(json.dumps(provenance, indent=2))
A TensorRT engine runs only on the GPU model and TensorRT version that built it. Cache engines under both, so a work directory shared between GPUs (for example an L4 and an RTX PRO 6000 session on the same drive) never loads an engine built on the other GPU; each GPU builds its own.
engine_key = f"{device} trt {provenance['tensorrt_version']}"
ENGINE_DIR = WORK_DIR / "engines" / re.sub(r"[^a-z0-9.]+", "-", engine_key.lower()).strip("-")
print(f"Engine cache: {ENGINE_DIR}")
Use the same explicit fields for loaded and new rows so changing the scored subset or installed revision cannot silently reuse a different run.
RESUME_FIELDS = ("model", "arm", "device", "rfdetr_version", "commit")
def resume_key(row: dict[str, Any]) -> tuple[Any, ...]:
"""Identify a score by its model, runtime and evaluation context."""
return tuple(row[field] for field in RESUME_FIELDS)
Load completed JSON lines before constructing models; an interrupted or malformed line fails visibly rather than discarding evidence.
WORK_DIR.mkdir(parents=True, exist_ok=True)
rows = (
[json.loads(line) for line in RESULTS_PATH.read_text().splitlines() if line.strip()]
if RESULTS_PATH.exists()
else []
)
completed = {resume_key(row) for row in rows}
print(f"Loaded {len(rows)} completed scores from {RESULTS_PATH}.")
Prepare held-out INT8 calibration¶
TensorRT INT8 uses the first N_CALIB train2017 images sorted by filename, matching Section 8.1 of the CUDA export cookbook's docs-table settings. These images are disjoint from val2017. Existing files are reused. Start a fresh work directory if you change N_CALIB; cached engines also depend on calibration settings, package versions and GPU hardware.
CALIBRATION_DIR = WORK_DIR / "calibration"
train_images = json.loads((COCO_ROOT / "annotations" / "instances_train2017.json").read_text())["images"]
calibration_names = sorted(record["file_name"] for record in train_images)[:N_CALIB]
del train_images
for name in calibration_names:
destination = CALIBRATION_DIR / name
if not destination.exists():
_download("http://images.cocodataset.org/train2017/" + quote(name), destination)
print(f"Calibration: {N_CALIB} train2017 images, first by filename, in {CALIBRATION_DIR}.")
Build predictors¶
Each builder returns the same public prediction facade. The PyTorch arm borrows the live eager FP32 model; it does not compile or change preprocessing. TensorRT arms export static batch-1 engines with the keyword sets used by the CUDA docs benchmark. Each model and arm has its own engine directory under ENGINE_DIR; a single existing engine is reused. Use a fresh WORK_DIR after changing calibration, weights or other software, because the engine path records only the GPU model and TensorRT version.
def build_pytorch_fp32(model: RFDETR, model_name: str) -> RFDETRInference:
"""Wrap the live eager model without compilation or dtype changes."""
return RFDETRInference(model)
Keep FP16 engines separate from INT8 so a resumed export cannot load the other precision.
def build_tensorrt_fp16(model: RFDETR, model_name: str) -> RFDETRInference:
"""Reuse or export one static batch-1 FP16 engine."""
engine_path = export_cached(
model,
ENGINE_DIR / model_name / "tensorrt-fp16",
"*.trt",
format="tensorrt",
batch_size=1,
verbose=False,
fp16=True,
)
assert not engine_path.name.endswith("_int8.trt"), f"Expected an FP16 engine, got {engine_path}"
return RFDETRInference(engine_path)
Pass the held-out directory and explicit image limit so INT8 uses all selected calibration images rather than the exporter's default limit.
def build_tensorrt_int8(model: RFDETR, model_name: str) -> RFDETRInference:
"""Reuse or export one static batch-1 calibrated INT8 engine."""
engine_path = export_cached(
model,
ENGINE_DIR / model_name / "tensorrt-int8",
"*_int8.trt",
format="tensorrt",
batch_size=1,
verbose=False,
quantization="int8",
calibration_data=str(CALIBRATION_DIR),
max_images=N_CALIB,
)
return RFDETRInference(engine_path)
Compact ONNX is the calibration-free CPU export of Atto, Femto and Pico: selected linear projections use dynamic INT8 and the other large filters are stored compactly. A fresh CPU copy of the model writes it, so the export does not depend on the CUDA device the other arms use, and the file is loaded on ONNX Runtime's CPU provider with device="cpu", the same way the CPU latency cookbook times it. RFDETRInference sizes the ONNX Runtime CPU pool to every logical core the process may use unless runtime_options={"num_threads": ...} says otherwise.
def build_onnx_compact(model: RFDETR, model_name: str) -> RFDETRInference:
"""Reuse or export the compact ONNX file of a PE-Core-T model and load it on the CPU provider."""
export_dir = ENGINE_DIR / model_name / "onnx-compact"
cached = sorted(export_dir.glob("*.onnx"))
assert len(cached) <= 1, f"Ambiguous compact ONNX cache: {export_dir} holds {len(cached)} files."
if cached:
path = cached[0]
else:
cpu_model = getattr(rfdetr, model_name)(accept_platform_model_license=True, device="cpu")
path = Path(cpu_model.export_cpu_onnx(export_dir))
del cpu_model
return RFDETRInference(path, device="cpu")
Score and save each run¶
List the model and arm pairs still to score. A pair whose resume key is already in the results file is skipped, so rerunning after an interruption continues where it stopped. The list keeps model order, so each checkpoint loads once for all its arms.
pairs = [
(model_name, arm) for model_name in MODELS for arm in ARMS if arm != "onnx-compact" or model_name in COMPACT_MODELS
]
pending = [(name, arm) for name, arm in pairs if resume_key({**provenance, "model": name, "arm": arm}) not in completed]
print(f"{len(pending)} of {len(pairs)} runs to score; {len(pairs) - len(pending)} already scored.")
Keep only one live model in memory: load a checkpoint when the model changes, after releasing the previous one, and release each predictor before the next run. One outer bar steps through the pending pairs and names the current one; the evaluator shows its own bar over the images of that run. The evaluator calls predict once per RGB image at batch 1 with default preprocessing. Read the task from the predictor rather than guessing it from the checkpoint name. Append each completed score immediately, including full provenance and elapsed scoring time in seconds; model initialization and engine builds are excluded from that time. A later interruption preserves all completed rows.
builders = {
"pytorch-fp32": build_pytorch_fp32,
"tensorrt-fp16": build_tensorrt_fp16,
"tensorrt-int8": build_tensorrt_int8,
"onnx-compact": build_onnx_compact,
}
loaded_name, model = None, None
progress = tqdm(pending, desc="Runs", unit="run")
for model_name, arm in progress:
progress.set_postfix_str(f"{model_name} / {arm}")
if model_name != loaded_name:
model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
# The license keyword exists only on the Plus constructors.
model_class = getattr(rfdetr, model_name)
model = model_class(accept_platform_model_license=True) if model_name in PLUS_MODELS else model_class()
loaded_name = model_name
predictor = builders[arm](model, model_name)
task = predictor.runtime_info["task"]
started = time.perf_counter()
score = evaluate_coco_ap(
partial(predictor.predict, threshold=0.001, include_source_image=False),
subset,
task,
)
row = {
**provenance,
"model": model_name,
"arm": arm,
"task": task,
"scored_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"n_images": score.n_images,
"wall_seconds": time.perf_counter() - started,
"box_ap": score.box_ap,
"box_ap50": score.box_ap50,
"task_ap": score.task_ap,
"task_ap50": score.task_ap50,
}
with RESULTS_PATH.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(row, allow_nan=False) + "\n")
completed.add(resume_key(row))
rows.append(row)
task_summary = "" if score.task_ap is None else f", {task} AP {100 * score.task_ap:.1f}%"
progress.write(
f"{model_name} / {arm}: box AP {100 * score.box_ap:.1f}%{task_summary}, {score.n_images} images, {row['wall_seconds']:.1f}s"
)
del predictor
model = None
if torch.cuda.is_available():
torch.cuda.empty_cache()
Inspect saved results¶
Reload the append-only file rather than relying on in-memory state, so the displayed DataFrame includes earlier completed runs as well as this session. These raw rows preserve both AP50:95 and AP50 for boxes and each model's additional task.
results = pd.DataFrame(json.loads(line) for line in RESULTS_PATH.read_text().splitlines() if line.strip())
Print the provenance fields that every saved row shares once, rather than repeating them on each row.
shared = [field for field in provenance if results[field].astype(str).nunique(dropna=False) == 1]
for field in shared:
print(f"{field}: {results[field].iloc[0]}")
Show one row per scored model and arm with only the columns that differ between rows, AP in percent.
ap_columns = ["box_ap", "box_ap50", "task_ap", "task_ap50"]
display(results.drop(columns=shared).assign(**{column: (100 * results[column]).round(1) for column in ap_columns}))
Next steps¶
Compare the scores with the official numbers on the Benchmarks page; the harness, runtime and device differ, so state them alongside any copied results. Continue with the CUDA export cookbook to measure engine latency and the FP16 versus INT8 accuracy trade-off on your GPU.