RF-DETR → TensorRT Export & Inference¶

Export an RF-DETR detector to a TensorRT engine (.trt), then run inference with inference-models — the recommended multi-backend runtime — using its TensorRT backend.

Step	What
Export	RFDETRSmall → rfdetr-small.trt (FP16) — raw-TensorRT deployment artifact
Infer	inference-models AutoModel(backend=TRT) → supervision detections

Two engines, on purpose. The .trt from the export step is a standalone artifact for raw TensorRT deployment, locked to this GPU + TensorRT version. inference-models builds and manages its own engine internally, so it does not load that file — it is the easier, portable path and handles preprocessing, postprocessing, and class names for you.

1. Install¶

rfdetr[tensorrt] provides the exporter; inference-models[trt10] provides the TensorRT inference runtime.

Colab ships mutually inconsistent preinstalled packages that otherwise crash import rfdetr, so two are aligned: torchaudio is uninstalled (RF-DETR never uses it, but transformers imports it when present and a torch/torchaudio CUDA-version mismatch then errors), and pillow is force-reinstalled to a clean version (a half-upgraded PIL breaks torchvision's import with cannot import name '_Ink').

Colab: select a GPU runtime (Runtime → Change runtime type → GPU), and after this cell Runtime → Restart session before running the next cell — Colab keeps old package versions loaded until a restart.

In [ ]:

!pip install -q "rfdetr[onnx,tensorrt]>=1.9.0" "inference-models[trt10]" supervision
!pip install -q --force-reinstall --no-deps "pillow==11.3.0"
!pip uninstall -q -y torchaudio

!pip install -q "rfdetr[onnx,tensorrt]>=1.9.0" "inference-models[trt10]" supervision !pip install -q --force-reinstall --no-deps "pillow==11.3.0" !pip uninstall -q -y torchaudio

2. Setup and GPU check¶

TensorRT export and inference both require CUDA — fail fast with a clear message if no GPU is visible.

In [ ]:

from pathlib import Path

import numpy as np
import torch

if not torch.cuda.is_available():
    raise RuntimeError("TensorRT export and inference require a CUDA GPU; none is available.")
print(f"GPU: {torch.cuda.get_device_name(0)}")

EXPORT_DIR = Path("export_tensorrt")
EXPORT_DIR.mkdir(exist_ok=True)
CONFIDENCE_THRESHOLD = 0.5

from pathlib import Path import numpy as np import torch if not torch.cuda.is_available(): raise RuntimeError("TensorRT export and inference require a CUDA GPU; none is available.") print(f"GPU: {torch.cuda.get_device_name(0)}") EXPORT_DIR = Path("export_tensorrt") EXPORT_DIR.mkdir(exist_ok=True) CONFIDENCE_THRESHOLD = 0.5

3. Sample image¶

A single street scene with several COCO classes (dog, bicycle, car) is enough to verify detections. The image is downloaded once and reused.

In [ ]:

import urllib.request

from PIL import Image

IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")

import urllib.request from PIL import Image IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg" IMAGE_PATH = EXPORT_DIR / "sample.jpg" if not IMAGE_PATH.exists(): urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH) image = Image.open(IMAGE_PATH).convert("RGB") print(f"Sample image: {image.size[0]}×{image.size[1]}")

4. Export a TensorRT engine¶

format="trt" (alias of "tensorrt") compiles the engine from the COCO-pretrained RFDETRSmall — pass pretrain_weights="<path/to/checkpoint.pth>" to export your own fine-tuned model instead. Compilation is the slow step (tens of seconds to a few minutes); it writes the .trt file to output_dir.

The engine is FP16 by default (lowest latency); it automatically falls back to FP32 (with a warning) if your TensorRT build doesn't expose the FP16 builder flag. Pass fp16=False to force FP32 explicitly.

This .trt is the raw-deployment artifact. The inference step below uses inference-models, which builds its own engine — it does not load this file (see the intro note).

In [ ]:

from rfdetr import RFDETRSmall

model = RFDETRSmall()

engine_path = model.export(format="trt", output_dir=str(EXPORT_DIR))
print(f"TensorRT engine: {engine_path}  ({engine_path.stat().st_size / 1e6:.1f} MB)")

from rfdetr import RFDETRSmall model = RFDETRSmall() engine_path = model.export(format="trt", output_dir=str(EXPORT_DIR)) print(f"TensorRT engine: {engine_path} ({engine_path.stat().st_size / 1e6:.1f} MB)")

5. Run inference with inference-models¶

AutoModel.from_pretrained("rfdetr-small", backend=BackendType.TRT) loads the model and builds a TensorRT engine for inference. Calling it on an image returns per-image predictions; .to_supervision() converts them to a supervision.Detections with class names already attached — no manual preprocessing, box decoding, or COCO-id mapping.

In [ ]:

from inference_models import AutoModel, BackendType

trt_model = AutoModel.from_pretrained("rfdetr-small", backend=BackendType.TRT)

predictions = trt_model(np.array(image))
detections = predictions[0].to_supervision()
detections = detections[detections.confidence > CONFIDENCE_THRESHOLD]
print(f"Kept {len(detections)} detections above {CONFIDENCE_THRESHOLD}")

from inference_models import AutoModel, BackendType trt_model = AutoModel.from_pretrained("rfdetr-small", backend=BackendType.TRT) predictions = trt_model(np.array(image)) detections = predictions[0].to_supervision() detections = detections[detections.confidence > CONFIDENCE_THRESHOLD] print(f"Kept {len(detections)} detections above {CONFIDENCE_THRESHOLD}")

6. Visualize with supervision¶

to_supervision() attaches class names under detections.data["class_name"]; fall back to the raw class_id if absent. Annotate the original image with boxes and labels.

In [ ]:

import supervision as sv

class_names = list(detections.data.get("class_name", [str(c) for c in detections.class_id]))
labels = [f"{name} {conf:.2f}" for name, conf in zip(class_names, detections.confidence)]
print(f"Detections: {labels}")

annotated = sv.BoxAnnotator(thickness=3).annotate(scene=np.array(image).copy(), detections=detections)
annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate(
    scene=annotated, detections=detections, labels=labels
)

OUTPUT_PATH = EXPORT_DIR / "annotated_tensorrt.jpg"
Image.fromarray(annotated).save(OUTPUT_PATH)
print(f"Saved annotated image: {OUTPUT_PATH}")
sv.plot_image(annotated)

import supervision as sv class_names = list(detections.data.get("class_name", [str(c) for c in detections.class_id])) labels = [f"{name} {conf:.2f}" for name, conf in zip(class_names, detections.confidence)] print(f"Detections: {labels}") annotated = sv.BoxAnnotator(thickness=3).annotate(scene=np.array(image).copy(), detections=detections) annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate( scene=annotated, detections=detections, labels=labels ) OUTPUT_PATH = EXPORT_DIR / "annotated_tensorrt.jpg" Image.fromarray(annotated).save(OUTPUT_PATH) print(f"Saved annotated image: {OUTPUT_PATH}") sv.plot_image(annotated)

Next steps¶

• Backend selection — AutoModel.from_pretrained also accepts backend="onnx" or backend="torch"; drop the backend= argument to let it pick the best available automatically.
• Local checkpoint — AutoModel.from_pretrained("<path/to/checkpoint.pth>", model_type="rfdetr-small") loads your own fine-tuned weights.
• Raw .trt deployment — use the engine exported in step 4 directly with the TensorRT runtime when you need a standalone artifact (no inference-models dependency).
• See the Export documentation for all formats and options.