RF-DETR → TFLite Export & Inference (INT8 Dynamic-Range)¶

Export an RF-DETR detector to TensorFlow Lite (.tflite) with INT8 dynamic-range quantization, then run the exported model directly with tensorflow.lite.Interpreter and visualize the detections. The TFLite pipeline converts ONNX → TensorFlow → TFLite via onnx2tf, and quantization="int8" always writes FP32 and FP16 .tflite files alongside the requested INT8 model.

Note: INT8 here means dynamic-range quantization — INT8 weights, float32 activations, with weight scales derived from the weights themselves. No calibration data is required (and supplying it does not change the result). This is not full-integer INT8, and the resulting model is not suitable for integer-only accelerators such as the Coral Edge TPU.

Python version: the [tflite] extra installs only on Python 3.12 exactly — its onnx2tf / tensorflow dependency stack pins numpy==1.26.4, which cannot be satisfied on 3.10, 3.11, 3.13, or 3.14 (see pyproject.toml). Check your interpreter before installing: a Colab runtime whose default Python has moved past 3.12 will fail the install cell below.

Warning: TFLite export is experimental. onnx2tf's output layout can change between versions, and the ONNX → TF → TFLite conversion chain introduces numerical rounding relative to the original PyTorch model — validate the exported .tflite file against a held-out evaluation set before deploying it.

1. Install¶

The [tflite] extra pulls in the full ONNX → TensorFlow → TFLite conversion stack (onnx2tf, tensorflow, onnxsim, onnx_graphsurgeon, ...). The assertion below fails fast on any interpreter other than 3.12, before the (slow) install runs.

In [ ]:

import sys

assert sys.version_info[:2] == (3, 12), (
    "rfdetr[tflite] requires Python 3.12 exactly (see pyproject.toml); this runtime is "
    f"{sys.version_info.major}.{sys.version_info.minor}. Use a Python 3.12 environment instead."
)

import sys assert sys.version_info[:2] == (3, 12), ( "rfdetr[tflite] requires Python 3.12 exactly (see pyproject.toml); this runtime is " f"{sys.version_info.major}.{sys.version_info.minor}. Use a Python 3.12 environment instead." )

In [ ]:

!pip install -q "rfdetr[tflite]>=1.10.1" supervision

!pip install -q "rfdetr[tflite]>=1.10.1" supervision

2. Setup¶

TFLite export itself runs on CPU — no GPU is needed.

In [ ]:

from pathlib import Path

from rfdetr import RFDETRSmall

EXPORT_DIR = Path("export_tflite")
EXPORT_DIR.mkdir(exist_ok=True)

from pathlib import Path from rfdetr import RFDETRSmall EXPORT_DIR = Path("export_tflite") EXPORT_DIR.mkdir(exist_ok=True)

3. Export to TFLite (INT8 dynamic-range)¶

The COCO-pretrained RFDETRSmall is exported directly — pass pretrain_weights="<path/to/checkpoint.pth>" to export a fine-tuned model instead.

quantization="int8" requests dynamic-range INT8; static full-integer INT8 (the mode that would need calibration data) is intentionally unsupported because RF-DETR's transformer activations do not survive it. This call also writes a *_fp32.tflite and *_fp16.tflite file alongside the requested INT8 file.

TFLite filenames are derived from the model variant (rfdetr-small, here). When GridSample ops are patched — the standard RF-DETR path, since RF-DETR's deformable attention uses GridSample — they also carry a _gs_patched infix, e.g. rfdetr-small_gs_patched_dynamic_range_quant.tflite. export() returns the resolved path directly, so nothing needs to be hardcoded here.

In [ ]:

model = RFDETRSmall()

tflite_path = model.export(format="tflite", quantization="int8", output_dir=str(EXPORT_DIR))
print(f"TFLite INT8 (dynamic-range) model: {tflite_path}  ({tflite_path.stat().st_size / 1e6:.1f} MB)")

model = RFDETRSmall() tflite_path = model.export(format="tflite", quantization="int8", output_dir=str(EXPORT_DIR)) print(f"TFLite INT8 (dynamic-range) model: {tflite_path} ({tflite_path.stat().st_size / 1e6:.1f} MB)")

4. Sample image¶

A single street scene with several COCO classes (dog, person, backpack, car) is enough to verify detections. The image is downloaded once and reused.

In [ ]:

import urllib.request

from PIL import Image

IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg"
IMAGE_PATH = EXPORT_DIR / "sample.jpg"
if not IMAGE_PATH.exists():
    urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH)

image = Image.open(IMAGE_PATH).convert("RGB")
print(f"Sample image: {image.size[0]}×{image.size[1]}")

import urllib.request from PIL import Image IMAGE_URL = "https://media.roboflow.com/notebooks/examples/dog.jpeg" IMAGE_PATH = EXPORT_DIR / "sample.jpg" if not IMAGE_PATH.exists(): urllib.request.urlretrieve(IMAGE_URL, IMAGE_PATH) image = Image.open(IMAGE_PATH).convert("RGB") print(f"Sample image: {image.size[0]}×{image.size[1]}")

5. FP32 baseline (PyTorch)¶

Before trusting the INT8 export, run the same image through the eager PyTorch model to get a fp32 baseline. predict() handles preprocessing, inference, and postprocessing (including the confidence threshold) in one call and returns a ready-to-use supervision.Detections.

In [ ]:

CONFIDENCE_THRESHOLD = 0.5

baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD)
print(
    f"PyTorch fp32 baseline: {len(baseline_detections)} detections, "
    f"top score {baseline_detections.confidence.max():.3f}"
)

CONFIDENCE_THRESHOLD = 0.5 baseline_detections = model.predict(image, threshold=CONFIDENCE_THRESHOLD) print( f"PyTorch fp32 baseline: {len(baseline_detections)} detections, " f"top score {baseline_detections.confidence.max():.3f}" )

6. Run the INT8 model with tensorflow.lite.Interpreter¶

rfdetr[tflite] already installs tensorflow, so tensorflow.lite.Interpreter needs no extra install (a standalone tflite-runtime package is a lighter-weight alternative on edge devices — see the Export docs).

The TFLite input is NHWC, not the NCHW layout ONNX/ExecuTorch/CoreML use. Preprocessing reuses infer_transforms for the resize + ImageNet-normalization pipeline, then permutes to (1, H, W, 3) before feeding the interpreter.

onnx2tf's SavedModel conversion route renames every output — the ONNX dets / labels names arrive as StatefulPartitionedCall:0 / :1 in the .tflite file and are no longer distinguishable by name. Match outputs by rank and last dimension instead: boxes are the rank-3 tensor with last dimension 4, logits are the other rank-3 tensor.

In [ ]:

import numpy as np
import tensorflow as tf
import torch

from rfdetr.export.benchmark import infer_transforms, post_process

resolution = model.model_config.resolution
image_tensor, _ = infer_transforms((resolution, resolution))(image, None)
# TFLite expects NHWC, not the NCHW layout used by the other export formats.
image_array = image_tensor.permute(1, 2, 0)[None].contiguous().numpy().astype(np.float32)

interpreter = tf.lite.Interpreter(model_path=str(tflite_path))
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
interpreter.set_tensor(input_details[0]["index"], image_array)
interpreter.invoke()

# onnx2tf strips the dets/labels names, so match by rank and last dimension instead — boxes are
# the rank-3 tensor with last dim 4, logits are the other rank-3 tensor.
rank3 = [detail for detail in output_details if len(detail["shape"]) == 3]
boxes_detail = next(detail for detail in rank3 if detail["shape"][-1] == 4)
labels_detail = next(detail for detail in rank3 if detail["shape"][-1] != 4)

dets = torch.from_numpy(interpreter.get_tensor(boxes_detail["index"]))
labels = torch.from_numpy(interpreter.get_tensor(labels_detail["index"]))

target_sizes = torch.tensor([[image.height, image.width]])
result = post_process({"dets": dets, "labels": labels}, target_sizes)[0]

keep = result["scores"] > CONFIDENCE_THRESHOLD
print(
    f"TFLite INT8: {int(keep.sum())} detections above {CONFIDENCE_THRESHOLD}, "
    f"top score {result['scores'][0]:.3f} "
    f"(fp32 baseline top score {baseline_detections.confidence.max():.3f})"
)

import numpy as np import tensorflow as tf import torch from rfdetr.export.benchmark import infer_transforms, post_process resolution = model.model_config.resolution image_tensor, _ = infer_transforms((resolution, resolution))(image, None) # TFLite expects NHWC, not the NCHW layout used by the other export formats. image_array = image_tensor.permute(1, 2, 0)[None].contiguous().numpy().astype(np.float32) interpreter = tf.lite.Interpreter(model_path=str(tflite_path)) interpreter.allocate_tensors() input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() interpreter.set_tensor(input_details[0]["index"], image_array) interpreter.invoke() # onnx2tf strips the dets/labels names, so match by rank and last dimension instead — boxes are # the rank-3 tensor with last dim 4, logits are the other rank-3 tensor. rank3 = [detail for detail in output_details if len(detail["shape"]) == 3] boxes_detail = next(detail for detail in rank3 if detail["shape"][-1] == 4) labels_detail = next(detail for detail in rank3 if detail["shape"][-1] != 4) dets = torch.from_numpy(interpreter.get_tensor(boxes_detail["index"])) labels = torch.from_numpy(interpreter.get_tensor(labels_detail["index"])) target_sizes = torch.tensor([[image.height, image.width]]) result = post_process({"dets": dets, "labels": labels}, target_sizes)[0] keep = result["scores"] > CONFIDENCE_THRESHOLD print( f"TFLite INT8: {int(keep.sum())} detections above {CONFIDENCE_THRESHOLD}, " f"top score {result['scores'][0]:.3f} " f"(fp32 baseline top score {baseline_detections.confidence.max():.3f})" )

7. Visualize with supervision¶

Wrap the kept TFLite detections in a supervision.Detections and annotate the original image.

Annotate the PIL image rather than np.array(image). supervision's annotators accept either, but a numpy scene is assumed to be OpenCV-style BGR — handing them an RGB array swaps the red and blue channels of the annotations, and sv.plot_image then swaps the photo itself. Passing PIL lets supervision convert in both directions and keeps the colours right.

In [ ]:

import supervision as sv

from rfdetr.assets.coco_classes import COCO_CLASSES

detections = sv.Detections(
    xyxy=result["boxes"][keep].numpy(),
    confidence=result["scores"][keep].numpy(),
    class_id=result["labels"][keep].numpy().astype(int),
)
print(f"Kept {len(detections)} detections above {CONFIDENCE_THRESHOLD}")

names = [COCO_CLASSES.get(int(c), str(c)) for c in detections.class_id]
annotation_labels = [f"{name} {conf:.2f}" for name, conf in zip(names, detections.confidence)]
print(f"Detections: {annotation_labels}")

annotated = sv.BoxAnnotator(thickness=3).annotate(scene=image.copy(), detections=detections)
annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate(
    scene=annotated, detections=detections, labels=annotation_labels
)

OUTPUT_PATH = EXPORT_DIR / "annotated_tflite.jpg"
annotated.save(OUTPUT_PATH)
print(f"Saved annotated image: {OUTPUT_PATH}")
sv.plot_image(annotated)

import supervision as sv from rfdetr.assets.coco_classes import COCO_CLASSES detections = sv.Detections( xyxy=result["boxes"][keep].numpy(), confidence=result["scores"][keep].numpy(), class_id=result["labels"][keep].numpy().astype(int), ) print(f"Kept {len(detections)} detections above {CONFIDENCE_THRESHOLD}") names = [COCO_CLASSES.get(int(c), str(c)) for c in detections.class_id] annotation_labels = [f"{name} {conf:.2f}" for name, conf in zip(names, detections.confidence)] print(f"Detections: {annotation_labels}") annotated = sv.BoxAnnotator(thickness=3).annotate(scene=image.copy(), detections=detections) annotated = sv.LabelAnnotator(text_scale=0.6, text_thickness=1, text_padding=4).annotate( scene=annotated, detections=detections, labels=annotation_labels ) OUTPUT_PATH = EXPORT_DIR / "annotated_tflite.jpg" annotated.save(OUTPUT_PATH) print(f"Saved annotated image: {OUTPUT_PATH}") sv.plot_image(annotated)

Next steps¶

• Deploy on-device — copy the .tflite file to your Android / edge app and run it with the TensorFlow Lite runtime for that platform.
• Fine-tuned weights — pass pretrain_weights="<path/to/checkpoint.pth>" when constructing the model.
• Other quantization modes — quantization="fp16" (or omit quantization for the FP32 / FP16 pair only) trades the INT8 size/latency win for tighter numeric parity with PyTorch.
• Other runtimes — see the ExecuTorch cookbook (mobile/edge via torch.export) or the TensorRT cookbook (NVIDIA GPU).
• See the Export documentation for every TFLite option, including calibration-data arguments and known instabilities.