OpenVINO IR Export

OpenVINO IR (Intermediate Representation) is a proprietary model format used by the OpenVINO Toolkit to optimize and deploy deep learning models.

Prerequisites

pip install "rfdetr[openvino]"

Basic OpenVINO Export

Object DetectionImage Segmentation

from rfdetr import RFDETRMedium

model = RFDETRMedium(pretrain_weights="<path/to/checkpoint.pth>")

model.export(format="openvino", output_dir="output")

from rfdetr import RFDETRSegMedium

model = RFDETRSegMedium(pretrain_weights="<path/to/checkpoint.pth>")

model.export(format="openvino", output_dir="output")

This produces two files (named after the model's variant):

• output/<model-variant>.xml - The model structure (Intermediate Representation)
• output/<model-variant>.bin - The model weights

OpenVINO Export with Custom Resolution

from rfdetr import RFDETRMedium

model = RFDETRMedium(pretrain_weights="<path/to/checkpoint.pth>")

model.export(format="openvino", shape=(608, 608))

OpenVINO Export with Precision

OpenVINO export defaults to FP16 weight compression. Pass openvino_precision="float32" to keep the stored IR weights at full precision (larger file, no compression) — this controls IR storage precision only. The precision the device computes in is a runtime setting, chosen when the model is compiled (see the warning under Runtime Options):

model.export(format="openvino", openvino_precision="float32")

Predict with RFDETRInference

from rfdetr import RFDETRInference

model = RFDETRInference("output/rfdetr-medium.xml")
detections = model.predict("image.jpg", threshold=0.5)

This returns the shared Supervision result types. See Predict with RFDETRInference for checkpoint inputs, metadata, and batch behavior.

runtime_options takes cache_dir, inference_precision and config, as described under Runtime Options below: reuse compiled kernels across process starts, set the precision the device computes in ("f32" by default, except on the NPU), and pass further compile properties.

model = RFDETRInference(
    "output/rfdetr-medium.xml",
    device="cpu",
    runtime_options={"cache_dir": "output/openvino-cache", "config": {"INFERENCE_NUM_THREADS": 4}},
)

Runtime Options

Execution precision: OpenVINO computes in float32 by default through RF-DETR

OpenVINO's CPU plugin picks its own execution precision, whatever precision the IR was stored in, and on hardware with native half-precision support it can compute in f16 or bf16. In f16 RF-DETR's class logits collapse: on an Apple M4 Max, RFDETRNano went from 3 detections above 0.5 to none. The same collapse in bf16 on x86 is expected but has not been measured. RFDETRInference therefore sets INFERENCE_PRECISION_HINT to "f32" unless told otherwise (on the NPU it sends no hint, because that plugin does not support f32; an explicit inference_precision is always passed through), which keeps the confident logits close to ONNX Runtime's (the CPU cookbook's parity check reports the actual drift). OpenVINO's own EXECUTION_MODE_HINT=ACCURACY is another way to ask for full precision. Pass runtime_options={"inference_precision": None} to keep the device default (faster on hardware that computes natively in reduced precision, at that accuracy cost), or another hint such as "f16" to set it explicitly. Further compile properties, such as {"INFERENCE_NUM_THREADS": 4}, go in runtime_options["config"]. Calling openvino.Core().compile_model(...) yourself leaves the device default in place, so pass {"INFERENCE_PRECISION_HINT": "f32"} there too.

Construct once, and use one instance per worker thread

Building an RFDETRInference for an OpenVINO model compiles it, which is the expensive step — do it once and reuse the instance for every image. Pass runtime_options={"cache_dir": "<dir>"} to reuse compiled kernels across process starts as well. Calls through a single instance are serialized by an internal lock, so sharing one instance across threads is safe but not faster; for parallel throughput give each worker thread its own instance.

Benchmark OpenVINO Model

Use OpenVINO's benchmark_app tool to measure performance. Pass -infer_precision f32 so it times the same execution precision RFDETRInference uses by default; without it, benchmark_app times the device's reduced-precision default:

benchmark_app -m output/rfdetr-medium.xml -data_shape [1,3,576,576] -infer_precision f32

OpenVINO Model Outputs

The exported OpenVINO IR model produces the following outputs:

• Object Detection Models:

  • Output 0: Bounding boxes [batch, 300, 4] — normalized cxcywh (center_x, center_y, width, height), not top-left xywh
  • Output 1: Class logits [batch, 300, num_classes]

• Segmentation Models:

  • Output 0: Bounding boxes [batch, 300, 4]
  • Output 1: Class logits [batch, 300, num_classes]
  • Output 2: Instance masks (if segmentation head is present)

• Keypoint Models:

  • Output 0: Bounding boxes [batch, 300, 4]
  • Output 1: Class logits [batch, 300, num_classes]
  • Output 2: Keypoints (if keypoint head is present)

INT8 Quantization

quantization="int8" compresses the IR with NNCF before it is written, so the .xml / .bin pair on disk is already 8-bit:

from rfdetr import RFDETRSmall

model = RFDETRSmall()
model.export(format="openvino", quantization="int8", calibration_data="calibration_images/")

NNCF is a separate install — pip install nncf — rather than part of the rfdetr[openvino] extra, since only this mode needs it and it carries a heavyweight dependency set an FP32 or FP16 export has no use for.

calibration_data is required: static quantization reads activation ranges from it, so it must be representative of your deployment domain. It accepts a directory of images (preprocessed exactly as predict() does, capped by max_images), a .npy file of shape (N, C, H, W) already normalized, or an equivalent array.

quantization and openvino_precision are different knobs

openvino_precision controls how IR weights are stored (FP32 vs FP16 compression) and leaves arithmetic alone. quantization="int8" changes the arithmetic. Set them independently.

The transformer recipe matters

The export asks NNCF for ModelType.TRANSFORMER rather than its generic defaults. Measured on a 500-image COCO val2017 subset, plain nncf.quantize(model, dataset) cost 5.34 mAP on Nano and 6.18 on Small — worse than a comparable ONNX Runtime configuration — because it quantizes normalization and the elementwise math around attention along with the matrix multiplies. The transformer recipe keeps those paths in float. Measure on your own data before deploying.

On the Colab x86 host of the measured table below, the INT8 IR cost 0.4 mAP against the fp32 IR (56.1 against 56.5) and ran at the same speed (274.2 against 274.6 ms at batch 1), for a file 3.6 times smaller (32.3 against 115.0 MB).

Measured Latency on an x86 CPU

Colab x86-64 host with 4 physical cores (8 logical), Linux 6.6, RFDETRSmall, rfdetr 1.12.0.dev0, ONNX Runtime 1.22.1, OpenVINO 2026.4.1, NNCF 3.4.0, 15 warmup + 50 timed runs per row, from the CPU cookbook. Every latency is the end-to-end predict() call (preprocessing, forward, postprocessing) through RFDETRInference; OpenVINO is pinned to the physical-core count. FPS is images per second. mAP@50:95 is a seeded 500-image COCO val2017 subset, scored on the same batch-1 artifact that was timed; the INT8 artifacts were calibrated on 100 other val2017 images held out of that subset. The INT8 artifacts are batch 1 only.

Runtime	Precision	Quantization	Batch	end2end [ms]	FPS [img/s]	mAP@50:95 [%]	Size [MB]	Memory [MB]
CPU	fp32	—	1	373.65 ± 61.10	2.7	56.5	115.0	319.7
CPU	fp16	—	1	452.30 ± 45.50	2.2	56.5	58.2	209.6
CPU	INT8	static NNCF	1	542.10 ± 74.40	1.8	56.1	32.3	138.1
CPU	fp32	—	4	1451.96 ± 223.26	2.8	—	115.0	639.9
CPU	fp16	—	4	1619.20 ± 249.61	2.5	—	58.2	523.8

What the table supports:

• On this host the fp32 IR (373.7 ms) is level with eager PyTorch (401.6 ms) and its TorchScript inference() snapshot (377.9 ms): the standard deviations are 15 to 16% of the mean, so none of the three can be ranked at batch 1 (see the PyTorch baseline). It is faster than ONNX Runtime fp32 (546.9 ms) by a gap well outside the noise.
• The fp16 IR (452.3 ms) is no faster than the fp32 IR, within about one standard deviation of it, and half the size. It saves disk and bandwidth, not arithmetic.
• The INT8 IR (542.1 ms) is slower than the fp32 IR on this host, though 3.6 times smaller and within 0.4 mAP. An earlier run on a different Colab host measured it at the speed of the fp32 IR, so INT8 speed on a CPU depends on the host's vector instructions and is worth measuring on the machine you deploy to.
• Batching helps little on this CPU: 2.7 FPS at batch 1, 2.8 at batch 4 for fp32.
• Predicting a folder of 64 images gave no more than batch 1: 2.8 img/s for the fp32 IR, 2.7 for fp16 and 2.0 for INT8. The model, not the input side, is the limit on a CPU.
• This run was slower and noisier than an earlier one on another Colab host (eager PyTorch 401.6 against 345.4 ms), so compare rows within this table, never against numbers from another run.

Memory column

Memory [MB] is host resident-memory growth across constructing the runtime plus its first call, in one shared process. Read it as a rough per-row indication.