Configuration

The server reads a YAML file (default config.yaml). A fully-commented example lives at configs/config.example.yaml.

The file has two top-level sections: server (global options) and models (a list of model definitions).

CLI

Serve mode (the default) takes a single flag:

rsinfer-server --config configs/config.example.yaml

rsinfer-server download <repo> <out>

Downloads a model's ONNX graph + tokenizer files from the Hugging Face Hub into <out> (created if missing), so it can be referenced with path: and served without any startup download:

rsinfer-server download Xenova/multilingual-e5-small models/e5-small

HF_HOME, HF_ENDPOINT (Hub mirror) and HF_TOKEN (gated repos) are honored; the download lands in <out> as a self-contained folder (graph keeps its repo-relative path, e.g. onnx/model.onnx + onnx/model.onnx_data, with tokenizer.json and config.json at the folder root).

Verbosity is controlled by the RUST_LOG environment variable (see Logging).

Server options

server:
  bind: 0.0.0.0:8080
  max_body_mb: 32
  request_timeout_ms: 120000
  max_queue: 256
  queue_timeout_ms: 30000
  # hf_cache_dir: .hf-cache

Model options

Each entry in models selects a kind (pipeline) and a model source (hf repo id or a local path). Options common to all kinds:

Kind-specific options

Embedding (kind: embedding)

Rerank (kind: rerank)

Zero-shot / true-false (kind: zeroshot)

PII / NER (kind: pii)

INT8 by default

On CPU-only execution (dtype: auto, the default) the server runs an int8 version of the model it quantizes itself from the fp32 graph, the first time the model loads:

• MatMul weights become int8 with one scale per output column (activations are quantized dynamically, like onnxruntime.quantization.quantize_dynamic), and large embedding tables become int8 with one scale per row. Same speed as the per-tensor int8 files published on the Hub, closer to fp32 (e5-small: embedding cosine to fp32 ≥ 0.995 vs ≥ 0.984), and much less memory because the embedding table stays int8 (e5-small, 4 replicas: 384 MB vs 879 MB idle). e5-small vs fp32 on a 14-core i7, 4 replicas: 1018 vs 582 req/s on 1-text requests, 170 vs 130 texts/s on 32-passage requests.
• A quality check then runs built-in sample inputs through both graphs and keeps int8 only if it stays close to fp32: embedding/logit cosine ≥ 0.99, rerank score within 0.05, PII token labels agreeing ≥ 97% (worst row). The verdict is logged (int8 quality check passed / ... drifts too far) and the model otherwise runs in fp32 — size memory for fp32 if the check may fail.
• The int8 graph and the verdict are cached under <hf cache>/rsinfer/int8/ (server.hf_cache_dir, else HF_HOME, else ~/.cache/huggingface), so later starts skip both steps. Delete that folder to redo them.

Models measured on CPU: e5-small and ms-marco pass; the ai4privacy DistilBERT PII model and Qwen3-Embedding-0.6B fail and stay in fp32. Decoder-style models have activation outliers that dynamic int8 cannot represent — the Hub's own model_int8.onnx for Qwen3-Embedding-0.6B only reaches cosine 0.77–0.84 to fp32.

Upgrading: int8 embeddings differ slightly from fp32 ones. If an existing vector index was built with fp32 embeddings, either rebuild it or set dtype: fp32 on the model.

Sizing memory and throughput

• Each replica holds its own copy of the weights (ONNX Runtime sessions cannot safely run concurrently, so concurrency comes from replicas). Budget roughly the model's file size plus runtime buffers per replica: e5-small fp32 is ~550 MB per replica (its 250k-token embedding table dominates), Qwen3-Embedding-0.6B int8 ~1 GB, fp32 well over 3 GB. dtype: int8 divides weight memory by ~4.
• Peak memory under load is driven by the batch size in padded tokens. batching.max_tokens bounds it per forward (times the number of replicas running at once). Lower it for large models: Qwen3-Embedding-0.6B int8 with 2 replicas and max_tokens: 4096 peaks at ~4.6 GB under sustained load.
• Replicas vs throughput: with batching on, 2–4 replicas keep the CPU busy; on a 14-core i7, 4 replicas served ~30–50% more short requests than 2.
• Disabling ONNX Runtime's CPU memory arena does not save memory — it made peak usage worse in tests (allocator fragmentation).

Execution providers

eps is a priority list. Entries not compiled into the binary, or absent from the linked ONNX Runtime build, are skipped with a warning; CPU is always the final fallback. GET /v1/models reports what each model actually uses.

OpenVINO (experimental). make cpu-openvino builds a server that loads ONNX Runtime at startup instead of linking it: fetch Intel's runtime (ONNX Runtime 1.24 + OpenVINO 2025.4, ~215 MB) with scripts/fetch-openvino-runtime.sh /opt/onnxruntime-openvino and start the server with ORT_DYLIB_PATH=/opt/onnxruntime-openvino/libonnxruntime.so.1.24.1 (Docker image: --build-arg FEATURES=ep-openvino --target runtime-openvino). Models opt in with eps: [openvino, cpu]; openvino_device picks CPU (default), GPU, NPU or AUTO. Models with eps: [cpu] keep running on ONNX Runtime's CPU kernels in the same server, int8 included.

Measured on a 14-core i7-13850HX CPU, it does not beat the default build:

• OpenVINO runs fp32 graphs ~1.2–1.3x faster than ONNX Runtime at a fixed input shape, but dynamic int8 graphs 2–4x slower, so dtype: auto serves the published fp32 graph on it — and the default build's int8 path is faster still.
• In the server (batched, varying shapes), a fp32 PII model on OpenVINO with 1 replica (it parallelizes internally; use replicas: 1) matched ONNX Runtime with 2 replicas: 294 vs 330 req/s on 1-text requests, 54 vs 51 texts/s on 32-text requests, with a lower memory peak (1.5 vs 2.6 GB).

Its use is therefore Intel GPUs/NPUs (Core Ultra) or fp32-only models on Intel hosts — measure on your hardware before adopting it.

CoreML caveat: for small BERT-size encoders CoreML often ends up slower than CPU (ORT splits the graph into many CPU↔CoreML partitions). Default to eps: [cpu] for small encoders and benchmark per model.

models:
  - name: e5-small
    kind: embedding
    hf: Xenova/multilingual-e5-small
    max_len: 512
    replicas: 2
    eps: [coreml, cpu]
    pooling: mean
    normalize: true

Logging

Logging uses tracing and is configured via the RUST_LOG environment variable:

RUST_LOG=info ./target/release/rsinfer-server --config config.yaml   # startup + request summaries
RUST_LOG=debug ./target/release/rsinfer-server --config config.yaml  # + EP/session/file-resolution detail

Per-crate overrides are supported, e.g. RUST_LOG=info,hf_hub=debug to watch downloads only.