Usage

Quick start (local)

# CPU-only build (works everywhere)
cargo build --release -p rsinfer-server

# or macOS + CoreML / Linux + CUDA / TensorRT variants — see the Overview page
make mac-coreml

# point at a config and run
cp configs/config.example.yaml config.yaml
./target/release/rsinfer-server --config config.yaml

# verify it is up (requires jq)
./scripts/smoke.sh

On first boot the configured models are downloaded from the Hugging Face Hub into the cache (HF_HOME respected, or the config's hf_cache_dir). Startup logs show each model as it is resolved and loaded.

Running with Docker

The Dockerfile builds a slim runtime image (debian:bookworm-slim) from the Rust source. The execution providers are chosen at compile time via the FEATURES build arg — GPU EP combos are ep-cuda, ep-cuda,ep-tensorrt, or ep-nvrtx,lax-ep-matching (the default).

# CPU-only image (works everywhere, smallest deps)
docker build --build-arg FEATURES= -t rsinfer:cpu .

# CUDA + TensorRT image (default)
docker build -t rsinfer:gpu .

The image runs the binary as a non-root user, listens on port 8080, and expects the config at /etc/rsinfer/config.yaml. Mount your model cache and config, then run:

docker run --rm -p 8080:8080 \
  -v "$PWD/config.yaml:/etc/rsinfer/config.yaml:ro" \
  -v rsinfer-hf-cache:/home/rsinfer/.cache/huggingface \
  rsinfer:cpu

# NVIDIA GPUs: pass the devices through and install the CUDA/cuDNN runtime libs
# (or mount them) — the base image does not ship the GPU libraries.
docker run --rm --gpus all -p 8080:8080 \
  -v "$PWD/config.yaml:/etc/rsinfer/config.yaml:ro" \
  rsinfer:gpu

Notes:

• bind in the config still controls the listen address; leave it on 0.0.0.0:8080 so the port is reachable from the host.
• Use a named volume (-v rsinfer-hf-cache:...) for the HF cache so downloads survive container restarts, or set server.hf_cache_dir.
• For GPU images, the CUDA/cuDNN/TensorRT shared libraries must be present in the container (bake them into a derived image or mount them); the slim base only installs ca-certificates, libgomp1, and curl.
• The server has no built-in auth/TLS — put it behind a reverse proxy.

Endpoints & examples

Every endpoint (embeddings, rerank, PII, zero-shot, true/false) has a full OpenAPI page with request/response schemas, sample payloads, and an interactive playground in the Server Endpoints.

Adding a model

1. Find a suitable ONNX export on the Hugging Face Hub (onnx-community/*, Xenova/* repos, or run python/convert_to_onnx.py).
2. Add an entry to your config under models:

models:
  - name: my-model
    kind: embedding          # embedding | rerank | zeroshot | pii
    hf: onnx-community/some-encoder
    max_len: 512
    replicas: 2

3. Restart the server. GET /v1/models lists it once loaded.

The engine is model-family agnostic within each kind: it intersects the tokenizer outputs (input_ids, attention_mask, token_type_ids) with the model's declared inputs and reads conventional output names (logits, sentence_embedding, last_hidden_state, token_embeddings, output0). Anything exported by optimum-cli export onnx (or the Xenova/onnx-community repos) is compatible.

See the Configuration page for the full list of options.