RS Infer
Self-hosted ONNX inference: embeddings, rerankers, PII, and zero-shot classification — one binary, one API.
$ make cpu
$ ./target/release/rsinfer-server --config config.yaml
ready models fetched from HF Hub · listening on :8080
$ curl -s localhost:8080/v1/embeddings -d '{"model":"bge-m3","input":"hello world"}'
{"data":[{"embedding":[0.012,-0.048,…],"index":0}]}Four model families, one API
Mix and match models in config.yaml — each type gets its own routes, defaults, and per-model limits.
Embeddings
OpenAI/vLLM-compatible vectors with Matryoshka truncation and base64 responses.
/v1/embeddings · /embed
Rerankers
Cross-encoder scoring for (query, document) pairs — Jina/Cohere-compatible API.
/v1/rerank · /v1/score
PII detection
Zero-shot NER over configurable labels, with span offsets and redacted output.
/pii/detect · /pii/redact
Zero-shot classification
Classify into arbitrary labels, or reduce NLI entailment to a true/false judgement.
/classify/zero-shot · /classify/true-false
Models from the Hub
Declare models in YAML — fetched from Hugging Face at startup, or served from a local directory. No manual export step.
Any execution backend
One binary, build profiles for CPU, CoreML/ANE on macOS, CUDA and TensorRT on Linux. EPs are matched automatically per model.
Operations-friendly
Prometheus metrics on /metrics, /health and /v1/models probes, request timeouts, body limits, and per-model queues.
Running in under five minutes
Build for your platform, point at a config, and the models take care of themselves.