Letting an application send RAG requests without ever holding a provider key

In an LLM feature that runs on your own documents, deciding to use RAG is the easy half. The hard half is what carries that retrieval in production.

What is left after the decision

The documents are embedded, the chunks are in the database, search works. Three questions remain: where the provider key sits, which application uses which model, and where spend gets cut off. None of the three is part of retrieval, and solving all three inside every application means copying the same code once per application.

Ragmux moves them out of the application and into the gateway between it and the providers. The application talks to one OpenAI-compatible endpoint; which provider the request reaches, which system prompt it carries and which document store it draws context from is decided by the project’s connection.

The model name the client sends selects nothing

The model field does no routing. Which model the request reaches is decided by the project’s connection; the client’s value is echoed back unchanged in the response and in every stream chunk, so SDKs that compare that field keep working.

What that buys: the provider credential stays on the server, encrypted, and is never read back through any interface. All the application holds is the project’s own key. The cost is plain too — the client can no longer choose a model; model choice stops being a deployment decision and becomes a configuration one.

Budgets are capped in tokens, not in money

Every request’s estimated cost is calculated and shown on the dashboard, but cost is informational: no request is ever refused because a monetary threshold was reached. The way to cap spend is to cap tokens — requests and tokens per minute per project, daily and monthly token budgets.

There is a limit here that is not hidden: token counts are only final once the provider has answered, so a single request can overshoot a budget by its own size, and it is the next one that gets refused.

One PostgreSQL, no second store

Users, connections, projects, documents, chunks, vectors, request logs and usage counters all live in one PostgreSQL with pgvector. No Redis, no separate vector database. Because a counter is a row rather than a process variable, several replicas pointed at the same database share one budget.

The review that found the product breaking its own rule

Three of the five recorded decisions came out of a pre-release review, two of them from places where code and documentation disagreed: the role matrix described viewer as read-only while the code let a viewer mint their own gateway key; a price-table row reported a connection pointed at a paid endpoint as “$0.00”.

The fifth decision was softened within a day of being taken. A hard per-tenant share had been put on the image-download ceiling; measurement showed that on a single-project install — the common case — it left half the ceiling permanently idle. The share was loosened the same day: when nobody is waiting, one tenant may use the whole ceiling.

The product’s own entry is in the portfolio.