You chose RAG. The real work starts now

RAG or fine-tuning is written about everywhere; what you operate after choosing RAG is written about nowhere. Where the key sits, which application uses which model, where spend is cut off — all three begin where those articles end.

This is the story of how the work that surrounds retrieval turned into four design decisions while I was building an LLM feature that runs on my own documents. The short answer: deciding on RAG is the easy half; the hard half is the gateway that carries it. What built those four is Ragmux.

What makes fine-tuning expensive is not the GPU

A correction first, because my own reasoning was incomplete. I used to think training or fine-tuning a model was out of reach for a small team. Half of that is no longer true: according to the framework in Winder.AI: RAG vs Fine-Tuning in 2026: A Decision Framework for LLM Teams, the GPU cost of a LoRA fine-tune now sits somewhere between fifty and five hundred dollars, and a few hundred good examples can be enough for classification-shaped tasks.

What has not come down is the cost of producing those examples. The same source puts a clean, labelled, deduplicated training set of a thousand examples at one to two weeks of a senior engineer’s time — so what was expensive was never the GPU, it was the data.

But the deciding factor for me was not even cost. A fine-tuned model carries frozen knowledge: when the documentation changes, you retrain it; you cannot ask which document an answer came from, and swapping the base model means doing the work again. What makes RAG the choice is not that it is cheap but that the source stays separate from the model: change the document and the answer changes; change the model and the document stays where it is.

What is left once the easy half is done

Say the decision is made and the work is done: the documents are parsed, chunked and embedded, and search works. At that point three questions remain, and none of them is part of retrieval:

• Where does the provider key sit?
• Which application uses which model?
• Where does spend get cut off?

Solving all three inside every application means copying the same code once per application — and spreading the key once per application. Ragmux moves them out of the application and into the gateway between it and the providers: the application talks to one OpenAI-compatible endpoint, and in the official SDKs the only things that change are base_url and api_key.

First decision: the model field selects nothing

The model field in the request does no routing. Which model the request reaches is decided by the project’s connection; the client’s value is only echoed back unchanged in the response and in every stream chunk, so SDKs that compare that field keep working.

What that buys is that the provider credential never reaches the application: it sits encrypted on the server and is never read back through any interface. All the application holds is the project’s own key, and if that key leaks, what you rotate is not your provider key.

The cost is plain too: the client can no longer choose a model. Model choice stops being a deployment decision and becomes a configuration one — changing provider no longer needs a redeploy, but an application can no longer say “send this one to that model” either.

Second decision: budgets are capped in tokens, not money

Every request’s estimated cost is calculated and shown on the dashboard. But cost is informational: no request is ever refused because a monetary threshold was reached. The way to cap spend is to cap tokens — requests and tokens per minute per project, daily and monthly token budgets.

That is a deliberate limit. A request’s real cost is only known once the provider has answered, so enforcing a ceiling in money would be stacking an estimate on an estimate. Tokens carry the same problem at a smaller scale, and it is not hidden: a single request can overshoot a budget by its own size, and it is the next one that gets refused.

A refused request never reaches the provider, so it costs nothing; and limits are checked before retrieval, so a throttled client does not even spend an embedding call.

Third decision: one database, no second store

Users, connections, projects, documents, chunks, vectors, request logs and usage counters all live in one PostgreSQL with pgvector. No Redis, no separate vector database.

That is more than a preference for simplicity. Because a counter is a row rather than a process variable, several replicas against the same database share one budget; a session cookie issued by one replica authenticates on another; document ingestion is distributed across replicas under a lease. Scaling out is not building a coordination layer, it is starting a few processes against the same database.

The cost: everything resting on one PostgreSQL means sizing that PostgreSQL accordingly — and because the default single-container layout hosts the database inside itself, it is deliberately single-instance.

Fourth decision: a retrieval failure does not drop the request

If the reranking service does not answer, if the search backend is unavailable, if context cannot be fetched — the request is not cancelled. The event is logged, whatever ordering exists is used, and the request continues without context. To be honest about it, nothing in the response tells the two apart: context that could not be fetched and context that matched nothing both come back as the same zero-hit count, and the difference lives only in the log.

The opposite — returning a 500 when there is no context — looks more honest and is worse in practice: instead of a slightly weaker answer, the user gets none. The choice here is not between answering wrongly in silence and failing visibly; it is between a weaker answer and no answer at all — with the log, not the response, carrying the reason.

The review that found the product breaking its own rule

Three of the five recorded decisions came out of a pre-release review, two of them from places where the code and the documentation disagreed. The role matrix described viewer as read-only while the code let a viewer mint their own gateway key — so a “reader” could spend money. A price-table row reported a connection pointed at a paid endpoint as “$0.00”; a missing price looked like a price of zero.

The fifth decision was softened within a day of being taken. A hard per-tenant share had been put on the image-download ceiling. Measurement showed that on a single-project install — the common case — it left half the ceiling permanently idle. The share was loosened the same day: when nobody is waiting, one tenant may use the whole ceiling.

The reason I write those three down: what interests me about a tool is less which decisions it made than how those decisions get audited. A project that can find itself contradicting its own documentation is more trustworthy than one that cannot.

What this costs in total

What I gained: the provider key sits in one place, which application uses which model is a line of configuration, and spend is capped per project. Adding an application is not handing out another key, it is opening another project.

What I pay: the client cannot choose a model, budgets are capped in tokens rather than money and a single request can overshoot by its own size, everything rests on one PostgreSQL, and an answer is still produced when context could not be fetched.

All four came out of a decision, and the opposite of all four is defensible. When you are building a gateway, the real question is not which one is right; it is which cost you are paying on purpose.

The route I took and the reasoning behind the decisions are in the Labs entry; the product itself is in the portfolio. The tool is open source and runs on your own server.

---

Sources

• Winder.AI: RAG vs Fine-Tuning in 2026: A Decision Framework for LLM Teams