# How should I abstract the LLM provider (Ollama → Bedrock/OpenAI)?

> Build one narrow complete(prompt, opts) interface with three implementations picked by config, and treat context window, latency and cost as config.

- Asked: 2026-05-03
- Answered: 2026-05-06
- Asked by: Deniz
- Tags: mimari
- Source: https://www.muhammetsafak.com.tr/en/just-ask/abstracting-the-llm-provider-from-ollama-to-bedrock-or-openai/
- Language: en-US
- Author: Muhammet Şafak

---
**Question:** We're adding an AI feature that does code analysis and text summarization. To avoid cost on developer machines we use Ollama + a local LLM (Llama 3); in production we're planning AWS Bedrock or OpenAI.

Which design pattern should I apply to abstract the LLM provider? How do I tolerate, in the architecture, the structural differences between local token limits/response times and production?


Short answer: build a narrow port/adapter (Strategy) around your **capability**, not around the provider. A small interface plus real implementations selected by config; nothing more.

Your real danger here is over-abstraction: people try to turn this into a generic "AI framework" and burn months on it. All you need is one thin interface.

1. **Define the interface by capability, not by vendor.** A tiny contract like `complete(prompt, opts): Result` is enough. Ollama, Bedrock and OpenAI become three implementations of it; `config`/`env` picks which one runs. Don't leak provider-specific params into the core logic — keep the interface small and capability-based.
2. **Tolerate differences with explicit timeouts + a cheap fallback.** In production the provider slows down or hits a rate limit; put a clear timeout on every call and fall back to a cheap/local option when needed. This frees the architecture from being hostage to one provider's bad day.
3. **Budget tokens at the edge.** Each model has a different context window; truncate/summarize the input at the edge so it fits the model. Differences like "small window locally, big in prod" should be config, not an `if` in business logic.
4. **Put streaming behind the same interface.** If you need it, keep streaming support optional in the same contract; the caller shouldn't know the provider's stream API.
5. **Keep a golden-prompt test suite.** Run the same prompt set against each provider; automate the output-quality and regression comparison. That's what catches what broke when you swap providers.

**Bottom line:** I'd build one small interface, three real implementations and config-based selection — anything more is YAGNI. Local Ollama is for dev cost and fast iteration; treat the production differences (context window, latency, rate limits, cost) as **configuration**, not as branches in business logic. The narrower the abstraction stays, the sturdier it is.

## Related Reading

- [My AI-Assisted Engineering Workflow](https://sade.dev/en/journal/ai-assisted-engineering-workflow) — sade.dev
- [How do I position feature flags when moving from Gitflow to Trunk-Based?](https://www.muhammetsafak.com.tr/en/just-ask/gitflow-vs-trunk-based-development-and-feature-flags/) — Just Ask
- [When should I apply CQRS, and how do I sync the read and write models?](https://www.muhammetsafak.com.tr/en/just-ask/when-to-apply-cqrs-and-syncing-read-and-write-models/) — Just Ask
- [Should I choose NoSQL or PostgreSQL for flexible product attributes?](https://www.muhammetsafak.com.tr/en/just-ask/nosql-vs-relational-for-flexible-product-attributes/) — Just Ask