# Should I put a Kafka/Redis buffer between Fluent Bit and OpenSearch when shipping logs?

> Turn on Fluent Bit's filesystem buffer first; move to Kafka once 40k/s is permanent, and keep a Redis list only for logs you can afford to lose.

- Asked: 2026-04-28
- Answered: 2026-05-02
- Asked by: Burak
- Tags: altyapi, observability, kafka
- Source: https://www.muhammetsafak.com.tr/en/just-ask/should-i-put-a-kafka-buffer-between-fluent-bit-and-opensearch/
- Language: en-US
- Author: Muhammet Şafak

---
**Question:** We aggregate logs from our Go and Laravel services into a central OpenSearch cluster. The flow: containers write to stdout, Fluent Bit on the hosts picks them up and ships straight to the OpenSearch API. But above ~40,000 log lines per second, OpenSearch indexing becomes the bottleneck; Fluent Bit can't keep up, backpressure builds, and the app containers' I/O blocks.

Should I put a Redis list or Kafka in between as a buffer? How do I avoid data loss (at-least-once) while keeping disk/memory from ballooning on the hosts?


Short answer: yes, a buffer is the right move — but the choice between a Redis list and Kafka comes down to the guarantee level you're willing to accept.

What you're hitting is classic **backpressure**: once OpenSearch's indexing rate drops below your production rate, the chain clogs backwards until the app container's stdout blocks. The fix is a durable layer that decouples the producer (your app) from the consumer (OpenSearch).

1. **Turn on Fluent Bit's own disk buffer first.** With `storage.type filesystem` and `storage.total_limit_size` on the output, Fluent Bit spools to disk instead of RAM when OpenSearch slows down, so no pressure reaches the app. Measure this before adding an external queue — it's often all you need.
2. **Redis list: cheap but risky.** In-memory; `RPUSH`/`LPOP` is simple, but with AOF off you lose data on a restart and a growing list eats Redis's RAM. Fine for logs you don't mind losing, not for financial/audit logs.
3. **Kafka: 40k/s is its home turf.** Disk-based, with retention, replayable, horizontal consumption via consumer groups. It gives you `at-least-once` naturally. The cost is operational — you have to stand up and feed a cluster. If the volume is permanent, it's the right tool.
4. **Make at-least-once idempotent at the consumer.** When writing to OpenSearch, give each document a deterministic `_id` (e.g. a hash of source+offset). A message delivered twice overwrites the same `_id` instead of creating a duplicate.

**Bottom line:** start with Fluent Bit filesystem buffering plus bulk tuning and ISM (rollover) on the OpenSearch side. If that's not enough, move to Kafka — keep Redis for logs you can afford to lose. And the golden rule: never let the app container block while logging. Logging should be fire-and-forget; don't carry critical data (payments, audit) over the log pipeline — run that on a separate, guaranteed path.

## Related Reading

- [Async Job Processing with Laravel Queue and Supervisor](/en/blog/async-job-processing-with-laravel-queue-and-supervisor/) — Blog
- [How do I prevent log storms and disk-full outages?](https://www.muhammetsafak.com.tr/en/just-ask/preventing-log-storms-and-disk-full-outages/) — Just Ask
- [How should I carry a correlation ID from the HTTP request all the way into queued jobs?](https://www.muhammetsafak.com.tr/en/just-ask/how-should-i-carry-a-correlation-id-from-the-http-request/) — Just Ask
- [How do I set a clear rule for when to log at WARN versus ERROR?](https://www.muhammetsafak.com.tr/en/just-ask/how-do-i-set-a-clear-rule-for-when-to-log/) — Just Ask