How do I prevent log storms and disk-full outages?

Short answer: there are two separate failures here — no throttling in the app, no limits/rotation on the host. Fixing one layer isn’t enough; you have to close both.

The real problem is this: a single broken dependency produced 10,000 lines per second, the disk filled, and once the OS couldn’t breathe the machine went down. So what killed you wasn’t the error, it was the unbounded logging of that error.

1. Throttle/sample repetitive logs in the app. Don’t write the same error 10,000 times; write a single counted line like “Connection refused (×10,000 in 60s).” Dedupe logs by error signature; if the same signature recurs, don’t emit a line — increment the count.
2. Back off retries so the error rate itself drops. Don’t retry instantly every time the connection fails; apply exponential backoff + jitter. This both reduces hammering the target and naturally cuts the log production rate.
3. Put a hard limit on the host with logrotate. Configure logrotate with a size and file-count cap (e.g. size 100M, rotate 5). Rotate the log file once it passes a size, delete the old ones; give total log space a ceiling.
4. Don’t write logs to the root/OS disk. Give /var/log its own volume. That way even if the log disk fills, the OS keeps running on the root disk and the machine doesn’t go down. This one change alone would have prevented your outage.
5. Run Filebeat with a bounded local spool and set early alerts. Cap Filebeat’s disk spool; and alert well before disk usage reaches 100% (at 70-80%). Catch the crisis while it’s still climbing, not at 100%.

Bottom line: I’d set up app-side throttling + a dedicated log volume + rotation together. Most importantly, a mindset shift: treat unbounded logging not as a feature but as a bug. I cover how to handle incidents like this with a disciplined post-mortem on sade.dev.