How do I set a clear rule for when to log at WARN versus ERROR?

Short answer: the level should be driven by “does a human need to act?”, not by “what the log line is about”. ERROR = a promise you broke; WARN = something unexpected that the system recovered from on its own.

1. Actionability is the axis. ERROR is when a request or job failed because of your fault and someone needs to look. WARN is something unexpected but tolerated: the retry succeeded, a fallback kicked in, a soft limit was approached. If your answer to “if this line fires at 3am, would I need to wake someone?” is yes, it’s ERROR.

2. Separate expected failures from bugs. A 404, a validation error, a user’s bad input — those aren’t your code breaking; they’re INFO or, at most, WARN. Rough rule: 4xx is the client’s problem, 5xx is yours. Reserve ERROR for when your own code failed to keep its promise.

3. Classify retries and fallbacks correctly. A single failed attempt that will be retried is WARN; a transient blip isn’t actionable. But once all retries are exhausted or the message lands in the dead-letter queue, it’s ERROR — because the work is permanently unfinished.

4. Alert on ERROR only, and by rate. Let your alerts trigger off ERROR (and fatal), and even then on a threshold/rate (“50 ERRORs in 5 minutes”), not on a single line. WARN feeds dashboards, not pagers.

5. Don’t log the same failure five times. In Go, if every layer both returns and logs the error, one failure becomes five lines. Log once at the boundary where the outcome is decided (the handler); lower layers should just wrap and return the error. In Laravel, do the same by using the report() level in the exception handler instead of sprinkling Log::error everywhere.

6. Separate ERROR from FATAL/critical too, and make every ERROR actionable. FATAL is when the process can’t continue (missing config, failed boot); ERROR is a request/job failure while the process stays up. And every ERROR line should carry enough context to assume someone will look at it in the middle of the night: request/trace id, the relevant identifiers, and the wrapped error chain. An ERROR nobody can act on isn’t really an ERROR.

A small example — let the outcome, not the call site, pick the level:

if err := publish(ctx, msg); err != nil {
    if attempt < maxRetries {
        log.Warn("publish failed, will retry", "attempt", attempt, "err", err)
        return retry
    }
    // retries exhausted: now it's genuinely actionable
    log.Error("publish failed permanently, sent to DLQ", "err", err)
}

Bottom line: personally I’d give the team a one-sentence rule: “If this log firing at 3am means someone must wake up, it’s ERROR; otherwise WARN.” On top of that I’d enforce logging a failure only once, at its boundary, and wire alerts to the ERROR rate. Those two changes dissolve most of the noise on their own; you clean up the rest over a few weeks by re-leveling the misclassified lines. The goal isn’t zero ERRORs — it’s that every ERROR actually means something.