Same message, different outcome: determinism in event-driven architecture

This morning an invoice came back marked “could not be sent.” One line in the log: the invoice’s series prefix is not on the allowed list. A message of exactly the same shape had gone through without trouble a few hours earlier. The code hadn’t changed. The contents of the message hadn’t changed. The only thing that had changed was that the same list of prefixes lives in two separate services, and one of them had been updated. A value that was valid on one side was treated as a forged payload on the other.

Back in June, writing about the outbox pattern and idempotent consumption, I made a note: solving one pattern gives birth to the next. This post is that next one. The outbox guaranteed that the event would be delivered. It did not guarantee that a delivered event would produce the same outcome. The difference looks small; in practice they are two separate classes of problem.

What determinism was, and where we left it behind

Determinism is a boring phrase: same input, same output. And because it is boring, it is the assumption engineering papers over most often.

Yet most of what we build sits on top of it. Retry needs determinism — “try again” is only a solution if you know the second attempt will do the same thing as the first; otherwise it is a gamble. Idempotency needs determinism — saying that processing the same message twice is harmless assumes the second pass will reach the same decision as the first. Tests need determinism — I wrote about flaky tests separately on sade.dev. Post-incident analysis needs determinism: with no answer to “why did this outcome happen,” you cannot know whether the thing you fixed is actually fixed.

In the synchronous world we got this for free. A user pressed a button; the read, the decision and the write all lived inside the same transaction, the same process, the same instant. There was no chance for the world to change between reading the data and deciding on it, because there was only one instant.

Event-driven architecture breaks that singularity on purpose. Time enters between the moment an event is produced and the moment it is consumed. It might be milliseconds; it might be twenty minutes when the queue backs up; it might be three days when a failed message is replayed. And in that gap the world can change: a setting is updated, a company record is edited, the year rolls over, a service is redeployed.

Determinism stopped being free. It is now a cost you have to pay deliberately.

The message is not the whole input

Looking at a queue, you think: the input is the message. It isn’t.

Everything a consumer reads is input. Every input not written in the message is hidden; a hidden input is the place where you cannot explain the outcome afterwards.

So the real equation is:

outcome = f(message, hidden inputs)

Hidden inputs don’t stand out in a consumer’s code, because each of them looks innocent. A few of them, in the shape they actually took in our flow:

Hidden input	What it looks like	The drift it produces
Time	The consumer reading the current clock for the document date	A retry across the year boundary: the document number belongs to one year, the document date to another; the far side rejects it
Current state of the database	The company’s prefix was X when the document was created, Y when it was sent	The number is drawn from an entirely different series
Configuration	The same allow-list lives in two separate services	One side accepts, the other rejects
An external system’s answer	”This already exists” — but what exists is unspecified	Two opposite actions can be derived from the same exception
The runtime host	The same library; one framework swallows the warning, the other turns it into an exception	Same code, same input, different outcome
Which worker won	Parallel replicas, locks, a redelivered message	The outcome becomes dependent on processing order

I want to dwell on those last two rows, because they were the ones that unsettled me most when I noticed them.

The same integration library is used by both our old synchronous application and our new queue service. In one particular case the library raises a PHP warning. The old application swallowed the warning and carried on; the new service turned it into an exception and stopped the flow. Same library, same call, same input. What determined the outcome was who was hosting the code. No unit test catches a difference like that, because the difference isn’t inside the code under test.

The second was more insidious: a non-critical log record failing to write was aborting the entire send pipeline. So even when the document had been successfully delivered to the far side, not a single record was written on ours. A side effect’s failure was changing the primary outcome. Losing a log line is acceptable; losing the record of a submission is not — but you can only write it that way once you have noticed the two need to be handled separately.

The critical point here: none of these items is a bug. Each one is a reasonable decision taken on its own. I said the same thing writing about role boundaries — every step was individually defensible; what was wrong was the sum. Architecture works the same way: determinism isn’t broken by one bad decision, it’s broken by ten reasonable ones that don’t know about each other.

Branch proliferation: what a condition actually costs

We think we know what an if costs: a little readability, a little complexity. In an event-driven consumer that isn’t what you pay. What you pay is a new outcome that has to be proven.

And the cost isn’t linear. The classic answer would be “N conditions, 2ᴺ paths,” but that isn’t the real problem. The real problem is that the conditions are not independent: they share the same hidden inputs. Read the clock in one place and read it again in another, and two different conditions drift at once — and in a correlated way.

My clearest example: the integrator returns a single error — “this document already exists.”

That sentence, on its own, means nothing. It corresponds to two completely opposite worlds:

What tells them apart	What actually happened	The correct action
The document identity reported by the far side is the same as ours	The document went through earlier	Recover the state, do not resend
The identity is different	That number belongs to a different document; ours never arrived	Draw a new number, resend

The two branches call for opposite actions. And the information that separates them is not in the message — it is in a field inside the far side’s response. Which means the input to this branch is how talkative an external system we don’t control happens to be.

The cost of picking the wrong branch isn’t symmetric either. Confuse it one way and you issue a duplicate document; confuse it the other way and you flag a document that never left as “lost” and raise a false alarm. The first is a financial event, the second is noise. Same if, two different weights of wrong.

And there is a third state, the part of this that taught me the most: on the send call the far side says “this document already exists,” but when you query with that same identity it says “document not found.” The identity is reserved on their side but was never forwarded to the authority. The document has neither been sent nor not been sent.

This is an undecidable state. This is exactly where determinism ends. Putting an assumption here and proceeding automatically — “it probably went through” or “it probably didn’t” — does not produce determinism; it just buries the uncertainty inside the code and makes it invisible. For some branches the right answer is not an action: don’t decide, flag it, hand it to a human.

Two paths, one outcome — mandatory

Anywhere a job can be done by two separate pieces of code, determinism is no longer something you look for inside a single service. Determinism lives in the semantic equality of the two paths.

That was exactly our situation: the same document can be sent by both the old synchronous path and the new queue path. The old path took a hard error code from the far side, rolled back, and counted it as failed. Had the new path taken the same code and written its records, the document would have been marked “sent.” Same error, same document, two different realities. The user sees a green tick on screen and there is no document on the other end — and you find out months later during a reconciliation.

The same problem shows up in a shared counter. If two writers draw numbers from the same series, “the next number” can no longer be read from a single source. The counter turns into arithmetic that has to heal itself: the counter’s own value, the highest committed record, and the highest reserved number — continue from whichever of the three is largest. That’s ugly. But the ugliness belongs to having allowed two writers, not to the code.

The principle I took from this: behavioural parity matters more than shared code — but parity needs an owner. You can achieve parity without extracting a shared library; what you cannot achieve is keeping unowned parity intact over time. The definition I used writing about role boundaries applies here too: work that is in nobody’s job description is work nobody notices when it stops being done. Two paths behaving identically is exactly that kind of work.

Winning determinism back: four moves

Deleting conditions isn’t the answer; the business is conditions. What you can do is fix their answers in place.

1. Freeze the context at production time. The consumer should not resolve anything it needs for its decision. Let the event carry its own context: which prefix to use, which date applies, which rate, which exchange rate, which template. We did this with the prefix — it is pinned when the document is created and is not re-read from the company record at send time. Otherwise, if the company’s prefix is changed between creation and sending, the document gets numbered from a different series. There is a price: the message gets fatter and the schema has to be versioned. The gain is just as clear: the same message always produces the same outcome.

2. Turn time into an input. Reading the current clock inside a consumer is inserting an unnamed variable into the middle of the flow. Take the decision instant from the message; if you can’t, fix it on the first attempt and persist it. The critical sentence: a retry must not change time. If it does, the retry mechanism itself becomes a new branch — which is precisely what happens at the year boundary. A retry that preserves the document number but shifts the date splits the year across the two, and it gets rejected. The most expensive thing about a bug that surfaces one day a year is how long it takes to find.

3. Name the branch, record the reasoning. Don’t let the log say “failed.” Let it say which condition, with which input, sent it down which arm. The only way to verify afterwards whether a decision was deterministic is for the reasoning behind it to have been recorded. When we added the hint “are the two lists still identical?” to the prefix rejection error, diagnosing the same problem a second time took minutes. The next person reading it won’t be you; putting direction into an error message isn’t a luxury, it’s part of the specification.

4. Don’t resolve ambiguity silently. Closing an ambiguous external answer with an assumption turns a correctable error into an uncorrectable one. When an unrecognised prefix arrived we could have chosen to fall back to the company default — the flow would have continued, nobody would have noticed anything, and the document would have been issued from the wrong series. A noisy failure beats a silent wrong outcome. A failed submission can be retried; a document issued from the wrong series needs a correcting document.

And a fifth, often the best of all: remove the branch instead of solving it. Some conditions in a consumer aren’t conditions at all — they are two different event types. Split them into separate message types and the if disappears entirely — and a branch that doesn’t exist has no wrong arm.

Is this determinism, or an impression of it?

Let’s be honest: there is no full determinism in a distributed system. The network isn’t yours. Delivery order isn’t yours. How many times a message arrives isn’t yours. You cannot solve any of those by freezing things into a payload.

So the goal has to be set correctly. What we are after is not outcome determinism but decision determinism: always reaching the same decision given the same set of inputs. When the message arrives is not in our hands; what we do when it arrives is.

“Then let’s put everything in the payload” is a trap too, and it backfires quickly. The message swells, the schema turns brittle; worse, when old messages waiting in the queue are consumed by new code, an entirely different determinism problem is born. The thing you thought you had solved comes back as replay.

The boundary rule that works for me: whatever enters the decision gets frozen; whatever doesn’t gets carried by reference. The document’s prefix enters the decision — freeze it. The company’s logo doesn’t — a reference is enough. The distinction isn’t always clean, but at least it makes you ask the right question.

Closing

That morning’s invoice went through in the end with a one-line configuration fix. But that wasn’t the real fix. The real fix was admitting that we had allowed that condition to take its answer from outside the message in the first place.

Your code can be deterministic; your system won’t be. Determinism is not a property of code but a discipline of context: binding everything that feeds a decision to the moment the event was born, not to the moment the decision is made.

The one question to ask before writing a condition: is the thing that changes this condition’s answer inside the message, or in the current state of the world?