Two languages sharing one queue without the serialization lock-in
A frozen JSON envelope in place of language-specific serialization; it became BabelQueue, a polyglot queue standard.
What it does today
The frozen JSON envelope has four languages — PHP, Python, Go, Node.js — reading the same bytes, with no sidecar and no broker plugin. Teams running a polyglot queue can pick up the spec and the SDKs today.
- Started
- March 2026 — June 2026
- Left Labs
Form
Technologies
This one became a product:
BabelQueuePHP’s serialize() and Python’s pickle do the same job; neither can read the
bytes the other produces. The problem is not the format itself but the fact
that the format belongs to a language — and a queue is the first place where
two different languages stand at either end.
Bytes locked inside a language
The insidious part of a polyglot architecture: one service writes to the queue in its own language’s format, a consumer in another language cannot open those bytes. Language-specific type information embedded in the message locks the job inside the language that produced it.
What was tried was a single envelope every language can parse with its standard
library, without putting a sidecar or a broker plugin in between. The envelope
carries the job identity, a trace_id for distributed tracing, the payload,
metadata and the attempt count.
A frozen envelope
A strict JSON envelope — frozen at schema_version 1 — plus a reference
implementation running on Redis and RabbitMQ. The measurement target was to
keep the overhead under 2%; the contract was written around one principle: one
language produces, another consumes, the same bytes.
Separating identity from the class name
Two decisions shaped this more than the envelope’s shape did.
Job identity is a stable URN-based identifier, not a class name. The
easiest way to say what a queued job is: write its class name, and in a
single-language system that works. Across two languages it does not:
App\Jobs\SendInvoice is not a class on the Python side, it is a meaningless
string. The identity had to be independent of the namespace that produced it.
The schema was frozen at schema_version 1. An extensible contract would
have been more flexible; adding fields would have stayed open. The cost would
have been the loss of a guarantee: that a producer written today and a consumer
added tomorrow share the same wire. The frozen schema gives that guarantee and
gives up room to grow in exchange — the right trade for a standard, the wrong
one for an application.
Together they make adoption incremental: existing non-BabelQueue jobs keep running untouched, and the migration happens queue by queue.
One adapter per language instead of one generic SDK
The third decision was about distribution. Since the envelope is a specification, one reference library and “the rest is up to you” would have been enough; anyone reading it would port it to their own language.
That road would have left the contract on paper. A queue standard gets adopted not by being understood but by plugging into the job infrastructure that is already there — and that infrastructure belongs to a framework, not a language. So each language ships with its own SDK and with the framework adapters that language actually uses. In the Labs stage the scope was four languages: Laravel and Symfony in PHP, Celery and Django in Python, Redis/RabbitMQ transports in Go, BullMQ and NestJS in Node.js.
The cost is the maintenance surface: one SDK, one release line and one package registry per language. In exchange, migrating means plugging in an adapter rather than rewriting the queue code.
The road to the specification
The envelope grew into a specification and a set of multi-language SDKs: four languages became six — Spring Boot in Java, MassTransit in .NET — and distribution moved to each ecosystem’s own channel: Packagist, PyPI, pkg.go.dev, npm, Maven Central, NuGet. BabelQueue is now a standard with its own domain. Details in the portfolio.
The research behind this work
All researchA partial index makes a queue table forty-one times smaller — for as long as the planner picks it
On a Postgres queue table with millions of dead rows, what does a partial index buy, and when does the planner refuse to use it?
Finding
At 10 million dead rows a partial index sustains 11,537 claims per second where the same table without one manages 7. Against a composite index the throughput difference is small (6.9%) but the size difference is not: 7.6 MB against 310.4 MB, and the partial one does not grow with the table because it indexes only the 5,000 live rows. None of that is the real finding. The moment the planner switches a prepared statement to a generic plan the partial index stops being used at all — 11,752 tps becomes 7, and 0.68 ms becomes 1.1 seconds. A factor of 1,673. The composite index is untouched under the same conditions.
measured 16 days ago
The partial index grew three hundred and eighty times in fifteen minutes — and autovacuum never ran
Under sustained churn, does a partial index stay small on a queue table, and do the default autovacuum settings keep up with it?
Finding
With the live set holding steady at five thousand rows the partial index went from 0.1 MB to 38.2 MB — three hundred and eighty times. Its smallness comes from the live set, its bloat rate comes from throughput, and nothing connects the two. The composite index bloated less in proportion (42%) and more in absolute terms (+126 MB), and while bloating it stopped fitting in memory: its latency went from 0.52 ms to 61 seconds and its backlog climbed to 126,000. Fifteen minutes produced 1.75 million dead rows and autovacuum **did not run once** — the default threshold scales with the whole table (50 + 0.2 × 10 million ≈ 2 million) while the churn happens in a tiny subset.
measured 16 days ago
Postgres never turned the partial index into a generic plan: forty executions, forty custom plans
Does Postgres switch a partial-index query to a generic plan on its own inside a prepared statement — or is the 1,673-fold cliff something you have to opt into?
Finding
Postgres declines. On the partial index all forty executions used a custom plan — the counter reads 40/0. The reason it declines is the disaster itself: a generic plan cannot use the partial index, so its estimated cost comes out high and the planner does not choose it. The composite index switches at the sixth execution exactly as documented (5/35) and loses nothing by it. So the 1,673-fold cliff is real but fenced: reaching it takes writing `plan_cache_mode = force_generic_plan`.
measured 16 days ago
