Expertise
Backend developer: the queue, the database and the server behind the API
Moving a synchronous flow onto RabbitMQ showed me that a user waiting on a spinner had been acting as an accidental queue for years; the real work began when that queue disappeared.
- Since
- 2010 Since
- years
- 16 years
- posts
- 26 posts
- projects
- 3 projects
Me in this ecosystem
Backend development, for me, is not a language but a boundary of responsibility: everything behind the screen the user sees. Contract, queue, database and server are four faces of the same work, and a decision about one binds the other three.
Observability, infrastructure managed as code and rollback strategy I write about as architectural subjects on sade.dev.
What I do on the backend
What they have in common is sitting behind the screen the user sees: the contract, the queue, the database and the server.
-
The API contract
The response shape, the error contract and the versioning decision are settled before code is written. Otherwise every client team writes its own assumption, and that assumption becomes your problem six months later.
-
The asynchronous delivery path
Queue, consumer and outbox relay; the structure that takes the synchronous wait out from in front of the user. The second half of the same discipline is idempotency: a request that can be repeated without side effects.
-
Database decisions
Index and plan behaviour is chosen by measurement, not intuition; intuition works particularly badly here. The measurement itself does not live on this page but in the research ledger, together with its method and environment.
-
Search and the server layer
Elasticsearch at the boundary where relational stops being enough; the environment with Docker, the deploy pipeline with GitHub Actions, nginx in front. The scope stays at application level.
A queue does not guarantee delivery
The mistake I see most often is assuming work becomes durable the moment it moves onto a queue. The chain is read in order; each step closes what the previous one left open.
- 01 Writing and publishing are two systems
- There is no atomicity between an INSERT into the database and a publish to the queue. If the write succeeds and the publish drops on a network hiccup, the row exists and the event never left.
- 02 The message is written in the same transaction
- The message is not published directly; it is written as a row into an outbox table inside the same transaction. The record and the message then share a single fate.
- 03 A relay publishes the rows
- A separate relay reads those rows and hands them to the queue. This is where the message stops getting lost — but not where the guarantee is complete.
- 04 An idempotent consumer completes it
- The outbox gives you at-least-once: the message is not lost, but it can repeat. Writing every processed event id into a processed_messages table inside the same transaction makes that repeat harmless.
My scope on data and the server
I do not make database decisions on intuition, and on the server side the scope stays at application level.
-
Index decisions made by measurement
Index choices, query plans and behaviour that shifts as a table grows are selected by measuring. A decision made without it can slow tomorrow the query it sped up today.
-
A full-text search layer
It takes over where the relational store stops being enough. Stemming, synonym expansion and relevance ranking exist there too; what brings in a separate layer is not their absence but how far they carry at scale and across languages.
-
The server at application level
Packaging the runtime environment, setting up the test and deploy pipeline, configuring the layer in front of the application.
The order that makes change safe
The hard part of back-end work is not writing a new feature but changing a running system without breaking it.
- 01 Plan the schema around the window where both versions are up
- Old and new code run side by side for a while; a schema change written without accounting for that window always breaks the running system first.
- 02 Split the data move into reversible steps
- A move that runs in one go has no way back. One split into steps can stop at the step that goes wrong.
- 03 Keep the test surface in three separate places
- Where the business rule is exercised, where the outside boundary is imitated, and where the flow runs end to end. The cost of a change has to stay smaller than the relief it brings.
I do not cover infrastructure as architecture here
Observability, IaC and rollback strategy do not stand on their own on this page; my scope here stays at application level. The place I write infrastructure as an architectural subject is sade.dev.
Recent writing in this area
Don't pick a partial index for speed
Four index strategies on a 10M-row Postgres queue table: the winner did not win for the reason I gave when I recommended it, and one setting turns the win into a 1,673x loss.
Same message, different outcome: determinism in event-driven architecture
Why does the same message produce different outcomes in an event-driven system? The hidden inputs that break determinism — and four moves to win it back.
From Dual-Write to Outbox: A Log of Idempotent Consumption and Field-Level Encryption
Lost events from dual-write pushed me to a transactional outbox, at-least-once delivery to idempotent consumption. Then I encrypted the GDPR fields at the row level with x-gdpr-sensitive. Notes from a real project.
Measurements in this area
Seven PHP frameworks under identical load: the gap narrows as soon as the request does real work
On the same hardware, the same PHP build and the same seven routes, how many requests a second do Laravel, Symfony, CodeIgniter, Yii2, Phalcon, Laminas and Slim serve, and at what latency?
Finding
On an empty route the fastest is 4.4× the slowest (Slim 25,975, Laravel 5,966 req/s). As soon as the request does real work the gap closes: 3.7× for a single row from the database, 3.5× for twenty rows. Phalcon is third on an empty route and fifth once a query is involved — being a C extension buys nothing while the process waits on MySQL. And the expensive decision is not the framework: Laravel's own default `web` middleware group takes the same response from 5,858 to 2,176 req/s, so one default costs more than most of the distance between the frameworks.
measured 17 days ago
The partial index grew three hundred and eighty times in fifteen minutes — and autovacuum never ran
Under sustained churn, does a partial index stay small on a queue table, and do the default autovacuum settings keep up with it?
Finding
With the live set holding steady at five thousand rows the partial index went from 0.1 MB to 38.2 MB — three hundred and eighty times. Its smallness comes from the live set, its bloat rate comes from throughput, and nothing connects the two. The composite index bloated less in proportion (42%) and more in absolute terms (+126 MB), and while bloating it stopped fitting in memory: its latency went from 0.52 ms to 61 seconds and its backlog climbed to 126,000. Fifteen minutes produced 1.75 million dead rows and autovacuum **did not run once** — the default threshold scales with the whole table (50 + 0.2 × 10 million ≈ 2 million) while the churn happens in a tiny subset.
measured 15 days ago
Postgres never turned the partial index into a generic plan: forty executions, forty custom plans
Does Postgres switch a partial-index query to a generic plan on its own inside a prepared statement — or is the 1,673-fold cliff something you have to opt into?
Finding
Postgres declines. On the partial index all forty executions used a custom plan — the counter reads 40/0. The reason it declines is the disaster itself: a generic plan cannot use the partial index, so its estimated cost comes out high and the planner does not choose it. The composite index switches at the sixth execution exactly as documented (5/35) and loses nothing by it. So the 1,673-fold cliff is real but fenced: reaching it takes writing `plan_cache_mode = force_generic_plan`.
measured 15 days ago
What I have built in this area
Personal and business finance on one data model
A finance core that keeps multi-account income and expense tracking in a single model; it became Parantaj, running on web, iOS and Android.
What it does today
One account/transaction model carries multi-account management, budgeting, goals and reporting on a single core; the web, iOS and Android clients are all live. Anyone who wants personal and business finances tracked in one place can use it.
Swipenor: if the client computes the score, what does the server trust?
A swipe-based true/false trivia game; the real subject is not the game but making a client-computed score verifiable through a signature.
What it does today
The swipe-based true/false quiz runs end to end in its core loop as a mobile app; the score is produced on the client but verified server-side with a server_nonce and an HMAC signature. swipenor.com is the app's landing page, not the game itself.
academia.sh: can theory and a real field problem live in the same lesson?
A free, publicly open, text-first learning platform that holds textbook theory and problems actually met in the industry inside a single lesson flow.
What it does today
Live as a free learning platform with no signup wall: the field–curriculum–course–unit–lesson hierarchy, full-text search, progress tracking and verifiable certificates all work. Anyone who wants to read the computer-science curriculum can use it today.
What shipped in this area
Parantaj
OngoingOwner
A personal and corporate financial management platform. It offers income-expense tracking, budget planning, detailed reporting, and multi-account management.
BabelQueue
OngoingFounder & Developer
BabelQueue is a language-agnostic message queue standard that lets services written in different languages share the same queue without getting trapped by serialization lock-in. Instead of language-specific formats like PHP's serialize(), it defines a strict JSON envelope — frozen at schema_version 1 — that every language can read natively. It runs on Redis and RabbitMQ with no sidecar or broker plugin, adding under 2% overhead.
Looplio
OngoingOwner
Looplio — a recurring automation platform (Web + iOS + Android) that turns repeating tasks, maintenance, and checks into reusable templates and automatically generated checklists.
Questions in this area
21 questions answered on this axis.
My non-root container can't write to the mounted storage directory—how do I fix the permissions?
Pin the user with a build-time `ARG UID`, use `COPY --chown`, and for named volumes chown only `storage` and `bootstrap/cache` in the entrypoint.
Should I author the OpenAPI spec first or generate it from my code?
Go contract-first with `openapi.yaml` as the single source of truth; what ends drift isn't the direction you pick but a green contract test running in CI.
Should I use a partial index on a queue table where I only ever scan 'pending' rows?
Build the partial index on `WHERE status='pending'`, include the `ORDER BY` column, pair it with `FOR UPDATE SKIP LOCKED`, and pass 'pending' literally.
Technologies I pair it with
-
RabbitMQ
Asynchronous delivery between services; the exit door of the outbox relay.
-
Redis
Cache, counters and short-lived locks.
-
PostgreSQL
The database I measured queue-table and index behaviour on.
-
MySQL
The relational store I have worked with longest.
-
Elasticsearch
Where relational stops being enough: full-text search and log analysis.
-
Docker
Development environment and deployment unit.
-
nginx
The layer in front of the application.
-
GitHub Actions
Test and deploy pipeline; the steps that SSH into the server live here.
Frequently asked
7 questions
-
How long have you worked on the backend?
Since 2010, so 16 years. Two years after I started with PHP the weight of the work moved from the contract to the queue and stayed there; that period is where the 26 posts on this site came from.
-
What do people usually come to you with on the backend?
Most often it is "we moved to a queue, but some jobs vanish." The answer is nearly always in the same place: writing to the database and publishing to the queue are two separate systems with no atomicity between them.
-
Where do you start when designing an API?
With the contract. If the response shape, the error contract and the versioning decision are not settled before code is written, every client team writes its own assumption instead. A good share of the 26 posts on this site circle those three headings.
-
Does using a queue guarantee delivery?
No. A transactional outbox stops messages from being lost, but what it gives you is at-least-once — the message can repeat. What completes the guarantee is not the queue; it is an idempotent consumer.
-
How do you make index decisions?
By measuring. Choosing a partial index purely because it is "small and fast" is a trap: the moment a generic plan takes over, the partial index becomes unusable. The good news is that the cliff is fenced — Postgres does not make that switch on its own; you have to force the setting by hand. The full measurement sits in the research ledger.
-
Do you set up the server side too?
Yes, at application level. I set up the environment with Docker, the deploy pipeline with GitHub Actions and nginx in front. Infrastructure as an architectural subject is written on sade.dev.
-
How many projects run on this stack?
3 of the projects recorded on this site sit directly on this queue-and-database stack.
Let us work together
If you have work in this ecosystem, tell me what you are building and we will talk about how to build it.