Skip to content
Muhammet Şafak
tr

Postgres never turned the partial index into a generic plan: forty executions, forty custom plans

Does Postgres switch a partial-index query to a generic plan on its own inside a prepared statement — or is the 1,673-fold cliff something you have to opt into?

Finding

Postgres declines. On the partial index all forty executions used a custom plan — the counter reads 40/0. The reason it declines is the disaster itself: a generic plan cannot use the partial index, so its estimated cost comes out high and the planner does not choose it. The composite index switches at the sixth execution exactly as documented (5/35) and loses nothing by it. So the 1,673-fold cliff is real but fenced: reaching it takes writing `plan_cache_mode = force_generic_plan`.

Partial index, generic plans
0 of 40 executions
Composite index, switch
execution 6
Partial, first 5 → last 5
0.36 → 0.20 ms
What reaching the cliff takes
force_generic_plan

Method

The same harness and the same 10-million-dead-plus-5,000-live table, across three index strategies. For each, one prepared statement was executed forty times in a single psql session and `pg_prepared_statements` was queried after every execution — the counters are session-local, which is why this measurement cannot be made from pgbench. The decision is not inferred from timings but read from the counters: `custom_plans` and `generic_plans` say directly what the planner chose on that execution. Timings arrive in pairs (the statement, then the counter query) and the parser keeps only the first. psql's aligned output pads the counter value with spaces, so output was switched to unaligned mode — the first run produced an empty series because of exactly that.

High confidence Repeated runs, controlled environment, raw data published.
Measured on

measured yesterday

Published

Environment

Postgres
17-alpine · plan_cache_mode at its default (auto)
Table
10M dead + 5,000 live rows
Measurement
one session · one prepared statement · 40 executions
Source of the decision
pg_prepared_statements.custom_plans / generic_plans
Hardware
Apple M4 Pro · 12 cores · 24 GB · Docker Desktop 29.7.2

Technologies

PostgreSQL SQL Docker

To reproduce

EXECUTIONS=40 ./bench/plan-switch.sh

The partial index measurement pinned plan_cache_mode to each extreme and found a factor of 1,673 between them: 11,752 tps on a custom plan, 7 on a generic one. What it did not measure is the setting everyone actually runs — auto.

That record wrote a sentence there: “the profile degrades as it warms up… it does not show up in staging.” It was a reasonable inference and it turned out to be wrong.

The decision was read, not guessed

Postgres runs a prepared statement on a custom plan for its first five executions, then compares the generic plan’s estimated cost against the average of the custom ones, and switches only if the generic looks cheaper. pg_prepared_statements keeps that decision as a counter, so nothing has to be inferred from timings.

Strategy First generic plan Final counter (custom/generic) First 5 (ms) Last 5 (ms)
partial the switch was declined never 40 / 0 0.36 0.20
composite execution 6 5 / 35 0.40 0.21
(status) never 40 / 0 0.98 0.78
One session, one prepared statement, forty executions. The counters were read after every one.

The composite index switches where the documentation says it will, right after the fifth:

#5  0.266 ms   custom=5  generic=0
#6  0.275 ms   custom=5  generic=1   ← switched
#7  0.185 ms   custom=5  generic=2

On the partial index it never happens. Forty executions, forty custom plans.

It declines because of the disaster itself

The planner does not choose the generic plan because it estimates its cost correctly. A generic plan does not know what $1 is; to use the partial index it would have to prove $1 = 'pending', it cannot, so that plan falls back to a sequential scan and its estimate goes through the roof. The average custom plan sits far below it, and the comparison comes out in favour of custom every time.

The real trade is smaller, and elsewhere

The protection has a price: a query on the partial index is re-planned on every execution. Forty executions, forty plans. The composite one reuses its plan from the sixth onwards.

At this scale the difference is unmeasurable — 0.20 ms against 0.21 ms. But planning is cheap on a cheap query, not on a many-table join or a long IN list. The partial index’s hidden costs are bloat (see the endurance measurement) and constant re-planning; both are small in this workload, and neither is guaranteed to stay small in another.

It corrects the earlier record

The “degrades as it warms up” paragraph in the partial index record went too far, and a note pointing at this finding has been added there. A measurement correcting its own publication is what this section’s contract asks for: if a number is published, so is the number that refutes it.

Share:

Other records

All records

The partial index grew three hundred and eighty times in fifteen minutes — and autovacuum never ran

Under sustained churn, does a partial index stay small on a queue table, and do the default autovacuum settings keep up with it?

Finding

With the live set holding steady at five thousand rows the partial index went from 0.1 MB to 38.2 MB — three hundred and eighty times. Its smallness comes from the live set, its bloat rate comes from throughput, and nothing connects the two. The composite index bloated less in proportion (42%) and more in absolute terms (+126 MB), and while bloating it stopped fitting in memory: its latency went from 0.52 ms to 61 seconds and its backlog climbed to 126,000. Fifteen minutes produced 1.75 million dead rows and autovacuum **did not run once** — the default threshold scales with the whole table (50 + 0.2 × 10 million ≈ 2 million) while the churn happens in a tiny subset.

measured yesterday

High confidence

A partial index makes a queue table forty-one times smaller — for as long as the planner picks it

On a Postgres queue table with millions of dead rows, what does a partial index buy, and when does the planner refuse to use it?

Finding

At 10 million dead rows a partial index sustains 11,537 claims per second where the same table without one manages 7. Against a composite index the throughput difference is small (6.9%) but the size difference is not: 7.6 MB against 310.4 MB, and the partial one does not grow with the table because it indexes only the 5,000 live rows. None of that is the real finding. The moment the planner switches a prepared statement to a generic plan the partial index stops being used at all — 11,752 tps becomes 7, and 0.68 ms becomes 1.1 seconds. A factor of 1,673. The composite index is untouched under the same conditions.

measured yesterday

High confidence

Laravel's preload curve: 123 files buy eight times what the last 1,912 do

How far can a curated preload take Laravel, and what does each slice cost in start-up time?

Finding

The curve is not proportional to volume. The first 1,592 files — Laravel's own framework — buy 30 ms and add 1.2 seconds to start-up. The next 1,094 Symfony files buy 9.5 ms for free. The **123 files** after that (psr, carbon) buy 15.7 ms, more than the 1,094 before them. And the last 1,912 buy 1.8 ms while adding another 1.2 seconds. So the blanket preload the earlier record measured as a ceiling is the worst point on the curve that is not the origin: stopping at 2,809 files gives 12.77 ms for 1,514 ms of start-up, while 4,721 files ask 2,691 ms to reach 10.96 ms.

measured yesterday

High confidence

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind