Should I use chunkById instead of chunk when the same job also updates the rows it iterates?
Question
In Laravel I'm processing a large table with a batch job. I filter records with `where('status', 'pending')`, iterate them with `chunk()`, and on each row I update the `status` column from `pending` to `processed`. The problem: some rows never get processed and are skipped. I think that as I update rows they drop out of the filter, the OFFSET shifts, and the records beneath it get passed over. Should I use `chunkById()` instead of `chunk()`? Does that actually fix the skip problem?
Answer
Short answer: yes, use chunkById(). The skipping you see is a well-known trap of chunk()’s OFFSET-based pagination, and yours is the textbook case. There’s one condition: never update the cursor column (usually id).
- Why
chunk()skips.chunk()runs a separate query per page withLIMIT/OFFSET. Once you process the first 500 rows and setstatustoprocessed, those rows fall out of thewhere status = pendingfilter; on the second pageOFFSET 500is now applied to a shrunken result set, and a slice right in the middle is skipped entirely. - How
chunkById()fixes it. Instead of OFFSET it does keyset (seek) pagination:WHERE id > ? ORDER BY id LIMIT n. The next page always continues from the last processed id; since processed rows are already behind the cursor, there’s no “shift” to speak of.Order::where('status', 'pending') ->chunkById(500, function ($orders) { foreach ($orders as $order) { $order->update(['status' => 'processed']); } }); - Critical rule: don’t update the cursor column. Updating
statusis fine, because it’s not the cursor. But if you change the column you order by (id) inside the loop, the seek logic breaks and you’re back to inconsistency. - Use
lazyById()for memory. It uses the same seek logic but returns a singleLazyCollectioninstead of a closure; if you prefer writing a fluentforeach— and without inflating RAM on huge tables — it’s cleaner. - The fastest path often isn’t iteration at all. If the update is deterministic (no per-row computation), you don’t need to loop; a single set-based query beats all of them and is atomic:
UPDATE orders SET status = 'processed' WHERE status = 'pending'; - Watch locks and deadlocks. In large batch updates keep the chunk size reasonable (500–1000 is a good start) and, if needed, process each chunk in its own transaction; one giant transaction inflates both lock duration and deadlock risk.
Bottom line: personally, if there’s real per-row work I’d use chunkById() (or lazyById() for fluency) and never touch the cursor column. But if the work is just updating a column, I’d drop the iteration entirely and switch to a single set-based UPDATE — it’s faster and the skip risk is zero.
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.