Chunked Batch Message Processing

ProcessMessageBatch used to process an entire alert's recipient list — 5,000 to 7,000 users for a large alert — inside one queue job. That job routinely ran long enough to exceed the queue worker's 90-second retry_after window, and on two real alerts only 17-25% of assigned recipients actually received a message before the job was considered dead and the queue moved on [@design-doc]. The fix splits one alert dispatch into many small ProcessMessageBatch jobs, each handling a fixed-size chunk of recipients, while keeping exactly one shared MessageBatch row as the source of truth for delivery counts.

Context

The failure mode was straightforward: a single long-running job silently produces partial delivery, and nothing about the system made that obvious until someone counted actual sends against an alert's recipient list. The queue's retry_after is a hard ceiling that a monolithic per-alert job could blow past well before finishing its recipient loop. Splitting recipients into chunks reduces each job's own runtime, but it also introduces a new problem the original single-job design never had to solve: multiple ProcessMessageBatch jobs now run concurrently against the same MessageBatch row, and any code that reads-then-overwrites that row's counters is a race condition waiting to clobber another chunk's progress.

Decision

BatchMessagingService::createBatch() creates one MessageBatch record, marks it Processing once up front, then dispatches recipients in fixed-size groups using collect($recipients)->chunk($chunkSize)->each(...), where $chunkSize comes from config('messaging.batch_chunk_size', 500) [@batch-messaging-service]. The config file itself ships with MESSAGING_BATCH_CHUNK_SIZE defaulting to 200, not the 500 used as a defensive fallback in the service call [@messaging-config] — so in practice a 7,000-recipient alert becomes 35 jobs of 200 recipients each rather than one job of 7,000, and the 500 in createBatch() only matters if the config key itself were ever missing.

Because multiple chunk jobs for the same MessageBatch now run concurrently, three things had to change inside ProcessMessageBatch:

• The completion guard no longer treats Dispatched as terminal. It only skips execution when the batch is already Completed or Failed [@process-message-batch] — otherwise the first chunk to finish would mark the shared batch Dispatched and every later chunk would see that status and skip its own recipients entirely.
• Counters are updated atomically, not overwritten. processed_count and failed_count are incremented with DB::raw('processed_count + '.$chunkProcessedCount) rather than recomputed and rewritten from scratch, and the batch's metadata is re-fetched immediately before merging in this chunk's channel_distribution so a concurrent chunk's own metadata writes (like chunk_errors) aren't lost to a stale read [@process-message-batch].
• A single chunk failing does not fail the whole batch. ProcessMessageBatch::failed() calls MessageBatch::recordChunkError() instead of marking the batch Failed outright [@process-message-batch] [@message-batch-model] — because marking the shared batch Failed would freeze incrementCounters() for every other chunk's still-in-flight SendChannelMessage jobs, silently discarding otherwise-successful sends from unrelated chunks. The batch is only marked failed by handle() itself, and only in the narrow case where a chunk created zero new messages, the batch's processed_count already reached total_count, and no Message rows exist for the batch at all [@process-message-batch] — a signal that the whole dispatch produced nothing, not just that one chunk had a transient problem.

The job also declares an explicit public int $timeout = 60;, with a comment noting it must stay shorter than the queue connection's 90-second retry_after so the queue never re-dispatches a chunk that is still legitimately running [@process-message-batch] — this is the concrete mechanism that fixes the original bug: a 200-recipient chunk finishes well inside 60 seconds, where a 7,000-recipient monolithic job could not.

Making concurrent chunk retries safe against duplicate sends required a database-level guarantee, not just application logic: a migration adds a unique composite index on messages(message_batch_id, user_id, conversation_id, direction) [@unique-index-migration], which is what makes the existing Message::firstOrCreate(...) pattern in ProcessMessageBatch safe under concurrency — without it, a retried chunk could race a SELECT-then-INSERT and create a duplicate outbound message to the same user.

See the messaging pipeline for how MessageBatch and ProcessMessageBatch fit into alert dispatch overall, and the debug message delivery guide for diagnosing a batch that still shows partial delivery.

Status

Approved and implemented.

Consequences

A large alert dispatch is now dozens of short-lived jobs instead of one long-lived job, so a single chunk running slow or crashing affects only its own 200 recipients rather than the entire alert. Delivery counters (processed_count, success_count, failed_count) are reliable under concurrency because they are atomic increments rather than read-modify-write cycles; channel_distribution inside metadata is documented as approximate under high concurrency and is explicitly not used for completion logic, since the real source of truth for "is this batch done" is the atomic counters, not the per-chunk distribution snapshot [@design-doc]. The chunk size is an operational knob (MESSAGING_BATCH_CHUNK_SIZE) rather than a constant, so it can be tuned per environment without a code change if send latency per recipient changes materially.