Designing Async Processing with Kafka and Worker Queues

"Just put it on a queue" is common advice, but Kafka and a traditional job queue (BullMQ, SQS, and similar) aren't interchangeable — they're optimized for different shapes of problem.

Job queues: point-to-point work dispatch

A job queue is built around the idea of a unit of work being picked up and completed exactly once by one worker. It's a natural fit for "process this file," "send this email," "charge this card" — tasks with one clear consumer and a defined completion state. Most job queues also give you retries, delay/backoff, and priority out of the box, which is why they're the default choice for background job processing.

Kafka: an ordered, replayable log with multiple independent consumers

Kafka is a different primitive: an append-only log that multiple, independent consumer groups can read from at their own pace, each tracking its own position. That makes it the better fit when:

• More than one system cares about the same event. A job queue models one consumer taking one job; Kafka models many consumers independently reading the same stream.
• You need replay. Because Kafka retains messages for a configurable period (not just until they're acknowledged), a new consumer can be added later and read historical events, or an existing consumer can be rewound after a bug fix.
• Ordering within a partition matters. Kafka guarantees order within a partition (e.g. all events for a given entity ID), which a generic job queue typically doesn't guarantee.

A concrete way to decide

Ask: "If I add a second consumer of this event next year, do I want it to see everything from the beginning, or only new events going forward?" If the answer is "everything from the beginning," or "I don't know yet, but I might," that's a sign toward Kafka. If there's exactly one clear consumer and the work is transactional in nature (do this thing, once), a job queue is simpler and does the job well.

They compose well together

These aren't mutually exclusive within one system. A common pattern: Kafka carries domain events between services, and a job queue inside a single consumer service manages retries and concurrency for the actual work that event triggers. The event bus handles fan-out and replay; the job queue handles per-worker task management.

In practice

The Distributed SaaS Platform system uses Kafka for fan-out to independent downstream consumers (payments, CRM, analytics, notifications), while the Data & Integration Platform uses a job queue (BullMQ) for single-consumer, retry-heavy processing of inbound integration data — the same underlying question, answered differently based on the actual consumer shape.