Integration / BullMQ Interview questions

BullMQ is a Node.js and TypeScript library for building background job queues on top of Redis. Instead of running slow or resource-heavy work inside a web request, you push a job onto a queue and a separate worker process picks it up later.

It grew out of the older Bull library and rewrote the internals in TypeScript, splitting responsibilities into dedicated classes: Queue for adding jobs, Worker for processing them, QueueEvents for cross-process notifications, and FlowProducer for chaining dependent jobs.

Typical use cases include sending emails, resizing images, generating reports, calling third-party APIs, and processing webhooks - anything that shouldn't block the main request-response cycle.

• Delayed and repeatable (scheduled) jobs
• Automatic retries with backoff
• Job priorities and rate limiting
• Parent-child job flows

BullMQ deliberately separates the job lifecycle into distinct classes rather than bundling everything into one object, which makes it easy to run producers and consumers on different machines.

Because these are separate classes, a producer service can only import Queue, while a dedicated worker fleet only needs Worker, keeping deployments lean.

A Queue instance is the entry point for adding work in BullMQ. It doesn't process jobs itself - it only writes job data into Redis and exposes management methods like pausing, draining, or cleaning old jobs.

import { Queue } from 'bullmq';

const connection = { host: 'localhost', port: 6379 };
const emailQueue = new Queue('emails', { connection });

await emailQueue.add('welcome-email', { userId: 42 });

Each queue is identified by name, and any number of Worker instances - even on different servers - can attach to that same queue name to consume jobs from it.

Because a Queue only talks to Redis, adding a job is fast and does not wait for the job to actually be processed.

A Worker connects to a named queue and continuously pulls jobs from it, running your processor function against each one.

import { Worker } from 'bullmq';

const worker = new Worker('emails', async job => {
  await sendEmail(job.data);
}, { connection: { host: 'localhost', port: 6379 }, concurrency: 5 });

The processor is an async function: returning a value marks the job completed and stores that value as the result, while throwing an error marks it failed (and triggers a retry if attempts remain).

The concurrency option controls how many jobs a single Worker instance handles in parallel - useful when jobs spend most of their time waiting on I/O rather than the CPU.

A Job is the object representing one unit of work sitting in a queue. It carries a name, a payload (data), configuration options, and metadata such as its ID, timestamps, and current state.

const job = await queue.add('resize-image', { url: 'https://...' }, { attempts: 3 });
console.log(job.id, job.data);

Inside a processor, the same job object is passed as the first argument, so you can read job.data, call job.updateProgress(), or inspect job.attemptsMade to see how many times it has already run.

Whatever value the processor returns is saved on the job as returnvalue, which other parts of your system (or a parent job in a flow) can later read.

QueueEvents is a dedicated class that subscribes to Redis pub/sub channels to receive job lifecycle notifications - completed, failed, progress, stalled, and more - regardless of which process or server actually ran the job.

import { QueueEvents } from 'bullmq';

const queueEvents = new QueueEvents('emails', { connection });
queueEvents.on('completed', ({ jobId, returnvalue }) => {
  console.log(`Job ${jobId} finished with`, returnvalue);
});

This differs from listening on a Worker instance directly: worker-level events only fire inside the same process that ran the job, while QueueEvents works across your whole distributed system - which is exactly what a monitoring dashboard or notification service needs.

Redis is the sole persistence and coordination layer for BullMQ - there's no separate database. Every job, its state, and its metadata live inside Redis data structures.

• Lists back the waiting and active states
• Sorted sets back delayed and prioritized jobs, ordered by timestamp or priority
• Hashes store each job's data and options

All state transitions - moving a job from waiting to active, for example - are implemented as Lua scripts executed atomically inside Redis. This matters because multiple workers can be polling the same queue simultaneously, and Redis's single-threaded script execution guarantees that two workers never grab the same job.

BullMQ connects through ioredis, and any Redis-protocol-compatible server can be used as the backing store.

A job in BullMQ moves through a well-defined set of states as it travels from creation to completion.

• waiting - ready to be picked up by a worker
• delayed - scheduled to become waiting at a future time
• active - currently being processed by a worker
• completed - the processor resolved successfully
• failed - the processor threw and no attempts remain
• waiting-children - a parent job in a flow, paused until its children finish
• paused - the whole queue is paused, so jobs sit here instead of waiting

You can query how many jobs sit in each state with queue.getJobCounts(), which is useful for building dashboards or health checks without inspecting individual jobs.

Jobs are added through the add() method on a Queue instance, which takes a job name, a data payload, and an optional options object.

await emailQueue.add(
  'welcome-email',
  { userId: 42, template: 'welcome' },
  { attempts: 3, delay: 2000, priority: 5 }
);

The name lets one queue handle several kinds of work - a worker can branch on job.name to run different logic for 'welcome-email' versus 'password-reset', for example.

Common options include delay (milliseconds before the job becomes eligible), attempts (retry budget), priority, and jobId for deduplication-style control. For adding many jobs efficiently at once, addBulk() avoids a round trip per job.

Instantiate a Worker with the queue name and an async processor function; BullMQ starts pulling jobs immediately once the connection is ready.

import { Worker } from 'bullmq';

const worker = new Worker('emails', async job => {
  if (job.name === 'welcome-email') {
    await sendWelcome(job.data);
  }
  return { sentAt: Date.now() };
}, { connection: { host: 'localhost', port: 6379 } });

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed:`, err.message);
});

The processor receives the job and, in newer versions, a cancellation token as a second argument, useful for extending locks on long-running work. Resolving the promise completes the job; rejecting or throwing fails it. Always attach a failed listener - unhandled failures are easy to miss otherwise.

Repeatable jobs are jobs that get re-created automatically on a schedule, instead of you calling add() yourself every time. In current BullMQ, they are managed by a Job Scheduler through upsertJobScheduler().

await myQueue.upsertJobScheduler(
  'daily-report',
  { pattern: '0 3 * * *' },   // cron: every day at 3am
  { name: 'generate-report', data: { type: 'daily' } }
);

A scheduler acts as a factory: it always keeps exactly one job in the delayed state representing the next run, and produces a new one once that job starts processing. You can also schedule with a simple every interval instead of a cron pattern.

The older repeat option passed directly to queue.add() still exists for backward compatibility but has been deprecated since v5.16 in favor of upsertJobScheduler(), which avoids accidentally creating duplicate schedulers.

A delayed job is added immediately but shouldn't be picked up until some time in the future. You set this with the delay option, specified in milliseconds.

await queue.add(
  'send-reminder',
  { userId: 7 },
  { delay: 60_000 } // becomes eligible to run in 1 minute
);

Internally the job sits in a Redis sorted set scored by the timestamp at which it should fire. A background process in BullMQ (the delay processor) moves jobs from delayed to waiting once their timestamp has passed, at which point any available worker can pick them up.

This is different from a repeatable Job Scheduler: a delayed job fires once, while a Job Scheduler keeps producing new delayed jobs on a recurring basis.

Priority lets you influence processing order instead of relying purely on FIFO. It's set with the priority option, ranging from 1 to 2,097,152.

await queue.add('urgent-task', data, { priority: 1 });  // processed first
await queue.add('normal-task', data, { priority: 10 }); // processed later

Counterintuitively for many developers, a lower number means higher priority - similar to how Unix process priorities work. Jobs added without any priority value are treated as the highest priority of all and are processed before any job that has an explicit priority number.

Prioritized jobs use a different underlying data structure than plain FIFO jobs, so adding one is a slightly more expensive operation - fine for occasional urgent work, but not something to apply to every single job by default.

FlowProducer adds a parent job together with one or more child jobs - potentially across different queues - in a single atomic call, so you never end up with a parent whose children failed to get created.

import { FlowProducer } from 'bullmq';

const flow = new FlowProducer({ connection });

await flow.add({
  name: 'process-order',
  queueName: 'orders',
  data: { orderId: 1 },
  children: [
    { name: 'charge-card', data: { orderId: 1 }, queueName: 'payments' },
    { name: 'reserve-stock', data: { orderId: 1 }, queueName: 'inventory' },
  ],
});

The parent job automatically enters the waiting-children state and won't run until all of its children complete, at which point it can read each child's returnvalue. Children can themselves have their own children, forming an arbitrarily deep tree.

BullMQ offers several ways to get rid of jobs, depending on whether you want it automatic or manual.

• removeOnComplete / removeOnFail options on add() - auto-remove after finishing, either as a boolean or a count/age limit
• job.remove() - manually delete one specific job
• queue.clean(grace, limit, status) - bulk-remove jobs older than a grace period in a given status
• queue.drain() - removes all waiting and delayed jobs, leaving active ones untouched
• queue.obliterate() - completely wipes the queue and all its jobs, including active ones (a destructive, rarely-used operation)

In production, relying on removeOnComplete/removeOnFail with a sensible retention count is the most common approach, since it keeps Redis memory usage bounded without needing a separate cleanup job.

Every job gets a unique id, either auto-generated by BullMQ (an incrementing counter by default) or explicitly supplied via the jobId option when adding a job.

await queue.add('sync-user', { userId: 7 }, { jobId: `sync-user-7` });

The ID is the key used to look the job up in Redis (queue.getJob(id)), to build progress or status URLs in a UI, and to prevent an exact duplicate: attempting to add a job with an ID that already exists in the queue's un-completed set will not add a second copy.

This last property makes explicit jobId values a simple way to achieve idempotent scheduling for cases like "only one pending sync job per user at a time" without extra bookkeeping.

There are two levels at which you can react to a job completing, and picking the right one matters for correctness.

// 1. Locally, on the Worker instance that ran the job
worker.on('completed', job => {
  console.log(`${job.id} done in this process`);
});

// 2. Globally, via QueueEvents - works even from a different process
const queueEvents = new QueueEvents('emails', { connection });
queueEvents.on('completed', ({ jobId, returnvalue }) => {
  console.log(`${jobId} done anywhere`, returnvalue);
});

The Worker's completed event only fires in the process that actually ran the job, so it's convenient for simple local logging. If you need to react from an API server, dashboard, or any process that isn't running the Worker, QueueEvents is the correct tool since it listens via Redis pub/sub instead of in-memory callbacks.

data is the payload you attach to a job when you add it - any JSON-serializable object describing what the processor needs to do the work.

await queue.add('resize-image', {
  fileUrl: 'https://cdn.example.com/photo.jpg',
  targetWidth: 800,
});

Because it has to be serialized into Redis, data should stay small and JSON-friendly - pass identifiers and let the processor fetch large payloads (like a file or full database row) itself, rather than embedding the whole file content in the job.

Job data is immutable once created via normal means; if you need to update it while a job is running, use job.updateData() rather than mutating job.data directly in memory, since the in-memory change alone won't be persisted back to Redis.

Bull was the original library; BullMQ is its ground-up TypeScript rewrite and the actively maintained successor, aimed at fixing architectural limits Bull had accumulated over time.

Migrating typically means separating your single Bull queue object into a Queue for producing and a Worker for consuming, and moving queue.process(fn) logic into new Worker(name, fn, opts).

In BullMQ v1, a separate QueueScheduler instance was required alongside every Queue and Worker - its job was to move delayed jobs into waiting once they were due, and to detect stalled jobs. Forgetting to run one was a very common source of "my jobs never start" bug reports.

Starting with BullMQ v2, that responsibility was folded directly into the Worker class itself. A Worker now checks for stalled jobs and promotes due delayed jobs as part of its own internal loop, so there's no extra component to remember to instantiate.

This simplified the mental model considerably: as long as at least one Worker is running against a queue, delayed and stalled-job handling happens automatically, removing an entire category of setup mistakes that plagued early BullMQ and Bull deployments.

BullMQ follows an "at-least-once" delivery model rather than "exactly-once" - a job is guaranteed to be processed, but under failure it might occasionally run more than once, so processors should be written to tolerate that.

The mechanism behind this is locking: when a worker picks up a job, it acquires a lock in Redis for lockDuration (30 seconds by default) and renews that lock roughly every half of that duration while processing continues. If the worker dies, blocks the event loop, or loses connectivity, it stops renewing the lock.

Once the lock expires without renewal, BullMQ's stall-detection logic (running inside another Worker's loop) notices the job is stalled and moves it back to waiting so a different worker can pick it up - up to maxStalledCount times before it's finally marked failed.

Because a job could be re-delivered after a worker crashes partway through, side effects inside your processor (charging a card, sending an email) should ideally be idempotent, for example by checking an external record before acting.

An in-memory array-based queue disappears the moment the Node.js process restarts or crashes, and it can't be shared between multiple processes or machines - two things any real production job system needs.

Redis solves both problems: it's a separate, persistent process, so job state survives an application redeploy, and any number of producer or worker processes can connect to the same Redis instance to add or claim jobs. Redis's atomic operations (via Lua scripts) also let BullMQ guarantee that two workers never grab the same job, something that's hard to coordinate safely across separate Node.js processes without a shared, atomic data store.

Redis's built-in data structures - sorted sets for time-ordered data, lists for FIFO queues - map naturally onto the states a job moves through, which is part of why BullMQ (and its predecessor Bull) chose it as the sole backing store rather than building a custom protocol.

The limiter option on a Worker caps how many jobs it will process within a rolling time window, which is useful when downstream systems (a third-party API, for instance) enforce their own request limits.

const worker = new Worker('api-calls', processor, {
  connection,
  limiter: { max: 10, duration: 1000 }, // max 10 jobs per second
});

Once the limit is hit, BullMQ pauses pulling new jobs until the window resets, rather than dropping or failing them - jobs simply wait longer in the queue. Note that this limiter is per-Worker-instance in the open-source version: if you scale to five worker instances, each enforces its own 10-per-second budget, so the effective ceiling across your fleet is roughly 50 per second, not 10.

Getting a hard global cap that holds no matter how many workers you run requires either external coordination or BullMQ Pro's more advanced group-based rate limiting.

Retries are configured per job through the attempts option, which sets the total number of times BullMQ will try to run the processor before giving up.

await queue.add('charge-card', data, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
});

When the processor throws or rejects, BullMQ checks job.attemptsMade against attempts. If retries remain, the job is moved back to delayed (respecting any configured backoff) rather than straight back to waiting, so failures don't hammer a struggling downstream service immediately. Only once the attempt budget is exhausted does the job transition to failed and stay there.

You can also retry a specific failed job manually after inspecting it with job.retry(), which is handy for admin tooling that lets a human re-trigger something after fixing a root cause.

These two options work together but control different things: one sets a ceiling on retry count, the other controls the spacing between retries.

You almost always want both together: attempts without a sensible backoff can hammer a failing dependency five times in a fraction of a second, while backoff without enough attempts gives the retry logic no room to actually help.

Pass a backoff object with type: 'exponential' and a base delay in milliseconds; BullMQ doubles the wait time on each successive retry.

await queue.add('call-flaky-api', data, {
  attempts: 4,
  backoff: { type: 'exponential', delay: 1000 },
});
// retry 1: wait ~1000ms, retry 2: ~2000ms, retry 3: ~4000ms

Built-in options are 'fixed' (same delay every time) and 'exponential'. If you need jitter, a cap on the maximum delay, or a completely different curve, register a custom backoff strategy function on the Queue's settings, which receives the attempt number and returns the delay to use.

Exponential backoff is especially useful for calls to external APIs, since it naturally backs off harder the longer an outage lasts instead of retrying at a constant, potentially overwhelming rate.

The job doesn't disappear - it stays marked active in Redis along with the lock the crashed worker had acquired, but because the process is gone, nothing renews that lock anymore.

After lockDuration passes with no renewal, another running Worker's internal stall-check notices the expired lock during its normal loop and moves the job back to waiting (incrementing its stalled count), where any healthy worker can pick it up again.

This means recovery depends on at least one other Worker instance still being alive to perform that check - if every worker for a queue goes down simultaneously, stalled jobs simply wait until a worker comes back online and runs its next check cycle. It also means the job may be executed twice in total, which is why idempotent processors matter for anything with real side effects.

Stall detection is built into the Worker class and driven by three settings: lockDuration (how long a lock lasts, default 30s), stalledInterval (how often the worker scans for stalled jobs), and maxStalledCount (how many times a job may stall before it's given up on, default 1).

const worker = new Worker('reports', processor, {
  connection,
  lockDuration: 30000,
  stalledInterval: 30000,
  maxStalledCount: 2,
});

worker.on('stalled', jobId => console.warn(`Job ${jobId} stalled`));

On each stalledInterval tick, a running worker checks active jobs whose lock has expired without renewal, emits a stalled event, and moves them back to waiting. If a job stalls more times than maxStalledCount allows, it's moved to failed with a "job stalled more than allowable limit" reason instead of being retried again.

Because normal lock renewal happens automatically in the background, stalling is usually a symptom of the event loop being blocked by synchronous work, or the process dying outright - both worth investigating rather than just raising the limits.

A job's life is a sequence of state transitions, each one implemented as an atomic Redis operation so it's always well-defined which state a job is in even under concurrent access.

stateDiagram-v2
    [*] --> waiting: queue.add() (no delay)
    [*] --> delayed: queue.add() (with delay/repeat)
    delayed --> waiting: scheduled time reached
    waiting --> active: picked up by a Worker
    active --> completed: processor resolves
    active --> failed: attempts exhausted
    active --> waiting: stalled, retried
    active --> delayed: fails, backoff retry pending
    waiting-children --> waiting: all children complete
    [*] --> waiting-children: parent job in a Flow

A plain job starts in waiting (or delayed if scheduled for later). Once a Worker claims it, it becomes active. From there it either resolves into completed, or - on failure - moves to delayed for a backoff-based retry if attempts remain, or straight to failed once they're exhausted. Flow parent jobs add one extra branch: they sit in waiting-children until every child job completes, before becoming eligible to run themselves.

Adding and processing a job involves a producer, Redis, and one or more workers coordinating purely through Redis - there's no direct network connection between producer and worker.

sequenceDiagram
    participant P as Producer (Queue.add)
    participant R as Redis
    participant W as Worker

    P->>R: Write job hash + push to "wait" list
    Note over W: Worker is blocked on BRPOPLPUSH-style call
    R-->>W: Job becomes available
    W->>R: Atomically move job to "active" + acquire lock
    W->>W: Run processor(job)
    alt processor resolves
        W->>R: Move job to "completed", store returnvalue
    else processor throws
        W->>R: Move job to "delayed" or "failed" based on attempts
    end
    R-->>P: (optional) QueueEvents pub/sub notifies listeners

The producer's add() call only talks to Redis; it never talks to a worker directly. A waiting worker is effectively parked on a blocking Redis read, so as soon as a job lands in the wait list it's picked up with very low latency, without any polling delay on the worker's side.

Once processing finishes, BullMQ writes the final state (completed or failed) back into Redis and, if any QueueEvents listeners exist, publishes to a Redis pub/sub channel so those listeners find out too - decoupling "who ran the job" from "who wants to know it finished."

Flows model a "run this only after these finish" relationship declaratively, instead of you manually chaining .then() calls or polling for other jobs to complete.

flowchart TD
    A[generate-invoice] --> B[charge-card]
    A --> C[reserve-stock]
    A --> D[send-confirmation-email]

await flowProducer.add({
  name: 'generate-invoice',
  queueName: 'billing',
  data: { orderId: 99 },
  children: [
    { name: 'charge-card', queueName: 'payments', data: { orderId: 99 } },
    { name: 'reserve-stock', queueName: 'inventory', data: { orderId: 99 } },
    { name: 'send-confirmation-email', queueName: 'emails', data: { orderId: 99 } },
  ],
});

The parent (generate-invoice) won't move out of waiting-children until all three child jobs report completed. Inside the parent's processor, you can call job.getChildrenValues() to collect each child's return value, which is how you aggregate results from a fan-out of work back into one place.

By default, if any child fails permanently, the parent stays stuck in waiting-children - you can change this with the failParentOnFailure option on a child, which instead fails the parent immediately when that specific child fails.

Manual chaining means writing your own logic - typically calling queue.add() for the next step from inside a completed handler or at the end of a processor. FlowProducer instead declares the whole dependency tree upfront as data.

Manual chaining still makes sense for a simple two-step "when A finishes, kick off B" case where you don't need atomicity guarantees or result aggregation - reaching for FlowProducer everywhere would be over-engineering for that.

Because a Worker only needs the queue name and Redis connection details, scaling out is mostly a matter of running more copies of the same worker process against the same Redis instance - no special coordination code required.

• Run multiple Worker processes (or containers/pods) pointed at the same queue name and Redis
• Redis's atomic job-claiming ensures each job still only goes to exactly one of them
• Tune per-worker concurrency for I/O-bound jobs so each process handles several jobs at once, in addition to running more processes
• Use queue.setGlobalConcurrency(n) if you need a hard ceiling on total in-flight jobs across the entire fleet, since per-worker concurrency alone only bounds each instance individually

Because coordination lives entirely in Redis, adding a tenth worker instance is as simple as deploying another copy of the same worker code - there's no leader election or partitioning logic for you to write, though Redis itself becomes the shared bottleneck to watch as scale grows.

By default, a worker's processor function runs inside the same Node.js process as the Worker itself. That's fine for light, async, I/O-bound work, but it becomes a liability for CPU-heavy tasks (image processing, PDF generation, large computations) or unstable third-party code, because a crash or a long synchronous block affects the entire worker.

// processor.js - runs in its own child process
module.exports = async job => {
  return heavyComputation(job.data);
};

// main.js
const worker = new Worker('cpu-heavy', path.join(__dirname, 'processor.js'), { connection });

Sandboxed processors run each job's logic in a separate Node.js child process. If that job's code crashes or throws an uncaught exception, only the child process dies - the main Worker keeps running and simply fails that one job, instead of taking down every other job currently in flight.

This isolation also means a memory leak in job logic gets cleaned up when the child process is recycled, and truly CPU-bound work no longer blocks the main event loop that's responsible for renewing locks and picking up new jobs.

The distinction comes down to where the job's code actually executes.

You don't have to choose exclusively for your whole application: it's common to use a lightweight inline processor for most job types and reserve a sandboxed processor specifically for the one or two job types that are CPU-intensive or depend on flaky native libraries.

Every job added to BullMQ has a name - if you don't pass one, jobs default to a generic name, but you can (and often should) give jobs meaningful names to distinguish different kinds of work on the same queue.

await queue.add('resize-thumbnail', data);
await queue.add('generate-pdf', data);

const worker = new Worker('media-tasks', async job => {
  switch (job.name) {
    case 'resize-thumbnail': return resizeThumbnail(job.data);
    case 'generate-pdf': return generatePdf(job.data);
  }
});

Named jobs are worth using whenever a single queue handles more than one type of task - it keeps related work together (sharing rate limits, concurrency, and monitoring) while still letting the processor branch cleanly instead of maintaining several near-identical queues.

If a queue only ever does one kind of thing, a single default name everywhere is perfectly fine and adds no real benefit from being distinguished further.

Throughput tuning in BullMQ usually comes down to matching concurrency to the actual bottleneck (CPU vs I/O) and reducing unnecessary Redis round trips.

• Raise concurrency on workers when jobs are I/O-bound (waiting on network calls) rather than CPU-bound, since idle time can be filled with other jobs
• Use addBulk() when adding many jobs at once instead of looping over add(), cutting down round trips to Redis
• Keep removeOnComplete/removeOnFail tuned so Redis doesn't accumulate millions of finished jobs, which slows down cleanup and memory usage over time
• Avoid using explicit priority on every job - prioritized jobs are more expensive to insert than plain FIFO ones
• Scale out more Worker processes rather than pushing one worker's concurrency arbitrarily high, since a single Node.js process still has one event loop
• Run Redis close (network-wise) to your workers, since each state transition is a round trip and latency compounds under load

Before tuning blindly, measure with queue.getJobCounts() and the built-in metrics feature to confirm whether jobs are piling up in waiting (a worker-capacity problem) or active (a per-job speed problem) - the fix looks different for each.

Jobs piling up in waiting almost always mean "nothing is available (or able) to pick them up," so the investigation starts with the worker side rather than the job data itself.

• Confirm at least one Worker process is actually running and connected - check its logs for connection errors
• Verify the Worker's queue name exactly matches the Queue's name; a typo means two completely separate queues in Redis
• Check that the Worker and Queue point at the same Redis instance/database index - an easy mistake across environments
• Look at whether the queue is paused (queue.isPaused()); a paused queue accepts new jobs but a healthy worker won't pull from it
• If a rate limiter or low concurrency is configured, jobs may simply be waiting their turn rather than being stuck

A quick sanity check is await queue.getWaitingCount() alongside await queue.getActiveCount() - if active is consistently zero while waiting keeps growing, no worker is successfully claiming jobs, which points at connectivity or naming rather than the job logic itself.

Both options control automatic cleanup, but they apply to opposite outcomes of a job, and mixing up their intent is a common source of "why did my failed jobs disappear" confusion.

await queue.add('sync', data, {
  removeOnComplete: { count: 1000 }, // keep last 1000 successes
  removeOnFail: { count: 5000 },     // keep more failures for debugging
});

A common mistake is setting removeOnFail: true globally, which deletes failed jobs immediately and makes root-cause debugging in production much harder than it needs to be.

Beyond reusing a fixed jobId (which blocks an identical ID from being added again while the original is still pending), BullMQ offers a dedicated deduplication option built specifically for this.

// Simple mode: dedupe until the job completes or fails
await queue.add('sync-user', data, {
  deduplication: { id: 'sync-user-42' },
});

// Throttle mode: dedupe for a fixed time window regardless of completion
await queue.add('send-digest', data, {
  deduplication: { id: 'digest-42', ttl: 60_000 },
});

In simple mode, a second add() call with the same deduplication ID is ignored as long as the first job hasn't finished. In throttle mode, duplicates are ignored for the given ttl in milliseconds, even if the original job already completed - useful for "don't send more than one digest per user per minute" style rules.

This is a more purpose-built tool than reusing jobId, since it clearly separates "identity of the job" from "deduplication policy," and supports the time-based throttle case that a bare jobId check doesn't.

Plain BullMQ jobs without priority are dequeued in FIFO order, but several everyday features intentionally break strict ordering once you look past the simplest case.

• Concurrency > 1 - if a worker (or several workers) process multiple jobs in parallel, a slower earlier job can finish after a faster later one
• Retries with backoff - a failed job goes back to delayed and re-enters waiting only after its backoff period, by which point jobs added after it may already have been processed and completed
• Priority - an explicitly prioritized job can jump ahead of plain FIFO jobs that were added earlier
• Multiple workers - independent workers each claim jobs as soon as they're free, so global add-order across the whole fleet isn't preserved in completion order

If an application genuinely needs strict ordering, the usual approach is concurrency: 1 on a single worker for that specific queue, and avoiding priority and backoff-based reordering for that workload - effectively trading throughput for a guarantee BullMQ doesn't provide by default.

Both let you react to job lifecycle changes, but they differ in scope, transport, and completeness - important details when deciding where to put monitoring or notification logic.

A practical rule of thumb: use worker events for lightweight, same-process logging or metrics, and reach for QueueEvents whenever the listener lives in a different process than the one running the jobs - an admin dashboard, a notification service, or a separate API server, for example.

A processor can report incremental progress while it runs, which is useful for long jobs like large file conversions or multi-step imports where a caller wants live feedback rather than a single final result.

const worker = new Worker('import-csv', async job => {
  const rows = await loadRows(job.data.fileUrl);
  for (let i = 0; i < rows.length; i++) {
    await processRow(rows[i]);
    await job.updateProgress(Math.round(((i + 1) / rows.length) * 100));
  }
  return { imported: rows.length };
}, { connection });

worker.on('progress', (job, progress) => {
  console.log(`Job ${job.id} is ${progress}% done`);
});

job.updateProgress() accepts either a number (commonly a 0-100 percentage) or an arbitrary JSON object if you need to report more than a single figure, such as { stage: 'uploading', percent: 40 }. Each call writes the progress value into Redis and emits a progress event, which both local Worker listeners and QueueEvents subscribers can pick up - so a frontend progress bar backed by QueueEvents updates in near real time.

The worker-level limiter option isn't implemented by delaying jobs client-side in JavaScript - it's enforced through the same Redis-script-based coordination BullMQ uses for everything else, so it stays accurate even with several workers pulling from the same queue.

flowchart LR
    A["Worker requests next job"] --> B{Rate limit window ok?}
    B -- yes --> C["Move job to active, process it"]
    B -- no --> D["Set queue's rate-limited flag with a TTL in Redis"]
    D --> E["Worker sleeps until window resets"]
    E --> A

Internally, each time a worker is about to take a job, an atomic Lua script checks a counter tied to the configured window (max jobs per duration milliseconds). If the limit for the current window has been reached, the script marks the queue as rate-limited in Redis with an expiry equal to the remaining window time, and the worker parks itself until that marker clears rather than busy-polling.

Because the limit and counter live in Redis, it correctly throttles a single worker's own consumption - but as covered earlier, in open-source BullMQ that counter is scoped per Worker instance rather than shared across your whole fleet, which is exactly the gap BullMQ Pro's group-based rate limiting is built to close.

Because a Job Scheduler is identified by an ID you choose, calling upsertJobScheduler() again with the same ID updates its schedule in place instead of creating a second, competing one.

// Originally every hour
await queue.upsertJobScheduler('cleanup', { every: 3_600_000 });

// Later, switch the same scheduler to run at 2am daily instead
await queue.upsertJobScheduler('cleanup', { pattern: '0 2 * * *' });

To inspect what's currently scheduled, queue.getJobSchedulers() lists every active scheduler with its next run time and settings - handy for an admin panel that shows "here's everything currently scheduled to run automatically." To stop a recurring job entirely, queue.removeJobScheduler(schedulerId) deletes the scheduler and its pending placeholder job, rather than trying to track down and remove the underlying delayed job by hand.

Treating the scheduler ID as a stable identifier - rather than generating a fresh one on every deploy - is what makes "upsert" meaningful: redeploying your app with the same ID simply reconciles the existing schedule instead of duplicating it.

These two operations sound similar but solve different problems: one is about temporarily halting work, the other about discarding pending work.

A common mistake is reaching for drain() when the actual intent was "stop processing for now, but keep the jobs" - that's what pause() is for. drain() is the right call only when you genuinely want to discard everything currently waiting.

A worker killed abruptly mid-job leaves that job to be recovered later as a stalled job - acceptable occasionally, but avoidable during a routine deploy by shutting down cleanly instead.

const worker = new Worker('emails', processor, { connection });

async function shutdown() {
  console.log('Shutting down worker...');
  await worker.close(); // waits for in-progress jobs to finish
  process.exit(0);
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

worker.close() stops the Worker from picking up new jobs and waits for any currently active job to finish (up to an optional timeout you can pass in) before resolving, rather than killing it instantly. Listening for both SIGTERM (what most orchestrators like Kubernetes or Docker send on a normal stop) and SIGINT (Ctrl+C locally) covers both production and local development shutdown paths.

It's also worth closing any Queue and QueueEvents instances the same process holds, since each keeps its own Redis connection that should be released cleanly on exit.

Neither wins universally - the right choice depends on how complex and how atomic the dependency structure genuinely needs to be, not on which one is newer.

Flows are the better fit when: the number of children is dynamic (fan-out across a variable list, such as one child per uploaded file), you need atomic all-or-nothing creation of parent and children, the tree is more than one level deep, or you need to aggregate every child's result through getChildrenValues(). A video-transcoding pipeline that spins up a rendition job per target resolution is a natural fit - the number of children isn't known until runtime.

Manual orchestration remains perfectly reasonable when the dependency is a simple, fixed two-step relationship - "once the order is paid, send a receipt" - handled from a single completed event listener. Reaching for FlowProducer here adds conceptual overhead (a separate queue/job tree to reason about) without buying you anything, since there's no fan-out and no result aggregation to simplify.

The practical guideline: default to manual chaining for a fixed, small number of sequential steps, and switch to flows as soon as the structure becomes dynamic, multi-branch, or needs atomic creation guarantees.

BullMQ ships native OpenTelemetry support through a companion package, bullmq-otel, rather than requiring a third-party instrumentation library to monkey-patch its internals.

import { Queue, Worker } from 'bullmq';
import { BullMQOtel } from 'bullmq-otel';

const telemetry = new BullMQOtel({ tracerName: 'orders-service' });

const queue = new Queue('orders', { connection, telemetry });
const worker = new Worker('orders', processor, { connection, telemetry });

When a job is added, the current trace context is captured and stored with the job's data in Redis. When a worker later picks that job up, it extracts that stored context and starts a child span linked back to the original trace - so a single request that adds a job and a worker that processes it minutes later (potentially on a different machine) still show up as one connected trace in your observability backend (Jaeger, Honeycomb, Datadog, etc.).

Spans are categorized by kind - PRODUCER for operations that add jobs, CONSUMER for operations that process them - which makes it straightforward to distinguish "time spent queued" from "time spent actually processing" when reading a trace.

BullMQ Pro is Taskforce.sh's commercial edition, built on top of the same open-source core but adding features that were judged too complex, niche, or support-intensive to maintain for free.

The flagship feature is Groups: imagine one queue handling video transcoding for thousands of users - without grouping, one user submitting thousands of jobs could starve everyone else. Groups process jobs round-robin across group keys (e.g., per user), so fair scheduling happens automatically instead of requiring you to build a custom queue-per-user architecture.

Teams typically reach for Pro when they hit a genuine multi-tenant fairness problem, need job cancellation semantics, or want commercial support backing a business-critical queue - not as a default upgrade from open-source BullMQ.