Background Jobs in Node.js: Building Production Queues with BullMQ

Every production Node.js app eventually needs background processing. Email sending shouldn't block API responses. Image resizing shouldn't timeout HTTP requests. Data exports shouldn't crash the event loop. BullMQ — a Redis-based queue system — provides the missing piece: reliable, observable, and resumable job processing.

Why BullMQ Over Alternatives?

BullMQ (successor to Bull) offers: persistence (jobs survive server restarts), retries with backoff, rate limiting, delayed jobs, job prioritization, and a built-in UI for monitoring. It's used by thousands of companies including Amazon, Microsoft, and Spotify for critical job processing.

Basic Setup: Producer and Worker

import { Queue, Worker } from 'bullmq';
import Redis from 'ioredis';
const connection = new Redis(process.env.REDIS_URL, {
maxRetriesPerRequest: null,
});
// Producer: Add jobs to queue
const emailQueue = new Queue(‘email’, { connection });
app.post(‘/api/send-welcome-email’, async (req, res) => {
await emailQueue.add(‘welcome’, {
to: req.body.email,
name: req.body.name,
}, {
attempts: 3,
backoff: { type: ‘exponential’, delay: 1000 },
removeOnComplete: true,
removeOnFail: false,
});
res.json({ queued: true });
});
// Worker: Process jobs
const emailWorker = new Worker(‘email’, async (job) => {
const { to, name } = job.data;
// Simulate email sending
await sendEmail({
to,
subject: ‘Welcome!’,
html: Hello ${name}, thanks for joining.
});
return { delivered: true, to };
}, { connection });
emailWorker.on(‘completed’, (job) => {
console.log(Job ${job.id} completed for ${job.data.to});
});
emailWorker.on(‘failed’, (job, err) => {
console.error(Job ${job.id} failed:, err);
});

Advanced Pattern: Image Processing with Rate Limiting

When processing images, you might want to limit concurrent operations to avoid exhausting memory:

const imageQueue = new Queue('image-processing', {
  connection,
  defaultJobOptions: {
    attempts: 2,
    timeout: 30000, // 30 seconds
  }
});

const imageWorker = new Worker('image-processing', async (job) => {
  const { originalPath, sizes } = job.data;
  
  const results = [];
  for (const size of sizes) {
    const processed = await sharp(originalPath)
      .resize(size.width, size.height)
      .toBuffer();
    await uploadToS3(processed, `uploads/${job.id}-${size.width}.jpg`);
    results.push(size);
  }
  return results;
}, {
  connection,
  concurrency: 2,  // Process only 2 images at once
  limiter: {
    max: 10,        // Max 10 jobs per
    duration: 60000 // 60 seconds
  }
});

Real-World: Multi-Step Order Processing Pipeline

Complex workflows can be chained using BullMQ flows:

import { FlowProducer } from 'bullmq';

const flowProducer = new FlowProducer({ connection });

await flowProducer.add({
  name: 'order-pipeline',
  queueName: 'order-flow',
  data: { orderId: 123, items: [...] },
  children: [
    {
      name: 'validate-stock',
      queueName: 'validation',
      data: { orderId: 123 },
      children: [
        { name: 'reserve-stock', queueName: 'inventory', data: { orderId: 123 } },
        { name: 'charge-payment', queueName: 'payment', data: { orderId: 123 } }
      ]
    },
    {
      name: 'send-confirmation',
      queueName: 'notification',
      data: { orderId: 123 }
    }
  ]
});

Monitoring and Observability

BullMQ provides a built-in UI (bull-board) and metrics:

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/admin/queues');

createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(imageQueue)
  ],
  serverAdapter,
});

app.use('/admin/queues', serverAdapter.getRouter());

// Programmatic monitoring
const metrics = await emailQueue.getJobCounts();
console.log({
  waiting: metrics.waiting,
  active: metrics.active,
  completed: metrics.completed,
  failed: metrics.failed,
});

// Get failed jobs for debugging
const failedJobs = await emailQueue.getFailed();
for (const job of failedJobs) {
  console.log(`Job ${job.id} failed:`, job.failedReason);
}

Production Configuration

For high-scale systems (1000+ jobs/second), configure Redis and BullMQ carefully:

// Redis optimized config
const connection = new Redis(process.env.REDIS_URL, {
  maxRetriesPerRequest: null,
  enableReadyCheck: false,
  lazyConnect: true,
  commandTimeout: 5000,
  keepAlive: 10000,
});

// Worker with graceful shutdown
const worker = new Worker('queue', processor, { connection });

process.on('SIGTERM', async () => {
  await worker.close(); // Stop receiving new jobs, finish current
  await connection.quit();
  process.exit(0);
});

// Job concurrency tuning: start with 1x CPU cores, benchmark
const optimalConcurrency = Math.max(1, require('os').cpus().length / 2);

Common Mistakes

• Not handling job idempotency: Jobs may be retried. Make your job processor idempotent (check if already done before processing).
• Storing large payloads in job data: Redis has memory limits. Store references (S3 paths, DB IDs) instead of 50MB JSON blobs.
• Infinite retries without backoff: Always set attempts (3-5) and exponential backoff to prevent log flooding.
• No dead-letter queue: Failed jobs after max retries disappear. Use removeOnFail: false and alert on failed jobs.

Alternatives to BullMQ

• Bull (v3): Mature but less maintained. BullMQ is successor.
• Agenda: MongoDB-backed, simpler but less scalable.
• Bee-Queue: Minimalist, Redis, but lacks UI and advanced features.
• Cloud solutions: AWS SQS/SNS, GCP Tasks, Azure Queue — more complex but fully managed.

Conclusion

BullMQ is the gold standard for background jobs in Node.js. Start with a simple email queue, then expand to image processing, data exports, and complex workflows. The Redis requirement is a small price for reliability and monitoring. Implement idempotency from day one, set sane retry policies, and always monitor failed jobs. Your API response times will drop dramatically, and your users will never wait for background tasks again.