Implementing a Background Job Queue with BullMQ & Redis
Why I Needed a Message Queue
In my project, I had a Quotation feature where:
Sales users submit quotation data
A DOCX template is filled dynamically
The DOCX is converted to PDF using LibreOffice
The PDF is uploaded to cloud storage
The PDF is emailed to the customer
Initially, all of this was done inside the API request.
Problems with that approach
PDF generation is slow (seconds)
LibreOffice is CPU-heavy
HTTP requests timeout
Multiple users cause server crashes
If PDF fails → entire API fails
So I needed a way to:
Respond to the client immediately
Do heavy work in the background
Retry on failure
Scale safely
This is where message queues come in.
What Is a Message Queue (In Simple Words)
A message queue lets you:
Push work into a queue
Process it later
Retry if it fails
Run it independently of API requests
In my case:
API creates a quotation
Queue handles PDF + Email
Tech Stack I Used
| Tool | Purpose |
| BullMQ | Job queue & retry logic |
| Redis (Upstash) | Job storage & coordination |
| Worker Process | Executes background tasks |
| LibreOffice | DOCX → PDF conversion |
| Bull Board | Queue dashboard |
| PostgreSQL | Business data |
High-Level Architecture
Client
↓
API (Express)
↓
Database (save quotation)
↓
BullMQ Queue
↓
Redis
↓
Worker
↓
PDF Generation + Email
↓
Update Database
Why BullMQ + Redis?
BullMQ (the brain)
Job lifecycle
Retry logic
Delays
Concurrency
Failure handling
Redis (the memory)
Stores jobs
Maintains job states
Locks jobs (no duplicates)
Survives crashes
BullMQ decides what to do
Redis remembers everything
Important Lesson: Worker ≠ API
Wrong
import "./worker"; // inside server.ts
Correct
node server.js # API
node worker.js # Worker
This separation avoids:
Duplicate job execution
Crashes affecting both systems
Scaling issues
What Happens If a Job Fails?
This was a key design question.
Flow:
Client already got success response
Worker fails (PDF error)
Job retries automatically
If still fails → marked FAILED
DB is updated with error
Quotation is never lost.
Database = source of truth
Queue = side effects
Final Takeaway
Message queues are not optional for heavy tasks.
They are a core backend pattern.
If your API:
Generates files
Sends emails
Talks to external systems
Does CPU-heavy work
👉 Use a queue
Author Note
This implementation helped me deeply understand:
Async systems
Eventual consistency
Background processing
Production-grade backend design