<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Building in Public]]></title><description><![CDATA[Building in Public]]></description><link>https://background-job-queue-with-bullmq-and-redis.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sat, 20 Jun 2026 19:43:22 GMT</lastBuildDate><atom:link href="https://background-job-queue-with-bullmq-and-redis.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Building a Secure Production Backend on AWS EC2 with CI/CD, HTTPS & PM2]]></title><description><![CDATA[In this blog, I’ll walk through how I deployed my Node.js backend on AWS EC2, configured Nginx, managed processes with PM2, secured the API using HTTPS (Let’s Encrypt), and finally connected it to a custom domain.
This wasn’t smooth — I ran into SSH ...]]></description><link>https://background-job-queue-with-bullmq-and-redis.hashnode.dev/building-a-secure-production-backend-on-aws-ec2-with-cicd-https-and-pm2</link><guid isPermaLink="true">https://background-job-queue-with-bullmq-and-redis.hashnode.dev/building-a-secure-production-backend-on-aws-ec2-with-cicd-https-and-pm2</guid><category><![CDATA[Devops]]></category><category><![CDATA[deployment]]></category><category><![CDATA[Reverse Proxy]]></category><category><![CDATA[AWS]]></category><category><![CDATA[cicd]]></category><category><![CDATA[ec2]]></category><category><![CDATA[Backend Deployment]]></category><dc:creator><![CDATA[Atharva G]]></dc:creator><pubDate>Mon, 19 Jan 2026 04:58:46 GMT</pubDate><content:encoded><![CDATA[<p>In this blog, I’ll walk through how I deployed my Node.js backend on <strong>AWS EC2</strong>, configured <strong>Nginx</strong>, managed processes with <strong>PM2</strong>, secured the API using <strong>HTTPS (Let’s Encrypt)</strong>, and finally connected it to a custom domain.</p>
<p>This wasn’t smooth — I ran into SSH issues, mixed content errors, missing packages, DNS delays, and broken deployments. But that’s exactly why this experience mattered.</p>
<h2 id="heading-tech-stack-used">Tech Stack Used</h2>
<p>Here’s what I used for this deployment:</p>
<ul>
<li><p><strong>Backend:</strong> Node.js + Express</p>
</li>
<li><p><strong>Database:</strong> PostgreSQL</p>
</li>
<li><p><strong>Cloud Provider:</strong> AWS</p>
</li>
<li><p><strong>Server:</strong> EC2 (Ubuntu)</p>
</li>
<li><p><strong>Process Manager:</strong> PM2</p>
</li>
<li><p><strong>Reverse Proxy:</strong> Nginx</p>
</li>
<li><p><strong>SSL:</strong> Let’s Encrypt (Certbot)</p>
</li>
<li><p><strong>CI/CD:</strong> GitHub Actions</p>
</li>
<li><p><strong>Domain &amp; DNS:</strong> Hostinger</p>
</li>
<li><p><strong>Version Control:</strong> Git &amp; GitHub</p>
</li>
</ul>
<h2 id="heading-architecture-diagram">Architecture diagram</h2>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1768798340187/2ea47c82-00a3-4284-b974-91079df9e129.png" alt class="image--center mx-auto" /></p>
<h2 id="heading-step-1-launching-an-ec2-instance">Step 1: Launching an EC2 Instance</h2>
<p>I started by launching an <strong>EC2 instance</strong> on AWS using Ubuntu.</p>
<h3 id="heading-key-decisions">Key decisions:</h3>
<ul>
<li><p>Instance type suitable for backend workloads (free-tier friendly)</p>
</li>
<li><p>Created a <strong>key pair (.pem file)</strong> for SSH access</p>
</li>
<li><p>Configured <strong>security groups</strong> to allow:</p>
<ul>
<li><p>SSH (22)</p>
</li>
<li><p>HTTP (80)</p>
</li>
<li><p>HTTPS (443)</p>
</li>
<li><p>Backend port (initially 5000)</p>
</li>
</ul>
</li>
</ul>
<p>Security groups act like a firewall. Without opening the correct ports, nothing works — learned this the hard way.</p>
<h2 id="heading-step-2-connecting-to-the-server-via-ssh">Step 2: Connecting to the Server via SSH</h2>
<p>Using the <code>.pem</code> key, I connected to the server:</p>
<pre><code class="lang-powershell">ssh <span class="hljs-literal">-i</span> <span class="hljs-string">"filename.pem"</span> ubuntu<span class="hljs-selector-tag">@</span>&lt;EC2_PUBLIC_IP&gt;
</code></pre>
<p>The first time you connect, AWS asks you to trust the server’s fingerprint — this is normal and ensures you’re connecting to the right machine.</p>
<h2 id="heading-step-3-setting-up-the-backend-environment">Step 3: Setting Up the Backend Environment</h2>
<h3 id="heading-installing-nodejs-via-nvm">Installing Node.js (via NVM)</h3>
<p>Instead of installing Node directly, I used <strong>NVM</strong>, which makes version management easier:</p>
<pre><code class="lang-powershell"><span class="hljs-built_in">curl</span> <span class="hljs-literal">-o</span>- https://raw.githubusercontent.com/nvm<span class="hljs-literal">-sh</span>/nvm/v0.<span class="hljs-number">39.7</span>/install.sh | bash
export NVM_DIR=<span class="hljs-string">"<span class="hljs-variable">$HOME</span>/.nvm"</span>
source <span class="hljs-string">"<span class="hljs-variable">$NVM_DIR</span>/nvm.sh"</span>
nvm install <span class="hljs-number">20</span>
nvm use <span class="hljs-number">20</span>
</code></pre>
<p>This step is critical — missing Node or NPM caused multiple deployment failures later when CI/CD ran.</p>
<h2 id="heading-step-4-running-the-backend-with-pm2">Step 4: Running the Backend with PM2</h2>
<p>Once the backend code was on the server, I used <strong>PM2</strong> to keep the app running even after crashes or reboots.</p>
<pre><code class="lang-powershell">npm install <span class="hljs-literal">-g</span> pm2
pm2 <span class="hljs-built_in">start</span> server.js -<span class="hljs-literal">-name</span> <span class="hljs-string">"backend"</span>
pm2 save
</code></pre>
<p>PM2 acts like a supervisor. Without it, your backend dies the moment your SSH session closes.</p>
<h2 id="heading-step-5-using-nginx-as-a-reverse-proxy">Step 5: Using Nginx as a Reverse Proxy</h2>
<p>Exposing the backend directly on port <code>5000</code> isn’t ideal.<br />Instead, I configured <strong>Nginx</strong> to act as a reverse proxy:</p>
<pre><code class="lang-powershell">Client → Nginx (<span class="hljs-number">80</span>/<span class="hljs-number">443</span>) → Node.js (<span class="hljs-number">5000</span>)
</code></pre>
<p>This gives:</p>
<ul>
<li><p>Cleaner URLs</p>
</li>
<li><p>Better security</p>
</li>
<li><p>HTTPS support</p>
</li>
<li><p>Easier scaling later</p>
</li>
</ul>
<p>Once Nginx was set up, I could access my API via a domain instead of an IP + port.</p>
<h2 id="heading-step-6-setting-up-a-custom-domain">Step 6: Setting Up a Custom Domain</h2>
<p>I bought a domain and created an <strong>A record</strong>:</p>
<div class="hn-table">
<table>
<thead>
<tr>
<td>Name</td><td>Value (IP)</td></tr>
</thead>
<tbody>
<tr>
<td>api</td><td>EC2 Public IP</td></tr>
</tbody>
</table>
</div><p>DNS takes time to propagate, so initially <code>ping</code> failed — this is expected. Waiting patiently fixed it.</p>
<h2 id="heading-step-7-enabling-https-with-lets-encrypt">Step 7: Enabling HTTPS with Let’s Encrypt</h2>
<p>This was one of the most satisfying steps.</p>
<p>Using <strong>Certbot</strong>, I generated a free SSL certificate:</p>
<pre><code class="lang-powershell">sudo certbot -<span class="hljs-literal">-nginx</span> <span class="hljs-literal">-d</span> api.caterview.online
</code></pre>
<p>After this:</p>
<ul>
<li><p>HTTPS was enabled</p>
</li>
<li><p>Certificates were auto-renewable</p>
</li>
<li><p>Browser warnings disappeared</p>
</li>
</ul>
<p>Seeing this message felt unreal:</p>
<blockquote>
<p>“Congratulations! You have successfully enabled HTTPS.”</p>
</blockquote>
<h2 id="heading-step-8-fixing-mixed-content-errors">Step 8: Fixing Mixed Content Errors</h2>
<p>At one point, my frontend (HTTPS) was calling an HTTP backend, which caused:</p>
<pre><code class="lang-powershell">blocked:mixed<span class="hljs-literal">-content</span>
</code></pre>
<p>The fix was simple but important:</p>
<ul>
<li><p>Make <strong>all backend URLs HTTPS</strong></p>
</li>
<li><p>Update frontend environment variables</p>
</li>
</ul>
<p>Security is only as strong as its weakest link.</p>
<h2 id="heading-step-9-automating-deployment-with-github-actions">Step 9: Automating Deployment with GitHub Actions</h2>
<p>Manually pulling code every time was inefficient, so I set up <strong>CI/CD</strong>.</p>
<p>Whenever I push to <code>main</code>:</p>
<ul>
<li><p>GitHub Actions SSHs into EC2</p>
</li>
<li><p>Pulls latest code</p>
</li>
<li><p>Installs dependencies</p>
</li>
<li><p>Restarts the backend via PM2</p>
</li>
</ul>
<p>This made deployments fast, consistent, and repeatable.</p>
<h2 id="heading-common-issues-i-faced-and-learned-from">Common Issues I Faced (and Learned From)</h2>
<ul>
<li><p>SSH timeouts due to security group misconfiguration</p>
</li>
<li><p><code>npm: command not found</code> (Node not installed on EC2)</p>
</li>
<li><p><code>pm2: command not found</code> (global install missing)</p>
</li>
<li><p>DNS not resolving immediately</p>
</li>
<li><p>Mixed content errors between frontend &amp; backend</p>
</li>
</ul>
<p>Every issue forced me to understand <strong>why things work</strong>, not just how.</p>
<h2 id="heading-final-result">Final Result</h2>
<ul>
<li><p>Backend running on AWS EC2</p>
</li>
<li><p>Secure HTTPS endpoint</p>
</li>
<li><p>Managed by PM2</p>
</li>
<li><p>Accessible via custom domain</p>
</li>
<li><p>Auto-deployed using CI/CD</p>
</li>
</ul>
<p>This felt like a real engineering milestone.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Deploying this backend taught me more than dozens of tutorials ever could.<br />It pushed me beyond writing code into <strong>owning software in production</strong>.</p>
<p>If you’re a student or fresher — don’t stop at <a target="_blank" href="http://localhost">localhost</a>.<br />Deploy something. Break it. Fix it. Learn from it.</p>
<p>That’s how you grow.</p>
]]></content:encoded></item><item><title><![CDATA[Implementing a Background Job Queue with BullMQ & Redis]]></title><description><![CDATA[Why I Needed a Message Queue
In my project, I had a Quotation feature where:

Sales users submit quotation data

A DOCX template is filled dynamically

The DOCX is converted to PDF using LibreOffice

The PDF is uploaded to cloud storage

The PDF is e...]]></description><link>https://background-job-queue-with-bullmq-and-redis.hashnode.dev/implementing-a-background-job-queue-with-bullmq-and-redis</link><guid isPermaLink="true">https://background-job-queue-with-bullmq-and-redis.hashnode.dev/implementing-a-background-job-queue-with-bullmq-and-redis</guid><category><![CDATA[General Programming]]></category><category><![CDATA[backend]]></category><category><![CDATA[message queue]]></category><category><![CDATA[Redis]]></category><category><![CDATA[bullmq]]></category><dc:creator><![CDATA[Atharva G]]></dc:creator><pubDate>Mon, 12 Jan 2026 04:34:20 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1768192212611/d83d8687-356c-477a-b037-604e35fdf782.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2 id="heading-why-i-needed-a-message-queue">Why I Needed a Message Queue</h2>
<p>In my project, I had a <strong>Quotation feature</strong> where:</p>
<ul>
<li><p>Sales users submit quotation data</p>
</li>
<li><p>A <strong>DOCX template</strong> is filled dynamically</p>
</li>
<li><p>The DOCX is converted to <strong>PDF using LibreOffice</strong></p>
</li>
<li><p>The PDF is uploaded to cloud storage</p>
</li>
<li><p>The PDF is emailed to the customer</p>
</li>
</ul>
<p>Initially, all of this was done <strong>inside the API request</strong>.</p>
<h3 id="heading-problems-with-that-approach">Problems with that approach</h3>
<ul>
<li><p>PDF generation is <strong>slow (seconds)</strong></p>
</li>
<li><p>LibreOffice is <strong>CPU-heavy</strong></p>
</li>
<li><p>HTTP requests <strong>timeout</strong></p>
</li>
<li><p>Multiple users cause <strong>server crashes</strong></p>
</li>
<li><p>If PDF fails → entire API fails</p>
</li>
</ul>
<p>So I needed a way to:</p>
<blockquote>
<p>Respond to the client immediately<br />Do heavy work <strong>in the background</strong><br />Retry on failure<br />Scale safely</p>
</blockquote>
<p>This is where <strong>message queues</strong> come in.</p>
<h2 id="heading-what-is-a-message-queue-in-simple-words">What Is a Message Queue (In Simple Words)</h2>
<p>A message queue lets you:</p>
<ul>
<li><p>Push work into a queue</p>
</li>
<li><p>Process it later</p>
</li>
<li><p>Retry if it fails</p>
</li>
<li><p>Run it independently of API requests</p>
</li>
</ul>
<p>In my case:</p>
<ul>
<li><p><strong>API creates a quotation</strong></p>
</li>
<li><p><strong>Queue handles PDF + Email</strong></p>
</li>
</ul>
<h2 id="heading-tech-stack-i-used">Tech Stack I Used</h2>
<div class="hn-table">
<table>
<thead>
<tr>
<td>Tool</td><td>Purpose</td></tr>
</thead>
<tbody>
<tr>
<td><strong>BullMQ</strong></td><td>Job queue &amp; retry logic</td></tr>
<tr>
<td><strong>Redis (Upstash)</strong></td><td>Job storage &amp; coordination</td></tr>
<tr>
<td><strong>Worker Process</strong></td><td>Executes background tasks</td></tr>
<tr>
<td><strong>LibreOffice</strong></td><td>DOCX → PDF conversion</td></tr>
<tr>
<td><strong>Bull Board</strong></td><td>Queue dashboard</td></tr>
<tr>
<td><strong>PostgreSQL</strong></td><td>Business data</td></tr>
</tbody>
</table>
</div><h2 id="heading-high-level-architecture"><strong>High-Level Architecture</strong></h2>
<p>Client<br />↓<br />API (Express)<br />↓<br />Database (save quotation)<br />↓<br />BullMQ Queue<br />↓<br />Redis<br />↓<br />Worker<br />↓<br />PDF Generation + Email<br />↓<br />Update Database</p>
<h2 id="heading-why-bullmq-redis">Why BullMQ + Redis?</h2>
<h3 id="heading-bullmq-the-brain">BullMQ (the brain)</h3>
<ul>
<li><p>Job lifecycle</p>
</li>
<li><p>Retry logic</p>
</li>
<li><p>Delays</p>
</li>
<li><p>Concurrency</p>
</li>
<li><p>Failure handling</p>
</li>
</ul>
<h3 id="heading-redis-the-memory">Redis (the memory)</h3>
<ul>
<li><p>Stores jobs</p>
</li>
<li><p>Maintains job states</p>
</li>
<li><p>Locks jobs (no duplicates)</p>
</li>
<li><p>Survives crashes</p>
</li>
</ul>
<blockquote>
<p>BullMQ decides <strong>what to do</strong><br />Redis remembers <strong>everything</strong></p>
</blockquote>
<h2 id="heading-important-lesson-worker-api">Important Lesson: Worker ≠ API</h2>
<p><strong>Wrong</strong></p>
<pre><code class="lang-javascript"><span class="hljs-keyword">import</span> <span class="hljs-string">"./worker"</span>; <span class="hljs-comment">// inside server.ts</span>
</code></pre>
<p><strong>Correct</strong></p>
<pre><code class="lang-powershell">node server.js   <span class="hljs-comment"># API</span>
node worker.js   <span class="hljs-comment"># Worker</span>
</code></pre>
<p>This separation avoids:</p>
<ul>
<li><p>Duplicate job execution</p>
</li>
<li><p>Crashes affecting both systems</p>
</li>
<li><p>Scaling issues</p>
</li>
</ul>
<h2 id="heading-what-happens-if-a-job-fails">What Happens If a Job Fails?</h2>
<p>This was a key design question.</p>
<h3 id="heading-flow">Flow:</h3>
<ol>
<li><p>Client already got success response</p>
</li>
<li><p>Worker fails (PDF error)</p>
</li>
<li><p>Job retries automatically</p>
</li>
<li><p>If still fails → marked FAILED</p>
</li>
<li><p>DB is updated with error</p>
</li>
</ol>
<p>Quotation is <strong>never lost</strong>.</p>
<blockquote>
<p>Database = source of truth<br />Queue = side effects</p>
</blockquote>
<h2 id="heading-final-takeaway">Final Takeaway</h2>
<blockquote>
<p>Message queues are not optional for heavy tasks.<br />They are a <strong>core backend pattern</strong>.</p>
</blockquote>
<p>If your API:</p>
<ul>
<li><p>Generates files</p>
</li>
<li><p>Sends emails</p>
</li>
<li><p>Talks to external systems</p>
</li>
<li><p>Does CPU-heavy work</p>
</li>
</ul>
<p>👉 <strong>Use a queue</strong></p>
<h2 id="heading-author-note">Author Note</h2>
<p>This implementation helped me deeply understand:</p>
<ul>
<li><p>Async systems</p>
</li>
<li><p>Eventual consistency</p>
</li>
<li><p>Background processing</p>
</li>
<li><p>Production-grade backend design</p>
</li>
</ul>
]]></content:encoded></item></channel></rss>