How a routine deploy infected our CI server, how we hunted it down, and what actually fixed it.
The setup
Our on-prem Ubuntu server runs ~20 GitHub Actions self-hosted runners and a dozen Next.js apps under pm2. One shared box, many repos — remember that detail, it matters later.
Day 0: "Is this suspicious?"
While running a routine security audit, one check jumped out — two strange processes:
node -e global.i='7-v1729';global.e='NPM';global.r=require;...
Heavily obfuscated JavaScript, running as our deploy user, with established HTTPS connections to an unknown IP: 181.214.149.148:443.
Deobfuscated, the loader did one thing: fetch JSON from a remote server and eval() whatever came back. A classic remote-code stager — the attacker could execute anything, anytime, on our production box.
The forensics trail
What we ruled out, step by step:
- ❌ SSH brute-force — auth logs clean, all logins from our LAN
- ❌ Rogue SSH keys, extra root accounts, cron/systemd persistence — clean
- ❌ Malicious install scripts — we reinstalled with ignore-scripts=true… and it came back anyway
- ❌ App source code & workflow file — clean
- ❌ NODE_OPTIONS/env injection, npmrc tampering, pm2 modules — clean
What we confirmed:
- ✅ The loaders spawned during npm install + build of one specific app, delivered through a CI deploy
- ✅ The parent chain: pm2 → npm start → next-server → malware. Every app boot respawned it
- ✅ global.e='NPM' — the malware itself tags its delivery vector: a poisoned npm package
The scariest moment: after a full wipe (rm -rf node_modules .next + fresh install), the malware respawned within a minute — spawned mid-install, then orphaned itself to PPID 1 so it survived pm2 stop.
The resolution
We never got the exact package name — and that's a realistic detail worth sharing. Here's what the evidence showed: a dependency version that was live-poisoned on the npm registry on deploy day, cached locally, and pulled/patched upstream within 48 hours (this is how real supply-chain waves work — npm yanks compromised versions fast). Once we cleaned the npm cache and reinstalled fresh, the exact same sequence came back clean — verified end-to-end with kernel-level auditd logging armed.
What actually fixed it
- Contain: iptables DROP on the C2 IP (made persistent!), stop the infected app, kill the orphaned loaders
- Freeze the vector: stop ALL runners — deploy freeze until clean
- Wipe properly: rm -rf node_modules .next plus npm cache clean --force — the poisoned tarballs live in ~/.npm/_cacache and reinfect you from cache
- Rotate everything: GitHub PATs, runner tokens, org secrets, every .env, SSH passwords. The malware had 24h of access — cleanup does not un-steal secrets
- Verify with a real deploy, then bring runners back one at a time
- Schedule a full host rebuild — after a remote-code stager, "no evidence of a second stage" ≠ "no second stage"
Lessons learned
- ignore-scripts=true is not enough. It blocks install hooks but NOT require-time payloads that run when your app boots.
- Self-hosted runners are a blast radius multiplier. One poisoned dependency in ONE repo = full compromise of every project on the box. Isolate runners.
- npm's cache can reinfect you. Wiping node_modules without npm cache clean is not a clean install.
- Orphaned processes survive pm2 restarts. Check for PPID 1 processes you don't recognize.
- Block C2 IPs at the firewall AND persist the rule — an in-memory iptables rule dies on reboot.
- Rotate credentials immediately. This is the step everyone delays and the one that matters most.
- A ufw status that says inactive on a production server is a finding in itself. 🙃
This was my first incident of this kind — if you run self-hosted runners on a shared box, I hope you never need this post. But if you do: contain, freeze, wipe (cache too!), rotate, verify, rebuild.
