- prod-opsgdpr
The client wanted production data in preprod. I said no, then gave them something better.
A reasonable request from a reasonable client: make preprod look like prod, with real data. Why that would have been the worst security decision in the project, how I pushed back without a fight, and the anonymised seed pipeline that made preprod more useful than a copy would have been.
- prod-opssecurity
The certificate expired in silence, and the scans never stopped
A renewal cron ran every night for months and never renewed anything, because of one wrong letter. Meanwhile the server took thousands of scans and brute-force attempts a day. Here is what CrowdSec actually blocked, the nginx rules that held, the ports I should never have exposed, and why Cloudflare is the next layer.
- prod-opsdevops
Users emailed me about 500s before I ever saw them
Every error in production was logged with console.error and then died in a log file nobody read. Customers became my error tracker. Here is how I wired Sentry-compatible tracking into Express and React with sourcemaps, grouping, and one alert rule that actually fires.
-
I was grepping JSON over SSH while production was down
During an incident, my observability stack was an SSH session, docker logs, and grep. Here is the zero-budget setup that replaced it on the same VPS: Uptime Kuma on the healthchecks I already had, Dozzle for live logs, and the alert that now reaches my phone.
- prod-opsdevops
The Friday push that taught me to ban commits on main
A git push at 6pm on a Friday deployed straight to production, untested, and took the API down. Nothing in the pipeline could have stopped it, because there was no pipeline. Here is the one I built: protected main, CI that runs the tests, images built off the box.
-
Database backups that actually restore: the pgdata I lost
I deleted a Postgres data directory that lived in a bind mount with nothing behind it. My backup plan was hope. Here is the cheap, single-VPS backup setup I should have had, including the step everyone skips.
-
Docker never restarted my database: the price of restart: no
A container died at night and nothing brought it back, because every service was set to restart: no. The healthchecks I did have could not help, and the reason why surprises most people.
-
The tests were already written. Nothing ever ran them.
A regression reached production, and the test that would have caught it was already in the repo. There was no test script, and the only CI workflow never triggered on a push. How I wired it up.
-
Docker bypassed my firewall: the database port was open to the internet
My host firewall blocked port 5432. It was reachable from the internet anyway, because Docker writes its own iptables rules that your INPUT and ufw rules never see. Why it happens, and how to actually close it.
-
The docker build that filled the disk and took down production
Production went down with no code change. The cause was a docker compose build orphaning a new <none> image on every deploy until the disk filled. What actually cleared it, and the deeper fix.
-
Shipping a SaaS on one VPS with Docker Compose: an honest audit
The whole product runs on a single VPS behind one docker-compose.yml. Here is why that was the right call at launch, and an honest map of every single point of failure it hides.