The certificate expired in silence, and the scans never stopped

The browser said the connection was not private. The certificate had expired at 02:00 that morning. The renewal cron had run at 03:00, as it had every night for months, and reported success, as it had every night for months. It had never renewed anything.

That was the loud incident. The quiet one was permanent: the server’s auth logs and nginx logs showed scans, credential stuffing and path probing all day, every day, from the moment the domain had a DNS record. Nothing about that is unusual. What matters is what stood between it and the database.

This is part 8 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the perimeter: what was actually protecting the box, what was not, and the cert that expired while a script said everything was fine.

The cron that lied for months

Here is the renewal script, trimmed and anonymised, as it sat on the server:

#!/bin/bash
# Remove set -e, handle errors manually so nginx always restarts

echo "Stopping Nginx container..."
sudo docker stop nginx

echo "Renewing certificates with Certbot..."
sudo certbot renew --standalone --non-interactive --cert-name api.exampel.com
sudo certbot renew --standalone --non-interactive --cert-name app.exampel.com
sudo certbot renew --standalone --non-interactive --cert-name storage.exampel.com

echo "Regenerating default self-signed certificate..."
bash ./generate-default-ssl.sh

echo "Starting Nginx container..."
sudo docker start nginx

echo "SSL renewal process completed."

Count the failures. There are four, and each alone would have been enough.

One wrong letter in the domain. Look at the --cert-name arguments. The certificates on disk were issued for example.com. The script asked certbot to renew exampel.com. Certbot, correctly, found no such certificate and said so. Nothing was renewed, ever, from the day the script was written.

set -e had been removed on purpose. The comment explains why: so that if renewal failed, the script would still restart nginx. Reasonable intent. The consequence was that certbot’s “no certificate found” exit code was swallowed, the script continued, and the final line printed SSL renewal process completed. into the log every single night. A log that says “completed” is not a log that says “succeeded”.

A hyphen where the filesystem had an underscore. The script calls generate-default-ssl.sh. The file in the repository is generate_default_ssl.sh. That step failed too, silently, for the same reason as the one before it.

The cron entry itself. The README documented it as 0 3 * * * sudo PATH/app/nginx/ssl_renewal.sh, with PATH as a literal placeholder to be filled in. Whether the placeholder ever got replaced is a question the expired certificate answered.

Four independent bugs, and the system reported success. The only external signal was a browser warning, on the day it expired. Part 6 now has a monitor that warns fourteen days before expiry, and that monitor exists because of this morning.

There is a fifth problem that is not a bug: the script stops nginx to renew. certbot --standalone needs port 80, nginx has port 80, so the site went down for the duration of every renewal attempt. Every night at 03:00, a few seconds of planned outage, to run a script that did nothing.

What was actually holding the line

Now the good news, because there was some. The server had two real layers of defence that did their job throughout.

CrowdSec on the host

CrowdSec reads the nginx and SSH logs, matches them against community-maintained attack scenarios, and bans the source IP at the firewall, with a nginx bouncer for the HTTP side. Pulling the decision list after a month gave a clear picture of what a small SaaS with no public profile attracts:

$ sudo cscli decisions list --all | wc -l
# roughly 2,300 active bans on a typical day

$ sudo cscli metrics
# scenarios firing most, in order:
#   crowdsecurity/http-probing         path scanners: /.env, /wp-login.php, /.git/config
#   crowdsecurity/http-bad-user-agent  known scanner signatures
#   crowdsecurity/ssh-bf               SSH brute force
#   crowdsecurity/http-crawl-non_statics

The community blocklist is the part that earns its keep: most of those bans were applied before the IP had sent a single request to this server, because it had already been caught attacking someone else. For zero cost and one package install, that is an extraordinary trade.

The nginx rules

In front of the application, nginx had a set of rules that had been added one incident at a time. Trimmed and anonymised:

# rate limits: one bucket for pages, a tighter one for the API
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=api:10m rate=5r/s;
limit_conn_zone $binary_remote_addr zone=perip:10m;

# scanner user agents get no response at all
map $http_user_agent $badagent {
    default 0;
    ~*(masscan|zgrab|nikto|sqlmap|nmap)  1;
    ~*python-requests                     1;
    "~*^$"                                1;   # empty UA
}

server {
    listen 443 ssl;
    server_name api.example.com;
    ssl_protocols TLSv1.2 TLSv1.3;

    if ($badagent) { return 444; }

    # nothing that starts with a dot is ever a legitimate request
    location ~ /\. { deny all; return 444; }

    location / {
        limit_req  zone=api burst=20 nodelay;
        limit_conn perip 30;
        proxy_pass http://api:8000;
    }
}

# any hostname we did not configure gets dropped, not served
server {
    listen 443 ssl default_server;
    return 444;
}

return 444 is nginx’s “close the connection without responding”, and it is the correct answer to a scanner: no status code, no headers, no server banner, nothing to fingerprint. The default_server block matters more than it looks. Without it, a request to the bare IP or a random hostname gets served the first configured site, which tells an attacker exactly what is behind the address.

Neither layer is exotic. Both had been quietly doing their job while I worried about other things, which is the highest compliment infrastructure can get.

What was not held: the ports

Then the part that undoes some of the above. The compose file, at the time:

# docker-compose.yml, trimmed and anonymised
  storage:
    command: ['server', '/data', '--console-address', ':9001']
    # TODO change to expose in production (only for 9000) /!\
    ports:
      - '9000:9000'
      - '9001:9001'

  api:
    ports:
      - '8000:8000'

  db:
    ports:
      - '127.0.0.1:5432:5432'

The database line is right, and it is right because of part 2, where I learned that Docker’s published ports bypass the host firewall entirely. Binding to 127.0.0.1 is the only thing that reliably keeps a published port off the internet.

The other two lines are wrong, and the TODO proves I knew it. The MinIO admin console on 9001 and the API on 8000 were both published to every interface, which means both were reachable directly from the internet, past nginx, past the rate limits, past the bad-agent filter, past CrowdSec’s nginx bouncer. The admin console is a login page for the storage of every file in the product. It was one guessed password away from the world, and the nginx rules I was proud of were not in the path.

The fix is the one part 2 already spelled out and I had not finished applying: internal services need no published ports at all, because nginx reaches them by container name over the Docker network.

  storage:
    command: ['server', '/data', '--console-address', ':9001']
    # no ports: nginx proxies storage:9000, the console is not exposed anywhere

  api:
    # no ports: nginx proxies api:8000

Two deletions. After them, nmap from outside shows 22, 80 and 443, and nothing else. The TODO had been in the file for eleven months.

While I was in there: the README contained the server’s public IP and the name of the SSH key file. Neither is a secret in the cryptographic sense, and both are exactly what you do not want in a repository that might one day have a second contributor. Both are gone, replaced with placeholders.

Cloudflare: the layer that removes the cron

The certificate problem has a small fix and a structural one. The small fix is to replace --standalone with --webroot so renewal never stops nginx, correct the domains, restore set -e, and put the expiry monitor from part 6 in front of it. That is a good afternoon’s work and I did it first, because the certificate was expired now.

The structural fix is to stop terminating public TLS on the box at all. Cloudflare in proxy mode sits in front of the server, holds the public certificate, and renews it as part of running a CDN, which is to say never as a thing I think about. Between Cloudflare and nginx, an origin certificate issued by Cloudflare, valid for fifteen years, does the encryption. The renewal cron is deleted, not fixed.

What it adds on top of removing the cron:

• The server’s IP disappears. DNS points at Cloudflare. Scanners hitting the domain hit Cloudflare’s edge. The direct-to-IP attacks stop reaching the box at all, provided the firewall only accepts 443 from Cloudflare’s published ranges, which is a short script with ufw and their IP list.
• The WAF and bot rules run before nginx ever sees the request. The badagent map is still there, and it now catches almost nothing, because the edge caught it first.
• Rate limiting at the edge, so a burst does not consume the VPS’s own bandwidth deciding to reject it.

None of this replaces CrowdSec, which still watches SSH and still applies the community blocklist. It replaces the certificate cron, and it puts the box behind an address nobody can scan.

I am aware of the trade. Cloudflare terminates TLS, which means it can read the traffic, and a product handling personal data has to be comfortable with that in its data processing agreement. This one is, and the agency’s other properties already sit behind it. That is a decision each product makes once, deliberately, and the alternative of running my own edge is not a real option for one person.

What is still not fixed

The Cloudflare migration is done for the public hostnames. SSH is still exposed on 22 to the world, protected by key-only auth and CrowdSec, which is adequate and not ideal; moving it behind a VPN or Cloudflare Tunnel is the next step.

MinIO’s admin console is now unreachable from outside, which also means it is unreachable by me. I use mc over an SSH tunnel when I need it, which is rarely. That is a feature.

And the fourteen-day certificate monitor now watches a Cloudflare edge certificate that Cloudflare renews automatically. It will almost certainly never fire. I am keeping it anyway, because “almost certainly” is exactly what I would have said about the cron.

The lesson

The dangerous part of the cron was not that it was broken. Broken things get fixed. It reported success, every night, with four independent reasons it could not possibly have succeeded, and I trusted the word “completed” for months because nothing else was watching.

Everything that held, held because it did not depend on me checking: CrowdSec banning IPs it had never met, nginx dropping connections without a reply. Everything that failed, failed because it did. The ports stayed open because a TODO is a promise to a future self who never came. The cert expired because a log said it had not.

Security for a solo-run product is mostly the discipline of not being in the loop.