I was grepping JSON over SSH while production was down

The message came in around lunchtime: “Is the site down for you too?” It was. I opened a terminal, SSHed into the server, ran docker ps, and started the ritual: docker logs api --tail 200, squint at a wall of single-line JSON, grep for error, grep for 500, scroll, guess, repeat for nginx, repeat for db.

Fourteen minutes later I found it. Fourteen minutes during which the only person who knew production was down had learned it from a customer, and had spent the whole time reading logs in the least efficient way a human being can read logs.

This is part 6 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the difference between having logs and being able to see them, and it costs nothing but an hour and a bit of RAM.

What I had, and why it was almost enough

I want to be fair to my past self, because the basics were there. Every service in the compose file had bounded logging:

# docker-compose.yml, trimmed and anonymised
services:
  api:
    logging:
      driver: json-file
      options: { 'max-size': '10m', 'max-file': '3' }
    labels:
      - 'logging'
    healthcheck:
      test: ['CMD', 'bun', '-e',
        "fetch('http://localhost:8000/health').then((r) => process.exit(r.ok ? 0 : 1)).catch(() => process.exit(1))"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Bounded log files are the one habit that meant a chatty container could never fill the disk. That was the disk’s job, apparently, and it found another way. The api and db services also had real healthchecks, which part 5 explains in detail, including the surprise that a failing healthcheck in plain Compose never restarts anything.

So Docker knew the API was unhealthy. It had known for several minutes before the customer’s message. It told nobody, because nothing was listening. And every log line I needed was already on disk, in a format designed for machines, being read by a person with grep.

That is the whole problem in one sentence: the information existed and there was no path from it to me.

The constraint

Same VPS, no new monthly bill. This is a solo-built product for a small agency, and “add Datadog” is not an answer I can give them. The stack below runs in two extra containers, uses about 150 MB of RAM between them, and costs zero.

Layer 1: Uptime Kuma on the healthchecks I already had

Uptime Kuma is a self-hosted status page and monitor. It pings things and tells you when they stop answering. It also has a Docker monitor type that reads container health directly, which means the healthchecks I already wrote become alerts without any duplication.

# docker-compose.yml, added service
  uptime-kuma:
    image: louislam/uptime-kuma:1
    volumes:
      - ./uptime-kuma/data:/app/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      my_network:
    restart: unless-stopped
    logging:
      driver: json-file
      options: { 'max-size': '10m', 'max-file': '3' }

Two things about that block. It mounts the Docker socket read-only, which is what lets it see container health. And it does not publish a port: it is reached through nginx on an internal hostname, behind the same auth as everything else, because a public status dashboard for a private product is a reconnaissance gift.

Then the monitors, set up in the UI, five minutes total:

Monitor	Type	Target	Interval
API health	HTTP	http://api:8000/health	60s
API container	Docker	api (healthy/unhealthy)	60s
Database container	Docker	db	60s
Dashboard	HTTP keyword	https://app.example.com, expects the app title	120s
TLS certificate	HTTP	https://api.example.com, expiry warning at 14 days	24h

The last row exists because of part 8, where a certificate silently failed to renew and I found out from a browser warning. Fourteen days of notice would have made that a non-event.

The HTTP monitor on /health and the Docker monitor on the same container look redundant. They are not. The HTTP check tells me the API answers from the network; the Docker check tells me what Docker thinks. When they disagree, that disagreement is itself the diagnosis: healthy container plus failing HTTP means nginx or networking, not the application.

Layer 2: an alert that reaches a phone

A dashboard nobody is looking at is a log file with a nicer font. The monitors need a notification channel, and Uptime Kuma supports about ninety of them. I use two: email for the record, and a push notification for the ones that matter.

The rule I settled on, which took a couple of false alarms to get right:

• Page me (push, immediately): API health down, database container unhealthy, certificate under 14 days.
• Notify me (email, batched): dashboard keyword missing, any monitor recovering.
• Nothing: a single failed check. Every monitor has retries set to 2 before it counts as down, because a 60-second blip at 3am is not worth waking up for and a real outage will still be there at the third check.

The lunchtime incident, replayed through this: the API healthcheck fails three times over 90 seconds, Docker marks the container unhealthy, Uptime Kuma sees both signals, and my phone buzzes about two minutes after the first failure. Instead of a customer telling me fourteen minutes in.

Layer 3: Dozzle, so the logs are readable while it is happening

Monitoring tells you that something is wrong. Logs tell you what. And my logs were in the worst possible place for a human: JSON lines, one file per container, on a server I had to SSH into.

Dozzle is a single container that tails Docker logs into a browser tab. No agent, no database, no indexing, no configuration. It reads the same socket Uptime Kuma does and renders every container’s output live, with search, filtering, and the ability to look at several containers side by side.

  dozzle:
    image: amir20/dozzle:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      DOZZLE_FILTER: 'label=logging'
    networks:
      my_network:
    restart: unless-stopped

That DOZZLE_FILTER: 'label=logging' line is why the compose file already had labels: ['logging'] on every service: it was added months earlier, for exactly this tool, and then the tool was never installed. Past me had the right idea and stopped one step short.

Dozzle also sits behind nginx with auth, on an internal hostname. The difference in an incident is hard to overstate. docker logs api --tail 200 | grep -i error becomes: open a tab, type error, watch it stream, click nginx to see whether the request even arrived. The fourteen minutes become two.

What I deliberately did not install

Loki plus Grafana was the obvious “proper” answer, and I wrote the compose block for it before deleting it. On one VPS with 8 GB of RAM, Loki, Promtail and Grafana add three containers, a query language to learn, retention to configure, and roughly 600 MB of memory for the privilege of searching logs I can already search in Dozzle. When there are more than a handful of services, or when I need to correlate across weeks, that trade changes. Today it does not.

Same reasoning for Prometheus and node exporters. The healthchecks plus Uptime Kuma answer the question I actually have, which is “is it up”, and disk usage is a single Uptime Kuma push monitor fed by a cron line:

# crontab, every 5 minutes: report disk usage to a push monitor
*/5 * * * * curl -fsS "http://localhost:3001/api/push/<token>?status=up&msg=$(df --output=pcent / | tail -1 | tr -d ' %')" > /dev/null

Uptime Kuma alerts if the push stops arriving, and I can read the last message to see the percentage. It is crude. It also would have caught the disk filling up a week before it took production down, which is more than the crude solution’s critics can say.

What is still not fixed

There is no error tracking. Dozzle shows me a stack trace if I am looking at the right container at the right moment; it does not group errors, count them, or tell me that a new one appeared after a deploy. That is a different tool and a different problem, and it is part 7.

Both new containers depend on the same machine they monitor. If the VPS itself goes down, Uptime Kuma goes down with it and sends nothing. The fix is a second, external monitor pointed at the public endpoints, and the free tier of any hosted uptime service is enough for that. I have not done it yet. It is on the list, and I am aware that “the monitor is on the thing it monitors” is the kind of sentence that ends up in a post-mortem.

The lesson

I had been treating observability as something you buy, and therefore something a small product could not afford. What I actually lacked was two containers and an hour. The healthchecks, the log limits, even the logging labels were all already in the compose file. The stack was 90% built and 0% visible.

The lunchtime incident would have been a two-minute push notification and a Dozzle tab. Instead it was a customer, an SSH session, and grep. The difference was never money.