The docker build that filled the disk and took down production

Production went down, and I had not shipped any code. No traffic spike, no bad migration, nothing in the application logs. The API just stopped answering, and so did every container on the box.

The cause was mundane and entirely self-inflicted: the disk was full. And the thing that filled it was my own deploy process, building Docker images on the exact machine that serves customer traffic.

This is part 1 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the first weak spot that bit me: building where you serve.

What “deploy” meant that day

There is no pipeline. A deploy is me, over SSH, running make deploy. Stripped down, the target does this:

# Makefile, roughly
docker compose build api                      # build ON the prod box
docker compose build --no-cache dashboard     # and this one with no cache
docker compose up -d --force-recreate --no-build
docker image prune -f                          # (added later, see below)

Read that with fresh eyes and the problem is right there. Every deploy builds images on the production host. The build pulls base layers, compiles the app, and writes new image layers, all onto the same disk that Postgres, MinIO uploads, and the container logs are living on. The build is a heavy, bursty batch job sharing a disk with a latency-sensitive service that must never stop.

But the resource contention was not what took prod down that day. Something quieter did.

Why the disk filled, and what cleared it

Here is the mechanism I had not thought about. docker compose build does not update an image in place. Each build produces a brand new image and moves the service’s tag to it. The previous image loses its tag and becomes a dangling image, still on disk, still taking space, tagged <none> and referenced by nothing.

Deploy after deploy, those orphaned <none> images piled up. The dashboard, built with --no-cache, wrote a fresh full set every single time. Nothing was cleaning them, so they grew until the disk hit 100 percent and the kernel started refusing writes. Postgres cannot write, so the API errors, so everything falls over.

The thing that saved me was exactly the command people underestimate:

docker image prune -f    # removes dangling <none> images

docker image prune -f removes dangling images, and dangling <none> images were precisely what had filled the disk. It reclaimed the space and brought production back in minutes. That is the whole reason that line is in my deploy today: I added the docker image prune -f at the end of make deploy after this incident, so the images the build orphans get cleared on every deploy instead of piling up.

Diagnosing it: two commands

When a Linux box misbehaves in a way the app logs cannot explain, check the disk before anything else:

df -h                    # is the filesystem full? almost always the answer

If Docker is the culprit, this is the command that shows you where the space went:

docker system df         # summary: images, containers, volumes, build cache
docker system df -v      # per-item breakdown, marks reclaimable space

The -v view is the one that matters. It lists images individually and splits “active” from “reclaimable”. The reclaimable images, the dangling <none> ones, were the space my deploys had been leaking. Seeing that stack of untagged images is what told me the deploy, not the data, had filled the disk.

One dangerous reflex to avoid

If the build cache has also grown, you can reclaim more, but be precise, because the aggressive commands can delete more than you mean to:

docker image prune -af      # all unused images, tagged or not
docker builder prune -f      # the build cache

What I did not run is docker system prune --volumes. That flag also removes volumes. On this setup the data lives in bind mounts rather than named volumes, so it would not have been caught, but the reflex is dangerous: on a box that does use named volumes, that one flag deletes your database. Never reach for --volumes on a production machine you have not audited. (That the data was in bind mounts with no backup is its own problem, and its own article later in this series.)

The one good habit already in place

Credit where it is due: the log side of this was already handled. Every service in the compose file caps its logs:

logging:
  driver: json-file
  options: { 'max-size': '10m', 'max-file': '3' }

Without that, Docker’s default json-file logs grow unbounded and are a classic way to fill a disk on their own. So the logs were not the problem. The orphaned images were. It is worth saying because “the disk is full” has several possible causes, and capping logs removes one of them cleanly. If your compose file does not do this, add it now.

The real fix: stop building where you serve

Auto-pruning after every deploy stops the disk from filling again, and it is the right immediate fix. But it patches one symptom of a deeper problem: I am still building images on the machine that serves customers. That has costs the prune does not touch. The build competes with live traffic for CPU, RAM, and disk while it runs. A deploy that fails before it reaches the prune line still leaves its garbage behind. And --force-recreate still means a few seconds of downtime on every deploy.

The clean answer is to take the build off the box entirely: build the image in CI, push it to a registry, and have the server only ever pull a finished image. The prod machine goes back to doing one job, running containers, and never sees a build again. That is a bigger change (it comes with banning direct pushes to main and an actual deploy pipeline), so it gets its own article later in this series.

The takeaway

The outage was my deploy process leaving orphaned <none> images on the machine it deployed to, one per build, until the disk was full. docker image prune -f clears exactly those, and wiring it into every deploy is the five-minute fix everyone building with Compose should have. But the deeper lesson is that a production box should not be building images at all.

Check docker system df -v right now. If you see a stack of dangling <none> images, your deploys are leaking disk, and a prune (ideally automatic) is one command away. The clean version of the fix, build in CI and pull on the server, is coming up in this series. The series overview tracks all of it.