The Friday push that taught me to ban commits on main

A git push at 6pm on a Friday deployed straight to production, untested, and took the API down. Nothing in the pipeline could have stopped it, because there was no pipeline. Here is the one I built: protected main, CI that runs the tests, images built off the box.

Aug 27, 2026

It was a Friday, a little after six. A small fix, one file, the kind of change you make with one eye on the door. I committed on main, pushed, SSHed into the server, ran git pull and make deploy SERVICE=api, and went to close my laptop.

The API did not come back up. The build had succeeded, the container had started, and the process died on boot because the one file I touched imported a module that existed on my machine and not in the image. Production was down for the twenty minutes it took me to notice, revert, and redeploy. Nobody was hurt. Everything about it was avoidable.

This is part 4 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the gap between “I have tests” and “nothing untested can reach production”, and about moving the Docker build off the machine that serves traffic.

What the deploy actually was

The honest description of the deployment process, as documented in the README, was this:

ssh <user>@<server>
cd app && git pull
make deploy SERVICE=api

And make deploy did this (trimmed and anonymised):

COMPOSE_PROD := docker compose --profile prod
DOCKER_PRUNE_DANGLING := docker image prune -f

deploy:
ifdef SERVICE
	$(COMPOSE_PROD) build $(SERVICE)
	$(COMPOSE_PROD) up -d --force-recreate --no-build $(SERVICE)
	$(DOCKER_PRUNE_DANGLING)
else
	$(COMPOSE_PROD) build api
	$(COMPOSE_PROD) build --no-cache dashboard
	$(COMPOSE_PROD) up -d --force-recreate --no-build api dashboard
	$(DOCKER_PRUNE_DANGLING)
endif

Read it as a list of trust assumptions. It trusts that whatever is on main is correct, because nothing checks. It trusts the person at the keyboard to have run the tests, because nothing runs them. It trusts the production box to have the CPU and disk to build an image, which it did not always have. And it trusts that the image built on the server matches the code I tested locally, which on that Friday it did not, because I had never built it locally at all.

None of this was stupid at launch. One developer, one server, a product that needed to ship: git pull && make deploy is the shortest path from a fix to a running fix, and I would choose it again for week one. It stops being defensible the first time a customer sees the consequences.

Three gates, in the order they matter

The fix has three parts, and the order is deliberate. Each one is useful alone, and each one makes the next one possible.

Gate 1: `main` refuses direct pushes

This is a GitHub setting, not code, and it is the cheapest change in the whole series. On the repository: Settings, Branches, add a protection rule for main:

Require a pull request before merging
Require status checks to pass before merging, and pick the check named test (it will not exist until gate 2 is in place, so come back for this one)
Do not allow bypassing the above settings, including for administrators, because this is the line that actually protects you from yourself on a Friday

The first time you try to push to main and get rejected is mildly annoying. That annoyance is the entire feature.

Gate 2: CI that runs the tests you already have

The repository had exactly one workflow, and it was not about code quality:

# .github/workflows/precision-check.yml
on:
  repository_dispatch:
    types: [jira-ticket-moved-to-progress]

It fires when a Jira ticket moves to “in progress” and asks an LLM whether the ticket is well specified. Useful, but it means the only automation in the repo was reviewing tickets, not code. Nothing ran on push. Nothing ran on pull request. Part 3 covers the discovery that real unit tests existed and had never executed once; this is the workflow that finally runs them, and the one gate 1 waits for:

# .github/workflows/test.yml
name: test
on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v2
      - run: bun install --frozen-lockfile
      - run: bun run lint
      - run: bun test

Five steps. It runs the linter and the Bun test runner on every pull request, and on main itself as a safety net. The job is called test, which is the name the branch protection rule requires. Once this is green once, go back to gate 1 and tick the box, and from then on a red test blocks the merge button.

Two details worth getting right. --frozen-lockfile fails the build if the lockfile is out of date, which is exactly the class of “works on my machine” bug that took the API down. And running on push to main as well as on pull requests catches anything that reaches main through a path you did not anticipate, like a merge done from the CLI with a stale branch.

Gate 3: build the image in CI, pull it on the server

This is the part that changes the shape of the deploy, and it fixes two of the pillar’s weak spots at once: the build on the prod box, and the SSH-and-hope deploy.

The idea: when main changes, GitHub Actions builds the Docker images, pushes them to a registry, and then tells the server to pull and restart. The server never builds anything again. Its only jobs are to pull an image that already passed the tests and to run it.

# .github/workflows/deploy.yml
name: deploy
on:
  push:
    branches: [main]

concurrency:
  group: deploy-prod
  cancel-in-progress: false

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    strategy:
      matrix:
        service: [api, dashboard]
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          file: apps/${{ matrix.service }}/Dockerfile
          target: prod
          push: true
          tags: |
            ghcr.io/${{ github.repository }}/${{ matrix.service }}:${{ github.sha }}
            ghcr.io/${{ github.repository }}/${{ matrix.service }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

  release:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.VPS_HOST }}
          username: ${{ secrets.VPS_USER }}
          key: ${{ secrets.VPS_SSH_KEY }}
          script: |
            cd app
            echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
            IMAGE_TAG=${{ github.sha }} docker compose --profile prod pull api dashboard
            IMAGE_TAG=${{ github.sha }} docker compose --profile prod up -d --no-build api dashboard
            docker image prune -f

The concurrency block matters more than it looks: two merges in quick succession must not race each other onto the server, and cancel-in-progress: false means the second one waits rather than killing the first mid-deploy.

On the compose side, the two services stop building and start pulling:

# docker-compose.yml, trimmed and anonymised
services:
  api:
    image: ghcr.io/<org>/<repo>/api:${IMAGE_TAG:-latest}
    # build: was here. It is gone.
    restart: unless-stopped
    ...
  dashboard:
    image: ghcr.io/<org>/<repo>/dashboard:${IMAGE_TAG:-latest}
    restart: unless-stopped
    ...

IMAGE_TAG is the commit SHA, so every running container can be traced back to exactly one commit that passed CI. Rolling back is IMAGE_TAG=<previous sha> docker compose up -d, and it takes seconds, because the image already exists in the registry. Compare that to the Friday procedure, which was “revert the commit and rebuild on the server while it is down”.

The Dockerfile already had a target: ${ENVIRONMENT} multi-stage setup with dev and prod stages, so pinning target: prod in CI required no changes to the images themselves.

What this did to the Makefile

make deploy still exists, because muscle memory is real, but it no longer builds:

deploy:
	@echo "Deploys happen from CI on merge to main."
	@echo "Emergency manual pull: make pull TAG=<sha>"

pull:
	IMAGE_TAG=$(TAG) $(COMPOSE_PROD) pull api dashboard
	IMAGE_TAG=$(TAG) $(COMPOSE_PROD) up -d --no-build api dashboard

The docker image prune -f that saved production when the disk filled is still in the deploy script, out of habit and as belt-and-braces. But it should have almost nothing to prune now, because the server no longer builds and therefore no longer orphans images. The disk problem was never really about pruning. It was about building in the wrong place.

What is still not fixed

Honest list.

The storage (MinIO) and nginx services are not in the matrix. MinIO has a small custom Dockerfile and nginx is stock with a mounted config, and both change rarely enough that I left them on the old path for now. That is a decision I will revisit the first time one of them needs a hot fix.

There is no smoke test after the deploy. The release job reports success when docker compose up returns, not when /health answers 200. Part 6 puts a monitor on that endpoint, and the right next step is to have the deploy job curl it and fail loudly if it does not come back.

And require pull request with one developer means I review my own code. That is not nothing: reading a diff in a browser five minutes after writing it catches a surprising amount, and it is where I would have seen the import that did not exist in the image. But it is not a second pair of eyes, and I am not going to pretend it is.

The lesson

The Friday incident was not a testing failure. The tests existed. It was not even a deployment failure in the usual sense: the deploy did exactly what I asked. It was a failure of the path. There was a route from my keyboard to production that skipped every check, and on a Friday evening I took it, because it was there.

The three gates do not make me a more careful engineer. They remove the route. main will not take a direct push, a red test will not merge, and the server will not run an image that CI did not build. Each is a small piece of configuration, and together they mean the worst thing that can happen at six on a Friday is a rejected push.