Three deploy mistakes that broke our production, and the rule each one left

The GHZ Creative backend handles accounts, licenses, Stripe webhooks and the AI calls behind GHZ Voice. In June 2026 it ran on a single VPS with Docker Compose, deployed by copying files over SSH. Three mistakes in that month broke it. None of them were in the code.

1. One rsync replaced the production secrets

On 19 June, a deploy synced the API folder to the server with rsync. The command did not exclude .env. The local copy of that file was out of date, so it overwrote the production one. Then docker compose up recreated the containers with the old values.

Everything failed at once: the database rejected the API’s password, and Stripe, Google sign-in, email and the voice providers all lost their keys.

Recovering the database had a trap of its own. Postgres stores the user password inside its data volume when it first starts. Changing POSTGRES_PASSWORD and recreating the container changes nothing. We had to connect through the local socket, which does not ask for a password, and reset it by hand:

docker compose exec db psql -U ghz
ALTER USER ghz WITH PASSWORD '<the value in .env>';

2. A variable in .env that never reached the app

The next day we added ADMIN_EMAILS to the server’s .env to enable the admin dashboard. It did not work. The account stayed a normal user after every restart.

Our docker-compose.yml had no env_file. Each variable was mapped one by one in the environment: block. A new line in .env was invisible to the container until the compose file mapped it too:

services:
  api:
    environment:
      ADMIN_EMAILS: ${ADMIN_EMAILS:-}

3. Another agent deployed from the wrong folder

We build with AI coding agents, and we often run several at once. On 14 June one of them deployed the backend from an old checkout that lived inside the website repository. That copy predated a week of security work.

The deploy silently reverted several fixes: the daily usage quota on the voice endpoints, the Redis password, and the CORS policy, which went back to accepting any origin. The service stayed up, so nothing looked wrong.

We caught it with the checks we had written for those fixes: Redis answered without a password, and a request from an unknown origin was accepted. The correct checkout was intact, and a redeploy restored everything. One change from the bad deploy was worth keeping: the Tauri app origins it had added to CORS, which GHZ Code needs to call the API from its webview.

What changed after June

Each of these was a process failure, so each fix was a process. Since October every backend we run lives on one server, deployed by one script with an automatic rollback, and backed up every hour. The details are in one server for every backend.