When Staging and Production Share the Same Server: A War Story

A client asked me to set up a staging environment for their platform. Whitelist my IP, SSH in, assess the situation. Simple enough.

The server was running 8+ Docker Swarm services on a single machine: databases, APIs, frontends, an AI microservice, nginx, certbot. All sharing the same resources. Zero isolation. No staging environment whatsoever. Every change to this server was a change to production.

The kind of infrastructure that works fine until someone needs to touch it.

The Landscape

The stack was not trivial:

MySQL 8, main database for 62 domain modules
Redis 7, caching layer for the professional catalog
Express API (Node 22), backend with auth, Stripe payments, RBAC
FastAPI (Python 3.12), AI agent service calling Google Gemini ~10 times per message
Two React SPAs, patient app and admin backoffice
Nginx, reverse proxy for all traffic plus prerender.io for SEO
Certbot for TLS certificates
A landing page, static but still running as a Swarm service

No cgroups. No memory limits. No CPU constraints. Docker Swarm with default settings on a single node.

The server also had 8 zombie containers sitting around eating memory. Standard cleanup:

docker container prune -f
docker image prune -a

3.3GB of RAM freed. About 18GB of disk recovered. The server could breathe again.

The Conflict Nobody Documents

The plan: install Coolify (a self-hosted PaaS) to manage a proper staging environment on the same server. Coolify's docs say nothing about Docker Swarm coexistence. CapRover's docs say nothing about Coolify. No warning anywhere that these two orchestrators will fight each other on the same host.

Three things broke at once.

1. Network pool collision

Coolify creates a bridge network with a default address pool. That pool overlapped with existing Swarm networks:

Swarm bridge:          172.16.0.0/24
Swarm captain-overlay: 10.0.1.0/24
Swarm ingress:         10.0.0.0/24
Coolify bridge:        10.0.2.0/24  ← overlap

Docker's response was immediate:

invalid pool request: Pool overlaps with other one on this address space

MySQL dropped to 0/1. Nginx dropped to 0/1. The API dropped to 0/1.

2. Daemon restart cascade

Coolify's installer modifies /etc/docker/daemon.json and restarts the Docker daemon. On a standalone host, that is a brief interruption. On a Swarm node, systemctl restart docker triggers a full recreation of every service container. Everything went down at once.

The behavior is documented in Docker's Swarm docs. Coolify's installer just does not check whether the host is a Swarm node before restarting.

3. The orphaned images problem

Something Docker does not make obvious: docker image prune -a removes images for Swarm services that currently have 0 replicas. If those images were never pushed to a registry, they are gone for good after pruning.

The AI agent backend and the landing page had been temporarily scaled to 0 replicas before my session. Their images were built locally during CapRover deployments, never pushed anywhere. After the daemon restart, Swarm tried to bring them back but had no image to pull.

Three services stuck permanently at 0/1:

ia-agent-backend (production): AI chat offline
ia-agent-backend-homolog: staging AI offline
landing-page: marketing site gone

This is not a pruning mistake. It is a pipeline gap. If your deployment process builds images locally without pushing to a registry, those images are one cleanup command away from disappearing. Docker will not warn you. Swarm will not warn you.

The Recovery

Step 1: Remove the conflicting orchestrator.

docker stop $(docker ps -q --filter "name=coolify")
docker rm $(docker ps -aq --filter "name=coolify")
docker network rm coolify

Step 2: Realign the address pool.

{
  "default-address-pools": [
    { "base": "172.20.0.0/12", "size": 24 }
  ]
}

Two daemon restarts (each causing a brief Swarm disruption) to stabilize the network. The bridge was recreated at 172.16.0.0/24. Services came back online.

Step 3: Triage the unrecoverable services.

The staging API had also lost its image. Fastest recovery: point it at the production image temporarily.

docker service update --image img-captain-api:38 api-homolog

Staging was now running production code. Technically operational, but every test against staging was actually testing production behavior. Not great. The three missing services needed a full rebuild from source.

Step 4: Document everything before the next session.

Post-incident state, confirmed via docker service ls:

8 services running at 1/1 (production stable)
3 services at 0/1 (pending image rebuild)
1 service running the wrong image (staging API on prod image)

What the Assessment Revealed

Before the Coolify conflict, I had been running a full diagnostic sweep. The problems went well beyond resource contention.

Security gaps:

AI agent service with zero authentication. Endpoints completely open, no JWT forwarding, no API key.
No rate limiting on the AI service. Each user message fires ~10 Gemini API calls. An attacker could drain the budget in hours.
CORS wide open on the backend: app.use(cors()).
Root SSH login enabled.

Reliability risks:

AI conversations stored in memory only. Container restart wipes all active sessions.
10 sequential Gemini calls per message with no parallelization. Latency was rough.
time.sleep() in retry logic inside an async FastAPI service. Blocking the event loop on a concurrent server.
13 tests for 62 backend modules.

Infrastructure debt:

No monitoring beyond an unwatched Prometheus endpoint.
Node 18 (EOL April 2025) in production.
Alpine 3.10 (EOL 2019) in Docker images.
Dual CI pipelines (GitHub Actions + Azure DevOps) for the same repos.

Each of these on its own? Acceptable in an early-stage product. All of them together? A system where any intervention carries cascading risk.

The Access Problem

While writing the post-incident document, my SSH connection dropped.

My ISP had rotated my public IP. The server uses IP whitelisting for SSH access, which is a reasonable security measure. But with a dynamic residential IP, access depends on something outside my control.

New IP. New whitelist request. Wait for the client's team. Next business day.

Production was stable. But I could not verify it remotely until the new IP was approved.

This is why tmux attach || tmux new is the first command I run after connecting to any server. If the connection drops, your shells survive. But when the IP itself changes, session persistence will not help. You need network-level access back. A VPN or bastion host would have made this a non-event.

What Should Have Existed Before Any of This

The root cause was not Coolify, not Docker pruning, not my ISP. It was the absence of basic infrastructure practices that would have prevented the whole thing.

1. Staging on a separate machine. Even a €5/month VPS eliminates 100% of the orchestrator collision risk. Staging and production on the same host is not saving money. It is borrowing time.

2. Every image in a registry. If a Docker image only lives on a local disk, it is not an artifact. It is a temp file. GHCR, Docker Hub, a private registry, whatever. It needs to survive a prune.

3. Network topology documentation. A docker network ls with CIDR ranges written down somewhere would have flagged the collision before it happened. You cannot make safe infrastructure changes without a network map.

4. Resource isolation. No service should consume unlimited memory on a shared host. cgroups, systemd limits, Docker resource constraints. The specific mechanism matters less than having any limit at all.

5. A deployment pipeline that does not need SSH. If deploying means logging into a server and running commands, every deploy depends on network access to one machine. CI/CD that pushes to a registry and triggers Swarm updates takes the human out of the critical path.

6. Access resilience. VPN or bastion host. IP whitelisting is a good layer but should not be the only path in, especially with a dynamic IP.

Production is stable. Three services are queued for image rebuilds. The staging API is temporarily wearing production's clothes. My ISP gave me a new IP this morning.

The real work starts next session: move staging to its own server, push every image to a registry, and build the deployment pipeline that should have existed before I ever SSH-ed in.

You do not always finish in one session. Sometimes you stabilize, document, and come back tomorrow.