The Docker-Compose Delusion: Why your ‘immutable’ stack is a house of cards

Listen, I’ve watched entire engineering departments sleepwalk into production disasters simply because they believed a YAML file made them invincible.

There is a comforting fiction that permeates modern infrastructure. We write a docker-compose.yml file, we specify a few services, we run docker-compose up -d, and we declare our environment “immutable.” We treat that file as a sacred contract. If the server dies, we simply pull the repository on a new machine, run the command, and everything returns exactly as it was.

Except it doesn’t. Your stack is a house of cards, and the wind is blowing. The delusion stems from a fundamental misunderstanding of what a container tag actually is, combined with an industry-wide hallucination about what we tend to call “reproducible builds.”

The Treachery of Tags

Let us look at the anatomy of a typical Compose file. You define a database: postgres:14. You define an application container: mycorp/backend:v2.1.5. You feel safe. You aren’t using the dreaded :latest tag, so you assume you have locked down your dependencies.

What you actually have is a pointer. A container tag is nothing more than a mutable string on a registry server. It is the cloud equivalent of a symlink, and it can be reassigned at any time by anyone with the right credentials. When a maintainer pushes a security patch for postgres:14, they overwrite the tag. The next time you pull it on a fresh server, you are getting a different set of bits. Different glibc, different underlying Alpine or Debian packages, different bugs.

If you think your CI/CD pipeline builds the exact same artifact twice just because you fed it the same Git commit, you are ignoring the network fetches happening inside your Dockerfile. Every apt-get update or npm install pulls whatever happens to be on the mirror at that exact millisecond. It might be that true immutability is mathematically impossible in a networked environment.

So, we pivot. We decide to pin everything to a SHA256 digest. postgres@sha256:3235b.... Now, the registry cannot lie to us. The content-addressable hash guarantees we download the exact same bytes every single time.

But we should pause here and ask ourselves what we are actually doing. We chase reproducible builds as if they are the ultimate technical truth. We pin every layer, we lock every hash, and we sit back, satisfied that our infrastructure cannot change underneath us. Yet, by freezing these bits in time, we are also freezing the rot. We are locking in yesterday’s zero-day vulnerabilities. It might be that the whole concept of an “immutable stack” is less of a technical requirement and more of a psychological defense mechanism. We are terrified of the relentless entropy of software, so we build a glass case around it and call it a feature. Are we actually trying to make our systems robust, or are we just trying to avoid dealing with the reality that software decays the second it is compiled?

Taking Matters Into Your Own Hands: The Cold Storage Method

If we accept that upstream registries are untrustworthy and that tags are lies, the only way to genuinely guarantee the survival of your specific, tested runtime environment is to physically possess the bits. Not as a cache. As a cold, hard tarball.

Below is a production-grade Bash script that inspects your running Docker-Compose environment, resolves the actual SHA256 digests currently executing in memory, and dumps them into compressed tarballs. This ensures that when the registry eventually deletes your image, or a maintainer forces a push over your tag, you own the original artifact.

Prerequisites

• A Linux environment with docker and docker-compose (or docker compose plugin) installed.
• bash version 4.0 or higher.
• jq installed (for parsing Docker inspect output).
• Massive amounts of free disk space (tarballs of uncompressed container layers are notoriously large).
• Root or docker group privileges.

The Paranoia-Driven Image Archiver

#!/usr/bin/env bash
# title           : compose-freeze.sh
# description     : Resolves running compose containers to their SHA256 digests and backs them up.
# author          : Senior Sysadmin
#==============================================================================

set -euo pipefail
IFS=$'\n\t'

# --- Configuration ---
COMPOSE_PROJECT_NAME=${1:-"my_stack"}
BACKUP_DIR="/var/backups/docker_freeze/${COMPOSE_PROJECT_NAME}_$(date +%Y%m%d_%H%M%S)"
LOG_FILE="/var/log/compose-freeze.log"

# --- Functions ---
log_info() {
    echo -e "[$(date +'%Y-%m-%dT%H:%M:%S%z')] [INFO] $*" | tee -a "$LOG_FILE"
}

log_err() {
    echo -e "[$(date +'%Y-%m-%dT%H:%M:%S%z')] [ERROR] $*" >&2 | tee -a "$LOG_FILE"
}

cleanup() {
    local exit_code=$?
    if [ $exit_code -ne 0 ]; then
        log_err "Script failed with exit code $exit_code. Partial backups may exist in $BACKUP_DIR."
    fi
    exit $exit_code
}
trap cleanup EXIT

# --- Execution ---
log_info "Starting image freeze for project: $COMPOSE_PROJECT_NAME"

if ! command -v jq >/dev/null 2>&1; then
    log_err "jq is required but not installed. Aborting."
    exit 1
fi

mkdir -p "$BACKUP_DIR"
log_info "Backup directory created at $BACKUP_DIR"

# Get all running container IDs for the project
CONTAINERS=$(docker ps -q --filter "label=com.docker.compose.project=$COMPOSE_PROJECT_NAME")

if [ -z "$CONTAINERS" ]; then
    log_err "No running containers found for project $COMPOSE_PROJECT_NAME."
    exit 1
fi

for CONTAINER in $CONTAINERS; do
    # Extract the true RepoDigest (the SHA256) and the Compose service name
    SERVICE_NAME=$(docker inspect "$CONTAINER" | jq -r '.[0].Config.Labels["com.docker.compose.service"]')
    IMAGE_DIGEST=$(docker inspect "$CONTAINER" | jq -r '.[0].Image')

    # We query the image itself to get the RepoDigest
    REPO_DIGEST=$(docker inspect "$IMAGE_DIGEST" | jq -r '.[0].RepoDigests[0]')

    if [ "$REPO_DIGEST" == "null" ] || [ -z "$REPO_DIGEST" ]; then
        log_err "Could not resolve RepoDigest for service $SERVICE_NAME (Image ID: $IMAGE_DIGEST). It may be locally built and untagged."
        continue
    fi

    SAFE_NAME="${SERVICE_NAME}_$(echo "$REPO_DIGEST" | awk -F'@sha256:' '{print $2}' | cut -c1-12)"
    TAR_PATH="${BACKUP_DIR}/${SAFE_NAME}.tar.gz"

    log_info "Archiving service '$SERVICE_NAME' ($REPO_DIGEST) to $TAR_PATH"

    # Save the exact image digest to a compressed tar archive
    docker image save "$REPO_DIGEST" | gzip > "$TAR_PATH"

    log_info "Successfully archived $SERVICE_NAME."
done

log_info "Freeze complete. All running images backed up to $BACKUP_DIR."
exit 0

Edge Cases and Hidden Traps

Do not assume this script solves all your problems. Infrastructure is rarely that forgiving.

First, consider the architecture trap. You run this backup on your production AMD64 servers. Two years later, the server dies, and management forces you onto a new ARM64 cloud instance to save a fraction of a cent per compute hour. Your tarballs are now completely useless. Docker will load them, but the binaries within will panic the moment the kernel tries to execute them.

Second, storage exhaustion. docker image save writes the entire uncompressed layer stack before it gets piped to gzip. If you have a monstrous 5GB Java monolith image and only 2GB of free disk space, this pipeline will silently choke on I/O, fill your root partition, and potentially take down the very production services you were trying to protect.

Finally, database consistency. This backs up the *binary environment*, not your data. If you restore an old image version over a newer PostgreSQL data directory, the database engine will refuse to start, citing incompatible data file headers. The container is immutable; the data volume is terrifyingly mutable.

The Restoration: Re-animating the Corpse

A backup that has never been restored is just a wish masquerading as a file. When your registry goes dark and your servers burn, here is how you put the pieces back together.

1. Load the Tarball

Transfer the tarball to the new host and feed it directly into Docker. It will reconstruct the layers and re-associate them with the SHA256 digest.

gunzip -c /var/backups/docker_freeze/my_stack_20231025/db_3235b123abcd.tar.gz | docker load

2. The Tagging Deception

Here is where things get messy. When you run docker load on a digested image, it does not magically recreate your friendly postgres:14 tag. If you just run docker-compose up -d now, Compose will look at your YAML, see postgres:14, realize it doesn’t have it locally, and try to reach out to Docker Hub—which defeats the entire purpose of this offline recovery.

You must manually map the loaded digest back to the tag your Compose file expects. Find the loaded Image ID and force the tag:

# Find the newly loaded image
docker images --digests

# Re-tag it to trick Docker-Compose
docker tag sha256:3235b123abcd... postgres:14

Now, when you spin up your stack, Docker Compose will see the local tag, assume it is satisfied, and start the container without consulting the network. You have successfully bypassed the registry and resurrected your exact binary state.

This is what it takes to actually guarantee state. It isn’t clean, it isn’t “GitOps,” and it certainly isn’t agile. It is heavy, cumbersome, and inherently paranoid.

Wait, the syslog just spat out a kernel panic on the SAN host.