Docker Compose in Production: Log Rotation, Healthchecks, and Resource Limits

1. Preventing Host Disk Exhaustion with JSON-File Log Rotation

Docker's default json-file logging driver writes every line of stdout/stderr from every container to an uncapped file on the host's disk. On a service that logs at even a moderate rate — request logs, ORM query logs, a chatty health-check endpoint hit every few seconds — that file grows without bound. It is one of the most common causes of a production VPS filling its disk unexpectedly, and unlike a slow memory leak, it can go from fine to 100% full within hours once a service starts erroring in a tight retry loop.

The fix is to cap log size and rotation at the Docker daemon level, not just per-service. Setting max-size and max-file in the daemon's default logging configuration applies the cap to every container that does not explicitly override it, which means a new service added to the stack later inherits safe defaults automatically instead of silently reintroducing the problem.

Per-service overrides in docker-compose.yml still matter for services with genuinely different log volume expectations — a database container's error log rarely needs more than a few megabytes of retention, while an application server under active debugging might warrant more. Set logging options explicitly per service rather than relying purely on daemon defaults when volumes differ meaningfully across your stack.

If you need to keep logs longer than local rotation allows — for debugging an incident after the fact, or for compliance — ship them off-box with a log driver like the local driver combined with a periodic collector, or a proper log aggregator (Loki, Vector) rather than raising max-file indefinitely. Retaining logs on the same disk you're trying to protect just delays the same disk-exhaustion problem rather than solving it.

2. Deep Application Health Probes

A container that is running is not the same as a container that is healthy. Docker's default healthcheck-less behavior only tells you a process has not exited, which misses the far more common failure mode of a Node.js or Next.js process that is alive but deadlocked, stuck waiting on a database connection pool, or serving 500s to every request because a downstream dependency died. A meaningful healthcheck exercises the actual request path, not just a TCP connect.

For an application with an API layer, the strongest healthcheck hits a dedicated /api/health endpoint that itself verifies the database connection and any critical downstream dependency (Redis, an external API) with a short timeout, rather than returning 200 unconditionally. This turns 'the process is up' into 'the process can actually serve traffic', which is the property your deploy pipeline and orchestrator actually care about.

Tune interval, timeout, retries, and start_period deliberately rather than accepting defaults. start_period matters enormously for Next.js and similar frameworks that can take several seconds to finish their startup phase (Prisma client generation checks, cache warming) — without a generous start_period, Docker marks the container unhealthy and your deploy script kills it before it ever got a fair chance to become ready, creating a self-inflicted crash loop.

3. Enforcing Memory Ceilings and CPU Limits

Without explicit resource limits, a single container with a memory leak or a runaway query can consume all available RAM on the host, triggering the Linux OOM killer, which then kills processes somewhat arbitrarily — often taking down PostgreSQL or another unrelated container rather than the actual offender. Setting a hard mem_limit on every service turns an unpredictable, host-wide incident into a contained, single-container restart that Compose's restart policy handles automatically.

Sizing these limits correctly requires knowing your application's real memory footprint under load, not guessing round numbers. Start by observing actual usage with docker stats over a representative traffic period, then set the limit at roughly 1.5-2x observed peak — tight enough to catch genuine leaks, loose enough to survive normal traffic spikes and garbage-collector sawtooth patterns without spurious kills.

CPU limits (cpus in Compose's deploy.resources or the legacy cpu_quota) matter less for isolation and more for fairness on a shared host running multiple services — a batch job or a background worker should not be able to starve your API of CPU time during a traffic spike. Reserve CPU limits for genuinely contended hosts; on a dedicated single-app VPS they add complexity without much benefit.

Watch for the difference between mem_limit (a hard ceiling enforced by the kernel cgroup, past which the OOM killer intervenes) and mem_reservation (a soft target used only when the host is under memory pressure). Setting only a hard limit without a reservation means Docker won't proactively prioritize which containers to squeeze first during contention, so pairing both gives the scheduler enough information to degrade gracefully rather than reactively killing whichever container happened to cross the line first.

4. Graceful Container Shutdown Signals

Docker sends SIGTERM to a container's main process on stop, waits for a grace period (10 seconds by default, configurable via stop_grace_period), and then sends SIGKILL if the process hasn't exited. If your application doesn't handle SIGTERM, every deploy or restart is effectively a hard kill of in-flight requests — a request that was 200ms from completing gets cut off mid-response, and any in-memory state or open database transaction is lost ungracefully.

Node.js processes need an explicit signal handler to drain in-flight connections before exiting: stop accepting new connections immediately, let existing ones finish within a bounded window, close the database pool cleanly, then exit. Without this, your zero-downtime deploy pipeline is not actually zero-downtime — it just moves the disruption from 'visible deploy' to 'silently dropped requests during every restart'.

There is a second, less obvious trap: if your image's entrypoint is a shell script (a common pattern for running migrations before starting the app), the shell process — not your Node.js process — receives the SIGTERM, and it typically does not forward it to its child. Use exec in shell entrypoints, or Docker's own PID 1 signal handling via init: true in Compose, so the signal actually reaches the process that knows how to handle it.

5. Restart Policies and Dependency Ordering

Compose's restart: unless-stopped is the right default for almost every production service — it survives host reboots and recovers from crashes automatically, while still respecting a deliberate docker compose stop. Avoid restart: always for services you might need to intentionally stop for maintenance, since it will fight you by restarting even after an explicit stop in some Docker versions' edge cases; unless-stopped avoids that ambiguity entirely.

depends_on with a plain service name only controls startup order, not readiness — your app container can start before Postgres is actually accepting connections, leading to a burst of connection errors on every fresh deploy. Use the condition: service_healthy form together with a healthcheck on the dependency, which makes Compose actually wait for the database to report healthy before starting the dependent service, eliminating an entire class of cold-start races.

6. Summary: A Hardened Compose Stack

Running Docker Compose reliably in production is less about any single setting and more about closing every silent failure mode that only shows up under real traffic: unbounded logs that fill the disk weeks after launch, healthchecks that lie about readiness, memory leaks that take down unrelated containers, and shutdown signals that get swallowed by a shell entrypoint. Each of these is invisible in local development and only bites once the service has been running under real load for days or weeks.

Treat the four pillars — log rotation, deep healthchecks, resource limits, and graceful shutdown — as non-negotiable defaults you apply to every service definition from day one, not retrofits you add after an incident. Concretely: cap json-file logs at the daemon level, write healthchecks that exercise a real dependency check, size mem_limit from observed docker stats data rather than guesswork, and make sure your application's entrypoint actually forwards SIGTERM to a handler that drains connections.

The payoff compounds once you layer automated deployments (covered next in this series) on top of a hardened Compose stack: a deploy pipeline that pulls a new image and swaps containers is only as safe as the healthchecks and shutdown behavior underneath it. Get this foundation right first, and zero-downtime deploys become a straightforward orchestration problem rather than a fight against containers that lie about their own state.

Keep a written record of the resource limits and healthcheck thresholds you settle on, along with the docker stats observations that justified them — six months later, when you're adding a new service or investigating a restart loop, that context is what turns a guessing exercise back into an informed adjustment.