Health check endpoints

How to implement /health/live, /health/ready and /version, wire them to each deploy target, and choose their timings. For what these endpoints are for and why they are shaped this way, read health checks first. The short version: the depth of a check must be inversely proportional to the blast radius of the reaction it triggers.

The endpoints

A FastAPI router implementing all three:

"""Operational endpoints. No authentication needed, so no business data here."""

import asyncio
import os
import time

from fastapi import APIRouter, Response
from sqlalchemy import text
from sqlalchemy.exc import SQLAlchemyError

from .database import engine

router = APIRouter()

VERSION = {
    "commit": os.environ["GIT_COMMIT"],
    "built": os.environ["BUILD_TIME"],
}

READY_TTL_SECONDS = 10
_ready_lock = asyncio.Lock()
_ready_cache = (0.0, False, "not checked yet")
draining = False


@router.get("/health/live")
async def live() -> dict[str, object]:
    """Report that the process is up. Wired to the liveness check, so it stays free of I/O."""
    return {"status": "UP", "checks": []}


@router.get("/health/ready")
async def ready(response: Response) -> dict[str, object]:
    """Report whether this instance should receive traffic.

    The lock makes the dependency check single-flight: concurrent probes wait for the
    one query in progress rather than each opening their own, which matters most when
    the database is already struggling. Only the exception class is reported, to keep
    connection strings and hostnames out of an unauthenticated response.
    """
    global _ready_cache
    if draining:
        response.status_code = 503
        return {"status": "DOWN", "checks": [{"name": "draining", "status": "DOWN"}]}

    async with _ready_lock:
        checked_at, healthy, detail = _ready_cache
        if time.monotonic() - checked_at > READY_TTL_SECONDS:
            try:
                async with engine.connect() as connection:
                    await connection.execute(text("set local statement_timeout = '2s'"))
                    await connection.execute(text("select 1"))
                healthy, detail = True, "ok"
            except SQLAlchemyError as exc:
                healthy, detail = False, type(exc).__name__
            _ready_cache = (time.monotonic(), healthy, detail)

    response.status_code = 200 if healthy else 503
    return {
        "status": "UP" if healthy else "DOWN",
        "checks": [{"name": "database", "status": "UP" if healthy else "DOWN", "data": {"detail": detail}}],
    }


@router.get("/version")
async def version() -> dict[str, str]:
    """Report which build answers on this hostname."""
    return VERSION

The body follows MicroProfile Health: UP or DOWN, with a checks array carrying the detail. Machines read the status code and not the body, so the shape costs nothing and saves inventing one.

Declare all three with async def. A plain def endpoint is handed to FastAPI’s threadpool, so under saturation the health check queues behind the very traffic that is overwhelming the service, and the probe times out on the one instance still doing its best. An async def that performs no I/O answers regardless.

The same starvation reaches any synchronous server, and most of our Python services are Flask on gunicorn rather than FastAPI. There a request occupies a worker or a thread for its whole duration, so --threads 4 --timeout 300 means four slow requests can park the health check for five minutes, and the container is marked unhealthy for being busy. There is no async def to reach for. Size the pool so a probe cannot be starved by ordinary concurrency, prefer a request timeout in the same order as the probe budget rather than minutes, and keep the endpoints themselves free of anything that can block.

Drop the engine block entirely if the service has no hard dependency and you deploy to Kubernetes, where drain state alone still changes routing. On Coolify, drop the endpoint: nothing there reads it, and what is left is a second spelling of /health/live.

The draining flag is the Kubernetes half of the example. On Coolify nothing acts on it, so drop the global and its branch and let /health/ready report the dependency alone. See shutting down for what to do instead, which is nothing.

Bounding the dependency call

Without timeouts the endpoint hangs instead of reporting DOWN, and a health check that hangs is worse than one that reports failure. Two different timeouts are needed, because a slow database and an unreachable one fail differently:

engine = create_async_engine(
    os.environ["DATABASE_URL"],
    connect_args={"timeout": 2},  # asyncpg; psycopg spells it connect_timeout
)

connect_args bounds reaching the host at all, which is the case that otherwise hangs until the TCP stack gives up. It sits on the engine the whole service shares, so it bounds every request rather than only the probe. That is usually what you want, and it is a change to production behaviour either way, so pick the number for the service and not for the health check. statement_timeout bounds a query that connects but never returns, and because the connection block exits without committing, the setting is discarded with the transaction rather than following the connection back into the pool. It only applies inside a transaction, so it silently does nothing on an engine set to autocommit.

Keep both shorter than the probe timeout in Timings below.

Frameworks that own the routing

The paths are a contract, and a framework with opinions about URLs will not hand them over for free. Next.js is the case we hit: with the Pages Router the handlers have to live under /api/*, and i18n claims everything else. So the handler goes where the framework wants it, and the contract path is rewritten onto it.

next.config.mjs

const operationalRewrites = [
  {source: "/health/live", destination: "/api/health/live"},
  {source: "/version", destination: "/api/version"},
];

Do not reach for locale: false here, natural as it looks on an endpoint that has nothing to do with language. With i18n configured, Next normalises an incoming path to include the locale before matching a rewrite that opts out of locale handling, so /health/live is compared against /nl-NL/health/live and never matches. Left to handle locales automatically, the rewrite answers on both the bare path and a prefixed one.

Check the mapping from outside the process rather than trusting it. A rewrite that fails to match returns the application’s own 404, which is indistinguishable from a healthy server being asked for a path that does not exist, because that is precisely what it is.

Making boot finish before serving

/health/live is only meaningful as a constant if the service cannot answer before it is usable. You do not work that out by querying migration state at runtime, you sequence it so the question cannot arise: the exit code of alembic upgrade head is the answer, and nothing listens until it is zero.

On Coolify, where there is one container per service, put it in the entrypoint:

ENTRYPOINT ["sh", "-c", "alembic upgrade head && exec uvicorn app.main:app --host 0.0.0.0 --port 8000"]

On Kubernetes, use an init container running the same image. Pod containers do not start until init containers exit 0, and the migration runs once for the pod rather than once per replica:

initContainers:
  - name: migrate
    image: registry.example.com/app:1.2.3  # the same image as the app container
    command: ["alembic", "upgrade", "head"]

Running migrations inside the application’s own startup works too, since the server does not serve until startup finishes, but every replica then races the others. Take a pg_advisory_lock first if you go that way; Alembic does not serialise itself.

Two things that follow:

• A migration must be compatible with the version still running. Both deploy targets start the new container while the old one is still serving, so the schema changes under live code. Expand first, contract in a later deploy.
• The startup budget has to cover the migration when it runs in the entrypoint, since nothing answers while it runs. On Coolify that budget is --start-period plus --retries times --interval, not the start period alone; on Kubernetes it is the startupProbe. An init container is outside either budget, because the app container has not started yet.

If you genuinely need to assert the schema version at runtime, compare alembic current against heads inside /health/ready, where a database round-trip already happens and the result is cached. It is redundant when the sequencing above holds.

Coolify

Coolify builds from git on push, so nothing is standing by to pass build arguments. Take the commit from SOURCE_COMMIT, which Coolify injects into the running container, and stamp the build time into the image, since the build is the only thing that knows it and it cannot know the commit:

RUN date -u +%Y-%m-%dT%H:%M:%SZ > .build-time

HEALTHCHECK --interval=10s --timeout=3s --start-period=15s --retries=6 \
  CMD curl -fsS "http://127.0.0.1:$PORT/health/live" || exit 1

Read SOURCE_COMMIT from the environment at runtime rather than baking it in. Coolify offers an Include SOURCE_COMMIT in build setting that would make it a build argument, and leaving it off is correct: the value changes every commit, so passing it into the build invalidates the layer cache every time.

$PORT is set by Coolify from the application’s exposed port, which is the port the check has to use. The image needs curl or wget for an HTTP check.

Address 127.0.0.1 and not localhost. In most images localhost resolves to ::1 before 127.0.0.1, and a server that binds IPv4 only then refuses the connection. curl hides this by trying both; busybox wget, which is what you get on Alpine, tries one and fails. The symptom is a health check reporting can't connect to remote host: Connection refused against a server whose own log says it is ready, and on Coolify that rolls back a deploy that had nothing wrong with it.

Leave the Coolify health check disabled, in Configuration > Healthcheck. That is what makes Coolify read the HEALTHCHECK out of the Dockerfile, parse its interval, timeout, start period and retries, and gate the deploy on it. Enabling the dashboard check does the opposite: Coolify writes its own healthcheck: into the compose file it generates, and that one wins. The two are mutually exclusive, and the dashboard is the one that takes precedence. You can tell which is in force from the deployment log, which says Custom healthcheck found in Dockerfile. when the image’s own check is being used.

--start-period also means something narrower here than it does to Docker. Docker treats it as a grace window in which failures do not count. Coolify sleeps for it, in full, before its first check, and then polls retries times at interval. So the budget for a slow boot is start-period + retries x interval, and every second of the start period is added to every deploy whether the service needs it or not. Keep it near the fastest plausible boot and let retries absorb the rest.

Docker allows one health check, and here it gates Coolify’s rolling update: the replacement container has to pass before the old one stops, so a service without a health check redeploys with a gap. This is why boot work has to finish before the server serves. A container that answers in two seconds and then migrates for ninety is declared healthy at two, and Coolify cuts traffic over to it.

Add one uptime-kuma monitor per public hostname to source/devices/provisioning/ansible/roles/uptime-kuma-config/defaults/main.yml in the infra repo, under the client’s instance. Every hostname that serves users gets one, including the ones nobody thinks about, because an unmonitored hostname is one that goes down silently.

Not the post-deployment command

Coolify has a Post-deployment Command field, and it is the obvious place to put alembic upgrade head. It is the wrong one. Coolify runs it after the deployment is already marked finished, and wraps it in a handler that logs a failure rather than raising it. A migration that fails there leaves a deployment reported green, with the new code serving against the old schema, and nothing in the pipeline notices.

The entrypoint has neither problem: nothing answers until the migration exits zero, so a failure means the container never turns healthy, Coolify rolls back to the old one, and the deploy fails. Keep the post-deployment command for work that genuinely follows a successful deploy and whose failure you are willing to miss, such as seeding reference data.

Kubernetes

Use all three probes and keep the dependency check out of the two that can kill a pod:

startupProbe:
  httpGet: {path: /health/live, port: http}
  periodSeconds: 10
  failureThreshold: 6       # 60s to finish booting before anything acts
livenessProbe:
  httpGet: {path: /health/live, port: http}
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: {path: /health/ready, port: http}
  periodSeconds: 10
  failureThreshold: 3

A restart cannot reach a database, so a livenessProbe that checks one turns a dependency outage into a fleet-wide crash loop, and the restarts then stampede the dependency with fresh connections exactly as it tries to recover. The startupProbe exists so a slow boot is not read as death, and it makes initialDelaySeconds on the liveness probe unnecessary. Its budget and the Coolify one describe the same thing, the application’s worst cold start, so measure that once and spend it on both targets. Spell it differently, though: the startupProbe here tolerates periodSeconds times failureThreshold and costs nothing when boot is fast, where Coolify charges the start period to every deploy. Migrations in an init container fall outside the budget; migrations in the entrypoint do not.

/version needs no probe. A response header on every request is an equally good option and better in one respect: you get the version on the very response you are already debugging, rather than asking again and hoping you reached the same pod.

Shutting down

Removing a pod from the Service endpoints is asynchronous, so a pod that exits the moment it receives SIGTERM drops requests that are still being routed to it. Every rolling deploy then sheds a few errors. The fix is to keep serving for a few seconds after Kubernetes decides to stop the pod:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "10"]
terminationGracePeriodSeconds: 40

preStop runs before SIGTERM, which gives the endpoints controller time to propagate the removal while the pod is still answering normally. The grace period must comfortably exceed the sleep plus the longest request.

Setting the draining flag in the router on shutdown is a refinement on top: it makes readiness fail immediately rather than waiting for the probe to notice, which shortens the window on load balancers that watch readiness directly. The preStop sleep is what does the real work, and it needs no application code.

On Coolify the rolling update provides the overlap, and nothing reads readiness, so leave the flag out. Drop the draining global and its branch along with it, the same way the engine block goes when there is no hard dependency.

Leave the server’s own SIGTERM handling alone

What actually sheds requests cleanly is the server’s graceful shutdown, and an application-level SIGTERM handler replaces it rather than adding to it. Under gunicorn the ordering makes this easy to do by accident: init_signals() runs first and installs Worker.handle_exit, whose whole body is self.alive = False, and load_wsgi() imports the application afterwards. So a signal.signal(signal.SIGTERM, ...) at import time, or inside an application factory, wins, and the worker never learns to stop.

The symptom is a shutdown that takes exactly graceful_timeout and then dies to a force kill. Measured on one of ours, kill -TERM on the gunicorn master took 30.07 seconds with such a handler installed and 1.32 seconds without. Docker stops a container 10 seconds after SIGTERM, so the difference is a SIGKILL through every request in flight, on every deploy.

If you need to run something on shutdown, use the hooks the server offers, such as gunicorn’s worker_exit, rather than installing a handler over its own.

Services with no HTTP server

Workers, queue consumers and scheduled jobs have nothing to probe over HTTP. The rule is unchanged, only the mechanism differs:

• Liveness. Have the worker touch a file each time it completes a loop, and probe it with an exec check that fails when the file is older than a few cycles. This detects a wedged loop, which is the failure a restart actually fixes. Do not check the broker: a broker outage restarted is still a broker outage.
• Readiness. Usually meaningless, since nothing routes traffic to a worker. Omit it.
• Version. Log it once at startup.

A container serving only static files, such as an SPA behind nginx, needs /health/live alone. There is nothing for it to be ready for beyond accepting connections, and /version can be a JSON file written at build time.

Timings

Two numbers matter for anything with an actuator behind it. The period sets how fast a fault is noticed, and period times threshold sets how long a wobble must last before something acts. Pick the reaction time first, then work backwards: ten seconds times three failures gives roughly 30 seconds of tolerance, long enough to ride out a garbage collection pause and short enough that nobody notices the recovery.

Check	Cadence	Timeout	Threshold	Why
/health/live liveness	10s	2s	3	Restarting is violent, so tolerate a wobble
/health/live startup	10s	2s	6	Covers the worst measured cold start
/health/ready readiness	10s	3s	3	Drains a pod, so it may react faster than liveness
/health/ready uptime-kuma	60s	10s	2	Wakes a human; a faster page is a noisier one, not an earlier fix
/version	never polled	n/a	n/a	Read on demand, at a deploy or when something looks wrong
Release verification	once per deploy	n/a	n/a	A one-shot script, run by a person at a boundary

Keep every timeout shorter than its interval, so a slow check cannot still be running when the next one starts.

The cache in /health/ready is what makes these numbers safe to choose freely. Each replica queries the database at most once per TTL however many probes and monitors point at it, so the load is the replica count divided by the TTL rather than every probe multiplied by every replica. The cache lives in the process, so it bounds the rate per replica, not across the fleet.

/version is never polled because polling it tells you nothing: it changes only when you deploy, and that is a moment you already know about. Release verification runs once per deploy and may do real work, such as counting rows through the application’s own connection. Both answer “did the thing I just did work”, not “is the thing still working”, and those questions want different instruments.