Health checks

A /ping that returns 200 OK proves one thing: the process accepts connections. That is worth having, and it is not enough. It does not tell you whether the service reaches its database, and it does not tell you whether the proxy in front of it still routes the public hostname to your container.

This page describes the operational endpoints we want on every service and which machine acts on each. For the code, the probe configuration and the timings, see health check endpoints.

The rule

A health check is a control signal for one automated actuator, not a status report. Before adding a check, answer: what machine reads this, and what does it do when the check fails? If the answer is “nothing, a human looks at it”, then it is monitoring, and it belongs in uptime-kuma or Grafana rather than in a gate.

From that follows the rule that keeps health checks from causing the outages they were meant to prevent:

The depth of a check must be inversely proportional to the blast radius of the reaction it triggers.

A check that restarts containers or removes them from routing has to be shallow, because a wrong answer takes the service down. A check that only pages a human can be as deep as you like.

The classic way to get this wrong is to let the container health check query the database. When the database blips, every container turns unhealthy at the same moment, the proxy stops routing to all of them, and a slow database becomes a hard outage. The service could have served a 500 with a useful error page. Instead the hostname 404s.

The second thing to get right is the axis a check sits on. A check running inside the container against localhost sees connection pools, migrations and queue depth, and is blind to DNS, TLS, the proxy and routing. A check from outside sees the whole user path and cannot say why it broke. Neither substitutes for the other, and deepening an inside-out check never moves it onto the other axis.

What acts on what

Our two deploy targets hand a failed check to different machinery, and which actuator reads a check is the whole of what makes it safe or dangerous.

On Coolify:

Signal	Read by	Reaction
Process exits	Docker `restart:` policy	The container restarts
Container `HEALTHCHECK`	Coolify	Gates the rolling update, and rolls a bad deploy back
Container `HEALTHCHECK`	Traefik	Routes only to a container whose check passes
HTTP probe on a public hostname	uptime-kuma	Pages the team in Mattermost
Metrics	Prometheus, Grafana	Alerts on a trend over a window, never on a single sample

Plain Docker does not restart a container because its health check failed; process death is what the restart: policy handles. Traefik does read the check: its docker provider routes only to a container Docker reports as healthy, and applies that filter again on every configuration rebuild, so a starting or unhealthy container gets no traffic. The itslanguage servers run caddy-docker-proxy instead, which builds its routes from container labels and offers no option to filter on health at all; there the container health check is a deploy signal and nothing else.

That makes the container health check on a Traefik server the routing signal for the whole service. With one replica, a check that fails retries times in a row takes the hostname down, which is the strongest reason to keep it on the shallow /health/live. A check that is too deep does not merely refuse to let you ship; it takes a healthy service out of routing.

On Kubernetes, every probe has its own actuator, and the restart one is real:

Signal	Read by	Reaction
`startupProbe`	kubelet	Holds off the other probes; kills the pod if boot never finishes
`livenessProbe`	kubelet	Restarts the container
`readinessProbe`	kubelet	Removes the pod from the Service endpoints, so it gets no traffic
Ingress or probe	uptime-kuma	Pages the team in Mattermost

The endpoints

Endpoint	Answers	Depth	Actuator
`/health/live`	Is the process up?	No I/O; drain state on Coolify	Restarts, gates a deploy, or routes
`/health/ready`	Should this instance get traffic?	Drain state, plus hard dependencies	Routing on Kubernetes, and pages a human
`/version`	Which build is answering?	Constant	A human reads it

/health/live answers UP and touches nothing; on Coolify it answers DOWN once the server is draining, which reads a flag in memory and is still no I/O. That is not a placeholder to improve on later, it is the correct and complete implementation. It has the most violent actuators behind it: on Kubernetes a failure restarts the container, on Coolify it blocks a deploy or takes the container out of routing, and none of these reactions repairs a broken dependency. Anything /health/live touches becomes a thing that can restart your fleet, empty your routing, or stop you shipping a fix during an incident. An existing /ping already does the right amount of work; only its path and body change.

For that UP to mean anything, finish boot work before serving. Run migrations and cache warming before the server answers, so “the port answers” is itself the signal that boot completed and no endpoint has to carry it. The guide covers how to sequence that.

/health/ready has two legitimate inputs. The first is whether the instance is draining because it is being shut down, which is worth reporting only where something reads readiness to make routing decisions. The second is the dependencies the service genuinely cannot work without. Be strict about “cannot”: the test is whether the service can still produce a useful response, not whether it can serve every route. A backend that still returns cached data, static assets or a real error page has no hard dependency.

That strictness matters because readiness is fleet-wide. Every replica evaluates the same dependency at the same moment, so one database blip empties the Service endpoints all at once, and neither Kubernetes nor our proxies have a fail-open to catch it. A total outage replaces what would have been a degraded service returning honest errors.

Probing /health/ready from uptime-kuma covers both axes in a single monitor. The request crosses DNS, TLS and the proxy or ingress to get there, and the response asserts what the application itself can reach. One monitor per public hostname is enough: backends point at /health/ready, frontends at /.

Whether a service needs the endpoint at all depends on the target, because the two inputs are not both live everywhere. On Kubernetes it is always warranted: the readiness probe gates Service endpoint membership, so both draining and dependency state change routing through it. On Coolify neither does: Traefik routes on the container health check, which reads /health/live, and that leaves uptime-kuma as the only reader of /health/ready. So a Coolify service with a hard dependency needs /health/ready, because that is what makes the monitor see past the front door, and one without a hard dependency does not: the endpoint would answer exactly what /health/live answers, and the monitor for a frontend points at / anyway.

On Coolify, report draining on /health/live. Coolify’s rolling update stops the old container with docker stop once the new one is healthy, and Traefik routes to a container until it exits. A server stops accepting connections when SIGTERM arrives and then waits for open requests and streams, so for as long as that wait lasts Traefik hands it requests that fail: uvicorn refuses them, and gunicorn leaves them queued unanswered until its workers exit. Reporting DOWN makes the container’s own health check fail, Docker marks the container unhealthy after retries failed checks, and Traefik stops routing to it; the server keeps serving until then, the same order a cloud load balancer follows when it deregisters a target. A Traefik loadbalancer.healthcheck label does not hold for this: Traefik starts every server of a rebuilt load balancer as up and checks it again only one interval later, so any configuration rebuild during the shutdown puts a container that has stopped listening back in rotation. See shutting down for the timings and the shutdown sequence.

/version is the cheapest of them to add. It turns “which commit is live on this hostname?” from an SSH session into a curl, and it is what makes verifying a deploy or a proxy change a diff instead of an investigation.

What not to check

Soft dependencies. Mail, object storage, analytics, a search index the UI degrades gracefully without. If the service can still answer requests without it, its failure does not belong in /health/ready. Expose it as a metric.
Other services’ health endpoints. Chaining readiness checks is how one service’s bad afternoon becomes everyone’s outage.
Anything expensive. These endpoints run every few seconds for the life of the container. No full table scans, no writes, no external API calls.
Business data. These endpoints are unauthenticated. Do not return record counts, configuration, dependency hostnames, or stack traces. If you want a detailed view, put it behind auth on a separate endpoint.

Verification that does real work, such as counting rows through the application’s own connection to prove a deploy landed correctly, is valuable and belongs in a one-shot script run at a deploy boundary. Never promote it to a continuous check.

Checklist

Naming and its roots

There is no standard path. There is a standard concept.

The liveness, readiness and startup split is settled across every major ecosystem. The URL is not. No RFC defines one, health does not appear in the IANA well-known URI registry, and the closest IETF work, Health Check Response Format for HTTP APIs, expired in 2022 without being adopted by a working group, so cite it as a draft rather than as a standard.

One ratified specification does fix the paths, and it is the one we follow. MicroProfile Health 4.0 (Eclipse Foundation, 2021) requires /health/live, /health/started and /health/ready, and defines the response body as UP or DOWN with a checks array, mapped onto 200 and 503. Quarkus, Open Liberty and WildFly implement it.

We take that wire contract and nothing else. The rest of the specification is Java: CDI annotations, a HealthCheck interface, a response builder, bean discovery. None of it ports, and none of it is needed to serve the same paths and the same JSON from any language.

We also serve two of its three paths rather than all three. /health/started is there for runtimes that accept requests while still booting; ours do not, since boot work completes before the server serves, so a third path would never answer anything /health/live does not. The startup semantic still exists, carried by the startup probe’s budget instead of by a URL. Deviating is the point of knowing what the spec says: take the parts that earn their place, and record which ones you left.

The alternatives are conventions rather than specifications:

Path	Used by
`/livez`, `/readyz`	Kubernetes API server, Node.js Reference Architecture
`/healthz/live`, `/healthz/ready`	ASP.NET Core
`/actuator/health/liveness`	Spring Boot Actuator, which also serves `/livez` and `/readyz` on request

The trailing z throughout is Google’s “z-pages” convention (/varz, /statusz, /rpcz), meant to stop operational endpoints colliding with real routes; Kubernetes documents its own family. The Node.js Reference Architecture rejects a bare /healthz because it does not say whether it means liveness or readiness, a criticism /health/live answers directly.

For gRPC there is a real cross-language standard, the gRPC Health Checking Protocol, which uses a service method rather than a path.