Image tags and pull policy

An image reference has two halves, and both decide what runs: the tag names a build, and the pull behaviour decides whether the machine goes and looks. Get either wrong and the same commit behaves differently on different machines, with nothing in the repository to explain why.

The rule

Reference a specific, immutable tag wherever you can. A tag that moves is a deliberate exception and has to earn itself.

Reference	Pulled fresh	Served from a local cache
Immutable tag	The build you named	The build you named
Floating tag	Whatever was last pushed to that tag	Whatever that machine happened to pull first

The top row does not care about the pull policy at all. Pinning removes a whole axis of the problem instead of managing it, and makes “which build is running?” answerable from a file rather than from the state of the machine that ran it. The bottom right cell is what you are avoiding: not a version, a per-machine accident, producing failures that look intermittent while being permanent. There is usually already something to pin to: CI pushes $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA on every build (see CI image builds), so the immutable tag exists before anyone needs it.

Pulling

always does not mean re-download. Docker checks the manifest and digest against the registry, uses the cached image when it matches, and pulls only the layers that changed. The cost is a manifest check, roughly 100 to 300 ms per start, so “it would slow the pipeline down” is not a reason to leave it off where it belongs.

Reference lives in	Default behaviour	What you do
A CI job’s image:	Set by the runner, not the project	Choose the tag; the fleet handles the pull
A Compose service	Pull only when the image is absent	Set pull_policy: always on tags that move
A Dockerfile FROM	Resolved once, at build time	Pin the upstream version

Compose is the one that needs writing by hand, and the one most often missed: a stack on :latest with no pull policy runs whatever each machine downloaded first. Moving is the test, though, and a tag pinned to at least a minor version does not move enough to matter: nobody needs the newest patch of redis:7.4, and a cached copy is right for it. Reserve always for tags that really do move, :latest, :test, a bare major or none at all, and above all for a dependency whose contents your tests assert against, where a stale copy changes what it means for a run to pass.

FROM upstream:latest fails later and louder: rebuilding an unchanged Dockerfile can jump a dependency a minor version underneath a running service.

Our runners are configured to check the registry, retry the remote pull once, and reuse a local copy only when the registry is genuinely unreachable, so a mirror outage cannot fail a job whose image is already on disk. GitLab tries the entries of pull_policy in order, which makes the list a retry sequence rather than a preference list. The current values live in the runner configuration. A job cannot override the pull, so in .gitlab-ci.yml the tag is the only lever you have.

Compose has no equivalent of that fallback, so always there makes every run depend on the registry being reachable. That is the trade: on a minor-pinned image it buys protection against an upstream re-pushing the same tag and charges every future run for it. Worth paying on the dependency under test, not on the base services.

Pinning where a version is a decision

always on a floating tag guarantees the newest build, not that anyone decided to ship it: an unrelated restart, a host reboot or a container moving machines redeploys whatever the default branch last built, at a moment nobody chose. So where a version change should be a decision, put the tag in a variable and pin it:

services:
  grafana:
    image: registry.gitlab.d-centralize.nl/dc/dcentralize/dc-monitoring/grafana:${MONITORING_IMAGE_TAG:?MONITORING_IMAGE_TAG is required}
    pull_policy: always

A restart now redeploys what is pinned, and shipping a new version is the deliberate act of setting the variable. Use the :? form rather than a :-latest default: an environment that has lost its pin should refuse to start, not quietly go back to floating. always earns its place beside a pin here because the variable can point anywhere, so a tag that does not exist fails at once instead of resolving to whatever is cached.

Working with a moving target

Sometimes the newest build is genuinely what you want, and sometimes pinning is not yours to choose. An image tracking a producer’s default branch is meant to move, a shared development stack wants the latest, and a consumer cannot pin what a producer never publishes. A moving target is workable, but it has to be worked at, because nothing else will tell you what you ran. Three practices make one safe.

Pull it every time. Without that you are back in the bottom right cell and the reference means nothing.

Record what you actually got. Log the resolved digest at the start of the run, before anything depends on it:

docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE"

One line, and a failure that reproduces on only some machines becomes diagnosable from the log instead of guessed at.

Fail fast on a definitive wrong answer. A gate waiting on a moving dependency has to separate “not up yet” from “up and answering the wrong thing”. A refused connection is worth retrying; a 404 on the route the gate polls is a final answer, and polling it to the end of the retry budget turns a permanent break into a slow one dressed up as flakiness. Stop on the first definitive response and report what came back. The health check endpoints make that distinction available.

Depending on another project’s image

This is the case that goes wrong most often, because both repositories look correct in isolation. A project’s tests run against an image another publishes, on a floating tag, with no pull policy, and neither repository records which build is under test. The producer ships a change, the consumer’s pipelines change behaviour with no commit in the consumer, and which runner a job lands on decides the answer.

The recommendation is one thing, not a menu. The consuming repository pins an immutable tag of the producer’s image, in a variable in its own CI configuration. Its Compose file reads that variable, so moving to a newer build of the dependency becomes a commit in the consumer, with a diff, a pipeline and a reviewer.

Publishing that tag is the producer’s half of the bargain, and it costs nothing. A tag other projects consume is an interface, so publish an immutable tag alongside the moving one and give consumers somewhere to pin.

Be honest about the cost: pinning buys a bumping habit. Somebody has to move the pin, and a pin left alone for months means testing against an image nobody runs any more. Hand it to Renovate where it can see the tag, or make moving consumers part of the producer’s release routine. It is smaller than the cost it replaces, paid by whoever debugs the next failure that reproduces on half the fleet. A consumer that cannot pin is working with a moving target, and owes itself the three practices above.

Checklist

• Every image reference names a specific tag, unless a moving one has a stated reason
• Every Compose service whose tag genuinely moves sets pull_policy: always; minor-pinned images do not
• Every Dockerfile FROM names a version, never latest
• Anything whose version should survive a restart reads its tag from a required variable
• Every image consumed from another project is pinned, or handled as a moving target
• Every run against a moving target logs the resolved digest
• Every gate waiting on a dependency stops on a definitive wrong answer instead of polling to its budget
• Something moves the pins, whether that is Renovate or a step in the producer’s release routine