build-image: push via in-cluster registry on runners that resolve it internally #14

Open

mail wants to merge 1 commit from registry-push-in-cluster into main

mail commented

2026-10-28 16:30:33 +00:00

Owner

build-image: push via the in-cluster registry on runners that resolve it internally

Why

Paperclip's image builds (and any build with layers big enough not to finish
within ~60s) have been failing since Oct 24 with:

502 Bad Gateway: coreflared: upstream unavailable
PUT https://reg.levior.io/v2/levior/paperclip/blobs/uploads/...?digest=sha256:...

infrastructure #147 made the registry reachable in-cluster from the runners
(hostAlias reg.levior.io -> zot ClusterIP, a 443-fronting Service, dind
--insecure-registry). That fixed dockerd, but builds still failed: with a
TCP DOCKER_HOST, buildx uses the docker-container driver, and the push is
performed by the nested buildkitd container — which has its own /etc/hosts
(docker-managed) and resolves reg.levior.io to the public coreflared edge.
Every push still hairpinned pod → public edge → tunnel → connector → Zot, and
coreflared kills any request whose response headers take longer than ~60s.
Large monolithic blob PUTs can never meet that.

What changes

A new "Detect in-cluster registry" step runs before docker/setup-buildx-action:

If the registry host resolves to a private IP (the levior runner pods map
it via hostAliases to the in-cluster Zot service), the builder is created with
driver-opts: network=host (buildkitd then shares the pod netns and sees the
hostAlias mapping) and a config-inline that marks the registry as plain-HTTP
(http = true, insecure = true) so buildkitd talks to the plain-HTTP Zot
front on port 443. Pushes then go pod → ClusterIP → Zot, no tunnel.
If it resolves publicly (runners outside the cluster), nothing changes — the
builder is created exactly as before.
The require_postgres driver-opts/buildkitd-flags behavior is unchanged.

Evidence

Measured on the runners (branch runs):

Nested buildkitd /etc/hosts has no hostAlias entry and resolves
reg.levior.io → flared.core.ci → 167.235.104.250 — the tunnel path.
With host networking + plain-HTTP config, a 600MB random blob pushed in
15s; through the tunnel the same PUT dies at 60s on every attempt.
Mirrors ([registry."..."].mirrors) are pull-only — they do not redirect
pushes (verified: push URL stayed https://reg.levior.io and 502'd), so the
host-network approach is the one that works.
template-smoke.yml (new, workflow_dispatch) builds Dockerfile.smoke, a
500MB incompressible layer that cannot fit through the tunnel: green, with
pushing layers 4.9s. Run it after future template changes to prove the
in-cluster path still works.
Two host-network builders boot fine concurrently on one runner pod (buildx
assigns each its own port); a hardcoded --addr in buildkitd-flags breaks
that negotiation and must not be used.

# build-image: push via the in-cluster registry on runners that resolve it internally ## Why Paperclip's image builds (and any build with layers big enough not to finish within ~60s) have been failing since Oct 24 with: ``` 502 Bad Gateway: coreflared: upstream unavailable PUT https://reg.levior.io/v2/levior/paperclip/blobs/uploads/...?digest=sha256:... ``` infrastructure #147 made the registry reachable in-cluster from the runners (hostAlias `reg.levior.io -> zot ClusterIP`, a 443-fronting Service, dind `--insecure-registry`). That fixed **dockerd**, but builds still failed: with a TCP `DOCKER_HOST`, buildx uses the **docker-container driver**, and the push is performed by the nested buildkitd container — which has its own `/etc/hosts` (docker-managed) and resolves `reg.levior.io` to the public coreflared edge. Every push still hairpinned pod → public edge → tunnel → connector → Zot, and coreflared kills any request whose response headers take longer than ~60s. Large monolithic blob PUTs can never meet that. ## What changes A new "Detect in-cluster registry" step runs before `docker/setup-buildx-action`: - If the registry host resolves to a **private IP** (the levior runner pods map it via hostAliases to the in-cluster Zot service), the builder is created with `driver-opts: network=host` (buildkitd then shares the pod netns and sees the hostAlias mapping) and a `config-inline` that marks the registry as plain-HTTP (`http = true`, `insecure = true`) so buildkitd talks to the plain-HTTP Zot front on port 443. Pushes then go pod → ClusterIP → Zot, no tunnel. - If it resolves publicly (runners outside the cluster), nothing changes — the builder is created exactly as before. - The `require_postgres` driver-opts/buildkitd-flags behavior is unchanged. ## Evidence Measured on the runners (branch runs): - Nested buildkitd `/etc/hosts` has no hostAlias entry and resolves `reg.levior.io → flared.core.ci → 167.235.104.250` — the tunnel path. - With host networking + plain-HTTP config, a **600MB** random blob pushed in **15s**; through the tunnel the same PUT dies at 60s on every attempt. - Mirrors (`[registry."..."].mirrors`) are pull-only — they do **not** redirect pushes (verified: push URL stayed `https://reg.levior.io` and 502'd), so the host-network approach is the one that works. - `template-smoke.yml` (new, workflow_dispatch) builds `Dockerfile.smoke`, a 500MB incompressible layer that cannot fit through the tunnel: green, with `pushing layers 4.9s`. Run it after future template changes to prove the in-cluster path still works. - Two host-network builders boot fine concurrently on one runner pod (buildx assigns each its own port); a hardcoded `--addr` in `buildkitd-flags` breaks that negotiation and must not be used.

mail added 1 commit

2026-10-28 16:30:33 +00:00

build-image: push via in-cluster registry on runners that resolve it internally 4149b336c5

The nested buildkitd container resolves the registry's public name, so
image pushes hairpin through the coreflared tunnel, whose 60s
time-to-first-response deadline kills large layer uploads with 502s
(paperclip image builds have failed this way since Oct 24).

When the runner pod resolves the registry host to a private IP (the
levior runners map it to the in-cluster Zot service), run buildkitd on
host networking so it sees that mapping, and let it speak plain HTTP to
the registry. Pushes then go pod -> ClusterIP -> Zot.

Add a smoke workflow that builds a 500MB incompressible layer: it cannot
fit through the tunnel, so a green run proves the in-cluster path.

This pull request is blocked because it's outdated.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin registry-push-in-cluster:registry-push-in-cluster

git switch registry-push-in-cluster

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

levior/pipeline-templates!14

No description provided.

Rows
Columns