Guides¶
Two ways in. If something is broken, start with the symptom. If you are setting something up, start with the task.
By symptom¶
| What you are seeing | Cause | Go to |
|---|---|---|
Cannot connect to the Docker daemon at tcp://docker:2375 |
TLS port mismatch, or no shared cert volume | Docker-in-Docker |
privileged pod rejected on admission |
Pod Security Admission restricted |
Rootless builds |
exec format error, helper container crashes at once |
amd64 helper image on an arm64 node | Node placement |
Waiting for pod ... then the job fails after 3 minutes |
poll_timeout, usually a slow image pull |
Jobs stuck in Pending |
0/N nodes are available |
selector or toleration matches nothing | Node placement |
| Exit code 137, job dies at the same step every time | OOM kill | Sizing jobs |
Job dies during uploading artifacts |
the helper container's own limit | Sizing jobs |
| Pod evicted, node looks healthy | ephemeral storage with no request | Persistent and ephemeral storage |
pods is forbidden |
RBAC, or a build namespace that is not allowed | RBAC and namespaces |
poddisruptionbudgets is forbidden |
the optional PDB grant, or an upgrade window | Spot and preemptible nodes |
| Mounted secret is missing inside the job | wrong namespace, or a key that needs items |
Mount secrets and configmaps |
ImagePullBackOff on the job's own image |
pull secret missing, or Docker Hub rate limit | Pull from a private registry |
| Connection refused reaching postgres or redis | service not ready yet, or OOM-killed | Service containers |
| Runner is online but picks up almost nothing | concurrent and request_concurrency are both 1 when unset |
Concurrency |
Runner stuck NotReady, connection error reaching GitLab |
egress proxy, and spec.environment does not reach the manager |
Outbound proxy |
Jobs fail as script_failure after a node disappeared |
spot eviction is not classified as a system failure | Spot and preemptible nodes |
cache: declared but never restores |
no distributed cache support yet | Limitations |
x509: certificate signed by unknown authority |
GitLab behind a private CA | Authentication |
| Runner NotReady, no obvious reason | read status.error and the Ready condition |
Watching a runner |
By task¶
Building images
- Docker-in-Docker: privileged builds, and the certificate share that makes TLS work
- Rootless builds: kaniko or buildkit, for clusters that forbid privileged
Scheduling
- Node placement: node pools, taints, arm64, and mixed-arch fleets
- Spot and preemptible nodes: cheap capacity, and making retry actually fire
Volumes and secrets
- Mount secrets and configmaps: deploy keys, kubeconfigs,
itemsmapping - Pull from a private registry: pull secrets, service accounts, rate limits
- Persistent and ephemeral storage: emptyDir, PVCs, CSI, and eviction
Capacity
- Sizing jobs: three containers, three budgets, and per-job overrides
- Concurrency: what
concurrentreally buys you here - Jobs stuck in Pending: finding out why, not just waiting longer
Isolation and networking
- Dedicated build namespace: the allow-list, PSA, quotas, NetworkPolicy
- Service containers: postgres, redis, aliases as hostnames
- Outbound proxy: HTTP_PROXY/HTTPS_PROXY/NO_PROXY on the manager, not the job
Operations
- Watching a runner: status fields, runner metrics on 9090, operator metrics
Before you file a bug¶
Two things are known gaps rather than misconfiguration, and each has an issue:
| Gap | Issue |
|---|---|
No distributed cache ([runners.cache]) |
#81 |
| Manager does not drain gracefully on rollout or eviction | #84 |
The full list, including fields that are accepted but inert, is in limitations.