Des notes d’ingénierie issues de l’exploitation d’infrastructures ouvertes : pannes, décisions de conception et travail upstream qui améliorent l’infrastructure ouverte.
Ephemeral CI Runners: When Every Build Should Start Clean
Ruchi RoyAuthor
Learn how ephemeral CI runners reduce cross-build contamination and security risk, where they add complexity, and when persistent runners still make sense.
A continuous integration job should be judged on the code and inputs it was handed. What happened to run on the same machine an hour earlier should not enter into it.
That sounds obvious. Yet many CI systems rely on persistent runners: long-lived virtual machines or physical hosts that execute one job after another. These runners can be fast and economical, especially when builds need specialized hardware or large local caches. They can also accumulate files, processes, packages, credentials, and configuration changes that are not part of the declared build.
An ephemeral CI runner takes a different approach. It receives a job, starts from a controlled image, executes the workload, sends logs and artifacts to durable storage, and is then destroyed instead of returned to a shared pool.
The payoff goes beyond cleaner CI. Ephemeral runners make the boundary between builds easier to reason about. They reduce the opportunity for one job to contaminate the next, and they limit how long a compromised build environment stays useful to an attacker.
They are not a complete security solution, however. A disposable runner can still boot from an unsafe base image, receive overly broad credentials, expose a privileged container runtime, or write poisoned data to a shared cache. So the useful question skips past whether ephemeral runners are automatically secure. Ask instead whether making runner state disposable improves the trust model for your particular builds, and what controls need to surround it.
What is an ephemeral CI runner?
An ephemeral CI runner is a compute environment created or assigned for a limited workload and removed from service when that workload finishes. Depending on the platform, it may be a virtual machine, Kubernetes pod, cloud instance, or another isolated node.
Lifecycle is what matters here. State created by one job should not survive on a runner that later executes a different job.
That makes an ephemeral runner different from:
A long-lived host that runs a cleanup script between jobs.
A persistent host that launches a new container for each job but retains host-level state.
A runner that disconnects from the CI control plane after one job while its underlying machine remains available.
Those designs may still provide useful separation, though they are not equivalent. If the host, container runtime, attached volumes, credentials, or caches survive, some state may cross the job boundary.
Ephemeral is also not the same as hermetic. A hermetic build controls all inputs and generally avoids undeclared network dependencies. An ephemeral build can still download packages, call external services, or use shared caches. It starts with a fresh environment. It does not automatically make every input reproducible.
Why persistent runners become unpredictable
Persistent runners often start simple. A team provisions a machine, installs the required toolchain, and connects it to the CI system. Over time, convenience becomes hidden state.
One build installs a package globally. Another leaves a background process running. A troubleshooting session changes a configuration file. A cache holds an artifact produced with an older compiler. A test passes because a dependency already sits on the runner, even though the pipeline never installs it.
The resulting failures are difficult to diagnose, because rerunning the same commit does not necessarily recreate the same conditions. A job may:
Pass on one runner and fail on another.
Fail only after a particular project used the machine.
Depend on a file or service that is absent from the pipeline definition.
Behave differently after a manual repair or operating-system update.
Not every flaky test is an infrastructure problem, and replacing runners will not repair nondeterministic application code. But when build outcomes depend on runner history, the CI system is testing more than the submitted change.
A disposable environment removes a large category of history. Teams still need versioned images, pinned dependencies, deterministic test data, and observable pipelines. A fresh starting point simply makes those controls easier to enforce.
The more important issue: CI is a code-execution boundary
Reliability is only half the argument.
A runner executes instructions supplied through the repository and pipeline configuration. That code may interact with source repositories, package registries, deployment systems, signing processes, internal networks, and secrets. Even a well-intentioned developer can introduce a dependency or script that changes the runner in an unsafe way.
The risk grows when the same persistent runner serves multiple projects. GitLab's security guidance explicitly warns that self-managed CI jobs create remote-code-execution risk, and that the risk sharpens when non-ephemeral runners are shared across projects. A malicious or compromised job may leave code on the runner, reach material from another project, or capture secrets exposed to later workloads.
An ephemeral runner narrows that opportunity. Destroying the environment after the job means an attacker cannot rely on host persistence to wait for a more valuable workload. It also lowers the chance that ordinary build residue from one repository becomes visible to another.
None of this removes the need for separate trust tiers. A pull request from an external contributor should not automatically receive the same network access and credentials as a release job from a protected branch. Ephemeral infrastructure works best alongside:
Short-lived, job-specific credentials.
Separate runner pools for different trust levels.
Restricted network paths and egress.
Non-privileged execution wherever possible.
Controlled, regularly rebuilt base images.
Durable logs and audit data stored outside the runner.
The lifecycle control limits persistence. The surrounding controls limit what a job can reach while it runs.
Clean runners support stronger software supply-chain controls
Build isolation has become a formal supply-chain concern in its own right.
SLSA's build requirements make this explicit at Build Level 3: an isolated build platform must prevent one build from persisting in or influencing a later build environment. The same requirements call out overlapping builds, access to platform secrets, cache poisoning, and undeclared external influence.
Ephemeral runners support that isolation objective, since the job environment is discarded instead of reused. By themselves, they do not make a build SLSA-compliant. Provenance generation, the trust boundary of the build platform, secret protection, cache behavior, and other requirements all still matter.
That distinction helps any team evaluating a CI architecture. A single-use runner offers evidence of a stronger build boundary, though not proof that the entire software supply chain is secure.
Ephemeral runners have real trade-offs
Destroying state after every job shifts work elsewhere. A sound design accounts for that operational cost instead of treating disposability as free.
Provisioning time
A new virtual machine or pod must become ready before the job starts. If images are large, capacity is scarce, or instances are created only after demand arrives, queue and startup times climb.
Common mitigations include maintaining a controlled ready pool, keeping images lean, preinstalling stable toolchains, and monitoring queue time separately from execution time. The aim is fast access to fresh capacity, achieved without preserving build state between jobs.
Image lifecycle
Persistent runners get patched in place. Ephemeral runners lean far more heavily on the quality of their source images.
Teams need a repeatable image pipeline: build, scan, test, version, promote, and retire. Pinning a job to an image version improves repeatability, while a clear update policy stops that repeatability from turning into permanent exposure to old vulnerabilities.
Caching
Removing the runner also removes its local cache. Without another strategy, builds may repeatedly download dependencies or reconstruct the same intermediate outputs.
Remote caches recover much of that performance, though they become shared infrastructure with their own trust boundary. Cache keys should include relevant dependency, platform, and toolchain information. Sensitive jobs may need separate caches, read-only caches, validation of retrieved content, or no shared cache at all.
SLSA's isolation requirements specifically address cache poisoning: using a cache should not change the resulting build output. Speed is worth having only when the cache cannot silently redefine the build.
Logs and debugging
When a failed runner disappears, so does any evidence that was not exported first. Pipelines should send console logs, test results, artifacts, image identifiers, and relevant environment metadata to durable systems before teardown.
Interactive access to a failed environment can help with debugging, though retaining it changes the risk profile. Where a platform supports an explicit hold or quarantine workflow, access should be time-limited, audited, and isolated from subsequent builds.
Capacity and cost
Ephemeral capacity scales with demand, yet it does not erase infrastructure limits. Burst traffic, cloud quotas, image distribution, specialized instance availability, and startup latency all shape how quickly jobs begin.
Measure the system as a queue, not only as a collection of runners. Useful signals include:
Time waiting for an appropriate runner.
Time from allocation to job readiness.
Runner provisioning failure rate.
Job execution time by image and workload class.
Idle capacity maintained to absorb bursts.
Infrastructure cost per completed workload, including failed jobs.
Those numbers give a better basis for tuning than an assumption that either maximum reuse or maximum disposability is always cheaper.
When persistent runners still make sense
Persistent runners are not automatically a mistake. They can be reasonable when a workload depends on:
Scarce accelerators, lab devices, or other physical equipment.
Licensed software tied to a specific host.
Very large datasets that cannot be staged for every job.
Trusted, single-project workloads with no cross-repository reuse.
A controlled environment where reset and verification are demonstrably effective.
Even then, persistence should be an explicit exception rather than an invisible default. Limit each runner to the narrowest practical trust domain, keep secrets away from untrusted changes, and verify what cleanup can and cannot remove.
A hybrid model is often the realistic answer: ephemeral workers for general tests and untrusted changes, with dedicated persistent runners for specialized jobs. The two pools should carry separate labels, permissions, credentials, and network policies, so a pipeline cannot casually select the more privileged environment.
Where Zuul and Nodepool fit
Zuul coordinates CI/CD workflows around code review, including project gating and testing changes across repositories. Nodepool manages the test resources those jobs use. The official Nodepool documentation describes support for launching single-use test nodes from cloud providers, along with managing predefined, pre-existing nodes.
This separation is useful. Pipeline logic does not have to bind to one permanent fleet. Teams can choose resource types for different jobs and use clean, single-use nodes where that boundary matters.
VEXXHOST's Managed Zuul offering connects that model to hosted, on-premises, or self-managed operating requirements. VEXXHOST states that its managed offering uses Zuul with Nodepool to launch blank, single-use test nodes and supports several infrastructure drivers, while also providing monitoring, upgrades, and support.
That does not remove the architecture decisions in this article. Teams still need to define trust tiers, credentials, images, cache boundaries, network access, log retention, and capacity. An experienced operator helps make those controls deliberate, and keeps the CI platform itself maintained while development teams focus on the software it tests.
Start with the trust boundary, not the runner count
The strongest case for ephemeral CI runners has little to do with being newer or more cloud native. It rests on one property: a job can begin from a controlled state and finish without leaving a hidden inheritance for the next one.
For teams running untrusted changes, sharing infrastructure across repositories, or strengthening software supply-chain controls, that property improves both reliability and security. For specialized, trusted workloads, a tightly scoped persistent runner may remain the better engineering choice.
Start by identifying what each job can execute, what it can access, and what survives after it ends. Then choose the runner lifecycle that makes those answers easiest to defend.
Virtual machines, Kubernetes & Bare Metal Infrastructure
Choose from Atmosphere Cloud, Hosted, or On-Premise. Simplify your cloud operations with our intuitive dashboard. Run it yourself, tap our expert support, or opt for full remote operations. Leverage Terraform, Ansible or APIs directly powered by OpenStack & Kubernetes
