
Air gapped cloud upgrades with Atmosphere release manifests
Air gapped cloud on OpenStack: how Atmosphere's digest-pinned release manifests move a release across the boundary, and what the manifest leaves out.
Lire la noteNotes de terrain / Dernières nouvelles
Des notes d’ingénierie issues de l’exploitation d’infrastructures ouvertes : pannes, décisions de conception et travail upstream qui améliorent l’infrastructure ouverte.
Parcourir toutes les notes
Air gapped cloud on OpenStack: how Atmosphere's digest-pinned release manifests move a release across the boundary, and what the manifest leaves out.
Lire la note
Learn when your team needs OpenStack consulting services and how expert support can help with deployments, migrations, upgrades, and operations.
Lire la note
Assess enterprise AI readiness before investing in GPUs. Explore 12 key factors covering data, models, infrastructure, governance, operations, and costs.
Lire la noteA merge queue, project gating on your own Zuul, or gating as a managed service: when each one holds up, where it breaks, and what Zuul 14 changed for teams running it themselves.
Most engineering teams protect their main branch the same way. A developer opens a pull request, CI runs the tests, a reviewer approves, and the change merges. The gap shows up when two changes each pass on their own and break the build once both have landed.
A merge queue closes that gap inside one repository. Approved pull requests wait in line, and each one is tested against the latest main branch plus every change queued ahead of it before it merges. Project gating, as Zuul runs it, stretches the same idea across several repositories. Zuul is an open source CI system built for gating. It keeps one queue for a group of repositories, and a change in one of them can declare that it needs a change in another. OpenStack tests its code through Zuul: about 1.6 million jobs ran during the 2026.2 cycle, which was released on September 30.
Teams that run their own Zuul have work ahead. Zuul jobs are written as Ansible playbooks, step-by-step files for Ansible, the widely used open source automation tool. Zuul also supplies build machines to the jobs that ask for them. Both halves are changing. In February, Zuul 14.0.0 added zuul-launcher, a new component that supplies build machines, and deprecated Nodepool, which did that job before. On September 29, Zuul 14.3.0 made Ansible 11 the default and deprecated Ansible 9. If your Zuul still uses Nodepool and Ansible 9, you now have two migrations to plan before either old piece is removed.
Engineering leaders of products split across several repositories meet this choice first, because those products are the ones that need gating. There are three paths: keep a merge queue on the code host, run project gating on a Zuul your engineers operate, or hand gating to a managed service. Too little gating and changes that cross repositories keep breaking main after they merge. Too much, and engineers who could be building the product end up maintaining CI. Each path below gets its strongest case and its weak point, and the last section turns one number from last quarter's work into your choice.
GitHub's built-in merge queue shows how one works. It builds a temporary branch for each queued pull request: the newest base branch, the changes ahead of it, and the pull request itself. Required checks run on that branch. If the change conflicts with the base branch, or fails a check while the queue is set to merge only passing pull requests, the pull request leaves the queue and the ones behind it are rebuilt without it. Standalone merge queue tools add the same kind of queue on top of a code host.
For a single repository or a monorepo, the best merge queue tool for CI/CD is usually the one your code host already offers. Your developers already work in pull requests, nobody has a CI system to host, and the queue sits next to the code review it depends on. If builds need your own hardware or private network, self-hosted runners fit underneath it, though patching and cleaning that runner fleet becomes your work. Ephemeral CI runners looks at when a runner should start clean for every build.
The limit appears when one feature needs changes in two repositories, such as an API service and the client library that calls it. Each queue tests its own repository against its own main branch. Someone has to merge the pair in the right order by hand, and until the second change lands, one side runs against code that doesn't match it yet.
Zuul's dependent pipelines test changes in parallel by assuming the changes ahead of them will pass. Each change is tested as if everything ahead of it had already merged. If a change fails, Zuul reports it, removes it, and retests the changes behind it without it. When repositories share a queue, the jobs for each change include, by default, the changes ahead of it from the other repositories.
Cross-repository dependencies set Zuul apart for products spread over several repositories. A developer adds a Depends-On footer that points at another change, in the commit message on Gerrit or the pull request description on GitHub. Zuul then tests the change with the one it depends on and won't merge it without that change. The dependency can even cross code review systems, so a Gerrit change can depend on a GitHub pull request. A 2019 walkthrough of multi-repository dependencies shows how this looks in practice.

Gating earns its cost when you can point to the problem in last quarter's work. Look for releases that needed coordinated merges across repositories, a written merge order that engineers follow by hand, or an integration test that checks out several repositories at once. If you find none of that, a merge queue covers you, and Zuul adds a system to run with no failure for it to prevent.
Self-hosting is strongest for teams that already operate an OpenStack or Kubernetes cloud for build nodes and have engineers who already write Ansible playbooks. Those teams keep job definitions, secrets and build machines inside infrastructure they already control, and they can read and patch every part of the system.
Installing Zuul is a small part of the cost. From 13.0.0 in September 2025 to 14.3.0 on September 29, 2026, Zuul shipped eight releases. Four of them belong to the 14.x series, which began with 14.0.0 on February 12. A self-hosted Zuul has to keep up with that pace, and two of those releases, 14.0.0 and 14.3.0, each started a migration.
Two of this year's changes reach self-hosted installs. One affects every Zuul still using Nodepool, and the other affects every job still running on Ansible 9. The first is the move from Nodepool to zuul-launcher, which now manages images and build nodes. Zuul's release notes promise a generous period in which both are supported before Nodepool support is removed. That gives you time to plan, but the work still needs an owner. Your image and node configuration moves into Zuul's own configuration, and custom image builds run as ordinary Zuul jobs. Plan the move on a test environment that mirrors production.
The second is Ansible. With 14.3.0, a job runs on Ansible 11 unless the job, its tenant or the scheduler sets another version, and Ansible 9 will be removed in a future Zuul version. Run every job written and tested against Ansible 9 on Ansible 11 before the upgrade reaches your gate. A job that breaks on 11 can start failing changes that are fine.
Keeping up is the real cost of building gating in-house. You need someone who reads every Zuul release note, a staging Zuul, and time set aside for each migration. If that work rests on one engineer, an absence or a resignation leaves your gate on a version nobody upgrades.
A managed service suits teams that want cross-repository gating but don't want to run CI infrastructure. With our managed Zuul, we run the infrastructure layer underneath Zuul, monitor it constantly and carry out its upgrades. Zuul upgrades become our job.
A service has limits a buyer should test before signing. Work above the infrastructure layer, such as writing jobs, deciding which repositories share a queue and fixing a test that fails for real, needs an agreed owner. Ask any provider, us included, who maintains your job definitions when an Ansible default changes. Onboarding takes time too. If you plan the handover for the same month as a major release of your product, the two will compete.
Count last quarter's merged changes that needed a matching change in another repository. If the count is near zero, keep the merge queue you have and stop there. If a pair turns up every sprint, move to project gating this quarter.
Once you gate, the remaining choice is about people. Keep Zuul in-house if two engineers can own its upgrades through this year's migrations. If you can't name them, hand the gate to a service and keep your engineers on the jobs that test your product.
If you hand it over, look at what the provider runs through its own gate. We have used Zuul for our own CI since before we launched managed Zuul in 2019, and Zuul had been tested on our own cloud before that launch. Since 2016 we have also provided infrastructure for OpenDev, which runs the Zuul that tests OpenStack.
Two of our open source projects have used Zuul too: magnum-cluster-api runs its tests as Zuul jobs, and Atmosphere 1.11.0, released in April 2024, used Zuul for speculative image builds. For teams staying on a merge queue, we maintain a public project that auto-scales self-hosted CI runners on OpenStack.
Run the count this quarter, and if it points to gating, tell our engineers what you found and talk through how your repositories and code review system would run on a managed gate.
Choose from Atmosphere Cloud, Hosted, or On-Premise.
Simplify your cloud operations with our intuitive dashboard.
Run it yourself, tap our expert support, or opt for full remote operations.
Leverage Terraform, Ansible or APIs directly powered by OpenStack & Kubernetes