Why an OpenStack Magnum cluster sticks in CREATE_IN_PROGRESS, DELETE_IN_PROGRESS or UPDATE_FAILED, where the cause shows up, and how to clear it.
You created a Kubernetes cluster through OpenStack Magnum, and hours later openstack coe cluster list still shows it in CREATE_IN_PROGRESS. Or a delete never finishes and the cluster sits in DELETE_IN_PROGRESS. Or an upgrade or resize ended in UPDATE_FAILED, and Magnum now refuses the next change.
Magnum is OpenStack's service for creating Kubernetes clusters. On clouds that use magnum-cluster-api, the Magnum driver we maintain, Magnum hands the work to Cluster API. Cluster API is a Kubernetes project whose controllers build each cluster as a set of objects on a management cluster. Magnum shows you one status word. The reason for it lives in those objects and in the controller logs.
This matters now because Kubernetes 1.34 goes out of upstream support on October 27, 2026. Clusters on 1.34 need an upgrade before then, and updates, upgrades and resizes can all end in UPDATE_FAILED. Magnum 23.0.0, tagged on September 30, still accepts updates and upgrades only from a completed state.
Until a stuck cluster leaves that state, you can't update or upgrade it. A delete that hangs also leaves ports, security groups and sometimes a load balancer behind, and they all count against the project's quota.
These steps are for cloud operators who run Magnum with magnum-cluster-api, on Atmosphere or another deployment, and who can run kubectl against the management cluster. For each state, you'll see which fields and objects to read, the causes users have reported in our issue tracker, and the fix our maintainers or docs give. Two of those fixes edit a database by hand, so find the right cause before you touch anything.
Start with status_reason in OpenStack Magnum
One command gives you the status, the reason and the name you need for everything else:
openstack coe cluster show <cluster> -c status -c status_reason -c cluster_id
Our driver builds status_reason from two messages: the latest event on the Cluster API Cluster object and the latest event on its OpenStackCluster. It does this while a create or update is waiting on a Cluster API condition that is not yet True, and for as long as a delete is in progress. Either message can be empty. The text starts with "CAPI Cluster status:". In the delete reports below, the OpenStack error showed up in this field.
From API microversion 1.13, which Magnum 23.0.0 supports, the last field is called cluster_id, and python-magnumclient 5.0.0 shows it under that name with any microversion. Older clients on an earlier microversion show the same value as stack_id. Our troubleshooting guide calls it stack_id, and so does the rest of this post.
The stack_id is the name of the Cluster API Cluster. The related objects carry it in the label cluster.x-k8s.io/cluster-name, and all of them live in the magnum-system namespace. On Atmosphere, Magnum runs directly on the management cluster. clusterctl describe cluster -n magnum-system <stack_id> --grouping=false prints the object tree with machines listed one per row. Add --show-conditions all to see every condition on those objects.
Magnum cluster stuck in CREATE_IN_PROGRESS
The legacy Heat driver moves a failed build to CREATE_FAILED. magnum-cluster-api behaves differently: when a Cluster API step fails, the cluster stays in CREATE_IN_PROGRESS until it is created or you delete it. A long CREATE_IN_PROGRESS can mean one step is failing over and over.
Magnum reports CREATE_COMPLETE once the Cluster API Cluster shows ControlPlaneReady, InfrastructureReady and Ready as True and every node group has finished. Work down the objects in order:
kubectl -n magnum-system get clusters <stack_id>: if the phase is Provisioned, move on.- The
OpenStackCluster with label cluster.x-k8s.io/cluster-name=<stack_id>: READY should be true. Its events cover the floating IP, the API load balancer and the security groups. - The
KubeadmControlPlane with the same label: compare READY with REPLICAS. - If fewer control plane nodes are ready, describe their
OpenStackMachine objects and find any condition that is False. Its Message field names the error.
kubectl -n magnum-system describe openstackmachines \
-l cluster.x-k8s.io/control-plane=,cluster.x-k8s.io/cluster-name=<stack_id>
Running out of capacity is one reported cause. In one report, a cluster with three control plane nodes and one worker booted two control plane nodes and the worker. The third control plane node then failed for lack of resources. The driver keeps retrying, so the step can succeed once you free quota or capacity. If you delete the cluster instead, read the DELETE_IN_PROGRESS section first: that cluster also got stuck in deletion.
Why does a Magnum cluster stay in DELETE_IN_PROGRESS?
During a delete, the driver keeps writing status_reason for as long as the Cluster API Cluster exists, so read it first. The reports fall into two groups: OpenStack resources that won't delete, and Keystone credentials the controllers can no longer use.
Security groups come up in two of the reports. In #188, status_reason read "Faileddeletesecuritygroup" with an HTTP 409 from Neutron. A 409 there means a port or instance still uses the group. That reporter found 24 ports tied to the cluster, nine of them on that group.
Our maintainers asked them to find which resources still held the group, and to check the Neutron logs for why those ports were not deleted. In #63, the API load balancer first sat in PENDING_DELETE, and then a leftover port held the security group. Cluster API Provider OpenStack fixed the bug behind #63 starting with 0.8.0.
Two older driver errors also stopped deletes, and both are fixed in later releases:
- A 403 Forbidden from Keystone while removing the cluster's application credential (#485). The reporter traced it to the Keystone policy on their cloud, which did not let an admin token list the cluster's application credentials. Release 0.38.2 skips a 403 at that step.
- "ApiError: clusterresourcesets.addons.cluster.x-k8s.io ... not found" on 0.26.0, fixed in 0.26.2.
If you see either, upgrade the driver before you dig further.
Magnum cluster delete when the Keystone project is gone
The Cluster API OpenStack controller talks to OpenStack with an application credential from the cluster's project. If that project is deleted first, nobody can delete the cluster, not even an admin. The controller log shows a "Reconciler error" with "Could not find project" and a 404 from Keystone's token endpoint. Read it with kubectl -n capo-system logs deploy/capo-controller-manager.
The same thing happened to a user who created a cluster and was then removed from the group that gave them access. Our maintainers pointed to the workaround in the magnum-cluster-api troubleshooting guide:
- Create a cleanup project, then change its
id in the Keystone database to the cluster's project_id. - Give your admin user the
member role on it and switch to that project. - Create a new application credential with the secret already stored in the cluster's
<stack_id>-cloud-config secret in magnum-system, then swap the new credential's ID into that secret. - Watch the controller log until the cluster is gone, then delete the cleanup project.
The guide warns that a wrong database edit can corrupt Keystone, so take a backup first. On Atmosphere, Keystone's database runs in the Percona XtraDB Cluster in the openstack namespace, and the guide includes the exact command for it.
A hand edit to Keystone on a production cloud is a good moment for a second opinion. Our engineers maintain magnum-cluster-api and its troubleshooting guide, and you can ask them to look at your case before you change the database.
UPDATE_FAILED after a Kubernetes cluster upgrade or resize
In Magnum 23.0.0, the conductor accepts openstack coe cluster update and openstack coe cluster upgrade only from completed states such as CREATE_COMPLETE and UPDATE_COMPLETE. If the cluster is in UPDATE_FAILED, it rejects the request with "Updating a cluster when status is "UPDATE_FAILED"" or "Upgrading a cluster when status is "UPDATE_FAILED"". Resize is the exception: it also accepts UPDATE_FAILED. #373 calls the cluster stuck for that reason.
To find what failed, list the node groups with openstack coe nodegroup list <cluster>, then read each one's reason with openstack coe nodegroup show <cluster> <nodegroup> -c status -c status_reason. The control plane node group takes its status_reason from the failure message on the KubeadmControlPlane. When the Cluster API validation webhook rejects a change, the rejection shows up in the magnum-conductor log.
Two causes have come up in the threads:
- Autoscaled clusters on 0.37 and 0.38.1. A node group update or resize on a cluster with
auto_scaling_enabled=true went to UPDATE_FAILED. The Cluster API validation webhook rejected the change with "replicas: Invalid value: null" (#1144). Release 0.38.2, published in August, fixes it. Upgrade the driver before you change autoscaled clusters again. - A skipped minor version. One team reported UPDATE_FAILED after upgrading a cluster directly from Kubernetes 1.29 to 1.31. Our maintainers noted that the driver finds out too late to stop it. The hops to take are laid out in planning the Kubernetes upgrade path for magnum-cluster-api.
Resetting a Magnum cluster stuck in UPDATE_FAILED
Magnum's update and upgrade calls refuse a cluster in UPDATE_FAILED. In #373, a user reset three columns in the cluster table of the Magnum database, and our maintainers called that acceptable for that case:
status back to a completed state (the user set CREATE_COMPLETE);status_reason to NULL;cluster_template_id back to the template the cluster ran before the failed upgrade.
Take a database backup first. The reset changes only what Magnum records, so fix the cause you found above before you run the upgrade again.
Is the Magnum cluster healthy again?
For a create or an update, openstack coe cluster show should report CREATE_COMPLETE or UPDATE_COMPLETE with an empty status_reason. Our driver clears the reason when it marks a cluster complete. In magnum-system, the Cluster should be Provisioned, and the KubeadmControlPlane should show READY equal to REPLICAS.
For a delete, the cluster should be gone from openstack coe cluster list, and no Cluster named after its stack_id should be left in magnum-system. After that, check the project's ports and security groups. Security groups the controller created carry the stack_id in their names, such as k8s-cluster-magnum-system-<stack_id>-secgroup-controlplane. If you made a cleanup project, delete it now.
Before you change an autoscaled cluster again, confirm your driver release is 0.38.2 or later. If you are moving clusters off 1.34 before October 27, plan the hops with the upgrade path post first. Once the cluster you fixed is back in CREATE_COMPLETE or UPDATE_COMPLETE, Magnum's conductor accepts its next update or upgrade request again. Retry the operation and check the result before you close the ticket.
