Learn why high availability doesn't guarantee resilience and how failure domains, redundancy, and failover testing help prevent outages.
High availability reduces downtime through redundancy and failover, but it doesn't guarantee infrastructure resilience. Shared dependencies, poorly isolated failure domains, configuration errors, and software failures can still cause major outages. Building resilient OpenStack, Kubernetes, and Ceph environments requires careful architecture, realistic failover testing, capacity planning, and ongoing operational readiness.
High availability is often treated as a guarantee that infrastructure will remain operational when something fails. By distributing workloads across multiple servers, replicating data, and configuring automatic failover, organizations can reduce the impact of individual component failures.
But redundant infrastructure can still experience major outages.
Two application servers may depend on the same storage system. A cluster designed to tolerate hardware failures may still be vulnerable to configuration errors, software bugs, or problems affecting an entire data center. Even when failover mechanisms work as intended, the remaining infrastructure may not have enough capacity to handle the additional workload.
The importance of looking beyond redundancy is reflected in the Uptime Institute's Annual Outage Analysis 2026. According to the report, 57% of respondents to its 2025 annual survey said their most recent major outage cost more than $100,000, while one in five reported costs exceeding $1 million. The report also highlights growing risks associated with infrastructure complexity, external dependencies, and automation. These findings illustrate why preventing outages requires attention to both infrastructure design and ongoing operations.
At VEXXHOST, we help organizations design, deploy, and operate OpenStack, Kubernetes, and Ceph infrastructure. Our services cover architecture, deployment, upgrades, troubleshooting, and ongoing operations, with hosted and on-premises options and expert support or fully managed services.
High availability is an important part of reliable infrastructure, but resilience requires a broader approach. Organizations also need to consider shared dependencies, failure domains, recovery procedures, and whether their systems can continue operating under conditions that go beyond the failure of a single component.
§1 Redundancy Doesn't Eliminate Every Single Point of Failure
Redundancy is one of the foundations of high availability. Running multiple instances of a service, replicating data, or deploying additional network connections can help infrastructure continue operating when individual components fail.
However, redundant components may still depend on the same underlying systems.
Consider an application running across several virtual machines. If those machines rely on a single storage system, network gateway, or authentication service, a failure in that shared dependency can affect the entire application. Adding more application servers does little to protect against an outage in the infrastructure they all use.
The same principle applies to cloud control planes. An OpenStack environment may have multiple compute nodes, but its ability to provision and manage resources also depends on services such as databases, message queues, and networking components. If these supporting services lack appropriate redundancy, parts of the cloud can become unavailable even while the compute nodes remain operational.
As we discuss in How to Design for Failure (with OpenStack), building resilient OpenStack environments involves making critical API services, databases, and messaging layers highly available, alongside appropriate networking and storage configurations.
Shared dependencies are not always obvious from an architecture diagram. Multiple services may use the same physical network path, rely on a common power source, or share infrastructure that becomes overloaded during failover.
Redundancy also needs to account for capacity. If one server fails and its workloads move to another, the remaining infrastructure must have sufficient resources to handle the additional demand. Otherwise, a successful failover may simply replace one outage with a broader performance problem.
Identifying shared dependencies and understanding what happens when they fail is therefore just as important as deploying multiple instances of critical services.
§2 Failure Domains Matter More Than Component Counts
Having multiple servers does not necessarily mean they are protected against the same failure. What matters is how those servers are distributed and which infrastructure components they share.
A failure domain is a group of resources that can be affected by a single event. Depending on the architecture, this might be a physical server, a rack, a network segment, or an entire data center.
Consider three virtual machines running an application. If all three are hosted on different physical servers but within the same rack, they may survive an individual server failure. However, a rack-level power or networking failure could take all three offline simultaneously.
Distributing those machines across separate racks or availability zones reduces the likelihood that one localized incident will affect every instance.
OpenStack provides availability zones and host aggregates that can help operators organize compute resources and influence workload placement. However, simply creating multiple availability zones does not guarantee physical isolation. Their effectiveness depends on how the underlying hardware, networking, and storage are arranged.
Storage requires similar consideration. Ceph uses the CRUSH algorithm to determine where data is placed across the cluster. With an appropriately configured CRUSH failure-domain hierarchy, replicas can be distributed across different hosts or racks rather than concentrated in resources vulnerable to the same failure.
VEXXHOST explores a related approach in Orchestrating Multi-Cloud Failover with Cluster API and OpenStack for Zero Downtime. The article describes an architecture using independent OpenStack regions, each with its own compute, networking, storage, and control-plane services, to reduce dependencies between regions.
The appropriate level of isolation depends on the failures an organization needs to withstand. Protecting against a failed disk requires a different architecture from protecting against the loss of an entire data center.
Resilience, therefore, depends less on how many components are deployed and more on whether a single failure can affect them all.
§3 High Availability Can Introduce Its Own Complexity
High availability mechanisms are designed to reduce downtime, but they also introduce additional components, dependencies, and operational requirements.
Clustering is a good example. Many distributed systems rely on quorum to make decisions and maintain consistency. If too many cluster members become unavailable, the remaining nodes may be unable to reach quorum, preventing certain operations even though some infrastructure is still running.
Failover mechanisms also require careful configuration. If two nodes incorrectly assume they should both become active, a split-brain situation can occur, potentially resulting in conflicting operations or data inconsistencies.
Software updates and configuration changes introduce further risks. A misconfigured load balancer, an incompatible cluster update, or a configuration error propagated across redundant nodes can disrupt services that were designed to tolerate hardware failures.
These challenges are particularly relevant in OpenStack, Kubernetes, and Ceph environments, where several distributed services must operate together.
Maintaining a highly available environment requires ongoing attention to cluster health, configuration consistency, quorum requirements, and failover behavior throughout the infrastructure's lifecycle.
§4 A Highly Available System Can Still Experience a Major Outage
High availability is typically designed around specific failure scenarios, such as the loss of a server, a network connection, or an individual service. But not every disruption can be addressed through redundancy, particularly when multiple components are affected by the same event.
Software bugs and configuration errors are good examples. If an incorrect configuration is deployed across every node in a cluster, additional nodes will not prevent the resulting disruption. A faulty software update can introduce the same problem across otherwise independent systems, while a misconfigured load balancer may direct traffic away from healthy application instances.
Data corruption presents another challenge. Replication helps maintain multiple copies of data, but it can also propagate corrupted or accidentally deleted information. A Ceph cluster may continue operating with its configured redundancy intact while an application loses access to valid data. Storage availability and data recoverability are different concerns, which is why snapshots, backups, and recovery procedures remain necessary even in highly available environments.
There is also an important distinction between infrastructure availability and application availability.
A virtual machine can remain operational while the application running inside it has stopped responding. The operating system may be healthy, the network connection active, and the storage accessible, yet users are still unable to complete requests.
VEXXHOST explores this distinction in Protecting the App Inside the VM with Heat. The article explains how application-level monitoring, OpenStack Heat orchestration, and automated recovery can work together to respond to failures that basic virtual machine health checks might miss. For example, monitoring application latency and health can help trigger scaling or instance replacement when the underlying infrastructure appears functional but the service is not performing as expected.
External dependencies can create similar problems. An application may rely on an identity provider, DNS service, or third-party API outside the organization's direct control. If that dependency becomes unavailable, users may experience an outage even when the application's own infrastructure remains healthy.
Large-scale disruptions introduce another limitation. An architecture designed to withstand the failure of a physical host may not survive a regional network outage or the loss of an entire data center. Supporting those scenarios requires additional planning around geographic isolation, application state, data consistency, and recovery procedures.
These examples show why availability cannot be evaluated solely by counting redundant components or measuring server uptime. The more useful question is whether critical applications can continue delivering their expected functionality when different parts of the infrastructure, including services outside the organization's control, fail.
§5 Resilience Requires Testing and Operational Readiness
A highly available architecture needs to be tested under realistic conditions. Failover mechanisms may be correctly configured, but that does not guarantee they will behave as expected during an actual incident.
Regular testing helps teams understand what happens when a node becomes unavailable, a network connection fails, or a storage service experiences problems. It can also reveal whether the remaining infrastructure has enough capacity to support workloads during failover.
Operational procedures are equally important. Infrastructure teams need clear maintenance processes, reliable monitoring, documented recovery steps, and an understanding of which services depend on one another.
Capacity planning also plays a role. An environment operating close to its resource limits may struggle to absorb workloads when a component fails, even if redundancy is available.
Testing these scenarios before an incident allows organizations to identify weaknesses, improve recovery procedures, and make more informed decisions about infrastructure design.
Conclusion
High availability reduces the impact of individual component failures, but it cannot eliminate every risk to infrastructure operations. Shared dependencies, configuration errors, software failures, and insufficient capacity can still disrupt services built with redundancy in mind.
Building resilient infrastructure requires understanding failure domains, testing failover mechanisms, and maintaining the operational processes needed to respond when systems behave unexpectedly.
At VEXXHOST, we help organizations design, deploy, and manage OpenStack, Kubernetes, and Ceph environments, combining infrastructure architecture with ongoing operational expertise.
If you're evaluating the reliability of your cloud infrastructure or planning improvements to an existing environment, contact our team to discuss your requirements.
