High Availability Explained: Redundancy, Failover, Clustering, Multi-AZ and Multi-Region Architecture

Modern infrastructure is not designed around the assumption that systems will never fail.

Servers fail.

Disks fail.

Network links fail.

Applications crash.

Availability zones can experience outages.

Databases can become unavailable.

Software deployments can introduce defects.

External dependencies can stop responding.

The engineering challenge is therefore not:

How do we build infrastructure that never fails?

A more realistic question is:

How do we design infrastructure that continues delivering an acceptable service when individual components fail?

That is the foundation of High Availability, commonly abbreviated as HA.

A simple application might begin like this:

Internet
   │
   ▼
Application Server
   │
   ▼
Database

This architecture contains several obvious single points of failure.

If the application server fails:

Internet
   │
   X
Application Server

the application becomes unavailable.

If the database fails:

Application
   │
   X
Database

the application may also stop functioning.

A highly available architecture introduces redundancy, failure detection, traffic redistribution, data replication, and recovery mechanisms:

                         Internet
                            │
                            ▼
                      Load Balancer
                            │
               ┌────────────┴────────────┐
               ▼                         ▼
           App Server A              App Server B
               │                         │
               └────────────┬────────────┘
                            ▼
                     Database Layer
                       ┌────┴────┐
                       ▼         ▼
                    Primary    Replica

But redundancy alone does not automatically create high availability.

The system must also know:

• when a component has failed;
• which component should take over;
• how traffic should move;
• how application state is preserved;
• how data remains consistent;
• how split-brain conditions are prevented;
• what happens when an entire availability zone fails;
• what happens when an entire region fails;
• how the system recovers after service is restored.

This guide explains High Availability architecture from first principles, including redundancy, failover, health checks, active-active and active-passive designs, clustering, quorum, split brain, load balancing, database replication, Multi-AZ, Multi-Region, disaster recovery, RTO, RPO, observability, testing, and common HA design mistakes.

To understand the traffic-distribution layer used throughout this architecture, read Load Balancing Explained. For the proxy layer immediately before many application clusters, see Reverse Proxy Explained.

What Is High Availability?

High Availability is an architectural approach designed to keep a service operational despite failures affecting individual components.

Consider:

User
 │
 ▼
Server

If that server fails:

User
 │
 X
Server

the entire service fails.

An HA design introduces alternatives:

                    Users
                      │
                      ▼
                 Load Balancer
                      │
             ┌────────┴────────┐
             ▼                 ▼
          Server A          Server B

Now Server A can fail:

Server A ✗
Server B ✓

while traffic continues through Server B.

But this is only the beginning.

If both servers depend on one database:

              Server A      Server B
                  │            │
                  └──────┬─────┘
                         ▼
                    Database

then the database remains a single point of failure.

High availability must therefore be evaluated across the complete dependency chain, not just the web tier.

Availability vs Reliability

Availability and reliability are related but different.

Reliability asks:

How consistently does a system operate without failing?

Availability asks:

How much of the required time is the system actually usable?

A component might fail occasionally but recover so quickly that availability remains high.

Another component may fail rarely but require hours to restore.

Therefore:

Failure Frequency
+
Recovery Time

both matter.

A useful conceptual relationship is:

Availability ≈
Uptime
──────────────
Uptime + Downtime

This is why reducing recovery time can be as important as reducing failure frequency.

Understanding the Nines of Availability

Availability is often expressed using percentages such as:

99%
99.9%
99.99%
99.999%

These are sometimes called the “nines.”

The difference between them may look small, but the permitted downtime changes dramatically.

Approximate downtime over a 365-day year:

Availability	Approximate Annual Downtime
99%	3 days 15 hours 36 minutes
99.9%	8 hours 46 minutes
99.99%	52 minutes 34 seconds
99.999%	5 minutes 15 seconds

Moving from:

99.9%

to:

99.99%

is not simply a small numerical improvement.

It requires substantially stronger architecture and operations.

Availability Targets Must Reflect Business Requirements

Not every website requires five-nines availability.

A personal blog and an international payment platform have very different consequences of downtime.

The correct target depends on factors such as:

Business Impact
Revenue Loss
User Expectations
Contractual SLAs
Regulation
Operational Cost
Engineering Complexity

Higher availability usually costs more.

For example:

Single Server

is cheaper than:

Multi-AZ Cluster

which is generally cheaper and simpler than:

Active-Active Multi-Region Architecture

High availability should therefore be driven by business requirements, not architecture fashion.

What Is a Single Point of Failure?

A Single Point of Failure, or SPOF, is a component whose failure can make the entire system unavailable.

Consider:

Internet
   │
   ▼
Load Balancer
   │
   ▼
App A ───── App B
   │          │
   └────┬─────┘
        ▼
     Database

There are two application servers.

That looks redundant.

But if there is only one database:

Database ✗

both application servers may become useless.

The database is therefore a SPOF.

Finding Single Points of Failure

Do not inspect only servers.

Trace the complete request path:

User
 ↓
DNS
 ↓
CDN
 ↓
Firewall
 ↓
Load Balancer
 ↓
Reverse Proxy
 ↓
Application
 ↓
Cache
 ↓
Database
 ↓
Storage
 ↓
External API

Ask for each layer:

What happens if this component disappears right now?

If the answer is:

The entire service stops.

you have probably found an important failure dependency.

Redundancy Explained

Redundancy means having additional resources capable of taking over when another resource fails.

Instead of:

Server A

use:

Server A
Server B

Instead of:

Network Link A

use:

Link A
Link B

Instead of:

Power Supply A

enterprise systems may use redundant power paths.

But redundancy only provides value if the redundant component is actually capable of assuming the workload.

Redundancy Is Not the Same as High Availability

Consider:

Server A
Server B

but DNS points permanently to Server A.

When A fails:

Server A ✗

Server B ✓

users still attempt to reach A.

The second server exists, but failover does not occur automatically.

Therefore:

Redundancy
+
Failure Detection
+
Failover
+
Correct State
+
Operational Testing

is much closer to real HA.

Failover Explained

Failover is the process of moving workloads from a failed component to another available component.

Before failure:

Traffic
   │
   ▼
Server A

Standby:

Server B

After failure:

Server A ✗
Server B ✓

traffic becomes:

Traffic
   │
   ▼
Server B

Failover can occur at many layers:

Server Failover
Database Failover
Network Failover
Storage Failover
DNS Failover
Regional Failover

Automatic vs Manual Failover

Manual failover requires human intervention:

Failure
   │
   ▼
Alert
   │
   ▼
Engineer
   │
   ▼
Change Configuration
   │
   ▼
Restore Service

Automatic failover attempts:

Failure
   │
   ▼
Detection
   │
   ▼
Automatic Decision
   │
   ▼
Traffic Switch

Automatic failover can dramatically reduce recovery time.

But it introduces an important danger:

What if the system incorrectly decides that a healthy node has failed?

This becomes especially dangerous for databases and stateful clusters.

Active-Active Architecture

In an active-active design, multiple nodes serve production traffic simultaneously.

                  Load Balancer
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
         Server A             Server B
          ACTIVE               ACTIVE

Both systems contribute capacity.

If Server A fails:

Server A ✗
Server B ✓

Server B continues serving traffic.

Advantages include:

• better resource utilization;
• horizontal scalability;
• potentially fast failover;
• no permanently idle application server.

Active-Active Challenges

Active-active architecture requires application state to be handled correctly.

If:

User Login
    │
    ▼
Server A

stores a session only in Server A memory, the next request to:

Server B

may fail to recognize the session.

Therefore active-active applications often benefit from:

Stateless Application Servers
+
Shared Session Storage

Architecture:

                  Load Balancer
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          App A                App B
             │                   │
             └─────────┬─────────┘
                       ▼
                  Shared State

Active-Passive Architecture

In active-passive architecture, one node serves traffic while another waits.

                Traffic
                   │
                   ▼
               Server A
                ACTIVE

               Server B
                PASSIVE

If A fails:

Server A ✗

B becomes active:

                Traffic
                   │
                   ▼
               Server B
                ACTIVE

This model is frequently easier for stateful workloads because only one node may own the active service at a time.

Active-Active vs Active-Passive

Characteristic	Active-Active	Active-Passive
Multiple nodes serve traffic	Yes	Usually no
Resource utilization	Higher	Standby capacity may be idle
Horizontal scaling	Strong	Limited by active node
Failover complexity	Depends on workload	Often conceptually simpler
State management	Can be complex	Often easier
Typical use	Web/app tiers	Stateful services, HA pairs

Neither architecture is universally superior.

The correct design depends on the workload.

Health Checks and Failure Detection

Failover depends on knowing when something is unhealthy.

Consider:

Server A

The HA system needs to distinguish between:

Healthy

and:

Unhealthy

A simple TCP check might test:

Can port 443 be reached?

An application-aware check might request:

GET /health

and expect:

200 OK

For the traffic layer responsible for routing around failed backends, see Load Balancing Explained.

Failure Detection Is Not Trivial

Imagine a temporary network problem:

Health Checker
      │
      X
      │
   Server A

The server itself may be healthy.

But the health checker cannot reach it.

Should Server A immediately be declared dead?

If failover occurs too aggressively, temporary network problems can trigger unnecessary state transitions.

Therefore health checks commonly use:

Intervals
Timeouts
Failure Thresholds
Recovery Thresholds

For example:

Check every 10 seconds
3 consecutive failures → Unhealthy
2 consecutive successes → Healthy

The correct values depend on workload and recovery objectives.

False Positives and False Negatives

Failure detection can be wrong in two directions.

A false positive says:

Healthy Server → Declared Failed

A false negative says:

Failed Server → Declared Healthy

Both are dangerous.

False positives can trigger unnecessary failover.

False negatives can keep routing traffic toward a broken system.

HA engineering therefore involves balancing:

Detection Speed
        ↕
Detection Confidence

What Is a High Availability Cluster?

A cluster is a group of systems cooperating to provide a service.

Conceptually:

                 Cluster
         ┌─────────┼─────────┐
         ▼         ▼         ▼
       Node A    Node B    Node C

The nodes may coordinate:

• service ownership;
• health;
• state;
• leadership;
• storage;
• replication;
• failover.

Not every cluster is an HA cluster.

Some clusters exist primarily for:

Performance
Parallel Computing
Distributed Storage
Analytics

HA clusters specifically focus on maintaining service during failures.

Cluster Membership

A cluster must understand which nodes are participating.

For example:

Node A ✓
Node B ✓
Node C ✓

then:

Node B ?

The remaining nodes need to decide:

Has B actually failed, or is communication temporarily broken?

This leads to one of the most important concepts in distributed systems:

quorum.

What Is Quorum?

Quorum is a mechanism used by distributed systems to require agreement from a sufficient number of participants before certain decisions are accepted.

Consider three nodes:

A
B
C

A majority requires:

2 of 3

If communication splits:

A + B

can still form a majority.

C alone cannot.

Conceptually:

A ─── B      C

2 nodes      1 node
Majority     Minority

This can help prevent both sides from independently believing they should control the same resource.

Why Odd Numbers of Voting Members Are Common

Consider:

3 voting nodes

majority:

2

You can lose one node and still have a majority.

With:

5 voting nodes

majority:

3

you can lose two.

Simply adding an even-numbered voting member does not always increase the number of failures the voting system can tolerate.

The exact behavior depends on the distributed system and consensus implementation, but the majority principle is central to many clustered systems.

What Is Split Brain?

Split brain occurs when separated parts of a distributed system both believe they should act as the authoritative side.

Imagine two database nodes:

Database A
     │
     X Network Failure
     │
Database B

A thinks:

B failed.
I should remain primary.

B thinks:

A failed.
I should become primary.

Now:

Primary A
+
Primary B

may both accept writes.

This can produce divergent data.

Split Brain Example

Before partition:

A ───── B
Primary Replica

Network partition:

A       B

Bad failover logic:

A = Primary
B = Primary

Client 1 writes:

Order #100 → A

Client 2 writes:

Order #101 → B

When connectivity returns, the system may contain conflicting histories.

Preventing this is one reason quorum, consensus, fencing, and carefully designed failover logic matter.

What Is Fencing?

Fencing prevents a failed or isolated node from continuing to access shared resources after another node takes control.

Conceptually:

Node A suspected failed
        │
        ▼
Fence Node A
        │
        ▼
Ensure A cannot write
        │
        ▼
Promote Node B

Without fencing:

A may still write
+
B starts writing

which can create corruption.

Fencing mechanisms depend on the infrastructure and may involve:

• power control;
• storage access control;
• network isolation;
• cloud instance control.

Stateless Application Architecture

Application servers are much easier to make highly available when they are stateless.

Bad architecture:

App A
 │
 └── User Session

App B
 │
 └── Different Session State

Better:

                  App A
                    │
                    │
                  App B
                    │
                    ▼
              Shared Session Store

Now the load balancer can send requests to either application server.

Why Stateless Design Matters

Suppose:

User → App A

and App A fails.

If state lives only on App A:

Session lost

If state lives in a shared external service:

User → App B → Shared State

the user can continue.

Stateless application servers therefore make:

Horizontal Scaling
Failover
Rolling Deployment
Autoscaling

significantly easier.

Load Balancing and High Availability

Load balancing and high availability are closely related but not identical.

Load balancing asks:

Which healthy server should receive this request?

High availability asks:

How does the service remain available when components fail?

A typical design combines both:

                     Load Balancer
                          │
               ┌──────────┴──────────┐
               ▼                     ▼
           Server A              Server B

When both are healthy:

A ✓
B ✓

traffic can be distributed.

When A fails:

A ✗
B ✓

traffic moves to B.

For the balancing algorithms and health-check layer, see Load Balancing Explained.

The Load Balancer Can Also Be a SPOF

Consider:

Internet
   │
   ▼
Single Load Balancer
   │
   ├── Server A
   └── Server B

The backend is redundant.

But:

Load Balancer ✗

still makes the service unavailable.

Therefore the load-balancing layer itself needs an HA design.

Conceptually:

                    Traffic
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          LB Node A           LB Node B
             │                   │
             └─────────┬─────────┘
                       ▼
                  Backend Pool

Managed cloud load balancers abstract much of this infrastructure, while self-managed Nginx or HAProxy deployments require explicit redundancy planning.

Database High Availability

Databases are often the most difficult part of HA architecture because they contain state.

Application servers can often be recreated.

Database data cannot simply disappear.

A basic database architecture:

Application
    │
    ▼
Database

contains a clear SPOF.

A replicated design might use:

Application
    │
    ▼
Primary Database
    │
    ▼
Replica

If the primary fails, a replica may potentially be promoted.

Database Replication

Replication copies data between database nodes.

Conceptually:

Writes
  │
  ▼
Primary
  │
  ├── Replication → Replica A
  └── Replication → Replica B

Replicas can provide:

• redundancy;
• read scaling;
• disaster recovery options;
• failover candidates.

But replication introduces difficult questions about consistency.

For the broader database layer, see Cloud Databases Explained.

Synchronous Replication

Conceptually:

Client Write
     │
     ▼
Primary
     │
     ▼
Replica confirms
     │
     ▼
Write acknowledged

This can reduce the amount of acknowledged data that is lost when a primary fails.

But it can increase write latency because additional coordination is required.

Asynchronous Replication

Conceptually:

Client Write
     │
     ▼
Primary
     │
     ├── Acknowledge Client
     │
     └── Replicate Later

This can provide lower write latency.

But if the primary fails before replication completes:

Latest Data

may not exist on the replica.

This introduces a trade-off between:

Performance
Consistency
Durability
Availability

depending on the database architecture.

Database Failover

A simplified database failover sequence might be:

Primary A
   │
   X
Failure
   │
   ▼
Failure Detection
   │
   ▼
Select Replica B
   │
   ▼
Promote B
   │
   ▼
Redirect Applications

Each step can fail.

For example:

• failure may be detected incorrectly;
• B may not contain the latest data;
• applications may still connect to A;
• DNS or connection pools may retain stale endpoints;
• A may return and create split brain.

Database HA therefore requires much more than maintaining a replica.

Multi-AZ Architecture

Cloud platforms divide regions into separate availability zones or similar failure domains.

Instead of:

Region
  │
  ▼
Zone A
  │
  ├── App 1
  └── App 2

a stronger architecture spreads workloads:

                         Region
                           │
               ┌───────────┴───────────┐
               ▼                       ▼
             Zone A                  Zone B
               │                       │
           ┌───┴───┐               ┌───┴───┐
           ▼       ▼               ▼       ▼
         App 1   App 2           App 3   App 4

A load balancer can distribute traffic across both zones.

If Zone A fails:

Zone A ✗
Zone B ✓

Zone B can continue serving traffic if it has sufficient capacity and all required dependencies remain available.

Multi-AZ Is More Than Multiple Application Servers

Suppose:

App A → Zone A
App B → Zone B

but:

Database → Zone A only

Then:

Zone A failure

still destroys the service.

A real Multi-AZ review must inspect:

Application
Database
Cache
Queue
Storage
Networking
Secrets
Identity
Load Balancing

The complete application dependency graph must tolerate the intended failure domain.

Capacity During Failure

Suppose normal traffic requires:

100 units of capacity

and each availability zone provides:

50 units

Under normal conditions:

Zone A = 50
Zone B = 50

If Zone A fails:

Zone B = 50
Required = 100

The application remains technically online but severely overloaded.

Therefore HA capacity planning asks:

Can the surviving infrastructure handle the failed component’s workload?

Redundancy without sufficient failover capacity is incomplete.

N+1 Redundancy

A common concept is:

N + 1

where N represents the resources required for normal operation and +1 provides additional failure capacity.

For example:

3 servers required
+
1 additional server
=
4 total

If one fails:

3 remain

and the system can continue at required capacity.

Other redundancy models exist depending on the criticality and infrastructure.

Multi-Region Architecture

Multi-AZ protects primarily against failures within a region.

Multi-Region architecture adds geographic redundancy.

                       Global Users
                            │
                            ▼
                     Global Routing
                            │
              ┌─────────────┴─────────────┐
              ▼                           ▼
          Region A                    Region B
              │                           │
         ┌────┴────┐                 ┌────┴────┐
         ▼         ▼                 ▼         ▼
       Zone A1   Zone A2           Zone B1   Zone B2

If Region A becomes unavailable:

Region A ✗
Region B ✓

global routing can direct traffic toward Region B.

This sounds straightforward.

The data layer makes it much harder.

Multi-Region Data

Suppose users write data in Region A:

User
 │
 ▼
Region A Database

If Region A fails, Region B needs sufficiently recent data.

Therefore:

Region A
   │
   ▼
Replication
   │
   ▼
Region B

becomes essential.

But geographic distance introduces:

Latency
Replication Delay
Consistency Challenges
Conflict Resolution
Cost

Multi-Region application servers are easy compared with Multi-Region state.

Active-Passive Multi-Region

One design uses:

Region A = Active
Region B = Standby

Normal traffic:

Users
  │
  ▼
Region A

During disaster:

Region A ✗
     │
     ▼
Region B becomes active

This can simplify data ownership compared with accepting writes in multiple regions simultaneously.

However, failover time may be longer.

Active-Active Multi-Region

Both regions serve traffic:

                    Global Traffic
                         │
               ┌─────────┴─────────┐
               ▼                   ▼
           Region A             Region B
            ACTIVE               ACTIVE

This can improve:

• geographic latency;
• capacity utilization;
• resilience.

But state becomes significantly more difficult.

If both regions accept writes:

Write A → Region A
Write B → Region B

the data architecture must define:

Replication
Ordering
Conflict Resolution
Consistency
Failure Semantics

Active-active Multi-Region is therefore not merely “deploy the application twice.”

DNS Failover

DNS can participate in regional failover.

Normal:

example.com
     │
     ▼
Region A

After failure:

example.com
     │
     ▼
Region B

But DNS records are cached.

If a resolver cached Region A for:

300 seconds

it may continue using the old answer until the record expires.

Therefore DNS failover is influenced by:

TTL
Resolver Behavior
Health Detection
Client Caching

For DNS fundamentals, see DNS Explained.

Global Traffic Management

More advanced architectures use a global traffic layer:

                     Global Traffic Manager
                              │
                 ┌────────────┴────────────┐
                 ▼                         ▼
              Region A                  Region B

Routing can potentially consider:

Health
Latency
Geography
Capacity
Policy

This layer becomes important in large Multi-Region systems.

Storage High Availability

Applications may depend on persistent storage.

A server can be redundant while its storage is not.

Bad architecture:

Server A ─┐
          ├── Single Storage Device
Server B ─┘

Storage failure:

Storage ✗

affects both servers.

Storage HA can involve:

• disk redundancy;
• replicated storage;
• distributed storage;
• storage clusters;
• cloud-managed replicated storage.

For underlying storage concepts, see Servers and Infrastructure Explained.

RAID Is Not Complete High Availability

RAID can protect against certain disk failures.

For example:

Disk A
Disk B
Disk C
Disk D

may provide redundancy depending on the RAID level.

But RAID does not automatically protect against:

Server Failure
Controller Failure
Data Center Failure
Accidental Deletion
Malware
Corruption
Application Bugs

RAID is one storage-resilience mechanism, not a complete HA or backup strategy.

High Availability Is Not Backup

This distinction is critical.

High availability asks:

Can the service continue operating when infrastructure fails?

Backup asks:

Can data be restored after loss or corruption?

Suppose database corruption is replicated:

Primary
  │
  ▼
Corrupted Data
  │
  ▼
Replica
  │
  ▼
Same Corruption

Replication worked perfectly.

High availability may still work.

But the data is damaged.

Backups solve a different problem.

Network High Availability

Applications depend on network paths.

Consider:

Internet
   │
   ▼
Router
   │
   ▼
Application

The router is a SPOF.

A more resilient design may involve:

                    Upstream Networks
                     │             │
                     ▼             ▼
                  Router A      Router B
                     │             │
                     └──────┬──────┘
                            ▼
                        Network

Redundancy can extend to:

Routers
Switches
Links
NICs
Firewalls
Internet Providers

depending on availability requirements.

Redundant Links Need Independent Paths

Two cables are not necessarily redundant.

Suppose:

Link A ─┐
        ├── Same switch
Link B ─┘

The switch remains a SPOF.

Similarly:

ISP A
ISP B

may appear redundant but could share the same upstream fiber path.

True redundancy requires understanding physical and logical failure domains.

High Availability vs Disaster Recovery

These concepts overlap but have different objectives.

High Availability attempts to maintain service during common component failures.

Disaster Recovery focuses on restoring service after a major disruption.

HA example:

Server A fails
     │
     ▼
Server B continues

DR example:

Entire Primary Region Unavailable
          │
          ▼
Recover in Secondary Region

The boundary depends on architecture.

A Multi-Region active-active design can make what would otherwise be a disaster-recovery event look like normal failover.

What Is RTO?

RTO means:

Recovery Time Objective.

It answers:

How long can the service remain unavailable after a disruption?

Example:

RTO = 1 hour

means the recovery architecture should aim to restore service within that objective.

What Is RPO?

RPO means:

Recovery Point Objective.

It answers:

How much recent data loss can the organization tolerate?

Example:

RPO = 15 minutes

means recovery may tolerate restoring to a point no more than roughly 15 minutes before the incident, depending on the implemented strategy.

RTO concerns time.

RPO concerns data.

RTO vs RPO

Incident
   │
   ▼
Service unavailable
   │
   │──── RTO ────► Service restored

while:

Last Recoverable Data
         │
         │──── RPO ────► Incident

These objectives strongly influence architecture.

Very low RTO and RPO generally require:

Replication
Automation
Redundancy
Continuous Monitoring
Fast Failover
Regular Testing

and therefore greater cost and complexity.

High Availability vs Fault Tolerance

High availability and fault tolerance are also not identical.

High availability may tolerate a short interruption while failover occurs:

Failure
  │
  ▼
5–30 seconds disruption
  │
  ▼
Failover

Fault-tolerant systems aim for even stronger continuity where individual component failures may have little or no externally visible interruption.

Fault tolerance generally requires greater duplication and coordination.

Observability Is Part of High Availability

A system cannot recover intelligently from failures it cannot detect.

Production HA requires visibility into:

Health
Latency
Error Rates
Capacity
Replication
Queue Depth
Storage
Network Connectivity
Database State

A basic monitoring architecture:

Infrastructure
     │
     ├── Metrics
     ├── Logs
     └── Traces
          │
          ▼
      Monitoring
          │
          ▼
        Alerts

Availability is partly an infrastructure problem and partly an operations problem.

Monitor Degraded Redundancy

Suppose:

Server A ✓
Server B ✓
Server C ✓

becomes:

Server A ✓
Server B ✗
Server C ✓

The application is still available.

A simplistic monitor says:

Website = UP

But the architecture has lost redundancy.

A better monitoring system alerts:

Service available
BUT
redundancy degraded

This allows engineers to repair the system before another failure causes an outage.

Monitor Replication Lag

For replicated databases:

Primary
   │
   ▼
Replica

a replica may technically be online but significantly behind.

Example:

Replication Lag = 30 minutes

Promoting it could produce significant data loss.

Therefore database health should include more than:

Server responds to ping

It may need to consider:

Replication State
Replication Lag
Role
Storage Capacity
Transaction Health

Test Failover

An HA architecture that has never experienced a controlled failure is largely theoretical.

Test scenarios such as:

Stop Application Server
Kill Proxy Process
Disconnect Database Replica
Block Network Path
Remove Availability Zone Capacity
Simulate Dependency Failure

Then observe:

Did failure detection work?

How long did detection take?

Did traffic move?

Did sessions survive?

Was data lost?

Did alerts fire?

Did the failed node return safely?

This converts assumptions into evidence.

Failure Injection

Advanced engineering teams deliberately introduce controlled failures.

Conceptually:

Healthy System
     │
     ▼
Inject Failure
     │
     ▼
Observe Behavior
     │
     ▼
Find Weakness
     │
     ▼
Improve Architecture

The purpose is not to break production recklessly.

It is to discover failure assumptions before real incidents do.

Test Recovery, Not Just Failover

Suppose:

Primary A fails
Replica B promoted

Failover succeeds.

Later A returns.

What happens?

Possible bad outcome:

A thinks it is Primary
B thinks it is Primary

Therefore HA testing must include:

Failure
Failover
Stabilization
Recovery
Reintegration

The recovery path can be as dangerous as the initial failure.

Deployment Availability

Infrastructure can be highly redundant and still experience outages during deployment.

Bad deployment:

Stop App A
Stop App B
Deploy
Start App A
Start App B

produces downtime.

A rolling deployment updates instances gradually:

A → New Version
B → Old Version
C → Old Version

then:

A → New
B → New
C → Old

and finally:

A → New
B → New
C → New

Healthy capacity remains available throughout the deployment.

Blue-Green Deployment

Another pattern uses two environments.

Blue = Current Production
Green = New Version

Before switch:

Users
  │
  ▼
Blue

After validation:

Users
  │
  ▼
Green

If serious problems appear, traffic may be switched back to Blue.

This reduces some deployment risks but requires sufficient duplicate infrastructure and careful database compatibility.

Canary Deployment

A canary rollout exposes a small amount of traffic to a new version.

Version 1 → 95%
Version 2 → 5%

Monitor:

Errors
Latency
Business Metrics

If healthy:

Version 2 → 25%

then:

Version 2 → 50%

and eventually:

Version 2 → 100%

Load balancing and observability therefore contribute directly to deployment availability.

Autoscaling and High Availability

Autoscaling can help maintain capacity when demand changes.

Architecture:

Traffic ↑
   │
   ▼
Load ↑
   │
   ▼
Autoscaler
   │
   ▼
Add Instances

But autoscaling is not automatically HA.

If all instances are launched into:

One Availability Zone

a zone failure can still remove all capacity.

Autoscaling policies should align with failure-domain strategy.

Infrastructure as Code and HA

Manual infrastructure creates recovery risk.

Suppose a region fails and engineers need to recreate:

Networks
Subnets
Load Balancers
Security Rules
Servers
Routing
Monitoring

from memory.

Recovery becomes slow and error-prone.

Infrastructure as Code allows:

Version-Controlled Definition
           │
           ▼
       Automation
           │
           ▼
     Infrastructure

This can significantly improve reproducibility and disaster recovery.

See Infrastructure as Code Explained for the complete workflow.

High Availability Anti-Patterns

Two Servers, One Database

App A ─┐
       ├── Database
App B ─┘

The application tier is redundant.

The service is not.

Two Servers on One Host

Physical Host
    │
    ├── VM A
    └── VM B

Host failure removes both.

Two Zones with Insufficient Capacity

Zone A = 50%
Zone B = 50%

If one fails, the survivor cannot handle full production load.

Replica Without Tested Failover

A replica that has never been promoted successfully should not automatically be considered a complete HA solution.

HA Without Monitoring

Primary failed
Replica took over

may appear successful.

But if nobody notices that redundancy is now gone:

Replica is now the only working node

the next failure becomes catastrophic.

Replication Treated as Backup

Replication can reproduce corruption, deletion, or malicious changes.

Backups remain necessary.

Multi-Region Application with Single-Region Data

Region A App ─┐
              ├── Database in Region A
Region B App ─┘

Region B exists, but Region A database failure can still disable both.

Failover That Has Never Been Tested

Documentation saying:

Region B will take over

is not the same as demonstrating that Region B actually can.

Production High Availability Architecture

A mature architecture might look like:

                              Global Users
                                   │
                                   ▼
                            Global DNS / Traffic
                                   │
                       ┌───────────┴───────────┐
                       ▼                       ▼
                    Region A                Region B
                       │                       │
                   CDN / Edge              CDN / Edge
                       │                       │
                       ▼                       ▼
                 Load Balancer            Load Balancer
                       │                       │
                ┌──────┴──────┐         ┌──────┴──────┐
                ▼             ▼         ▼             ▼
             Zone A1       Zone A2    Zone B1       Zone B2
                │             │         │             │
             App Nodes     App Nodes  App Nodes     App Nodes
                │             │         │             │
                └──────┬──────┘         └──────┬──────┘
                       ▼                       ▼
                  Data Layer  ◄──────────► Data Layer
                       │                       │
                       └──────────┬────────────┘
                                  ▼
                           Backup / Recovery

This architecture has multiple layers of resilience.

But it is not automatically highly available.

Engineers must still define:

Routing Behavior
Replication Semantics
Failure Detection
Failover Ownership
Capacity
Consistency
RTO
RPO
Recovery Procedures

Architecture diagrams do not guarantee operational resilience.

A Practical HA Design Process

Start with the service rather than technology.

Step 1: Define the Availability Requirement

Example:

Target availability: 99.95%

Then define what downtime actually means for the service.

Step 2: Define RTO and RPO

Example:

RTO = 15 minutes
RPO = 5 minutes

These numbers immediately constrain architecture options.

Step 3: Map Dependencies

Create:

User
 ↓
DNS
 ↓
CDN
 ↓
Load Balancer
 ↓
Application
 ↓
Redis
 ↓
Database
 ↓
Storage

Include external services.

Step 4: Identify Failure Domains

Ask what happens when you lose:

One Process
One VM
One Physical Host
One Switch
One Availability Zone
One Region
One Cloud Dependency

Step 5: Remove Critical SPOFs

Add redundancy where justified.

But ensure redundant components do not share the same hidden failure domain.

Step 6: Design Failure Detection

Define:

What is checked?
How often?
What counts as failure?
Who makes the decision?

Step 7: Define Failover

Document:

What takes over?
How?
How quickly?
Where does state come from?

Step 8: Design Recovery

Define what happens when the failed component returns.

Do not stop at failover.

Step 9: Monitor Degradation

Alert when redundancy is lost even if users are not yet affected.

Step 10: Test Everything

Simulate the failures the architecture claims to survive.

High Availability Checklist

Before calling an architecture highly available, verify:

Application

• Are there multiple instances?
• Are they distributed across failure domains?
• Is application state externalized where appropriate?
• Can deployments occur without full downtime?

Load Balancing

• Is the load-balancing layer redundant?
• Are health checks application-aware?
• Are unhealthy instances removed automatically?
• Is failover time acceptable?

Database

• Is the database replicated?
• Is replication health monitored?
• Is failover tested?
• Is split brain prevented?
• Are backups separate from replication?

Storage

• Is storage redundant?
• Is there a backup strategy?
• Can data be restored?
• Has restoration been tested?

Networking

• Are critical links redundant?
• Are routers/firewalls SPOFs?
• Are redundant paths actually independent?

Cloud

• Are workloads distributed across availability zones?
• Can surviving zones handle production load?
• Is regional failure part of the threat model?

Operations

• Are logs centralized?
• Are metrics monitored?
• Are alerts actionable?
• Are runbooks documented?
• Is failure testing performed?

Recovery

• Is RTO defined?
• Is RPO defined?
• Is disaster recovery tested?
• Is failback documented?

Frequently Asked Questions

What is high availability?

High availability is an architectural approach designed to keep services operational despite failures affecting individual infrastructure components.

What is a single point of failure?

A single point of failure is a component whose failure can cause the entire service or a critical function to become unavailable.

Is redundancy the same as high availability?

No. Redundancy provides alternative resources. High availability also requires failure detection, failover, correct state management, adequate capacity, monitoring, and tested recovery.

What is failover?

Failover is the process of moving service or traffic from a failed component to a healthy alternative.

What is active-active architecture?

Active-active architecture uses multiple active systems simultaneously to serve production traffic.

What is active-passive architecture?

Active-passive architecture uses one active system while another remains ready to take over after failure.

What is quorum?

Quorum is a mechanism in distributed systems that requires sufficient agreement among participants before certain cluster decisions are accepted.

What is split brain?

Split brain occurs when separated parts of a distributed system independently believe they are authoritative, potentially resulting in conflicting operations or data.

What is fencing?

Fencing prevents an isolated or failed node from continuing to access resources after another node has taken ownership.

What is Multi-AZ?

Multi-AZ architecture distributes workloads across separate availability zones or equivalent failure domains within a cloud region.

What is Multi-Region architecture?

Multi-Region architecture deploys infrastructure across multiple geographic cloud regions to provide geographic resilience, lower latency, disaster recovery, or active-active service delivery.

What is RTO?

Recovery Time Objective defines the targeted maximum time for restoring a service after a disruption.

What is RPO?

Recovery Point Objective defines the targeted maximum amount of recent data loss measured in time.

Is database replication a backup?

No. Replication improves availability and can support failover, but unwanted changes or corruption may also replicate. Independent backups are still required.

Does a load balancer automatically provide high availability?

No. It can improve backend availability, but the load balancer, database, storage, networking, and other dependencies must also be designed for failure.

Is Multi-AZ enough for high availability?

It can protect against important zone-level failures, but only if all critical dependencies are appropriately distributed and surviving capacity is sufficient.

Is Multi-Region always necessary?

No. Multi-Region infrastructure adds significant cost and complexity. It should be justified by business availability, latency, regulatory, or disaster-recovery requirements.

Final Thoughts

High Availability is frequently misunderstood as:

Add another server

Real HA architecture is closer to:

Redundancy
     +
Failure Detection
     +
Failover
     +
State Management
     +
Replication
     +
Capacity Planning
     +
Observability
     +
Recovery
     +
Testing

The most important change in mindset is accepting that:

Failures are inevitable.

A strong architecture therefore does not ask:

Will this component fail?

It asks:

What happens when it fails?

For every critical component:

DNS
CDN
Load Balancer
Reverse Proxy
Application
Database
Cache
Queue
Storage
Network

you should know:

How failure is detected.

What takes over.

How long takeover requires.

Whether data can be lost.

Whether surviving infrastructure has enough capacity.

How engineers know redundancy has been degraded.

How the failed component safely rejoins the system.

This transforms a simple architecture:

Internet
   │
   ▼
Server
   │
   ▼
Database

into a resilient service architecture:

                          Global Users
                               │
                               ▼
                         Traffic Layer
                               │
                               ▼
                         Load Balancing
                               │
                    ┌──────────┴──────────┐
                    ▼                     ▼
               Failure Domain A     Failure Domain B
                    │                     │
                 App Nodes             App Nodes
                    │                     │
                    └──────────┬──────────┘
                               ▼
                         Resilient Data
                               │
                               ▼
                        Backup / Recovery

But the objective is not maximum complexity.

The objective is the minimum architecture capable of meeting the required availability target.

A small website may need:

Reliable hosting
+
Backups
+
Monitoring

A business-critical application may require:

Multi-AZ
+
Load Balancing
+
Database Failover
+
Automated Recovery

A globally critical platform may justify:

Multi-Region
+
Global Traffic Management
+
Distributed Data
+
Automated Failover
+
Continuous Failure Testing

Each additional layer should exist because it mitigates a defined failure scenario.

Continue with Load Balancing Explained to understand how traffic is redistributed among healthy systems.

Read Reverse Proxy Explained for the application traffic gateway layer.

Use CDN Caching Explained to understand how cached responses can reduce dependency on origin infrastructure.

For the underlying compute and storage architecture, continue with Servers and Infrastructure Explained.

For cloud network design, read Cloud Networking Explained.

And for rebuilding infrastructure predictably after failure, continue with Infrastructure as Code Explained.

The CorgiNetwork infrastructure path now becomes:

DNS → Edge → CDN Caching → Reverse Proxy → Load Balancing → High Availability → Clustering → Disaster Recovery → Cloud Networking → Infrastructure as Code.