Table of Contents

Server Clustering Explained: HA Clusters, Quorum, Failover, Fencing and Distributed Architecture

A single server is easy to understand.

It has its own CPU, memory, storage, operating system, network interfaces, applications, and processes.

But it also creates an obvious architectural problem:

Users
  │
  ▼
Server
  │
  X
Failure

If that server provides a critical service, the failure of one machine can become the failure of the entire application.

Server clustering addresses this problem by allowing multiple systems—called nodes—to cooperate as part of a larger architecture.

                    Cluster
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
        Node A       Node B       Node C

But clustering is much more than connecting several servers.

A production cluster must answer difficult questions:

Which nodes are currently alive?

Which node owns a resource?

Who becomes leader?

What happens when nodes cannot communicate?

How is state synchronized?

How many nodes must agree before a decision is valid?

How do we prevent two nodes from becoming primary?

How does a failed node safely rejoin?

Where is application data stored?

What happens when the network itself is partitioned?

These questions introduce some of the most important concepts in modern infrastructure engineering:

cluster membership, heartbeats, quorum, consensus, leader election, failover, fencing, split brain, shared storage, shared-nothing architecture, replication, service discovery, and distributed state.

Clustering concepts appear throughout modern infrastructure.

Traditional Linux HA systems can use technologies such as Pacemaker and Corosync, while container platforms such as Kubernetes apply clustering principles to orchestrated workloads at a much larger scale.

Database systems, virtualization platforms, storage systems, container orchestrators, and distributed applications all rely on variations of the same fundamental problem:

How can multiple machines cooperate safely while some machines or network paths may fail?

This guide explains server clustering from first principles through production architecture.

For the availability concepts underneath this topic, start with High Availability Explained. For traffic distribution across application servers, read Load Balancing Explained.

What Is a Server Cluster?
Why Server Clusters Exist
Cluster vs Multiple Independent Servers
Cluster Nodes
Types of Server Clusters
High Availability Clusters
Load-Balancing Clusters
Compute Clusters
Storage Clusters
Database Clusters
Cluster Membership
Heartbeats
Failure Detection
Leader Election
Quorum
Consensus
Split Brain
Fencing
STONITH
Shared-Storage Clusters
Shared-Nothing Clusters
Cluster Failover
Pacemaker and Corosync
Kubernetes Clustering
Database Clustering
Load Balancers vs Clusters
Cluster Networking
Cluster Security
Observability
Troubleshooting Cluster Failures
Production Architecture
Best Practices
Frequently Asked Questions

What Is a Server Cluster?

A server cluster is a group of independent servers that cooperate to provide one or more services.

Each server is commonly called a:

Node

A three-node cluster might look like:

               Cluster
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
     Node A     Node B     Node C

The nodes communicate through a network and exchange information about:

Health
Membership
State
Resources
Leadership
Services

Depending on the architecture, they may also share or replicate data.

To external users, several nodes may appear as one logical service:

Users
  │
  ▼
Logical Service
  │
  ▼
Cluster
  │
 ┌┼┐
 ▼▼▼
 A B C

The implementation behind that service can vary dramatically depending on the type of cluster.

Why Server Clusters Exist

Clustering usually addresses one or more of four broad requirements:

Availability
Scalability
Performance
Data Resilience

A single server has finite resources:

CPU
RAM
Storage
Network Capacity

and creates a failure boundary.

Clustering allows infrastructure designers to distribute workloads or maintain redundant service capacity across several machines.

Cluster vs Multiple Independent Servers

Three servers do not automatically form a cluster.

Consider:

Server A

Server B

Server C

If they do not coordinate with each other, they are simply independent systems.

A cluster introduces some form of coordination:

Node A ◄────► Node B
  ▲             ▲
  │             │
  └────► Node C ◄────┘

Coordination may include:

Membership
Health Monitoring
Leader Election
Resource Ownership
Replication
Distributed Scheduling
Failover

The exact mechanism depends on the cluster.

What Is a Cluster Node?

A node is an individual participant in the cluster.

It may be:

Physical Server
Virtual Machine
Cloud Instance
Container Host

depending on architecture.

A node typically has:

Operating System
CPU
Memory
Network Interfaces
Cluster Software
Applications or Services

The cluster software allows nodes to coordinate.

Cluster Identity

Nodes require unique identities.

Conceptually:

cluster-01
cluster-02
cluster-03

The cluster must distinguish between:

Node A is temporarily unreachable

and:

A new node has joined

or:

Node A has restarted

Identity becomes especially important when making distributed decisions.

Types of Server Clusters

“Cluster” is a broad term.

Different clusters solve different problems.

Major categories include:

High Availability Cluster
Load-Balancing Cluster
Compute Cluster
Storage Cluster
Database Cluster
Container Cluster

Some platforms combine several categories.

High Availability Cluster

An HA cluster focuses on keeping a service available when a node fails.

Basic architecture:

              Cluster
                 │
       ┌─────────┴─────────┐
       ▼                   ▼
    Node A              Node B
    ACTIVE              STANDBY

Normal operation:

Service → Node A

Failure:

Node A ✗

Failover:

Service → Node B

The objective is continuity.

For the broader architecture surrounding this model, see High Availability Explained.

Active-Passive Cluster

A common HA design uses:

Node A = Active
Node B = Passive

The passive node monitors cluster state and remains prepared to assume resources.

Conceptually:

                  Virtual Service
                       │
                       ▼
                    Node A
                     ACTIVE

                    Node B
                    PASSIVE

After failure:

Node A ✗

the cluster can attempt:

Stop / Fence A
      │
      ▼
Promote B
      │
      ▼
Move Service

The fencing step can be extremely important for stateful workloads.

Active-Active Cluster

An active-active cluster uses multiple nodes simultaneously.

                     Service
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
           Node A              Node B
           ACTIVE              ACTIVE

This can improve:

Resource Utilization
Scalability
Availability

but requires careful state management.

Active-active application servers are relatively straightforward when stateless.

Active-active databases can be considerably more difficult because multiple nodes may modify shared data.

Load-Balancing Cluster

A load-balancing cluster distributes client traffic across multiple servers.

                    Clients
                       │
                       ▼
                 Load Balancer
                       │
             ┌─────────┼─────────┐
             ▼         ▼         ▼
           Node A    Node B    Node C

All nodes may be active.

The load balancer chooses a destination using algorithms such as:

Round Robin
Least Connections
Weighted Distribution
Hashing

For those algorithms and Layer 4 vs Layer 7 routing, see Load Balancing Explained.

Compute Cluster

A compute cluster focuses primarily on processing large workloads.

Instead of one machine executing a large computational task:

Large Job
   │
   ▼
Server

the workload can be distributed:

                     Large Job
                        │
                        ▼
                     Scheduler
                        │
             ┌──────────┼──────────┐
             ▼          ▼          ▼
         Compute A  Compute B  Compute C

Compute clusters are common in:

Scientific Computing
Simulation
Machine Learning
Rendering
High-Performance Computing

Their primary objective may be performance rather than service failover.

Storage Cluster

A storage cluster distributes data or storage services across multiple nodes.

                    Storage Service
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
          Storage A   Storage B   Storage C

Data may be:

Replicated
Erasure-Coded
Partitioned

depending on the system.

Storage clustering attempts to avoid relying on one physical storage system.

Database Cluster

Database clustering coordinates multiple database nodes.

A simple primary-replica design:

                    Applications
                         │
                         ▼
                       Primary
                      Database
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
           Replica A             Replica B

A distributed database may instead spread ownership across several nodes.

Database clustering introduces difficult problems involving:

Consistency
Replication
Transactions
Leader Election
Quorum
Failover
Conflict Resolution

For the broader cloud database layer, see Cloud Databases Explained.

Cluster Membership

Before a cluster can make decisions, it must know:

Which nodes are members right now?

Suppose the expected cluster is:

A
B
C

Normal state:

A ✓
B ✓
C ✓

Then B disappears:

A ✓
B ?
C ✓

What happened?

Possible explanations include:

B crashed.

B lost network connectivity.

A and C lost connectivity to B.

A network switch failed.

The cluster network partitioned.

B is overloaded and responding slowly.

The cluster cannot directly observe “failure” as an abstract fact.

It observes symptoms.

Heartbeats Explained

Clusters frequently exchange periodic messages called heartbeats.

Conceptually:

Node A ─── heartbeat ───► Node B
Node B ─── heartbeat ───► Node A

The message effectively says:

I am alive.

With three nodes:

        A
       / \
      /   \
     B─────C

nodes continuously exchange membership or health information depending on the cluster protocol.

Missed Heartbeats

Suppose:

A → B heartbeat ✓
A → B heartbeat ✓
A → B heartbeat ✗
A → B heartbeat ✗
A → B heartbeat ✗

The cluster may conclude:

B is unavailable.

But this conclusion is probabilistic.

B might still be running.

The network may simply be broken.

This distinction is fundamental to distributed systems.

Failure Detection

Cluster failure detection usually involves parameters such as:

Heartbeat Interval
Timeout
Failure Threshold
Membership Rules

Aggressive settings detect failures quickly.

But they can increase false positives.

Conservative settings reduce false positives.

But they increase failover time.

Conceptually:

Fast Detection
     │
     ▼
Potential False Failover

versus:

Slow Detection
     │
     ▼
Longer Service Disruption

There is no universal timeout that is correct for every cluster.

Network Partition

One of the hardest failures is not a server crash.

It is a network partition.

Before:

A ───── B ───── C

After:

A ───── B       C

All three nodes may still be running.

But C cannot communicate with A and B.

From C’s perspective:

A and B disappeared.

From A and B’s perspective:

C disappeared.

Which side is allowed to continue?

That is where quorum becomes important.

Quorum Explained

Quorum defines how much agreement is required before a cluster is allowed to perform certain operations.

In a three-voter system:

A
B
C

a majority is:

If the cluster partitions:

A + B     C

then:

A + B = majority
C = minority

The majority side can remain authoritative according to the system’s rules.

The minority side may need to stop serving certain operations.

Why Quorum Matters

Without quorum, consider:

A       B

Network connection fails.

A thinks:

B failed.

B thinks:

A failed.

Both may attempt to become primary.

Now:

A = Primary
B = Primary

This is split brain.

Quorum provides a mechanism to prevent isolated minorities from independently taking ownership.

Majority Calculation

For a simple majority system:

Majority = floor(N / 2) + 1

For three voters:

floor(3/2) + 1 = 2

For five:

floor(5/2) + 1 = 3

For seven:

floor(7/2) + 1 = 4

This explains why odd numbers of voting participants frequently appear in consensus architectures.

Why Two-Node Clusters Are Difficult

Consider:

Node A
Node B

The network between them fails.

Now:

A       B

A cannot determine whether:

B crashed

or:

the network failed.

B has the same problem.

There is no natural majority.

This is why two-node HA designs often require an additional mechanism such as:

Quorum Device
Witness
Fencing
External Tie-Breaker

depending on the cluster technology.

Witness Node

A witness can provide an additional vote or arbitration signal.

Conceptually:

        Witness
        /     \
       /       \
      A         B

If A and B lose communication but A can reach the witness:

A + Witness

may form the authoritative side.

The exact behavior is implementation-specific, so a witness should not be treated as a generic replacement for understanding the cluster’s quorum model.

Consensus Explained

Quorum and consensus are related but not identical.

Quorum answers questions about sufficient participation.

Consensus protocols coordinate agreement about state or decisions among distributed nodes.

The cluster may need to agree on:

Who is leader?

Which configuration version is current?

Which transaction was committed?

Which node owns a resource?

Which log entry comes next?

Distributed consensus protocols are designed to provide safe agreement despite failures within defined assumptions.

Leader Election

Some clusters designate one node as leader.

          Cluster
             │
     ┌───────┼───────┐
     ▼       ▼       ▼
     A       B       C
  LEADER  FOLLOWER FOLLOWER

The leader may coordinate:

Writes
Scheduling
Resource Ownership
Metadata
Configuration Changes

If the leader fails:

A ✗

the remaining nodes may elect another leader:

B = New Leader

Leader election must avoid having two leaders simultaneously when the system requires only one.

Split Brain Explained

Split brain occurs when separated parts of a cluster act as if they independently own the same authoritative role or resource.

Before partition:

A ───── B
Primary   Standby

Network failure:

A       B

Bad outcome:

A = Primary
B = Primary

Both may now modify data.

For a database:

Client 1 → A → UPDATE balance
Client 2 → B → UPDATE balance

The histories diverge.

When communication returns, there may be no trivial way to determine which history is correct.

Why Split Brain Is Dangerous

Split brain can produce:

Data Corruption
Conflicting Transactions
Duplicate Jobs
Duplicate IP Ownership
Filesystem Corruption
Inconsistent Application State

For shared storage, two nodes writing simultaneously without proper coordination can be catastrophic.

Therefore HA cluster design often prioritizes:

Preventing unsafe dual ownership is more important than keeping every partition writable.

This is one of the central trade-offs in distributed infrastructure.

Fencing Explained

Fencing ensures that a node believed to be unsafe cannot continue accessing protected resources.

Suppose Node A stops responding.

The cluster wants Node B to take over.

Unsafe sequence:

A appears failed
     │
     ▼
Immediately start service on B

A may still be running.

Now both can own the resource.

Safer sequence:

A appears failed
     │
     ▼
Fence A
     │
     ▼
Confirm A cannot access resource
     │
     ▼
Start service on B

Fencing creates stronger guarantees around ownership.

STONITH

In traditional Linux HA terminology, fencing is often associated with STONITH:

Shoot The Other Node In The Head

The name is intentionally memorable.

Its purpose is to force an uncertain node into a known safe state.

Possible mechanisms include:

Power Off Node
Reboot Node
Disable Storage Access
Cloud API Shutdown
Hypervisor Control

Pacemaker documentation treats fencing as a critical mechanism for protecting shared resources and avoiding unsafe concurrent access.

Why “Ping Failed” Is Not Enough

Suppose:

Cluster Network
      X
    Node A

A cannot answer heartbeats.

But:

Storage Network
      │
      ▼
    Node A

still works.

If Node B takes over the same shared disk while A continues writing:

Node A ───► Shared Disk ◄─── Node B

corruption can occur.

This demonstrates why:

Failure Detection

and:

Fencing

solve different problems.

Shared-Storage Cluster

Traditional HA clusters often use shared storage.

Architecture:

               Node A
                 │
                 │
               Node B
                 │
                 ▼
            Shared Storage

Only the authorized node may mount or modify certain resources at a particular time, depending on the filesystem and architecture.

Failover might involve:

Stop Service on A
      │
      ▼
Fence A
      │
      ▼
Move Storage Ownership
      │
      ▼
Mount on B
      │
      ▼
Start Service on B

This requires careful ordering.

Shared-Nothing Architecture

A shared-nothing cluster avoids a single shared storage subsystem.

Each node owns local resources:

Node A
 ├── CPU
 ├── RAM
 └── Storage A

Node B
 ├── CPU
 ├── RAM
 └── Storage B

Node C
 ├── CPU
 ├── RAM
 └── Storage C

Data is distributed or replicated between nodes.

Conceptually:

Storage A
   ⇅
Storage B
   ⇅
Storage C

Many modern distributed systems use variations of shared-nothing architecture because it can scale horizontally and reduce dependence on centralized storage.

Shared Storage vs Shared Nothing

Characteristic	Shared Storage	Shared Nothing
Storage location	Common subsystem	Local/distributed
Data coordination	Shared access	Replication/distribution
Central storage dependency	Possible	Reduced
Horizontal scaling	Can be limited	Often strong
Complexity	Storage coordination	Distributed data coordination
Split-brain risk	Shared-resource corruption	Divergent distributed state

Neither design is universally better.

The workload determines the appropriate architecture.

Cluster Failover

A complete failover is usually more than:

A failed → start B

A safer conceptual workflow is:

Detect failure
     │
     ▼
Validate cluster membership
     │
     ▼
Establish quorum
     │
     ▼
Fence failed owner if required
     │
     ▼
Select recovery node
     │
     ▼
Acquire resources
     │
     ▼
Start service
     │
     ▼
Validate health
     │
     ▼
Restore traffic

Each stage should be observable.

Resource Dependencies

Clustered services often have dependencies.

For example:

Virtual IP
    │
    ▼
Filesystem
    │
    ▼
Database
    │
    ▼
Application

The startup order matters.

You cannot safely start:

Application

before the required database or filesystem exists.

Shutdown order may be reversed:

Application
   ↓
Database
   ↓
Filesystem
   ↓
Virtual IP

Cluster resource managers model these dependencies.

Pacemaker Explained

Pacemaker is an open-source cluster resource manager used in Linux high-availability environments.

Conceptually:

                 Pacemaker Cluster
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
           Node A                 Node B

Pacemaker can manage resources such as:

Virtual IP Addresses
Services
Filesystems
Databases
Custom Resource Agents

and coordinate where they should run.

Corosync Explained

Corosync provides group communication and membership capabilities used by clustered systems.

A simplified Linux HA stack can be viewed as:

Applications / Resources
          │
          ▼
       Pacemaker
          │
          ▼
       Corosync
          │
          ▼
        Network

Very roughly:

Corosync
→ cluster communication / membership

Pacemaker
→ resource management / recovery decisions

Production behavior is more detailed, but this distinction provides a useful mental model.

Pacemaker Resource Example

A clustered web service might contain:

Virtual IP
    │
    ▼
Filesystem
    │
    ▼
Nginx

The cluster must ensure that these resources run:

On the correct node
In the correct order
Under the correct conditions

If Node A fails:

Node A ✗

the resources can move to Node B according to cluster policy.

Virtual IP Failover

A traditional HA service may use a virtual IP address.

Normal:

Virtual IP: 192.0.2.50
          │
          ▼
        Node A

After failover:

Virtual IP: 192.0.2.50
          │
          ▼
        Node B

Clients continue using the same service address.

The cluster changes which node owns it.

This is different from load balancing, where multiple servers may receive traffic simultaneously.

Cluster vs Load Balancer

Consider:

Load Balancer
    │
    ├── Node A
    ├── Node B
    └── Node C

The load balancer distributes requests.

The nodes do not necessarily coordinate directly.

A cluster may instead contain direct coordination:

A ◄────► B
▲        ▲
│        │
└──► C ◄─┘

with shared knowledge of membership, leadership, or resource ownership.

Therefore:

Multiple load-balanced servers

are not automatically the same thing as:

A coordinated cluster

although systems frequently combine both approaches.

Kubernetes as a Clustered System

Kubernetes provides a useful modern example of cluster architecture.

At a simplified level:

                   Kubernetes Cluster
                           │
             ┌─────────────┴─────────────┐
             ▼                           ▼
        Control Plane                Worker Nodes
             │                    ┌──────┼──────┐
             ▼                    ▼      ▼      ▼
          Cluster State          Pod    Pod    Pod

The control plane manages the desired state of the cluster.

Workers execute workloads.

Kubernetes Control Plane

Important components conceptually include:

API Server
Scheduler
Controller Manager
etcd

The Kubernetes API becomes the primary interface for cluster state.

Controllers continuously compare:

Desired State

with:

Actual State

and attempt to reconcile the difference.

Desired State

Suppose you declare:

I want 3 replicas of my application.

Current state:

Pod A ✓
Pod B ✓
Pod C ✓

If Pod B fails:

Pod A ✓
Pod B ✗
Pod C ✓

the platform attempts to return toward:

3 desired replicas

by creating a replacement.

This is different from traditional active-passive service failover but solves a related availability problem through orchestration.

etcd and Cluster State

Kubernetes uses etcd as a consistent key-value store for critical cluster data.

This makes the availability and integrity of etcd fundamental to control-plane architecture.

The general lesson extends beyond Kubernetes:

The service that stores cluster coordination state is itself part of the cluster’s critical failure domain.

A perfectly redundant worker tier cannot compensate for a badly designed control plane.

Kubernetes Node Failure

Suppose:

Worker A
 ├── Pod 1
 └── Pod 2

fails.

The cluster can eventually schedule replacement workloads on surviving nodes:

Worker B
 └── Replacement Pod 1

Worker C
 └── Replacement Pod 2

provided:

Capacity Exists
Storage Supports It
Scheduling Rules Allow It
Dependencies Remain Available

Again, redundancy is not enough without capacity and state.

Database Clustering

Database clustering deserves special attention because databases contain durable state.

A database cluster might use:

Primary
   │
   ├── Replica A
   └── Replica B

or a distributed architecture:

Node A ◄────► Node B
  ▲             ▲
  │             │
  └────► Node C ◄────┘

The architecture must define:

Who accepts writes?

How is data replicated?

When is a write committed?

How is a leader selected?

What happens during a network partition?

How is stale data handled?

Database Leader Election

Suppose:

A = Leader
B = Follower
C = Follower

A fails.

The remaining nodes may attempt to elect:

B = New Leader

But B must generally have sufficiently current state according to the database’s protocol.

Promoting an outdated node can cause acknowledged data to disappear.

This is why database failover cannot safely be reduced to:

Ping failed → promote random replica

Replication Lag

In asynchronous replication:

Primary
   │
   ├── Write 100
   ├── Write 101
   └── Write 102

Replica
   │
   ├── Write 100
   └── Write 101

If the primary fails before Write 102 reaches the replica, promoting that replica can lose Write 102.

Monitoring therefore needs to include:

Replication Lag
Replication Position
Replica Health

not merely node uptime.

Cluster Networking

Cluster traffic may include:

Heartbeats
Replication
Membership
Client Traffic
Storage Traffic
Management Traffic

Putting all traffic on one overloaded network can create failure coupling.

Conceptually:

Application Traffic
       +
Replication
       +
Cluster Heartbeats
       │
       ▼
Single Network

Under congestion, heartbeats may be delayed.

The cluster may interpret delay as node failure.

This can trigger unnecessary failover.

Network Redundancy

Critical clusters may require redundant network paths.

Instead of:

Node A ─── Switch ─── Node B

a stronger architecture can provide multiple paths:

          Switch A
         /        \
     Node A      Node B
         \        /
          Switch B

But redundancy is real only when paths do not share hidden failure points.

For networking fundamentals, see Computer Networking Explained.

Cluster DNS

Clustered services often depend on DNS for:

Service Discovery
Node Discovery
Client Routing
Failover

DNS problems can therefore appear as cluster problems.

Useful diagnostic tools include:

dig
nslookup
getent hosts

For the underlying system, see DNS Explained.

Time Synchronization

Distributed systems rely heavily on timestamps for:

Logs
Certificates
Monitoring
Events
Auditing

Significant clock differences make troubleshooting extremely difficult.

Example:

Node A log: 10:01:05 failure
Node B log: 09:58:32 election
Node C log: 10:03:14 reconnect

Without synchronized clocks, reconstructing the incident becomes confusing.

Reliable time synchronization is therefore an important operational dependency.

Cluster Security

Cluster communication itself must be protected.

Potential risks include:

Unauthorized Node Joining
Spoofed Cluster Messages
Credential Theft
Management API Exposure
Unencrypted Replication
Compromised Cluster Administrator

A cluster increases the number of systems that must trust one another.

That trust should be explicit and minimized.

Management Plane vs Data Plane

It is useful to distinguish:

Management / Control Plane

from:

Data Plane

The control plane makes decisions:

Who is leader?
Where should workload run?
Which nodes are members?

The data plane performs workload traffic:

HTTP Requests
Database Queries
Storage I/O

A control-plane failure does not always mean existing data-plane traffic stops immediately.

But it may prevent recovery, scaling, or configuration changes.

This distinction is especially important in cloud and Kubernetes architecture.

Observability

Cluster monitoring should include more than CPU and RAM.

Important signals include:

Node Membership
Quorum Status
Leader Identity
Heartbeat Latency
Resource Ownership
Failover Events
Fencing Events
Replication Lag
Network Latency
Storage Latency
Cluster Errors

A useful dashboard should answer:

How many nodes exist?

How many are healthy?

Do we currently have quorum?

Who is leader?

Where are critical resources running?

Has failover occurred?

Is redundancy degraded?

Monitor Degraded State

Suppose:

A ✓
B ✓
C ✓

becomes:

A ✓
B ✓
C ✗

The service may still function perfectly.

But the cluster has lost failure tolerance.

Monitoring should therefore distinguish:

Service Down

from:

Service Up but Redundancy Degraded

This distinction is essential for preventing second-failure outages.

Troubleshooting a Cluster

Cluster troubleshooting should be systematic.

Start with:

1. Node State
2. Network Connectivity
3. Cluster Membership
4. Quorum
5. Resource State
6. Fencing
7. Storage
8. Replication
9. Application
10. Logs

Do not immediately restart every node.

Restarting nodes can destroy evidence and potentially worsen quorum or split-brain conditions.

Step 1: Check Node Health

On Linux, start with system health:

uptime

free -h

df -h

systemctl --failed

A node may appear as a cluster problem when it is actually suffering from:

Disk Full
Memory Exhaustion
CPU Saturation
Kernel Problem
Service Crash

For broader administration, see Linux Server Administration.

Step 2: Check Network Connectivity

Test node-to-node communication.

Examples:

ping node-b

where ICMP is permitted, and:

nc -vz node-b <port>

for specific services.

Check routes:

ip route

Check interfaces:

ip addr

Check sockets:

ss -lntp

For more diagnostics, see the Linux Network Tools Guide.

Step 3: Check DNS Resolution

Use:

getent hosts node-b

and where DNS is involved:

dig node-b.example.com

A DNS change or resolver failure can break node discovery while basic IP connectivity remains healthy.

Step 4: Check Quorum

Before moving resources manually, determine:

Does the cluster currently have quorum?

If the cluster has lost quorum, forcing resources online can create exactly the split-brain scenario the cluster is attempting to prevent.

Step 5: Check Resource Ownership

Determine:

Which node owns the resource?

Which node believes it owns the resource?

Is the resource stopped elsewhere?

Never assume the cluster’s view matches the physical reality.

Step 6: Check Fencing History

If failover did not occur, determine whether the cluster was waiting for fencing.

Conceptually:

Node A uncertain
      │
      ▼
Fence request
      │
      X
Fencing failed
      │
      ▼
Cluster refuses unsafe failover

This may look like a failure of HA.

In reality, the cluster may be protecting data integrity.

Step 7: Check Storage

For shared-storage systems:

Can the node see the storage?

Is the filesystem mounted?

Is storage latency excessive?

Does another node still own it?

For replicated storage:

Is replication healthy?

Is data synchronized?

Which node has the latest copy?

Step 8: Check Logs

Correlate:

Cluster Logs
System Logs
Application Logs
Network Logs
Storage Logs

Use synchronized timestamps to reconstruct:

Failure
→ Detection
→ Election
→ Fencing
→ Failover
→ Recovery

This timeline is usually more valuable than one isolated error message.

Common Cluster Failure: Quorum Lost

Example:

Five-node cluster

A ✓
B ✓
C ✗
D ✗
E ✗

Only two of five voters remain.

If majority quorum requires three:

2 < 3

the remaining nodes may stop certain operations.

This can look like:

Servers are running,
but cluster refuses service.

That behavior may be intentional.

Common Cluster Failure: Split Brain

Symptoms may include:

Two leaders
Two primaries
Conflicting resource ownership
Divergent database writes
Filesystem corruption

Do not blindly reconnect the partitions.

First determine which side contains authoritative state and how the technology expects split-brain recovery to be performed.

Common Cluster Failure: Flapping Node

A node repeatedly:

joins
leaves
joins
leaves

Possible causes include:

Network Packet Loss
CPU Starvation
Memory Pressure
Heartbeat Timeout
Faulty NIC
Switch Problem
Storage Stall
Virtualization Pause

Repeated membership changes can destabilize the entire cluster.

Common Cluster Failure: Fencing Loop

A node is repeatedly fenced:

Start
 ↓
Join
 ↓
Problem
 ↓
Fence
 ↓
Restart
 ↓
Join
 ↓
Fence

Do not simply disable fencing.

Investigate why the cluster considers the node unsafe.

Removing a safety mechanism can convert an availability problem into data corruption.

Common Cluster Failure: Resource Cannot Start

Possible causes:

Dependency Missing
Port Already Used
Filesystem Not Mounted
Configuration Error
Permissions
Missing Secret
Database Unavailable
Wrong Node Placement

Test the resource itself as well as the cluster manager.

Cluster Capacity Planning

A three-node cluster does not automatically tolerate one node failure at full workload.

Suppose:

Node A = 40% workload
Node B = 40%
Node C = 20%

Total:

100%

If A fails:

Remaining capacity must absorb 40%.

If B and C are already near their limits, the cluster can remain logically healthy but become overloaded.

HA planning therefore includes:

Failure Capacity

not merely node count.

N+1 Cluster Capacity

Suppose production requires:

3 nodes worth of capacity

An N+1 design provides:

4 nodes

so one can fail while three remain capable of carrying the workload.

More critical systems may require stronger redundancy models.

Failure Domains

Do not place all cluster nodes inside the same failure domain.

Bad:

Physical Host
 ├── Cluster Node A
 ├── Cluster Node B
 └── Cluster Node C

Host failure removes the entire cluster.

Better:

Host A → Node A
Host B → Node B
Host C → Node C

Cloud architecture extends this concept:

Availability Zone A → Nodes
Availability Zone B → Nodes
Availability Zone C → Nodes

For broader failure-domain design, see High Availability Explained.

Multi-AZ Clustering

A cloud cluster may span availability zones:

                      Cluster
                         │
            ┌────────────┼────────────┐
            ▼            ▼            ▼
          Zone A       Zone B       Zone C
            │            │            │
          Node A       Node B       Node C

This protects against some zone-level failures.

But cross-zone clustering introduces:

Network Latency
Replication Traffic
Inter-Zone Cost
Failure Detection Complexity

The cluster technology must support the latency and network behavior of the intended topology.

Multi-Region Clustering

Stretching a tightly coupled cluster across geographic regions is substantially more difficult.

Latency becomes much larger:

Region A
    │
    │ Long Distance
    ▼
Region B

Synchronous coordination may become expensive.

Many systems therefore prefer:

Regional Cluster A
       │
       ▼
Replication
       │
       ▼
Regional Cluster B

rather than one low-latency cluster stretched globally.

This separates local consensus from geographic disaster recovery.

Clustering vs Disaster Recovery

A local cluster can protect against:

Node Failure
Process Failure
Local Hardware Failure

but may not protect against:

Entire Data Center Failure
Region Failure
Catastrophic Storage Corruption
Ransomware
Operator Error

Disaster recovery addresses broader failure scenarios.

A mature design might use:

Region A
   │
   └── HA Cluster
          │
          ▼
   Replication / Backup
          │
          ▼
Region B
   │
   └── Recovery Cluster

High availability and disaster recovery complement each other.

Backups Still Matter

A five-node database cluster can still replicate:

DELETE FROM critical_table;

to all five nodes.

Now every node consistently contains the unwanted change.

Clustering provides:

Availability

not necessarily:

Historical Recovery

Independent backups and tested restore procedures remain essential.

Infrastructure as Code for Clusters

Cluster infrastructure should be reproducible where possible.

Instead of manually creating:

Networks
Instances
Firewall Rules
Load Balancers
Cluster Configuration
Monitoring

use controlled automation.

Conceptually:

Git
 │
 ▼
Infrastructure Code
 │
 ▼
Automation
 │
 ▼
Cluster

This improves:

Consistency
Reviewability
Repeatability
Disaster Recovery

For the complete approach, see Infrastructure as Code Explained.

Cluster Security Best Practices

A production cluster should consider:

Restrict Cluster Ports

Only cluster members and required management systems should reach internal coordination services.

Protect Administrative Interfaces

Cluster control interfaces can have enormous privileges.

Use Strong Authentication

Nodes should not trust arbitrary systems claiming to be cluster members.

Encrypt Sensitive Traffic

Replication and management traffic may contain sensitive data.

Separate Privileges

Application administrators do not necessarily need unrestricted cluster-management privileges.

Monitor Membership Changes

Unexpected nodes joining or leaving should be visible.

Patch Cluster Nodes

Redundancy does not compensate for identical vulnerabilities across every node.

Server Cluster Anti-Patterns

Calling Two Servers a Cluster Without Coordination

Server A
Server B

is not enough.

Disabling Quorum to “Fix” Availability

If quorum prevents a resource from starting, forcing it online may create split brain.

Understand why quorum was lost first.

Disabling Fencing

Fencing exists because an unreachable node may still be alive.

Removing it can create concurrent ownership and data corruption.

All Nodes on One Host

Virtual machines do not create meaningful hardware redundancy if the same hypervisor failure removes all of them.

All Nodes Behind One Critical Switch

Compute redundancy does not help if one network device isolates the entire cluster.

No Spare Capacity

If surviving nodes cannot absorb failed-node workloads, failover can create overload instead of availability.

No Recovery Procedure

Failover is only half the lifecycle.

The failed node eventually needs to return safely.

Cluster Without Backups

Replication and clustering do not replace recoverable historical copies.

No Failure Testing

A cluster that has never survived a controlled failure is an unverified architecture.

Production Cluster Architecture

A mature architecture might look like:

                            Global Users
                                 │
                                 ▼
                               DNS
                                 │
                                 ▼
                           CDN / Edge
                                 │
                                 ▼
                          Load Balancer
                                 │
                   ┌─────────────┼─────────────┐
                   ▼             ▼             ▼
                Zone A        Zone B        Zone C
                   │             │             │
                Node A        Node B        Node C
                   │             │             │
                   └──── Cluster Coordination ─┘
                                 │
                                 ▼
                         Distributed State
                                 │
                 ┌───────────────┼───────────────┐
                 ▼               ▼               ▼
              Database         Cache           Queue
                 │
                 ▼
          Replication / Backup
                 │
                 ▼
         Disaster Recovery Site

Several different availability mechanisms coexist:

CDN
→ edge resilience

Load Balancer
→ traffic distribution

Application Cluster
→ compute redundancy

Database Cluster
→ state resilience

Backup
→ historical recovery

DR Environment
→ catastrophic recovery

No single cluster solves every failure scenario.

Practical Cluster Design Workflow

Step 1: Define the Workload

Is it:

Stateless Web Application?
Database?
Filesystem?
Queue?
Container Platform?
Compute Workload?

Cluster architecture depends heavily on state.

Step 2: Define Failure Requirements

Must the system survive:

Process Failure?
Server Failure?
Hypervisor Failure?
Switch Failure?
Availability Zone Failure?
Region Failure?

Do not build architecture before defining the intended failure boundary.

Step 3: Choose the Cluster Model

Potential models include:

Active-Passive
Active-Active
Load Balanced
Leader-Follower
Distributed Shared-Nothing

Step 4: Define Membership

How do nodes discover one another?

How is membership stored?

What happens when a node disappears?

Step 5: Define Quorum

Ask:

How many voters exist?

What constitutes majority?

What happens when quorum is lost?

Is a witness required?

Step 6: Define Fencing

For stateful systems:

How do we guarantee the old owner cannot continue writing?

This must have a concrete answer.

Step 7: Define Data Architecture

Is data:

Shared?
Replicated?
Partitioned?
External?

What happens to data during failover?

Step 8: Define Resource Dependencies

Document startup and shutdown ordering.

Step 9: Design Capacity

Ensure the cluster can carry production traffic after the failures it claims to tolerate.

Step 10: Design Observability

Monitor:

Membership
Quorum
Leadership
Resources
Replication
Fencing
Capacity

Step 11: Test Failure

Deliberately stop a node.

Then test:

Network Partition
Storage Failure
Application Failure
Leader Failure

under controlled conditions.

Step 12: Test Recovery

Bring the failed node back.

Verify:

Safe Reintegration
State Synchronization
No Split Brain
Redundancy Restored

Server Clustering Checklist

Before calling a system a production cluster, verify:

Membership

Are node identities unique?
Is membership monitored?
Can nodes join and leave safely?

Networking

Is node-to-node connectivity reliable?
Are critical network paths redundant?
Is cluster traffic protected?

Quorum

Is the voting model understood?
What happens when quorum is lost?
Are two-node scenarios handled safely?

Fencing

Is fencing configured where required?
Has it been tested?
Can it actually isolate a failed node?

Resources

Are dependencies modeled?
Is startup order correct?
Is shutdown order correct?

Data

Is state shared or replicated?
Is replication monitored?
Is split-brain recovery documented?

Capacity

Can surviving nodes handle failover load?
Is N+1 or stronger capacity available where required?

Failure Domains

Are nodes distributed across independent infrastructure?
Could one host, switch, rack, or zone remove all nodes?

Monitoring

Is quorum visible?
Is leader identity visible?
Are resource movements logged?
Are fencing events alerted?

Recovery

Is failback tested?
Can failed nodes safely rejoin?
Are backups independent of the cluster?

Frequently Asked Questions

What is a server cluster?

A server cluster is a group of independent servers that coordinate to provide one or more services, workloads, or distributed resources.

What is a cluster node?

A node is an individual server, virtual machine, or other computing participant that belongs to a cluster.

What is a high availability cluster?

An HA cluster coordinates redundant nodes so that services can continue or recover when one node fails.

What is an active-passive cluster?

One node actively provides the service while another remains ready to take over after failure.

What is an active-active cluster?

Multiple nodes actively provide service at the same time.

What is quorum?

Quorum is a mechanism requiring sufficient cluster participation or agreement before certain distributed decisions are permitted.

Why do clusters need quorum?

Quorum can help prevent isolated parts of a cluster from independently assuming authority and causing split-brain conditions.

What is split brain?

Split brain occurs when separated parts of a cluster simultaneously act as authoritative owners of the same role or resource.

What is fencing?

Fencing isolates a node so that it cannot continue accessing protected resources after the cluster decides another node should take ownership.

What is STONITH?

STONITH is a traditional Linux HA term for a fencing mechanism that forces an unsafe node into a known inactive state, often through power or infrastructure control.

What are Pacemaker and Corosync?

Pacemaker is a Linux cluster resource manager, while Corosync provides cluster communication and membership capabilities commonly used in Linux HA stacks.

Is Kubernetes a cluster?

Yes. Kubernetes organizes control-plane and worker systems into a cluster and continuously manages workloads toward declared desired state.

Is a load-balanced server farm the same as a cluster?

Not necessarily. Load-balanced servers may operate independently, while a cluster usually implies some form of coordination, membership, shared state, or distributed decision-making.

Does clustering replace backups?

No. Clustering primarily addresses availability and service continuity. Independent backups are required for historical recovery from corruption, deletion, malware, or other data-loss events.

Does clustering replace disaster recovery?

No. A local cluster may survive node failures while still being vulnerable to an entire data-center or regional outage.

How many nodes should a cluster have?

There is no universal number. The correct node count depends on quorum rules, capacity, workload, failure requirements, data architecture, and the specific clustering technology.

Final Thoughts

Server clustering is often introduced with a deceptively simple diagram:

A ─── B ─── C

But the lines between those nodes represent some of the hardest problems in infrastructure engineering.

A real cluster must reason about:

Membership
Failure
Network Partitions
Ownership
Leadership
Quorum
Consensus
State
Replication
Recovery

The most important problem is uncertainty.

When Node A cannot communicate with Node B, A does not automatically know:

B is dead.

It only knows:

I cannot currently communicate with B.

That difference explains why clusters need mechanisms such as:

Heartbeats
Quorum
Consensus
Fencing

and why unsafe failover can be worse than temporary downtime.

The architecture therefore evolves from:

Single Server

to:

                    Cluster
                       │
           ┌───────────┼───────────┐
           ▼           ▼           ▼
        Node A      Node B      Node C
           │           │           │
           └──── Coordination ─────┘
                       │
                       ▼
                  Shared Service

But clustering does not eliminate failure.

It changes the failure model.

Instead of asking:

What happens when one server dies?

you now ask:

What happens when one node dies?

What happens when the network partitions?

What happens when quorum disappears?

What happens when two nodes think they are primary?

What happens when the leader fails?

What happens when storage remains reachable from a supposedly failed node?

What happens when a node returns with stale state?

These are much more sophisticated questions.

That is precisely why clustering is such an important step in understanding modern infrastructure.

The learning path should therefore continue from High Availability Explained into this clustering layer.

Use Load Balancing Explained to understand how client traffic reaches healthy application instances.

Use Cloud Databases Explained to understand the stateful systems that make clustering significantly more difficult.

Use Cloud Networking Explained to understand the networks connecting distributed nodes.

Use Linux Server Administration and the Linux Network Tools Guide when troubleshooting the operating systems underneath the cluster.

Finally, use Infrastructure as Code Explained to make clustered infrastructure reproducible instead of manually assembled.

The CorgiNetwork infrastructure learning path now becomes:

DNS → Edge → CDN Caching → Reverse Proxy → Load Balancing → High Availability → Server Clustering → Disaster Recovery → Storage Architecture → Cloud Networking → Infrastructure as Code.