Database Cluster: Cluster Split-Brain Event Caused by Unintended Node Count Reduction

Incident Report for Dribl

Postmortem

The backup process was incorrectly configured and performed an unintended cluster node downgrade/removal, violating the requirement for an odd-numbered cluster size.

Impact:

Service instability and intermittent outages
Elevated risk of data inconsistency (no permanent data loss confirmed)
Degraded system performance during incident window

Resolution:

Affected node was restored and cluster returned to an odd number configuration
Cluster quorum and leader election normalised
System integrity verified

Address Root Cause Actions:

Update backup processes to ensure disconnecting node approach is no longer utilised on backup nodes
Enforce configuration policies requiring odd-numbered node counts

The above actions will ensure the root cause will be addressed.

Posted Feb 15, 2026 - 19:02 AEDT

Resolved

Critical system incident occurred within the production database cluster resulting in a split-brain condition. The cluster was originally configured with an odd number of nodes to maintain quorum integrity and ensure proper leader election.

During a scheduled backup operation, an automated backup process unintentionally triggered a downgrade action, which removed one active node from the cluster, reducing the total node count to an even number. This change resulted in the cluster operating with an even number of nodes, which compromised the quorum mechanism.

Without a clear majority, the cluster partitioned into two separate node groups, each believing it had authority to act as the primary. This caused a split-brain event, leading to:

* Conflicting cluster state decisions
* Intermittent service availability
* Risk of data inconsistency between partitions

The condition persisted until the issue was identified and the cluster was restored to an odd-numbered node configuration, re-establishing quorum and stabilising operations. All services were fully restored 15/02/2026 @ 6PM AEST.

Posted Feb 15, 2026 - 02:30 AEDT