According to the Splunk Search Head Clustering Troubleshooting Guide, when a Search Head Cluster (SHC) member is reverted from a backup or experiences configuration drift (e.g., an outdated Raft state), it can fail to rejoin the cluster due to inconsistent Raft metadata. The Raft database stores the SHC’s internal consensus and replication state, including knowledge object synchronization, captain election history, and peer membership information.
If this Raft metadata becomes corrupted or outdated (as in the scenario where a node is restored from backup), the recommended and Splunk-supported remediation is to clean the Raft metadata using:
splunk clean raft
This command resets the node’s local Raft state so it can re-synchronize with the current SHC captain and rejoin the cluster cleanly.
The steps generally are:
Stop the affected SHC member.
Run splunk clean raft on that node.
Restart Splunk.
Verify that it successfully rejoins the SHC.
Deleting configuration stanzas or forcing re-addition (Options B and C) can lead to further inconsistency or data loss. Reviewing logs (Option A) helps diagnose issues but does not resolve Raft corruption.
References (Splunk Enterprise Documentation):
• Troubleshooting Raft Metadata Corruption in Search Head Clusters
• splunk clean raft Command Reference
• Search Head Clustering: Recovering from Backup and Membership Failures
• Splunk Enterprise Admin Manual – Raft Consensus and SHC Maintenance
