Problem

Recently I’ve been setting up Redis Cluster to run in Kubernetes on GKE. During the setup, I’ve been experimenting with various configurations of Redis Nodes and clustering. Unfortunately, at some point, I’ve rendered my cluster unusable. Logs showed the following:

1:C 20 Feb 2019 16:57:45.189 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Feb 2019 16:57:45.189 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 20 Feb 2019 16:57:45.189 # Configuration loaded
1:M 20 Feb 2019 16:57:45.192 # Unrecoverable error: corrupted cluster config file.

Solution

The fix was simple - delete your Redis Cluster configuration file (default # cluster-config-file nodes-6379.conf ) and restart Redis. This may not work in your case and there could be reasons not to delete existing records. Exercise with care.

After this, I’ve looked up the same Redis Cluster configuration file. It contained the following:

# cat /redis/nodes.conf
2c120eb1eaf37ea04b1880fd5933898167317b40 :0@0 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0

Please be aware

Once recovered, your Redis Node will no longer be able to recognise that it is part of the cluster as we’ve deleted its records. Below command will introduce node back into the cluster. cluster-node-ip is an IP of the live, healthy node running in the cluster. Redis will handle the rest of handshakes via the gossip protocol. More info here.

redis-cli CLUSTER MEET cluster-node-ip cluster-node-port