etcd集群节点异常处理

某个节点异常如何处理

• 使用命令移除节点
• 删除异常节点数据目录
• 重新将节点添加至集群

移除节点

假设etcd3出现了异常。

leader节点操作：

$ etcdctl member list
3fccfd9e945d331b, started, etcd-3, http://10.220.43.208:2380, http://10.220.43.208:2379, false
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false
$ etcdctl member remove 3fccfd9e945d331b
Member 3fccfd9e945d331b removed from cluster cee4a9895463ca7d
$ etcdctl member list
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false

删除异常节点的数据目录

etcd-3节点操作：

$ rm -rf /var/lib/etcd
cat /etc/etcd/etcd.conf 
ETCD_NAME=etcd-3
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="http://10.220.43.208:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://10.220.43.208:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER="etcd-1=http://10.220.43.206:2380,etcd-2=http://10.220.43.207:2380,etcd-3=http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER_STATE="existing" ##修改为existing
ETCD_INITIAL_CLUSTER_TOKEN="singless"
ETCD_ADVERTISE_CLIENT_URLS="http://10.220.43.208:2379"

将节点添加至集群

leader节点操作：

$ etcdctl member add etcd-3 --peer-urls=http://10.220.43.208:2380
Member 765137d296bf17e7 added to cluster cee4a9895463ca7dETCD_NAME="etcd-3"
ETCD_INITIAL_CLUSTER="etcd-2=http://10.220.43.207:2380,etcd-3=http://10.220.43.208:2380,etcd-1=http://10.220.43.206:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.220.43.208:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
$ etcdctl member list
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
765137d296bf17e7, unstarted, , http://10.220.43.208:2380, , false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false

etcd-3启动etcd。 

$ systemctl start etcd

leader节点操作：

$ etcdctl member list
5d19435c38496e2c, started, etcd-2, http://10.220.43.207:2380, http://10.220.43.207:2379, false
765137d296bf17e7, started, etcd-3, http://10.220.43.208:2380, http://10.220.43.208:2379, false
9f23d9fd9d308b96, started, etcd-1, http://10.220.43.206:2380, http://10.220.43.206:2379, false