Ubuntu24搭建k8s高可用集群

环境信息

主机名	IP	k8s版本	备注
vm-master	192.168.103.250	1.28.2	master1
vm-master2	192.168.103.249	1.28.2	master2
vm-master3	192.168.103.254	1.28.2	master3
vm-node1	192.168.103.251	1.28.2	node1
vm-node2	192.168.103.252	1.28.2	node2

容器进行时：container
网络插件为：flannel

配置中有不清楚的地方可以查看官方文档 https://kubernetes.io/zh-cn/docs/home/

1. 基础配置（所有节点）

配置/etc/hosts文件

192.168.103.253 vip.cluster.local
192.168.103.250 vm-master
192.168.103.249 vm-master2
192.168.103.251 vm-node1
192.168.103.252 vm-node2
192.168.103.254 vm-master3

配置时间同步

sudo apt install chrony -y

sudo vim /etc/chrony/chrony.conf# master节点配置
confdir /etc/chrony/conf.d
pool ntp.aliyun.com iburst maxsources 4
pool time1.cloud.tencent.com iburst maxsources 2
refclock PHC /dev/ptp0 poll 0 dpoll -2 offset 0
local stratum 10
allow 192.168.103.0/24
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
makestep 1 3
maxdistance 16.0
leapsecmode slew
rtcsync
leapsectz right/UTC

sudo vim /etc/chrony/chrony.conf# node节点配置
confdir /etc/chrony/conf.d
server vm-master iburst
server vm-master2 iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
makestep 1 3
maxdistance 16.0
leapsecmode slew
rtcsync
leapsectz right/UTC

sudo systemctl restart chrony

更新系统并关闭交换分区

sudo apt-get update
sudo apt-get upgrade -y
sudo swapoff -a
sudo vim /etc/fstab
#注释掉swap所在行
#/swap
#重启查看是否成功，swap显示0就是成功sudo reboot
free -h
#Swap:             0B          0B          0B

修改内核参数

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOFsudo modprobe overlay
sudo modprobe br_netfilter

配置网络参数

# 默认情况下，Linux 内核不允许 IPv4 数据包在接口之间路由。
# 手动启用 IPv4 数据包转发
sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl --system

生产环境需要配置防火墙放行端口

IPVS负载均衡

# 安装
sudo apt install -y ipset ipvsadm# 内核加载ipvs
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF# 加载模块
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack

2. 安装组件（所有节点）

容器运行时containerd

sudo apt install -y containerd# 配置containerd文件
sudo mkdir -p /etc/containerd/containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1sudo sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
sudo cat /etc/containerd/config.toml | grep SystemdCgroup
# 修改沙箱镜像源
sudo sed -i "s#registry.k8s.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
sudo cat /etc/containerd/config.toml | grep sandbox_image

安装kubeadm，kubelet，kubectl

# 安装依赖
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gpg# 添加kubernetes的key
curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg# 添加kubernetes apt仓库，使用阿里云镜像源
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list# 更新apt索引
sudo apt update# 查看版本列表
apt-cache madison kubeadm# 不带版本默认会安装最新版本，本文安装的版本为1.28.2
sudo apt-get install -y kubelet kubeadm kubectl# 锁定版本，不随 apt upgrade 更新
sudo apt-mark hold kubelet kubeadm kubectl# kubectl命令补全
sudo apt install -y bash-completionkubectl completion bash | sudo tee /etc/profile.d/kubectl_completion.sh > /dev/null. /etc/profile.d/kubectl_completion.sh

说明：

kubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环。

3. 高可用方案部署keepalived、haproxy（所有master节点）

所有master节点安装配置haproxy

sudo apt install keepalived haproxy -y

sudo vim /etc/haproxy/haproxy.cfg

### 在文件末尾添加
frontend apiserverbind *:16443mode tcpoption tcplogdefault_backend apiserverbackendbackend apiserverbackendoption httpchkhttp-check connect sslhttp-check send meth GET uri /healthzhttp-check expect status 200mode tcpbalance     roundrobinserver vm-master 192.168.103.250:6443 check verify none	#有几个master节点就添加几个server vm-master2 192.168.103.249:6443 check verify none
###

#重启haproxy服务，监听16443端口
sudo systemctl restart haproxy

所有master节点安装配置keepalived

cd /etc/keepalived
sudo cp keepalived.conf.sample keepalived.conf

sudo vim keepalived.conf
###全部替换成以下内容，主备节点略有不同，需要修改注释的地方! Configuration File for keepalivedglobal_defs {router_id LVS_DEVEL
}vrrp_script check_apiserver {script "/etc/keepalived/check_apiserver.sh"interval 3weight -2fall 10rise 2
}vrrp_instance VI_1 {state MASTER	# 主master节点为MASTER，备master节点为BACKUPinterface eth0virtual_router_id 51priority 150	#优先级，主master比备master多50。例如主150，备100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.103.253	#虚拟IP，指定的vip，所有节点一致，可添加多个vip}track_script {check_apiserver}
}

#添加检查脚本

sudo vim /etc/keepalived/check_apiserver.sh

#!/bin/sh
errorExit() {echo "*** $*" 1>&2exit 1
}
curl -sfk --max-time 2 https://localhost:16443/healthz -o /dev/null || errorExit "Error GET https://localhost:16443/healthz"

#脚本添加可执行权限
sudo chmod +x /etc/keepalived/check_apiserver.sh

#重启keepalived
sudo systemctl restart keepalived

所有master节点拉取镜像

kubeadm config images listkubeadm config images list --image-repository registry.cn-hangzhou.aliyuncs.com/google_containerskubeadm config images pull --kubernetes-version=v1.28.2 --image-repository registry.aliyuncs.com/google_containers

4. 初始化k8s集群

仅主master节点生成配置文件模板

kubeadm config print init-defaults > init.default.yaml

###内容如下，根据情况修改注释的地方
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: 192.168.103.250	#主master节点IPbindPort: 6443
nodeRegistration:criSocket: unix:///var/run/containerd/containerd.sockimagePullPolicy: IfNotPresentname: vm-master	#主master名称taints: null
---
apiServer:timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
controlPlaneEndpoint: vip.cluster.local:16443	#添加高可用配置
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:local:dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers	#配置镜像库
kind: ClusterConfiguration
kubernetesVersion: 1.28.2
networking:dnsDomain: cluster.localpodSubnet: 10.200.0.0/16	#pod子网serviceSubnet: 10.96.0.0/12
scheduler: {}

仅主master初始化节点

sudo kubeadm init --config init.default.yaml --upload-certs

###完成后输出结果如下，输出内容中有后续操作说明
Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:# 创建配置文件操作mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of the control-plane node running the following command on each as root:# 添加其他master节点操作kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4 \--control-plane --certificate-key f01a6c541e26097e4c90557256e376352a71490f149db45516ac542c58eef883Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:
# 添加worker节点操作
kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4

#配置kubeconfig文件
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(i d - u) :$ (id -g) $HOME/.kube/config

#如果初始化出错其他问题，可以清除初始化操作，重新初始化
sudo kubeadm reset -f

仅主master节点配置网络组件flannel

#先下载配置文件，若需要代理则加上代理参数，我使用的是clash代理
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml -e "https_proxy=http://127.0.0.1:7890"

vim kube-flannel.yml #更改子网网段和init.default.yaml中配置的podSubnet一致

### 若container拉取容器时需要配置代理则配置以下操作，不需要代理则跳过下列操作
# 在containerd层面设置代理（所有节点执行）
sudo mkdir -p /etc/systemd/system/containerd.service.d
sudo tee /etc/systemd/system/containerd.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://127.0.0.1:7890"
Environment="HTTPS_PROXY=http://127.0.0.1:7890"
EOF# 重启服务生效
sudo systemctl daemon-reload
sudo systemctl restart containerd kubelet

#应用并创建flannel，需等待kube自行配置
kubectl apply -f kube-flannel.yml

#查看状态为running则配置成功
例如：
root@vm-master:~# kubectl get pods -n kube-flannel
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-4ct2x   1/1     Running   0          81m

备master节点配置加入集群，所有备用master节点都执行

#测试能否ping通虚拟IP
ping vip.cluster.local#加入节点命令（初始化后的信息中有命令）：
kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4 \--control-plane --certificate-key f01a6c541e26097e4c90557256e376352a71490f149db45516ac542c58eef883

成功加入后显示信息

This node has joined the cluster and a new control plane instance was created:

Certificate signing request was sent to apiserver and approval was received.
The Kubelet was informed of the new secure connection details.
Control plane label and taint were applied to the new node.
The Kubernetes control plane instances scaled up.
A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(i d - u) :$ (id -g) $HOME/.kube/config

Run ‘kubectl get nodes’ to see this node join the cluster.

执行配置文件操作

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

耐心等待从主节点同步配置

worker节点加入

#执行命令
kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4

查看状态

kubectl get node

5. 添加新的master节点或node节点

如果前期操作时就部署了3个master节点则不用配置此操作。

由于k8s高可用集群最少需要3个master节点，所以如果只配置了两个master节点就需要再添加一个master节点。

新server配置，和之前的配置一样

基础配置（所有节点）
安装组件（所有节点）
高可用方案部署keepalived、haproxy（所有master节点，如果是添加node节点则不用配置）

获取加入节点的证书和token

主master节点操作

# 获取证书
kubeadm init phase upload-certs --upload-certs

在这里插入图片描述

# 生成加入集群的token
kubeadm token create --print-join-command

在这里插入图片描述

需要新添加的节点上执行

拼接获取的命令，添加master节点需要在token和证书之间用–control-plane --certificate-key连接。

如果只使用token则会添加为node节点

kubeadm join vip.cluster.local:16443 --token x12ydm.5w9y2439fmrfbll3 --discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4 --control-plane --certificate-key 6ac8c3d9baead5aac03895f2e9ba3f37de4067f2f34b8524f9cc5de0dc063c09

加入集群成功后输出结果如下
在这里插入图片描述

添加节点超时故障处理

如果新节点卡住，如图

在这里插入图片描述

且主master查看如下，

kubectl get node
kubectl get pods -n kube-flannel

在这里插入图片描述

则需要在新节点上检查错误日志

journalctl -u kubelet --since "10 minutes ago" | grep -i error

如果日志显示连接无法访问 registry.k8s.io 的443端口，则需要配置国内仓库或者使用代理，我使用配置代理的方法解决问题

# 在新节点上运行clash代理，
sudo mkdir -p /etc/systemd/system/containerd.service.d
sudo tee /etc/systemd/system/containerd.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://127.0.0.1:7890"
Environment="HTTPS_PROXY=http://127.0.0.1:7890"
EOF# 重启服务生效
sudo systemctl daemon-reload
sudo systemctl restart containerd kubelet

配置代理后再运行加入节点的命令即可，输出如图则配置完成

在这里插入图片描述