二进制方式部署K8s高可用集群

1 二进制方式部署K8s高可用集群

1.1 kubeadm 和二进制安装 k8s 适用场景分析

• kubeadm 是官方提供的开源工具，是一个开源项目，用于快速搭建 kubernetes 集群，目前是比较方便和推荐使用的。kubeadm init 以及 kubeadm join 这两个命令可以快速创建 kubernetes 集群。Kubeadm初始化 k8s，所有的组件都是以 pod 形式运行的，具备故障自恢复能力。

• kubeadm 是工具，可以快速搭建集群，也就是相当于用程序脚本帮我们装好了集群，属于自动部署，简化部署操作，自动部署屏蔽了很多细节，使得对各个模块感知很少，如果对 k8s 架构组件理解不深的话，遇到问题比较难排查。

• kubeadm 适合需要经常部署 k8s，或者对自动化要求比较高的场景下使用。

• 二进制：在官网下载相关组件的二进制包，如果手动安装，对 kubernetes 理解也会更全面。

• Kubeadm 和二进制都适合生产环境，在生产环境运行都很稳定，具体如何选择，可以根据实际项目进行评估。

1.2 机器规划

• 部署版本Kubernetes（k8s）v1.30.1

主机	IP地址	操作系统	配置
k8s-master-01	192.168.110.21	CentOS Linux release 7.9.2009	4颗CPU 8G内存 100G硬盘
k8s-master-02	192.168.110.22	CentOS Linux release 7.9.2009	4颗CPU 8G内存 100G硬盘
k8s-master-03	192.168.110.23	CentOS Linux release 7.9.2009	4颗CPU 8G内存 100G硬盘
k8s-node-01	192.168.110.24	CentOS Linux release 7.9.2009	4颗CPU 8G内存 100G硬盘
k8s-node-02	192.168.110.25	CentOS Linux release 7.9.2009	4颗CPU 8G内存 100G硬盘
k8s-node-03	192.168.110.26	CentOS Linux release 7.9.2009	4颗CPU 8G内存 100G硬盘

• VIP：192.168.110.20/24

1.3 基础环境部署(所有节点执行)

• 运行初始化脚本

[root@k8s-all ~]# vim /root/init.sh
#!/bin/bash
​
echo_red() { echo -e "\e[31m$1\e[0m"; }
echo_green() { echo -e "\e[32m$1\e[0m"; }
echo_yellow() { echo -e "\e[33m$1\e[0m"; }
​
interface=$(ip route | grep default | sed -e "s/^.*dev.//" -e "s/.proto.*//")
LocalIP=$(ip addr show $interface | awk '/inet / {print $2}' | cut -d/ -f1 | tail -1)
linux=$(awk -F "("  '{print $1}' /etc/redhat-release)
hostname=$(hostname)
​
echo_yellow "当前系统发行版为  $linux"
echo_yellow "当前系统网卡名为  $interface"
echo_yellow "本机IP地址为  $LocalIP"
echo_yellow "本机主机名为  $hostname"
​
sleep 2
echo_yellow '正在初始化请稍后...'
​
autoconnect=$(nmcli dev show $interface | grep 'GENERAL.AUTOCONNECT:' | awk '{print $2}')
if [[ "$autoconnect" != "yes" ]]; thenecho_yellow "检查网卡是否为开机自动连接..."nmcli con mod $interface connection.autoconnect yes &>/dev/nullnmcli con up $interface &>/dev/nullsed -i '/^ONBOOT/ c ONBOOT=yes' /etc/sysconfig/network-scripts/"ifcfg-${interface}" &>/dev/nullsystemctl restart network &>/dev/nullecho_green "网卡已设置为开机自动连接"
elseecho_green "网卡已设置为开机自动连接"
fi
​
ip_mode=$(nmcli dev show $interface | grep 'IP4.ADDRESS[1]' | awk '{print $2}')
gateway_dns=$(ip addr show $interface | awk '/inet / {split($2,a,"."); print a[1]"."a[2]"."a[3]}')
ip_mode2=$(grep BOOTPROTO /etc/sysconfig/network-scripts/"ifcfg-$interface" | awk -F "=" '{print $2}')
​
if [[ -z "$ip_mode" ]] || [ "$ip_mode2" != static ] ; thenecho_green "正在检查网卡是否为静态IP地址..."nmcli con mod $interface ipv4.addresses "${LocalIP}/24" &>/dev/nullnmcli con mod $interface ipv4.gateway "${gateway_dns}.2" &>/dev/nullnmcli con mod $interface ipv4.dns "${gateway_dns}.2" &>/dev/nullnmcli con mod $interface ipv4.method manual &>/dev/nullnmcli con up $interface &>/dev/nullsed -i '/^BOOTPROTO/ c BOOTPROTO=static' /etc/sysconfig/network-scripts/"ifcfg-${interface}" &>/dev/nullsystemctl restart network &>/dev/nullecho_green "网卡已设置静态IP"
elseecho_green "网卡已设置静态IP"
fi
​
​
for i in {1..3}; doif ping -c 4 -i 0.2 223.5.5.5 &>/dev/null; thenecho_green "网络连通性正常"breakelseecho_red "网络错误请检查网络配置"echo_yellow "正在尝试重启网络（尝试次数：$i）"systemctl restart network &>/dev/nullnmcli con up $interface &>/dev/nullsleep 5fi
done
​
manage_selinux_firewall() {SELINUXSTATUS=$(getenforce)
​if [[ "$SELINUXSTATUS" == "Disabled" ]] || [[ "$SELINUXSTATUS" == "Permissive" ]]; thenecho_green "SELinux已成功关闭或当前处于宽容模式"elseecho_red "SELinux未能关闭，正在尝试手动关闭..."sed -i '/^SELINUX=/ c\SELINUX=disabled' /etc/selinux/config &>/dev/nullsetenforce 0if [[ "$(getenforce)" == "Disabled" ]] || [[ "$(getenforce)" == "Permissive" ]]; thenecho_green "SELinux已成功关闭"elseecho_red "SELinux未能关闭，请手动解决"fifi
}
​
manage_selinux_firewall
​
FIREWALLSTATUS=$(systemctl is-active firewalld)
​
if [[ "$FIREWALLSTATUS" == "active" ]]; thenecho_yellow "防火墙状态为开启，正在关闭防火墙..."systemctl disable --now firewalld &>/dev/nullecho_green "防火墙已关闭"
elseecho_green "防火墙无需操作"
fi
​
PACKAGES="lrzsz ntpdate sysstat net-tools wget vim bash-completion dos2unix tree psmisc chrony rsync lsof gcc"
echo_yellow "正在安装常用软件..."
yum -y install $PACKAGES &>/dev/null
​
if [ $? -eq 0 ]; thenecho_green "安装成功"
elseecho_red "安装失败，请检测yum镜像仓库"
fi
​
echo_yellow "准备开启 $linux 系统的体验吧！！！"
​
# Prompt for shutdown, reboot, or exit
echo_yellow "请选择一个操作："
echo_yellow "          关机 : 1"
echo_yellow "          重启 : 2"
echo_yellow "          退出 : 3"
​
read -p "请输入您的选择: " choice
​
delete_script() {echo_yellow "脚本执行完毕，正在删除自身..."rm -f "$0"echo_green "脚本已删除。"
}
​
case $choice in1)echo_yellow "正在执行关机操作..."delete_scriptinit 0# 关机前删除脚本自身;;2)echo_green "正在执行重启操作..."delete_scriptreboot# 重启前删除脚本自身;;3)echo_yellow "请继续操作，脚本将退出。"# 退出前删除脚本自身delete_scriptexit 0;;*)echo_red "无效的输入"# 退出前删除脚本自身delete_scriptexit 1;;
esac
​
[root@k8s-all ~]# bash /root/init.sh

注意：

• 若虚拟机是进行克隆的那么网卡的UUID会重复

• 若UUID重复需要重新生成新的UUID

• UUID重复无法获取到IPV6地址

• 克隆出来的虚拟机 CentOS系统需要删除DUID

  rm -rf /etc/machine-id systemd-machine-id-setup reboot

• 查看当前的网卡列表和 UUID：

  nmcli con show

• 删除要更改 UUID 的网络连接：

  nmcli con delete uuid <原 UUID>

• 重新生成 UUID：

  nmcli con add type ethernet ifname <接口名称> con-name <新名称>

• 重新启用网络连接：

  nmcli con up <新名称>

• 所有节点配置Hosts解析

[root@K8s-all ~]# cat >> /etc/hosts << EOF
192.168.110.21 k8s-master-01
192.168.110.22 k8s-master-02
192.168.110.23 k8s-master-03
192.168.110.24 k8s-node-01
192.168.110.25 k8s-node-02
192.168.110.26 k8s-node-03
EOF

• k8s-master-01生成密钥，其他节点可以免密钥访问

[root@k8s-master-01 ~]# ssh-keygen -f ~/.ssh/id_rsa -N '' -q
[root@k8s-master-01 ~]# ssh-copy-id k8s-master-02
[root@k8s-master-01 ~]# ssh-copy-id k8s-master-03
[root@k8s-master-01 ~]# ssh-copy-id k8s-node-01
[root@k8s-master-01 ~]# ssh-copy-id k8s-node-02
[root@k8s-master-01 ~]# ssh-copy-id k8s-node-03

• 配置NTP时间同步

sed -i '3,6 s/^/# /' /etc/chrony.conf
sed -i '6 a server ntp.aliyun.com iburst' /etc/chrony.conf 
systemctl restart chronyd.service 
chronyc sources

• 禁用Swap交换分区

[root@K8s-all ~]# swapoff -a   #临时关闭
[root@K8s-all ~]# sed -i 's/.*swap.*/# &/' /etc/fstab  #永久关闭

• 升级操作系统内核

[root@k8s-all ~]# wget -c http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-ml-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# wget -c http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-ml-devel-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# rpm -ivh kernel-ml-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# rpm -ivh kernel-ml-devel-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux (6.0.3-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.119.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.71.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-35f6b014eff0419881bbf71f1d9d4943) 7 (Core)
[root@k8s-all ~]# grub2-set-default 0
[root@k8s-all ~]# reboot
[root@k8s-all ~]# uname -r
6.0.3-1.el7.elrepo.x86_64

• 配置内核转发及网桥过滤

[root@K8s-all ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
​
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
​
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
EOF
​
sysctl --system

• 开启IPVS

[root@K8s-all ~]# yum install ipset ipvsadm -y
[root@K8s-all ~]# vim /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
​
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_vip ip_vs_sed ip_vs_ftp nf_conntrack"
​
for kernel_module in $ipvs_modules; 
do/sbin/modinfo -F filename $kernel_module >/dev/null 2>&1if [ $? -eq 0 ]; then/sbin/modprobe $kernel_modulefi
done
​
chmod 755 /etc/sysconfig/modules/ipvs.modules
​
[root@K8s-all ~]# bash /etc/sysconfig/modules/ipvs.modules

1.4 安装docker作为Runtime

• 二进制包下载地址：Index of linux/static/stable/x86_64/

[root@K8s-all ~]# wget -c https://mirrors.ustc.edu.cn/docker-ce/linux/static/stable/x86_64/docker-25.0.3.tgz
#解压
[root@K8s-all ~]# tar xf docker-*.tgz 
#拷贝二进制文件
[root@K8s-all ~]# cp docker/* /usr/bin/

• 创建containerd的service文件

[root@K8s-all ~]# cat >/etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999[Install]
WantedBy=multi-user.target
EOF[root@k8s-all ~]# systemctl enable --now containerd.service
[root@k8s-all ~]# systemctl is-active containerd.service
active

• 准备docker的service文件

[root@K8s-all ~]# cat > /etc/systemd/system/docker.service <<EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service
​
[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
OOMScoreAdjust=-500
​
[Install]
WantedBy=multi-user.target
EOF

• 准备docker的socket文件

[root@K8s-all ~]# cat > /etc/systemd/system/docker.socket <<EOF
[Unit]
Description=Docker Socket for the API
​
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
​
[Install]
WantedBy=sockets.target
EOF

• 配置加速器

[root@K8s-all ~]# mkdir -p /etc/docker
[root@K8s-all ~]# tee /etc/docker/daemon.json <<-'EOF'
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": [
"https://dbckerproxy.com",
"https://hub-mirror.c.163.com",
"https://mirror.baidubce.com",
"https://ccr.ccs.tencentyun.com"
]
}
EOF

• 启动Docker

[root@K8s-all ~]# groupadd docker
#创建docker组[root@K8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。[root@K8s-all ~]# systemctl enable --now docker.socket
# 启用并立即启动docker.socket单元。docker.socket是一个systemd的socket单元，用于接收来自网络的Docker API请求。[root@K8s-all ~]# systemctl enable --now docker.service
# 启用并立即启动docker.service单元。docker.service是Docker守护进程的systemd服务单元。[root@K8s-all ~]# docker info
#验证

• 安装部署cri-docker

  • 注意：K8s从1.24版本后不支持docker了所以这里需要用cri-dockererd

  • 下载地址：https://github.com/Mirantis/cri-dockerd/releases/

[root@K8s-all ~]# wget -c https://mirrors.chenby.cn/https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.10/cri-dockerd-0.3.10.amd64.tgz# 解压cri-docker
[root@K8s-all ~]# tar xf cri-dockerd-*.amd64.tgz 
[root@K8s-all ~]# cp -r cri-dockerd/  /usr/bin/
[root@K8s-all ~]# chmod +x /usr/bin/cri-dockerd/cri-dockerd

• 写入启动cri-docker配置文件

[root@K8s-all ~]# cat >  /usr/lib/systemd/system/cri-docker.service <<EOF
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process[Install]
WantedBy=multi-user.target
EOF

• 写入cri-docker的socket配置文件

[root@K8s-all ~]# cat > /usr/lib/systemd/system/cri-docker.socket <<EOF
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker[Install]
WantedBy=sockets.target
EOF

• 启动cri-docker

[root@K8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。[root@K8s-all ~]# systemctl enable --now cri-docker.service
# 启用并立即启动cri-docker.service单元。cri-docker.service是cri-docker守护进程的systemd服务单元。[root@K8s-all ~]# systemctl restart cri-docker.service
# 重启cri-docker.service单元，即重新启动cri-docker守护进程。[root@K8s-all ~]# systemctl status docker.service
# 显示docker.service单元的当前状态，包括运行状态、是否启用等信息。

1.5 k8s与etcd下载及安装

仅在k8s-master-01操作

• 下载并解压k8s安装包

[root@k8s-master-01 ~]# wget -c https://mirrors.chenby.cn/https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz
[root@k8s-master-01 ~]# wget -c https://dl.k8s.io/v1.30.1/kubernetes-server-linux-amd64.tar.gz
[root@k8s-master-01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}# 命令的解释如下：
# - tar：用于处理tar压缩文件的命令。
# - -xf：表示解压操作。
# - kubernetes-server-linux-amd64.tar.gz：要解压的文件名。
# - --strip-components=3：表示解压时忽略压缩文件中的前3级目录结构，提取文件时直接放到目标目录中。
# - -C /usr/local/bin：指定提取文件的目标目录为/usr/local/bin。
# - kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}：要解压和提取的文件名模式，用花括号括起来表示模式中的多个可能的文件名。# 解压etcd安装文件
[root@k8s-master-01 ~]# tar -xf etcd*.tar.gz && mv etcd-*/etcd /usr/local/bin/ && mv etcd-*/etcdctl /usr/local/bin/

• 查看版本

[root@k8s-master-01 ~]# kubelet --version
Kubernetes v1.30.1
[root@k8s-master-01 ~]# etcdctl version
etcdctl version: 3.5.12
API version: 3.5

• 将组件发送至其他k8s节点

# 定义变量
[root@k8s-master-01 ~]# Master='k8s-master-02 k8s-master-03'
[root@k8s-master-01 ~]# Work='k8s-node-01 k8s-node-02 k8s-node-03'# 拷贝master组件
[root@k8s-master-01 ~]# for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done# 拷贝Work节点
[root@k8s-master-01 ~]# for NODE in $Work; do echo $NODE; scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done# 所有节点创建目录
[root@k8s-all ~]# mkdir -p /opt/cni/bin

1.6 相关证书生成

• k8s-master-01节点下载证书生成工具

[root@k8s-master-01 ~]# wget -c "https://mirrors.chenby.cn/https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64" -O /usr/local/bin/cfssl
[root@k8s-master-01 ~]# wget -c "https://mirrors.chenby.cn/https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64" -O /usr/local/bin/cfssljson# 添加执行权限
[root@k8s-master-01 ~]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

1.6.1 生成etcd证书

• 所有master节点创建证书存放目录

[root@k8s-master-all ~]# mkdir /etc/etcd/ssl -p

• master01节点生成etcd证书

[root@k8s-master-01 ~]# cd /etc/etcd/ssl/# 写入生成证书所需的配置文件
[root@k8s-master-01 ssl]# cat > ca-config.json << EOF 
{"signing": {"default": {"expiry": "876000h"},"profiles": {"kubernetes": {"usages": ["signing","key encipherment","server auth","client auth"],"expiry": "876000h"}}}
}
EOF[root@k8s-master-01 ssl]# cat > etcd-ca-csr.json  << EOF 
{"CN": "etcd","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "etcd","OU": "Etcd Security"}],"ca": {"expiry": "876000h"}
}
EOF[root@k8s-master-01 ssl]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca#cfssl是一个用于生成TLS/SSL证书的工具，它支持PKI、JSON格式配置文件以及与许多其他集成工具的配合使用。
# gencert参数表示生成证书的操作。-initca参数表示初始化一个CA（证书颁发机构）。CA是用于签发其他证书的根证书。etcd-ca-csr.json是一个JSON格式的配置文件，其中包含了CA的详细信息，如私钥、公钥、有效期等。这个文件提供了生成CA证书所需的信息。# 使用cfssl工具根据配置文件ca-csr.json生成一个CA证书
[root@k8s-master-01 ssl]# cat > etcd-csr.json << EOF 
{"CN": "etcd","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "etcd","OU": "Etcd Security"}]
}
EOF# 用cfssl生成etcd证书
[root@k8s-master-01 ssl]# cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master-01,k8s-master-02,k8s-master-03,192.168.110.21,192.168.110.22,192.168.110.23 \
-profile=kubernetes \
etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

• 将证书复制到其他节点

[root@k8s-master-01 ssl]# Master='k8s-master-02 k8s-master-03'
[root@k8s-master-01 ssl]# for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done

1.6.2 生成k8s相关证书

• 所有k8s节点创建证书存放目录

[root@k8s-all ~]# mkdir -p /etc/kubernetes/pki

• master01节点生成k8s证书

[root@k8s-master-01 ~]# cd /etc/kubernetes/pki/
[root@k8s-master-01 pki]# cat > ca-csr.json   << EOF 
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "Kubernetes","OU": "Kubernetes-manual"}],"ca": {"expiry": "876000h"}
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
​
[root@k8s-master-01 pki]# cat > apiserver-csr.json << EOF 
{"CN": "kube-apiserver","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "Kubernetes","OU": "Kubernetes-manual"}]
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert   \
-ca=/etc/kubernetes/pki/ca.pem   \
-ca-key=/etc/kubernetes/pki/ca-key.pem   \
-config=/etc/etcd/ssl/ca-config.json   \
-hostname=10.96.0.1,192.168.110.20,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,x.oiox.cn,k.oiox.cn,l.oiox.cn,o.oiox.cn,192.168.110.21,192.168.110.22,192.168.110.23,192.168.110.24,192.168.110.25,192.168.110.26,192.168.110.20 \
-profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

• 生成apiserver聚合证书

[root@k8s-master-01 pki]# cat > front-proxy-ca-csr.json  << EOF 
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"ca": {"expiry": "876000h"}
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca  # 生成CA证书
​
[root@k8s-master-01 pki]# cat > front-proxy-client-csr.json  << EOF 
{"CN": "front-proxy-client","key": {"algo": "rsa","size": 2048}
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert  \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem   \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   \
-config=/etc/etcd/ssl/ca-config.json   \
-profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

1.6.3 生成controller-manage的证书

• 选择使用那种高可用方案

  • 若使用 haproxy 那么为 --server=https://192.168.110.20:9443

  • 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443

[root@k8s-master-01 pki]# cat > manager-csr.json << EOF 
{"CN": "system:kube-controller-manager","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:kube-controller-manager","OU": "Kubernetes-manual"}]
}
EOF[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager# 设置一个集群项
# 若使用 haproxy、keepalived 那么为 `--server=https://192.168.110.20:9443`
# 若使用 nginx方案，那么为 `--server=https://127.0.0.1:8443`
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://127.0.0.1:8443 \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig# 设置一个环境项，一个上下文
[root@k8s-master-01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig# 设置一个用户项
[root@k8s-master-01 pki]# kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.pem \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig# 设置默认环境
[root@k8s-master-01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

1.6.4 生成kube-scheduler的证书

[root@k8s-master-01 pki]# cat > scheduler-csr.json << EOF 
{"CN": "system:kube-scheduler","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:kube-scheduler","OU": "Kubernetes-manual"}]
}
EOF[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler# 配置一个名为"kubernetes"的集群
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://127.0.0.1:8443 \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig# 设置 kube-scheduler 组件的身份验证凭据
[root@k8s-master-01 pki]# kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.pem \
--client-key=/etc/kubernetes/pki/scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig# 设置一个名为"system:kube-scheduler@kubernetes"的上下文
[root@k8s-master-01 pki]# kubectl config set-context system:kube-scheduler@kubernetes \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig# 配置Kubernetes集群中的调度器组件
[root@k8s-master-01 pki]# kubectl config use-context system:kube-scheduler@kubernetes \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

1.6.5 生成admin的证书配置

[root@k8s-master-01 pki]# cat > admin-csr.json << EOF 
{"CN": "admin","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:masters","OU": "Kubernetes-manual"}]
}
EOF
​
# 生成Kubernetes admin的证书
[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
​
# 配置一个名为"kubernetes"的集群
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     \
--server=https://127.0.0.1:8443     \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
​
# 设置 kubernetes-admin 组件的身份验证凭据
[root@k8s-master-01 pki]# kubectl config set-credentials kubernetes-admin  \
--client-certificate=/etc/kubernetes/pki/admin.pem     \
--client-key=/etc/kubernetes/pki/admin-key.pem     \
--embed-certs=true     \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
​
# 设置一个名为"kubernetes-admin@kubernetes"的上下文
[root@k8s-master-01 pki]# kubectl config set-context kubernetes-admin@kubernetes    \
--cluster=kubernetes     \
--user=kubernetes-admin     \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
​
# 配置Kubernetes集群中的调度器组件
[root@k8s-master-01 pki]# kubectl config use-context kubernetes-admin@kubernetes  --kubeconfig=/etc/kubernetes/admin.kubeconfig

1.6.6 创建kube-proxy证书

[root@k8s-master-01 pki]# cat > kube-proxy-csr.json  << EOF 
{"CN": "system:kube-proxy","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:kube-proxy","OU": "Kubernetes-manual"}]
}
EOF# 生成Kubernetes admin的证书
[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy# 配置一个名为"kubernetes"的集群
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     \
--server=https://127.0.0.1:8443     \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig# 设置 kube-proxy 组件的身份验证凭据
[root@k8s-master-01 pki]# kubectl config set-credentials kube-proxy  \
--client-certificate=/etc/kubernetes/pki/kube-proxy.pem     \
--client-key=/etc/kubernetes/pki/kube-proxy-key.pem     \
--embed-certs=true     \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig# 设置一个名为"kube-proxy@kubernetes"的上下文
[root@k8s-master-01 pki]# kubectl config set-context kube-proxy@kubernetes    \
--cluster=kubernetes     \
--user=kube-proxy     \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig# 配置Kubernetes集群中的调度器组件
[root@k8s-master-01 pki]# kubectl config use-context kube-proxy@kubernetes  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

1.6.7 创建ServiceAccount Key — secret

[root@k8s-master-01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
[root@k8s-master-01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

1.6.8 将证书发送到其他master节点

[root@k8s-master-01 pki]# for NODE in k8s-master-02 k8s-master-03; do  for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do  scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done;  for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do  scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done

1.7 K8s系统组件配置

1.7.1 etcd配置

• k8s-master-01配置

[root@k8s-master-01 ~]# cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master-01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.110.21:2380'
listen-client-urls: 'https://192.168.110.21:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.110.21:2380'
advertise-client-urls: 'https://192.168.110.21:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master-01=https://192.168.110.21:2380,k8s-master-02=https://192.168.110.22:2380,k8s-master-03=https://192.168.110.23:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
peer-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'peer-client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

• k8s-master-02配置

[root@k8s-master-02 ~]# cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master-02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.110.22:2380'
listen-client-urls: 'https://192.168.110.22:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.110.22:2380'
advertise-client-urls: 'https://192.168.110.22:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master-01=https://192.168.110.21:2380,k8s-master-02=https://192.168.110.22:2380,k8s-master-03=https://192.168.110.23:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
peer-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'peer-client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

• k8s-master-03配置

[root@k8s-master-03 ~]# cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master-03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.110.23:2380'
listen-client-urls: 'https://192.168.110.23:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.110.23:2380'
advertise-client-urls: 'https://192.168.110.23:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master-01=https://192.168.110.21:2380,k8s-master-02=https://192.168.110.22:2380,k8s-master-03=https://192.168.110.23:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
peer-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'peer-client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

1.7.2 创建service（所有master节点操作）

• 创建etcd.service并启动

[root@k8s-master-all ~]# cat > /usr/lib/systemd/system/etcd.service << EOF
​
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
​
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
​
[Install]
WantedBy=multi-user.target
Alias=etcd3.service
​
EOF

• 创建etcd证书目录

[root@k8s-master-all ~]# mkdir /etc/kubernetes/pki/etcd
[root@k8s-master-all ~]# ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
[root@k8s-master-all ~]# systemctl daemon-reload
[root@k8s-master-all ~]# systemctl enable --now etcd.service

• 查看etcd状态

[root@k8s-master-all ~]# export ETCDCTL_API=3
[root@k8s-master-all ~]# etcdctl --endpoints="192.168.110.21:2379,192.168.110.22:2379,192.168.110.23:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.110.21:2379 | 968dab376a943298 |  3.5.12 |   20 kB |      true |      false |         2 |          8 |                  8 |        |
| 192.168.110.22:2379 | 86c98fa15965d9ab |  3.5.12 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
| 192.168.110.23:2379 | 4a6158ec2cc2b579 |  3.5.12 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

1.8 高可用配置

• 在Master服务器上操作

1.8.1 NGINX高可用方案

# 安装编译环境
[root@k8s-master-01 ~]# yum install gcc -y
​
# 下载解压nginx二进制文件
[root@k8s-master-01 ~]# wget -c http://nginx.org/download/nginx-1.25.3.tar.gz
[root@k8s-master-01 ~]# tar xf nginx-*.tar.gz
[root@k8s-master-01 ~]# cd nginx-*
​
# 进行编译
[root@k8s-master-01 ~]# ./configure --with-stream --without-http --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
[root@k8s-master-01 ~]# make && make install 
​
# 拷贝编译好的nginx
[root@k8s-master-01 ~]# node='k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 k8s-node-03'
[root@k8s-master-01 ~]# for NODE in $node; do scp -r /usr/local/nginx/ $NODE:/usr/local/nginx/; done

• 写入启动配置

[root@k8s-all ~]# cat > /usr/local/nginx/conf/kube-nginx.conf <<EOF
worker_processes 1;
events {worker_connections  1024;
}
stream {upstream backend {least_conn;hash $remote_addr consistent;server 192.168.110.21:6443        max_fails=3 fail_timeout=30s;server 192.168.110.22:6443        max_fails=3 fail_timeout=30s;server 192.168.110.23:6443        max_fails=3 fail_timeout=30s;}server {listen 127.0.0.1:8443;proxy_connect_timeout 1s;proxy_pass backend;}
}
EOF
​
# 写入启动配置文件
[root@k8s-all ~]# cat > /etc/systemd/system/kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
​
[Service]
Type=forking
ExecStartPre=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx
ExecReload=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF
​
# 启动
[root@k8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。
​
[root@k8s-all ~]# systemctl enable --now kube-nginx.service
# 启用并立即启动kube-nginx.service单元。kube-nginx.service是kube-nginx守护进程的systemd服务单元。

1.8.2 配置Keepalived

[root@k8s-master-01 ~]# yum install keepalived -y
[root@k8s-master-01 ~]# cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {router_id master1script_user rootenable_script_security
}
​
vrrp_script check_nginx {script "/etc/keepalived/check_nginx.sh"interval 3fall 3rise 2
}
​
vrrp_instance Nginx {state MASTERinterface ens33virtual_router_id 51priority 200advert_int 1authentication {auth_type PASSauth_pass XCZKXY}track_script {check_nginx}virtual_ipaddress {192.168.110.20/24}
}
EOF
​
# 创建健康检测脚本
[root@k8s-master-01 ~]# cat > /etc/keepalived/check_nginx.sh<<EOF
#!/bin/sh
# nginx down
pid=`ps -C nginx --no-header | wc -l`
if [ $pid -eq 0 ]
thensystemctl start kube-nginx.servicesleep 5if [ `ps -C nginx --no-header | wc -l` -eq 0 ]thensystemctl stop kube-nginx.serviceelseexit 0fi
fi
EOF
​
[root@k8s-master-01 ~]# chmod +x /etc/keepalived/check_nginx.sh
​
[root@k8s-master-01 ~]# rsync -avz /etc/keepalived/{keepalived.conf,check_nginx.sh} k8s-master-02:/etc/keepalived/
​
[root@k8s-master-01 ~]# rsync -avz /etc/keepalived/{keepalived.conf,check_nginx.sh} k8s-master-03:/etc/keepalived/

• 其他两个节点修改Keepalivbed配置

[root@k8s-master-02 ~]# sed -i 's/MASTER/BACKUP/' /etc/keepalived/keepalived.conf 
[root@k8s-master-02 ~]# sed -i 's/200/150/' /etc/keepalived/keepalived.conf[root@k8s-master-03 ~]# sed -i 's/MASTER/BACKUP/' /etc/keepalived/keepalived.conf 
[root@k8s-master-03 ~]# sed -i 's/200/100/' /etc/keepalived/keepalived.conf# 启动服务
[root@k8s-master-all ~]# systemctl daemon-reload
[root@k8s-master-all ~]# systemctl enable --now keepalived.service

• 高可用测试

[root@k8s-master-01 ~]# ip a | grep 192.168.110.20/24inet 192.168.110.20/24 scope global secondary ens33# 模拟Keepalived宕机
​
[root@k8s-master-01 ~]# systemctl stop keepalived
​
[root@k8s-master-02 ~]# ip a | grep 192.168.110.20/24   # VIP漂移到master-02inet 192.168.110.20/24 scope global secondary ens33
[root@k8s-master-03 ~]# ip a | grep 192.168.110.20/24   # master-02宕机inet 192.168.110.20/24 scope global secondary ens33[root@k8s-master-03 ~]# ip a | grep 192.168.110.20/24inet 192.168.110.20/24 scope global secondary ens33  # VIP漂移到master-03[root@k8s-master-01 ~]# systemctl start keepalived.service  # 恢复后正常
[root@k8s-master-01 ~]# ip a | grep 192.168.110.20/24inet 192.168.110.20/24 scope global secondary ens33[root@k8s-all ~]# ping -c 2 192.168.110.20   #确保集群内部可以通讯
PING 192.168.110.20 (192.168.110.20) 56(84) bytes of data.
64 bytes from 192.168.110.20: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 192.168.110.20: icmp_seq=2 ttl=64 time=2.22 ms
​
--- 192.168.110.20 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 1.034/1.627/2.220/0.593 ms

1.9 k8s组件配置

• 所有k8s节点创建以下目录

[root@k8s-all ~]# mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

1.9.1 创建apiserver

• master-01节点配置

[root@k8s-master-01 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
​
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\--v=2  \\--allow-privileged=true  \\--bind-address=0.0.0.0  \\--secure-port=6443  \\--advertise-address=192.168.110.21 \\--service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\--service-node-port-range=30000-32767  \\--etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \\--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\--etcd-certfile=/etc/etcd/ssl/etcd.pem  \\--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\--client-ca-file=/etc/kubernetes/pki/ca.pem  \\--tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\--service-account-key-file=/etc/kubernetes/pki/sa.pub  \\--service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\--service-account-issuer=https://kubernetes.default.svc.cluster.local \\--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \--authorization-mode=Node,RBAC  \\--enable-bootstrap-token-auth=true  \\--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\--requestheader-allowed-names=aggregator  \\--requestheader-group-headers=X-Remote-Group  \\--requestheader-extra-headers-prefix=X-Remote-Extra-  \\--requestheader-username-headers=X-Remote-User \\--enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
​
[Install]
WantedBy=multi-user.target
​
EOF

• master-02节点配置

[root@k8s-master-02 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\--v=2  \\--allow-privileged=true  \\--bind-address=0.0.0.0  \\--secure-port=6443  \\--advertise-address=192.168.110.22 \\--service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\--service-node-port-range=30000-32767  \\--etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \\--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\--etcd-certfile=/etc/etcd/ssl/etcd.pem  \\--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\--client-ca-file=/etc/kubernetes/pki/ca.pem  \\--tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\--service-account-key-file=/etc/kubernetes/pki/sa.pub  \\--service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\--service-account-issuer=https://kubernetes.default.svc.cluster.local \\--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\--authorization-mode=Node,RBAC  \\--enable-bootstrap-token-auth=true  \\--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\--requestheader-allowed-names=aggregator  \\--requestheader-group-headers=X-Remote-Group  \\--requestheader-extra-headers-prefix=X-Remote-Extra-  \\--requestheader-username-headers=X-Remote-User \\--enable-aggregator-routing=true
​
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
​
[Install]
WantedBy=multi-user.target
​
EOF

• master-03节点配置

[root@k8s-master-03 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service  << EOF
​
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\--v=2  \\--allow-privileged=true  \\--bind-address=0.0.0.0  \\--secure-port=6443  \\--advertise-address=192.168.110.23 \\--service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\--service-node-port-range=30000-32767  \\--etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \\--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\--etcd-certfile=/etc/etcd/ssl/etcd.pem  \\--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\--client-ca-file=/etc/kubernetes/pki/ca.pem  \\--tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\--service-account-key-file=/etc/kubernetes/pki/sa.pub  \\--service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\--service-account-issuer=https://kubernetes.default.svc.cluster.local \\--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\--authorization-mode=Node,RBAC  \\--enable-bootstrap-token-auth=true  \\--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\--requestheader-allowed-names=aggregator  \\--requestheader-group-headers=X-Remote-Group  \\--requestheader-extra-headers-prefix=X-Remote-Extra-  \\--requestheader-username-headers=X-Remote-User \\--enable-aggregator-routing=true
​
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
​
[Install]
WantedBy=multi-user.target
​
EOF

1.9.2 启动apiserver

[root@k8s-master-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。
​
[root@k8s-master-all ~]# systemctl enable --now kube-apiserver.service
# 启用并立即启动kube-apiserver.service单元。kube-apiserver.service是kube-apiserver守护进程的systemd服务单元。
​
[root@k8s-master-all ~]# systemctl restart kube-apiserver.service
# 重启kube-apiserver.service单元，即重新启动etcd守护进程。

1.10 配置kube-controller-manager service

# 所有master节点配置，且配置相同
# 172.16.0.0/12为pod网段
[root@k8s-master-all ~]# cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\--v=2 \\--bind-address=0.0.0.0 \\--root-ca-file=/etc/kubernetes/pki/ca.pem \\--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \\--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \\--service-account-private-key-file=/etc/kubernetes/pki/sa.key \\--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \\--leader-elect=true \\--use-service-account-credentials=true \\--node-monitor-grace-period=40s \\--node-monitor-period=5s \\--controllers=*,bootstrapsigner,tokencleaner \\--allocate-node-cidrs=true \\--service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\--cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\--node-cidr-mask-size-ipv4=24 \\--node-cidr-mask-size-ipv6=120 \\--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pemRestart=always
RestartSec=10s[Install]
WantedBy=multi-user.targetEOF

• 启动kube-controller-manager

[root@k8s-master-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。[root@k8s-master-all ~]# systemctl enable --now kube-controller-manager.service
# 启用并立即启动kube-controller-manager.service单元。kube-controller-manager.service是kube-controller-manager守护进程的systemd服务单元。[root@k8s-master-all ~]# systemctl restart kube-controller-manager.service
# 重启kube-controller-manager.service单元，即重新启动etcd守护进程。

1.11 配置kube-scheduler service

[root@k8s-master-all ~]# cat > /usr/lib/systemd/system/kube-scheduler.service << EOF[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target[Service]
ExecStart=/usr/local/bin/kube-scheduler \\--v=2 \\--bind-address=0.0.0.0 \\--leader-elect=true \\--kubeconfig=/etc/kubernetes/scheduler.kubeconfigRestart=always
RestartSec=10s[Install]
WantedBy=multi-user.targetEOF

• 启动并查看服务状态

[root@k8s-master-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。[root@k8s-master-all ~]# systemctl enable --now kube-scheduler.service
# 启用并立即启动kube-scheduler.service单元。kube-scheduler.service是kube-scheduler守护进程的systemd服务单元。[root@k8s-master-all ~]# systemctl restart kube-scheduler.service
# 重启kube-scheduler.service单元，即重新启动etcd守护进程。

1.12 TLS Bootstrapping配置

1.12.1 在master01上配置

# 设置 Kubernetes 集群配置
[root@k8s-master-01 ~]# kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     --server=https://127.0.0.1:8443     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig# 设置凭证信息
[root@k8s-master-01 ~]# kubectl config set-credentials tls-bootstrap-token-user     \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig# 设置上下文信息
[root@k8s-master-01 ~]# kubectl config set-context tls-bootstrap-token-user@kubernetes     \
--cluster=kubernetes     \
--user=tls-bootstrap-token-user     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig# 设置当前上下文
[root@k8s-master-01 ~]# kubectl config use-context tls-bootstrap-token-user@kubernetes     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig[root@k8s-master-01 ~]# mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

1.12.2 查看集群状态

[root@k8s-master-01 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok # 写入bootstrap-token
[root@k8s-master-01 ~]# cat > bootstrap.secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:name: bootstrap-token-c8ad9cnamespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:description: "The default bootstrap token generated by 'kubelet '."token-id: c8ad9ctoken-secret: 2e4d610cf3e7426eusage-bootstrap-authentication: "true"usage-bootstrap-signing: "true"auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: kubelet-bootstrap
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.iokind: Groupname: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: node-autoapprove-bootstrap
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.iokind: Groupname: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: node-autoapprove-certificate-rotation
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.iokind: Groupname: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsname: system:kube-apiserver-to-kubelet
rules:- apiGroups:- ""resources:- nodes/proxy- nodes/stats- nodes/log- nodes/spec- nodes/metricsverbs:- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: system:kube-apiservernamespace: ""
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:kube-apiserver-to-kubelet
subjects:- apiGroup: rbac.authorization.k8s.iokind: Username: kube-apiserver
EOF[root@k8s-master-01 ~]# kubectl apply -f bootstrap.secret.yaml

1.13 node节点配置

• 在master01上将证书复制到node节点

[root@k8s-master-01 ~]# cd /etc/kubernetes/[root@k8s-master-01 ~]# for NODE in k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 k8s-node-03; do ssh $NODE mkdir -p /etc/kubernetes/pki; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done

1.13.1 当使用docker作为Runtime

# 所以节点操作
[root@k8s-all ~]# cat > /usr/lib/systemd/system/kubelet.service << EOF[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service[Service]
ExecStart=/usr/local/bin/kubelet \\--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig  \\--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\--config=/etc/kubernetes/kubelet-conf.yml \\--container-runtime-endpoint=unix:///run/cri-dockerd.sock  \\--node-labels=node.kubernetes.io/node= [Install]
WantedBy=multi-user.target
EOF

1.13.2 所有k8s节点创建kubelet的配置文件

[root@k8s-all ~]# cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:anonymous:enabled: falsewebhook:cacheTTL: 2m0senabled: truex509:clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:mode: Webhookwebhook:cacheAuthorizedTTL: 5m0scacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:imagefs.available: 15%memory.available: 100Minodefs.available: 10%nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF

1.13.3 启动kubelet

[root@k8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。[root@k8s-all ~]# systemctl enable --now kubelet.service
# 启用并立即启动kubelet.service单元。kubelet.service是kubelet守护进程的systemd服务单元。[root@k8s-all ~]# systemctl restart kubelet.service
# 重启kubelet.service单元，即重新启动kubelet守护进程。

1.13.4 查看集群

[root@k8s-master-01 ~]# kubectl get nodes
NAME            STATUS     ROLES    AGE     VERSION
k8s-master-01   NotReady   <none>   13m     v1.30.1
k8s-master-02   NotReady   <none>   8m53s   v1.30.1
k8s-master-03   NotReady   <none>   8m52s   v1.30.1
k8s-node-01     NotReady   <none>   5m37s   v1.30.1
k8s-node-02     NotReady   <none>   5m38s   v1.30.1
k8s-node-03     NotReady   <none>   5m39s   v1.30.1
​
# 查看容器运行时
[root@k8s-master-01 ~]# kubectl describe node | grep RuntimeContainer Runtime Version:  docker://25.0.3Container Runtime Version:  docker://25.0.3Container Runtime Version:  docker://25.0.3Container Runtime Version:  docker://25.0.3Container Runtime Version:  docker://25.0.3Container Runtime Version:  docker://25.0.3

1.14 kube-proxy配置

1.14.1将kubeconfig发送至其他节点

# master-01执行
[root@k8s-master-01 ~]# for NODE in k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 k8s-node-03; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done

1.14.2 所有k8s节点添加kube-proxy的service文件

[root@k8s-all ~]# cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-proxy \\--config=/etc/kubernetes/kube-proxy.yaml \\--cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\--v=2
Restart=always
RestartSec=10s
​
[Install]
WantedBy=multi-user.target
​
EOF

1.14.3 所有k8s节点添加kube-proxy的配置

[root@k8s-all ~]# cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:acceptContentTypes: ""burst: 10contentType: application/vnd.kubernetes.protobufkubeconfig: /etc/kubernetes/kube-proxy.kubeconfigqps: 5
clusterCIDR: 172.16.0.0/12,fc00:2222::/112
configSyncPeriod: 15m0s
conntrack:max: nullmaxPerCore: 32768min: 131072tcpCloseWaitTimeout: 1h0m0stcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:masqueradeAll: falsemasqueradeBit: 14minSyncPeriod: 0ssyncPeriod: 30s
ipvs:masqueradeAll: trueminSyncPeriod: 5sscheduler: "rr"syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF

1.14.4 启动kube-proxy

[root@k8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。
​
[root@k8s-all ~]# systemctl enable --now kube-proxy.service
# 启用并立即启动kube-proxy.service单元。kube-proxy.service是kube-proxy守护进程的systemd服务单元。
​
[root@k8s-all ~]# systemctl restart kube-proxy.service
# 重启kube-proxy.service单元，即重新启动kube-proxy守护进程。

1.15 安装网络插件

• CentOS 7 要升级libseccomp 不然 无法安装网络插件

# 升级runc
[root@k8s-master-01 ~]# wget -c https://mirrors.chenby.cn/https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64
​
[root@k8s-master-01 ~]# install -m 755 runc.amd64 /usr/local/sbin/runc
[root@k8s-master-01 ~]# cp -p /usr/local/sbin/runc  /usr/local/bin/runc
[root@k8s-master-01 ~]# cp -p /usr/local/sbin/runc  /usr/bin/runc
​
#查看当前版本
[root@k8s-master-01 ~]# rpm -qa | grep libseccomp
libseccomp-2.3.1-4.el7.x86_64

• 安装Calico

[root@k8s-master-01 ~]# wget -c https://gitee.com/kong-xiangyuxcz/svn/releases/download/V3.25.0/calico.yaml
​
# 所以节点手动拉取镜像
docker pull docker.io/calico/cni:v3.25.0
docker pull docker.io/calico/node:v3.25.0
docker pull docker.io/calico/kube-controllers:v3.25.0
​
[root@k8s-master-01 ~]# kubectl apply -f calico.yaml 
​
[root@k8s-master-01 ~]# kubectl get pod -n kube-system
NAME                                       READY   STATUS     RESTARTS   AGE
calico-kube-controllers-5b9b456c66-n9lz6   1/1     Running    0          2m52s
calico-node-49v62                          1/1     Running    0          2m52s
calico-node-64blt                          1/1     Running    0          2m52s
calico-node-668qt                          1/1     Running    0          2m52s
calico-node-9ktxk                          1/1     Running    0          2m52s
calico-node-njgvp                          1/1     Running    0          2m52s
​
[root@k8s-master-01 ~]# kubectl get node
NAME            STATUS     ROLES    AGE   VERSION
k8s-master-01   Ready      <none>   52m   v1.30.1
k8s-master-02   Ready      <none>   47m   v1.30.1
k8s-master-03   Ready      <none>   47m   v1.30.1
k8s-node-01     Ready      <none>   44m   v1.30.1
k8s-node-02     Ready      <none>   44m   v1.30.1
k8s-node-03     Ready      <none>   44m   v1.30.1

1.16 安装命令行自动补全功能

yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc