kubeadm-单节点-centos

# 架构

| 集群角色 | 主机名 | IP | 系统架构 |
| --- | --- | --- | --- |
|  master主控  | master | 192.168.1.100 | CentOS7.6 |
| node节点1 | node1 | 192.168.1.101 | CentOS7.6 |
| node节点2 | node2 | 192.168.1.102 | CentOS7.6 |

# 环境初始化

三台机器分别添加解析
```asp
cat << EOF >> /etc/hosts
192.168.1.100 master
192.168.1.101 node1
192.168.1.102 node2
EOF
```

三台机器开启时间同步
```asp
systemctl enable --now chronyd
```

三台机器关闭swap分区
```asp
vim /etc/fstab
```

三台设备添加网桥过滤和地址转发功能
```asp
cat <<EOF> /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
```
三台机器重新加载配置
```asp
sysctl -p
```
三台机器加载网桥过滤模块
```asp
modprobe br_netfilter
```
三台机器查看网桥过滤模块是否加载成功
```asp
lsmod | grep br_netfilter
```

# 安装ipvs

三台机器安装ipset和ipvsadm（k8s中service有两种代理模型，基于iptables、基于ipvs，后者性能略高）
```asp
yum -y install ipset ipvsadm
```
三台机器添加需要加载的模块写入脚本
```asp
cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
```
三台机器添加执行权限
```asp
chmod +x  /etc/sysconfig/modules/ipvs.modules
```
三台机器执行脚本加载模块
```asp
/bin/bash /etc/sysconfig/modules/ipvs.modules
```

三台机器查看是否启动成功
```asp
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
```

三台服务器重启
```asp
reboot
```

# 安装docker

三台机器添加docker源
```asp
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
```
三台机器查看镜像中的docker版本
```asp
yum list docker-ce --showduplicates
```
三台机器安装docker（--setopt=obsoletes=0：指定对应版本）
```asp
yum -y install --setopt=obsoletes=0 docker-ce-18.06.3.ce-3.el7
```
三台机器查看安装的版本为：Docker version 18.06.3-ce, build d7080c1
```asp
docker -v
```
三台机器配置Docker镜像加速器
```asp
mkdir -p /etc/docker
```
```asp
cat <<EOF > /etc/docker/daemon.json
{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "registry-mirrors": ["https://kn0t2bca.mirror.aliyuncs.com"]
}
EOF
```
```asp
systemctl daemon-reload
```
三台机器启动docker
```asp
systemctl enable --now docker
```
三台机器查看docker状态
```asp
systemctl status docker
```

# 集群组件安装

三台机器添加K8s镜像源
```asp
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
```
三台机器安装kubeadm、kubelet、kubectl三个组件
```asp
yum -y install --setopt=obsoletes=0 kubeadm-1.17.4-0 kubelet-1.17.4-0 kubectl-1.17.4-0
```
三台机器配置Kubelet使用systemd作为cgroup驱动，并将Kube-Proxy模式设置改为ipvs
```asp
cat <<EOF > /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
KUBE_PROXY_MODE="ipvs"
EOF
```
三台机器设置kubelet开机自启
```asp
systemctl enable kubelet
```

# 集群镜像拉取

三台机器查看推荐镜像版本
```asp
kubeadm config images list
```

三台机器自定义镜像版本选择1.17.4
```asp
images=(
    kube-apiserver:v1.17.4
    kube-controller-manager:v1.17.4
    kube-scheduler:v1.17.4
    kube-proxy:v1.17.4
    pause:3.1
    etcd:3.4.3-0
    coredns:1.6.5
)
```
```asp
images=(
    kube-apiserver:v1.21.0
    kube-controller-manager:v1.21.0
    kube-scheduler:v1.21.0
    kube-proxy:v1.21.0
    pause:3.4.1
    etcd:3.4.13-0
    coredns:v1.8.0
)
```

三台机器批量Pull镜像并替换官方标签
```asp
for imageName in "${images[@]}" ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
    docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done
```

三台机器查看镜像
```asp
docker images
```

# Master初始化

在master节点执行：初始化集群（第一行IP修改为master服务器的实际IP）
```asp
kubeadm init \
  --apiserver-advertise-address=192.168.1.100 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.18.0 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16
```

在master节点执行：设置Kubernetes配置文件
```asp
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

查看kubeadm加入集群的token信息
```asp
kubeadm token create --print-join-command --ttl 0
```

# Node  初始化

在node1和node2执行（token信息是集群初始化之后生成，根据自己集群的输出信息在node节点执行）
```asp
kubeadm join 192.168.1.100:6443 --token fgbmpk.p2629mxjy1yd0la5     --discovery-token-ca-cert-hash
sha256:31ac28d970a3d6cca982f309e6a07391cf9ea0498de902803717ade45d45e9e2
```

查看集群信息（notready状态是网络不通）
```asp
kubectl get nodes
```
```asp
NAME     STATUS     ROLES    AGE     VERSION
master   NotReady   master   10m     v1.17.4
node1    NotReady   <none>   6m11s   v1.17.4
node2    NotReady   <none>   3m2s    v1.17.4
```

# 集群网络安装
> 在Master节点执行

K8s支持多种网络插件如flannel、calico、canal等，本次使用calico插件，k8s.1.17.4对应calico的3.14版本

下载calico.yaml
```asp
curl https://docs.projectcalico.org/v3.14/manifests/calico.yaml -O
```
执行calico.yaml
```asp
kubectl create -f calico.yaml
```

查看安装进度（ctrl+c退出）
```asp
watch kubectl get pods -n kube-system
```
```asp
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-65f8bc95db-6cm49   1/1     Running   1          3m
calico-node-72rtg                          1/1     Running   1          3m
calico-node-t9rh4                          1/1     Running   1          3m
calico-node-xc8f9                          1/1     Running   1          3m
coredns-5dbbf58dbf-czp64                   1/1     Running   1          3m
coredns-5dbbf58dbf-z85f9                   1/1     Running   1          3m
etcd-master                                1/1     Running   1          3m
kube-apiserver-master                      1/1     Running   1          3m
kube-controller-manager-master             1/1     Running   1          3m
kube-proxy-92f5q                           1/1     Running   1          3m
kube-proxy-j9g64                           1/1     Running   1          3m
kube-proxy-nbpvh                           1/1     Running   1          3m
kube-scheduler-master                      1/1     Running   1          3m
```

查看集群状态
```asp
kubectl get nodes
```
```asp
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   19m   v1.17.4
node1    Ready    <none>   18m   v1.17.4
node2    Ready    <none>   18m   v1.17.4
```

# 集群监控安装
> 在Master节点执行

下载metrics-server（v0.3.6）
```asp
wget https://github.com/kubernetes-sigs/metrics-server/archive/v0.3.6.tar.gz
```
解压进入目录
```asp
tar -zxvf v0.3.6.tar.gz && cd metrics-server-0.3.6/deploy/1.8+/
```
修改配置文件
```asp
vim metrics-server-deployment.yaml
```
```asp
    spec:
      #新增1行
      hostNetwork: true
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        #修改为国内镜像源
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
        imagePullPolicy: Always
        #新增3行
        args:
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
```
部署启动
```asp
kubectl create -f ./
```
查看服务状态
```asp
kubectl get pod -n kube-system
```
```asp
NAME                                       READY   STATUS    RESTARTS   AGE
...
metrics-server-5f55b696bd-xndm8            1/1     Running   0          60s
```
查看node资源监控
```asp
kubectl top node
```
```asp
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master   314m         7%     954Mi           25%
node1    200m         5%     482Mi           13%
node2    241m         6%     492Mi           13%
```

# 集群测试

创建pod容器nginx服务
```asp
kubectl create deployment nginx --image=nginx:1.17.1
```

创建svc暴露nginx对外端口
```asp
kubectl expose deployment nginx --port=80 --type=NodePort
```

查看pod和svc
```asp
kubectl get deployment,svc nginx
```
```asp
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           100s

NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/nginx   NodePort   10.105.117.217   <none>        80:30584/TCP   98s
```
curl验证nginx服务是否正常（集群100、101、102均能访问nginx）
```asp
curl 192.168.1.100:30735
```
![](/media/202406/2024-06-16_151124_8900090.9470536339870345.png)

删除svc
```asp
kubectl delete svc nginx
```
删除pod
```asp
kubectl delete deployment nginx
```

# 节点授权

node节点授权kubectl命令管理集群，将kube文件发送到node1节点
```asp
scp -r /root/.kube/ node1:/root/
```
node1节点执行管理命令生效
```asp
kubectl get node
```

# 集群重置

重置集群
```asp
kubeadm reset
```
删除配置文件
```asp
rm -rf /root/.kube/
rm -rf /etc/kubernetes/
rm -rf /var/lib/kubelet/
rm -rf /var/lib/dockershim
rm -rf /var/run/kubernetes
rm -rf /var/lib/cni
rm -rf /var/lib/etcd
rm -rf /etc/cni/net.d
```
清理防火墙
```asp
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
```

# 网络错误

>错误日志：
Warning  Unhealthy  24s (x196 over 29m)  kubelet  (combined from similar events): Readiness probe failed: 2024-03-22 02:39:47.813 [INFO][7095] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.51.10.4,10.51.10.5

修改calico.yaml
```asp
vim calico.yaml
```
```asp
在650行新添加配置：
----------------------------------------------------------------------
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            #声明集群网卡所在网卡名（查看自己实际的）
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens33"
            # Auto-detect the BGP IP address.
            - name: IP
----------------------------------------------------------------------
```
重建网络
```asp
kubectl apply -f calico.yaml
```