吃透 CRI / kubeadm 原理手把手搭建生产级实验集群综合实战指南 · 含 Harbor 私有仓库 · Ansible 自动化 · 完整代码一、K8s 集群部署核心原理1.1 整体架构与组件职责Kubernetes 集群由控制平面Master和工作节点Worker两部分组成。控制平面负责全局决策调度、API 入口、状态存储Worker 负责真正运行业务容器。图 1Kubernetes 集群架构与组件交互关系控制平面组件Masterkube-apiserver集群唯一入口所有组件通过它交互REST 接口无状态可水平扩展。etcd分布式键值数据库存储集群所有状态Pod、Service、配置等是集群的真相之源。kube-controller-manager运行各类控制器Node、Deployment、Service 等通过 watch apiserver 持续调和实际状态与期望状态。kube-scheduler监听未分配节点的 Pod根据资源、亲和性等规则将其调度到合适的 Worker。工作节点组件Workerkubelet节点代理接收 apiserver 指令通过 CRI 接口调用容器运行时创建 Pod并上报节点状态。kube-proxy在每个节点维护网络规则iptables/IPVS实现 Service 的负载均衡。容器运行时真正负责拉取镜像、创建和运行容器containerd / Dockercri-dockerd / CRI-O。1.2 容器运行时与 CRI 接口kubelet 并不直接操作容器而是通过CRIContainer Runtime Interface容器运行时接口这一 gRPC 标准与底层运行时通信。这使得 K8s 可以插拔不同的容器运行时。图 2kubelet → CRI → 容器运行时 → runc 的调用链路✅ containerdK8s 1.24 默认运行时轻量稳定Docker 本身底层也是它。推荐首选。 Docker cri-dockerd1.24 后 kubelet 移除了内置 dockershim需额外安装 cri-dockerd 适配器才能继续用 Docker。⚡ CRI-O专为 K8s 设计的轻量运行时去掉了 Docker 的多余功能红帽系常用。本文选择 Docker cri-dockerd 方案因为实验环境中大家对 Docker 更熟悉且 Harbor 私有仓库、镜像操作体验更好。生产环境推荐 containerd。1.3 kubeadm 部署工作流kubeadm是官方推荐的集群引导工具它把复杂的证书生成、静态 Pod 清单、配置文件等全部自动化。理解它的内部流程排障时才能有的放矢。图 3kubeadm init / join 完整工作流程kubeadm init 六步走预检查验证内核版本、端口占用、cgroup 驱动、swap 是否关闭等。生成证书在/etc/kubernetes/pki下生成 CA、apiserver、etcd 等全套证书。生成 kubeconfig为 admin、controller-manager、scheduler 生成连接 apiserver 的配置文件。生成静态 Pod 清单在/etc/kubernetes/manifests下生成 apiserver、etcd、controller-manager、scheduler 的 Pod YAML由本节点 kubelet 直接拉起这就是为什么控制平面组件以 Pod 形式运行。安装附加组件通过 apiserver 创建 CoreDNS 和 kube-proxy 的 DaemonSet/Deployment。生成 bootstrap token输出kubeadm join命令供 Worker 节点加入使用。二、从 0 搭建 K8s 实验环境手动版2.1 节点规划与环境拓扑图 4实验环境网络拓扑1 Master 2 Worker 1 Harbor主机名IP 地址配置角色harbor172.25.254.2541C1GHarbor 私有镜像仓库可选k8s-master172.25.254.1004C4GMaster 控制平面k8s-node1172.25.254.102C2GWorker 工作节点k8s-node2172.25.254.202C2GWorker 工作节点系统要求CentOS 7.9 / Rocky Linux 8 / Ubuntu 20.04 均可本文以 CentOS 7 为例。Master 建议 ≥2C4GWorker ≥2C2G否则容易因资源不足导致组件起不来。2.2 系统初始化所有节点执行所有节点以下操作在master、node1、node2上都要执行。① 关闭防火墙systemctl stop firewalld systemctl disable firewalld② 关闭 SELinuxsetenforce 0 sed -i s/^SELINUX.*/SELINUXdisabled/ /etc/selinux/config③ 关闭 SwapK8s 强制要求swapoff -a sed -i /swap/s/^/#/ /etc/fstab④ 加载内核模块与设置内核参数cat /etc/modules-load.d/k8s.conf EOF br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF modprobe br_netfilter modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr modprobe ip_vs_sh modprobe nf_conntrack cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl --system⑤ 时间同步yum install -y chrony systemctl enable --now chronyd chronyc sources⑥ 配置主机名与 hosts 解析# 在 master 上执行 hostnamectl set-hostname k8s-master # 在 node1 上执行 hostnamectl set-hostname k8s-node1 # 在 node2 上执行 hostnamectl set-hostname k8s-node2 # 所有节点配置 hosts cat /etc/hosts EOF 172.25.254.254 reg.timingleee.org harbor 172.25.254.100 k8s-master 172.25.254.10 k8s-node1 172.25.254.20 k8s-node2 EOF2.3 搭建 Harbor 私有镜像仓库可选可选Harbor 提供企业级镜像管理配合自签名证书即可在实验环境中安全使用。以下在harbor 节点172.25.254.254执行。① 安装 Dockeryum install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io mkdir -p /etc/docker cat /etc/docker/daemon.json EOF { registry-mirrors: [https://reg.timingleee.org], exec-opts: [native.cgroupdriversystemd], insecure-registries: [reg.timingleee.org] } EOF systemctl enable --now docker② 生成自签名证书mkdir -p /data/certs openssl req -newkey rsa:4096 -nodes -sha256 \ -keyout /data/certs/timingleee.org.key \ -x509 -days 3650 \ -out /data/certs/timingleee.org.crt \ -subj /CCN/STShanxi/LXian/OTimingLee Ltd/OUkubernetes/CNreg.timingleee.org③ 安装并配置 Harbor# 下载离线安装包版本可自行选择 tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/ cd /opt/harbor cp harbor.yml.tmpl harbor.yml # 编辑 harbor.yml关键配置如下 # hostname: reg.timingleee.org # certificate: /data/certs/timingleee.org.crt # private_key: /data/certs/timingleee.org.key # harbor_admin_password: Harbor12345 # data_volume: /data sed -i s/^hostname:.*/hostname: reg.timingleee.org/ harbor.yml sed -i s|^ certificate:.*| certificate: /data/certs/timingleee.org.crt| harbor.yml sed -i s|^ private_key:.*| private_key: /data/certs/timingleee.org.key| harbor.yml sed -i s/^harbor_admin_password:.*/harbor_admin_password: Harbor12345/ harbor.yml # 安装并启动 ./install.sh --with-chartmuseum # 配置 systemd 管理 cat /usr/lib/systemd/system/harbor.service EOF [Unit] DescriptionHarbor with Docker Compose Requiresdocker.service Afterdocker.service [Service] Typeoneshot WorkingDirectory/opt/harbor ExecStart/usr/bin/docker compose up -d ExecStop/usr/bin/docker compose down RemainAfterExityes [Install] WantedBymulti-user.target EOF systemctl daemon-reload systemctl enable harbor④ 所有 K8s 节点信任 Harbor 证书# 在 master / node1 / node2 上执行 mkdir -p /etc/docker/certs.d/reg.timingleee.org scp root172.25.254.254:/data/certs/timingleee.org.crt \ /etc/docker/certs.d/reg.timingleee.org/ca.crt # 同时加入系统信任 cp /etc/docker/certs.d/reg.timingleee.org/ca.crt \ /etc/pki/ca-trust/source/anchors/ update-ca-trust extract systemctl restart docker # 验证登录 docker login reg.timingleee.org -u admin -p Harbor123452.4 安装 Docker cri-dockerd所有 K8s 节点所有 K8s 节点在 master、node1、node2 上执行。① 安装 Docker CEyum install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io # 配置 cgroup 驱动为 systemd与 kubelet 一致否则会报错 mkdir -p /etc/docker cat /etc/docker/daemon.json EOF { registry-mirrors: [https://reg.timingleee.org], exec-opts: [native.cgroupdriversystemd], insecure-registries: [reg.timingleee.org] } EOF systemctl enable --now docker docker info | grep -i cgroup关键坑点Docker 的 cgroup 驱动必须与 kubelet 一致都用 systemd否则 kubelet 启动失败报misconfiguration: kubelet cgroup driver: cgroupfs is different from docker cgroup driver: systemd。② 安装 cri-dockerd 适配器# 下载二进制版本可自行选择本文用 0.3.14 wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.14/cri-dockerd-0.3.14.amd64.tgz tar zxf cri-dockerd-0.3.14.amd64.tgz install -o root -g root -m 0755 cri-dockerd/cri-dockerd /usr/local/bin/ # 创建 systemd 服务 cat /usr/lib/systemd/system/cri-docker.service EOF [Unit] DescriptionCRI Interface for Docker Application Container Engine Documentationhttps://docs.mirantis.com Afternetwork-online.target firewalld.service docker.service Wantsnetwork-online.target [Service] Typenotify ExecStart/usr/local/bin/cri-dockerd \ --container-runtime-endpoint fd:// \ --network-plugincni \ --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.9 ExecReload/bin/kill -s HUP $MAINPID TimeoutSec0 RestartSec2 Restartalways [Install] WantedBymulti-user.target EOF cat /usr/lib/systemd/system/cri-docker.socket EOF [Unit] DescriptionCRI Docker Socket for the API PartOfcri-docker.service [Socket] ListenStream%t/cri-dockerd.sock SocketMode0660 SocketUserroot SocketGroupdocker [Install] WantedBysockets.target EOF systemctl daemon-reload systemctl enable --now cri-docker systemctl status cri-docker2.5 安装 kubeadm / kubelet / kubectl所有 K8s 节点所有 K8s 节点# 配置阿里云 K8s YUM 源本文以 1.28 为例稳定且文档丰富 cat /etc/yum.repos.d/kubernetes.repo EOF [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck0 repo_gpgcheck0 gpgkeyhttps://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg excludekubelet kubeadm kubectl EOF # 安装指定版本三个组件版本必须一致 yum install -y kubelet-1.28.2 kubeadm-1.28.2 kubectl-1.28.2 --disableexcludeskubernetes # 配置 kubelet 的 cgroup 驱动为 systemd cat /var/lib/kubelet/config.yaml EOF apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd EOF # 启用 kubelet此时还不会正常运行init 后才会真正启动 systemctl enable kubelet # 配置命令补全 echo source (kubectl completion bash) ~/.bashrc echo source (kubeadm completion bash) ~/.bashrc source ~/.bashrc2.6 初始化 Master 节点仅 Master在k8s-master172.25.254.100上执行。kubeadm init \ --apiserver-advertise-address172.25.254.100 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.28.2 \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --cri-socketunix:///var/run/cri-dockerd.sock \ --v5参数说明--apiserver-advertise-addressapiserver 对外通告的 IP必须是其他节点可访问的 IP。--image-repository镜像仓库地址用阿里云代替 gcr.io避免国内拉取失败。--pod-network-cidrPod 网络网段Flannel 默认用10.244.0.0/16必须与后续 CNI 配置一致。--cri-socket指定 cri-dockerd 的 socket因为我们用的不是默认 containerd。初始化成功后会输出类似如下的kubeadm join命令务必保存kubeadm join 172.25.254.100:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx配置 kubectl 访问凭证mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config # 验证 kubectl get nodes # 此时 master 状态为 NotReady因为还没装网络插件正常如果 token 过期了可以在 master 上重新生成kubeadm token create --print-join-command2.7 Worker 节点加入集群仅 Worker在node1 和 node2上执行刚才保存的 join 命令注意加上--cri-socketkubeadm join 172.25.254.100:6443 \ --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \ --cri-socketunix:///var/run/cri-dockerd.sock加入成功后在 master 上查看kubectl get nodes # NAME STATUS ROLES AGE VERSION # k8s-master NotReady control-plane 5m v1.28.2 # k8s-node1 NotReady none 1m v1.28.2 # k8s-node2 NotReady none 1m v1.28.2所有节点都是NotReady因为还没有部署 CNI 网络插件接下来安装 Flannel。2.8 部署 Flannel 网络插件仅 MasterFlannel 是最轻量的 CNI 插件适合实验环境。在 master 上执行# 应用 Flannel YAML kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml # 国内如果拉不到镜像可以先手动拉取并重标签 # docker pull rancher/mirrored-flannelcni-flannel:v0.24.4 # docker pull rancher/mirrored-flannelcni-flannel-cni-plugin:v1.4.1-flannel1 # 查看 Pod 状态 kubectl get pods -n kube-system # 每个节点会运行一个 kube-flannel PodFlannel 默认使用10.244.0.0/16网段如果kubeadm init时--pod-network-cidr不是这个值需要修改 kube-flannel.yml 中的Network字段否则网络不通。2.9 验证集群状态# 1. 节点状态应为 Ready kubectl get nodes -o wide # 2. 所有系统 Pod 应为 Running kubectl get pods -n kube-system -o wide # 3. 部署一个测试 Pod 验证网络 kubectl create deployment nginx --imagenginx --replicas3 kubectl expose deployment nginx --port80 --typeNodePort kubectl get pods,svc # 4. 访问测试NodePort 端口随机分配 curl http://任意节点IP:NodePort # 5. 清理测试资源 kubectl delete deployment nginx kubectl delete svc nginx三、Ansible 自动化部署进阶手动部署适合学习理解原理但节点一多就很繁琐。下面给出一套完整的 Ansible 剧本可一键完成从初始化到集群就绪的全部操作。3.1 Ansible 管理机准备可复用 master 节点作为管理机也可单独一台。以下在管理机上执行。# 安装 Ansible yum install -y epel-release yum install -y ansible # 配置免密登录 ssh-keygen -t rsa -N -f ~/.ssh/id_rsa for ip in 172.25.254.100 172.25.254.10 172.25.254.20; do ssh-copy-id root$ip done3.2 Inventory 清单mkdir -p /opt/k8s-ansible cd /opt/k8s-ansible cat inventory EOF [masters] 172.25.254.100 [workers] 172.25.254.10 172.25.254.20 [k8s:children] masters workers EOF cat ansible.cfg EOF [defaults] inventory /opt/k8s-ansible/inventory remote_user root host_key_checking False [privilege_escalation] become true become_method sudo become_user root become_ask_pass false EOF # 测试连通性 ansible all -m ping3.3 剧本一系统初始化 init.ymlcat init.yml EOF - name: K8s 集群系统初始化 hosts: k8s gather_facts: true tasks: - name: 关闭防火墙 service: namefirewalld statestopped enabledno - name: 关闭 SELinux selinux: statedisabled - name: 关闭 swap shell: swapoff -a sed -i /swap/s/^/#/ /etc/fstab changed_when: false - name: 加载内核模块 modprobe: name{{ item }} statepresent loop: [br_netfilter, ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack] - name: 设置内核参数 copy: dest: /etc/sysctl.d/k8s.conf content: | net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 vm.swappiness 0 notify: 重载 sysctl - name: 时间同步 yum: namechrony statepresent - service: namechronyd statestarted enabledyes handlers: - name: 重载 sysctl shell: sysctl --system EOF ansible-playbook init.yml3.4 剧本二安装 Docker cri-dockerd docker.ymlcat docker.yml EOF - name: 安装 Docker 与 cri-dockerd hosts: k8s tasks: - name: 配置 Docker YUM 源 yum_repository: name: docker description: Docker CE baseurl: https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable gpgcheck: no - name: 安装 Docker yum: name{{ item }} statepresent loop: [docker-ce, docker-ce-cli, containerd.io] - name: 配置 daemon.json copy: dest: /etc/docker/daemon.json content: | { registry-mirrors: [https://reg.timingleee.org], exec-opts: [native.cgroupdriversystemd], insecure-registries: [reg.timingleee.org] } - name: 启动 Docker service: namedocker statestarted enabledyes - name: 下载 cri-dockerd unarchive: src: https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.14/cri-dockerd-0.3.14.amd64.tgz dest: /tmp/ remote_src: yes - name: 安装 cri-dockerd 二进制 copy: src/tmp/cri-dockerd/cri-dockerd dest/usr/local/bin/ mode0755 remote_srcyes - name: 创建 cri-docker service copy: dest: /usr/lib/systemd/system/cri-docker.service content: | [Unit] DescriptionCRI Interface for Docker Afterdocker.service [Service] Typenotify ExecStart/usr/local/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugincni --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.9 Restartalways [Install] WantedBymulti-user.target - name: 启动 cri-docker systemd: namecri-docker statestarted enabledyes daemon_reloadyes EOF ansible-playbook docker.yml3.5 剧本三安装 K8s 组件 k8s.ymlcat k8s.yml EOF - name: 安装 K8s 组件 hosts: k8s vars: k8s_version: 1.28.2 tasks: - name: 配置 K8s YUM 源 copy: dest: /etc/yum.repos.d/kubernetes.repo content: | [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck0 - name: 安装 kubeadm/kubelet/kubectl yum: name: {{ item }}-{{ k8s_version }} state: present disable_excludes: kubernetes loop: [kubelet, kubeadm, kubectl] - name: 配置 kubelet cgroup 驱动 copy: dest: /var/lib/kubelet/config.yaml content: | apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd - name: 启用 kubelet service: namekubelet enabledyes EOF ansible-playbook k8s.yml3.6 剧本四初始化 Master init-master.ymlcat init-master.yml EOF - name: 初始化 Master 节点 hosts: masters tasks: - name: 执行 kubeadm init shell: | kubeadm init \ --apiserver-advertise-address{{ ansible_default_ipv4.address }} \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.28.2 \ --pod-network-cidr10.244.0.0/16 \ --cri-socketunix:///var/run/cri-dockerd.sock args: creates: /etc/kubernetes/admin.conf register: init_result - name: 保存 init 输出 copy: content{{ init_result.stdout }} dest/root/kubeadm-init.log - name: 配置 kubectl shell: | mkdir -p $HOME/.kube cp -f /etc/kubernetes/admin.conf $HOME/.kube/config changed_when: false - name: 获取 join 命令 shell: kubeadm token create --print-join-command register: join_cmd changed_when: false - name: 输出 join 命令 debug: varjoin_cmd.stdout EOF ansible-playbook init-master.yml3.7 剧本五Worker 加入 join-worker.ymlcat join-worker.yml EOF - name: Worker 节点加入集群 hosts: workers vars_prompt: - name: join_command prompt: 请输入 kubeadm join 命令从 init-master 输出复制 private: no tasks: - name: 执行加入 shell: {{ join_command }} --cri-socketunix:///var/run/cri-dockerd.sock args: creates: /etc/kubernetes/kubelet.conf EOF ansible-playbook join-worker.yml3.8 剧本六部署 Flannel flannel.ymlcat flannel.yml EOF - name: 部署 Flannel 网络插件 hosts: masters tasks: - name: 应用 Flannel shell: kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml changed_when: false - name: 等待节点 Ready shell: kubectl get nodes | grep -c Ready register: ready_count until: ready_count.stdout | int 3 retries: 30 delay: 10 EOF ansible-playbook flannel.yml一键执行顺序init.yml → docker.yml → k8s.yml → init-master.yml → join-worker.yml → flannel.yml全部跑完集群即就绪。也可以用import_playbook合成一个总入口。四、常见问题与总结 常见排障现象原因解决kubelet 启动失败cgroup driver 不一致Docker 用 systemdkubelet 默认 cgroupfs统一为 systemd见 2.4 / 2.5 配置kubeadm init 镜像拉取超时默认从 gcr.io 拉取国内被墙加--image-repository registry.aliyuncs.com/google_containers节点一直 NotReadyCNI 网络插件未部署或网段不匹配检查 Flannel Pod 状态确认 pod-network-cidr 一致Worker join 提示 token 过期token 默认 24 小时有效master 上执行kubeadm token create --print-join-commandHarbor 镜像拉取失败 x509节点不信任自签名证书将 ca.crt 放入/etc/docker/certs.d/域名/并 update-ca-trustCoreDNS Pod 一直 Pendingmaster 默认有污点且网络未就绪安装 CNI 后自动调度如需 master 运行业务 Pod 可去除污点核心要点回顾原理层理解 K8s 架构apiserver/etcd/controller/scheduler kubelet/kube-proxy/runtime、CRI 接口的插拔设计、kubeadm init 的六步内部流程。运行时1.24 默认 containerd用 Docker 必须装 cri-dockerdcgroup 驱动必须统一为 systemd。网络CNI 插件是节点 Ready 的前提Flannel 轻量适合实验pod-network-cidr 必须前后一致。镜像国内用阿里云镜像源 Harbor 私有仓库解决 gcr.io 拉取问题。自动化Ansible 剧本把六步操作固化适合多节点和重复部署。 从 0 到 1：Kubernetes 集群部署方法原理与实战
吃透 CRI / kubeadm 原理手把手搭建生产级实验集群综合实战指南 · 含 Harbor 私有仓库 · Ansible 自动化 · 完整代码一、K8s 集群部署核心原理1.1 整体架构与组件职责Kubernetes 集群由控制平面Master和工作节点Worker两部分组成。控制平面负责全局决策调度、API 入口、状态存储Worker 负责真正运行业务容器。图 1Kubernetes 集群架构与组件交互关系控制平面组件Masterkube-apiserver集群唯一入口所有组件通过它交互REST 接口无状态可水平扩展。etcd分布式键值数据库存储集群所有状态Pod、Service、配置等是集群的真相之源。kube-controller-manager运行各类控制器Node、Deployment、Service 等通过 watch apiserver 持续调和实际状态与期望状态。kube-scheduler监听未分配节点的 Pod根据资源、亲和性等规则将其调度到合适的 Worker。工作节点组件Workerkubelet节点代理接收 apiserver 指令通过 CRI 接口调用容器运行时创建 Pod并上报节点状态。kube-proxy在每个节点维护网络规则iptables/IPVS实现 Service 的负载均衡。容器运行时真正负责拉取镜像、创建和运行容器containerd / Dockercri-dockerd / CRI-O。1.2 容器运行时与 CRI 接口kubelet 并不直接操作容器而是通过CRIContainer Runtime Interface容器运行时接口这一 gRPC 标准与底层运行时通信。这使得 K8s 可以插拔不同的容器运行时。图 2kubelet → CRI → 容器运行时 → runc 的调用链路✅ containerdK8s 1.24 默认运行时轻量稳定Docker 本身底层也是它。推荐首选。 Docker cri-dockerd1.24 后 kubelet 移除了内置 dockershim需额外安装 cri-dockerd 适配器才能继续用 Docker。⚡ CRI-O专为 K8s 设计的轻量运行时去掉了 Docker 的多余功能红帽系常用。本文选择 Docker cri-dockerd 方案因为实验环境中大家对 Docker 更熟悉且 Harbor 私有仓库、镜像操作体验更好。生产环境推荐 containerd。1.3 kubeadm 部署工作流kubeadm是官方推荐的集群引导工具它把复杂的证书生成、静态 Pod 清单、配置文件等全部自动化。理解它的内部流程排障时才能有的放矢。图 3kubeadm init / join 完整工作流程kubeadm init 六步走预检查验证内核版本、端口占用、cgroup 驱动、swap 是否关闭等。生成证书在/etc/kubernetes/pki下生成 CA、apiserver、etcd 等全套证书。生成 kubeconfig为 admin、controller-manager、scheduler 生成连接 apiserver 的配置文件。生成静态 Pod 清单在/etc/kubernetes/manifests下生成 apiserver、etcd、controller-manager、scheduler 的 Pod YAML由本节点 kubelet 直接拉起这就是为什么控制平面组件以 Pod 形式运行。安装附加组件通过 apiserver 创建 CoreDNS 和 kube-proxy 的 DaemonSet/Deployment。生成 bootstrap token输出kubeadm join命令供 Worker 节点加入使用。二、从 0 搭建 K8s 实验环境手动版2.1 节点规划与环境拓扑图 4实验环境网络拓扑1 Master 2 Worker 1 Harbor主机名IP 地址配置角色harbor172.25.254.2541C1GHarbor 私有镜像仓库可选k8s-master172.25.254.1004C4GMaster 控制平面k8s-node1172.25.254.102C2GWorker 工作节点k8s-node2172.25.254.202C2GWorker 工作节点系统要求CentOS 7.9 / Rocky Linux 8 / Ubuntu 20.04 均可本文以 CentOS 7 为例。Master 建议 ≥2C4GWorker ≥2C2G否则容易因资源不足导致组件起不来。2.2 系统初始化所有节点执行所有节点以下操作在master、node1、node2上都要执行。① 关闭防火墙systemctl stop firewalld systemctl disable firewalld② 关闭 SELinuxsetenforce 0 sed -i s/^SELINUX.*/SELINUXdisabled/ /etc/selinux/config③ 关闭 SwapK8s 强制要求swapoff -a sed -i /swap/s/^/#/ /etc/fstab④ 加载内核模块与设置内核参数cat /etc/modules-load.d/k8s.conf EOF br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF modprobe br_netfilter modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr modprobe ip_vs_sh modprobe nf_conntrack cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl --system⑤ 时间同步yum install -y chrony systemctl enable --now chronyd chronyc sources⑥ 配置主机名与 hosts 解析# 在 master 上执行 hostnamectl set-hostname k8s-master # 在 node1 上执行 hostnamectl set-hostname k8s-node1 # 在 node2 上执行 hostnamectl set-hostname k8s-node2 # 所有节点配置 hosts cat /etc/hosts EOF 172.25.254.254 reg.timingleee.org harbor 172.25.254.100 k8s-master 172.25.254.10 k8s-node1 172.25.254.20 k8s-node2 EOF2.3 搭建 Harbor 私有镜像仓库可选可选Harbor 提供企业级镜像管理配合自签名证书即可在实验环境中安全使用。以下在harbor 节点172.25.254.254执行。① 安装 Dockeryum install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io mkdir -p /etc/docker cat /etc/docker/daemon.json EOF { registry-mirrors: [https://reg.timingleee.org], exec-opts: [native.cgroupdriversystemd], insecure-registries: [reg.timingleee.org] } EOF systemctl enable --now docker② 生成自签名证书mkdir -p /data/certs openssl req -newkey rsa:4096 -nodes -sha256 \ -keyout /data/certs/timingleee.org.key \ -x509 -days 3650 \ -out /data/certs/timingleee.org.crt \ -subj /CCN/STShanxi/LXian/OTimingLee Ltd/OUkubernetes/CNreg.timingleee.org③ 安装并配置 Harbor# 下载离线安装包版本可自行选择 tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/ cd /opt/harbor cp harbor.yml.tmpl harbor.yml # 编辑 harbor.yml关键配置如下 # hostname: reg.timingleee.org # certificate: /data/certs/timingleee.org.crt # private_key: /data/certs/timingleee.org.key # harbor_admin_password: Harbor12345 # data_volume: /data sed -i s/^hostname:.*/hostname: reg.timingleee.org/ harbor.yml sed -i s|^ certificate:.*| certificate: /data/certs/timingleee.org.crt| harbor.yml sed -i s|^ private_key:.*| private_key: /data/certs/timingleee.org.key| harbor.yml sed -i s/^harbor_admin_password:.*/harbor_admin_password: Harbor12345/ harbor.yml # 安装并启动 ./install.sh --with-chartmuseum # 配置 systemd 管理 cat /usr/lib/systemd/system/harbor.service EOF [Unit] DescriptionHarbor with Docker Compose Requiresdocker.service Afterdocker.service [Service] Typeoneshot WorkingDirectory/opt/harbor ExecStart/usr/bin/docker compose up -d ExecStop/usr/bin/docker compose down RemainAfterExityes [Install] WantedBymulti-user.target EOF systemctl daemon-reload systemctl enable harbor④ 所有 K8s 节点信任 Harbor 证书# 在 master / node1 / node2 上执行 mkdir -p /etc/docker/certs.d/reg.timingleee.org scp root172.25.254.254:/data/certs/timingleee.org.crt \ /etc/docker/certs.d/reg.timingleee.org/ca.crt # 同时加入系统信任 cp /etc/docker/certs.d/reg.timingleee.org/ca.crt \ /etc/pki/ca-trust/source/anchors/ update-ca-trust extract systemctl restart docker # 验证登录 docker login reg.timingleee.org -u admin -p Harbor123452.4 安装 Docker cri-dockerd所有 K8s 节点所有 K8s 节点在 master、node1、node2 上执行。① 安装 Docker CEyum install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io # 配置 cgroup 驱动为 systemd与 kubelet 一致否则会报错 mkdir -p /etc/docker cat /etc/docker/daemon.json EOF { registry-mirrors: [https://reg.timingleee.org], exec-opts: [native.cgroupdriversystemd], insecure-registries: [reg.timingleee.org] } EOF systemctl enable --now docker docker info | grep -i cgroup关键坑点Docker 的 cgroup 驱动必须与 kubelet 一致都用 systemd否则 kubelet 启动失败报misconfiguration: kubelet cgroup driver: cgroupfs is different from docker cgroup driver: systemd。② 安装 cri-dockerd 适配器# 下载二进制版本可自行选择本文用 0.3.14 wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.14/cri-dockerd-0.3.14.amd64.tgz tar zxf cri-dockerd-0.3.14.amd64.tgz install -o root -g root -m 0755 cri-dockerd/cri-dockerd /usr/local/bin/ # 创建 systemd 服务 cat /usr/lib/systemd/system/cri-docker.service EOF [Unit] DescriptionCRI Interface for Docker Application Container Engine Documentationhttps://docs.mirantis.com Afternetwork-online.target firewalld.service docker.service Wantsnetwork-online.target [Service] Typenotify ExecStart/usr/local/bin/cri-dockerd \ --container-runtime-endpoint fd:// \ --network-plugincni \ --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.9 ExecReload/bin/kill -s HUP $MAINPID TimeoutSec0 RestartSec2 Restartalways [Install] WantedBymulti-user.target EOF cat /usr/lib/systemd/system/cri-docker.socket EOF [Unit] DescriptionCRI Docker Socket for the API PartOfcri-docker.service [Socket] ListenStream%t/cri-dockerd.sock SocketMode0660 SocketUserroot SocketGroupdocker [Install] WantedBysockets.target EOF systemctl daemon-reload systemctl enable --now cri-docker systemctl status cri-docker2.5 安装 kubeadm / kubelet / kubectl所有 K8s 节点所有 K8s 节点# 配置阿里云 K8s YUM 源本文以 1.28 为例稳定且文档丰富 cat /etc/yum.repos.d/kubernetes.repo EOF [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck0 repo_gpgcheck0 gpgkeyhttps://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg excludekubelet kubeadm kubectl EOF # 安装指定版本三个组件版本必须一致 yum install -y kubelet-1.28.2 kubeadm-1.28.2 kubectl-1.28.2 --disableexcludeskubernetes # 配置 kubelet 的 cgroup 驱动为 systemd cat /var/lib/kubelet/config.yaml EOF apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd EOF # 启用 kubelet此时还不会正常运行init 后才会真正启动 systemctl enable kubelet # 配置命令补全 echo source (kubectl completion bash) ~/.bashrc echo source (kubeadm completion bash) ~/.bashrc source ~/.bashrc2.6 初始化 Master 节点仅 Master在k8s-master172.25.254.100上执行。kubeadm init \ --apiserver-advertise-address172.25.254.100 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.28.2 \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --cri-socketunix:///var/run/cri-dockerd.sock \ --v5参数说明--apiserver-advertise-addressapiserver 对外通告的 IP必须是其他节点可访问的 IP。--image-repository镜像仓库地址用阿里云代替 gcr.io避免国内拉取失败。--pod-network-cidrPod 网络网段Flannel 默认用10.244.0.0/16必须与后续 CNI 配置一致。--cri-socket指定 cri-dockerd 的 socket因为我们用的不是默认 containerd。初始化成功后会输出类似如下的kubeadm join命令务必保存kubeadm join 172.25.254.100:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx配置 kubectl 访问凭证mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config # 验证 kubectl get nodes # 此时 master 状态为 NotReady因为还没装网络插件正常如果 token 过期了可以在 master 上重新生成kubeadm token create --print-join-command2.7 Worker 节点加入集群仅 Worker在node1 和 node2上执行刚才保存的 join 命令注意加上--cri-socketkubeadm join 172.25.254.100:6443 \ --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \ --cri-socketunix:///var/run/cri-dockerd.sock加入成功后在 master 上查看kubectl get nodes # NAME STATUS ROLES AGE VERSION # k8s-master NotReady control-plane 5m v1.28.2 # k8s-node1 NotReady none 1m v1.28.2 # k8s-node2 NotReady none 1m v1.28.2所有节点都是NotReady因为还没有部署 CNI 网络插件接下来安装 Flannel。2.8 部署 Flannel 网络插件仅 MasterFlannel 是最轻量的 CNI 插件适合实验环境。在 master 上执行# 应用 Flannel YAML kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml # 国内如果拉不到镜像可以先手动拉取并重标签 # docker pull rancher/mirrored-flannelcni-flannel:v0.24.4 # docker pull rancher/mirrored-flannelcni-flannel-cni-plugin:v1.4.1-flannel1 # 查看 Pod 状态 kubectl get pods -n kube-system # 每个节点会运行一个 kube-flannel PodFlannel 默认使用10.244.0.0/16网段如果kubeadm init时--pod-network-cidr不是这个值需要修改 kube-flannel.yml 中的Network字段否则网络不通。2.9 验证集群状态# 1. 节点状态应为 Ready kubectl get nodes -o wide # 2. 所有系统 Pod 应为 Running kubectl get pods -n kube-system -o wide # 3. 部署一个测试 Pod 验证网络 kubectl create deployment nginx --imagenginx --replicas3 kubectl expose deployment nginx --port80 --typeNodePort kubectl get pods,svc # 4. 访问测试NodePort 端口随机分配 curl http://任意节点IP:NodePort # 5. 清理测试资源 kubectl delete deployment nginx kubectl delete svc nginx三、Ansible 自动化部署进阶手动部署适合学习理解原理但节点一多就很繁琐。下面给出一套完整的 Ansible 剧本可一键完成从初始化到集群就绪的全部操作。3.1 Ansible 管理机准备可复用 master 节点作为管理机也可单独一台。以下在管理机上执行。# 安装 Ansible yum install -y epel-release yum install -y ansible # 配置免密登录 ssh-keygen -t rsa -N -f ~/.ssh/id_rsa for ip in 172.25.254.100 172.25.254.10 172.25.254.20; do ssh-copy-id root$ip done3.2 Inventory 清单mkdir -p /opt/k8s-ansible cd /opt/k8s-ansible cat inventory EOF [masters] 172.25.254.100 [workers] 172.25.254.10 172.25.254.20 [k8s:children] masters workers EOF cat ansible.cfg EOF [defaults] inventory /opt/k8s-ansible/inventory remote_user root host_key_checking False [privilege_escalation] become true become_method sudo become_user root become_ask_pass false EOF # 测试连通性 ansible all -m ping3.3 剧本一系统初始化 init.ymlcat init.yml EOF - name: K8s 集群系统初始化 hosts: k8s gather_facts: true tasks: - name: 关闭防火墙 service: namefirewalld statestopped enabledno - name: 关闭 SELinux selinux: statedisabled - name: 关闭 swap shell: swapoff -a sed -i /swap/s/^/#/ /etc/fstab changed_when: false - name: 加载内核模块 modprobe: name{{ item }} statepresent loop: [br_netfilter, ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack] - name: 设置内核参数 copy: dest: /etc/sysctl.d/k8s.conf content: | net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 vm.swappiness 0 notify: 重载 sysctl - name: 时间同步 yum: namechrony statepresent - service: namechronyd statestarted enabledyes handlers: - name: 重载 sysctl shell: sysctl --system EOF ansible-playbook init.yml3.4 剧本二安装 Docker cri-dockerd docker.ymlcat docker.yml EOF - name: 安装 Docker 与 cri-dockerd hosts: k8s tasks: - name: 配置 Docker YUM 源 yum_repository: name: docker description: Docker CE baseurl: https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable gpgcheck: no - name: 安装 Docker yum: name{{ item }} statepresent loop: [docker-ce, docker-ce-cli, containerd.io] - name: 配置 daemon.json copy: dest: /etc/docker/daemon.json content: | { registry-mirrors: [https://reg.timingleee.org], exec-opts: [native.cgroupdriversystemd], insecure-registries: [reg.timingleee.org] } - name: 启动 Docker service: namedocker statestarted enabledyes - name: 下载 cri-dockerd unarchive: src: https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.14/cri-dockerd-0.3.14.amd64.tgz dest: /tmp/ remote_src: yes - name: 安装 cri-dockerd 二进制 copy: src/tmp/cri-dockerd/cri-dockerd dest/usr/local/bin/ mode0755 remote_srcyes - name: 创建 cri-docker service copy: dest: /usr/lib/systemd/system/cri-docker.service content: | [Unit] DescriptionCRI Interface for Docker Afterdocker.service [Service] Typenotify ExecStart/usr/local/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugincni --pod-infra-container-imageregistry.aliyuncs.com/google_containers/pause:3.9 Restartalways [Install] WantedBymulti-user.target - name: 启动 cri-docker systemd: namecri-docker statestarted enabledyes daemon_reloadyes EOF ansible-playbook docker.yml3.5 剧本三安装 K8s 组件 k8s.ymlcat k8s.yml EOF - name: 安装 K8s 组件 hosts: k8s vars: k8s_version: 1.28.2 tasks: - name: 配置 K8s YUM 源 copy: dest: /etc/yum.repos.d/kubernetes.repo content: | [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled1 gpgcheck0 - name: 安装 kubeadm/kubelet/kubectl yum: name: {{ item }}-{{ k8s_version }} state: present disable_excludes: kubernetes loop: [kubelet, kubeadm, kubectl] - name: 配置 kubelet cgroup 驱动 copy: dest: /var/lib/kubelet/config.yaml content: | apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd - name: 启用 kubelet service: namekubelet enabledyes EOF ansible-playbook k8s.yml3.6 剧本四初始化 Master init-master.ymlcat init-master.yml EOF - name: 初始化 Master 节点 hosts: masters tasks: - name: 执行 kubeadm init shell: | kubeadm init \ --apiserver-advertise-address{{ ansible_default_ipv4.address }} \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.28.2 \ --pod-network-cidr10.244.0.0/16 \ --cri-socketunix:///var/run/cri-dockerd.sock args: creates: /etc/kubernetes/admin.conf register: init_result - name: 保存 init 输出 copy: content{{ init_result.stdout }} dest/root/kubeadm-init.log - name: 配置 kubectl shell: | mkdir -p $HOME/.kube cp -f /etc/kubernetes/admin.conf $HOME/.kube/config changed_when: false - name: 获取 join 命令 shell: kubeadm token create --print-join-command register: join_cmd changed_when: false - name: 输出 join 命令 debug: varjoin_cmd.stdout EOF ansible-playbook init-master.yml3.7 剧本五Worker 加入 join-worker.ymlcat join-worker.yml EOF - name: Worker 节点加入集群 hosts: workers vars_prompt: - name: join_command prompt: 请输入 kubeadm join 命令从 init-master 输出复制 private: no tasks: - name: 执行加入 shell: {{ join_command }} --cri-socketunix:///var/run/cri-dockerd.sock args: creates: /etc/kubernetes/kubelet.conf EOF ansible-playbook join-worker.yml3.8 剧本六部署 Flannel flannel.ymlcat flannel.yml EOF - name: 部署 Flannel 网络插件 hosts: masters tasks: - name: 应用 Flannel shell: kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml changed_when: false - name: 等待节点 Ready shell: kubectl get nodes | grep -c Ready register: ready_count until: ready_count.stdout | int 3 retries: 30 delay: 10 EOF ansible-playbook flannel.yml一键执行顺序init.yml → docker.yml → k8s.yml → init-master.yml → join-worker.yml → flannel.yml全部跑完集群即就绪。也可以用import_playbook合成一个总入口。四、常见问题与总结 常见排障现象原因解决kubelet 启动失败cgroup driver 不一致Docker 用 systemdkubelet 默认 cgroupfs统一为 systemd见 2.4 / 2.5 配置kubeadm init 镜像拉取超时默认从 gcr.io 拉取国内被墙加--image-repository registry.aliyuncs.com/google_containers节点一直 NotReadyCNI 网络插件未部署或网段不匹配检查 Flannel Pod 状态确认 pod-network-cidr 一致Worker join 提示 token 过期token 默认 24 小时有效master 上执行kubeadm token create --print-join-commandHarbor 镜像拉取失败 x509节点不信任自签名证书将 ca.crt 放入/etc/docker/certs.d/域名/并 update-ca-trustCoreDNS Pod 一直 Pendingmaster 默认有污点且网络未就绪安装 CNI 后自动调度如需 master 运行业务 Pod 可去除污点核心要点回顾原理层理解 K8s 架构apiserver/etcd/controller/scheduler kubelet/kube-proxy/runtime、CRI 接口的插拔设计、kubeadm init 的六步内部流程。运行时1.24 默认 containerd用 Docker 必须装 cri-dockerdcgroup 驱动必须统一为 systemd。网络CNI 插件是节点 Ready 的前提Flannel 轻量适合实验pod-network-cidr 必须前后一致。镜像国内用阿里云镜像源 Harbor 私有仓库解决 gcr.io 拉取问题。自动化Ansible 剧本把六步操作固化适合多节点和重复部署。