K8S自定义CRD

1_引入

随着 Kubernetes 生态系统的持续发展，越来越多高层次的对象将会不断涌现。比起目前使用的对象，新对象将更加专业化。

有了它们，开发者将不再需要逐一进行 Deployment、Service、configMap 等步骤，而是创建并管理一些用于表达整个应用程序或者软件服务的对象。

我们能使用自定义控制器观察高阶对象，并在这些高阶对象的基础上创建底层对象。

例如，你想在 Kubernetes 集群中运行一个 messaging 代理，只需要创建一个队列资源实例，而自定义队列控制器将自动完成所需的 Secret、Deployment 和 Service。目前，Kubernetes 已经提供了类似的自定义资源添加方式。

2_CRD 的概念

CustomResourceDefinitions（CRD）允许开发者向 Kubernetes API 服务提交 CRD 对象，即可以定义新的资源类型。在成功提交后，开发者可以通过 API 服务提交 JSON 清单或 YAML 清单来创建自定义资源，以及其他 Kubernetes 资源实例。

注意：在 Kubernetes 1.7 之前的版本中，需要通过 ThirdPartyResource 对象的方式来定义自定义资源，ThirdPartyResource 于 Kubernetes 1.8 中被 CRD 替代。

开发者可以通过创建 CRD 来创建新的对象类型。不过，如果创建的对象无法在集群中解决实际问题，那么它就是一个无效特性。通常，CRD 与所有 Kubernetes 核心资源都有一个基于自定义对象有效实现目标的控制器。

CRD 的创建流程

3_CRD 安装

CRD 添加

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: # 资源名复数.组名name: websites.extensions.example.com
spec: # API 组名group: extensions.example.comversions:- name: v1served: truestorage: trueschema:openAPIV3Schema: # 资源的结构模式，用于校验资源字段type: object # 顶层资源是一个对象properties: # 定义了资源的 spec 部分，类型为 objectspec:type: objectproperties:gitRepo:type: stringrequired: # 指定 spec 是必须字段- specscope: Namespacednames:plural: websites # 定义资源的复数形式，供 API 使用singular: website # 定义资源的单数形式kind: Website # 定义资源的 Kubernetes 对象类型（kind 字段值）

代理当前接口，主要为了跳过一些认证

kubectl proxy

发起对当前接口的监听

curl http://localhost:8001/apis/extensions.example.com/v1/websites?watch=true

测试 CRD 是否有效，创建 Website 的资源清单文件

apiVersion: extensions.example.com/v1
kind: Website
metadata:name: websitenamespace: default
spec:gitRepo: https://gitee.com/efewagtehqwedqw/website.git

虽然资源出来了，但是没有人为其做实际的动作

查看结果

Json数据模型创建：

{"type": "ADDED","object": {"apiVersion": "extensions.example.com/v1","kind": "Website","metadata": {"creationTimestamp": "2025-01-14T17:56:32Z","generation": 1,"managedFields": [{"apiVersion": "extensions.example.com/v1","fieldsType": "FieldsV1","fieldsV1": {"f:spec": {".": {},"f:gitRepo": {}}},"manager": "kubectl-create","operation": "Update","time": "2025-01-14T17:56:32Z"}],"name": "website","namespace": "default","resourceVersion": "448560","uid": "b0ef73a5-cd1c-4f18-b8d2-08e2bb612bcb"},"spec": {"gitRepo": "https://gitee.com/efewagtehqwedqw/website.git"}}
}

数据模型——删除

{"type": "DELETED","object": {"apiVersion": "extensions.example.com/v1","kind": "Website","metadata": {"creationTimestamp": "2025-01-14T18:54:00Z","generation": 1,"managedFields": [{"apiVersion": "extensions.example.com/v1","fieldsType": "FieldsV1","fieldsV1": {"f:spec": {".": {},"f:gitRepo": {}}},"manager": "kubectl-create","operation": "Update","time": "2025-01-14T18:54:00Z"}],"name": "website","namespace": "default","resourceVersion": "456357","uid": "e6467fdd-1583-4b1a-84c6-d391f0fb691b"},"spec": {"gitRepo": "https://gitee.com/efewagtehqwedqw/website.git"}}
}

我们只需要根据监听到的消息做出动作即可，自定义案例逻辑：只需要提供 Website 类型资源清单文件，我们就直接自动创建好对应的 deployment、Service，还能根据给出的 gitRepo 准备好 index.html

可以发现两个容器和保证两个容器数据一致性的 emptyDir

创建 website-controller 代码，主要逻辑就是接收到 JSON 格式消息后根据自定义逻辑请求 ApiServer

package org.example.controller;import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.example.WebsiteControllerApplication;
import org.example.models.WebsiteWatchEvent;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Flux;import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;@Slf4j
@Service
public class WebsiteController {private final WebClient webClient;@Value("${k8s.api.url}") // 用于 Kubernetes API 地址配置private String k8sApiUrl;public WebsiteController() {this.webClient = WebClient.create(); // 使用 WebClient 创建基础客户端}private final ObjectMapper objectMapper = new ObjectMapper();public void startWatching() {log.info("website-controller started.");// 开启一个新的线程处理任务Runnable runnable = this::watchWebsites;Thread thread = new Thread(runnable);thread.start();}public void watchWebsites() {// 使用响应式流的方式处理连续数据流Flux<String> flux = webClient.get().uri(k8sApiUrl + "/apis/extensions.example.com/v1/websites?watch=true").retrieve().bodyToFlux(String.class);  // 以 String 流的形式处理返回的每一部分数据flux.subscribe(eventJson -> {try {// 这里是处理每个事件的逻辑WebsiteWatchEvent eventObj = objectMapper.readValue(eventJson, WebsiteWatchEvent.class);log.info("Received event: {}: {}: {}: {}", eventObj.type, eventObj.object.getApiVersion(), eventObj.object.metadata.name, eventObj.object.spec.gitRepo);if ("ADDED".equals(eventObj.type)) {createWebsite(eventObj.object);} else if ("DELETED".equals(eventObj.type)) {deleteWebsite(eventObj.object);} else {log.warn("Unexpected event: {}", eventObj.type);}} catch (IOException e) {log.error("Error occurred while watching websites IO", e);}});}private void createWebsite(WebsiteWatchEvent.Website website) {createResource(website, "api/v1", "services", "service-template.json");createResource(website, "apis/apps/v1", "deployments", "deployment-template.json");}private void deleteWebsite(WebsiteWatchEvent.Website website) {deleteResource(website, "api/v1", "services");deleteResource(website, "apis/apps/v1", "deployments");}private void createResource(WebsiteWatchEvent.Website website, String apiGroup, String kind, String filename) {try {log.info("Creating {} with name {} in namespace {}", kind, website.metadata.name, website.metadata.namespace);// 读取模板文件InputStream inputStream = WebsiteControllerApplication.class.getClassLoader().getResourceAsStream(filename);if (inputStream == null) {log.error("Resource file not found: {}", filename);return;}String template = new String(inputStream.readAllBytes(), StandardCharsets.UTF_8);template = template.replace("[NAME]", website.metadata.name);template = template.replace("[GIT-REPO]", website.spec.gitRepo);// 发送 POST 请求String url = String.format("%s/%s/namespaces/%s/%s", k8sApiUrl, apiGroup, website.metadata.namespace, kind);HttpRequest request = HttpRequest.newBuilder().uri(URI.create(url)).POST(HttpRequest.BodyPublishers.ofString(template)).header("Content-Type", "application/json").build();HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());log.info("Resource {} created successfully.", kind);} catch (IOException | InterruptedException e) {log.error("Error creating resource", e);}}private void deleteResource(WebsiteWatchEvent.Website website, String apiGroup, String kind) {try {log.info("Deleting {} with name {} in namespace {}", kind, website.metadata.name, website.metadata.namespace);String url = String.format("%s/%s/namespaces/%s/%s/%s", k8sApiUrl, apiGroup, website.metadata.namespace, kind, website.metadata.name);HttpRequest request = HttpRequest.newBuilder().uri(URI.create(url)).DELETE().build();HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());log.info("Resource {} deleted successfully.", kind);} catch (IOException | InterruptedException e) {log.error("Error deleting resource", e);}}}

准备好的资源清单模版，deployment

{"apiVersion": "apps/v1","kind": "Deployment","metadata": {"name": "[NAME]","labels": {"webserver": "[NAME]"}},"spec": {"replicas": 1,"selector": {"matchLabels": {"webserver": "[NAME]"}},"template": {"metadata": {"name": "[NAME]","labels": {"webserver": "[NAME]"}},"spec": {"containers": [{"image": "nginx:1.27.3-alpine","name": "main","volumeMounts": [{"name": "html","mountPath": "/usr/share/nginx/html","readOnly": true}],"ports": [{"containerPort": 80,"protocol": "TCP"}]},{"image": "assigned/website:gitsync","name": "git-sync","env": [{"name": "GIT_SYNC_REPO","value": "[GIT-REPO]"},{"name": "GIT_SYNC_DEST","value": "/gitrepo"},{"name": "GIT_SYNC_BRANCH","value": "master"},{"name": "GIT_SYNC_REV","value": "FETCH_HEAD"},{"name": "GIT_SYNC_WAIT","value": "10"}],"volumeMounts": [{"name": "html","mountPath": "/gitrepo"}]}],"volumes": [{"name": "html","emptyDir": {}}]}}}
}

Service

{"apiVersion": "v1","kind": "Service","metadata": {"labels": {"webserver": "[NAME]"},"name": "[NAME]"},"spec": {"type": "NodePort","ports": [{"port": 80,"protocol": "TCP","targetPort": 80}],"selector": {"webserver": "[NAME]"}}
}

打包后封装容器镜像

FROM openjdk:11-jdk
LABEL maintainer="sy<1463476251@qq.com>"
COPY website-controller.jar /usr/local
WORKDIR /usr/local
ENTRYPOINT [ "java","-jar","website-controller.jar" ]

使用docker build命令构建镜像并分发到其他各个节点上，封装镜像已上传至 hub.docker.com，镜像名为assigned/website:controller

docker build -t assigned/website:controller .

配置 kubectl proxy 容器权限

apiVersion: v1
kind: Namespace
metadata:name: website
---
apiVersion: v1
kind: ServiceAccount
metadata:name: website-controllernamespace: website
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: website-controller
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:- kind: ServiceAccountname: website-controllernamespace: website

部署 website-controller

apiVersion: apps/v1
kind: Deployment
metadata:name: website-controllernamespace: website
spec:replicas: 1selector:matchLabels:app: website-controllertemplate:metadata:labels:app: website-controllerspec:containers:- image: assigned/website:controllerimagePullPolicy: IfNotPresentname: main- image: assigned/website:kubectl-proxyname: proxyserviceAccount: website-controllerserviceAccountName: website-controller

再次使用 Website 的资源清单文件进行创建，过一段时间后查看结果

[root@k8s-master 3]# kubectl get pod,svc,deploy -o wide
NAME                           READY   STATUS    RESTARTS      AGE     IP              NODE         NOMINATED NODE   READINESS GATES
pod/website-86bc8fb756-pzh2q   2/2     Running   0             4m36s   10.244.85.241   k8s-node01   <none>           <none>NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/kubernetes   ClusterIP   10.0.0.1      <none>        443/TCP        21d     <none>
service/website      NodePort    10.2.104.81   <none>        80:30418/TCP   4m36s   webserver=websiteNAME                      READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS      IMAGES                                         SELECTOR
deployment.apps/website   1/1     1            1           4m36s   main,git-sync   nginx:1.27.3-alpine,assigned/website:gitsync   webserver=website

在浏览器中访问 masterIP:30418 即可