### 部署Harbor私有镜像仓库

安装docker

```shell
# 通过 wget 命令获取 docker 软件仓库信息
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
# 安装 docker-ce
yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# 开启 docker 服务
systemctl enable docker && systemctl start docker
# 官方软件源默认启用了最新的软件，您可以通过编辑软件源的方式获取各个版本的软件包。例如官方并没有将测试版本的软件源置为可用，您可以通过以下方式开启。同理可以开启各种测试版本等。
# vim /etc/yum.repos.d/docker-ce.repo
#   将[docker-ce-test]下方的enabled=0修改为enabled=1
#
# 安装指定版本的Docker-CE:
# Step 1: 查找Docker-CE的版本:
# yum list docker-ce.x86_64 --showduplicates | sort -r
#   Loading mirror speeds from cached hostfile
#   Loaded plugins: branch, fastestmirror, langpacks
#   docker-ce.x86_64            17.03.1.ce-1.el7.centos            docker-ce-stable
#   docker-ce.x86_64            17.03.1.ce-1.el7.centos            @docker-ce-stable
#   docker-ce.x86_64            17.03.0.ce-1.el7.centos            docker-ce-stable
#   Available Packages
# Step2: 安装指定版本的Docker-CE: (VERSION例如上面的17.03.0.ce.1-1.el7.centos)
# sudo yum -y install docker-ce-[VERSION]
```

下载harbor离线安装包

```shell
wget https://github.com/goharbor/harbor/releases/download/v2.13.1/harbor-offline-installer-v2.13.1.tgz
tar -xvf harbor-offline-installer-v2.13.1.tgz
```

#### http方式

```shell
cp harbor.yml.tmpl harbor.yml
vim harbor.yml
# 设置域名
hostname: harbor.local.com
# 注释https相关配置
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
#https:
  # https port for harbor, default is 443
  #port: 443
  # The path of cert and key files for nginx
  #certificate: /your/certificate/path
  #private_key: /your/private/key/path
```

#### https方式

##### **生成自签名证书**

```shell
# 克隆项目
git clone https://github.com/Fishdrowned/ssl.git
# 一键生成证书
cd ssl
./gen.cert.sh harbor.local.com # 生成harbor.local.com域名的证书
Removing dir out
Creating output structure
Done
Generating a RSA private key
...................................+++++
....+++++
writing new private key to 'out/root.key.pem'
-----
Generating RSA private key, 2048 bit long modulus (2 primes)
.............+++++
....................................+++++
e is 65537 (0x010001)
Using configuration from ./ca.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'CN'
stateOrProvinceName   :ASN.1 12:'Guangdong'
localityName          :ASN.1 12:'Guangzhou'
organizationName      :ASN.1 12:'Fishdrowned'
organizationalUnitName:ASN.1 12:'harbor.local.com'
commonName            :ASN.1 12:'*.harbor.local.com
Certificate is to be certified until Aug 12 10:49:02 2025 GMT (730 days)

Write out database with 1 new entries
Data Base Updated

Certificates are located in:
lrwxrwxrwx 1 root root 43 8月  13 18:49 /opt/ssl/out/harbor.local.com/harbor.local.com.bundle.crt -> ./20230813-1849/harbor.local.com.bundle.crt
lrwxrwxrwx 1 root root 36 8月  13 18:49 /opt/ssl/out/harbor.local.com/harbor.local.com.crt -> ./20230813-1849/harbor.local.com.crt
lrwxrwxrwx 1 root root 15 8月  13 18:49 /opt/ssl/out/harbor.local.com/harbor.local.com.key.pem -> ../cert.key.pem
lrwxrwxrwx 1 root root 11 8月  13 18:49 /opt/ssl/out/harbor.local.com/root.crt -> ../root.crt

# 拷贝证书至harbor目录
cd out/harbor.local.com/
cp harbor.local.com.crt /opt/harbor/
cp harbor.local.com.key.pem /opt/harbor/
```

##### **修改配置文件**

```shell
cp harbor.yml.tmpl harbor.yml
vim harbor.yml
# 设置域名
hostname: harbor.local.com
# 注释http相关配置
# http related config
# http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  # port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /opt/harbor/harbor.local.com.crt 
  private_key: /opt/harbor/harbor.local.com.key.pem
data_volume: /data/harbor
```

### 运行install.sh

![image-20250704005555580](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250704005555580.png)

### 访问Harbor

```shell
[root@k8s-harbor harbor]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.5    harbor.local.com
```

![image-20250703211711694](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250703211711694.png)

![image-20250703211735964](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250703211735964.png)

- 初始用户名admin
- 初始密码Harbor12345

### 创建systemd服务管理脚本

```shell
cat << 'EOF' > /lib/systemd/system/harbor.service
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /usr/local/harbor/docker-compose.yml up
ExecReload=/usr/local/bin/docker-compose -f /usr/local/harbor/docker-compose.yml restart
ExecStop=/usr/local/bin/docker-compose -f /usr/local/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF
```

```shell
systemctl enable harbor --now
```

### Docker授权访问

docker配置文件私有仓库设置

```
cat /etc/docker/daemon.json
{
    "registry-mirrors": [
        "https://mirror.ccs.tencentyun.com",
        "https://o2j0mc5x.mirror.aliyuncs.com"
    ],
    "insecure-registries": [
        "https://harbor.local.com"
    ]
}
```

重启docker

```
systemctl daemon-reload && systemctl restart docker
```

集群配置映射

```
cat << EOF >> /etc/hosts
10.0.0.5 harbor.local.com
EOF
```

**firewalld**放行443

```
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --reload
```

访问测试

```
curl -k https://harbor.local.com
```

查看服务状态

```
systemctl status harbor.service
```

登录测试

```shell
docker login harbor.local.com -u admin
```

![image-20250704011714192](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250704011714192.png)

### 推送镜像





### kubernets授权访问

创建认证secret

```shell
kubectl create secret docker-registry registry-secret --namespace=default --docker-server=harbor.local.com --docker-username=admin --docker-password=Harbor12345
```

```
[root@k8s-master ~]# cat harbor-registry-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: registry-secret
  namespace: default
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJoYXJib3IubG9jYWwuY29tIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6IkhhcmJvcjEyMzQ1IiwiZW1haWwiOiJ5b3VAZXhhbXBsZS5jb20iLCJhdXRoIjoiWVdSdGFXNDZjR0Z6YzNkdmNtUT0ifX19
```

```
kubectl apply -f harbor-registry-secret.yaml
```

查看secret

```shell
[root@k8s-master ~]# kubectl get secret registry-secret -n default
NAME              TYPE                             DATA   AGE
registry-secret   kubernetes.io/dockerconfigjson   1      11s
[root@k8s-master ~]# kubectl describe secret registry-secret -n default
Name:         registry-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/dockerconfigjson

Data
====
.dockerconfigjson:  132 bytesxxxxxxxxxx [root@k8s-master ~]# kubectl get secret registry-secret -n defaultNAME              TYPE                             DATA   AGEregistry-secret   kubernetes.io/dockerconfigjson   1      11s[root@k8s-master ~]# kubectl describe secret registry-secret -n defaultName:         registry-secretNamespace:    defaultLabels:       <none>Annotations:  <none>Type:  kubernetes.io/dockerconfigjsonData====.dockerconfigjson:  132 byteskubectl get secrets NAME              TYPE                             DATA   AGEregistry-secret   kubernetes.io/dockerconfigjson   1      9s
```

使用相应的私有registry中镜像的Pod资源的定义，即可通过imagePullSecrets字段使用此Secret对象

```yaml
apiVersion: v1
kind: Pod 
metadata:
  name: secret-imagepull-demo
  namespace: default
spec:
  imagePullSecrets:
  - name: registry-secret
  containers:
  - image: 192.168.10.14/k8s/nginx:v1
    name: myapp
```

将 CA 证书添加到 Docker 守护进程 获取 CA 证书 从 harbor.local.com 获取 CA 证书。可以通过浏览器访问 [https://harbor.local.com ](https://www.kimi.com/chat/方法 1：将 CA 证书添加到 Docker 守护进程获取 CA 证书从 harbor.local.com 获取 CA 证书。你可以通过浏览器访问 https://harbor.local.com  ，然后导出证书。或者，如果你有权限，可以从 Harbor 服务器上获取 CA 证书文件。将 CA 证书添加到 Docker 守护进程将 CA 证书复制到 Docker 守护进程的证书目录中。通常，这个目录是 /etc/docker/certs.d/harbor.local.com/。重启 Docker 服务：bash复制sudo systemctl restart docker我是用这个方法可以正常拉取harbor镜像了 我之前创建的harbor-registry-secret.yaml还有用吗)  ，然后导出证书。 或者，如果你有权限，可以从 Harbor 服务器上获取 CA 证书文件。 将 CA 证书添加到 Docker 守护进程 将 CA 证书复制到 Docker 守护进程的证书目录中。通常，这个目录是 /etc/docker/certs.d/harbor.local.com/。 重启 Docker 服务

```shell
[root@k8s-harbor harbor]# scp /usr/local/harbor/harbor.local.com.crt root@10.0.0.2:/root
The authenticity of host '10.0.0.2 (10.0.0.2)' can't be established.
ECDSA key fingerprint is SHA256:BQCBv/PyzW2XHb2Ud77URNV029tFeitQou9hNMjsRQ0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.0.0.2' (ECDSA) to the list of known hosts.
root@10.0.0.2's password:
harbor.local.com.crt                                                                                   100% 1533   609.1KB/s   00:00
```

```shell
[root@k8s-worker-1 ~]# mkdir /etc/docker/certs.d/harbor.local.com -p
[root@k8s-worker-2 ~]# mkdir /etc/docker/certs.d/harbor.local.com -p
[root@k8s-master ~] mkdir /etc/docker/certs.d/harbor.local.com/ -p
```

```shell
[root@k8s-master ~]# scp ../harbor.local.com.crt root@10.0.0.4:/etc/docker/certs.d/harbor.local.com/
root@10.0.0.4's password:
harbor.local.com.crt                                                                                   100% 1533   801.7KB/s   00:00
[root@k8s-master ~]# scp ../harbor.local.com.crt root@10.0.0.3:/etc/docker/certs.d/harbor.local.com/
root@10.0.0.3's password:
harbor.local.com.crt                                                                                   100% 1533   793.8KB/s   00:00
[root@k8s-master ~]# cp ../harbor.local.com.crt /etc/docker/certs.d/harbor.local.com/
```

```shell
[root@k8s-master kube-prometheus-0.15.0]# systemctl restart docker
[root@k8s-worker-1 ~]# systemctl restart docker
[root@k8s-worker-2 ~]# systemctl restart docker
```



Harbor私有镜像仓库部署

### Kubernetes部署开源Excalidraw实现在线绘图

创建tools命名空间

```bash
kubectl create namespace tools
```

harbor镜像推送

```
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/excalidraw/excalidraw:latest
```

```
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/excalidraw/excalidraw:latest harbor.local.com/local/excalidraw:latest
```

```
docker push harbor.local.com/local/excalidraw:latest
```

```
docker rmi swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/excalidraw/excalidraw:latest
```

编写excalidraw.yaml

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: excalidraw
  namespace: tools
  labels:
    app: excalidraw
spec:
  replicas: 1
  selector:
    matchLabels:
      app: excalidraw
  template:
    metadata:
      labels:
        app: excalidraw
    spec:
      containers:
      - name: excalidraw
        image: harbor.local.com/local/excalidraw:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        # 可选：健康检查
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: excalidraw-service
  namespace: tools
spec:
  selector:
    app: excalidraw
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  # 根据你的需求选择服务类型
  # type: ClusterIP   # 默认类型，仅在集群内部可访问
  # type: NodePort    # 通过节点IP访问
  type: NodePort
```

查看pod，svc

```
kubectl get pod,svc -n tools|grep excalidraw
pod/excalidraw-56c8757458-24jw9   1/1     Running   0          8m9s
service/excalidraw-service   NodePort   10.110.237.0   <none>        80:30364/TCP   8m9s
```

查看pod节点

```
kubectl get pod -o wide -ntools
NAME                          READY   STATUS    RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATES
excalidraw-56c8757458-24jw9   1/1     Running   0          9m25s   10.244.140.3   k8s-worker-2   <none>           <none>
```

查看节点IP

```
hostname -I
10.0.0.4 172.17.0.1 10.244.140.0
```

web访问

```
http://10.0.0.4:30364/
```

![image-20250805192031676](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250805192031676.png)

kubernetes部署在线绘图Excalidraw

kubernetes集群部署

### k8s+prometheus+grafana

kube-prometheus是一个完整的监控解决方案，可以轻松地将其部署到 Kubernetes 集群中，它包括以下内容

1. Prometheus 用于度量收集
2. Alertmanager 用于指标警报和通知
3. [Grafana](https://zhida.zhihu.com/search?content_id=237913738&content_type=Article&match_order=1&q=Grafana&zhida_source=entity) 用于图形用户界面
4. 一组特定于K8s的exporters，用作指标收集代理
5. 使用 Prometheus Operator 来简化和自动化该堆栈的设置

#### **下载**

执行`kubectl version` 查看k8s 版本，下载对应版本

```
wget https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.15.0.tar.gz
```

#### 解压

```
tar -zxvf v0.15.0.tar.gz && cd kube-prometheus-0.15.0
```

#### 修改镜像地址

网络原因，某些镜像在国内无法下载，因此需要修改地址

```shell
[root@k8s-master kube-prometheus-0.15.0]# grep "image:" manifests/*yaml
manifests/alertmanager-alertmanager.yaml:  image: quay.io/prometheus/alertmanager:v0.28.1
manifests/blackboxExporter-deployment.yaml:        image: quay.io/prometheus/blackbox-exporter:v0.26.0
manifests/blackboxExporter-deployment.yaml:        image: ghcr.io/jimmidyson/configmap-reload:v0.15.0
manifests/blackboxExporter-deployment.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.19.1
manifests/grafana-deployment.yaml:        image: grafana/grafana:12.0.1
manifests/kubeStateMetrics-deployment.yaml:        image: harbor.local.com/local/kube-state-metrics:v2.15.0
manifests/kubeStateMetrics-deployment.yaml:        image: harbor.local.com/local/kube-rbac-proxy:v0.19.1
manifests/kubeStateMetrics-deployment.yaml:        image: harbor.local.com/local/kube-rbac-proxy:v0.19.1
manifests/nodeExporter-daemonset.yaml:        image: quay.io/prometheus/node-exporter:v1.9.1
manifests/nodeExporter-daemonset.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.19.1
manifests/prometheusAdapter-deployment.yaml:        image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0
manifests/prometheusOperator-deployment.yaml:        image: quay.io/prometheus-operator/prometheus-operator:v0.83.0
manifests/prometheusOperator-deployment.yaml:        image: quay.io/brancz/kube-rbac-proxy:v0.19.1
manifests/prometheus-prometheus.yaml:  image: quay.io/prometheus/prometheus:v3.4.0
```

![image-20250710231416537](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250710231416537.png)

#### 获取需镜像列表

```
[root@k8s-master kube-prometheus-0.15.0]# grep "image:" manifests/*yaml | awk '{print $3}'
quay.io/prometheus/alertmanager:v0.28.1
quay.io/prometheus/blackbox-exporter:v0.26.0
ghcr.io/jimmidyson/configmap-reload:v0.15.0
quay.io/brancz/kube-rbac-proxy:v0.19.1
grafana/grafana:12.0.1
harbor.local.com/local/kube-state-metrics:v2.15.0
harbor.local.com/local/kube-rbac-proxy:v0.19.1
harbor.local.com/local/kube-rbac-proxy:v0.19.1
quay.io/prometheus/node-exporter:v1.9.1
quay.io/brancz/kube-rbac-proxy:v0.19.1
registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0
quay.io/prometheus-operator/prometheus-operator:v0.83.0
quay.io/brancz/kube-rbac-proxy:v0.19.1
quay.io/prometheus/prometheus:v3.4.0
```

#### 下载镜像并推送至harbor

```
拉取镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/prometheus/alertmanager:v0.28.1
标记镜像
docker tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/prometheus/alertmanager:v0.28.1 harbor.local.com/local/alertmanager:v0.28.1
推送镜像
docker push harbor.local.com/local/alertmanager:v0.28.1
删除本地
docker rmi swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/prometheus/alertmanager:v0.28.1
```

![image-20250710232648855](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250710232648855.png)

![image-20250710232249483](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250710232249483.png)

#### 修改Yaml中镜像地址为harbor地址

```
[root@k8s-master kube-prometheus-0.15.0]# grep "image:" manifests/*yaml | awk -F':' '{print $1}'
manifests/alertmanager-alertmanager.yaml
manifests/blackboxExporter-deployment.yaml
manifests/blackboxExporter-deployment.yaml
manifests/blackboxExporter-deployment.yaml
manifests/grafana-deployment.yaml
manifests/kubeStateMetrics-deployment.yaml
manifests/kubeStateMetrics-deployment.yaml
manifests/kubeStateMetrics-deployment.yaml
manifests/nodeExporter-daemonset.yaml
manifests/nodeExporter-daemonset.yaml
manifests/prometheusAdapter-deployment.yaml
manifests/prometheusOperator-deployment.yaml
manifests/prometheusOperator-deployment.yaml
manifests/prometheus-prometheus.yaml
```

使用创建的harbor-registry-secret拉取 harbor.local.com上的私有镜像，需要修改yaml文件，在 spec部分添加 imagePullSecrets字段

```yaml
  imagePullSecrets:
  - name: registry-secret
```

![image-20250710235810720](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250710235810720.png)

#### 查看镜像地址

```
[root@k8s-master kube-prometheus-0.15.0]# grep "image:" manifests/*yaml
manifests/alertmanager-alertmanager.yaml:  image: harbor.local.com/local/alertmanager:v0.28.1
manifests/blackboxExporter-deployment.yaml:        image: harbor.local.com/local/blackbox-exporter:v0.26.0
manifests/blackboxExporter-deployment.yaml:        image: harbor.local.com/local/configmap-reload:v0.15.0
manifests/blackboxExporter-deployment.yaml:        image: harbor.local.com/local/kube-rbac-proxy:v0.19.1
manifests/grafana-deployment.yaml:        image: harbor.local.com/local/grafana:12.0.1-security-01
manifests/kubeStateMetrics-deployment.yaml:        image: harbor.local.com/local/kube-state-metrics:v2.15.0
manifests/kubeStateMetrics-deployment.yaml:        image: harbor.local.com/local/kube-rbac-proxy:v0.19.1
manifests/kubeStateMetrics-deployment.yaml:        image: harbor.local.com/local/kube-rbac-proxy:v0.19.1
manifests/nodeExporter-daemonset.yaml:        image: harbor.local.com/local/node-exporter:v1.9.1
manifests/nodeExporter-daemonset.yaml:        image: harbor.local.com/local/kube-rbac-proxy:v0.19.1
manifests/prometheusAdapter-deployment.yaml:        image: harbor.local.com/local/prometheus-adapter:v0.12.0
manifests/prometheusOperator-deployment.yaml:        image: harbor.local.com/local/prometheus-operator:v0.83.0
manifests/prometheusOperator-deployment.yaml:        image: harbor.local.com/local/kube-rbac-proxy:v0.19.1
manifests/prometheus-prometheus.yaml:  image: harbor.local.com/local/prometheus:v3.4.0
```

![image-20250710235906472](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250710235906472.png)

#### 访问配置

为了可以从外部访问 `Prometheus`、`Grafana`、`Alertmanager`，需要修改 `service` 类型为 `NodePort` 类型。

##### 修改 Prometheus 的 service

```
[root@k8s-master kube-prometheus-0.15.0]# cat manifests/prometheus-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 3.4.0
  name: prometheus-k8s
  namespace: monitoring
spec:
  ports:
  - name: web
    nodePort: 31922		#增加
    port: 9090
    targetPort: web
  - name: reloader-web
    nodePort: 30981
    port: 8080
    targetPort: reloader-web
  selector:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
  sessionAffinity: ClientIP
  type: NodePort	#增加
```

##### 修改 Grafana 的 service

```
[root@k8s-master kube-prometheus-0.15.0]# cat manifests/grafana-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 12.0.1
  name: grafana
  namespace: monitoring
spec:
  ports:
  - name: http
    nodePort: 30300		#增加
    port: 3000
    targetPort: http
  selector:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/name: grafana
    app.kubernetes.io/part-of: kube-prometheus
  type: NodePort	#增加
```

##### 修改 Alertmanager 的 service

```
 cat manifests/alertmanager-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: alert-router
    app.kubernetes.io/instance: main
    app.kubernetes.io/name: alertmanager
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.28.1
  name: alertmanager-main
  namespace: monitoring
spec:
  ports:
  - name: web
    nodePort: 30200		#增加
    port: 9093
    targetPort: web
  - name: reloader-web
    nodePort: 31160
    port: 8080
    targetPort: reloader-web
  selector:
    app.kubernetes.io/component: alert-router
    app.kubernetes.io/instance: main
    app.kubernetes.io/name: alertmanager
    app.kubernetes.io/part-of: kube-prometheus
  sessionAffinity: ClientIP
  type: NodePort	#增加
```

#### 安装

在kube-prometheus-0.15.0目录下执行以下命令进行安装

```text
kubectl apply --server-side -f manifests/setup
```

![image-20250711000640212](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711000640212.png)

```
[root@k8s-master kube-prometheus-0.15.0]# kubectl apply -f manifests/
alertmanager.monitoring.coreos.com/main created
networkpolicy.networking.k8s.io/alertmanager-main created
poddisruptionbudget.policy/alertmanager-main created
prometheusrule.monitoring.coreos.com/alertmanager-main-rules created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager-main created
clusterrole.rbac.authorization.k8s.io/blackbox-exporter created
clusterrolebinding.rbac.authorization.k8s.io/blackbox-exporter created
configmap/blackbox-exporter-configuration created
deployment.apps/blackbox-exporter created
networkpolicy.networking.k8s.io/blackbox-exporter created
service/blackbox-exporter created
serviceaccount/blackbox-exporter created
servicemonitor.monitoring.coreos.com/blackbox-exporter created
secret/grafana-config created
secret/grafana-datasources created
configmap/grafana-dashboard-alertmanager-overview created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-grafana-overview created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-multicluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-windows-cluster created
configmap/grafana-dashboard-k8s-resources-windows-namespace created
configmap/grafana-dashboard-k8s-resources-windows-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-k8s-windows-cluster-rsrc-use created
configmap/grafana-dashboard-k8s-windows-node-rsrc-use created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes-aix created
configmap/grafana-dashboard-nodes-darwin created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
networkpolicy.networking.k8s.io/grafana created
prometheusrule.monitoring.coreos.com/grafana-rules created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
prometheusrule.monitoring.coreos.com/kube-prometheus-rules created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
networkpolicy.networking.k8s.io/kube-state-metrics created
prometheusrule.monitoring.coreos.com/kube-state-metrics-rules created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
prometheusrule.monitoring.coreos.com/kubernetes-monitoring-rules created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
networkpolicy.networking.k8s.io/node-exporter created
prometheusrule.monitoring.coreos.com/node-exporter-rules created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
networkpolicy.networking.k8s.io/prometheus-k8s created
poddisruptionbudget.policy/prometheus-k8s created
prometheus.monitoring.coreos.com/k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-prometheus-rules created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-k8s created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
networkpolicy.networking.k8s.io/prometheus-adapter created
poddisruptionbudget.policy/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
servicemonitor.monitoring.coreos.com/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
networkpolicy.networking.k8s.io/prometheus-operator created
prometheusrule.monitoring.coreos.com/prometheus-operator-rules created
service/prometheus-operator created
serviceaccount/prometheus-operator created
servicemonitor.monitoring.coreos.com/prometheus-operator created
```

访问monitoring 空间，查看部署状态

```shell
kubectl get pods -n monitoring
```

![image-20250711012351802](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711012351802.png)

![image-20250711013056877](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711013056877.png)

![image-20250711013249825](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711013249825.png)

#### Prometheus验证

选一台 node 节点ip+31922，即可访问prometheus的 Web UI

![image-20250711014628381](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711014628381.png)

#### Alertmanager验证

选一台 node 节点ip+30200，即可访问alertmanager的 Web UI，可以看到有一些报警，由于alertmanager的报警配置比较复杂同时对国内的通讯工具支持有限，因此可以使用`PrometheusAlert`进行告警配置

![image-20250711014721502](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711014721502.png)

#### Grafana验证

选一台 node 节点ip+30300，即可访问grafana的 Web UI，默认用户名密码：admin/admin，登录会提示更改密码Admin12345，登录以后，可以看到已经内置了不少监控大盘

![image-20250711021720241](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711021720241.png)

![image-20250711022220138](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711022220138.png)

Grafana密码重置

```
[root@k8s-master kube-prometheus-0.15.0]# kubectl exec -it -nmonitoring grafana-9988b94d4-rwngx -- grafana-cli admin reset-admin-password Admin@12345
```

![image-20250711095456721](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711095456721.png)

如果出现无法访问grafan、prometheus以及alertmanger，因为prometheus operator内部默认配置了NetworkPolicy，需要删除对应的资源，才可以通过外网访问
kubectl delete -f manifests/prometheus-networkPolicy.yaml
kubectl delete -f manifests/grafana-networkPolicy.yaml
kubectl delete -f manifests/alertmanager-networkPolicy.yaml

部署后dashboard监控指标不显示

```shell
[root@k8s-master kube-prometheus-0.15.0]# kubectl delete -f ../metrics-server.yaml
[root@k8s-master kube-prometheus-0.15.0]# kubectl apply  -f ../metrics-server.yaml
```

![image-20250711025831906](C:\Users\liutaoang\AppData\Roaming\Typora\typora-user-images\image-20250711025831906.png)

kubernetes部署prometheus+grafana

### 集群升级

[升级 kubeadm 集群 |Kubernetes](https://v1-33.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)

yum仓库替换版本号，集群所有节点

```
sed -i 's/v1.32/v1.33/g' /etc/yum.repos.d/kubernetes.repo
```

### Master节点

**更新 kubeadm**

```
yum install -y kubeadm-'1.33.4-*' --disableexcludes=kubernetes
```

**检查升级计划**

```
kubeadm upgrade plan
```

**执行升级命令**

```
kubeadm upgrade apply v1.33.4
```

```
[root@k8s-master ~]# kubeadm upgrade apply v1.33.4
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
        [WARNING SystemVerification]: cgroups v1 support is in maintenance mode, please migrate to cgroups v2
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.33.4"
[upgrade/versions] Cluster version: v1.32.6
[upgrade/versions] kubeadm version: v1.33.4
[upgrade] Are you sure you want to proceed? [y/N]: y
。。。
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.4".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
```

```
yum install -y kubelet-'1.33.4-*' kubectl-'1.33.4-*' --disableexcludes=kubernetes
```

```
systemctl daemon-reload && systemctl restart kubelet
```

### Worker 节点

```
yum install -y kubelet-'1.33.4-*' kubectl-'1.33.4-*' --disableexcludes=kubernetes
```

```
[root@k8s-master ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master     Ready    control-plane   65d   v1.33.4
k8s-worker-1   Ready    worker          65d   v1.32.6
k8s-worker-2   Ready    worker          65d   v1.32.6
```

```
kubeadm upgrade node
```

```
[root@k8s-worker-1 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running pre-flight checks
        [WARNING SystemVerification]: cgroups v1 support is in maintenance mode, please migrate to cgroups v2
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubeconfig] Skipping phase. Not a control plane node.
W0829 12:35:18.060987  102454 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config3221878225 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config3221878225/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/addon] Skipping the addon/coredns phase. Not a control plane node.
[upgrade/addon] Skipping the addon/kube-proxy phase. Not a control plane node.
```

```shell
systemctl daemon-reload && systemctl restart kubelet
```

```bash
[root@k8s-master ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
k8s-master     Ready    control-plane   65d   v1.33.4
k8s-worker-1   Ready    worker          65d   v1.33.4
k8s-worker-2   Ready    worker          65d   v1.33.4
```

