k8s使用rbd作为存储
如果需要使用rbd作为后端存储的话,需要先安装ceph-common
1. ceph集群创建rbd
需要提前在ceph集群上创建pool,然后创建image
[root@ceph01 ~]# ceph osd pool create pool01 [root@ceph01 ~]# ceph osd pool application enable pool01 rbd [root@ceph01 ~]# rbd pool init pool01 [root@ceph01 ~]# rbd create pool01/test –size 10G –image-format 2 –image-feature layerin [root@ceph01 ~]# rbd info pool01/test2. k8s编写yaml文件
apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: rbd name: rbd spec: replicas: 1 selector: matchLabels: app: rbd strategy: {} template: metadata: creationTimestamp: null labels: app: rbd spec: volumes: – name: test rbd: fsType: xfs keyring: /root/admin.keyring monitors: – 192.168.200.230:6789 pool: pool01 image: test user: admin readOnly: false containers: – image: nginx imagePullPolicy: IfNotPresent volumeMounts: – mountPath: /usr/share/nginx/html name: test name: nginx resources: {} status: {} [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE rbd-888b8b747-n56wr 1/1 Running 0 26m这个时候k8s就使用了rbd作为存储
如果这个地方一直显示ContainerCreating的话,可能是没有安装ceph-common,也可能是你的keyring或者ceph.conf没有发放到node节点,具体可以使用describe来看
2.1 进入容器查看挂载
[root@master euler]# kubectl exec -it rbd-5db4759c-nj2b4 — bash root@rbd-5db4759c-nj2b4:/# df -hT |grep /dev/rbd0 /dev/rbd0 xfs 10G 105M 9.9G 2% /usr/share/nginx/html可以看到,/dev/rbd0已经被格式化成xfs并且挂载到了/usr/share/nginx/html
2.2 进入容器修改内容
root@rbd-5db4759c-nj2b4:/usr/share/nginx# cd html/ root@rbd-5db4759c-nj2b4:/usr/share/nginx/html# ls root@rbd-5db4759c-nj2b4:/usr/share/nginx/html# echo 123 > index.html root@rbd-5db4759c-nj2b4:/usr/share/nginx/html# chmod 644 index.html root@rbd-5db4759c-nj2b4:/usr/share/nginx/html# exit [root@master euler]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rbd-5db4759c-nj2b4 1/1 Running 0 8m5s 192.168.166.131 node1 <none> <none>访问容器查看内容
[root@master euler]# curl 192.168.166.131 123内容可以正常被访问到,我们将容器删除,然后让他自己重新启动一个来看看文件是否还存在
[root@master euler]# kubectl delete pods rbd-5db4759c-nj2b4 pod “rbd-5db4759c-nj2b4” deleted [root@master euler]# kubectl get pods NAME READY STATUS RESTARTS AGE rbd-5db4759c-v9cgm 0/1 ContainerCreating 0 2s [root@master euler]# kubectl get pods -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rbd-5db4759c-v9cgm 1/1 Running 0 40s 192.168.166.132 node1 <none> <none> [root@master euler]# curl 192.168.166.132 123可以看到,也是没有问题的,这样k8s就正常的使用了rbd存储
有一个问题,那就是开发人员他们并不是很了解yaml文件里面改怎么去写挂载,每种类型的存储都是不同的写法,那有没有一种方式屏蔽底层的写法,直接告诉k8s集群我想要一个什么样的存储呢?
有的,那就是pv
3. pv使用rbd
[root@master euler]# vim pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim spec: accessModes: – ReadWriteOnce volumeMode: Block resources: requests: storage: 8Gi这里的pvc使用的是块设备,8个G,目前还没有这个pv可以给到他
具体的这里不细说,CKA里面有写
注意,这里是pvc,并不是pv,pvc就是开发人员定义想要的存储类型,大小,然后我可以根据你的pvc去给你创建pv,或者提前创建好pv你直接申领
[root@master euler]# vim pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: rbdpv spec: capacity: storage: 8Gi volumeMode: Block accessModes: – ReadWriteOnce persistentVolumeReclaimPolicy: Recycle mountOptions: – hard – nfsvers=4.1 rbd: fsType: xfs image: test keyring: /etc/ceph/ceph.client.admin.keyring monitors: – 172.16.1.33 pool: rbd readOnly: false user: admin3.1 查看pvc状态
[root@master euler]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myclaim Bound rbdpv 8Gi RWO 11s这个时候pv和就和pvc绑定上了,一个pv只能绑定一个pvc,同样,一个pvc也只能绑定一个pv
3.2 使用pvc
[root@master euler]# vim pod-pvc.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: pvc-pod name: pvc-pod spec: volumes: – name: rbd persistentVolumeClaim: claimName: myclaim readOnly: false containers: – image: nginx imagePullPolicy: IfNotPresent name: pvc-pod volumeDevices: # 因为是使用的块设备,所以这里是volumeDevices – devicePath: /dev/rbd0 name: rbd resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} ~ [root@master euler]# kubectl get pods NAME READY STATUS RESTARTS AGE pvc-pod 1/1 Running 0 2m5s rbd-5db4759c-v9cgm 1/1 Running 0 39m3.3 进入容器查看块设备
root@pvc-pod:/# ls /dev/rbd0 /dev/rbd0可以看到,现在rbd0已经存在于容器内部了
这样做我们每次创建pvc都需要创建对应的pv,我们可以使用动态制备
4. 动态制备
使用storageClass,但是目前欧拉使用的k8s太老了,所以需要下载欧拉fork的一个storageClass
[root@master ~]# git clone https://gitee.com/yftyxa/ceph-csi.git [root@master ~]# cd ceph-csi/deploy/ [root@master deploy]# ls ceph-conf.yaml csi-config-map-sample.yaml rbd cephcsi Makefile scc.yaml cephfs nfs service-monitor.yaml4.1 动态制备rbd
我们需要修改/root/ceph-csi/deploy/rbd/kubernetes/csi-config-map.yaml
# 先创建一个csi命名空间 [root@master ~]# kubectl create ns csi修改文件内容
[root@master kubernetes]# vim csi-rbdplugin-provisioner.yaml # 将第63行的内容改为false 63 – “–extra-create-metadata=false” # 修改第二个文件 [root@master kubernetes]# vim csi-config-map.yaml apiVersion: v1 kind: ConfigMap metadata: name: “ceph-csi-config” data: config.json: |- [ { “clusterID”: “c1f213ae-2de3-11ef-ae15-00163e179ce3”, “monitors”: [“172.16.1.33″,”172.16.1.32″,”172.16.1.31”] } ] 这里面的clusterID可以通过ceph -s去查看修改第三个文件
[root@master kubernetes]# vim csidriver.yaml — apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: “rbd.csi.ceph.com” spec: attachRequired: true podInfoOnMount: false # seLinuxMount: true # 将这一行注释 fsGroupPolicy: File自行编写一个文件
[root@master kubernetes]# vim csi-kms-config-map.yaml — apiVersion: v1 kind: ConfigMap metadata: name: ceph-csi-encryption-kms-config data: config-json: |- {}4.2 获取admin的key
[root@ceph001 ~]# cat /etc/ceph/ceph.client.admin.keyring [client.admin] key = AQC4QnJmng4HIhAA42s27yOflqOBNtEWDgEmkg== caps mds = “allow *” caps mgr = “allow *” caps mon = “allow *” caps osd = “allow *” AQC4QnJmng4HIhAA42s27yOflqOBNtEWDgEmkg== 只要这部分然后自行编写一个csi-secret.yaml的文件
[root@master kubernetes]# vim csi-secret.yaml apiVersion: v1 kind: Secret metadata: name: csi-secret stringData: userID: admin userKey: AQC4QnJmng4HIhAA42s27yOflqOBNtEWDgEmkg== adminID: admin adminKey: AQC4QnJmng4HIhAA42s27yOflqOBNtEWDgEmkg== [root@master kubernetes]# kubectl apply -f csi-secret.yaml -n csi secret/csi-secret created [root@master kubernetes]# cd ../../ [root@master deploy]# kubectl apply -f ceph-conf.yaml -n csi configmap/ceph-config created [root@master deploy]# cd – /root/ceph-csi/deploy/rbd/kubernetes4.3替换所有的namespace
[root@master kubernetes]# sed -i “s/namespace: default/namespace: csi/g” *.yaml4.4 部署
[root@master kubernetes]# [root@master kubernetes]# kubectl apply -f . -n csi注意:如果你的worker节点数量少于3个的话,是需要将 csi-rbdplugin-provisioner.yaml这个文件里面的replicas改小一点的。
[root@master kubernetes]# kubectl get pods -n csi NAME READY STATUS RESTARTS AGE csi-rbdplugin-cv455 3/3 Running 1 (2m14s ago) 2m46s csi-rbdplugin-pf5ld 3/3 Running 0 4m36s csi-rbdplugin-provisioner-6846c4df5f-dvqqk 7/7 Running 0 4m36s csi-rbdplugin-provisioner-6846c4df5f-nmcxf 7/7 Running 1 (2m11s ago) 4m36s5.使用动态制备
5.1 创建storageClass
[root@master rbd]# /root/ceph-csi/examples/rbd [root@master rbd]# grep -Ev “s*#|^$” storageclass.yaml — apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rbd-sc provisioner: rbd.csi.ceph.com parameters: clusterID: <cluster-id> pool: <rbd-pool-name> imageFeatures: “layering” csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: default csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: default csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: default csi.storage.k8s.io/fstype: ext4 reclaimPolicy: Delete allowVolumeExpansion: true mountOptions: – discard将这里的内容复制出来
— apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rbd-sc provisioner: rbd.csi.ceph.com parameters: clusterID: c1f213ae-2de3-11ef-ae15-00163e179ce3 pool: rbd imageFeatures: “layering” csi.storage.k8s.io/provisioner-secret-name: csi-secret csi.storage.k8s.io/provisioner-secret-namespace: csi csi.storage.k8s.io/controller-expand-secret-name: csi-secret csi.storage.k8s.io/controller-expand-secret-namespace: csi csi.storage.k8s.io/node-stage-secret-name: csi-secret csi.storage.k8s.io/node-stage-secret-namespace: csi csi.storage.k8s.io/fstype: ext4 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: – discard修改成这个样子,这里面的clusterID改成自己的,secret-name自己查一下
5.2 创建pvc
[root@master euler]# cp pvc.yaml sc-pvc.yaml [root@master euler]# vim sc-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: sc-pvc spec: accessModes: – ReadWriteOnce volumeMode: Block storageClassName: “csi-rbd-sc” resources: requests: storage: 15Gi storageClassName 可以使用 kubectl get sc查看现在我们只需要创建pvc,他就自己可以创建pv了
[root@master euler]# kubectl apply -f sc-pvc.yaml persistentvolumeclaim/sc-pvc created [root@master euler]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myclaim Bound rbdpv 8Gi RWO 111m sc-pvc Bound pvc-dfe3497f-9ed7-4961-9265-9e7242073c28 15Gi RWO csi-rbd-sc 2s回到ceph集群查看rbd
[root@ceph001 ~]# rbd ls csi-vol-56e37046-b9d7-4ef1-a534-970a766744f3 test [root@ceph001 ~]# rbd info csi-vol-56e37046-b9d7-4ef1-a534-970a766744f3 rbd image csi-vol-56e37046-b9d7-4ef1-a534-970a766744f3: size 15 GiB in 3840 objects order 22 (4 MiB objects) snapshot_count: 0 id: 38019ee708da block_name_prefix: rbd_data.38019ee708da format: 2 features: layering op_features: flags: create_timestamp: Wed Jun 19 04:55:35 2024 access_timestamp: Wed Jun 19 04:55:35 2024 modify_timestamp: Wed Jun 19 04:55:35 20245.3 将sc设为默认
如果不设置为默认的话,每次写yaml文件都需要指定sc,将sc设为默认的话就不用每次都指定了
[root@master euler]# kubectl edit sc csi-rbd-sc # 在注释里面写入这一行 annotations: storageclass.kubernetes.io/is-default-class: “true”5.4 测试默认pvc
[root@master euler]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE csi-rbd-sc (default) rbd.csi.ceph.com Retain Immediate true 29m再去查看sc就会有一个default的显示
[root@master euler]# cp sc-pvc.yaml sc-pvc1.yaml [root@master euler]# cat sc-pvc1.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: sc-pvc1 spec: accessModes: – ReadWriteOnce volumeMode: Block resources: requests: storage: 20Gi这个文件里面是没有指定storageClassName的
[root@master euler]# kubectl apply -f sc-pvc1.yaml persistentvolumeclaim/sc-pvc1 created [root@master euler]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE myclaim Bound rbdpv 8Gi RWO 138m sc-pvc Bound pvc-dfe3497f-9ed7-4961-9265-9e7242073c28 15Gi RWO csi-rbd-sc 27m sc-pvc1 Bound pvc-167cf73b-4983-4c28-aa98-bb65bb966649 20Gi RWO csi-rbd-sc 6s这样就好了
暂无评论内容