Container中的文件在磁盘上是临时存放的，这会给Container中运行的较重要的应用程序带来问题，问题之一是当容器崩溃时文件丢失。 kubelet 会重新启动容器，但容器会以干净的状态重启。 第二个问题会在同一 Pod 中运行多个容器并共享文件时出现。 kubernetes的volume解决了该问题。
——kubernetes官方文档
kubernetes支持多种类型的volume,其中较为常用的为secret、emptyDir、condfigMap、hostPath以及persistentVolumeClaim,使用场景与语法可通过k8s官网进行了解，本次旨在测试上述几种volume 权限的问题。
环境描述： 集群信息
ip
系统版本
内核版本
硬件资源
k8s版本
192.9.180.42
CentOS 7.5.1804
3.10.0-862.el7.x86_64
8c 16g
v1.17.0
服务镜像 Dockerfile:
1 2 3 4 5 FROM library/tomcat:7 RUN groupadd -g 7295 slions && useradd -g 7295 -u 7295 -d /slions slions USER slionsCMD [ "catalina.sh" ,"run" ] WORKDIR /slions
基于该Dockerfile构建tomcat:7-sjy3镜像供后续的实验服务使用。
secret secret 卷用来给 Pod 传递敏感信息，例如密码。secret 卷由 tmpfs（基于 RAM 的文件系统）提供存储，因此它们永远不会被写入持久化存储。
Secret:
1 2 3 4 5 6 7 8 apiVersion: v1 kind: Secret metadata: name: slions-secret type: Opaque data: username: c2xpb25z password: c2xpb25zLWt1YmVybmV0ZXM=
Deployment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 apiVersion: apps/v1 kind: Deployment metadata: generation: 1 labels: app: sjy name: sjy namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: sjy strategy: rollingUpdate: maxSurge: 25 % maxUnavailable: 25 % type: RollingUpdate template: metadata: labels: app: sjy name: sjy spec: containers: - env: - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: SERVICE_NAME value: sjy image: library/tomcat:7-sjy3 imagePullPolicy: Always name: sjy ports: - containerPort: 80 name: ll protocol: TCP resources: limits: cpu: 500m memory: 500Mi requests: cpu: 200m memory: 200Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /testsecret name: test-secret dnsPolicy: ClusterFirst imagePullSecrets: - name: default-image-secret restartPolicy: Always schedulerName: default-scheduler terminationGracePeriodSeconds: 30 volumes: - name: test-secret secret: secretName: slions-secret
分别创建上述的Secret与Deployment资源，进入Pod内，查看secret的权限。
1 2 3 4 5 6 7 8 9 $ kubectl get pods |grep sjy sjy-75896cb868-9qs6p 1/1 Running 0 4m40s $ kubectl exec -it sjy-75896cb868-9qs6p bash slions@sjy-75896cb868-9qs6p:~$ ls -l /testsecret/ total 0 lrwxrwxrwx 1 root root 15 Jun 14 02:50 password -> ..data/password lrwxrwxrwx 1 root root 15 Jun 14 02:50 username -> ..data/username slions@sjy-75896cb868-9qs6p:~$ cat /proc/mounts |grep testsecret tmpfs /testsecret tmpfs ro,relatime 0 0
当前secret属主属组都为root，且默认被设置为了只读。
将Deployment进行如下修改：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 apiVersion: apps/v1 kind: Deployment ... spec: ... template: ... spec: containers: ... name: sjy volumeMounts: - mountPath: /testsecret name: test-secret ... securityContext: fsGroup: 7295 ... volumes: - name: test-secret secret: secretName: slions-secret
securityContext.fsGroup 是kubernetes提供的Pod安全策略的选项，符合条件的volume可被kubelet更改为指定的属组。
更新Deployment，验证是否生效：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 slions@sjy-658c5786f6-5879p:~$ ls -l / ... drwxrwsrwt 3 root slions 120 Jun 14 02:57 testsecret drwxrwxrwt 1 root root 31 Jun 14 02:57 tmp drwxr-xr-x 1 root root 19 Oct 11 2018 usr drwxr-xr-x 1 root root 17 Oct 11 2018 var slions@sjy-658c5786f6-5879p:~$ ls -la /testsecret/ total 0 drwxrwsrwt 3 root slions 120 Jun 14 02:57 . drwxr-xr-x 1 root root 57 Jun 14 02:57 .. drwxr-sr-x 2 root slions 80 Jun 14 02:57 ..2022_06_14_02_57_40.151713228 lrwxrwxrwx 1 root root 31 Jun 14 02:57 ..data -> ..2022_06_14_02_57_40.151713228 lrwxrwxrwx 1 root root 15 Jun 14 02:57 password -> ..data/password lrwxrwxrwx 1 root root 15 Jun 14 02:57 username -> ..data/username slions@sjy-658c5786f6-5879p:~$ ls -la /testsecret/..data/ total 8 drwxr-sr-x 2 root slions 80 Jun 14 02:57 . drwxrwsrwt 3 root slions 120 Jun 14 02:57 .. -rw-r--r-- 1 root slions 17 Jun 14 02:57 password -rw-r--r-- 1 root slions 6 Jun 14 02:57 username slions@sjy-658c5786f6-5879p:~$ cat /testsecret/username slions slions@sjy-658c5786f6-5879p:~$ cat /proc/mounts |grep testsecret tmpfs /testsecret tmpfs ro,relatime 0 0
可以看到，secret volumes的属组虽然已经被修改为了普通用户的（slions）,但还是只读的状态。
再次修改Deployment声明，指定volume的权限，禁用只读。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 apiVersion: apps/v1 kind: Deployment ... spec: ... template: ... spec: containers: ... name: sjy volumeMounts: - mountPath: /testsecret name: test-secret readOnly: false ... volumes: - name: test-secret secret: secretName: slions-secret defaultMode: 0777
更新后再次验证，依旧无法进行写入操作。
1 2 3 4 slions@sjy-84fd8c4957-nbtzj:~$ echo "aaaa" >> /testsecret/password bash: /testsecret/password: Read-only file system slions@sjy-84fd8c4957-nbtzj:~$ cat /proc/mounts |grep testsecret tmpfs /testsecret tmpfs ro,relatime 0 0
emptyDir 当 Pod 分派到某个 Node 上时，emptyDir 卷会被创建，并且在 Pod 在该节点上运行期间，卷一直存在。尽管 Pod 中的容器挂载 emptyDir 卷的路径可能相同也可能不同，这些容器都可以读写 emptyDir 卷中相同的文件。 当 Pod 因为某些原因被从节点上删除时，emptyDir 卷中的数据也会被永久删除。
Deployment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 apiVersion: apps/v1 kind: Deployment metadata: generation: 1 labels: app: sjy name: sjy namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: sjy strategy: rollingUpdate: maxSurge: 25 % maxUnavailable: 25 % type: RollingUpdate template: metadata: labels: app: sjy name: sjy spec: containers: - env: - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: SERVICE_NAME value: sjy image: library/tomcat:7-sjy3 imagePullPolicy: Always name: sjy ports: - containerPort: 80 name: ll protocol: TCP resources: limits: cpu: 500m memory: 500Mi requests: cpu: 200m memory: 200Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /emptydir name: empty dnsPolicy: ClusterFirst imagePullSecrets: - name: default-image-secret restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 7295 terminationGracePeriodSeconds: 30 volumes: - emptyDir: {} name: empty
创建此Deployment，进入Pod验证emptyDir目录访问权限与属组。
1 2 3 4 5 6 7 8 9 $ kubectl exec -it sjy-6f845654bf-5d5rc bash slions@sjy-6f845654bf-5d5rc:~$ ls -l / |grep emptydir drwxrwsrwx 2 root slions 6 Jun 14 04:15 emptydir slions@sjy-6f845654bf-5d5rc:~$ echo "abc" >> /emptydir/testfile slions@sjy-6f845654bf-5d5rc:~$ ls -l /emptydir/ total 4 -rw-r--r-- 1 slions slions 4 Jun 14 05:25 testfile slions@sjy-6f845654bf-5d5rc:~$ cat /proc/mounts |grep emptydir /dev/mapper/centos-home /emptydir xfs rw,relatime,attr2,inode64,noquota 0 0
设置了securityContext.FsGroup的服务，emptyDir属组被设置为了指定的组，且普通用户可以对其进行读写操作。
configMap configMap 卷提供了向 Pod 注入配置数据的方法。 ConfigMap 对象中存储的数据可以被 configMap 类型的卷引用，然后被 Pod 中运行的容器化应用使用。(应用配置文件外置的常见做法)
Map:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 apiVersion: v1 data: testfile: |- 1 22 333 4444 55555 666666 222 kind: ConfigMap metadata: name: sjyconfig namespace: default
Deployment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 apiVersion: apps/v1 kind: Deployment metadata: generation: 1 labels: app: sjy name: sjy namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: sjy strategy: rollingUpdate: maxSurge: 25 % maxUnavailable: 25 % type: RollingUpdate template: metadata: labels: app: sjy name: sjy spec: containers: - env: - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: SERVICE_NAME value: sjy image: library/tomcat:7-sjy3 imagePullPolicy: Always name: sjy ports: - containerPort: 80 name: ll protocol: TCP resources: limits: cpu: 500m memory: 500Mi requests: cpu: 200m memory: 200Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /slions/test name: configmap-volume-0-0 subPath: testfile readOnly: false dnsPolicy: ClusterFirst imagePullSecrets: - name: default-image-secret restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 7295 terminationGracePeriodSeconds: 30 volumes: - configMap: items: - key: testfile path: testfile name: sjyconfig name: configmap-volume-0-0
分别创建上述的ConfigMap与Deployment资源，进入Pod内，查看ConfigMap volume的权限。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 $ kubectl exec -it sjy-6dc89b49bb-7kchg bash slions@sjy-6dc89b49bb-7kchg:~$ ls -l /slions/test -rw-r--r-- 1 root slions 30 Jun 14 05:50 /slions/test slions@sjy-6dc89b49bb-7kchg:~$ cat /slions/test 1 22 333 4444 55555 666666 222slions@sjy-6dc89b49bb-7kchg:~$ echo "abc" >> /slions/test bash: /slions/test: Permission denied slions@sjy-6dc89b49bb-7kchg:~$ cat /proc/mounts |grep test /dev/mapper/centos-home /slions/test xfs ro,relatime,attr2,inode64,noquota 0 0
与secret现象一致，设置fsgroup生效，但为只读状态。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 $ kubectl describe deployments.apps sjy Name: sjy Namespace: default CreationTimestamp: Tue, 14 Jun 2022 13:50:52 +0800 Labels: app=sjy Annotations: deployment.kubernetes.io/revision: 2 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"generation":1,"labels":{"app":"sjy"},"name":"sjy","namespace":"d... Selector: app=sjy Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=sjy Annotations: sidecar.istio.io/inject: false Containers: sjy: Image: library/tomcat:7-sjy3 Port: 80/TCP Host Port: 0/TCP Limits: cpu: 500m memory: 500Mi Requests: cpu: 200m memory: 200Mi Environment: MY_POD_NAME: (v1:metadata.name) POD_NAMESPACE: (v1:metadata.namespace) NODE_NAME: (v1:spec.nodeName) POD_IP: (v1:status.podIP) HOST_IP: (v1:status.hostIP) SERVICE_NAME: sjy Mounts: /slions/test from configmap-volume-0-0 (rw,path="testfile") Volumes: configmap-volume-0-0: Type: ConfigMap (a volume populated by a ConfigMap) Name: sjyconfig Optional: false Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: sjy-7bbd8cfb97 (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 4m12s deployment-controller Scaled up replica set sjy-6dc89b49bb to 1 Normal ScalingReplicaSet 119s deployment-controller Scaled up replica set sjy-7bbd8cfb97 to 1 Normal ScalingReplicaSet 117s deployment-controller Scaled down replica set sjy-6dc89b49bb to 0
查看deployment详情，configmap 挂载点显示rw，但实际并为生效。
hostPath hostPath 卷能将主机节点文件系统上的文件或目录挂载到 Pod 中。
Deployment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 apiVersion: apps/v1 kind: Deployment metadata: generation: 1 labels: app: sjy name: sjy namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: sjy strategy: rollingUpdate: maxSurge: 25 % maxUnavailable: 25 % type: RollingUpdate template: metadata: labels: app: sjy name: sjy spec: containers: - env: - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: SERVICE_NAME value: sjy image: library/tomcat:7-sjy3 imagePullPolicy: Always name: sjy ports: - containerPort: 80 name: ll protocol: TCP resources: limits: cpu: 500m memory: 500Mi requests: cpu: 200m memory: 200Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /hostPath name: hostpath-volume-0 subPathExpr: $(POD_NAMESPACE)/$(SERVICE_NAME)/$(MY_POD_NAME) dnsPolicy: ClusterFirst nodeName: kubemaster imagePullSecrets: - name: default-image-secret restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 7295 terminationGracePeriodSeconds: 30 volumes: - hostPath: path: /tmp/hostpathDir type: DirectoryOrCreate name: hostpath-volume-0
创建该Deployment，查看运行该Pod的宿主机的hostpath路径权限，都是root。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ ls -lR /tmp/hostpathDir/ /tmp/hostpathDir/: 总用量 0 drwxr-xr-x 3 root root 17 6月 14 14:10 default /tmp/hostpathDir/default: 总用量 0 drwxr-xr-x 3 root root 34 6月 14 14:10 sjy /tmp/hostpathDir/default/sjy: 总用量 0 drwxr-xr-x 2 root root 6 6月 14 14:10 sjy-78d4cc59c5-txhbh /tmp/hostpathDir/default/sjy/sjy-78d4cc59c5-txhbh: 总用量 0
进入该pods内的查看对应目录权限：
1 2 3 4 5 $ kubectl exec -it sjy-78d4cc59c5-txhbh bash slions@sjy-78d4cc59c5-txhbh:~$ ls -l / |grep host drwxr-xr-x 2 root root 6 Jun 14 06:10 hostPath slions@sjy-78d4cc59c5-txhbh:~$ echo "abc" >> /hostPath/testfile bash: /hostPath/testfile: Permission denied
可以看到hostPath volume的属主属组都为root，普通用户无法进行读写操作。设置了securityContext.FsGroup对hostPath类型的volume并不生效。
官方并没有提供修改hostPath目录权限的方式，从hostPath的实现方式可知道，要想修改hostPath volume的属主属组，要提前修改宿主机对应的源目录权限，所以可采用曲线救国的方式来达到目的。
其中会使用inotify工具来实时监听宿主机目录，当有创建新目录动作时执行chown操作，将宿主机hostPath路径赋权为指定的应用用户。
inotify 是linux内核的一个特性，在内核 2.6.13 以上都可以使用。主要的功能是监听文件的变化。
首先在宿主机上创建应用用户：
1 $ sudo groupadd -g 7295 slions && sudo useradd -u 7295 -g 7295 slions && echo "slions:slions" |sudo chpasswd
安装inotify软件包：
1 $ yum install inotify-tools
编写监听脚本：（根据实际环境修改Dir与OwnerUser）
1 2 3 4 5 6 7 8 9 10 11 12 $ cat changehpowner.sh #!/bin/bash Dir=/tmp/hostpathDir OwnerUser=slions inotifywait -mrq --format '%e %w%f' -e create $Dir | while read msg do if [[ $msg =~ "CREATE,ISDIR" ]] then sudo chown -R $OwnerUser:$OwnerUser "${msg:13}" fi done $ chmod +x /home/user/shijingyu/hpowner/changehpowner.sh
编写systemd service:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 $ cat /usr/lib/systemd/system/changehpowner.service [Unit] Description=Change the owner of the host directory After=default.target Before=docker.service [Service] Type=simple ExecStart=/bin/bash /home/user/shijingyu/hpowner/changehpowner.sh ExecStop=/bin/kill -s HUP $MAINPID Restart=always [Install] WantedBy=multi-user.target
设置开机自启并启动服务。
1 2 3 $ sudo systemctl enable changehpowner.service Created symlink from /etc/systemd/system/multi-user.target.wants/changehpowner.service to /usr/lib/systemd/system/changehpowner.service. $ sudo systemctl restart changehpowner.service
重启之前的Deployment，进行验证：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 $ kubectl exec -it sjy-78d4cc59c5-fcwgq bash slions@sjy-78d4cc59c5-fcwgq:~$ ls -l / |grep host drwxr-xr-x 2 slions slions 6 Jun 14 06:34 hostPath slions@sjy-78d4cc59c5-fcwgq:~$ echo "abc" >> /hostPath/testfile slions@sjy-78d4cc59c5-fcwgq:~$ ls /hostPath/ testfile slions@sjy-78d4cc59c5-fcwgq:~$ ls -l /hostPath/ total 4 -rw-r--r-- 1 slions slions 4 Jun 14 06:35 testfile slions@sjy-78d4cc59c5-fcwgq:~$ exit exit $ ls -lR /tmp/hostpathDir/ /tmp/hostpathDir/: 总用量 0 drwxr-xr-x 3 slions slions 17 6月 14 14:34 default /tmp/hostpathDir/default: 总用量 0 drwxr-xr-x 3 slions slions 34 6月 14 14:34 sjy /tmp/hostpathDir/default/sjy: 总用量 0 drwxr-xr-x 2 slions slions 22 6月 14 14:35 sjy-78d4cc59c5-fcwgq /tmp/hostpathDir/default/sjy/sjy-78d4cc59c5-fcwgq: 总用量 4 -rw-r--r-- 1 slions slions 4 6月 14 14:35 testfile
可以看到，此时普通用户可对hostPath volume进行读写操作。
persistentVolumeClaim persistentVolumeClaim 卷用来将持久卷（PersistentVolume）挂载到 Pod 中。
pvc:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 $ kubectl get pvc sjydemo-default-pvc -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs creationTimestamp: "2022-06-14T07:08:07Z" finalizers: - kubernetes.io/pvc-protection labels: agree-app-type: agree-pvc name: sjydemo-default-pvc namespace: default resourceVersion: "210865311" selfLink: /api/v1/namespaces/default/persistentvolumeclaims/sjydemo-default-pvc uid: 26acc08d-fb9e-401c-9d2d-eaaa4c56754d spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: gluster-heketi volumeMode: Filesystem volumeName: pvc-26acc08d-fb9e-401c-9d2d-eaaa4c56754d
Deployment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 apiVersion: apps/v1 kind: Deployment metadata: generation: 1 labels: app: sjy name: sjy namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: sjy strategy: rollingUpdate: maxSurge: 25 % maxUnavailable: 25 % type: RollingUpdate template: metadata: labels: app: sjy name: sjy spec: containers: - env: - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: SERVICE_NAME value: sjy image: library/tomcat:7-sjy3 imagePullPolicy: Always name: sjy ports: - containerPort: 80 name: ll protocol: TCP resources: limits: cpu: 500m memory: 500Mi requests: cpu: 200m memory: 200Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /slions/mypv name: sjydemo-default-pvc dnsPolicy: ClusterFirst imagePullSecrets: - name: default-image-secret restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 7295 terminationGracePeriodSeconds: 30 volumes: - name: sjydemo-default-pvc persistentVolumeClaim: claimName: sjydemo-default-pvc
创建pvc与Deployment资源后进入Pod，查看挂载目录权限。
1 2 3 4 5 6 7 $ kubectl exec sjy-b746f9cb8-8r85f -it bash slions@sjy-b746f9cb8-8r85f:~$ ls -l /slions/ total 0 drwxrwsr-x 3 root 2060 24 Jun 14 07:08 mypv slions@sjy-b746f9cb8-8r85f:~$ echo "abc" >> /slions/mypv/testfile slions@sjy-b746f9cb8-8r85f:~$ cat /slions/mypv/testfile abc
可以看到，设置了securityContext.FsGroup对Pvc类型的volume并不生效。
kubernetes pvc只是资源的声明，对于pv卷的权限交给了具体的存储供应商，如果想要赋予pvc volume普通用户的权限，提供两种思路：
一是可通过init容器实现，在启动应用服务前完成目录赋权操作。
样例如下：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 apiVersion: apps/v1 kind: Deployment metadata: generation: 1 labels: app: sjy name: sjy namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: sjy strategy: rollingUpdate: maxSurge: 25 % maxUnavailable: 25 % type: RollingUpdate template: metadata: annotations: sidecar.istio.io/inject: "false" creationTimestamp: null labels: app: sjy name: sjy spec: containers: - env: - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: SERVICE_NAME value: sjy image: library/tomcat:7-sjy3 imagePullPolicy: Always name: sjy ports: - containerPort: 80 name: ll protocol: TCP resources: limits: cpu: 500m memory: 500Mi requests: cpu: 200m memory: 200Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /slions/mypv name: sjydemo-default-pvc initContainers: - command: - /bin/chown - -R - 7295 :7295 - /slions/mypv image: library/busybox:1.29.3 imagePullPolicy: IfNotPresent name: change-ownership-container resources: {} securityContext: privileged: true runAsUser: 0 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /slions/mypv name: sjydemo-default-pvc dnsPolicy: ClusterFirst imagePullSecrets: - name: default-image-secret restartPolicy: Always schedulerName: default-scheduler terminationGracePeriodSeconds: 30 volumes: - name: sjydemo-default-pvc persistentVolumeClaim: claimName: sjydemo-default-pvc
创建后进入Pod内查看，普通用户可实现正常读写。
1 2 3 4 5 6 7 $ kubectl exec -it sjy-6f69f5b786-frnvr bash slions@sjy-6f69f5b786-frnvr:~$ ls -l /slions/ total 0 drwxrwsr-x 3 slions slions 40 Jun 14 07:10 mypv slions@sjy-6f69f5b786-frnvr:~$ ls -l /slions/mypv/ total 1 -rw-r--r-- 1 slions slions 4 Jun 14 07:10 testfile
第二种可以在应用运行cmd出下文章，在执行应用进程前执行服务操作，或者是写在运行脚本中。
总结
volume类型
支持fsGroup
普通用户是否可读写
secret
√
只读
emptyDir
√
√
configMap
√
只读
hostPath
×
×
persistentVolumeClaim
×
×
从表中可看出，emptyDir支持fsGroup且普通用户可对其读写；secret大量用于密钥存储，不支持普通用户写操作；configMap在kubernetes 1.9.4版本之前允许普通用户写操作，但系统会以任意时间间隔恢复这些更改，在后续的版本中取消了该功能，通过上面的测试可发现，当明确定义了volumeMount的readOnly为false，k8s也会置为只读挂载；hostPath目录的权限由宿主机的源路径决定，可以使用linux 监听程序来自动化完成hostPath目录的赋权；pvc的目录赋权操作交给了存储提供商，我们可以利用初始化容器或者修改应用cmd来实现目的。
kubernetes在1.18版本添加了ConfigurableFSGroupPolicy的特性。
使用1.22.8版本开启该特性后，secret、configMap、hostPath与上表测试结果一致。
根据官方描述，高版本开启该特性后，securityContext支持fsGroupChangePolicy 字段，主要解决的是csi卷在设置了fsGroup时，递归地更改每个卷中的内容的属主和访问权限的效率问题。
