S_lion's Studio

手工安装heketi管理glusterfs

glusterfs heketi
字数统计: 1.5k阅读时长: 8 min
2021/07/14 Share

GlusterFS（gfs）由此名字也可看出是文件系统存储相关的软件，它是一个开源的分布式文件系统，具有强大的横向扩展能力。Heketi是一个GlusterFs管理软件，可以管理glusterFS集群的卷创建、删除等操作。Glusterfs作为kubernetes支持的多种卷类型之一，可以为上层应用提供多种挂载形式。

heketi + glusterfs提供两种部署形式：

  • 容器化
  • 传统服务

以下我会先介绍以传统服务的部署形态对接kubernetes。

  1. 本文的原稿是我在19年初编写的，以下版本可能过于老旧，同学们进行搭建测试时建议下载较新的版本。
  2. 阅读本文前希望您能对glusterfs,kubernetes,storageclass,pv有所了解。

环境描述

主机名 ip地址 系统环境 角色
k8s-1 IP:192.168.186.10 cpu:x2 mem:2GB disk:/dev/sdb 10GB K8s_master,Gluster_master,Heketi_master
K8s-2 IP:192.168.186.11 cpu:x2 mem:2GB disk:/dev/sdb 10GB K8s_node,Gluster_node
K8s-3 IP:192.168.186.12 cpu:x2 mem:2GB disk:/dev/sdb 10GB K8s_master,Gluster_node

如果存在iptable限制，需执行以下命令开通以下port

1
2
3
4
5
6
iptables -N heketi
iptables -A heketi -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT
iptables -A heketi -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT
iptables -A heketi -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
iptables -A heketi -p tcp -m state --state NEW -m multiport --dports 49152:49251 -j ACCEPT
service iptables save

安装配置gfs

三台机器都要安装gfs软件并启动服务

1
2
3
4
[root@k8s-1 ~]# yum -y install centos-release-gluster
[root@k8s-1 ~]# yum -y install glusterfs-server
[root@k8s-1 ~]# systemctl enable glusterd
[root@k8s-1 ~]# systemctl start glusterd

配置/etc/hosts,IP和主机名都一一对应

image-20210714161502089

安装glusterfs client客户端命令

image-20210714161514389

为存储池添加节点Node:（k8s-1操作，不用添加自己）

1
2
[root@k8s-1 ~]# gluster peer probe k8s-2
[root@k8s-1 ~]# gluster peer probe k8s-3

安装配置heketi

Heketi使用SSH来配置GlusterFS的所有节点。创建SSH密钥对:

1
2
3
4
[root@k8s-1 ~]# mkdir /etc/heketi
ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ‘’
[root@k8s-1 ~]# chown heketi:heketi /etc/heketi/heketi_key*

ssh公钥传递，这里只以一个节点为例：

1
[root@k8s-1 ~]# ssh-copy-id -i /etc/heketi/heketi_key.pub root@192.168.186.11

制作完成后会在当前目录下生成heketi_key、heketi_key.pub，将公钥heketi_key.pub拷贝到所有glusterfs节点上/etc/heketi/keketi_key.pub（包括你登陆的第一个节点）

安装heketi（在k8s-1操作）：

1
2
[root@k8s-1 ~]#  yum install -y https://mirrors.aliyun.com/centos/7.6.1810/storage/x86_64/gluster-5/heketi-8.0.0-1.el7.x86_64.rpm
[root@k8s-1 ~]#  yum install -y https://mirrors.aliyun.com/centos/7.6.1810/storage/x86_64/gluster-5/heketi-client-8.0.0-1.el7.x86_64.rpm

创建存储db的文件夹：

1
2
[root@k8s-1 ~]#  mkdir /dcos/heketi
[root@k8s-1 ~]#  chown -R heketi:heketi /dcos/heketi

配置 heketi.json：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
{
  "_port_comment": "Heketi Server Port Number",
  "port": "8088",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": false,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "123456"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "123456"
    }
  },

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",

    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/etc/heketi/heketi_key",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab"
    },

    "_kubeexec_comment": "Kubernetes configuration",
    "kubeexec": {
      "host" :"https://kubernetes.host:8443",
      "cert" : "/path/to/crt.file",
      "insecure": false,
      "user": "kubernetes username",
      "password": "password for kubernetes user",
      "namespace": "OpenShift project or Kubernetes namespace",
      "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
    },

    "_db_comment": "Database file name",
    "brick_min_size_gb": 1,
    "db": "/dcos/heketi/heketi.db",

    "_loglevel_comment": [
      "Set log level. Choices are:",
      "  none, critical, error, warning, info, debug",
      "Default is warning"
    ],
    "loglevel" : "debug"
  }
}

注：这里需要注意只是测试的话用mock 授权，standalone模式就 ssh 授权，k8s下就 kubernetes授权。

重启heketi：

1
2
[root@k8s-1 ~]#  systemctl enable heketi
[root@k8s-1 ~]#  systemctl restart heketi

测试heketi是否好用：

1
[root@k8s-1 ~]#  curl http://localhost:8088/hello

通过topology文件对接glusterfs集群：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "192.168.186.10"
              ],
              "storage": [
                "192.168.186.10"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "192.168.186.11"
              ],
              "storage": [
                "192.168.186.11"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "192.168.186.12"
              ],
              "storage": [
                "192.168.186.12"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        }
      ]
    }
  ]
}

将该文件发送给heketi创建：

1
[root@k8s-1 ~]# heketi-cli --server http://192.168.186.10:8088 --user admin --secret 123456 topology load --json=/etc/heketi/topology.json

创建成功后，heketi会在每个gluster节点上创建一个逻辑卷组，通过vgscan或vgdisplay可以看到：

image-20210714163441286

创建卷测试：

1
[root@k8s-1 ~]# heketi-cli --server http://192.168.186.10:8088 --user admin --secret 123456  volume create --size=1

配置kubernetes使用glusterfs

以密文的方式创建heketi userkey的secret

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s-1 k8s_gfs]# echo 123456|base64
MTIzNDU2Cg==
[root@k8s-1 k8s_gfs]# cat glusterfs-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  # base64 encoded password. E.g.: echo -n "mypassword" | base64
  key: MTIzNDU2Cg==
type: kubernetes.io/glusterfs

创建Storageclass

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@k8s-1 k8s_gfs]# cat storageclass_glusterfs.yaml 
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: glusterfs
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://192.168.186.10:8088"
  clusterid: "a06343355a5f3e4240662d3963ec7d90"
  restauthenabled: "true"
  secretNamespace: "default"
  secretName: "heketi-secret"
  restuser: "admin"
  #restuserkey: "123456"
  gidMin: "40000"
  gidMax: "50000"
  volumetype: "replicate:3"
allowVolumeExpansion: true

创建PVC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-1 k8s_gfs]# cat pvc_glusterfs.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: glusterfs-pvc
  namespace: default
  annotations:
    volume.beta.kubernetes.io/storage-class: "glusterfs"
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

创建pods测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[root@k8s-1 k8s_gfs]# cat nginx-pod.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-gfs
  labels:
    name: nginx-gfs
spec:
  replicas: 2
  selector:
    matchLabels:
      name: nginx-gfs
  template:
    metadata:
      labels:
        name: nginx-gfs
    spec:
      containers:
      - name: nginx-gfs
        image: nginx
        ports:
        - name: web
          containerPort: 80
        volumeMounts:
        - name: gfs
          mountPath: /usr/share/nginx/html
      volumes:
      - name: gfs
        persistentVolumeClaim:
          claimName: glusterfs-pvc

参考文档

https://kubernetes.io/zh/docs/concepts/storage/volumes/

CATALOG
  1. 1. 环境描述
  2. 2. 安装配置gfs
  3. 3. 安装配置heketi
  4. 4. 配置kubernetes使用glusterfs
    1. 4.1. 以密文的方式创建heketi userkey的secret
    2. 4.2. 创建Storageclass
    3. 4.3. 创建PVC
    4. 4.4. 创建pods测试
  5. 5. 参考文档
Total : 85
2024
2023
2022
2021
docker 虚拟机相关 prometheus glusterfs kubernetes storage harbor heketi linux ceph Distributed protocol ansible hexo shadowsocks git kubernetes核心组件 kubernetes Mysql logging Network shell openshift 观点与感悟
存储类 linux系统 虚拟机 k8s存储 镜像仓库 监控 docker hello-world 分布式一致性协议 ansible回炉 hexo使用 科学上网 git 好文推荐 kubernetes笔记 kubernetes Mysql 云原生 Network shell 观点与感悟