Featured image of post k8s部署longhorn
linux

k8s部署longhorn

longhorn是Kubernetes云原生分布式块存储解决方案，下面是部署步骤：

前提条件：

  • k8s已经部署，并且正常运行

  • 能够正常拉取到外网（墙外）镜像

  • 安装了open-iscsi和nfs的客户端

1
2
3
4
5
6
7
8

# open-iscsi is installed, and the iscsid daemon is running on all the nodes. 
dnf update -y
dnf install -y iscsi-initiator-utils
systemctl enable iscsid --now

# each node has a NFSv4 client installed
cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_1
cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_2

  • 主机文件系统支持file extents存储数据的功能，目前支持：XFS 、 ext4

  • bash, curl, findmnt, grep, awk, blkid, lsblk 已经安装

  • 必须启用挂载传播， Volumes | Kubernetes

安装longhorn

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# 直接执行longhorn.yaml
$ kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.2/deploy/longhorn.yaml

# 等待程序启动
$ kubectl get pods \
  --namespace longhorn-system \
  --watch
# 如果就绪了应该这样
#NAME                                                READY   STATUS    RESTARTS   AGE
#longhorn-ui-b7c844b49-w25g5                         1/1     Running   0          2m41s
#longhorn-manager-pzgsp                              1/1     Running   0          2m41s
#longhorn-driver-deployer-6bd59c9f76-lqczw           1/1     Running   0          2m41s
#longhorn-csi-plugin-mbwqz                           2/2     Running   0          100s
#csi-snapshotter-588457fcdf-22bqp                    1/1     Running   0          100s
#csi-snapshotter-588457fcdf-2wd6g                    1/1     Running   0          100s
#csi-provisioner-869bdc4b79-mzrwf                    1/1     Running   0          101s
#csi-provisioner-869bdc4b79-klgfm                    1/1     Running   0          101s
#csi-resizer-6d8cf5f99f-fd2ck                        1/1     Running   0          101s
#csi-provisioner-869bdc4b79-j46rx                    1/1     Running   0          101s
#csi-snapshotter-588457fcdf-bvjdt                    1/1     Running   0          100s
#csi-resizer-6d8cf5f99f-68cw7                        1/1     Running   0          101s
#csi-attacher-7bf4b7f996-df8v6                       1/1     Running   0          101s
#csi-attacher-7bf4b7f996-g9cwc                       1/1     Running   0          101s
#csi-attacher-7bf4b7f996-8l9sw                       1/1     Running   0          101s
#csi-resizer-6d8cf5f99f-smdjw                        1/1     Running   0          101s
#instance-manager-b34d5db1fe1e2d52bcfb308be3166cfc   1/1     Running   0          114s
#engine-image-ei-df38d2e5-cv6nc                      1/1     Running   0          114s

调整longhorn配置

如果需要访问longhorn的管理页面，可以编辑 longhorn-frontend 把这个service改成 NodePort ，然后访问页面可以编辑节点、卷的信息。

测试

我们新建一个nginx的svc+deployment，让pod挂载一个pv，这个pv由pvc定义并依赖longhorn的StorageClass:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

# nginx-longhorn-test.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30081

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-data
          mountPath: /usr/share/nginx/html
      volumes:
      - name: nginx-data
        persistentVolumeClaim:
          claimName: nginx-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi

然后部署这个应用：

1

$ kubectl apply -f nginx-longhorn-test.yaml

我们可以看看pvc、pv和pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


$ kubectl get pvc
#NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
#nginx-pvc   Bound    pvc-d2d80ab6-1a2e-41fd-b771-81bff9a5a3f9   1Gi        RWO            longhorn       <unset>                 19m
$ kubectl get pv
#NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
#pvc-d2d80ab6-1a2e-41fd-b771-81bff9a5a3f9   1Gi        RWO            Delete           Bound    default/nginx-pvc   longhorn       <unset>                          19m
$ kubectl get po
#NAME                                READY   STATUS    RESTARTS   AGE
#nginx-deployment-7fc8dbbc67-bvhqb   1/1     Running   0          19m

这个时候nginx已经起来了，如果卷创建有问题，那pod会无法启动，我们可以测试下，是不是正常写入数据，我们这个时候访问nginx的话会提示403，我们执行下面的操作：

1
2
3
4

$ kubectl exec -it $(kubectl get po | grep nginx-deployment |awk '{print $1}') -- sh -c "echo 'Longhorn storage test' > /usr/share/nginx/html/index.html"
$ kubectl exec -it $(kubectl get po | grep nginx-deployment |awk '{print $1}') -- cat /usr/share/nginx/html/index.html
# Longhorn storage test
# 直接curl访问或者浏览器访问也会返回这个。