K8S 离线安装 dolphinscheduler2.0.5

由于环境原因,不能在生产环境访问外网,所以只能在外网下载好DS相关的依赖镜像上传到内网环境,在内网环境安装,k8s部署流程参考官方文档:快速试用 Kubernetes 部署

一、下载镜像到本地

[root@hombd03 dolphinscheduler] cd /opt/modules/apache-dolphinscheduler-2.0.5-src/docker/kubernetes/dolphinscheduler

values.yaml 文件中修改镜像,以实现离线安装(air-gap install);

image:
  repository: "apache/dolphinscheduler"
  tag: "2.0.5"
  pullPolicy: "IfNotPresent"
  pullSecret: ""

针对公司内部安装好的harbor,或者其他公有云的私仓,进行pull,tag,以及push。这里我们假定私仓地址是harbor.abc.com,你所在构建镜像的主机已经进行了docker login harbor.abc.com, 且已经建立和授权私仓下面新建apache项目。

执行shell命令:

# 下载镜像到本地
docker pull apache/dolphinscheduler:2.0.5

# 将下载的镜像打成tar包,然后导入到内网(该内网环境不能访问外网)
dock tag apache/dolphinscheduler:2.0.5 harbor.abc.com/apache/dolphinscheduler:2.0.5
docker push apache/dolphinscheduler:2.0.5

再替换value文件中的镜像信息,这里我们推荐使用Always的方式拉取镜像,生产环境中尽量每次去检查是否是最新的镜像内容,保证软件制品的正确性。此外,很多同学会把tag写成latest(制作镜像不写tag信息,这样在生产环境非常危险,任何人push了镜像,就相当于改变了latest的tag的镜像,而且也无法判断latest是什么版本,所以建议要明确每次发版的tag,并且使用Always。

image:
  repository: "harbor.abc.com/apache/dolphinscheduler"
  tag: "2.0.5"
  pullPolicy: "Always"

更新DS依赖

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm dependency update .

报如下的错,postgresql:10.3.18, zookeeper:6.5.3 版本找不到对应的包:

[root@homaybd03 dolphinscheduler]# helm dependency update .
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Error: can't get a valid version for repositories postgresql, zookeeper. Try changing the version constraint in Chart.yaml

如果找不到对应的包,则通过 helm search repo postgresql -l 命令来搜索可使用的包:

[root@homaybd03 dolphinscheduler]# helm search repo postgresql -l
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
bitnami/postgresql      11.6.13         14.4.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.6.12         14.4.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.6.11         14.4.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.6.10         14.4.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.6.9          14.4.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.6.8          14.4.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.6.7          14.4.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.6.6          14.3.0          PostgreSQL (Postgres) is an open source object-...
...
bitnami/postgresql      11.1.13         14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.12         14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.11         14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.10         14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.9          14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.8          14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.7          14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.6          14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.5          14.2.0          PostgreSQL (Postgres) is an open source object-...
bitnami/postgresql      11.1.4          14.2.0          PostgreSQL (Postgres) is an open source object-...
...
bitnami/postgresql      10.14.4         11.14.0         Chart for PostgreSQL, an object-relational data...
bitnami/postgresql      10.14.3         11.14.0         Chart for PostgreSQL, an object-relational data...
bitnami/postgresql-ha   9.2.0           14.4.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.9           14.4.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.8           14.4.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.7           14.4.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.6           14.4.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.5           14.3.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.4           14.3.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.2           14.3.0          This PostgreSQL cluster solution includes the P...
bitnami/postgresql-ha   9.1.1           14.3.0          This PostgreSQL cluster solution includes the P...
...
bitnami/postgresql-ha   8.2.1           11.14.0         Chart for PostgreSQL with HA architecture (usin...
bitnami/postgresql-ha   8.2.0           11.14.0         Chart for PostgreSQL with HA architecture (usin...
bitnami/postgresql-ha   8.1.5           11.14.0         Chart for PostgreSQL with HA architecture (usin...
[root@homaybd03 dolphinscheduler]# 

查看zookeepr可用的版本:

[root@homaybd03 dolphinscheduler]# helm search repo zookeeper -l
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
bitnami/zookeeper               10.0.1          3.8.0           Apache ZooKeeper provides a reliable, centraliz...
bitnami/zookeeper               10.0.0          3.8.0           Apache ZooKeeper provides a reliable, centraliz...
bitnami/zookeeper               9.2.7           3.8.0           Apache ZooKeeper provides a reliable, centraliz...
bitnami/zookeeper               9.2.6           3.8.0           Apache ZooKeeper provides a reliable, centraliz...
...
bitnami/zookeeper               8.0.3           3.7.0           Apache ZooKeeper provides a reliable, centraliz...
bitnami/zookeeper               8.0.2           3.7.0           Apache ZooKeeper provides a reliable, centraliz...
bitnami/zookeeper               8.0.1           3.7.0           A centralized service for maintaining configura...
bitnami/zookeeper               8.0.0           3.7.0           A centralized service for maintaining configura...
bitnami/zookeeper               7.6.2           3.7.0           A centralized service for maintaining configura...
bitnami/zookeeper               7.6.1           3.7.0           A centralized service for maintaining configura...
bitnami/zookeeper               7.6.0           3.7.0           A centralized service for maintaining configura...
...
bitnami/dataplatform-bp2        10.0.10         1.0.1           This Helm chart can be used for the automated d...
bitnami/dataplatform-bp2        10.0.9          1.0.1           This Helm chart can be used for the automated d...
bitnami/dataplatform-bp2        10.0.8          1.0.1           OCTO Data platform Kafka-Spark-Elasticsearch He...

bitnami/kafka                   16.2.5          3.1.0           Apache Kafka is a distributed streaming platfor...
bitnami/kafka                   16.2.4          3.1.0           Apache Kafka is a distributed streaming platfor...
bitnami/kafka                   16.2.3          3.1.0           Apache Kafka is a distributed streaming platfor...
...
bitnami/kafka                   15.3.5          3.1.0           Apache Kafka is a distributed streaming platfor...
bitnami/solr                    4.1.0           8.11.1          Apache Solr is an extremely powerful, open sour...
bitnami/solr                    4.0.12          8.11.1          Apache Solr is an extremely powerful, open sour...
bitnami/solr                    4.0.11          8.11.1          Apache Solr is an extremely powerful, open sour...
bitnami/solr                    4.0.10          8.11.1          Apache Solr is an extremely powerful, open sour...
bitnami/solr                    4.0.9           8.11.1          Apache Solr is an extremely powerful, open sour...
bitnami/solr                    4.0.8           8.11.1          Apache Solr is an extremely powerful, open sour...
bitnami/solr                    4.0.7           8.11.1          Apache Solr is an extremely powerful, open sour...
...
bitnami/solr                    2.4.2           8.11.1          Apache Solr is an open source enterprise search...
bitnami/solr                    2.3.0           8.11.1          Apache Solr is an open source enterprise search...
bitnami/solr                    2.2.0           8.11.1          Apache Solr is an open source enterprise search...
[root@homaybd03 dolphinscheduler]# 

更改 Chart.yaml 文件的版本:

dependencies:
- name: postgresql
  version: 10.14.3
  # Due to a change in the Bitnami repo, https://charts.bitnami.com/bitnami was truncated only
  # containing entries for the latest 6 months (from January 2022 on).
  # This URL: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
  # contains the full 'index.yaml'.
  # See detail here: https://github.com/bitnami/charts/issues/10833
  repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
  condition: postgresql.enabled
- name: zookeeper
  version: 8.0.2
  # Same as above.
  repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
  condition: zookeeper.enabled

然后再重新加载:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm dependency update .

安装DS:

# [root@hombd03 dolphinscheduler]# kubectl create ns test

helm install dolphinscheduler . --set image.tag=2.0.5 -n test

将名为 dolphinscheduler 的版本(release) 发布到 test 的命名空间中:

[root@hombd03 dolphinscheduler]# helm install dolphinscheduler . --set image.tag=2.0.5 -n test
NAME: dolphinscheduler
LAST DEPLOYED: Tue Jul  5 00:08:05 2022
NAMESPACE: test
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

** Please be patient while the chart DolphinScheduler 2.0.5 is being deployed **

Access DolphinScheduler UI URL by:

  kubectl port-forward -n test svc/dolphinscheduler-api 12345:12345

  DolphinScheduler UI URL: http://127.0.0.1:12345/dolphinscheduler
[root@homaybd03 dolphinscheduler]# 

查看Pod启动状况:

[root@hombd03 dolphinscheduler]# kubectl get pods -n test
NAME                                      READY   STATUS              RESTARTS   AGE
dolphinscheduler-alert-69bb5f4f95-w2dft   0/1     Running             0          2m52s
dolphinscheduler-api-84889f7f64-b9mzp     0/1     ContainerCreating   0          2m52s
dolphinscheduler-master-0                 0/1     ContainerCreating   0          2m52s
dolphinscheduler-master-1                 0/1     ContainerCreating   0          2m52s
dolphinscheduler-master-2                 0/1     Running             0          2m52s
dolphinscheduler-postgresql-0             1/1     Running             0          2m52s
dolphinscheduler-worker-0                 0/1     Running             0          2m52s
dolphinscheduler-worker-1                 0/1     Running             0          2m52s
dolphinscheduler-worker-2                 0/1     ContainerCreating   0          2m52s
dolphinscheduler-zookeeper-0              1/1     Running             0          2m52s
[root@hombd03 dolphinscheduler]# kubectl get pods -n test -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP                NODE        NOMINATED NODE   READINESS GATES
dolphinscheduler-alert-69bb5f4f95-w2dft   1/1     Running   0          3m22s   192.200.72.222    hombd05   <none>           <none>
dolphinscheduler-api-84889f7f64-b9mzp     0/1     Running   0          3m22s   192.200.245.154   hombd04   <none>           <none>
dolphinscheduler-master-0                 0/1     Running   0          3m22s   192.200.245.150   hombd04   <none>           <none>
dolphinscheduler-master-1                 0/1     Running   0          3m22s   192.200.245.153   hombd04   <none>           <none>
dolphinscheduler-master-2                 0/1     Running   0          3m22s   192.200.72.226    hombd05   <none>           <none>
dolphinscheduler-postgresql-0             1/1     Running   0          3m22s   192.200.72.224    hombd05   <none>           <none>
dolphinscheduler-worker-0                 1/1     Running   0          3m22s   192.200.72.223    hombd05   <none>           <none>
dolphinscheduler-worker-1                 1/1     Running   0          3m22s   192.200.72.225    hombd05   <none>           <none>
dolphinscheduler-worker-2                 0/1     Running   0          3m22s   192.200.245.152   hombd04   <none>           <none>
dolphinscheduler-zookeeper-0              1/1     Running   0          3m22s   192.200.245.151   hombd04   <none>           <none>
[root@hombd03 dolphinscheduler]# 

等一会在看pod启动:

[root@homaybd03 dolphinscheduler]# kubectl get pods -n test
NAME                                      READY   STATUS             RESTARTS   AGE
dolphinscheduler-alert-69bb5f4f95-w2dft   1/1     Running            0          27m
dolphinscheduler-api-84889f7f64-7vlnz     1/1     Running            0          4m53s
dolphinscheduler-master-0                 1/1     Running            0          98s
dolphinscheduler-master-1                 1/1     Running            0          97s
dolphinscheduler-master-2                 1/1     Running            5          27m
dolphinscheduler-postgresql-0             1/1     Running            0          27m
dolphinscheduler-worker-0                 1/1     Running            6          27m
dolphinscheduler-worker-1                 0/1     CrashLoopBackOff   6          27m
dolphinscheduler-worker-2                 1/1     Running            1          2m23s
dolphinscheduler-zookeeper-0              1/1     Running            0          100s
[root@homaybd03 dolphinscheduler]# kubectl get pods -n test -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP                NODE        NOMINATED NODE   READINESS GATES
dolphinscheduler-alert-69bb5f4f95-w2dft   1/1     Running   0          27m     192.200.72.222    homaybd05   <none>           <none>
dolphinscheduler-api-84889f7f64-7vlnz     1/1     Running   0          5m36s   192.200.72.233    homaybd05   <none>           <none>
dolphinscheduler-master-0                 1/1     Running   0          2m21s   192.200.245.161   homaybd04   <none>           <none>
dolphinscheduler-master-1                 1/1     Running   0          2m20s   192.200.245.162   homaybd04   <none>           <none>
dolphinscheduler-master-2                 1/1     Running   5          27m     192.200.72.226    homaybd05   <none>           <none>
dolphinscheduler-postgresql-0             1/1     Running   0          27m     192.200.72.224    homaybd05   <none>           <none>
dolphinscheduler-worker-0                 1/1     Running   6          27m     192.200.72.223    homaybd05   <none>           <none>
dolphinscheduler-worker-1                 0/1     Running   7          27m     192.200.72.225    homaybd05   <none>           <none>
dolphinscheduler-worker-2                 1/1     Running   1          3m6s    192.200.245.155   homaybd04   <none>           <none>
dolphinscheduler-zookeeper-0              1/1     Running   0          2m23s   192.200.245.159   homaybd04   <none>           <none>
[root@homaybd03 dolphinscheduler]# 

查看Pod日志:

[root@hombd03 dolphinscheduler]# kubectl logs dolphinscheduler-api-84889f7f64-b9mzp -n test
Error from server: Get "https://homaybd04:10250/containerLogs/test/dolphinscheduler-api-84889f7f64-b9mzp/dolphinscheduler-api": net/http: TLS handshake timeout
[root@homaybd03 dolphinscheduler]# 

如果 values.yaml 文件中的 ingress.enabled 被设置为 true, 在浏览器中访问 http://${ingress.host}/dolphinscheduler 即可

提示: 如果 ingress 访问遇到问题,请联系 Kubernetes 管理员并查看 Ingress

否则,当 api.service.type=ClusterIP 时,你需要执行 port-forward 端口转发命令:

kubectl port-forward --address 0.0.0.0 -n test svc/dolphinscheduler-api 12345:12345 # 使用 test 命名空间
[root@hombd03 dolphinscheduler]# kubectl get svc -n test
NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
dolphinscheduler-alert                 ClusterIP   192.233.2.127    <none>        50052/TCP                    34m
dolphinscheduler-api                   ClusterIP   192.233.214.70   <none>        12345/TCP                    34m
dolphinscheduler-master-headless       ClusterIP   None             <none>        5678/TCP                     34m
dolphinscheduler-postgresql            ClusterIP   192.233.60.228   <none>        5432/TCP                     34m
dolphinscheduler-postgresql-headless   ClusterIP   None             <none>        5432/TCP                     34m
dolphinscheduler-worker-headless       ClusterIP   None             <none>        1234/TCP,50051/TCP           34m
dolphinscheduler-zookeeper             ClusterIP   192.233.241.93   <none>        2181/TCP,2888/TCP,3888/TCP   34m
dolphinscheduler-zookeeper-headless    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   34m
[root@hombd03 dolphinscheduler]# 

导出镜像包:

# https://blog.csdn.net/m0_67402235/article/details/124174771
[root@hombd03 saveimage]# docker save > dolphinscheduler-2.0.5.tar apache/dolphinscheduler:2.0.5
[root@hombd03 saveimage]# docker save > dolphinscheduler-mysql-2.0.5.tar apache/dolphinscheduler:mysql-driver
[root@hombd03 saveimage]# docker save > dolphinscheduler-scarf-2.0.5.tar dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler
docker pull bitnami/zookeeper:3.7.0-debian-10-r295
docker save > zookeeper-3.7.0.tar bitnami/zookeeper:3.7.0-debian-10-r295

导入镜像包:

# export 导出容器,对应的import导入
# docker import dolphinscheduler-mysql-2.0.5.tar apache/dolphinscheduler:2.0.5

# save对应的是load 使用 -i 导入
[root@dce88 ~]# docker load -i /opt/images.tar.gz
# 使用 < 导入
[root@dce88 ~]# docker load  < /opt/images.tar.gz

export和save命令区别

export命令是从容器(container)中导出tar文件,而save命令则是从镜像(images)中导出

  • 文件大小不同:export 导出的镜像文件体积小于 save 保存的镜像
  • 是否可以对镜像重命名
     docker import 可以为镜像指定新名称
     docker load 不能对载入的镜像重命名
  • 是否可以同时将多个镜像打包到一个文件中
    docker export 不支持
    docker save 支持
  • 是否包含镜像历史
    export 导出(import 导入)是根据容器拿到的镜像,再导入时会丢失镜像所有的历史记录和元数据信息(即仅保存容器当时的快照状态),所以无法进行回滚操作
    而 save 保存(load 加载)的镜像,没有丢失镜像的历史,可以回滚到之前的层(layer)

应用场景不同
docker export 的应用场景:主要用来制作基础镜像,比如我们从一个 ubuntu 镜像启动一个容器,然后安装一些软件和进行一些设置后,使用 docker export 保存为一个基础镜像。然后,把这个镜像分发给其他人使用,比如作为基础的开发环境。

docker save 的应用场景:如果我们的应用是使用 docker-compose.yml 编排的多个镜像组合,但我们要部署的客户服务器并不能连外网。这时就可以使用 docker save 将用到的镜像打个包,然后拷贝到客户服务器上使用 docker load 载入。

部署服务

启动zookeeper 服务,本地端口 8181 映射到容器的 2181 端口

docker run  --privileged=true  -d -e TZ="Asia/Shanghai"  -e ALLOW_ANONYMOUS_LOGIN=yes -p 8181:2181 -v /data/zookeeper:/data --name zookeeper --restart always bitnami/zookeeper:3.7.0-debian-10-r295

运行一个 DolphinScheduler 实例

$ docker run -d --name dolphinscheduler \
-e DATABASE_TYPE="mysql"  -e DATABASE_DRIVER="com.mysql.cj.jdbc.Driver"  -e DATABASE_PARAMS="characterEncoding=utf8&useSSL=false" \
-e DATABASE_HOST="192.168.1.125" -e DATABASE_PORT="3306" -e DATABASE_DATABASE="ds-k8s" \
-e DATABASE_USERNAME="root" -e DATABASE_PASSWORD="homch" \
-e ZOOKEEPER_QUORUM="192.168.1.123:8181" \
-e -e REGISTRY_SERVERS="192.168.1.123:8181" \
-p 12345:12345 \
apache/dolphinscheduler:mysql-driver all

相关文章:
DolphinScheduler在Kubernetes体系中的技术实战
快速试用 Kubernetes 部署
docker--导出镜像 save/export、导入镜像 load/import区别及使用场景
DolphinScheduler部署中的一些坑(Docker & Kubernetes)

为者常成,行者常至