docker搭建kafka集群的方法实现

目录一、原生Docker命令二、镜像选择三、集群规划四、Zookeeper集群安装五、Kafka集群安装

一、原生Docker命令

1. 删除所有dangling数据卷（即无用的Volume，僵尸文件）

docker volume rm $(docker volume ls -qf dangling=true)

2. 删除所有dangling镜像（即无tag的镜像）

docker rmi $(docker images | grep “^<none>” | awk “{print $3}”

3. 删除所有关闭的容器

docker ps -a | grep Exit | cut -d -f 1 | xargs docker rm

二、镜像选择

环境为M1版本的mbp:

Zookeeper采用zookeeperKafka采用wurstmeister/kafkaKafka-Manager采用scjtqs/kafka-managerMysql采用mysql/mysql-server

三、集群规划

1. 新建docker网络

docker network create docker-net –subnet 172.20.10.0/16 docker network ls

2. 集群规划

hostnameIp addrportlistenerzook1172.20.10.112184:2181zook2172.20.10.122185:2181zook3172.20.10.132186:2181kafka1172.20.10.14内部9092:9092，外部9192:9192kafka1kafka2172.20.10.15内部9093:9093，外部9193:9193kafka2Kafka3172.20.10.16内部9094:9094，外部9194:9194Kafka3本机（宿主机Mbp）172.20.10.2kafka manager172.20.10.109000:9000

四、Zookeeper集群安装

1. 新建文件zk-docker-compose.yml

version: 3.4 services:   zook1:     image: zookeeper:latest     restart: always     hostname: zook1     container_name: zook1 #容器名称，方便在rancher中显示有意义的名称     ports:     – 2183:2181 #将本容器的zookeeper默认端口号映射出去     volumes: # 挂载数据卷     – “/Users/konsy/Development/volume/zkcluster/zook1/data:/data”     – “/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog”     – “/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs”     environment:         ZOO_MY_ID: 1  #即是zookeeper的节点值，也是kafka的brokerid值         ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181     networks:         docker-net:             ipv4_address: 172.20.10.11   zook2:        image: zookeeper:latest     restart: always     hostname: zook2     container_name: zook2 #容器名称，方便在rancher中显示有意义的名称     ports:     – 2184:2181 #将本容器的zookeeper默认端口号映射出去     volumes:     – “/Users/konsy/Development/volume/zkcluster/zook2/data:/data”     – “/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog”     – “/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs”     environment:         ZOO_MY_ID: 2  #即是zookeeper的节点值，也是kafka的brokerid值         ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181     networks:         docker-net:             ipv4_address: 172.20.10.12                zook3:        image: zookeeper:latest     restart: always     hostname: zook3     container_name: zook3 #容器名称，方便在rancher中显示有意义的名称     ports:     – 2185:2181 #将本容器的zookeeper默认端口号映射出去     volumes:     – “/Users/konsy/Development/volume/zkcluster/zook3/data:/data”     – “/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog”     – “/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs”     environment:         ZOO_MY_ID: 3  #即是zookeeper的节点值，也是kafka的brokerid值         ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181     networks:         docker-net:             ipv4_address: 172.20.10.13 networks:   docker-net:     external:       name: docker-net

2. 执行脚本部署zookeeper至Docker:

docker compose -f ./zk-docker-compose.yml up -d

五、Kafka集群安装

1. 新建文件kafka-docker-compose.yml

version: 2 services:   kafka1:     image: docker.io/wurstmeister/kafka     restart: always     hostname: kafka1     container_name: kafka1     ports:       – 9093:9093       – 9193:9193     environment:       KAFKA_BROKER_ID: 1       KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9193       #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094       SKAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093,OUTSIDE://localhost:9193       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE       KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181       ALLOW_PLAINTEXT_LISTENER : yes       JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据     volumes:       – /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka       – /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka     external_links:       – zook1       – zook2       – zook3     networks:       docker-net:         ipv4_address: 172.20.10.14   kafka2:     image: docker.io/wurstmeister/kafka     restart: always     hostname: kafka2     container_name: kafka2     ports:       – 9094:9094       – 9194:9194     environment:       KAFKA_BROKER_ID: 2       KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9194       #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094       KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9094,OUTSIDE://localhost:9194       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE       KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181       ALLOW_PLAINTEXT_LISTENER : yes       JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据     volumes:       – /Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka       – /Users/konsy/Development/volume/kafka/kafka2/kafka:/kafka     external_links:       – zook1       – zook2       – zook3     networks:       docker-net:         ipv4_address: 172.20.10.15   kafka3:     image: docker.io/wurstmeister/kafka     restart: always     hostname: kafka3     container_name: kafka3     ports:       – 9095:9095       – 9195:9195     environment:       KAFKA_BROKER_ID: 3       KAFKA_LISTENERS: INSIDE://:9095,OUTSIDE://:9195       #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094       KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9095,OUTSIDE://localhost:9195       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE       KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181       ALLOW_PLAINTEXT_LISTENER : yes       JMX_PORT: 9999 #开放JMX监控端口，来监测集群数据     volumes:       – /Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka       – /Users/konsy/Development/volume/kafka/kafka3/kafka:/kafka     external_links:       – zook1       – zook2       – zook3     networks:       docker-net:         ipv4_address: 172.20.10.16 networks:   docker-net:     external:       name: docker-net

2. 执行脚本部署kafka至Docker:

docker compose -f ./kafka-docker-compose.yml up -d

3. listeners 和 advertised.listeners

listeners: 学名叫监听器，其实就是告诉外部连接者要通过什么协议访问指定主机名和端口开放的 Kafka 服务。advertised.listeners：和 listeners 相比多了个 advertised。Advertised 的含义表示宣称的、公布的，就是说这组监听器是 Broker 用于对外发布的。

比如说：

listeners: INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094 advertised_listeners: INSIDE://172.17.0.10:9092,OUTSIDE://<公网 ip>:端口 kafka_listener_security_protocol_map: “INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT” kafka_inter_broker_listener_name: “INSIDE”

advertised_listeners 监听器会注册在 zookeeper 中；

当我们对 172.17.0.10:9092 请求建立连接，kafka 服务器会通过 zookeeper 中注册的监听器，找到 INSIDE 监听器，然后通过 listeners 中找到对应的 通讯 ip 和 端口；

同理，当我们对 <公网 ip>:端口 请求建立连接，kafka 服务器会通过 zookeeper 中注册的监听器，找到 OUTSIDE 监听器，然后通过 listeners 中找到对应的 通讯 ip 和 端口 172.17.0.10:9094；

总结：advertised_listeners 是对外暴露的服务端口，真正建立连接用的是 listeners。

4.安装kafka-manager

4.1 新建文件kafka-manager-docker-compose.yml

version: 2 services:   kafka-manager:     image: scjtqs/kafka-manager:latest     restart: always     hostname: kafka-manager     container_name: kafka-manager     ports:       – 9000:9000     external_links:  # 连接本compose文件以外的container       – zook1       – zook2       – zook3       – kafka1       – kafka2       – kafka3     environment:       ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181       KAFKA_BROKERS: kafka1:9093,kafka2:9094,kafka3:9095       APPLICATION_SECRET: letmein       KM_ARGS: -Djava.net.preferIPv4Stack=true     networks:       docker-net:         ipv4_address: 172.20.10.10 networks:   docker-net:     external:       name: docker-net

4.2 执行脚本部署kafka-manager至Docker:

docker compose -f ./kafka-manager-docker-compose.yml up -d

4.3 配置Cluster

5. 测试kafka

5.1连接容器

进入kafka容器的命令行：

docker exec -ti kafka /bin/bash

进入kafka所在目录：

cd opt/kafka_2.13-2.8.1/

5.2 创建Replication为2，Partition为2的topic

bin/kafka-topics.sh –create –zookeeper zook1:2181 –replication-factor 2 –partitions 2 –topic partopic

5.3 查看topic的状态

在kafka容器中的 opt/kafka_2.12-1.1.0/ 目录下输入

bin/kafka-topics.sh –describe –zookeeper zook1:2181 –topic partopic