Docker Compose ve Kafka

Giriş

Bazı image'lar şöyle

- confluentinc/cp-zookeeper + confluentinc/cp-server + confluentinc/cp-kafka

- confluentinc/cp-kafka + confluentinc/cp-zookeeper

- debezium/kafka + debezium/zookeeper

confluentinc/cp-kafka : Kafka'yı çalıştırır

confluentinc/cp-zookeeper : Zookeeper'ı çalıştırır

confluentinc/cp-server : Bu da Kafka'yı çalıştırır ancak içinde Confluent tarafından sağlanan ticari bazı özellikler de var

Kafka Listeners

Açıklaması şöyle.

There are two types of listeners, which we need to understand from Docker's perspective. First, LISTENERS, are what interfaces Kafka binds to. This is for clients connecting on docker. Secondly, ADVERTISED_LISTENERS, are how external clients can connect to Kafka and this is for clients connecting outside of the Docker machine. In the below diagram, Listener A acts as an internal listener and exposes Kafka to clients running on a Docker host. And, Listener B acts as an external listener and allows connections to Kafka from outside the Docker host to remote clients.

Yani 

- LISTENERS docker içindeki veya localhost üzerindekilerin Kafka'ya bağlanabilmesi içindir. 

- ADVERTISED_LISTENERS ise uzak bilgisayardakilerin  Kafka'ya bağlanabilmesi içindir.

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP

Açıklaması şöyle.

defines key/value pairs for the security protocol to use per listener name. (SSL or no SSL)

PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT yaparsak SSL kullanılmaz

Metrics and monitoring İçin JMX Exporters

Açıklaması şöyle

Kafka brokers can be monitored using prometheus and grafana. We will use the JMX exporter to expose Kafka’s metrics for Prometheus to scrape and then visualize the metrics on a Grafana dashboard. JMX exporter acts as a collector which exposes Kafka metrics over an HTTP endpoint and then can be consumed by any system such as Prometheus.

We have added EXTRA_ARGS to use JMX exporter for exposing Kafka metrics and port mapping for exposing the same for Prometheus target to scrape. JMX exporter jar is required for the same. We also need to provide Kafka metrics configuration (kafka-broker.yaml) in order to expose the required metrics.

Örnek - JMX Exporters

Şöyle yaparız

services:
  kafka:
    logging:      
      driver: local
    ports:
      - 29094:29092
      - 29103:29101
    image: confluentinc/cp-kafka:latest
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://127.0.0.1:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 
      EXTRA_ARGS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=29101:/usr/share/jmx_exporter/kafka-broker.yaml  
    networks:
      - kafka
    volumes:      
      - ./jmx-exporter:/usr/share/jmx_exporter/      
      - kafka-jmx-volume:/jmx-exporter
    depends_on:
      - zookeeper
  zookeeper:
    ports:
      - 32181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    image: confluentinc/cp-zookeeper:latest
    networks:
      - kafka
volumes:  
  kafka-jmx-volume:
networks:
  kafka:
    external: false

Örnek - Kafka Listeners

Şöyle yaparız

services:
  kafka:
    ports:
      - 29094:29092
    image: confluentinc/cp-kafka:latest
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://127.0.0.1:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1   
    networks:
      - kafka
    depends_on:
      - zookeeper
  zookeeper:
    ports:
      - 32181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    image: confluentinc/cp-zookeeper:latest
    networks:
      - kafka
networks:
  kafka:
    external: false

- Docker içindekiler kafka isimli servise 9092 portundan bağlanır

- Docker dışındakiler ama aynı bilgisayarda çalışan uygulamalar localhost bilgisayarına 29092 portundan bağlanır.

- Uzak bilgisayardaki uygulamalar  localhost bilgisayarına 29094 portundan bağlanır.

Varsayıaln network içinde şöyle yaparız

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.0.1
    container_name: broker
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

Örnek

confluentinc/cp-zookeeper:5.4.0 : zookeeper için

confluentinc/cp-server:5.4.0 : kafka için

confluentinc/cp-kafka:5.4.0 : kafka tools için

Şöyle yaparız

version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:5.4.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

  kafka-tools:
    image: confluentinc/cp-kafka:5.4.0
    hostname: kafka
    container_name: kafka
    command: ["tail", "-f", "/dev/null"]
    network_mode: "host"

Örnek

Şöyle yaparız

 zookeeper:
    image: debezium/zookeeper:1.2
    ports:
      - 2181:2181
  kafka:
    image: debezium/kafka:1.2
    ports:
      - 9092:9092
    links:
      - zookeeper
    depends_on:
      - zookeeper
    environment:
      - ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092