Broker Ayarları

zookeeper Ayarları

zookeeper.connect Alanı

Bu alanda zookeeper sunucusunun adresini belirtilir. Zookeeper varsayılan port olarak 2181'i kullanır

Örnek

Şöyle yaparız

# A comma separated list of directories under which to store logs 
log.dirs=[Path to log directory]\logs
...
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

advertised.listeners Alanı

Bu alan Kafka'ya bağlanmak için gerekli

Örnek

Şöyle yaparız

advertised.listeners=PLAINTEXT://your.host.name:9092)

Örnek 

Şöyle yaparız.

#Set the broker.id as 0,1,2,3,… 
#Specify the zookeeper.connect as hostname1:port1,hostname2:port2,hostname3:port3 and 
#advertised.listener as the IP (private) of the instance. 
#The log directories can be set using log.dirs.
broker.id=0
advertised.listeners=PLAINTEXT://1.2.3.7:9092
zookeeper.connect=1.2.3.4:2181,1.2.3.5:2181,1.2.3.6:2181
listeners=PLAINTEXT://:9092
log.dirs=/data/logs/kafka

Mesaj Ayarları

message.max.bytes

Açıklaması şöyle. Daha büyük mesajlar için Conduktor kullanılabilir

... default limit on the record size it accepts. Apache Kafka is set to handle records up to 1MB

Threads Ayarları

Örnek 

Şöyle yaparız.

#Set the number of threads required for the various processes.
num.network.threads=150
num.io.threads=80
background.threads=80
num.recovery.threads.per.data.dir=3
num.replica.fetchers=3

#Set the size of the socket buffers, maximum request size and the fetch size.
socket.send.buffer.bytes=10485760
socket.receive.buffer.bytes=10485760
socket.request.max.bytes=41943040
message.max.bytes=10485760
fetch.max.bytes=41943040
replica.fetch.max.bytes=10485760
replica.fetch.response.max.bytes=41943040

#Set the default replication factor and number of partitions based on the requirement. 
#Replication factor of the offset and transaction topics need to be set to 2 or 3.
default.replication.factor=3
num.partitions=1

offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3

#Set Kafka broker log configuration such as retention hours and 
#retention check interval.
retention.hours=48
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Socket Ayarları

Örnek 

Şöyle yaparız.

#Set the size of the socket buffers, maximum request size and the fetch size.
socket.send.buffer.bytes=10485760
socket.receive.buffer.bytes=10485760
socket.request.max.bytes=41943040
message.max.bytes=10485760
fetch.max.bytes=41943040
replica.fetch.max.bytes=10485760
replica.fetch.response.max.bytes=41943040

Plugin Ayarları

Kafka Connect için kullanılan pluginleri belirtmek gerekir

Örnek

Şöyle yaparız

plugin.path=/home/kafka/kafka_2.13–3.3.2/libs/connect-file-3.3.2.jar

Replication Ayarları

Örnek 

Şöyle yaparız.

#Set the default replication factor and number of partitions based on the requirement. 
#Replication factor of the offset and transaction topics need to be set to 2 or 3.
default.replication.factor=3
num.partitions=1

offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3

#Set Kafka broker log configuration such as retention hours and 
#retention check interval.
retention.hours=48
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Compaction Ayarları

Broker Ayarları - Compaction Of Topic Data yazısına taşıdım

 

Retention Ayarları

Mesajların ne kadar saklanacağını belirtir. Bayatlayan mesajlar cleanup policy ayarına göre silinir. Açıklaması şöyle

Retention is the policy that determines how long data is kept in Kafka before being deleted or compacted. Retention can be based on time or size limits.
- Time-based retention deletes records that are older than a specified duration.
- Size-based retention deletes records when the total size of a partition exceeds a specified limit. You can also use log compaction to retain only the latest value for each record key

Örnek

Şöyle yaparız. Mesajlar hiç bir zaman silinmezler

cleanup.policy=compact 
log.rentention.ms=-1

Örnek

Şöyle yaparız.

#Set Kafka broker log configuration such as retention hours and 
#retention check interval.
retention.hours=48
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Log ayarları

log.dirs

Örnek

Şöyle yaparız

# A comma separated list of directories under which to store logs 
log.dirs=[Path to log directory]\logs

log.flush.interval.messages=1 

Açıklaması şöyle.

.. forces Kafka to fsync on each message batch