config/server.properties Dosyası

broker id Alanı

Her broker tekil bir id değerine sahip olmalıdır

broker.rack Alanı

Kafka Rack Awareness için gerekir. Açıklaması şöyle

Kafka uses rack awareness during replica assignment. When creating a topic, Kafka assigns replicas of partitions to brokers in such a way that no two replicas of the same partition are in the same rack. 

Açıklaması şöyle

You can still lose data in Kafka, even with a replication factor of 3.

All 3 brokers are healthy. No disk fails. Kafka has no bug.

Because when Kafka acknowledges a write, it is not on disk yet.

Each broker hands the record to the operating system. The operating system keeps it in memory and writes it to disk later, on its own schedule. Kafka does not force every record to disk, and its own documentation recommends that you do not make it.

So Kafka's durability does not come from the disk. It comes from the copies. Kafka is betting that 3 machines will not lose their memory at the same moment.

Now put all 3 brokers in one availability zone.

The power goes off in that zone for a few seconds. Every broker restarts. The disks are fine. But the last few seconds of writes were only in memory, on all 3 machines, and they are gone.

This is why you can tell each broker which zone it sits in (broker.rack). Kafka then places a partition's 3 copies in 3 different zones, so one zone going down costs you one copy, not all of them.

It does not make the risk zero. It makes losing all 3 copies at once far less likely.

And it has a price:

- Every write now waits for a copy in another zone, so each write gets slower.

- acks=all only waits for the replicas that are in sync right now. If the other two fall behind, the leader acknowledges alone, and that write sits in one zone after all. Set min.insync.replicas to 2 and the leader refuses the write instead. So you are also choosing to reject writes when too few copies are in sync.

- Every copy crosses a zone boundary, and cloud providers bill you for traffic between zones.

Kafka does not trust the disk to keep your data.

It trusts that your zones will not fail together.

JMX

broker_java_opts değişkenine şunlar ilave ederiz

-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=9393