Kafka Producer KafkaProducer Sınıfı Batch Şeklinde Gönderir

Giriş

Açıklaması şöyle

Kafka producer sends messages in batches rather than one-by-one. Messages are first accumulated in RecordAccumulator, then a sender thread asynchronously sends the batched messages to Kafka broker. Batching is a major reason why Kafka can achieve high throughout in message processing.

RecordAccumulator Sınıfı

İki tane önemli bileşeni vardır

1. memory buffer pool

Açıklaması şöyle

Kafka does not directly rely on JVM for memory management, rather, it has BufferPool to optimize dynamic memory allocation and recycle.

2. a set of queues where messages are stored and batched before sending to brokers.

Çalışması şöyle. Yani her dequeue içinde ProducerBatch var. Periyodik yazma zamanı gelince ProducerBatch bir bütün olarak yazılır.

- messages assigned to the same TopicPartition are stored in the same Deque.
- Each Deque contains multiple ProducerBatches, each ProducerBatch contains multiple messages.
- Messages are sent to the broker at the ProducerBatch granularity.
- Buffer pool has default size of 32 MB, each pooled memory block has default size of 16 KB.