Kafka Connect Kullanımı

Giriş

Açıklaması şöyle

Kafka Streams, like Kafka Connect, are part of open-source Apache Kafka. Hence, the Java library is included if you download Kafka from the Apache website. It is already included in the data streaming landscape with the Kafka logo. You should always ask yourself if you need another framework besides Kafka Streams for stream processing. The significant benefit: One technology, one vendor, one infrastructure.

Many vendors exclude or do not focus on Kafka Streams and Kafka Connect and only offer incomplete Kafka; they want to sell their own integration and processing products instead.

Kafka Connect ve SourceConnector İlişkisi

Kafka Connect alt yapısı bir SourceConnector kullanarak List<SourceRecord> listesi çeker. SourceRecord kalıtımı şöyle

public class SourceRecord extends ConnectRecord<SourceRecord>

ConnectRecord şöyle. Yani okunan şeyin hangi Kafka topic'e yazılacağını SourceConnector söylüyor. Yazma işlemini Kafka Connect alt yapısı yapıyor

public abstract class ConnectRecord<R extends ConnectRecord<R>> {
  private final String topic;
  private final Integer kafkaPartition;
  private final Schema keySchema;
  private final Object key;
  private final Schema valueSchema;
  private final Object value;
  private final Long timestamp;
  private final Headers headers;
  ...
}