Debezium ve Kafka Connect İlişkisi - Debezium Connector Altta Kafka Connect Kullanır

Giriş

Açıklaması şöyle. Kısaca Debezium Connector Altta Kafka Connect Kullanır

It leverages the Kafka Connect framework to capture changes from various database systems and publish them as messages to Kafka topics.

Debezium provides a set of connectors for different databases, such as MySQL, PostgreSQL, MongoDB, and Oracle. These connectors read the transaction logs or other change streams of the source database and generate Kafka messages for each change. The messages are then consumed by downstream systems, such as data lakes, data warehouses, or other databases.

By leveraging Kafka, Debezium provides a scalable and fault-tolerant platform for capturing and processing database changes in real-time. Kafka's distributed architecture allows for parallel processing of messages across multiple nodes, and its built-in replication and fault-tolerance features ensure that data is reliably delivered to downstream systems.

Debezium also provides a set of features for data transformation, such as filtering, enrichment, and masking of sensitive data. These features can be applied to the Kafka messages before they are consumed by downstream systems.

Örnek

Debezium kendi içinde bir Kafka Connect bileşeni içeriyor. Kafka Connect'i direkt çalıştırmak istersek şöyle yaparız

cd /opt/kafka/bin
connect-standalone.sh \
  config/connect-standalone.properties \
  config/debezium-oracle-connector.properties

Loglama ayarlarını değiştirmek için şöyle yaparız

$KAFKA_HOME/config/connect-log4j.properties

log4j.rootLogger=DEBUG, stdout, connectAppender