Kafka Connect SourceConnector Sınıfı

Giriş

Şu satırı dahil ederiz. Soyut bir sınıftır

import org.apache.kafka.connect.source.SourceConnector;

config metodu

Medotun için şuna benzer

@Override
public ConfigDef config() {
    return CONFIG_DEF;
}

Açıklaması şöyle

create fewer tasks if it cannot achieve this level of parallelism.aaThe ConfigDef exposes what configuration you require for your connector and hopefully how it manipulates your connector.

context metodu

Metodu şöyle

@Override
protected SourceConnectorContext context() {
  return (SourceConnectorContext) context;
}

taskConfigs metodu

İmzası şöyle

public abstract List<Map<String, String>> taskConfigs(int maxTasks);

Açıklaması şöyle

The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this level of parallelism.

maxTasks alanı kullanılan connector tipine göre bazen dikkate alınıyor bazen alınmıyor. 

Örneğin io.debezium.connector.mysql.MySqlConnector, io.debezium.connector.postgresql.PostgresConnector bu değeri dikkate almıyor. Açıklaması şöyle

Q : How can I increase the tasks.max for debezium sql connnector?

A : It's not possible.

The database bin log must be read sequentially by only one task.

Run multiple connectors for different tables if you want to distribute workload

Ama mesela io.confluent.connect.jdbc.JdbcSourceConnector dikkate alıyor. O yüzden dokümantasyona bakmak lazım

Docker ve Debezium

Örnek

Şöyle yaparız

docker run -it --name connect --net=host -p 8083:8083 \
-e GROUP_ID=1 \
-e CONFIG_STORAGE_TOPIC=my-connect-configs \
-e OFFSET_STORAGE_TOPIC=my-connect-offsets \
-e BOOTSTRAP_SERVERS=localhost:9092 \
-e CONNECT_TOPIC_CREATION_ENABLE=true \
-v ~/.aws/config:/kafka/.aws/config \
-v ~/Downloads/kafka-connect-iceberg-sink-0.1.3-shaded.jar:/kafka/connect/kafka-connect-iceberg-sink-0.1.3-shaded.jar \
debezium/connect