Kafka Connect Mimarisi

Deployment Model

1. Standalone

2. Distributed 

Eğer dağıtık ise bir cluster şeklinde çalışır. Şeklen şöyle

Connector Mimarisi

Bileşenler şöyle

1. Connector Arayüzü

 Connector yazısına bakabilirsiniz

Açıklaması şöyle

Connectors are instances responsible for instantiating and defining/configuring a set of child processes called Tasks.

Açıklaması şöyle

Example to understand Connector and Tasks Difference

Say we have a Postgres database source connector which is responsible to pull data from 5 different tables into Kafka.

The connector would come up first and will figure out that there are 5 tables that need to be polled.

Based on that it will spin up new tasks and each task will have responsibility to poll data from a subset of tables.

2. Tasks

Açıklaması şöyle

Tasks are the most basic unit of pulling/pushing data into/out-of Kafka.

SourceTask : Poll işlemini yapar

SinkTask  : Put işlemini yapar

3. Workers

Açıklaması şöyle

As a user when you submit a certain connector configuration to the connect worker. This config would create the type of connector plugin and in addition to that some more plugin specific configs, a connector instance would be instantiated on one of the workers in the connect cluster.