Skip to content

Commit be936f3

Browse files
committed
feat: kafka connect - continuation
1 parent 4a3b6d5 commit be936f3

File tree

1 file changed

+13
-6
lines changed

1 file changed

+13
-6
lines changed

blog/2025-03-15-kafka-connect.md

+13-6
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,16 @@ tags: [java, kafka]
88

99
# How to stream data from Kafka to relational database
1010

11-
In this post I would like to show how to stream data from any topic to relational database.
11+
In this post I would like to show how to stream data from any Kafka topic to relational database with Kafka Connect.
1212

1313
<!-- truncate -->
1414

15-
## What is Kafka Connect
15+
## What is Kafka Connect ?
1616

1717
In short Kafka Connect is a framework facilitating streaming integrations beetwen Kafka and other systems.
1818
You can learn more about it [here](https://developer.confluent.io/courses/kafka-connect/intro/).
1919

20-
There are two types of Kafka Conect workflows. One is *Source* to Kafka.
20+
There are two types of Kafka Connect workflows. One is *Source* to Kafka.
2121

2222
```mermaid
2323
graph LR;
@@ -28,13 +28,20 @@ and the other on Kafka to *Sink*
2828
graph LR;
2929
Kafka --> kc["Kafka Connect"] --> Sink
3030
```
31-
where *Source* and *Sink* are kind of abstractions representing any external system like MongoDb, FTP, File or any relational database like PostgreSql, Oracle or Sql Server.
32-
31+
where *Source* and *Sink* are abstractions representing any external system like MongoDb, FTP, File or any relational database like PostgreSql, Oracle or Sql Server or even othe kafka cluster.
3332

3433
## Kafka Connect Plugins
3534

3635

37-
## How it works
36+
## The Problem
37+
38+
Confluent Kafka JdbcSinkConnector which I will be using requires that the incoming message schema is known to be able map between message fields and sql collumns.
39+
40+
## The Solution
41+
```mermaid
42+
graph LR;
43+
Kafka --> conv["SimpleWrappingConverter"] --> tr["AddMetadataTransformer"] --> PostgreSQL
44+
```
3845

3946

4047
https://github.com/tomaszkubacki/kafka_connect_demo/blob/master/kafka_to_postgresql/kafka_to_postgres.md

0 commit comments

Comments
 (0)