CDC Connectors for Apache Flink<sup>®</sup> is a set of source connectors for <ahref="https://flink.apache.org/">Apache Flink<sup>®</sup></a>, ingesting changes from different databases using change data capture (CDC).
The CDC Connectors for Apache Flink<sup>®</sup> integrate Debezium as the engine to capture data changes. So it can fully leverage the ability of Debezium. See more about what is [Debezium](https://github.com/debezium/debezium).
@ -9,15 +9,15 @@ The CDC Connectors for Apache Flink<sup>®</sup> integrate Debezium as the engin
CDC Streaming ELT Framework is a stream data integration framework that aims to provide users with a more robust API. It allows users to configure their data synchronization logic through customized Flink operators and job submission tools. The framework prioritizes optimizing the task submission process and offers enhanced functionalities such as whole database synchronization, sharding, and schema change synchronization.
The data types flowing in the Flink CDC 3.0 framework are referred to as **Event**, which represent the change events generated by external systems.
Each event is marked with a **Table ID** for which the change occurred. Events are categorized into `SchemaChangeEvent` and `DataChangeEvent`, representing changes in table structure and data respectively.
**Data Source** Connector captures the changes in external systems and converts them into events as the output of the synchronization task. It also provides a `MetadataAccessor` for the framework to read the metadata of the external systems.
**Data Sink** connector receives the change events from **Data Source** and applies them to the external systems. Additionally, `MetadataApplier` is used to apply metadata changes from the source system to the target system.
Since events flow from the upstream to the downstream in a pipeline manner, the data synchronization task is referred as a **Data Pipeline**. A **Data Pipeline** consists of a **Data Source**, **Route**, **Transform** and **Data Sink**. The transform can add extra content to events, and the router can remap the `Table ID`s corresponding to events.
Now let's introduce more details about the concepts you need to know when using the CDC Streaming ELT Framework.
### Table ID
When connecting to external systems, it is necessary to establish a mapping relationship with the storage objects of the external system. This is what `Table Id` refers to.
To be compatible with most external systems, the `Table ID` is represented by a 3-tuple : (namespace, schemaName, table). Connectors need to establish the mapping between Table ID and storage objects in external systems.
For instance, a table in MySQL/Doris is mapped to (null, database, table) and a topic in a message queue system such as Kafka is mapped to (null, null, topic).
### Data Source
Data Source is used to access metadata and read the changed data from external systems.
A Data Source can read data from multiple tables simultaneously.
To describe a data source, the follows are required:
* Type: The type of the source, such as MySQL, Postgres.
* Name: The name of the source, which is user-defined (optional, with a default value provided).
* Other custom configurations for the source.
For example, we could use `yaml` files to define a mysql source
```yaml
source:
type: mysql
name: mysql-source #optional，description information
* escription: Routing rule description(optional, default value provided)
For example, if synchronize the table 'web_order' in the database 'mydb' to a Kafka topic 'ods_web_order', we can use this yaml file to define this route：
```yaml
route:
source-table: mydb.default.web_order
sink-table: ods_web_order
description: sync table to one destination table with given prefix ods_
```
### Data Pipeline
Since events flow from the upstream to the downstream in a pipeline manner, the data synchronization task is also referred as a Data Pipeline.
To describe a Data Pipeline, the follows are required:
* Name: The name of the pipeline, which will be submitted to the Flink cluster as the job name.
* Other advanced capabilities such as automatic table creation, schema evolution, etc., will be implemented.
For example, we can use this yaml file to define a pipeline:
Hongshun Wang
1 year ago
committed by
GitHub
