d5981c9acb
[cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
...
When the CDC job runs on Flink cluster, DataSinkWriterOperator is
loaded by user code classloader, while SinkWriterOperator is loaded
by app classloader, and the protection mechanism forbids us from
accessing SinkWriterOperator across different classloaders.
Therefore we have to manually load the SinkWriterOperator in user
code classloader, and use reflections to control it.
This closes #2742
Co-authored-by: lvyanquan <decq12ybhl@gmail.com>
1 year ago
f0bf38e0fe
[cdc-composer] Implementation of Flink pipeline composer for chaining everything together
...
This closes #2609
1 year ago
b52e88a43f
[cdc-composer] Introduce partitioning related runtime functions and translator
1 year ago
bf5914e6ac
[cdc-composer] Introduce data sink and schema operator translators
...
This closes #2618
1 year ago
9e64b8f0b6
[cdc-composer] Introduce DataSourceTranslator for composing source
...
This closes #2617
1 year ago
7ab1960466
[cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
...
This closes #2741
1 year ago
5f0343e29e
[cdc-composer] Add FlinkEnvironmentUtils for adding JAR to StreamExecutionEnvironment
1 year ago
32de7776d2
[cdc-common] Define pipeline options and schema change behaviors
...
This closes #2740
1 year ago
6506ba788d
[cdc-common] Support adding Column directly in schema builder
...
This closes #2739
1 year ago
51554d8630
[hotfix][vitess] Fix invalidate package name
...
This closes #2735 .
1 year ago
b4deda97be
[cdc-pipeline-connector][values] Introduce `values` cdc pipeline connector
...
This closes #2675 .
1 year ago
e88bea3514
[3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
1 year ago
40184ae19f
[3.0][cdc-common][minor] Add toString method for some classes.
1 year ago
be286110f1
[3.0][cdc-common] Add SchemaUtil to perform schema evolution ( #2732 )
...
This closes #2732 .
1 year ago
ace6080c93
[3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes ( #2685 )
1 year ago
4770f360d0
[sqlserver] Fix timestamp type parse of sqlserver ( #2708 )
1 year ago
6c21046382
[3.0][cdc-runtime] Add TypeInformation for Event class ( #2728 )
1 year ago
70bbac99d5
[3.0][cdc-runtime][tests] Add tests for type serializers
...
This closes #2713 .
1 year ago
5223d6b69d
[3.0][cdc-runtime][hotfix] Move type serializers to runtime module
1 year ago
74f032aad6
[3.0][cdc-common] Introduce data type for record field in DataChangeEvent
1 year ago
fb9a8a4257
[3.0][cdc-common][minor] Improve public methods for schema event
1 year ago
9e149efcc1
[3.0][cdc-common][minor] Move FlushEvent to common module
1 year ago
18936a3676
[mongodb] Incremental snapshot source supports comma-separated format parameter for databases and collections ( #2724 )
1 year ago
3a80de7676
[3.0][cdc-common] Use built-in ConfigOption for cdc-common module ( #2716 )
...
This closes #2716 .
1 year ago
a1b67a7585
[3.0][cdc-common] Add ConText for Source/Sink initialization. ( #2672 )
1 year ago
af5a0cef3f
[3.0][cdc-common] Add TypeSerializer implementations ( #2683 )
1 year ago
00f8156671
[build] Add header copyright check for java files
1 year ago
dab7ba4cb5
[common] Introduce Configuration and related utilities ( #2681 )
1 year ago
7192e5300b
[oracle] Skip unsupported nested table for legacy debezium source function
1 year ago
0565bf844c
[oracle] Skip unsupported nested table for Incremental OracleSource
1 year ago
995d524416
[3.0][common] Use built-in API annotations to make common module free with flink
...
This closes #2695 .
1 year ago
ce4621a549
[3.0][common] Introduce built-in API annotations for project
...
This closes #2693 .
1 year ago
1e6b983bdd
[build] Bump Flink version to 1.18.0 ( #2463 )
...
This closes #2670 .
1 year ago
07818221d2
[build][tests] Enable CI tests for CDC 3.0 modules ( #2678 )
...
This closes #2671
1 year ago
4930698a9b
[3.0][cdc-common] Introduce internal data structures and generic implementations ( #2688 )
...
This closes #2651 .
1 year ago
98c929e6e4
[composer] Connector factory and JAR discovery utilities ( #2662 )
1 year ago
7da3eaef77
[postgres-cdc] Add tests for latest-offset startup strategy ( #2527 )
1 year ago
de45676faa
[3.0][cdc-common] Change DataChangeEvent from interface to class, and add implementation for DataChangeEvent. ( #2677 )
...
* [3.0][cdc-common] Change DataChangeEvent from interface to class, and add implementation for DataChangeEvent.
* [3.0][cdc-common] Remove @param for code cleaning.
* [3.0][cdc-common] Make the DataChangeEvent constructor private.
1 year ago
c7568c3eab
[build][tests] Enable CI tests for CDC 3.0 modules
...
This closes #2671 .
1 year ago
80dfc515a1
[3.0][cdc-runtime] add DataSinkWriterOperator to process Event ( #2649 )
1 year ago
c52ccfa5c6
[3.0][cdc-common] Add SchemaChangeEvent implementations ( #2664 )
1 year ago
1329a48bcc
[cli] Implementation of CLI frontend ( #2660 )
...
* [cli] Introduce parser interface and YAML implementation for parsing pipeline definition files into objects
* [cli] Implementation of CLI frontend
1 year ago
112b835743
[docs] Remove flink 1.13 from supported flink versions
...
This closes #2526 .
1 year ago
2fa2787214
[3.0][cdc-common] Add GenericRecordData implementation for RecordData
...
This closes #2644 .
1 year ago
8826e22932
[3.0][cdc-common] Keep columns in Schema in fixed order
1 year ago
5c2ff1ef78
[3.0][cdc-common] Use RecordData to represent before and after record.
1 year ago
80603a6465
[build] Introduce -Dfast property for skipping code style checks
1 year ago
07cbbace52
[test] Bump maven-surefire-plugin version to 3.0.0-M5
1 year ago
38cb96e67c
[test] Introduce JUnit 5 and AssertJ dependencies
1 year ago
99d1ad5e52
[flink-cdc-composer] Base interfaces and models for composer and pipeline definition ( #2656 )
...
* [flink-cdc-composer] Base interfaces and models for Flink CDC composer and pipeline definition
* [flink-cdc-composer] Flink implementation of composer and execution.
Note that Flink composer doesn't have any solid implementation now.
1 year ago
Qingsheng Ren
gongzhongqiang
lvyanquan
Hang Ruan
Hongshun Wang
Shawn Huang
joyCurry30
Leonard Xu
zyi728
Maciej Bryński