26ff6d2a08
[FLINK-35237][cdc-common] Improve the interfaces and reorganize the directory.
7 months ago
9a4dede3fe
[FLINK-35237][cdc-common] Allow Sink to choose HashFunction in PrePartitionOperator
7 months ago
ad1f554c0e
[FLINK-35647][route] Support symbol replacement to enrich routing rules
...
This closes #3428 .
Co-authored-by: 张田 <zhang_tian@inspur.com>
Co-authored-by: yangshuaitong <duguhoney@gmail.com>
7 months ago
3441a1e2ab
[FLINK-35648][runtime] Allow applying multiple route rules for one single source table ( #3425 )
8 months ago
414720d1c5
[FLINK-35464][cdc] Fixes operator state backwards compatibility from CDC 3.0.x and add migration tests ( #3369 )
8 months ago
88afc5f18d
[FLINK-35323][cdc-runtime] Fix transform failure when one rule matches multiple tables with incompatible schema
...
This closes #3313 .
8 months ago
f61f0f44bd
[FLINK-35264][cdc][runtime] Fix multiple transform rules do not take effect ( #3280 )
9 months ago
23a67dcdb9
[FLINK-35255][cdc][runtime] DataSinkWriterOperator adds overrides for the snapshotState and processWatermark methods ( #3271 )
9 months ago
75a553eb92
[FLINK-34878][cdc][transform] Flink CDC pipeline transform supports CASE WHEN ( #3228 )
9 months ago
0108d0e5d1
[FLINK-35259][cdc][transform] Fix FlinkCDC pipeline transform can't deal timestamp field ( #3278 )
9 months ago
6258bec5bb
[FLINK-35256][cdc][runtime] Fix transform node does not respect type nullability ( #3272 )
10 months ago
9cc3451ddf
[FLINK-35251][cdc][runtime] Fix bug of serializing derivation mapping in SchemaDerivation
...
This closes #3267 .
10 months ago
c1e1aa48d9
[FLINK-34648][cdc] Avoid RPC timeout when applying SchemaChangeEvent to downstream external systems
...
This closes #3128 .
10 months ago
6017b16528
[FLINK-34653][cdc][route] Support table merging with route rules
...
This closes #3129 .
10 months ago
18dd51fe22
[FLINK-34905][cdc-connector][mysql] Fix the default length of CHAR/BINARY data type of Add column DDL ( #3145 )
10 months ago
8c5437a1f1
[FLINK-34952][cdc-composer][sink] Flink CDC pipeline supports SinkFunction ( #3204 )
10 months ago
6510e670fa
[FLINK-34959][cdc] Update old flink-cdc-connectors artifactId ( #3200 )
10 months ago
a35b8dd44c
[build] Optimize pom to solve the CI error
10 months ago
bcad5d9d11
[cdc-runtime] Introduce TransformSchemaOperator and TransformDataOperator to support transformation
10 months ago
5f4f86fb31
[hotfix] Change old com.ververica dependencies to org.apache.flink ( #3110 )
11 months ago
e10c8691ad
[FLINK-34584][cdc][build] Rename package from com.ververica to org.apache.flink
...
This closes #3089 .
11 months ago
eeb4bab92d
[FLINK-34184][cdc][license] Update Copyright from Ververica to Apache
11 months ago
c98f899af8
[cdc-common][cdc-pipelines][cdc-runtime] Use name instead of column in DropColumnEvent/AddColumnEvent. ( #2888 )
1 year ago
f2c81c7d08
[hotfix] Fix BinaryRecordData deserialize
1 year ago
e6bb8e9a55
[hotfix][cdc-rumtime][minor] Clean up useless code in BinaryRecordDataSerializer ( #2845 )
...
Co-authored-by: xuyu <11161569@vivo.com>
1 year ago
daceb42168
[build] Use ${project.version} instead of ${revision} for dependency version
...
This is documented in https://maven.apache.org/maven-ci-friendly.html#dependencies
1 year ago
30c8254af3
[3.0][cdc-runtime] Remove waitForFlushSuccess field in request handler ( #2812 )
1 year ago
1766e82d7c
[3.0][cdc-common & runtime] Remove useless flink imports
1 year ago
9ce8450cf2
[cdc-runtime][hotfix] Setup waitFlushSuccess before responding operator to flush. ( #2805 )
...
This closes #2805 .
There might be a case that sink flushes faster than schema operator requesting release upstream. In that case waitFlushSuccess has not been set when sink has flushed successfully, which will lead to a NullPointerException.
1 year ago
e435004eda
[cdc-common][hotfix] Remove useless flink imports ( #2803 )
...
This closes #2803 .
1 year ago
a8f4ba8cc6
[cdc-composer][hotfix] Always chain operator by default
1 year ago
4843ab2432
[cdc-runtime] SchemaRegistry should complete the future after making the checkpoint ( #2800 )
...
This closes #2800 .
1 year ago
416562c9ba
[cdc-runtime] Add RouteFunction for routing change events
1 year ago
a81d4a16b9
[cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. ( #2762 )
...
This closes #2762 .
1 year ago
da26c36d50
[cdc-runtime] Improve DataSinkWriterOperator logic that only emit latest schema for events that are not CreateTableEvent ( #2771 )
1 year ago
3806c72ef8
[cdc-common] add PublicEvolving annotation to SchemaUtils ( #2756 )
1 year ago
a02454572f
[cdc-runtime] Improve partitioning logics and add tests for PrePartitionOperator( #2761 )
...
This closes #2761 .
1 year ago
1dae8674d4
[cdc-runtime][tests] Introduce Harness test utilities for CDC customized operators
1 year ago
ab3f7d8135
[cdc-runtime] Make sure CreateTableEvent is always sent before DataChangeEvent even if during restoration ( #2767 )
...
This closes #2767 .
1 year ago
70614be28c
[3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
...
This closes #2734 .
1 year ago
cad7424a1d
[3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
1 year ago
ac9bab3dce
[3.0][cdc-common] Introduce binaryRecordData to avoid type serialization and improve performance as well
...
This closes #2748 .
1 year ago
8ee3ea52d5
[cdc-runtime] Use CollectCoordinationResponse as a wrapper of CoordinationResponse
...
The reason we do this is that there is potentially a bug in Flink RPC system, that CoordinationResponse can only be deserialized by AppClassloader instead of user code classloader, so we can't use customize CoordinationResponse class in RPCs (will lead to ClassNotFoundException). As CollectCoordinationResponse is predefined in Flink and shipped in flink-dist, it will be always visible to AppClassloader, and we use its payload for holding the actual serialized custom CoordinationResponse.
This closes #2744
1 year ago
d5981c9acb
[cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
...
When the CDC job runs on Flink cluster, DataSinkWriterOperator is
loaded by user code classloader, while SinkWriterOperator is loaded
by app classloader, and the protection mechanism forbids us from
accessing SinkWriterOperator across different classloaders.
Therefore we have to manually load the SinkWriterOperator in user
code classloader, and use reflections to control it.
This closes #2742
Co-authored-by: lvyanquan <decq12ybhl@gmail.com>
1 year ago
b52e88a43f
[cdc-composer] Introduce partitioning related runtime functions and translator
1 year ago
bf5914e6ac
[cdc-composer] Introduce data sink and schema operator translators
...
This closes #2618
1 year ago
7ab1960466
[cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
...
This closes #2741
1 year ago
e88bea3514
[3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
1 year ago
ace6080c93
[3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes ( #2685 )
1 year ago
6c21046382
[3.0][cdc-runtime] Add TypeInformation for Event class ( #2728 )
1 year ago
70bbac99d5
[3.0][cdc-runtime][tests] Add tests for type serializers
...
This closes #2713 .
1 year ago
5223d6b69d
[3.0][cdc-runtime][hotfix] Move type serializers to runtime module
1 year ago
9e149efcc1
[3.0][cdc-common][minor] Move FlushEvent to common module
1 year ago
80dfc515a1
[3.0][cdc-runtime] add DataSinkWriterOperator to process Event ( #2649 )
1 year ago
d01f0a9791
[pom] Use ${revision} replace specific version. ( #2623 )
1 year ago
e26470fff0
[3.0][cdc-runtime] introduce schema evolution framework API
...
This closes #2604
1 year ago
82a652ca6f
[3.0][kickoff] Introduce basic modules for Flink CDC 3.0
...
This closes #2610
1 year ago
Leonard Xu
鼎昕
yuxiqian
yanghuaiGit
Wink
Qingsheng Ren
Kunni
Qishang Zhong
Hongshun Wang
xleoken
wenmo
Hang Ruan
joyCurry30
GOODBOY008
xuzifu666
Leonard Xu
Kunni