a81d4a16b9
[cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. ( #2762 )
...
This closes #2762 .
1 year ago
da26c36d50
[cdc-runtime] Improve DataSinkWriterOperator logic that only emit latest schema for events that are not CreateTableEvent ( #2771 )
1 year ago
3806c72ef8
[cdc-common] add PublicEvolving annotation to SchemaUtils ( #2756 )
1 year ago
a02454572f
[cdc-runtime] Improve partitioning logics and add tests for PrePartitionOperator( #2761 )
...
This closes #2761 .
1 year ago
1dae8674d4
[cdc-runtime][tests] Introduce Harness test utilities for CDC customized operators
1 year ago
ab3f7d8135
[cdc-runtime] Make sure CreateTableEvent is always sent before DataChangeEvent even if during restoration ( #2767 )
...
This closes #2767 .
1 year ago
70614be28c
[3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
...
This closes #2734 .
1 year ago
cad7424a1d
[3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
1 year ago
ac9bab3dce
[3.0][cdc-common] Introduce binaryRecordData to avoid type serialization and improve performance as well
...
This closes #2748 .
1 year ago
8ee3ea52d5
[cdc-runtime] Use CollectCoordinationResponse as a wrapper of CoordinationResponse
...
The reason we do this is that there is potentially a bug in Flink RPC system, that CoordinationResponse can only be deserialized by AppClassloader instead of user code classloader, so we can't use customize CoordinationResponse class in RPCs (will lead to ClassNotFoundException). As CollectCoordinationResponse is predefined in Flink and shipped in flink-dist, it will be always visible to AppClassloader, and we use its payload for holding the actual serialized custom CoordinationResponse.
This closes #2744
1 year ago
d5981c9acb
[cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
...
When the CDC job runs on Flink cluster, DataSinkWriterOperator is
loaded by user code classloader, while SinkWriterOperator is loaded
by app classloader, and the protection mechanism forbids us from
accessing SinkWriterOperator across different classloaders.
Therefore we have to manually load the SinkWriterOperator in user
code classloader, and use reflections to control it.
This closes #2742
Co-authored-by: lvyanquan <decq12ybhl@gmail.com>
1 year ago
b52e88a43f
[cdc-composer] Introduce partitioning related runtime functions and translator
1 year ago
bf5914e6ac
[cdc-composer] Introduce data sink and schema operator translators
...
This closes #2618
1 year ago
7ab1960466
[cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
...
This closes #2741
1 year ago
e88bea3514
[3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
1 year ago
ace6080c93
[3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes ( #2685 )
1 year ago
6c21046382
[3.0][cdc-runtime] Add TypeInformation for Event class ( #2728 )
1 year ago
70bbac99d5
[3.0][cdc-runtime][tests] Add tests for type serializers
...
This closes #2713 .
1 year ago
5223d6b69d
[3.0][cdc-runtime][hotfix] Move type serializers to runtime module
1 year ago
9e149efcc1
[3.0][cdc-common][minor] Move FlushEvent to common module
1 year ago
80dfc515a1
[3.0][cdc-runtime] add DataSinkWriterOperator to process Event ( #2649 )
1 year ago
d01f0a9791
[pom] Use ${revision} replace specific version. ( #2623 )
1 year ago
e26470fff0
[3.0][cdc-runtime] introduce schema evolution framework API
...
This closes #2604
1 year ago
82a652ca6f
[3.0][kickoff] Introduce basic modules for Flink CDC 3.0
...
This closes #2610
1 year ago
Kunni
Qingsheng Ren
Hang Ruan
joyCurry30
Leonard Xu