c98f899af8
[cdc-common][cdc-pipelines][cdc-runtime] Use name instead of column in DropColumnEvent/AddColumnEvent. ( #2888 )
1 year ago
f2c81c7d08
[hotfix] Fix BinaryRecordData deserialize
1 year ago
e6bb8e9a55
[hotfix][cdc-rumtime][minor] Clean up useless code in BinaryRecordDataSerializer ( #2845 )
...
Co-authored-by: xuyu <11161569@vivo.com>
1 year ago
daceb42168
[build] Use ${project.version} instead of ${revision} for dependency version
...
This is documented in https://maven.apache.org/maven-ci-friendly.html#dependencies
1 year ago
30c8254af3
[3.0][cdc-runtime] Remove waitForFlushSuccess field in request handler ( #2812 )
1 year ago
1766e82d7c
[3.0][cdc-common & runtime] Remove useless flink imports
1 year ago
9ce8450cf2
[cdc-runtime][hotfix] Setup waitFlushSuccess before responding operator to flush. ( #2805 )
...
This closes #2805 .
There might be a case that sink flushes faster than schema operator requesting release upstream. In that case waitFlushSuccess has not been set when sink has flushed successfully, which will lead to a NullPointerException.
1 year ago
e435004eda
[cdc-common][hotfix] Remove useless flink imports ( #2803 )
...
This closes #2803 .
1 year ago
a8f4ba8cc6
[cdc-composer][hotfix] Always chain operator by default
1 year ago
4843ab2432
[cdc-runtime] SchemaRegistry should complete the future after making the checkpoint ( #2800 )
...
This closes #2800 .
1 year ago
416562c9ba
[cdc-runtime] Add RouteFunction for routing change events
1 year ago
a81d4a16b9
[cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. ( #2762 )
...
This closes #2762 .
1 year ago
da26c36d50
[cdc-runtime] Improve DataSinkWriterOperator logic that only emit latest schema for events that are not CreateTableEvent ( #2771 )
1 year ago
3806c72ef8
[cdc-common] add PublicEvolving annotation to SchemaUtils ( #2756 )
1 year ago
a02454572f
[cdc-runtime] Improve partitioning logics and add tests for PrePartitionOperator( #2761 )
...
This closes #2761 .
1 year ago
1dae8674d4
[cdc-runtime][tests] Introduce Harness test utilities for CDC customized operators
1 year ago
ab3f7d8135
[cdc-runtime] Make sure CreateTableEvent is always sent before DataChangeEvent even if during restoration ( #2767 )
...
This closes #2767 .
1 year ago
70614be28c
[3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
...
This closes #2734 .
1 year ago
cad7424a1d
[3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
1 year ago
ac9bab3dce
[3.0][cdc-common] Introduce binaryRecordData to avoid type serialization and improve performance as well
...
This closes #2748 .
1 year ago
8ee3ea52d5
[cdc-runtime] Use CollectCoordinationResponse as a wrapper of CoordinationResponse
...
The reason we do this is that there is potentially a bug in Flink RPC system, that CoordinationResponse can only be deserialized by AppClassloader instead of user code classloader, so we can't use customize CoordinationResponse class in RPCs (will lead to ClassNotFoundException). As CollectCoordinationResponse is predefined in Flink and shipped in flink-dist, it will be always visible to AppClassloader, and we use its payload for holding the actual serialized custom CoordinationResponse.
This closes #2744
1 year ago
d5981c9acb
[cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
...
When the CDC job runs on Flink cluster, DataSinkWriterOperator is
loaded by user code classloader, while SinkWriterOperator is loaded
by app classloader, and the protection mechanism forbids us from
accessing SinkWriterOperator across different classloaders.
Therefore we have to manually load the SinkWriterOperator in user
code classloader, and use reflections to control it.
This closes #2742
Co-authored-by: lvyanquan <decq12ybhl@gmail.com>
1 year ago
b52e88a43f
[cdc-composer] Introduce partitioning related runtime functions and translator
1 year ago
bf5914e6ac
[cdc-composer] Introduce data sink and schema operator translators
...
This closes #2618
1 year ago
7ab1960466
[cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
...
This closes #2741
1 year ago
e88bea3514
[3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
1 year ago
ace6080c93
[3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes ( #2685 )
1 year ago
6c21046382
[3.0][cdc-runtime] Add TypeInformation for Event class ( #2728 )
1 year ago
70bbac99d5
[3.0][cdc-runtime][tests] Add tests for type serializers
...
This closes #2713 .
1 year ago
5223d6b69d
[3.0][cdc-runtime][hotfix] Move type serializers to runtime module
1 year ago
9e149efcc1
[3.0][cdc-common][minor] Move FlushEvent to common module
1 year ago
80dfc515a1
[3.0][cdc-runtime] add DataSinkWriterOperator to process Event ( #2649 )
1 year ago
d01f0a9791
[pom] Use ${revision} replace specific version. ( #2623 )
1 year ago
e26470fff0
[3.0][cdc-runtime] introduce schema evolution framework API
...
This closes #2604
1 year ago
82a652ca6f
[3.0][kickoff] Introduce basic modules for Flink CDC 3.0
...
This closes #2610
1 year ago
joyCurry30
GOODBOY008
xuzifu666
Qingsheng Ren
Hang Ruan
Leonard Xu
Kunni