6eb980a614
[hotfix][mysql][tests] Avoid canceling a finished job in tests of SpecificStartingOffsetITCase ( #2687 )
...
This closes #2687 .
1 year ago
f7dda3b67b
[docs][zh][hotfix] Remove redundant sections from mysql-cdc documentation ( #2777 )
...
This closes #2777 .
1 year ago
cc9dbc6660
[mongodb][hotfix] Fix MongoDB's rewriteOutputBuffer to emit +I rather than +U ( #2760 ) ( #2760 )
1 year ago
a81d4a16b9
[cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. ( #2762 )
...
This closes #2762 .
1 year ago
99ffbe03d2
[cdc-composer] use DataSinkWriterOperatorFactory to replace all SinkWriterOperatorFactory. ( #2773 )
1 year ago
8edf2a60f0
[cdc-pipeline-connector][values] Avoid using `default` as name of namespace and schema. ( #2774 )
1 year ago
da26c36d50
[cdc-runtime] Improve DataSinkWriterOperator logic that only emit latest schema for events that are not CreateTableEvent ( #2771 )
1 year ago
3806c72ef8
[cdc-common] add PublicEvolving annotation to SchemaUtils ( #2756 )
1 year ago
0cdfd36c08
[3.0][cdc-composer] Introduce pipeline.name config to support custom flink job name ( #2768 )
...
Co-authored-by: Leonard Xu <leonard@apache.org>
This closes #2769 .
1 year ago
57e4793b21
[cdc-base] Extract the common parts of each ScanFetchTask into AbstractScanFetcherTask.
...
This closes #2690 .
1 year ago
369be8c035
[docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial-zh.md ( #2763 )
...
This closes #2763 .
1 year ago
a02454572f
[cdc-runtime] Improve partitioning logics and add tests for PrePartitionOperator( #2761 )
...
This closes #2761 .
1 year ago
1dae8674d4
[cdc-runtime][tests] Introduce Harness test utilities for CDC customized operators
1 year ago
ab3f7d8135
[cdc-runtime] Make sure CreateTableEvent is always sent before DataChangeEvent even if during restoration ( #2767 )
...
This closes #2767 .
1 year ago
c853514a0d
[docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial.md ( #2764 )
...
This closes #2764 .
1 year ago
6e53733df2
[oceanbase] Use global timestamp to start log client ( #2565 )
...
* use global timestamp to start log client
* remove redundant fields
* update log info
1 year ago
4a5f2161aa
[cdc-pipeline-connector][values] Add event set MULTI_SPLITS_SINGLE_TABLE to mock source with multiple splits ( #2737 )
...
This closes #2737 .
1 year ago
071523eae6
[hotfix][cdc-composer] Use correct factory to discovery DataSink ( #2757 )
...
This closes #2757 .
1 year ago
70614be28c
[3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
...
This closes #2734 .
1 year ago
cad7424a1d
[3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
1 year ago
ac9bab3dce
[3.0][cdc-common] Introduce binaryRecordData to avoid type serialization and improve performance as well
...
This closes #2748 .
1 year ago
6a92546340
[build][hotfix] Use the correct format for labeler.yml ( #2758 )
...
This closes #2758 .
1 year ago
30031d0a14
[build] Add github labeler for pull request
...
This closes #2564 .
1 year ago
0bf378dd22
[docs] Add DataStream application package guidance ( #2583 )
...
This closes #2583 .
Co-authored-by: Leonard Xu <leonard@apache.org>
1 year ago
9c3c7faae4
[cdc-common] Introduce TableFilter to filter table objects from given TableId pattern ( #2673 )
...
This closes #2669 .
1 year ago
a26607a027
[cdc-composer] Initialize context with existing config of SourceDef and SinkDef ( #2749 )
...
This closes #2749 .
1 year ago
f697c5fff3
[hotfix][docs] Adjust the correct doc versions ( #2679 )
...
This closes #2679 .
1 year ago
9aed54d70f
[docs][oracle] Fix oracle cdc quick start ( #2727 )
...
This closes #2727 .
1 year ago
8ee3ea52d5
[cdc-runtime] Use CollectCoordinationResponse as a wrapper of CoordinationResponse
...
The reason we do this is that there is potentially a bug in Flink RPC system, that CoordinationResponse can only be deserialized by AppClassloader instead of user code classloader, so we can't use customize CoordinationResponse class in RPCs (will lead to ClassNotFoundException). As CollectCoordinationResponse is predefined in Flink and shipped in flink-dist, it will be always visible to AppClassloader, and we use its payload for holding the actual serialized custom CoordinationResponse.
This closes #2744
1 year ago
1973f64aa3
[cdc-common] MetadataApplier should extends Serializable
...
This closes #2743
1 year ago
d5981c9acb
[cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
...
When the CDC job runs on Flink cluster, DataSinkWriterOperator is
loaded by user code classloader, while SinkWriterOperator is loaded
by app classloader, and the protection mechanism forbids us from
accessing SinkWriterOperator across different classloaders.
Therefore we have to manually load the SinkWriterOperator in user
code classloader, and use reflections to control it.
This closes #2742
Co-authored-by: lvyanquan <decq12ybhl@gmail.com>
1 year ago
f0bf38e0fe
[cdc-composer] Implementation of Flink pipeline composer for chaining everything together
...
This closes #2609
1 year ago
b52e88a43f
[cdc-composer] Introduce partitioning related runtime functions and translator
1 year ago
bf5914e6ac
[cdc-composer] Introduce data sink and schema operator translators
...
This closes #2618
1 year ago
9e64b8f0b6
[cdc-composer] Introduce DataSourceTranslator for composing source
...
This closes #2617
1 year ago
7ab1960466
[cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
...
This closes #2741
1 year ago
5f0343e29e
[cdc-composer] Add FlinkEnvironmentUtils for adding JAR to StreamExecutionEnvironment
1 year ago
32de7776d2
[cdc-common] Define pipeline options and schema change behaviors
...
This closes #2740
1 year ago
6506ba788d
[cdc-common] Support adding Column directly in schema builder
...
This closes #2739
1 year ago
51554d8630
[hotfix][vitess] Fix invalidate package name
...
This closes #2735 .
1 year ago
b4deda97be
[cdc-pipeline-connector][values] Introduce `values` cdc pipeline connector
...
This closes #2675 .
1 year ago
e88bea3514
[3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
1 year ago
40184ae19f
[3.0][cdc-common][minor] Add toString method for some classes.
1 year ago
be286110f1
[3.0][cdc-common] Add SchemaUtil to perform schema evolution ( #2732 )
...
This closes #2732 .
1 year ago
ace6080c93
[3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes ( #2685 )
1 year ago
4770f360d0
[sqlserver] Fix timestamp type parse of sqlserver ( #2708 )
1 year ago
6c21046382
[3.0][cdc-runtime] Add TypeInformation for Event class ( #2728 )
1 year ago
70bbac99d5
[3.0][cdc-runtime][tests] Add tests for type serializers
...
This closes #2713 .
1 year ago
5223d6b69d
[3.0][cdc-runtime][hotfix] Move type serializers to runtime module
1 year ago
74f032aad6
[3.0][cdc-common] Introduce data type for record field in DataChangeEvent
1 year ago
Hang Ruan
ice
Hongshun Wang
Kunni
Qingsheng Ren
gongzhongqiang
l568288g
He Wang
Kunni
skylines