a35b8dd44c
[build] Optimize pom to solve the CI error
10 months ago
bcad5d9d11
[cdc-runtime] Introduce TransformSchemaOperator and TransformDataOperator to support transformation
10 months ago
f7f248cda1
[build] fix inconsistent Kafka shading among cdc connectors ( #2988 )
10 months ago
5f4f86fb31
[hotfix] Change old com.ververica dependencies to org.apache.flink ( #3110 )
10 months ago
14b81dc6ec
[FLINK-34715][cdc][cdc-connector][mysql] Correctly close BinlogSplitReader in MySQL UTCase
...
This closes #3161 .
11 months ago
26eb6a99b9
[hotfix][cdc][build] Include cdc-common module when shaded for module flink-sql-connector-mysql-cdc
11 months ago
86272bf102
[FLINK-34183][cdc][license] Update NOTICE files ( #3107 )
11 months ago
e10c8691ad
[FLINK-34584][cdc][build] Rename package from com.ververica to org.apache.flink
...
This closes #3089 .
11 months ago
eeb4bab92d
[FLINK-34184][cdc][license] Update Copyright from Ververica to Apache
11 months ago
6da9684999
[cdc-connector][oracle] Use database name in upper case at the snapshot phase to fix fetching two different db name for the same table at the snapshot phase and incremental phase( #2088 )
11 months ago
ea7b0e8dbb
[cdc-base] Support to parse Oracle/SqlServer/PostgresSQL CDC schema change events ( #2315 )
11 months ago
a2ad627fed
[cdc-connector][mongodb] Expose capture newly added tables ability to Mongodb CDC source
11 months ago
03a287a9c2
[cdc-connector][postgres] Expose capture newly added tables ability to Postgres CDC source
11 months ago
1eeda75eaa
[cdc-connector][cdc-base] Support capture newly added tables with un-blocking stream read for CDC Framework（#3024）
11 months ago
1562bc29a3
[cdc-base] Support Scan Newly Added Tables feature
11 months ago
0ca718c31f
[cdc-connector][oracle] Expose SNAPSHOT_ONLY mode for Oracle CDC Source ( #3005 )
1 year ago
d058102c5c
[cdc-connector][oracle]Add oracle 21 version to OracleValidator ( #3054 )
1 year ago
82ccf00e6a
[hotfix][cdc-connector][mongodb] Fix scanFullChangelog requires all collections
1 year ago
7ea197242f
[debezium] Upgrade to Debezium 1.9.8.Final( #3033 ) ( #3034 )
1 year ago
10e542c3ff
[cdc-connector][sqlserver] sqlserver support use specific chunk column as a split key ( #2972 )
...
This closes #2971 .
1 year ago
3be3417de2
[cdc-connector][postgres] Fix potential PostgresSQL database connection leak ( #3013 )
...
This closes #2980 .
1 year ago
20e44e6966
[cdc-connector][oracle] Oracle supports user specific chunk column ( #2976 )
...
This closes #2973 .
1 year ago
c98f899af8
[cdc-common][cdc-pipelines][cdc-runtime] Use name instead of column in DropColumnEvent/AddColumnEvent. ( #2888 )
1 year ago
a2d36a892e
[cdc-connector][mongodb][hotfix] Stop executing log reading in MongoDBStreamFetchTask#close ( #3029 )
...
This closes #3029 .
1 year ago
9ce36569fb
[cdc-connector][postgres] Fix data lost problem when new lsn committed to slot between snapshotState and notifyCheckpointComplete ( #2539 )
...
This closes #2538 .
Co-authored-by: sammieliu <sammieliu@tencent.com>
1 year ago
e3d6c7e0aa
[cdc-connector][postgres] Apply DBZ-5398 postgres connector to handle functional unique index ( #2842 )
...
This closes #2710 .
1 year ago
9abf2cf900
[hotfix][cdc-connector][mysql] Skip SchemaChangeEvents that were not included in capturedTableFilter ( #2986 )
...
This closes #2981
1 year ago
e3f6829263
[cdc-connector][mongodb] Add SNAPSHOT_ONLY mode for Mongodb CDC Source
1 year ago
0e4f835f6d
[cdc-connector][Postgres] Add SNAPSHOT_ONLY mode for Postgres CDC Source
1 year ago
1cbdb4f8ae
[cdc-connector][cdc-base] Add SNAPSHOT_ONLY mode for Incremental CDC Source framework ( #2901 )
1 year ago
3bf886b94d
[cdc-connector][mysql] Add SNAPSHOT_ONLY mode for Mysql CDC Source ( #2901 )
1 year ago
35d3c0ff28
[cdc-base][hotfix] Fix SourceSplitSerializer to get totalFinishedSplitSize from recovery in 4th version
...
This closes #2992 .
1 year ago
a24c5d1b24
[cdc][e2e] Unify logic to clean docker image when connector e2e test finished ( #2978 )
1 year ago
f3762a1b30
[pipeline-connector][mysql] fix timestamp with timezone format ( #2952 )
...
* fix ts with tz parser
* test timestamp with default value
* fix related test
* use timestamp string in test cases
1 year ago
15a734efa0
[cdc-connector][oracledb] Fix oracle read exceed high_watermark in backfill phase
1 year ago
1b643026fe
[pipeline-connector][starrocks] Optimize char type mapping for starrocks pipeline connector ( #2895 )
1 year ago
812c33d27b
[pipeline-connector][doris][starrocks] Remove SimpleDateFormat for thread safe ( #2954 )
1 year ago
b42bc60ab9
[source-connector][mysql] speedup searching binlog by timestamp ( #2924 )
...
* [feature] speedup search mysql binlog when mysql-cdc-connector consume binlog by timestamp
* improve code
* improve code
* code improve
* test case add and code improve
* code improve
* code improve
* code improve, ut add and bugfix for purged log
---------
Co-authored-by: TJX2014 <tangjinxin@deepexi.com>
1 year ago
055b0af477
[cdc-common][pipeline-doris] Fix typo and wrong value for 'precision' of time type. ( #2886 )
1 year ago
a5fa2dc6c8
[hotfix][cdc-connector][mysql] Expose row_kind metadata column for mysql cdc source. ( #2927 )
...
We use 'row_kind' name instead of 'op' or 'op_type', the operation type is a concept of database and database only has 'update'
type and hasn't 'update_before' and 'update_after' type.
1 year ago
8eba910063
[cdc-connector][mysql] Support metadata 'operation' virtual column for MySql Connector. ( #2913 )
1 year ago
6b6d9fcb0b
[cdc-connector][mongodb] Avoid mongodb source to read data after high_watermark in backfill phase ( #2893 )
1 year ago
26dfeeb563
[cdc-connector][oceanbase] Fallback to current timestamp when query gts failure ( #2868 )
1 year ago
2c557c6a78
[cdc-connector][sqlserver][tests] Fix UT errors by correcting right output ( #2864 )
1 year ago
08beca52d0
[cdc] Add Deprecated annotation to source function which connector implemented incremental
1 year ago
568ed79005
[postgres] Add commits of PostgresConnection back to distinguish with debezium's origin class ( #2839 )
1 year ago
08d6b8178f
[cdc-connector][sqlserver] Fix SqlServerTableSource params order error ( #2847 )
1 year ago
f15faac298
[docs] Fix overview docs to remove prefix of pipeline options ( #2833 )
1 year ago
72bcb1df02
[pipeline-connector][starrocks] Fix char/varchar length inconsistency between cdc and starrocks ( #2829 )
1 year ago
c484bb447c
[build][hotfix] Remove duplicate dependency ( #2828 )
1 year ago
daceb42168
[build] Use ${project.version} instead of ${revision} for dependency version
...
This is documented in https://maven.apache.org/maven-ci-friendly.html#dependencies
1 year ago
8a08ded410
[cdc-pipeline-connector][mysql] Fix precision problem of BIT type conversion ( #2820 )
...
This closes #2820 .
1 year ago
f970444efa
[cdc-source-connector][mysql] fix deserialization issue on table metadata binlog event ( #2682 )
...
Co-authored-by: EchoLee5 <39044001+EchoLee5@users.noreply.github.com>
1 year ago
7d7c1af553
[pipeline-connector][mysql] Remove unnecessary serverTimeZone in DebeziumEventDeserializationSchema ( #2816 )
1 year ago
63b39ba22c
[cdc-source-connector][mysql] Fix attempted load of isSuspended in MySqlSplitSerializer::deserializeSplit when cdc version <= 2.2.0 ( #2550 )
...
Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
1 year ago
8c75db145d
[cdc-pipeline-connector][mysql] Ensure the inference of MEDIUMINT_UNSIGNED type matches INT type in table schema ( #2811 )
1 year ago
a10e07fd36
[cdc-connector][oceanbase] Add option for obcdc extra configs ( #2543 )
...
This closes #2543 .
1 year ago
0eae214183
[pipeline-connector][mysql] Enable send schema change by default ( #2815 )
...
This closes #2815 .
1 year ago
33b7c85093
[pipeline-connector][starrocks] Add starrocks pipeline connector ( #2765 )
...
This closes #2645 .
1 year ago
4abd86a500
[pipeline-connector][doris] add doris pipeline connector. ( #2810 )
...
Co-authored-by: wudi <676366545@qq.com>
1 year ago
8ca3091805
[cdc-pipeline-connector][mysql] Ensure the precision inference of DECIMAL type matches table schema
1 year ago
1766e82d7c
[3.0][cdc-common & runtime] Remove useless flink imports
1 year ago
08c5dfed83
[cdc-pipeline-connector][mysql] Add tests for mysql pipeline connector
...
Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
1 year ago
174282d3ef
[cdc-pipeline-connector][mysql] Support more mysql types
1 year ago
ec3d558485
[cdc-pipeleine-connecotr][mysql] Send CreateTableEvent in the mysql pipeline connector
1 year ago
10ccbde023
[cdc-pipeline-connector][mysql] Parse the alter statement to generate SchemaChangeEvent
1 year ago
d0cce26b76
[cdc-pipeline-connector][mysql] Introduce mysql cdc pipeline DataSource
1 year ago
95921d5620
[cdc-common] Introduce "pipeline.local-time-zone" config option which help handle time zone well ( #2797 )
...
This closes #2797 .
1 year ago
2d9dc018c5
[cdc-connector][sqlserver][tests] Fix UT errors by increasing the wait time after committing SQL ( #2799 )
...
This closes #2799 .
1 year ago
173deb5248
[cdc-composer] Add common and runtime JAR when submit ( #2795 )
...
This closes #2795 .
1 year ago
5f318346ff
[cdc-connector][debezium] Supports conversion from binary bytes to base64 encoded string ( #2435 )
...
This closes #2435 .
1 year ago
d1c3f8500b
[cdc-connector][mysql] MysqlTableSource return metadata in a fixed order ( #2578 )
1 year ago
72d33c8965
[cdc-connector][mysql] Fix snapshot fetch size conf does not take effect ( #2766 )
1 year ago
5c00f76d0f
[cdc-connector][oracle] Expose ability to skip backfill in snapshot period for Oracle CDC ( #2793 )
...
This closes #2793 .
1 year ago
ef1c095763
[cdc-connector][oracle] Fix wrong table path during backfill phase which leads to potential data loss ( #2707 )
...
This closes #2707 .
1 year ago
cc9d368b63
[cdc-composer][tests] Add integration tests for FlinkPipelineComposer ( #2776 )
...
This closes #2766 .
1 year ago
f57d6f6e67
[cdc-connector][sqlserver] Expose ability to skip backfill in snapshot period for SqlServer CDC.( #2783 )
...
This closes #2783 .
1 year ago
a6379358e4
[cdc-connector][mysql] Expose ability to skip backfill in snapshot period for MySQL CDC.
1 year ago
e7c713eca6
[cdc-connector][mongodb] Expose ability to skip backfill in snapshot period for Mongodb CDC.
1 year ago
c6cb3b2371
[cdc-connector][postgres] Expose ability to skip backfill in snapshot period for Postgres CDC.
1 year ago
4e248fad33
[cdc-connector][base] Support skip snapshot backfill for incremental snapshot connector
1 year ago
13be14fe3f
[cdc-connector][base][tests] Introduce snapshot phase hooks for better snapshot test
1 year ago
6eb980a614
[hotfix][mysql][tests] Avoid canceling a finished job in tests of SpecificStartingOffsetITCase ( #2687 )
...
This closes #2687 .
1 year ago
cc9dbc6660
[mongodb][hotfix] Fix MongoDB's rewriteOutputBuffer to emit +I rather than +U ( #2760 ) ( #2760 )
1 year ago
a81d4a16b9
[cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. ( #2762 )
...
This closes #2762 .
1 year ago
8edf2a60f0
[cdc-pipeline-connector][values] Avoid using `default` as name of namespace and schema. ( #2774 )
1 year ago
57e4793b21
[cdc-base] Extract the common parts of each ScanFetchTask into AbstractScanFetcherTask.
...
This closes #2690 .
1 year ago
6e53733df2
[oceanbase] Use global timestamp to start log client ( #2565 )
...
* use global timestamp to start log client
* remove redundant fields
* update log info
1 year ago
4a5f2161aa
[cdc-pipeline-connector][values] Add event set MULTI_SPLITS_SINGLE_TABLE to mock source with multiple splits ( #2737 )
...
This closes #2737 .
1 year ago
70614be28c
[3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
...
This closes #2734 .
1 year ago
cad7424a1d
[3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
1 year ago
51554d8630
[hotfix][vitess] Fix invalidate package name
...
This closes #2735 .
1 year ago
b4deda97be
[cdc-pipeline-connector][values] Introduce `values` cdc pipeline connector
...
This closes #2675 .
1 year ago
4770f360d0
[sqlserver] Fix timestamp type parse of sqlserver ( #2708 )
1 year ago
18936a3676
[mongodb] Incremental snapshot source supports comma-separated format parameter for databases and collections ( #2724 )
1 year ago
a1b67a7585
[3.0][cdc-common] Add ConText for Source/Sink initialization. ( #2672 )
1 year ago
00f8156671
[build] Add header copyright check for java files
1 year ago
7192e5300b
[oracle] Skip unsupported nested table for legacy debezium source function
1 year ago
0565bf844c
[oracle] Skip unsupported nested table for Incremental OracleSource
1 year ago
995d524416
[3.0][common] Use built-in API annotations to make common module free with flink
...
This closes #2695 .
1 year ago
1e6b983bdd
[build] Bump Flink version to 1.18.0 ( #2463 )
...
This closes #2670 .
1 year ago
7da3eaef77
[postgres-cdc] Add tests for latest-offset startup strategy ( #2527 )
1 year ago
d01f0a9791
[pom] Use ${revision} replace specific version. ( #2623 )
1 year ago
82a652ca6f
[3.0][kickoff] Introduce basic modules for Flink CDC 3.0
...
This closes #2610
1 year ago
wenmo
Paul Lin
xleoken
Xin Gong
Kunni
Hang Ruan
e-mhui
Hongshun Wang
Yaroslav Tkachenko
Hongshun Wang
gongzhongqiang
yuxiqian
joyCurry30
lzshlzsh
He Wang
张田
Kunni
FocusComputing
Leonard Xu
Xianxun Ye
PengFei Li
Qingsheng Ren
Jiabao Sun
Dian Qi
Josh Mahonin
yunqingmo
Shawn Huang
zyi728
Maciej Bryński