64f996b504
[FLINK-35891][pipeline-connector][paimon] Support dynamic bucket in Paimon sink
...
This closes #3499 .
7 months ago
3bb1f3ca45
[FLINK-35938][pipeline-connector][paimon] Use filterAndCommit API for retry Committable to avoid duplicate commit
...
This closes #3504 .
7 months ago
e0d6d1d1a8
[FLINK-35442][cdc-connect][kafka] add key.format and partition.strategy option to make sure the same record sending to the same partition. ( #3522 )
7 months ago
4bf5a395a5
[FLINK-34688][cdc-connector][mysql] Make scan newly table trigger condition strictly
...
This closes #3519 .
7 months ago
07446d1f9d
[FLINK-35715][common] Ignore the compact optimize for mysql timestamp type in BinaryRecordData ( #3511 )
7 months ago
09f36a4a05
[FLINK-35143][pipeline-connector][mysql] Expose newly added tables capture in mysql pipeline connector. ( #3411 )
...
Co-authored-by: Muhammet Orazov <916295+morazow@users.noreply.github.com>
Co-authored-by: north.lin <north.lin@yunlsp.com>
7 months ago
b937db25d5
[hotfix][cdc-connector][mongodb] Fix unstable test cases for snapshot back-filling ( #3506 )
7 months ago
9d6154f123
[FLINK-35791][kafka] Add database and table info of Canal / Debezium json format for Kafka sink ( #3461 )
7 months ago
21fade0e22
[FLINK-35432][pipeline-connector][mysql] Support catch modify event in mysql to send AlterColumnTypeEvent. ( #3352 )
...
Co-authored-by: haoke <haoke@bytedance.com>
7 months ago
5ed9e05bdf
[FLINK-35242][cdc-common][cdc-runtime] Support TRY_EVOLVE and LENIENT schema evolution behavior
...
This closes #3339 .
7 months ago
4561a8a32b
[FLINK-34638][cdc-common] Support column with default value
7 months ago
1778ef685a
Revert "[hotfix][starrocks] Fix StarRocks FE startup failure due to insufficient disk space available "
...
This reverts commit 3315be3c8f
7 months ago
7ca11359d7
[FLINK-35743][cdc-runtime] Correct the temporal function semantics
...
This closes #3449 .
Co-authored-by: wenmo <32723967+wenmo@users.noreply.github.com>
Co-authored-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>
7 months ago
3315be3c8f
[hotfix][starrocks] Fix StarRocks FE startup failure due to insufficient disk space available
...
This closes #3508 .
7 months ago
f6d1d4810a
[FLINK-35912][cdc-connector] SqlServer CDC doesn't chunk UUID-typed columns correctly ( #3497 )
...
* resolve conficts
* polish code to trigger ci
---------
Co-authored-by: Kael <kael@fts.dev>
Co-authored-by: gongzhongqiang <gongzhongqiang@gigacloudtech.com>
7 months ago
986f37b307
[FLINK-35674][cdc-connector][mysql]Fix blocking caused by searching for timestamp in binlog file ( #3432 )
7 months ago
0c49959f67
[FLINK-35524][cdc-base] Clear connections pools when reader exist. ( #3388 )
7 months ago
47f5660055
[FLINK-35344][cdc-base] Move same code from multiple subclasses to JdbcSourceChunkSplitter ( #3319 )
7 months ago
a39959f4b3
[FLINK-35072][doris] Support applying AlterColumnTypeEvent to Doris pipeline sink
...
This closes #3473
7 months ago
b15a226ccd
[FLINK-35391][cdc-connector][paimon] Bump dependency of Paimon Pipeline connector to 0.8.2
...
This closes #3335
Co-authored-by: wangjunbo <wangjunbo@qiyi.com>
7 months ago
241eb03a74
[FLINK-35868][cdc-connector][mongodb] Bump dependency version to support MongoDB 7.0
...
This closes #3489 .
7 months ago
91fc7bb506
[FLINK-35740][cdc-connector][mysql] Allow column as chunk key even it's not primary key
...
This closes #3448 .
7 months ago
ea71b2302d
[FLINK-35874][cdc-connector][mysql] Check pureBinlogPhaseTables set before call getBinlogPosition method
...
This closes #3488 .
7 months ago
313bace106
[minor][doc][cdc-connector][oracle] Update OracleSchema#getTableSchema doc description
...
This closes #3443 .
7 months ago
c5f391cf6a
[FLINK-35865][base] Support Byte and Short in ObjectUtils ( #3481 )
7 months ago
332e1f330d
[minor][cdc-connector][sqlserver] Fix some typos ( #3421 )
7 months ago
9a97182fbb
[minor][cdc-connector][mysql] Code optimization for constants
...
This closes #3385
7 months ago
2d50e43137
[minor][cdc-connector][postgres] PostgresDialect removes useless code
...
This closes #3416 .
7 months ago
94a0415475
[FLINK-34883] Fix postgres uuid column as PK ( #3282 )
...
* [FLINK-34883] Fix postgres uuid column as PK
* [FLINK-34883] Fix column comment.
7 months ago
a2539f5cfa
[hotfix][docs][postgres] Remove unsupported erroneous example code
...
This closes #3464
7 months ago
2d1eb0aff1
[FLINK-34990][cdc-connector][oracle] Oracle cdc support newly add table ( #3203 )
...
* [cdc-connector][oracle] Oracle cdc support newly add table
* [cdc-connector][oracle] Fix code style
* [cdc-connector][oracle] Address comment
7 months ago
302a691225
[FLINK-35354] Support host mapping in Flink tikv cdc ( #3336 )
...
* [FLINK-35337][cdc] Keep up with the latest version of tikv client
* [FLINK-35354]Support host mapping in Flink tikv cdc
* [FLINK-35354] Add doc for host mapping feature
* [FLINK-35354] fixed annotation import
8 months ago
80920aaa28
[FLINK-35297][mysql] Add validation for option connect.timeout because of HikariConfig limitation ( #3295 )
8 months ago
2bd2e4ce24
[FLINK-35121][common] Adds validation for pipeline definition options
9 months ago
c958daf8ab
[FLINK-35540][cdc-common] Fix table missed when database and table are with the same name
...
This closes #3396 .
9 months ago
17ff659ff7
[hotfix][build] Remove test-jar from parent pom and only add to necessary modules
...
This closes #3397 .
Co-authored-by: Qingsheng Ren <renqschn@gmail.com>
9 months ago
e2ccc836a0
[FLINK-34908][pipeline-connector][doris] Fix MySQL to doris pipeline will lose precision for timestamp type
...
This closes #3207 .
9 months ago
1a360635c7
[FLINK-35092][cdc][starrocks] Add starrocks integration test cases
9 months ago
8eb7e53965
[FLINK-35120][doris] Add Doris integration test cases
9 months ago
8a7fdd5ae3
[minor][cdc-connector][paimon] Remove duplicate interface implements
9 months ago
0214166049
[FLINK-35325][cdc-connector][paimon] Support specify column order when adding new columns to a table
...
This closes #3323 .
9 months ago
5b28d1a579
[FLINK-35129][postgres] Introduce scan.lsn-commit.checkpoints-num-delay option to control LSN offset commit lazily
...
This close #3349 .
9 months ago
0e8b2c7cc5
[FLINK-35430][cdc-connector][kafka] Pass the time zone infor to JsonSerializationSchema
...
This closes #3359 .
9 months ago
19c382c0f2
[FLINK-35409][cdc][mysql] Request more splits if all splits are filtered from addSplits method
...
This closes #3342 .
9 months ago
b2cb30f240
[FLINK-35408][mysql] Introduce 30 minutes tolerance when validate the time-zone setting
...
This closes #3341 .
9 months ago
6350eec66c
[FLINK-35301][cdc] Avoid deadlock when loading driver classes
...
This closes #3300 .
9 months ago
18b0627e34
[FLINK-35295][mysql] Improve jdbc connection pool initialization failure message
...
This closes #3293 .
9 months ago
bddcaae784
[FLINK-35294][mysql] Use source config to check if the filter should be applied in timestamp starting mode
...
This closes #3291 .
9 months ago
8e8fd304af
[FLINK-35298][cdc][metrics] Improve the fetch delay metrics
...
This closes #3298 .
9 months ago
fa6e7ea512
[FLINK-35245][cdc-connector][tidb] Add metrics for flink-connector-tidb-cdc
10 months ago
a7cb46f762
[FLINK-35274][cdc-connector][db2] Fix occasional failure issue with Flink CDC Db2 UT
10 months ago
002b16ed4e
[FLINK-35244][cdc-connector][tidb] Correct the package for flink-connector-tidb-cdc test
10 months ago
ec643c9dd7
[FLINK-35235][pipeline-connector][kafka] Fix missing dependencies in the uber jar of Kafka pipeline sink
10 months ago
6232d84052
[FLINK-35173][cdc][mysql] Debezium custom time serializer for MySQL connector ( #3240 )
10 months ago
05281e5d6f
[FLINK-34639][cdc][oceanbase] Support debezium deserializer in OceanBase source connector ( #3124 )
10 months ago
bdca0e328b
[FLINK-35128][cdc-connector][cdc-base] Re-calculate the starting changelog offset after the new table added ( #3230 )
10 months ago
ddf557c8e1
[tests][cdc-connector][db2] Improve the DB2 tests
...
This closes #2870
10 months ago
8deddab6eb
[minor][cdc-connector][db2] Fix typos and improve the code style
10 months ago
3385ae8fbc
[cdc-connector][db2] Db2 support incremental source
10 months ago
ef2eece256
[pipeline-connector][paimon] add paimon pipeline data sink connector. ( #2916 )
10 months ago
01ec7da989
[FLINK-35055][cdc] Exclude connector dependencies with questionable licenses from fat jar packages ( #3212 )
10 months ago
253ef9256b
[pipeline-connector][kafka] Add kafka pipeline data sink connector. ( #2938 )
10 months ago
5e52d8620c
[FLINK-35102][cdc][doris-sink] Correct Doris connector varchar type mapping
...
This closes #3224 .
10 months ago
0d797a6021
[FLINK-34903][cdc][mysql] Add mysql-pipeline-connector with tables.exclude option to exclude unnecessary tables ( #3186 )
10 months ago
d4ed7db8dc
[FLINK-35127][cdc][values] Remove HybridSource from ValuesSource to avoid CI failure. ( #3237 )
10 months ago
0562e35da7
[FLINK-35079][cdc-connector][mongo] Fallback to timestamp startup mode when resume token has expired
10 months ago
48ca8623bb
[FLINK-34634][cdc-base][mysql] Fix that Restarting the job will not read the changelog anymore if it stops before the synchronization of meta information is complete and some table is removed ( #3134 )
11 months ago
af7665d338
[FLINK-34689][MySQL][Feature] check binlog_row_value_options ( #3148 )
11 months ago
1b0b9f868d
[minor][cdc-connector][mongo] Fix warning during MongoDB tests ( #3210 )
11 months ago
025653d772
[hotfix] run mvn spotless to fix code style violations. ( #3219 )
11 months ago
18dd51fe22
[FLINK-34905][cdc-connector][mysql] Fix the default length of CHAR/BINARY data type of Add column DDL ( #3145 )
11 months ago
abb98ee257
[FLINK-34690] Cast decimal to VARCHAR as primary key in starrocks sink ( #3150 )
11 months ago
80d461b1b8
[MySQL] Fix for DBZ-7037 and #2217 to not clear tableMapEventByTableId for fake Rotates Event ( #3065 )
11 months ago
8c5437a1f1
[FLINK-34952][cdc-composer][sink] Flink CDC pipeline supports SinkFunction ( #3204 )
11 months ago
6510e670fa
[FLINK-34959][cdc] Update old flink-cdc-connectors artifactId ( #3200 )
11 months ago
a35b8dd44c
[build] Optimize pom to solve the CI error
11 months ago
bcad5d9d11
[cdc-runtime] Introduce TransformSchemaOperator and TransformDataOperator to support transformation
11 months ago
f7f248cda1
[build] fix inconsistent Kafka shading among cdc connectors ( #2988 )
11 months ago
5f4f86fb31
[hotfix] Change old com.ververica dependencies to org.apache.flink ( #3110 )
11 months ago
14b81dc6ec
[FLINK-34715][cdc][cdc-connector][mysql] Correctly close BinlogSplitReader in MySQL UTCase
...
This closes #3161 .
11 months ago
26eb6a99b9
[hotfix][cdc][build] Include cdc-common module when shaded for module flink-sql-connector-mysql-cdc
12 months ago
86272bf102
[FLINK-34183][cdc][license] Update NOTICE files ( #3107 )
12 months ago
e10c8691ad
[FLINK-34584][cdc][build] Rename package from com.ververica to org.apache.flink
...
This closes #3089 .
12 months ago
eeb4bab92d
[FLINK-34184][cdc][license] Update Copyright from Ververica to Apache
12 months ago
6da9684999
[cdc-connector][oracle] Use database name in upper case at the snapshot phase to fix fetching two different db name for the same table at the snapshot phase and incremental phase( #2088 )
1 year ago
ea7b0e8dbb
[cdc-base] Support to parse Oracle/SqlServer/PostgresSQL CDC schema change events ( #2315 )
1 year ago
a2ad627fed
[cdc-connector][mongodb] Expose capture newly added tables ability to Mongodb CDC source
1 year ago
03a287a9c2
[cdc-connector][postgres] Expose capture newly added tables ability to Postgres CDC source
1 year ago
1eeda75eaa
[cdc-connector][cdc-base] Support capture newly added tables with un-blocking stream read for CDC Framework（#3024）
1 year ago
1562bc29a3
[cdc-base] Support Scan Newly Added Tables feature
1 year ago
0ca718c31f
[cdc-connector][oracle] Expose SNAPSHOT_ONLY mode for Oracle CDC Source ( #3005 )
1 year ago
d058102c5c
[cdc-connector][oracle]Add oracle 21 version to OracleValidator ( #3054 )
1 year ago
82ccf00e6a
[hotfix][cdc-connector][mongodb] Fix scanFullChangelog requires all collections
1 year ago
7ea197242f
[debezium] Upgrade to Debezium 1.9.8.Final( #3033 ) ( #3034 )
1 year ago
10e542c3ff
[cdc-connector][sqlserver] sqlserver support use specific chunk column as a split key ( #2972 )
...
This closes #2971 .
1 year ago
3be3417de2
[cdc-connector][postgres] Fix potential PostgresSQL database connection leak ( #3013 )
...
This closes #2980 .
1 year ago
20e44e6966
[cdc-connector][oracle] Oracle supports user specific chunk column ( #2976 )
...
This closes #2973 .
1 year ago
c98f899af8
[cdc-common][cdc-pipelines][cdc-runtime] Use name instead of column in DropColumnEvent/AddColumnEvent. ( #2888 )
1 year ago
a2d36a892e
[cdc-connector][mongodb][hotfix] Stop executing log reading in MongoDBStreamFetchTask#close ( #3029 )
...
This closes #3029 .
1 year ago
9ce36569fb
[cdc-connector][postgres] Fix data lost problem when new lsn committed to slot between snapshotState and notifyCheckpointComplete ( #2539 )
...
This closes #2538 .
Co-authored-by: sammieliu <sammieliu@tencent.com>
1 year ago
e3d6c7e0aa
[cdc-connector][postgres] Apply DBZ-5398 postgres connector to handle functional unique index ( #2842 )
...
This closes #2710 .
1 year ago
9abf2cf900
[hotfix][cdc-connector][mysql] Skip SchemaChangeEvents that were not included in capturedTableFilter ( #2986 )
...
This closes #2981
1 year ago
e3f6829263
[cdc-connector][mongodb] Add SNAPSHOT_ONLY mode for Mongodb CDC Source
1 year ago
0e4f835f6d
[cdc-connector][Postgres] Add SNAPSHOT_ONLY mode for Postgres CDC Source
1 year ago
1cbdb4f8ae
[cdc-connector][cdc-base] Add SNAPSHOT_ONLY mode for Incremental CDC Source framework ( #2901 )
1 year ago
3bf886b94d
[cdc-connector][mysql] Add SNAPSHOT_ONLY mode for Mysql CDC Source ( #2901 )
1 year ago
35d3c0ff28
[cdc-base][hotfix] Fix SourceSplitSerializer to get totalFinishedSplitSize from recovery in 4th version
...
This closes #2992 .
1 year ago
a24c5d1b24
[cdc][e2e] Unify logic to clean docker image when connector e2e test finished ( #2978 )
1 year ago
f3762a1b30
[pipeline-connector][mysql] fix timestamp with timezone format ( #2952 )
...
* fix ts with tz parser
* test timestamp with default value
* fix related test
* use timestamp string in test cases
1 year ago
15a734efa0
[cdc-connector][oracledb] Fix oracle read exceed high_watermark in backfill phase
1 year ago
1b643026fe
[pipeline-connector][starrocks] Optimize char type mapping for starrocks pipeline connector ( #2895 )
1 year ago
812c33d27b
[pipeline-connector][doris][starrocks] Remove SimpleDateFormat for thread safe ( #2954 )
1 year ago
b42bc60ab9
[source-connector][mysql] speedup searching binlog by timestamp ( #2924 )
...
* [feature] speedup search mysql binlog when mysql-cdc-connector consume binlog by timestamp
* improve code
* improve code
* code improve
* test case add and code improve
* code improve
* code improve
* code improve, ut add and bugfix for purged log
---------
Co-authored-by: TJX2014 <tangjinxin@deepexi.com>
1 year ago
055b0af477
[cdc-common][pipeline-doris] Fix typo and wrong value for 'precision' of time type. ( #2886 )
1 year ago
a5fa2dc6c8
[hotfix][cdc-connector][mysql] Expose row_kind metadata column for mysql cdc source. ( #2927 )
...
We use 'row_kind' name instead of 'op' or 'op_type', the operation type is a concept of database and database only has 'update'
type and hasn't 'update_before' and 'update_after' type.
1 year ago
8eba910063
[cdc-connector][mysql] Support metadata 'operation' virtual column for MySql Connector. ( #2913 )
1 year ago
6b6d9fcb0b
[cdc-connector][mongodb] Avoid mongodb source to read data after high_watermark in backfill phase ( #2893 )
1 year ago
26dfeeb563
[cdc-connector][oceanbase] Fallback to current timestamp when query gts failure ( #2868 )
1 year ago
2c557c6a78
[cdc-connector][sqlserver][tests] Fix UT errors by correcting right output ( #2864 )
1 year ago
08beca52d0
[cdc] Add Deprecated annotation to source function which connector implemented incremental
1 year ago
568ed79005
[postgres] Add commits of PostgresConnection back to distinguish with debezium's origin class ( #2839 )
1 year ago
08d6b8178f
[cdc-connector][sqlserver] Fix SqlServerTableSource params order error ( #2847 )
1 year ago
f15faac298
[docs] Fix overview docs to remove prefix of pipeline options ( #2833 )
1 year ago
72bcb1df02
[pipeline-connector][starrocks] Fix char/varchar length inconsistency between cdc and starrocks ( #2829 )
1 year ago
c484bb447c
[build][hotfix] Remove duplicate dependency ( #2828 )
1 year ago
daceb42168
[build] Use ${project.version} instead of ${revision} for dependency version
...
This is documented in https://maven.apache.org/maven-ci-friendly.html#dependencies
1 year ago
8a08ded410
[cdc-pipeline-connector][mysql] Fix precision problem of BIT type conversion ( #2820 )
...
This closes #2820 .
1 year ago
f970444efa
[cdc-source-connector][mysql] fix deserialization issue on table metadata binlog event ( #2682 )
...
Co-authored-by: EchoLee5 <39044001+EchoLee5@users.noreply.github.com>
1 year ago
7d7c1af553
[pipeline-connector][mysql] Remove unnecessary serverTimeZone in DebeziumEventDeserializationSchema ( #2816 )
1 year ago
63b39ba22c
[cdc-source-connector][mysql] Fix attempted load of isSuspended in MySqlSplitSerializer::deserializeSplit when cdc version <= 2.2.0 ( #2550 )
...
Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
1 year ago
8c75db145d
[cdc-pipeline-connector][mysql] Ensure the inference of MEDIUMINT_UNSIGNED type matches INT type in table schema ( #2811 )
1 year ago
a10e07fd36
[cdc-connector][oceanbase] Add option for obcdc extra configs ( #2543 )
...
This closes #2543 .
1 year ago
0eae214183
[pipeline-connector][mysql] Enable send schema change by default ( #2815 )
...
This closes #2815 .
1 year ago
33b7c85093
[pipeline-connector][starrocks] Add starrocks pipeline connector ( #2765 )
...
This closes #2645 .
1 year ago
4abd86a500
[pipeline-connector][doris] add doris pipeline connector. ( #2810 )
...
Co-authored-by: wudi <676366545@qq.com>
1 year ago
8ca3091805
[cdc-pipeline-connector][mysql] Ensure the precision inference of DECIMAL type matches table schema
1 year ago
1766e82d7c
[3.0][cdc-common & runtime] Remove useless flink imports
1 year ago
08c5dfed83
[cdc-pipeline-connector][mysql] Add tests for mysql pipeline connector
...
Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
1 year ago
174282d3ef
[cdc-pipeline-connector][mysql] Support more mysql types
1 year ago
ec3d558485
[cdc-pipeleine-connecotr][mysql] Send CreateTableEvent in the mysql pipeline connector
1 year ago
10ccbde023
[cdc-pipeline-connector][mysql] Parse the alter statement to generate SchemaChangeEvent
1 year ago
d0cce26b76
[cdc-pipeline-connector][mysql] Introduce mysql cdc pipeline DataSource
1 year ago
95921d5620
[cdc-common] Introduce "pipeline.local-time-zone" config option which help handle time zone well ( #2797 )
...
This closes #2797 .
1 year ago
2d9dc018c5
[cdc-connector][sqlserver][tests] Fix UT errors by increasing the wait time after committing SQL ( #2799 )
...
This closes #2799 .
1 year ago
173deb5248
[cdc-composer] Add common and runtime JAR when submit ( #2795 )
...
This closes #2795 .
1 year ago
5f318346ff
[cdc-connector][debezium] Supports conversion from binary bytes to base64 encoded string ( #2435 )
...
This closes #2435 .
1 year ago
d1c3f8500b
[cdc-connector][mysql] MysqlTableSource return metadata in a fixed order ( #2578 )
1 year ago
72d33c8965
[cdc-connector][mysql] Fix snapshot fetch size conf does not take effect ( #2766 )
1 year ago
5c00f76d0f
[cdc-connector][oracle] Expose ability to skip backfill in snapshot period for Oracle CDC ( #2793 )
...
This closes #2793 .
1 year ago
ef1c095763
[cdc-connector][oracle] Fix wrong table path during backfill phase which leads to potential data loss ( #2707 )
...
This closes #2707 .
1 year ago
Kunni
Hongshun Wang
Hang Ruan
North Lin
yuxiqian
hk__lrzy
lvyanquan
gongzhongqiang
Wink
lipl
Thorne
Junbo wang
SeungMin
Zhongmin Qiao
ChengJie1053
gongzhongqiang
Xin Gong
Joao Boto
skylines
ouyangwulin
Shawn Huang
joyCurry30
Muhammet Orazov
Xie Yi
ConradJam
He Wang
Leonard Xu
Qishang Zhong
shikai93
xleoken
wenmo
Paul Lin
e-mhui
Hongshun Wang
Yaroslav Tkachenko
lzshlzsh
张田
FocusComputing
Leonard Xu
Xianxun Ye
PengFei Li
Qingsheng Ren
Jiabao Sun
Dian Qi
Josh Mahonin
yunqingmo