Commit Graph

249 Commits (2fd03e683e1e2f1a8c7790e6bb1a168bc1ab9266)

Author SHA1 Message Date
yuxiqian 39fd00ce49
[FLINK-36610] MySQL CDC supports parsing gh-ost / pt-osc generated schema changes (#3668)
Co-authored-by: MOBIN-F <18814118038@163.com>
1 month ago
Shawn Huang 085684b773
[FLINK-36811][mysql] MySQL cdc setIsProcessingBacklog in snapshot phase and exit when snapshot phase finished (#3793) 1 month ago
Kunni 7b01d9ca40
[hotfix][ci] Fix CI failure due to implicit conflicts
This closes #3846
2 months ago
Jzjsnow 630e0d70c2
[FLINK-36701][cdc-runtime] Obtain latest evolvedSchema when SinkDataWriterOperator handles FlushEvent from failover
This closes #3802

Co-authored-by: jzjsnow <snow.jiangzj@gmail.com>
2 months ago
Kunni 28985db062
[FLINK-36877][pipeline-connector/kafka] Correct canal-json output for delete record
This closes #3788
2 months ago
Kunni d3c049d8a7
[FLINK-36524][pipeline-connector][paimon] Bump Paimon version to 0.9.0
This closes #3644
2 months ago
Jason Zhang 92081dfe58
[FLINK-35167][pipeline-connector/maxcompute] Introduce MaxCompute pipeline DataSink
This closes  #3254.
2 months ago
moses 98f9f15132
[hotfix][docs] Add missed enumeration in MySQL Source
This closes #3832
2 months ago
Umesh Dangat baba9c6ae6
[FLINK-37017][cdc-common] Supports map and array types for binary record data
This closes #3434.
2 months ago
yuanoOo 6a345aaa80
[FLINK-34545][pipeline-connector/ob]Add OceanBase pipeline connector to Flink CDC
This closes #3360.
2 months ago
stayrascal dca94bdd71
[FLINK-36790][cdc-connector][paimon] Set waitCompaction to true in PaimonWriter to avoid CME problem
This closes  #3760

Co-authored-by: wuzhiping <wuzhiping.007@bytedance.com>
2 months ago
yuxiqian ddb5f00df5
[hotfix][tests] Fix unstable `OceanBaseMySQLModelITCase` (#3831)
Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>
2 months ago
yuxiqian 9f44c523e4
[hotfix][tests] Fix unstable `testInitialStartupModeWithOpTs` case (#3809)
Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>
2 months ago
yuxiqian 2e03f9ad49 [FLINK-36763][cdc-runtime] Introduce distributed schema evolution topology for sources with parallelized metadata
This closes #3801
2 months ago
Hang Ruan a0ae3320fa
[FLINK-36784][common] Support to add metadata columns for data in the meta fields of DataChangeEvent at transform (#3758)
Co-authored-by: Kunni <lvyanquan.lyq@alibaba-inc.com>
2 months ago
yuxiqian 6d21941661
[hotfix] Fix Java 11 target compatibility & add tests (#3633)
* [hotfix] Fix Java 11 target compatibility

Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>

# Conflicts:
#	.github/workflows/flink_cdc_java_8.yml
#	.github/workflows/flink_cdc_migration_test_base.yml
#	pom.xml

* fix: clarify GiHub workflow names

Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>

---------

Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>
2 months ago
yuxiqian dfbaf617e2
[FLINK-36558][source-connector/mysql] Fix column metadata parsing compatibility with MySQL 8.0.17&8.0.18
This closes  #3647.
2 months ago
ouyangwulin 0b0bb02023
[FLINK-36895][cdc-connector][base] The JdbcSourceChunkSplitter#queryMin method passed a parameter with tableName/coiumnName reversed. (#3797) 2 months ago
Hongshun Wang 12cf22f7b4
[FLINK-34688][cdc-connector][base] CDC framework split snapshot chunks asynchronously (#3510) 2 months ago
Sergei Morozov b50d1728e0
[FLINK-36891[[source-connector][mysql] Fix corrupted state in case of serialization failure in MySQL CDC Source
This closes #3794.
2 months ago
Xin Gong 0037c4379e
[FLINK-36771][cdc-connector][base&mysql] Fix UT trigger error: Invalid assigner status {} [NEWLY_ADDED_ASSIGNING_FINISHED] (#3755) 3 months ago
yuxiqian 2e54caf9dd
[hotfix][mysql] Remove deprecated MySQL incremental sources used in tests
This closes #3792
3 months ago
Hang Ruan fd82126c7d
[FLINK-36878][pipeline-connector][kafka] Shade org.apache.kafka with org.apache.flink.kafka.shaded.org.apache.kafka instead of cdc path
This closes #3790.
3 months ago
liuxiaodong ee9cd828c3
[FLINK-36315][cdc-connector][base&pg&mongodb]The flink-cdc-base module supports source metric statistics (#3619)
Co-authored-by: molin.lxd <molin.lxd@alibaba-inc.com>
Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
3 months ago
yuxiqian 26f5880fbf
[hotfix][tests][oceanbase] Fix oceanbase test failure, possibly caused by some interactions among cases (#3712) 3 months ago
Hang Ruan 23e8149e6a
[FLINK-36803][cdc-connector][base & mysql] Use the same format `tableId:chunkId` for splitId in SnapshotSplit (#3763) 3 months ago
Xin Gong 1d4e366969
[FLINK-36772][mysql][cdc-base] Fix error placeholder for errorMessageTemplate of Preconditions
This closes  #3754
3 months ago
Leonard Xu d0b2fe38a2
[hotfix][source-connector/mysql] Fix conflicts after flink version bumped to 1.19 (#3748) 3 months ago
Kunni 3a2d799fd8
[FLINK-36750][pipeline-connector/paimon] Restore SinkWriter from state to keep the information of previous SinkWrite when schema evolution happen
This closes #3744.
3 months ago
Leonard Xu 01a1e5fba5
[FLINK-36656][mysql] Fix type conversion failure for newly-added sharding table with mysql boolean type (#3683) 3 months ago
ConradJam dd79a02f0f
[FLINK-36586][build] Update flink version to 1.19 (#3660)
Co-authored-by: ConradJam <czy006@apache.com>
Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
3 months ago
yuxiqian 8e6c361f96
[FLINK-36285][doris] Fix unable to alter column type without default value specified
This closes #3691.
3 months ago
ConradJam 13248faa22
[FLINK-35592][cdc-connector][mysql] Fix MysqlDebeziumTimeConverter miss timezone when convert to timestamp
This closes #3332

Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
3 months ago
Kunni ae986ab3f6
[FLINK-36609][pipeline-connector][paimon] Add partition columns to primary keys
This closes  #3641.
4 months ago
Kunni f38341b02e
[FLINK-36541][pipeline-connector][paimon] Pass checkpointId to StoreSinkWrite#prepareCommit correctly
This closes #3652.
4 months ago
Junbo wang ff77a490bd
[FLINK-36517][pipeline-connector][paimon] Use filterAndCommit API to avoid committing the same datafile twice
This closes  #3639
4 months ago
North Lin 8647fe4986
[FLINK-36375][cdc-runtime] Fix missed default value in AddColumnEvent/RenameColunEvent
This closes  #3622.
4 months ago
yuxiqian 57a1a7df93
[FLINK-36408][pipeline-connector][mysql] Fix MySQL pipeline connector failed to parse FLOAT type with precision (#3625) 4 months ago
Kunni ecfee702dc
[FLINK-36560][pipeline-connector][paimon] Fix the issue that timestamp_ltz field is not correctly converted
This closes  #3648.
4 months ago
Runkang He be9b52750c
[FLINK-36572][pipeline-connector][starrocks] Fix the issue that the local time zone is wrongly set
This closes  #3655.
4 months ago
wudi 7f6d9115ae
[FLINK-36649][cdc-connector][oracle] Fix oracle connection close error (#3678) 4 months ago
Xin Gong d5f1b9ace8
[FLINK-36247][cdc-connector][mysql] Fix potential transaction leak during MySQL snapshot phase (#3602) 4 months ago
yuxiqian 84f97ed4c1
[FLINK-36474][route] Support merging timestamp columns when routing
This closes  #3636.
4 months ago
Junbo wang e692e7b4bc
[FLINK-36052][docs][cdc-pipeline-connector/es] Add flink cdc pipeline elasticsearch connector docs
This closes #3649.

Co-authored-by: wangjunbo <wangjunbo@qiyi.com>
4 months ago
Hongshun Wang 086bd9996a
[FLINK-36618][cdc-connector][postgres] Improve PostgresDialect.discoverDataCollections to reduce the bootstrap time
This closes #3672.
4 months ago
Kunni f24399cf5a
[FLINK-36211][pipeline-connector/kafka] shade org.apache.flink.streaming.connectors.kafka to avoid conflict with flink-connector-kafka jar. (#3595) 4 months ago
Shawn Huang ecce75949a
[FLINK-36463][cdc-connector][mysql] Forbid to override some debezium options (#3631) 4 months ago
yuxiqian d5f549f91f
[FLINK-36326][cdc-connector][mysql] Send BinlogSplitUpdateRequestEvent only once to fix auto scan newly-added table failure (#3613)
Signed-off-by: yuxiqian <34335406+yuxiqian@users.noreply.github.com>
Co-authored-by: Hang Ruan <ruanhang1993@hotmail.com>
4 months ago
yuxiqian daf27fa7a8 [hotfix][test] Reorganize test cases 4 months ago
gongzhongqiang a5b666a325
[FLINK-35503] Add support for running Oracle connector unit test on ARM architecture (#3600) 5 months ago