Commit Graph

191 Commits (dd19da6ff1721942e6da37c4e3464e9c09be1455)

Author SHA1 Message Date
luoyuxia eeae69cdf9
[mysql] support all mysql data types (#438)
This closes #438
3 years ago
Leonard Xu 63a071cb82 [mysql] Support binlog split with huge snapshot split meta 3 years ago
Leonard Xu 2bc715e76a [mysql] Support tables with different split key 3 years ago
luoyuxia 5cf940d160
[mysql] Validate the regex pattern of database-name and table-name in table factory (#526) 3 years ago
mincwang 98ab02fed2
[docs] Use new MySqlSource API in documentation
Co-authored-by: Leonard Xu <xbjtdcq@gmail.com>
3 years ago
Leonard Xu 0f1b493d6a
[hotfix][mysql] Correct the usage of includeSchemaChanges parameter (#501) 3 years ago
Leonard Xu 5ce7adb619
[mysql] Expose MySqlSource API that supports incremental snapshot (#495)
Co-authored-by: Jark Wu <jark@apache.org>
3 years ago
Jark Wu 12f27acda2
[mysql] Support metadata columns for mysql-cdc connector (#496) 3 years ago
luoyuxia 64fc3e34b5
[common] Support to read tables of inconsistent schema (#494) 3 years ago
Leonard Xu 004d1768ac
[mysql] Support wide table with huge number of columns 3 years ago
Leonard Xu f0dede1934
[tests] Correct the argument order of assertArrayEqualInOrder/assertArrayEqualAnyOrder 3 years ago
Leonard Xu ee3d7f4584
[tests] Introduce assertEqualsInOrder/assertEqualsInAnyOrder methods for test
this closes #437
3 years ago
Leonard Xu d89373b294 [tests] Improve the unstable test testStartupFromLatestOffset in MySqlConnectorITCase 3 years ago
Leonard Xu a785efab5a [mysql] Fix the restoration failure when checkpoint happened in one MySQL transaction 3 years ago
Leonard Xu 22f2ac8a75 [mysql] Optimization that skips the binlog read if the low watermark equals high watermark 3 years ago
luoyuxia cfe3d3e400
[mysql] Add source metrics for MySqlParallelSource
Co-authored-by: Leonard Xu <xbjtdcq@gmail.com>
3 years ago
Leonard Xu d4d4eb32e2
[mysql] Use min/max of split key and row count to describe the distribution of table
this closes  #392
3 years ago
Leonard Xu 70a0e80d86
[mysql] Fix the ClassCastException in RecordUtils
this closes #330
3 years ago
mincwang 7f12859304
[common] Support json deserialization schema 3 years ago
Leonard Xu c52576f6d4 [mysql] Improve the tests code 3 years ago
Leonard Xu 15985d9269 [mysql] Fix the MySqlSnapshotSplitAssigner may assign duplicate splits 3 years ago
Leonard Xu a6eccf5789
[mysql] Add tests to check mysql-cdc works well under various timezones (#367)
Co-authored-by: Leonard Xu <xbjtdcq@gmailcom>
4 years ago
Leonard Xu a26096910e
[mysql] Optimize the checkpoint be optional under single parallelism (#365)
Co-authored-by: Leonard Xu <xbjtdcq@gmailcom>
4 years ago
taox e34ff84a43
[mysql] Correct the initial binlog offset for MySqlHybridSplitAssigner
Co-authored-by: Leonard Xu <xbjtdcq@gmailcom>
4 years ago
Jiabao Sun d76b17495e
[mysql] Add default value for 'database.history.instance.name' in MySqlParallelSource
This closes #326
4 years ago
ili zh 28faecdf04
[mysql] Improve the code format in SignalEventDispatcher 4 years ago
gongzhongqiang 08e9c756e1
[docs] Update the debezium document link to version 1.5 4 years ago
manmao 23bdc7a95b
[postgres] Fix Validator didn't implement Serializable
Co-authored-by: mao.man@rootcloud.com <mao.man@rootcloud.com>
4 years ago
Leonard Xu 9ee80c5239 [build] Bump version to 2.1-SNAPSHOT 4 years ago
Leonard Xu 7b5f6aab9c [mysql] Forbid using problematic startup options earliest-offset/specific-offset/timestamp 4 years ago
Shengkai 2f644007e2
[mysql] Validate MySQL server configuration (version, binlog_format, binlog_row_image) before start job
This closes #211
4 years ago
Leonard Xu 8162dc7ae3
[build] Rename group.id and packge name from 'com.alibaba.ververica' to 'com.ververica' (#287) 4 years ago
Leonard Xu 848a6bfefc
[debezium] Avoid uncompleted state of schema history when engine stop before checkpoint
* This close (#282)
4 years ago
Jark Wu 5180bc984f
[mysql] Refactor split assigner and simplify chunk splitting (#283) 4 years ago
Leonard Xu 2edf6d3778 [mysql] Enable single server id for MySqlParallelSource 4 years ago
Leonard Xu e894b0a69e
[mysql] Keep the global order of between snapshot records and binlog records 4 years ago
Leonard Xu 8638720d71 [mysql] Support latest-offset mode for MySqlParallelSource 4 years ago
Leonard Xu 75c01c530e [mysql] Simplify the options for Mysql Source 4 years ago
Jark Wu ff6b638262
[mysql] Improve split state and enumerator state structure 4 years ago
Jark Wu 7d369548b0
[mysql] Move MySqlSplitReader from 'split' to 'reader' package 4 years ago
Jark Wu a42f6ae61c
[mysql] Simplify EmbeddedFlinkDatabaseHistory to not record schema changes 4 years ago
Leonard Xu ae54f0b11a
[mysql] Throw underlying debezium reader exception 4 years ago
Leonard Xu 35c636224d
[mysql] Make every subtask used different database history 4 years ago
Leonard Xu a0e5e7a65b
[mysql] Optimize the data structure of split and splitState
This closes #268
4 years ago
Leonard Xu 5542233278 [mysql] Add JM/TM failover tests to check checkpoint works fine 4 years ago
Jark Wu c4a7d6ad56 [build] Bump version to 2.0-SNAPSHOT 4 years ago
Jark Wu ce2ee32521 [mysql][test] Rename "custom.sql" to "customer.sql" 4 years ago
Jark Wu e9e78f10d6 [mysql] Refactor class names to use "MySql" prefix instead of "MySQL" 4 years ago
Jark Wu 6827aad0a4 [mysql] Use LOG instead of LOGGER as Logger variable name 4 years ago
Jark Wu 6e5e14dc34 [mysql] Rename BinlogPosition to BinlogOffset 4 years ago
Leonard Xu 5786506394
[mysql] Support binlog reader merge optimization
This close #256
4 years ago
Leonard Xu 6231e990c0
[mysql] Support numeric type optimization for split boundary calculation
This close  (#248)
4 years ago
Leonard Xu c6ca6c1874 [mysql] Support new Source in SQL 4 years ago
Leonard Xu dd3826be11 [mysql] Use SchemaRecord for MySQL cdc source 4 years ago
Leonard Xu 1a81915cd1 [mysql] Support table with combined primary key 4 years ago
Leonard Xu 088dc5c049 [mysql] Add more tests for assigner and new Source 4 years ago
Leonard Xu a781a88817 [mysql] First draft for MySQL-CDC 2.0 4 years ago
Jiabao Sun ff9bf31b19
[common] Fix ClassCastException in type conversion
[common] Fix ClassCastException in type conversion, this close(#226)
4 years ago
Shengkai fcea3d14f7
[common] Fix schema history run out of memory
Fix schema history run out of memory, this close (#207)

Co-authored-by: Leonard Xu <xbjtdcq@gmail.com>
4 years ago
Leonard Xu 5195647576
[debezium] Bump debezium version to 1.5.2.Final (#205)
* [debezium] Upgrade the Debezium to 1.5.2.Final
Co-authored-by: Shengkai <1059623455@qq.com>
4 years ago
Leonard Xu f37f3998e2 [build] Bump version to 1.5-SNAPSHOT 4 years ago
Leonard Xu 1436410adf
[debezium] Bump Flink version to 1.13.0 (#176) 4 years ago
Jark Wu d6814f7a27
[build] Bump version to 1.4-SNAPSHOT 4 years ago
Jark Wu ea01014c64
[mysql] Add tests for monitoring empty MySQL table 4 years ago
Jark Wu 116cea690d
[mysql] Minor code improvement 4 years ago
Jark Wu 8925d943bf
[build] Format code with Spotless/google-java-format 4 years ago
Jark Wu c1c96a6ac7
[build] Bump version to 1.3-SNAPSHOT 4 years ago
Jark Wu 1ee4264e9f
[debezium] Remove logging security properties 4 years ago
Jark Wu 5264e80a7b
[mysql] Support read binlog from specific timestamp 4 years ago
Jark Wu 5fcfe8ca05
[mysql] Support "initial", "earliest-offset", "latest-offset" startup mode options 4 years ago
Jark Wu c47526737e
[mysql] Improve implementation of specifying binlog position and add tests 4 years ago
shizhengchao befb4adbd9
[mysql] Support read from specific MySQL Binlog position 4 years ago
jiabao.sun 2dfb59f3c7
[build] Bump up Flink version to 1.12.1 (#95) 4 years ago
Jark Wu 05a28a4428
[build] Set up CI with Azure Pipelines (#108)
This requires to upgrade testcontainers to 1.15.1 to fix can't pull images problem
4 years ago
陈小东 6c4f1f8528 [fix bug] offset state missing if no binlog fetched tile checkpoint after restore. 4 years ago
Zhenghua Gao e2d4ff735c
[ISSUE-69] DebeziumSourceFunction only stores monitored table DDLs which cause error in some situation
This closes #70
4 years ago
Jark Wu 0c8fcbbeca
[build] Bump version to 1.2-SNAPSHOT 4 years ago
Jark Wu 3dc2f0d5cf
[mysql][postgres] Use underscore for the database.server.name to avoid Kafka warning log 4 years ago
Jark Wu 5e9aa0b733
[mysql] Ignore exceptions when encountered unparseable DDL 4 years ago
Jark Wu f6b7b35a35
Support to parse MySQL TIMESTAMP type and support "server-time-zone" option
This fixes #17 and #16
5 years ago
Jark Wu bfa1fc3f91
[build] Bump project version to 1.1.0-SNAPSHOT 5 years ago
qinshiwei c89b46f14a
Support "debezium.*" option to pass through the Debezium's properties
This closes #15
5 years ago
Jark Wu 97b95eda77
Move test utils to a separate module 5 years ago
Jark Wu 5120a5b35b
[build] Update version to 1.0.0-SNAPSHOT 5 years ago
Jark Wu ad6eeb0fa6
[mysql][postgres] Rename MySqlBinlogSource and PostgreSqlChangelogSource to MySQLSource and PostgreSQLSource 5 years ago
Jark Wu 94e7703f96
[postgres] Add PostgreSQL CDC source connector 5 years ago
Jark Wu ab3a2ca1a6
[test] Move AssertUtils to flink-connector-debezium 5 years ago
Jark Wu ffae96de9d
Add README and LICENSE 5 years ago
Jark Wu 524af1c78d
[build] Fix shading problems 5 years ago
Jark Wu 6ed359c694
[build] Add flink-sql-connector-mysql-cdc to shade dependencies 5 years ago
Jark Wu a0b21b6092
[build] Rename modules to 'flink-connector-debezium' and 'flink-connector-mysql-cdc' 5 years ago