From 78b28df791e17b503bf9b1b5797a29f33ed9f754 Mon Sep 17 00:00:00 2001 From: Leonard Xu Date: Tue, 9 Nov 2021 10:32:11 +0800 Subject: [PATCH] [docs] Add checkpoint configuration limitation for Oracle and Postgres connector (#580) --- docs/content/connectors/oracle-cdc.md | 13 +++++++++++++ docs/content/connectors/postgres-cdc.md | 12 ++++++++++++ 2 files changed, 25 insertions(+) diff --git a/docs/content/connectors/oracle-cdc.md b/docs/content/connectors/oracle-cdc.md index 15252415b..4385d473c 100644 --- a/docs/content/connectors/oracle-cdc.md +++ b/docs/content/connectors/oracle-cdc.md @@ -231,6 +231,19 @@ Connector Options +Limitation +-------- + +### Can't perform checkpoint during scanning snapshot of tables +During scanning snapshot of database tables, since there is no recoverable position, we can't perform checkpoints. In order to not perform checkpoints, Oracle CDC source will keep the checkpoint waiting to timeout. The timeout checkpoint will be recognized as failed checkpoint, by default, this will trigger a failover for the Flink job. So if the database table is large, it is recommended to add following Flink configurations to avoid failover because of the timeout checkpoints: + +``` +execution.checkpointing.interval: 10min +execution.checkpointing.tolerable-failed-checkpoints: 100 +restart-strategy: fixed-delay +restart-strategy.fixed-delay.attempts: 2147483647 +``` + Features -------- diff --git a/docs/content/connectors/postgres-cdc.md b/docs/content/connectors/postgres-cdc.md index 1cb1088f9..a4c983060 100644 --- a/docs/content/connectors/postgres-cdc.md +++ b/docs/content/connectors/postgres-cdc.md @@ -189,6 +189,18 @@ The following format metadata can be exposed as read-only (VIRTUAL) columns in a +Limitation +-------- + +### Can't perform checkpoint during scanning snapshot of tables +During scanning snapshot of database tables, since there is no recoverable position, we can't perform checkpoints. In order to not perform checkpoints, Postgres CDC source will keep the checkpoint waiting to timeout. The timeout checkpoint will be recognized as failed checkpoint, by default, this will trigger a failover for the Flink job. So if the database table is large, it is recommended to add following Flink configurations to avoid failover because of the timeout checkpoints: + +``` +execution.checkpointing.interval: 10min +execution.checkpointing.tolerable-failed-checkpoints: 100 +restart-strategy: fixed-delay +restart-strategy.fixed-delay.attempts: 2147483647 +``` Features --------