数据集成 - Sharding

工具

• Flink CDC
• SeaTunnel - MySQL-CDC
• dts

文档

• 基于 Flink CDC 同步 MySQL 分库分表构建实时数据湖
• 多库多表场景下使用Amazon EMR CDC实时入湖最佳实践

Doris

flink-doris-connector

• Flink Doris Connector
• 使用FlinkCDC接入多表或整库示例
• CDCSchemaChangeExample
• DDL 毫秒级同步，Light Schema Change 的设计与实现｜新版本揭秘

flink

• File Systems：Hadoop/Presto S3 文件系统插件 https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/cli/ https://nightlies.apache.org/flink/flink-docs-master/zh/docs/dev/datastream/application_parameters/

flink-kubernetes-operator

• flink-kubernetes-operator
• CRD Reference
• Job Lifecycle Management
• Kubernetes HA

flink-cdc

MySQL CDC Connector

SeaTunnel

https://doris.apache.org/zh-CN/docs/dev/ecosystem/seatunnel https://seatunnel.apache.org/docs/2.3.1/start-v2/kubernetes/

xx

• Flink kubernetes deployment - how to provide S3 credentials from Hashicorp Vault?