Executing `start-cluster.sh` multiple times can start multiple `TaskManager`s.
多次执行 `start-cluster.sh` 可以拉起多个 TaskManager。
### Prepare docker compose
The following tutorial will prepare the required components using `docker-compose`.
### 准备 Docker 环境
接下来的教程将以 `docker-compose` 的方式准备所需要的组件。
1. Host Machine Configuration
Since `Doris` requires memory mapping support for operation, execute the following command on the host machine:
1. 宿主机配置
由于 Doris 的运行需要内存映射支持，需在宿主机执行如下命令:
```shell
sysctl -w vm.max_map_count=2000000
```
Due to the different ways of implementing containers internally on MacOS, it may not be possible to directly modify the value of max_map_count on the host during deployment. You need to create the following containers first:
docker run -it --privileged --pid=host --name=change_count debian nsenter -t 1 -m -u -n -i sh
```
The container was created successfully executing the following command:
容器创建成功执行以下命令：
```shell
sysctl -w vm.max_map_count=2000000
```
Then `exit` exits and creates the Doris Docker cluster.
然后 `exit` 退出，创建 Doris Docker 集群。
2. Start docker compose
Create a `docker-compose.yml` file using the content provided below:
2. docker 镜像启动
使用下面的内容创建一个 `docker-compose.yml` 文件：
```yaml
version: '2.1'
@ -104,76 +101,77 @@ Then `exit` exits and creates the Doris Docker cluster.
- MYSQL_PASSWORD=mysqlpw
```
The Docker Compose should include the following services (containers):
- MySQL: include a database named`app_db`
- Doris: to store tables from MySQL
该 Docker Compose 中包含的容器有：
- MySQL: 包含商品信息的数据库`app_db`
- Doris: 存储从 MySQL 中根据规则映射过来的结果表
To start all containers, run the following command in the directory that contains the `docker-compose.yml` file.
在 `docker-compose.yml` 所在目录下执行下面的命令来启动本教程需要的组件：
```shell
docker-compose up -d
```
This command automatically starts all the containers defined in the Docker Compose configuration in a detached mode. Run docker ps to check whether these containers are running properly. You can also visit [http://localhost:8030/](http://localhost:8030/) to check whether Doris is running.
Then, modify schema and record in MySQL, and the tables of Doris will change the same in real time：
1. insert one record in `orders` from MySQL:
接下来，修改 MySQL 数据库中表的数据，Doris 中显示的订单数据也将实时更新：
1. 在 MySQL 的 `orders` 表中插入一条数据
```sql
INSERT INTO app_db.orders (id, price) VALUES (3, 100.00);
```
2. add one column in `orders` from MySQL:
2. 在 MySQL 的 `orders` 表中增加一个字段
```sql
ALTER TABLE app_db.orders ADD amount varchar(100) NULL;
```
3. update one record in `orders` from MySQL:
3. 在 MySQL 的 `orders` 表中更新一条数据
```sql
UPDATE app_db.orders SET price=100.00, amount=100.00 WHERE id=1;
```
4. delete one record in `orders` from MySQL:
4. 在 MySQL 的 `orders` 表中删除一条数据
```sql
DELETE FROM app_db.orders WHERE id=2;
```
Refresh the Doris Web UI every time you execute a step, and you can see that the `orders` table displayed in Doris will be updated in real-time, like the following：
每执行一步就刷新一次 Doris Web UI，可以看到 Doris 中显示的 orders 数据将实时更新，如下所示：
@ -321,8 +318,8 @@ Here is an example file for using `route` feature:
parallelism: 2
```
Using the upper `route` configuration, we can synchronize the table schema and data of `app_db.orders` to `ods_db.ods_orders`, thus achieving the function of database migration.
Specifically, `source-table` support regular expression matching with multiple tables to synchronize sharding databases and tables. like the following：
@ -330,16 +327,14 @@ Specifically, `source-table` support regular expression matching with multiple t
sink-table: ods_db.ods_orders
```
In this way, we can synchronize sharding tables like `app_db.order01`、`app_db.order02`、`app_db.order03` into one ods_db.ods_orders tables.
Warning that there is currently no support for scenarios where the same primary key data exists in multiple tables, which will be supported in future versions.
## Clean up
After finishing the tutorial, run the following command to stop all containers in the directory of `docker-compose.yml`:
The Docker Compose should include the following services (containers):
- MySQL: include a database named`app_db`
- StarRocks: to store tables from MySQL
该 Docker Compose 中包含的容器有：
- MySQL: 包含商品信息的数据库`app_db`
- StarRocks: 存储从 MySQL 中根据规则映射过来的结果表
To start all containers, run the following command in the directory that contains the `docker-compose.yml` file.
在 `docker-compose.yml` 所在目录下执行下面的命令来启动本教程需要的组件：
```shell
docker-compose up -d
```
This command automatically starts all the containers defined in the Docker Compose configuration in a detached mode. Run docker ps to check whether these containers are running properly. You can also visit [http://localhost:8030/](http://localhost:8030/) to check whether StarRocks is running.
Connect to jdbc through database connection tools such as Dbeaver using `mysql://127.0.0.1:9030`. You can view the data written to three tables in StarRocks.
Then, modify schema and record in MySQL, and the tables of StarRocks will change the same in real time：
1. insert one record in `orders` from MySQL:
接下来，修改 MySQL 数据库中表的数据，StarRocks 中显示的订单数据也将实时更新：
1. 在 MySQL 的 `orders` 表中插入一条数据
```sql
INSERT INTO app_db.orders (id, price) VALUES (3, 100.00);
```
2. add one column in `orders` from MySQL:
2. 在 MySQL 的 `orders` 表中增加一个字段
```sql
ALTER TABLE app_db.orders ADD amount varchar(100) NULL;
```
3. update one record in `orders` from MySQL:
3. 在 MySQL 的 `orders` 表中更新一条数据
```sql
UPDATE app_db.orders SET price=100.00, amount=100.00 WHERE id=1;
```
4. delete one record in `orders` from MySQL:
4. 在 MySQL 的 `orders` 表中删除一条数据
```sql
DELETE FROM app_db.orders WHERE id=2;
```
Refresh the Dbeaver every time you execute a step, and you can see that the `orders` table displayed in StarRocks will be updated in real-time, like the following：
@ -276,38 +273,37 @@ Here is an example file for using `route` feature:
table.create.properties.replication_num: 1
route:
- source-table: app_db.orders
sink-table: ods_db.ods_orders
- source-table: app_db.shipments
sink-table: ods_db.ods_shipments
- source-table: app_db.products
sink-table: ods_db.ods_products
- source-table: app_db.orders
sink-table: ods_db.ods_orders
- source-table: app_db.shipments
sink-table: ods_db.ods_shipments
- source-table: app_db.products
sink-table: ods_db.ods_products
pipeline:
name: Sync MySQL Database to StarRocks
parallelism: 2
name: Sync MySQL Database to StarRocks
parallelism: 2
```
Using the upper `route` configuration, we can synchronize the table schema and data of `app_db.orders` to `ods_db.ods_orders`, thus achieving the function of database migration.
Specifically, `source-table` support regular expression matching with multiple tables to synchronize sharding databases and tables. like the following：
In this way, we can synchronize sharding tables like `app_db.order01`、`app_db.order02`、`app_db.order03` into one ods_db.ods_orders tables.
Warning that there is currently no support for scenarios where the same primary key data exists in multiple tables, which will be supported in future versions.
Hongshun Wang
1 year ago
committed by
GitHub