For systems running on a single database that urgently need to securely and simply migrate data to a horizontally sharded database.
git clone --depth 1 https://github.com/apache/shardingsphere.git
cd shardingsphere
mvn clean install -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true -Djacoco.skip=true -DskipITs -DskipTests -Prelease
Release package：
Or you can get the installation package through the Download Page
Decompress the proxy release package and modify the configuration file conf/config-sharding.yaml. Please refer to proxy startup guide for details.
Modify the configuration file conf/server.yaml. Please refer to mode configuration for details.
Currently, mode must be Cluster, and the corresponding registry must be started in advance.
Configuration sample:
mode:
type: Cluster
repository:
type: ZooKeeper
props:
namespace: governance_ds
server-lists: localhost:2181
retryIntervalMilliseconds: 500
timeToLiveSeconds: 60
maxRetries: 3
operationTimeoutMilliseconds: 500
overwrite: false
Proxy has included JDBC driver of PostgreSQL.
If the backend is connected to the following databases, download the corresponding JDBC driver jar package and put it into the ${shardingsphere-proxy}/ext-lib directory.
| Database | JDBC Driver | Reference |
|---|---|---|
| MySQL | mysql-connector-java-5.1.47.jar | Connector/J Versions |
| openGauss | opengauss-jdbc-3.0.0.jar |
If you are migrating to a heterogeneous database, then you could use more types of database, e.g. Oracle. Introduce JDBC driver as above too.
sh bin/start.sh
logs/stdout.log. If you see the following statements:[INFO ] [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy start success
The startup will have been successful.
7.1. Query configuration.
SHOW MIGRATION PROCESS CONFIGURATION;
The default configuration is as follows.
+--------------------------------------------------------------+--------------------------------------+------------------------------------------------------+
| read | write | stream_channel |
+--------------------------------------------------------------+--------------------------------------+------------------------------------------------------+
| {"workerThread":40,"batchSize":1000,"shardingSize":10000000} | {"workerThread":40,"batchSize":1000} | {"type":"MEMORY","props":{"block-queue-size":10000}} |
+--------------------------------------------------------------+--------------------------------------+------------------------------------------------------+
7.2. New configuration (Optional).
A default value is available if there is no configuration.
A completely configured DistSQL is as follows.
CREATE MIGRATION PROCESS CONFIGURATION (
READ(
WORKER_THREAD=40,
BATCH_SIZE=1000,
SHARDING_SIZE=10000000,
RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='500')))
),
WRITE(
WORKER_THREAD=40,
BATCH_SIZE=1000,
RATE_LIMITER (TYPE(NAME='TPS',PROPERTIES('tps'='2000')))
),
STREAM_CHANNEL (TYPE(NAME='MEMORY',PROPERTIES('block-queue-size'='10000')))
);
Configuration item description:
CREATE MIGRATION PROCESS CONFIGURATION (
READ( -- Data reading configuration. If it is not configured, part of the parameters will take effect by default.
WORKER_THREAD=40, -- Obtain the thread pool size of all the data from the source side. If it is not configured, the default value is used.
BATCH_SIZE=1000, -- The maximum number of records returned by a query operation. If it is not configured, the default value is used.
SHARDING_SIZE=10000000, -- Sharding size of all the data. If it is not configured, the default value is used.
RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
TYPE( -- Algorithm type. Option: QPS
NAME='QPS',
PROPERTIES( -- Algorithm property
'qps'='500'
)))
),
WRITE( -- Data writing configuration. If it is not configured, part of the parameters will take effect by default.
WORKER_THREAD=40, -- The size of the thread pool on which data is written into the target side. If it is not configured, the default value is used.
BATCH_SIZE=1000, -- The maximum number of records for a batch write operation. If it is not configured, the default value is used.
RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
TYPE( -- Algorithm type. Option: TPS
NAME='TPS',
PROPERTIES( -- Algorithm property.
'tps'='2000'
)))
),
STREAM_CHANNEL ( -- Data channel. It connects producers and consumers, used for reading and writing procedures. If it is not configured, the MEMORY type is used by default.
TYPE( -- Algorithm type. Option: MEMORY
NAME='MEMORY',
PROPERTIES( -- Algorithm property
'block-queue-size'='10000' -- Property: blocking queue size.
)))
);
DistSQL sample: configure READ for traffic limit.
CREATE MIGRATION PROCESS CONFIGURATION (
READ(
RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='500')))
)
);
Configure data reading for traffic limit. Other configurations use default values.
7.3. Modify configuration.
ALTER MIGRATION PROCESS CONFIGURATION, and its internal structure is the same as that of CREATE MIGRATION PROCESS CONFIGURATION.
DistSQL sample: modify traffic limit parameter
ALTER MIGRATION PROCESS CONFIGURATION (
READ(
RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='1000')))
)
);
---
ALTER MIGRATION PROCESS CONFIGURATION (
READ(
RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='1000')))
), WRITE(
RATE_LIMITER (TYPE(NAME='TPS',PROPERTIES('tps'='1000')))
)
);
7.4. Clear configuration.
DistSQL sample: clear the configuration of READ and restore it to the default value.
DROP MIGRATION PROCESS CONFIGURATION '/READ';
DistSQL sample: clear the configuration of READ/RATE_LIMITER.
DROP MIGRATION PROCESS CONFIGURATION '/READ/RATE_LIMITER';