This section mainly introduces how to build the GraalVM Native Image
of ShardingSphere Proxy through the native-image
command line tool of GraalVM CE
,
and the Docker Image
containing this GraalVM Native Image
. The GraalVM Native Image
of ShardingSphere Proxy refers to ShardingSphere Proxy Native in this article.
All Docker Images involved in this section are not distributed through ASF official channels such as https://downloads.apache.org and https://repository.apache.org.
Docker Images are only provided in downstream channels such as GitHub Packages
and Docker Hub
for easy use.
ShardingSphere Proxy Native can execute DistSQL, which means that no YAML file defining the logical database is actually required. By default, ShardingSphere Proxy Native only contains,
This section assumes one of the following system environments,
This section is still limited by the recorded content of GraalVM Native Image on the ShardingSphere JDBC side.
If users need to use third-party JAR in ShardingSphere Proxy Native, or use UPX to compress and compile GraalVM Native Image,
then they need to modify the source code of the Maven module org.apache.shardingsphere:shardingsphere-proxy-native-distribution
.
Refer to the Build from source code
section below.
If you do not need to modify the default configuration of ShardingSphere Proxy Native, developers can start from the Use through the nightly built Docker Image
section.
The Docker Image containing ShardingSphere Proxy Native is built nightly at https://github.com/apache/shardingsphere/pkgs/container/shardingsphere-proxy-native .
The default port of ShardingSphere Proxy Native is 3307
, and the configuration file is loaded from /opt/shardingsphere-proxy/conf
.
The nightly built Docker Image has multiple variant Docker Image Tags of GraalVM Native Image.
Assuming that there is a conf
folder containing global.yaml
as ./custom/conf
,
developers can test ShardingSphere Proxy Native in the form of dynamically linked GraalVM Native Image
through the following Docker Compose file.
services:
apache-shardingsphere-proxy-native:
image: ghcr.io/apache/shardingsphere-proxy-native:da826af47804dae79b1ba5717af86792726745fd
volumes:
- ./custom/conf:/opt/shardingsphere-proxy/conf
ports:
- "3307:3307"
As a supplement, the Docker Image Tag of ghcr.io/apache/shardingsphere-proxy-native:latest
will point to the dynamically linked GraalVM Native Image
.
This section is limited to the Container Runtime that supports running linux/amd64
OS/Arch Containers.
Assuming that there is a conf
folder containing global.yaml
as ./custom/conf
,
developers can test ShardingSphere Proxy Native in the form of mostly statically linked GraalVM Native Image
through the following Docker Compose file.
Just add the -mostly
suffix to the Docker Image Tag corresponding to the specific, dynamically linked GraalVM Native Image
.
services:
apache-shardingsphere-proxy-native:
image: ghcr.io/apache/shardingsphere-proxy-native:da826af47804dae79b1ba5717af86792726745fd-mostly
volumes:
- ./custom/conf:/opt/shardingsphere-proxy/conf
ports:
- "3307:3307"
This section is limited to Container Runtime that supports running linux/amd64
OS/Arch Containers.
Assuming that there is a conf
folder containing global.yaml
as ./custom/conf
,
Developers can test ShardingSphere Proxy Native in the form of fully statically linked GraalVM Native Image
through the following Docker Compose file.
Just add the -static
suffix to the Docker Image Tag corresponding to the specific, dynamically linked GraalVM Native Image
.
services:
apache-shardingsphere-proxy-native:
image: ghcr.io/apache/shardingsphere-proxy-native:da826af47804dae79b1ba5717af86792726745fd-static
volumes:
- ./custom/conf:/opt/shardingsphere-proxy/conf
ports:
- "3307:3307"
If you build from source code, developers have two options,
Build a Linux Docker Image
containing ShardingSphere Proxy Native products without installing a local toolchain
Build a ShardingSphere Proxy Native product with a local toolchain installed. For Windows, you can create a GraalVM Native Image in the form of .exe
in this way
Developers may need to use JARs with custom SPI implementations or third-party dependent JARs. Before building from source code, modify the dependencies
section of the distribution/proxy-native/pom.xml
file.
An example of adding a MySQL JDBC Driver dependency is as follows. The relevant JAR should be pre-placed in the local Maven repository or a remote Maven repository such as Maven Central.
<dependencies>
<dependency>
<groupId>com.mysql</groupId>
<artifactId>mysql-connector-j</artifactId>
<version>9.3.0</version>
</dependency>
</dependencies>
Contributors must have installed on their devices,
OpenJDK 11 or higher
Docker Engine that can run Linux Containers
The following sections discuss possible required operations under Ubuntu and Windows respectively.
It is assumed that the contributor is on a fresh Ubuntu 22.04.5 LTS instance with git configured.
OpenJDK 21 can be installed using SDKMAN!
in bash using the following command.
sudo apt install unzip zip -y
curl -s "https://get.sdkman.io" | bash
source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install java 21.0.7-ms
sdk use java 21.0.7-ms
You can install Docker Engine in rootful mode by running the following command in bash. This article does not discuss changing the default logging driver in /etc/docker/daemon.json
.
sudo apt update && sudo apt upgrade -y
sudo apt-get remove docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc
cd /tmp/
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
Assuming the contributor is on a fresh Windows 11 Home 24H2 instance with git-for-windows/git
and PowerShell/PowerShell
installed and configured.
OpenJDK 21 can be installed using version-fox/vfox
in Powershell 7 using the following command.
winget install version-fox.vfox
if (-not (Test-Path -Path $PROFILE)) { New-Item -Type File -Path $PROFILE -Force }; Add-Content -Path $PROFILE -Value 'Invoke-Expression "$(vfox activate pwsh)"'
# At this time, you need to open a new Powershell 7 terminal
vfox add java
vfox install java@21.0.7-ms
vfox use --global java@21.0.7-ms
When Windows pops up a window asking you to allow an application with a path like C:\users\lingh\.version-fox\cache\java\v-21.0.7-ms\java-21.0.7-ms\bin\java.exe
to pass through Windows Firewall,
you should approve it.
Background reference https://support.microsoft.com/en-us/windows/risks-of-allowing-apps-through-windows-firewall-654559af-3f54-3dcf-349f-71ccd90bcc5c .
You can enable WSL2 and set Ubuntu WSL
as the default Linux distribution in Powershell 7 with the following command.
wsl --install
After enabling WSL2, download and install rancher-sandbox/rancher-desktop
from https://rancherdesktop.io/ and set up Container Engine
using dockerd(moby)
.
This article does not discuss changing the default logging driver in /etc/docker/daemon.json
of the Linux distribution rancher-desktop
.
You can execute the following command to build.
git clone git@github.com:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -am -pl distribution/proxy-native -T1C "-Pdocker.build.native.linux" clean validate
A possible Docker Compose example is,
services:
apache-shardingsphere-proxy-native:
image: apache/shardingsphere-proxy-native:5.5.3-SNAPSHOT
volumes:
- ./custom/conf:/opt/shardingsphere-proxy/conf
ports:
- "3307:3307"
You can execute the following command to build.
git clone git@github.com:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -am -pl distribution/proxy-native -T1C "-Pdocker.build.native.linux" "-Dproxy.native.dockerfile=Dockerfile-linux-mostly" "-Dproxy.native.image.tag=5.5.3-SNAPSHOT-mostly" clean validate
A possible Docker Compose example is,
services:
apache-shardingsphere-proxy-native:
image: apache/shardingsphere-proxy-native:5.5.3-SNAPSHOT-mostly
volumes:
- ./custom/conf:/opt/shardingsphere-proxy/conf
ports:
- "3307:3307"
You can execute the following command to build.
git clone git@github.com:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -am -pl distribution/proxy-native -T1C "-Pdocker.build.native.linux" "-Dproxy.native.dockerfile=Dockerfile-linux-static" "-Dproxy.native.image.tag=5.5.3-SNAPSHOT-static" clean validate
A possible Docker Compose example is,
services:
apache-shardingsphere-proxy-native:
image: apache/shardingsphere-proxy-native:5.5.3-SNAPSHOT-static
volumes:
- ./custom/conf:/opt/shardingsphere-proxy/conf
ports:
- "3307:3307"
Contributors must have installed on their devices,
The possible required operations under Ubuntu and Windows are consistent with Development and test.
Developers who want to build a mostly statically linked GraalVM Native Image
or a fully statically linked GraalVM Native Image
,
will need to build musl from source as described in https://www.graalvm.org/latest/reference-manual/native-image/guides/build-static-executables/ .
You can execute the following command to build it.
git clone git@github.com:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -am -pl distribution/proxy-native -T1C -DskipTests "-Prelease.native" clean package
You can execute the following command to build.
git clone git@github.com:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -am -pl distribution/proxy-native -T1C -DskipTests "-Prelease.native" "-DbuildArgs=-H:+AddAllCharsets,-H:+StaticExecutableWithDynamicLibC" clean package
You can execute the following command to build.
git clone git@github.com:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -am -pl distribution/proxy-native -T1C -DskipTests "-Prelease.native" "-DbuildArgs=-H:+AddAllCharsets,--static,--libc=musl" clean package
No matter what variant the GraalVM Native Image is, you need to bring 3 parameters to start the Native Image through the command line.
global.yaml
configuration file written by the user,0.0.0.0
, any database client can access ShardingSphere Proxy Native.The binary file of the built GraalVM Native Image can only set command line parameters. This means that,
On Ubuntu, assuming that the conf
folder containing global.yaml
is /tmp/conf
, possible example is,
cd ./shardingsphere/
cd ./distribution/proxy-native/target/apache-shardingsphere-5.5.3-SNAPSHOT-shardingsphere-proxy-bin/bin
./proxy-native "3307" "/tmp/conf" "0.0.0.0"
On Windows, assuming that a conf
folder containing global.yaml
already exists at C:\Users\shard\Downloads\conf
, a possible example is,
cd ./shardingsphere/
cd ./distribution/proxy-native/target/apache-shardingsphere-5.5.3-SNAPSHOT-shardingsphere-proxy-bin/bin
./proxy-native.exe "3307" "C:\Users\shard\Downloads\conf" "0.0.0.0"
In general, developers only need to use dynamically linked GraalVM Native Image
.
When developers only use the Container Runtime that can run linux/amd64
OS/Arch Containers and want to get a smaller Docker Image,
consider using mostly statically linked GraalVM Native Image
or fully statically linked GraalVM Native Image
.
For background, see https://www.graalvm.org/latest/reference-manual/native-image/guides/build-static-executables/ and https://github.com/oracle/graal/issues/2589.
Mostly statically linked executables are an alternative to statically linked musl libc implementations promoted by golang/go
.
For ShardingSphere Proxy Native, the observability capabilities it provides are not consistent with Observability.
Users can use a series of command-line tools or visualization tools provided by https://www.graalvm.org/latest/reference-manual/tools/ to observe the internal behavior of GraalVM Native Image, and use VSCode under Linux to complete the debugging work according to their requirements. If the user is using IntelliJ IDEA and wants to debug the generated GraalVM Native Image, the user can follow https://blog.jetbrains.com/idea/2022/06/intellij-idea-2022-2-eap-5/#Experimental_GraalVM_Native_Debugger_for_Java and its successors. If the user is not using Linux, the GraalVM Native Image cannot be debugged. Please pay attention to https://github.com/oracle/graal/issues/5648 which has not been closed.
For the use of Java Agents such as ShardingSphere Agent
, the native-image
component of GraalVM does not fully support the use of javaagent when building Native Image.
Users need to pay attention to https://github.com/oracle/graal/issues/8177 which has not been closed.
If users expect to use such Java Agents under ShardingSphere Proxy Native, they need to pay attention to the changes involved in https://github.com/oracle/graal/pull/8077.
linux/riscv64
OS/Arch limitationCurrently, ShardingSphere Proxy Native does not provide availability on linux/riscv64
OS/Arch. If developers use the linux/riscv64
device,
they should refer to https://medium.com/graalvm/graalvm-native-image-meets-risc-v-899be38eddd9 to modify the build configuration of Proxy Native.
Since https://github.com/oracle/graal/issues/6855, LLVM backend
needs to be built from the source code of GraalVM to be used.
ShardingSphere Proxy Native can build GraalVM Native Image on Windows out of the box with a local toolchain containing Microsoft.VisualStudio.2022.Community
.
Currently affected by https://github.com/graalvm/container/issues/106,
ShardingSphere does not provide the build configuration required to build Docker Image for Dynamically Linked GraalVM Native Image
compiled through Windows.
Although Oracle GraalVM Early Access Builds For JDK 25 EA 24
already supports building GraalVM Native Image in the form of Wasm Module
,
ShardingSphere is not yet ready to test CI under OpenJDK 25.
Currently, ShardingSphere Proxy Native does not provide the build configuration required to compile to Wasm Module
.