Kubernetes Clusters
Publish services from Kubernetes clusters in three clouds on one overlay, with conflux running without privileges in proxy mode inside a container.
- Needs
- A cluster in each cloud, with outbound internet access from its pods
- Uses
- conflux proxy in each cluster, conflux up on the machines that call in
Before and After VeilNet
Use this setup when machines need to reach services in clusters on several clouds, without VPN gateways between the clouds.
Today, linking clusters across clouds needs:
- A VPN gateway in every cloud.
- A tunnel to every other gateway.
- Route tables on both sides.
- Pod ranges that never overlap.
The tunnels and the TLS inside them use ECDHE and RSA, and a quantum computer can break both.
With VeilNet, each cluster runs a conflux container that publishes selected services on the overlay. Machines in any region reach those services directly, or through a relay when NAT blocks a direct connection. Pod ranges never appear on the overlay, so they can overlap.
Today, IPsec, TLS 1.2, ECDHE and RSA protect these links. A quantum computer can break all of them. With VeilNet, ML-DSA-87, ML-KEM-1024 and AES-256-GCM protect every link.
The drawing joins whole clusters. In this guide, one conflux container in each cluster publishes its services, and pods get no overlay addresses of their own.
Container Requirements
Conflux has no Helm chart or operator. You run it in your own container, one per cluster or one per service, in proxy mode. Peers reach only the ports you publish. The container needs:
- No extra privileges. In proxy mode, conflux needs no
CAP_NET_ADMIN, no TUN device and no host network. - Root inside the container. Conflux keeps its configuration and identity where only root can read them. Every command that changes state needs root.
- systemd as init. On Linux, conflux runs as a systemd service, so the container must run systemd as its init process.
- A volume per pod. Keep
/etc/confluxand/var/lib/confluxon a volume that outlives the pod.
A pod that starts with an empty volume enrols again, with a new identity and a new overlay IPv6 address. It keeps the IPv4 address you give it, so peers that use IPv4 see no change.
To call services on the overlay from a workload, run the workload where conflux up can give it an interface, such as a VM, a CI runner or a laptop.
Publishing the Services
Choose a private IPv4 address from one range for each cluster's conflux container. This guide uses
10.128.0.11in AWS,10.128.0.12in GCP and10.128.0.13in Azure. Give a different address to each container that must be reachable on its own.
On a sovereign network, the guardian commissions each container and machine and issues its manifest. Install it with conflux enrol --manifest FILE, then run each command without --taint, because the credential carries the taints. See Joining a sovereign network.
Verifying the Setup
On the machine from the last step, run sudo conflux peers. You should see the three cluster containers with DATA yes. Peers take 15 to 60 seconds to find each other:
$ sudo conflux peers ANCHOR OVERLAY IPV4 HARDWARE STATE CONN DATA REACH RTT hh7566tfjg fd80:c4b9:99ae:87b0:9701:37cf:fdd4:5fe7 10.128.0.11 ba:8b:ab:16:86:a2 connected out* yes nat-cone 71ms n6a6wfhym6 fd80:c4b9:99ae:7257:f858:d584:1ce2:4af2 10.128.0.12 8e:98:01:21:0c:77 connected out* yes nat-cone 148ms 4lajlj6hdu fd80:c4b9:99ae:c973:1bd4:6d07:c19:48a8 10.128.0.13 36:f3:ee:c5:80:dc connected out* yes unknown 212ms $ pg_isready -h 10.128.0.11 -p 5432 10.128.0.11:5432 - accepting connections
In a cluster's conflux container, conflux anchorctl proxy lists each published port with its live, opened and failed connections, and the last error from its backend. connection refused means that the service does not answer at the name in the spec.