Load Balancing
Several anchors can share one private IPv4 address and act as one destination, with no DNS, load balancer or gateway in front.
Choosing a Holder
Each client's own anchor chooses a holder. No coordinator assigns clients, and the holders take no part in the choice.
The anchor chooses once per flow. A flow is all the traffic from one host to one address, whatever the ports or protocol. So all the connections from one client host reach the same holder.
- A flow stays with its holder for as long as it keeps sending, even when costs change or new holders join.
- A flow gets a new holder only after 15 minutes without traffic, or when its holder leaves.
- New flows go more often to the holders that are cheaper to reach. The cost is the round-trip time to the holder plus the time the holder takes to answer, measured from the client's own traffic. Every holder gets some flows.
- Taints decide which holders are in a client's group. A client never uses a holder it may not exchange data with.
A holder that becomes slower keeps the flows it already has, because a change in cost affects only new flows.
Failover
A holder leaves a client's group in one of two ways:
| How the holder stops | When clients drop it |
|---|---|
Stopped with sudo conflux down | At once. The anchor announces that it is leaving as it stops. |
| Killed, powered off or cut off | When the client's traffic to it has failed for two seconds. Each client decides this for itself, without waiting for the rest of the realm. |
Only the flows of the holder that left move. Each one moves to one of the remaining holders. Flows on the other holders stay where they are.
Open connections to the holder that left do not survive, because the next holder is a different host. The application must reconnect, and then reaches one of the remaining holders.
A holder that comes back does not take its old flows back. It rejoins the group and gets new flows only.
Replacing a holder
To replace a holder, start the new one at the shared address, then stop the old one with sudo conflux down. The old holder's clients move to the remaining holders, including the new one.
Before you take a service down, stop its anchor with sudo conflux down, so that its clients move to the other holders.
DNS and Load Balancers
A shared address replaces three kinds of infrastructure:
- Unlike DNS round-robin, there are no records to publish and no caches to wait for. Clients drop a holder at once, or after two seconds of failed traffic.
- Unlike a load balancer, nothing sits between the clients and the holders. Each client's own anchor chooses, and traffic stays encrypted from the client to the holder.
- Unlike an API gateway, a shared address needs no public address. Clients reach it across the overlay, and only if they may exchange data with its holders.
Planning a Shared Address
Consider these points when you plan a shared address:
- Use a private address. An anchor advertises only a private IPv4 address, so clients cannot reach a group at a public one.
- Use an address outside
198.18.0.0/15. Anchors use that range for the clients they translate, and conflux refuses an address inside it. - Load is balanced per client host, not per connection. One busy host reaches one holder however many connections it opens. The load evens out across many clients.
- Each host behind a subnet router counts as a separate client host.
- An anchor has one IPv4 address, so a holder has no other IPv4 address. To reach one particular holder, use its overlay IPv6 address.
- A holder that connects to the shared address reaches itself.
- Taints decide each client's group, so clients in different compartments can reach different holders at one address.
- The service on a holder never sees a client's own IPv4 address. With a host interface, each client arrives from an address in
198.18.0.0/15by default.
Setting Up a Shared Address
To create a shared address, give several machines the same --ipv4 value. Nothing else is needed. On the instance VeilNet hosts, every machine joins with the taint that the first machine created in Quick Start.
Run the same service on every holder, listening on all addresses or on the shared address. In this example, the service is a web server on port 80.
Each holder is a separate host, so a session stored on one holder is unknown to the others.
To publish chosen ports instead of giving each holder an interface, run conflux proxy with the same --ipv4. Internal APIs shows this setup with an HTTP API.
Checking the Group
On a client, sudo conflux routes lists the shared address once, with every holder in the VIA column:
$ sudo conflux routes NETWORK VIA STATE WHY 10.128.0.9/32 (64:ff9b:1::a80:9/128) dfyspsc5lk,qh5av2ricp,z7kok3zv2m accepted - 10.128.0.101/32 (64:ff9b:1::a80:65/128) - served -
This list shows every holder that advertises the address, not which holders are answering. A holder that failed without warning stays listed for a while after traffic has moved away from it.
sudo conflux peers shows each holder as a separate peer, with the shared address in the IPV4 column:
$ sudo conflux peers ANCHOR OVERLAY IPV4 HARDWARE STATE CONN DATA REACH RTT dfyspsc5lk fd80:c4b9:99ae:3e07:9c2a:41d6:b0f3:8e15 10.128.0.9 6a:1f:c3:08:d2:94 connected out*,in yes public 1.4ms qh5av2ricp fd80:c4b9:99ae:7b52:e0c9:16a4:5d38:c2a1 10.128.0.9 be:43:9e:71:05:ca connected out*,in yes nat-cone 2.1ms z7kok3zv2m fd80:c4b9:99ae:d4ad:1a8a:1eca:5e42:c961 10.128.0.9 f2:8d:26:b4:7e:13 connected out yes public 990µs
On a holder, the shared address shows as served, because the holder answers at it itself:
$ sudo conflux routes NETWORK VIA STATE WHY 10.128.0.9/32 (64:ff9b:1::a80:9/128) - served - 10.128.0.101/32 (64:ff9b:1::a80:65/128) k3vq7xmb2a accepted -
anchor_route_groups counts the networks that more than one answering anchor offers to this anchor. On a client with one shared address and no other route group, it reads 1:
$ sudo conflux metrics | grep anchor_route_groups anchor_route_groups 1
Which holder answers
To see which holder a client reaches, make the service on each holder reply with its own host name. Every request from one client then shows the same name, while other clients may see other names.
When flow records are exported, each record names the holder in anchor.peer and the shared address in anchor.flow.destination.ipv4. See Observability.
Failing over by hand
To test failover, stop the holder that answered:
$ sudo conflux down Stopped. The boot service is still registered and the configuration is intact, so the next reboot brings this machine back exactly as it was. conflux start start it again now conflux uninstall remove it permanently
The client's next request shows another holder's name. To bring the stopped holder back, run sudo conflux start. It then gets new flows only.