Subnet Routers and Exits

Subnet Routers and Exit Nodes

A subnet router offers its realm the private networks it is attached to. Devices that cannot run an anchor, such as printers, cameras and controllers, then become reachable across the overlay. An exit node offers the internet, so clients can send their internet traffic through it.

The destination address decides the role. Private networks go through subnet routers, and public addresses go through exit nodes. An exit node never carries traffic to a private network, even one it is attached to.

• Name each network by interface or by prefix. An interface follows whatever addresses DHCP assigns, while a prefix must match exactly.
• If an entry matches nothing on the host, the anchor stops. This catches a typo or an unplugged cable.
• You can also serve every private network. The router then offers all of them on every interface, including container bridges and pod networks, and follows them as they come and go.
• To offer a network to one compartment only, add a taint to the entry. For example, eth1@lab offers eth1 only to peers that are comparable with lab and with the router's own taints. Two anchors are comparable when one carries every taint the other does.
• An exit node blocks traffic to private addresses, except to the networks that the same anchor offers as a subnet router.
• An anchor can serve an exit or use one, but not both.

Choosing a Router

Each router advertises its networks to the realm. A client accepts a route only from an anchor it may exchange data with, so taints decide which routers a client can use.

A client accepts subnet routes automatically, because they only add more specific networks. A client uses an exit node only when you tell it to, because that replaces the machine's default route.

Several routers for one network

You can run several routers for one network, or several exit nodes. They give you redundancy without conflict, and share the load flow by flow.

A flow is all the traffic between two hosts, whatever the ports or protocol. A flow keeps the router it started on, so a connection never changes router midway.

• A flow moves to another router only after 15 minutes without traffic, or when its router leaves. A router leaves when the realm drops it, for example when it stops, or when traffic to it has failed for two seconds.
• A router that comes back does not take its old flows back.
• New flows go more often to the routers that are cheaper to reach. The cost is the round-trip time through the router to the destination. Every router gets some flows.

The Egress Filter

An anchor forwards traffic only to the networks it advertises. Traffic to any other destination has no route and is answered as unreachable. You do not keep a separate filter list for this.

An anchor also never forwards traffic from one peer to another. Two anchors that may not exchange data therefore cannot do so through a third.

IPv4 Translation

The overlay carries IPv6 only, so a subnet router offers an IPv4 network as the IPv6 prefix that stands for it. For example, 192.168.5.0/24 is advertised as 64:ff9b:1::c0a8:500/120. IPv6 networks are carried as they are.

• Between two LANs, through two subnet routers, each host sees the other's real IPv4 address. Neither host runs an anchor.
• An anchor that reaches into a LAN arrives from a pool address, which the router's host translates onto the LAN.
• An exit node translates traffic toward the internet only. It translates traffic from anchors, and from hosts behind other subnet routers that it forwards for.
• An anchor drops translated IPv4 traffic from a peer unless its source is in a network that the peer advertises and this anchor accepted, or it comes from an exit node this anchor uses.

Host Setup

On Linux with a host interface, a router sets up its host to forward traffic when it starts, and restores the host when it stops. You cannot turn this off. Without it, a router could advertise a route that its host does not forward, and every client's traffic on that route would be lost.

• The anchor keeps its rules in one nftables table, inet anchor_router, or in its own iptables chains and ipsets when nft is not available. It never edits your own rules.
• Nothing is saved. After a reboot, the host follows its own configuration until the anchor starts again.
• The setup needs CAP_NET_ADMIN, both for the anchor and for the commands it runs. Root has it, and so does a systemd service with AmbientCapabilities=. A binary given the capability with setcap does not pass it on to those commands.
• The setup also needs a writable /proc/sys, and either nft on kernel 5.2 or later, or iptables with ipset and ip6tables.
• If the setup fails, the anchor undoes its changes and stops with REASON_HOST_REFUSED. It offers nothing until the setup succeeds.
• On macOS, the BSDs and Windows, the anchor changes nothing on the host. It prints the commands to run instead when it starts.

Userspace Routers

Without a host interface, a router forwards traffic from its own process. It needs no privileges and changes nothing on the host. The realm sees no difference.

The router ends each connection it receives and opens a new one to the destination from an ordinary socket.

• It carries TCP and UDP, and the ICMP errors about them, but not ping. A ping through it goes unanswered even when TCP works.
• The destination sees the router host's address.
• It carries traffic into the destination network, but not out of it. A host on that LAN without an anchor cannot reach the overlay through it.
• If a destination does not answer within 30 seconds, the router resets the connection. Its only limit is the process's limit on open files.

Use a router with an interface where the anchor may have one. Use a userspace router where it may not, such as in an unprivileged container or on a host whose firewall the anchor must not change. To run a userspace router with conflux, use conflux proxy --subnet.

Changing a Router's Networks

A member whose credential carries the subnets power can send another member a Subnets order. The order replaces the networks that member serves. Machines are never given a power.

On a sovereign network, the order comes from the guardian's root node in the realm, which the console calls the admin node. On the instance VeilNet hosts, only you can change your machine's networks.

$ sudo conflux subnets -peer anchorgopyjl472a… -set eth2
gopyjl472a now serves eth2
  it keeps the list in its directory, over the one it was started with
  offers 10.40.0.0/24

• The router checks each entry against its host and refuses any entry that does not match. It saves the new list in its state directory and uses it from then on, across restarts. A member without a state directory refuses the order.
• If an entry in that list later matches nothing, the anchor holds it back until it matches again, instead of stopping. The subnets-withheld event names the entry.
• An anchor that started with nothing to serve starts forwarding when an order gives it something to serve.
• On a conflux machine, the next up or proxy with --subnet or --no-subnet sets the list again.
• When a network leaves the host, the router stops offering it. Another router for that network then takes its new flows.

To read a router's list, run sudo conflux subnets -peer ID. The output shows whose order set the list, and when.

Using an Exit Node

When a client uses an exit node, the overlay becomes its default route. The anchor's own traffic to its peers must still use the underlay, the physical network below the overlay. So the anchor first routes each peer's underlay address through the host's own gateway.

It then adds the default route as two halves, so your own default route stays in place underneath:

$ ip route
0.0.0.0/1 dev anchor0 scope link              # the overlay default, as two halves...
128.0.0.0/1 dev anchor0 scope link            # ...so the operator's own default survives
default via 192.168.10.2 dev eth0             # and it does: untouched
192.168.10.0/24 dev eth0 scope link           # on-link, more specific, still wins
203.0.113.2 via 192.168.10.2 dev eth0         # a peer's underlay address, pinned

• IPv6 works the same way.
• A peer on a directly connected network needs no extra route.
• If the host has no gateway to route peers through, the anchor does not add the default route, and reports a route-rejected event. This keeps the host reachable.

$ ip route get 1.1.1.1          → dev anchor0            the internet, over the overlay
$ ip route get 203.0.113.2      → via 192.168.10.2 eth0  a relay: the real uplink
$ ip route get 192.168.20.3     → via 192.168.10.2 eth0  a peer's own address on that LAN
$ ip route get 192.168.20.9     → dev anchor0            a host on the same LAN

Joining Two Sites

To connect two sites, run a subnet router on each site's LAN. Every machine on either site can then reach the other site, including machines that run no anchor.

Each router learns the other site's network from its advertisement. A machine without an anchor needs only one route, to the other site's network through the local router:

ip route add 192.168.20.0/24 via <the local anchor's LAN address>

Run two routers per site, so that one router is not a single point of failure. Give them the same taint to make them interchangeable, or different taints to control who reaches what.

Checking a Route