VeilNet

Run Modes

Choose how an anchor runs on a host: with a network interface, in userspace, as a reverse proxy, over an uplink or in low-latency mode.

Address Ownership

The main choice is who holds the overlay addresses: the host, or the anchor itself. Every other difference between the modes follows from it.

With an interface, the host holds the addresses, so every program on the machine can use the overlay. Creating the interface needs administrator rights. In userspace mode, the anchor holds the addresses in its own network stack. Nothing on the host can see the overlay, and no privileges are needed.

Fig. 01 — Publishing a service, two ways01 / 02
CLIENTlaptopCONFLUX UPanchor0OVERLAYSERVER · CONFLUX PROXYdb-1ANCHORD · VIRTUAL KERNEL[fd80:…]:543210.128.0.5:5432SERVICEpostgres · 127.0.0.1:5432NO HOST SOCKET · SEES 127.0.0.1

Userspace. The anchor holds the overlay addresses, and nothing is bound on the host. For each incoming connection, a reverse proxy opens a new connection to the service on loopback. Conflux runs this mode as conflux proxy.

Interface. The host has a network interface that holds the overlay addresses, and the service listens on it as on any other interface. Peers that connect over IPv4 appear at translated addresses. Conflux runs this mode as conflux up.

Comparing the Modes

ModeWhat the host seesPrivilegeWhen to use it
InterfaceA network interface that any program can useAdministrator rights, and IPv6 on the hostServers and laptops that should simply be on the network. One per host
UserspaceNothing. The overlay runs inside the anchorNoneContainers, CI and unprivileged hosts. It can still act as a subnet router or an exit node
Reverse proxyOnly the services it publishesNonePublishing a service at an overlay port without an interface. The service needs no changes
UplinkA serial line or another file descriptor, in place of a networkWhatever access the device needsA machine with no IP network beneath it. One link carries one peer
Low latencyA smaller interface MTU, measured from the routeAs for the mode it runs inReal-time traffic. A lost frame stays lost and does not hold up the frames behind it

Low latency and uplink are not separate modes. They change how frames travel, or what they travel over, in interface or userspace mode.

To see the same choice as conflux presents it, see Proxy and TUN Modes.

Interface Mode

In interface mode, the anchor creates a TUN device, named anchor0 unless you choose another name. The device holds the anchor's overlay IPv6 address and its IPv4 address, so ping, ssh and every other program can use the overlay.

PlatformDevicePrivilegeInterface name
Linux/dev/net/tunCAP_NET_ADMINThe name you choose, such as anchor0
macOSutunrootAlways utunN. A name you choose is only a preference
FreeBSD, OpenBSDtunroottunN
WindowsWintunAdministratorThe name you choose
Other platforms——Use userspace mode
  • The host needs IPv6, because every overlay address is IPv6. If the kernel boots without IPv6, such as Linux with ipv6.disable=1, the anchor refuses to create the interface with REASON_TUN_UNAVAILABLE. Use userspace mode on such a host. If IPv6 is only turned off by sysctl, the anchor turns it on for its own interface and nothing else.
  • Run one interface anchor per host. It manages the host's routes, its own ports and, on a router, the forwarding and firewall rules, and only one anchor can own these. To run a second anchor, put it in its own container or run it in userspace mode.
  • While it runs, the anchor opens its UDP port and the discovery port, 4747, in the host firewall. It closes them when it stops.
  • The interface MTU is 65,498 bytes on every route. On OpenBSD, the tun device limits it to 16,384 bytes, and the anchor reports this.
  • On Windows, put wintun.dll next to the executable or in system32. The DLL is not bundled, and the error message says where to put it.

Userspace Mode

A userspace anchor needs no privileges and no device. The overlay runs inside the anchor, and the host cannot see it. Use this mode in an unprivileged container or a CI job.

Every TCP connection that a userspace anchor carries uses keepalive, so a peer that disappears cannot hold a connection open.

A userspace anchor can still act as a subnet router or an exit node, and forwards TCP and UDP itself. It does not answer ping for the destinations it forwards to. See Userspace Routers.

Reverse Proxy

A reverse proxy publishes a TCP or UDP service on the host at a port on the overlay. It needs no interface and binds no port on the host. Peers reach the service at the anchor's private IPv4 address, as on a VPN, and at its overlay IPv6 address.

Write each entry as OVERLAYPORT[/tcp|udp]=BACKEND, with the backend in its usual form. For example: 8080=127.0.0.1:3000, 5432=[::1]:5432 or 53/udp=127.0.0.1:53.

  • The proxy opens a new connection to the backend for each incoming connection, so it picks up a restarted service by itself.
  • An anchor with a public IPv4 address, or with no IPv4 address, answers on its overlay IPv6 address only.
  • A UDP flow ends two minutes after its last datagram in either direction.
  • The proxy sets no limit on connections.
  • You cannot use a reverse proxy together with an interface. With an interface, bind the service to the overlay address instead.

On a conflux machine, run sudo conflux proxy 8080=127.0.0.1:3000 to publish a service. Conflux remembers it. To change a published service without a restart, use the anchor's own proxy command. Changes made this way are not remembered.

$ sudo conflux anchorctl proxy
OVERLAY                                 PROTO  BACKEND          LIVE  OPENED  FAILED  LAST ERROR
[fd0a:…:2040]:8080, 10.4.0.7:8080       tcp    127.0.0.1:3000   2     47      0
[fd0a:…:2040]:5432, 10.4.0.7:5432       tcp    [::1]:5432       0     3       3       connection refused

$ sudo conflux anchorctl proxy -rm 8080 -add 8080=127.0.0.1:3001

A FAILED count with connection refused means the backend is down. Use these columns to tell a proxy with a broken backend from one that nobody has used yet.

Low-latency Mode

By default, all frames to a peer travel on one reliable stream. In low-latency mode, frames travel as separate datagrams instead. Nothing waits behind a lost frame, and a lost frame is not sent again. The mode is off by default.

  • The interface MTU is measured from the route instead of being fixed at 65,498. It starts at 1280 and rises to what one datagram can carry, such as 1353 on a route with a 1500-byte MTU. The route with the smallest MTU sets it, and it never goes below 1280.
  • A frame too big for a datagram travels on the stream instead, and anchor_frames_degraded_total counts it. The mtu-changed event fires only in this mode.
  • In testing, round-trip time during a saturating transfer dropped from about 55 ms to 16 ms. Bulk throughput is lower.
  • The default mode already shares each peer's queue fairly between flows, so a ping does not wait behind a bulk copy.
  • Anchors in either mode can share a realm and exchange data.

On a conflux machine, run sudo conflux up --low-latency to turn the mode on. For the measured cost, see Performance.

Checking the Mode

The mode row at the top of sudo conflux status shows whether the machine runs with an interface or in userspace, for example proxy — userspace, no host interface. The anchor status below it shows the MTU:

10:12:22  reachability=nat-cone  mtu=65498  peers=3  up=4m12s
  • mtu=65498 means the default mode. A lower value with an interface means the platform does not accept an MTU that large.
  • In low-latency mode, mtu=1280 means no connection has answered yet. If the value never rises above 1280, a route is dropping large packets and no ICMP message is getting back.
  • On Linux, run ip addr show anchor0 to see the interface and its addresses. A userspace anchor has no interface.