Discovery and NAT Traversal
How anchors find the members of their realm and reach each other across NAT and firewalls.
How Anchors Connect
Every anchor knows every member of its realm, so it never asks a directory server where a peer is. To reach a peer, an anchor uses one of three kinds of connection:
| Connection | When it is used |
|---|---|
| Direct | The peer is publicly reachable, has a port forward, or is on the same LAN |
| Hole punching | Both anchors are behind NAT. They send to each other at the same moment, and each NAT lets the other side in |
| Relayed | Hole punching cannot work, for example behind a symmetric NAT. Another member forwards the traffic, which stays encrypted end to end |
You do not choose between them. If a direct connection is not up quickly, traffic starts on a relayed route. The anchors then try to punch through, and move the traffic to the direct connection once it works.
Your anchor dials one address it was given, here a reachable member. The member sends everything your anchor does not yet know about the realm, so your anchor learns about the peer without contacting it.
This first packet out also makes your router's NAT map the machine's inside address and port to an outside one.
Behind a NAT, an anchor cannot see its own outside address. Each peer reports the address it sees, and the anchor uses these reports to learn its outside address and its NAT type.
Here, members on two networks see the same outside port. The result is nat-cone, and hole punching can work. If they saw different ports, the result would be nat-symmetric.
Port mapping is on by default. The anchor asks your router to forward a port, over UPnP and NAT-PMP.
When the router grants a forward, the anchor advertises it. Peers then dial the forward directly, and no hole punching is needed.
Without a forward, neither anchor can dial the other. Most NATs let a packet in only from an address and port they have already sent to.
The two anchors agree on a moment through a relay they share. Both then send to each other's outside address at once. Each NAT has now sent a packet to the other side, so it lets the other side's packets in.
Some NATs use a new outside port for every destination. Hole punching cannot work through them, because the peer cannot know which port to send to.
The anchor does not try. Traffic stays on the relayed route, encrypted end to end.
All addresses are examples from the documentation ranges. One reachable member plays every helping role here. In a realm, the bootstrap entry, the peers that report and the relay can be different members.
Bootstrapping
To join a realm, an anchor needs the address of one member. Any member works, because every anchor answers requests from its realm. From that member, the anchor learns about all the others.
- Write an entry as
host:port, or asanchorID@host:portto name the anchor you expect there. Any other anchor at a named address does not answer. - The anchor dials all its entries in parallel, so an address that does not answer does not delay the others.
- The anchor remembers the members it has connected to before. It can rejoin even after the member it first joined through has moved.
- The anchor resolves host names again on every attempt, so it finds a member whose host has moved to a new address.
- An anchor that loses all its connections bootstraps again, every five seconds, until it connects.
On a conflux machine, the bootstrap list comes from the manifest, and --peers replaces it. On a sovereign network, the guardian puts your organisation's own publicly reachable bootstrap nodes first in the manifest, and every renewal brings the list up to date.
Local Network Discovery
An anchor also looks for members on its local networks. Two machines on one LAN find each other with no address configured.
The anchor sends a multicast probe, and members of the same tree answer. A tree is the set of realms under one root. Discovery only finds members to dial. Admission and taints still decide what they may do once they are connected.
| Setting | Value |
|---|---|
| Multicast groups | 239.255.4.70 and ff02::4:70. Neither crosses a router |
| Port | UDP 4747. It is fixed and cannot be changed |
| When it probes | Every 2 seconds while the anchor has no connection. It stops probing once it connects, and keeps answering probes while it runs |
On a host that faces the internet, drop UDP 4747 from outside your local networks, or turn discovery off with sudo conflux up --lan-discovery no. Discovery is always off on an uplink.
Gossip and Liveness
Anchors share what they know by gossip. A change, such as a new member, a route advertisement or a departure, goes at once to every connected peer. From there it spreads across the realm.
When two anchors connect, they compare what they hold and exchange only the differences. A network that was split heals at the first connection across the split.
- Each anchor reissues its own record every 30 seconds. With the default interval, its peers forget it after 4 minutes 15 seconds without a new record. A shorter interval does not make changes spread faster. It only makes peers forget a crashed anchor sooner.
- An anchor that stops cleanly announces that it is leaving, and its peers drop it at once.
- Each anchor advertises its routes: the networks it forwards, the relays that carry traffic for it, whether it relays, and how to reach it. An advertisement lasts 15 minutes and is renewed halfway through.
Detecting the NAT Type
An anchor works out what kind of NAT it is behind, because that decides whether hole punching can work. It needs no STUN server. Instead, every peer reports the address it sees the anchor at, and the anchor compares the reports.
| NAT | Outside port | Hole punching |
|---|---|---|
| None | — | Not needed. Peers connect directly |
| Full cone | The same for every destination | Works |
| Restricted cone | The same for every destination | Works |
| Port-restricted cone | The same for every destination | Works. This is the most common case |
| Symmetric | A new port for every destination | Does not work. Traffic is always relayed |
No single peer can decide the result. The anchor counts one vote per network, and only from networks other than its own. It shows the result as one of four values:
| Value | Meaning | Effect |
|---|---|---|
public | A peer on another network reached the anchor without being dialled first, and saw no NAT in between | Peers dial it directly |
nat-cone | Peers on different networks see the same outside port | Hole punching can work |
nat-symmetric | Peers on different networks see different outside ports | No hole punching, only relays. The anchor also stops relaying for others |
unknown | Not enough reports yet | Treated as if hole punching can work. This is normal in a realm of two, or when all members share one network |
When the machine moves to another network, the anchor forgets what it learned and starts again. The log then says: the network moved; forgetting what was learned about the old one.
Failed punches
If hole punching to peers on three different networks fails within ten minutes, the anchor tells its peers it is symmetric. They then stop trying to punch. It still relays for others. A successful punch, an unprompted connection from outside, a network move or ten minutes without new failures clears this. While it applies, status shows observed= beside the NAT type.
Port mapping
Port mapping is on by default. The anchor asks your router for a port forward over UPnP and NAT-PMP, keeps it renewed, and advertises it to peers. It does not advertise a forward when the router's own outside address is private or carrier-grade NAT, because another NAT sits above it. Port mapping is off on an uplink.
If your router does not support UPnP or NAT-PMP, forward a port by hand. First give the machine a fixed port:
$ sudo conflux up --port 4700
Then forward that UDP port on your router to the machine.
Hole Punching
Most NATs let a packet in only from an address they have already sent to. Two anchors behind such NATs therefore cannot dial each other. Instead, both send at the same moment, and each outgoing packet opens its own NAT for the other side.
The two anchors agree on the moment through a relay they already share. This exchange is encrypted end to end, so the relay cannot read or change it. The relay keeps carrying the traffic during the punch, so a failed punch only means higher latency.
- The anchor does not try to punch if either end is behind a symmetric NAT.
- The anchor retries a failed punch with the same peer at once, up to three times. After that, it waits before each new attempt, starting at 5 seconds and doubling up to one minute.
- When a punch succeeds, the anchor reports a
path-upgradedevent. In a test with two cone NATs, this happened about fifteen seconds after the anchors started.
Relays
When hole punching cannot work, traffic stays on a relayed route through a member that both ends can reach. Every anchor relays for the members of its realm that it is connected to, unless it is behind a symmetric NAT. You do not need to request or configure a relay.
- A relayed route has at most five relays.
- A relay carries traffic only while it is needed. Once the two ends punch through, traffic moves to the direct connection.
Reachable Members
Every realm needs at least one member that others can dial. New members join through it, hole punches are arranged through it, and pairs that cannot punch through relay through it.
A reachable member does not need a public IPv4 address. Any of these is enough:
| Way | What you need |
|---|---|
| A public IPv4 address | A hosted server, for example |
| Port mapping | A router that supports UPnP or NAT-PMP. The anchor sets up the forward with no configuration |
| Global IPv6 | Most home IPv6 has no NAT, only a firewall. The anchor tries IPv6 first |
| The same LAN | A realm that stays on one network needs nothing reachable from outside |
Existing connections keep working without a reachable member. New members and new pairs of peers need one.
Firewalls
- An anchor listens on one UDP port. Any firewall between a reachable member and its peers must let that port through.
- While it runs, an anchor with a host interface opens its own UDP port and UDP 4747 in the host firewall. It removes both rules when it stops.
- Configure the host to drop unsolicited UDP rather than reject it. The anchor's port then stays as silent as every closed port.
Inspecting Peers and Routes
On a conflux machine, run anchor commands through sudo conflux. Where a command takes an ID, use the full anchor… form from sudo conflux status.
To list every member this anchor knows, run sudo conflux peers:
$ sudo conflux peers ANCHOR OVERLAY HARDWARE STATE CONN DATA REACH RTT zvukrxk67n fd3f:d1b7:6962:9e12:b18d:b0:bd9e:8f4a ca:98:3e:86:05:d9 connected out yes unknown 0s
| Column | Meaning |
|---|---|
| STATE | connected means a live connection. known means the anchor knows the peer from gossip but has no connection. Both are normal |
| CONN | out if this anchor dialled, in if the peer did, - for none. A star marks the connection in use |
| DATA | Whether the two anchors may exchange data |
| REACH | The peer's NAT type, as the peer reports it |
| RTT | The round-trip time, measured every five seconds. 0s means not measured yet |
peers -long adds each peer's addresses, relays and realm, and when it was last heard from.
To see this anchor's own NAT type, read the first line of the anchor status at the end of sudo conflux status:
10:12:22 reachability=unknown mtu=65498 peers=1 up=4m12s
To check the route to one peer, run route. It prints direct or relayed -> hop -> hop. To connect or punch through now, run connect or punch:
$ sudo conflux route -to anchorzvukrxk67n… direct $ sudo conflux connect -to anchorzvukrxk67n… connected, directly $ sudo conflux punch -to anchorzvukrxk67n… the path is now direct
If peers shows a peer as connected but route does not show direct, something is wrong.
If hole punching cannot work, punch answers still relayed: holepunch: the peer is behind a symmetric NAT. This is an expected result, not an error.
$ sudo conflux events 03:55:04 peer-up zvukrxk67n 03:55:11 path-upgraded zvukrxk67n a relayed peer is now directly reachable
| Metric | What it tells you |
|---|---|
anchor_punch_attempts_total | Compare it with anchor_punch_successes_total. The ratio shows how well NAT traversal works |
anchor_punch_suppressed_total | Punches skipped before anything was sent. A high value with few attempts means a pair is waiting out its retry delay |
anchor_punch_deferred_total | Punches where the other end started at the same moment. It rises on a healthy anchor |
anchor_peers_known | How many members this anchor knows. If it stays flat while connections come and go, gossip is not reaching this anchor |
anchor_handshake_failures_total | Bootstrap dials that did not connect, most of them unanswered |
anchor_knocks_refused_total | On the receiving end, connection attempts it refused, by reason. The dialling anchor gets no answer, so a refusal shows only here |
anchor_lan_peers_found_total | Anchors found by local discovery, by iface |
anchor_lan_probes_refused_total | Probes not answered, by reason. unreadable means another tree, credential a credential it does not accept, and stale a clock too far off |
anchor_lan_groups_joined_total | Multicast groups joined. It rises when a network is attached to a running host |