Telemetry
Learn how the guardian collects telemetry from every machine, and how to read the health and traffic of your fleet in the console.
- Who
- Every role reads telemetry. Auditors and above read flow records
- Where
- Console → Telemetry, and a machine's panel
How Telemetry Is Collected
Every machine the guardian commissions reports telemetry, and it is always on. The guardian collects it in two ways. In-band answers show the present, and exported metrics give the history.
| In-band | Exported | |
|---|---|---|
| How it works | The realm's admin node sends the machine a signed telemetry order, and the machine answers with its current readings | The machine sends its metrics and flow records to the guardian, with its own collector credential |
| Travels over | The overlay, within the realm's own rules | The IP network, straight to the guardian |
| How often | Every 30 seconds by default | Every 60 seconds by default |
| Kept | Not kept. It is the console's view of the present | 90 days, in the metrics store |
The console shows the in-band answer while it is fresh, and the last export after that. If neither has a reading, the console shows a blank.
In-Band Telemetry
A telemetry order is a signed request for a machine's current readings. It reaches only the realm of the anchor that sends it, so the guardian runs an admin node in every realm. The admin node is the guardian's own anchor in that realm, and it runs inside the guardian stack.
The admin node carries no data and relays nothing for anyone. It reaches your network only by connecting out to your own bootstrap nodes.
Between operator actions, the admin node holds only the telemetry power. It asks for telemetry in rounds:
- Every 30 seconds by default, it asks every active machine in its realm for a report. To change the interval, set
GUARDIAN_INBAND_INTERVAL_SECONDSto a value from 5 to 300. - If it has no route to a machine yet, the answer reads as unreachable. The admin node then connects to the machine, so the next round can reach it.
- An answer stays current for three rounds. After that, the console takes the machine's figures from its export.
- It does not ask revoked machines.
The Telemetry page shows each realm's in-band state, and the reason when the state is not running:
| State | What it means | What to do |
|---|---|---|
running | The realm's admin node is up and asking every machine | Nothing |
starting | The admin node is starting and will be up within one round. This is normal for a new realm, or just after the guardian starts | Wait one round |
withheld | Your realm credential does not grant the telemetry power. The admin nodes still run for realm control, and the console reads the fleet from exports only | Ask VeilNet for a realm credential that grants telemetry |
failed | The admin node could not start. The console shows the reason, and the next round tries again | Read the reason |
off | Nothing can run yet, because there is no realm credential or no sealing key | Finish deploying the guardian |
Reading the Fleet
Open Telemetry and choose the realm. The overview shows:
- How many machines report, and how many are silent.
- The in-band state, with its admin node.
- The state of the metrics store.
- A chart of the last hour.
- How machines are reached.
- The relay load on each bootstrap node.
Guardian console — TelemetryNorthern Commandtier2 · au-ismPlatformDashboardRealmsNodes2RenewalsTelemetryRealmsguardian realm3office14field11lab9DeploymentAccountsStack & complianceSettingsGuardianTelemetryguardian-adminTelemetryIn-band from this guardian's own admin node in each realm — every node's own report of now — and OTLP into the metrics store for history. A node that never reported still shows.OverviewNodesRealmguardian realmReporting3 / 3active nodes in this realmSilent0never reached, or gone quietIn-bandthis realm's admin node · asks every 30 sMetrics storeOTLP history · 90 d retentionThe last hourconnections 120relay circuits 55Summed across this realm's nodes, as they exported it over OTLP, every 30 s. Each node's own history is in its panel.11:0011:2011:4012:00How nodes are reachedreachability gauge, nowDirect3100%Cone NAT00%Symmetric NAT00%Silent00%Symmetric NAT is the one to watch: those nodes need a bootstrap node's relay to be met.Relay loadcircuits each bootstrap node carries, nowbootstrap-131bootstrap-218bootstrap-36A bootstrap node carries a sub-realm's circuits without being able to read them.
Flow Records
Flow records log the traffic a machine exchanges with its peers. They are on by default for every machine the guardian commissions. You can turn them off for a machine when you commission it.
Each flow record names the peer, both overlay addresses, the transport protocol and the destination port. It counts the bytes and packets sent in each direction. Traffic the machine relays for others is recorded only as its two hops and its byte count. Payloads are never reported.
Auditors and above can read flow records on a machine's Flows tab. The guardian keeps them for 90 days. To turn flow records off on a running machine, edit its conflux.json and restart conflux.
Exported Telemetry
Each machine exports telemetry with its own collector credential, which is separate from the secret it renews with. A leak of one reveals nothing about the other. Revoking a machine cuts off its reporting at once.
Machines export every 60 seconds by default, and GUARDIAN_METRIC_INTERVAL_SECONDS sets the interval. The guardian keeps exported telemetry for 90 days. See Telemetry intervals for both interval settings.
The fleet view reads seven gauges for each machine: connections, peers known, neighbours, reachability, overlay MTU, routes served and relay circuits. A machine's panel shows every reading it reports.