Docs Self-hosting

Nodes and mirrors

Run Obscenter on several machines: a primary and mirrors that replicate every report, agents that fail over, automatic promotion, and one-command joins.

An install can run on more than one machine. One node is the primary, and the others are mirrors. Every measurement reaches every node, agents move to another node when one stops answering, and a mirror takes over if the primary goes away. Nodes are managed by the install's administrator under Settings → Nodes.

What is replicated

Every accepted report: browser beacons, agent reports, device readings and OTLP traces. Each one is put in an outbox as it is answered, and handed to each peer in order. The peer replays it through its own collector with the original client address and receive time. A peer that is down gets the backlog when it returns. Seven days are kept (CLUSTER_OUTBOX_DAYS).
A reconcile every ten minutes compares the browser tables, per application and hour over 14 days, and fills in anything a node lacks.
Configuration: mirrors follow the primary's accounts, applications, keys, channels, checks, objectives, workflows, dashboards, funnels and models, and the switches people set.
Tickets come from the primary. A mirror runs no detectors while it stands by.

A write made on a mirror's dashboard is forwarded to the primary as the signed-in person. Sign-in works on either node. With the primary down, such a write answers 503 and says so. Only the primary sends mail, webhooks and workflow calls, and runs synthetic checks. A mirror takes over sending after three minutes without the primary, and hands back after it returns.

Agents

Every answer to an agent carries the list of nodes. The agent keeps it in nodes.json beside itself. It moves to the next node the moment one does not answer, and tries its configured node again every five minutes. Each report carries an id, so a report resent to the other node is stored once on each.

Roles and promotion

Roles live in the database, with an epoch that rises on every change. In Settings → Nodes:

Make primary hands the role over. On a mirror, it works only for itself, and only once the primary has not answered for a minute.
Take out removes a node.
Put back and Join a node… add one.

A primary that is silent for ten minutes is replaced automatically (CLUSTER_AUTO_PROMOTE_MIN, 0 to turn it off). The first reachable mirror by name promotes itself and sends a mail. When the old primary answers again, it reads the later epoch and becomes a mirror.

Two nodes have no majority. If the link between two running nodes is cut, both act as primary until it heals. Reports still reach both through the outbox. Settings changed on the losing side in the meantime are replaced by the winner's.

Joining a node

A new or returning node is built from a copy of the primary's database, so it has no hole. Join a node… on the primary gives a one-use code, valid for an hour, and the command to run. For a fresh machine it is one line:

curl -fsS 'https://primary.example/join/node.sh?code=…&name=node3&url=https://node3.example' | sudo bash

It fetches the primary's own Node runtime and code from the primary, and nothing from anywhere else. It installs the systemd units and runs the join. That means a consistent snapshot of the primary, streamed over, with the primary's outbox cleared from the copy, so nothing is applied twice and nothing is missed. It then starts the node, and fails unless the node answers as a mirror. With nginx and certbot on the machine, it also serves the node's address. On a machine that already has the code, the join is node server/joinrun.mjs --primary <url> --code <code> --name <name> --url <its address>, run with the service stopped.

Configuration

Variable	Meaning
`NODE_NAME`, `NODE_URL`	This node's name and its own address. Never use a shared name here.
`CLUSTER_TOKEN`	The same secret on every node.
`NODE_ROLE`, `CLUSTER_PEERS`	The first roles and peers (`name=url,…`). After the first start, roles are kept in the database.
`COLLECT_URL`	A name with an address record for every node, for tags and agents.

Give every node its own address for traffic between nodes. A shared public name that reaches either node must not be used as a node's own address.

Backups across nodes

Each node's daily backup is also handed to every peer, which keeps the newest two (BACKUP_PEER_KEEP). It travels in 8 MB pieces, resumes where a broken transfer stopped, and is kept only if its SHA-256 matches. A restored node catches up from its peer by itself: through the outbox for the last seven days, and through the reconcile for older browser data. See Restoring.

Taking Obscenter off a machine

sudo bash deploy/uninstall-node.sh --yes removes the node's services and code. It moves its data aside rather than deleting it. It refuses a node that is still a primary or a mirror, so Take out comes first.