Install the server monitoring agent

One agent runs on each machine, whatever that machine serves. It needs root or administrator rights once, to register a service. After that it only makes outbound calls to your install. It opens no inbound port, and nothing dials into the machine. On Linux the agent runs as root and does change the machine in a few specific ways, all listed under What it changes with the way to switch each one off; the installer prints the same list when it finishes.

Linux

Copy the command from Launchpad → Install → Host and run it as root, or with sudo:

curl -fsSL https://your-install/agent/install.sh | sudo sh -s -- --key agt_… --url https://your-install

The installer needs systemctl and curl, and checks that it can reach your install before it writes anything. It uses Node.js 18 or newer if the machine has it. Otherwise it downloads a runtime from your install, not from nodejs.org, for x64, arm64 or armv7l, and checks it against a SHA-256 sum. The runtime download works on glibc systems only. On Alpine (musl), point it at a Node binary with --node.

What it writes:

• The agent under /opt/obscenter-agent. Each file is checked against the SHA-256 your install publishes for it before it is put in place.
• The key in a root-only file there, agent.env. It never goes into a unit file or onto a command line. Re-running the installer keeps every other line you added to that file.
• One systemd service, obscenter-agent.service, hardened with ProtectSystem=strict, ProtectHome and PrivateTmp.
• The command /usr/local/bin/obscenter-agent.

Before it finishes, it runs one real report and prints what it read and what came back. If that fails, it prints the command that shows the agent's log and exits with an error. Re-running the same one-liner is safe, and it is also how you reinstall over an existing agent.

Installer options

Option	Meaning
--key, --url	Required: the agent key and your install's address.
--ignore a,b	Path prefixes not to record.
--no-allow-instrument	Do not let the agent change configuration on this machine: no systemd drop-ins, no nginx changes, no Postgres restarts (see below). --allow-instrument is the default.
--allow-terminal	Let a signed-in person with the Console capability open a root shell on this machine from the dashboard. Off by default since agent 0.95.1; --no-allow-terminal says so explicitly. Machines installed before 0.95.1 keep accepting sessions, also when the installer is re-run on them. See Console.
--node <path>	Use this Node.js binary.
--own-node	Fetch a runtime even if the machine has one.
--prefix <dir>	Install somewhere other than /opt/obscenter-agent.

What it collects

The agent samples the machine every 15 seconds and sends a report once a minute.

• The machine: processor, memory, swap, load average, every filesystem, every network interface, disk throughput and IOPS, page faults, TCP retransmissions and failed connections, boots, and the virtualisation it runs on.
• Processes: every process with its processor time, memory, threads, disk I/O and open files. Node, Python, PHP, Ruby, Java, .NET, Erlang and Perl processes are recognised. Anything that looks like a credential on a command line is redacted.
• Services, containers and ports: systemd units, Docker containers with their cgroup limits and throttling, listening ports, and which process is retransmitting.
• Packages: the dpkg or rpm inventory, which installed packages a running process actually uses, and which processes still run old code after an upgrade. The inventory feeds vulnerability scanning.
• Logs: journal units and log files. Every source arrives switched off and is offered, not collected. See Logs and log patterns.
• Web sites and nginx: the virtual hosts nginx serves, and a timed access log the agent can turn on. See Automatic injection.
• Postgres in Docker: statement and database statistics. See Database monitoring.

It never collects credentials, cookies or authorization headers. Visitor addresses are not kept, only a country and region. The CPU and memory profilers stay off until you switch them on. See Profiling.

What it changes on the machine

The agent runs as root. Its service is sandboxed, but what it writes outside its own directory it asks systemd to run outside that sandbox, so treat it as root. These are everything it changes on its own, and how to stop each. A line marked agent.env goes in /opt/obscenter-agent/agent.env, followed by sudo systemctl restart obscenter-agent.

What	When	Off
A systemd drop-in, /etc/systemd/system/<unit>.d/obscenter.conf, for each Node.js service it finds. It takes effect at that service's next restart; nothing is restarted for it.	Configuration allowed	OBSCENTER_AUTO_PREPARE=0 (agent.env)
nginx: /etc/nginx/conf.d/00-obscenter-rum.conf (the tag in each site's pages), /etc/nginx/conf.d/obscenter-log.conf (a second, timed access log), and one marked line, proxy_set_header Accept-Encoding "" # obscenter-ae, inside a vhost that sets its own proxy headers. Then a reload. nginx -t runs first, and if it fails every file is put back and nothing is reloaded.	Configuration allowed	obscenter-agent rum --disable (the tag and the line), obscenter-agent nginx --disable (the log), or the two switches in Settings → Server agent
Postgres in Docker: adds pg_stat_statements to shared_preload_libraries and restarts that container, once. See Database monitoring.	Configuration allowed	OBSCENTER_NO_PG_FIX=1 (agent.env)
A service restart, to add or remove the in-process agent, only when somebody asks for that unit by name. It checks the service came back and puts it back if not.	Configuration allowed	the switches below
Its own files, when your install serves a newer version. See Updates.	Always	OBSCENTER_AUTO_UPDATE=0 (agent.env)
The execute bit on a root-owned script in /usr/local/bin that starts with #! and lost it. Each repair is reported.	Always	OBSCENTER_NO_FIX=1 (agent.env)
A root shell, opened from Backend → Console and relayed over the agent's outbound connection.	Only if installed with --allow-terminal (or installed before 0.95.1)	obscenter-agent deny-terminal, or OBSCENTER_NO_TERMINAL=1 (agent.env)

"Configuration allowed" is one permission. --no-allow-instrument at install, sudo obscenter-agent deny-instrument on the machine, or the switch on the host's settings page takes all four of those rows away at once. sudo obscenter-agent uninstall takes everything back.

Tracing Node.js services

A Node.js process can only load an agent when it starts. So the server agent prepares every Node.js service it finds and leaves the restart to you. It writes a systemd drop-in that sets NODE_OPTIONS and runs daemon-reload. The running process is not touched. Backend → Processes shows three states:

State	Meaning
off	Nothing is written.
ready · restart it	The drop-in is in place. The process still runs without the agent.
on	The process runs with the agent inside it.

The next deploy, or sudo systemctl restart <unit>, turns ready into on. The toggle on the Processes page does both steps, writing and restarting, after asking first. If the unit does not come back healthy, it is rolled back. A drop-in is one file, so removing it undoes everything.

The agent never prepares its own unit or the install's own unit. You can exclude more units in Settings. To stop it preparing anything on a machine, set OBSCENTER_AUTO_PREPARE=0. PM2 applications have no systemd unit of their own. If systemd runs the PM2 daemon (pm2 startup, then pm2 save), the drop-in goes on the pm2-* unit, and each app reports under its own PM2 name.

Two switches decide whether the dashboard may do this:

• The machine's own: --no-allow-instrument, or sudo obscenter-agent deny-instrument. While it is off, the agent cannot write drop-ins at all.
• The collector's, per machine: Let this dashboard instrument services on the host, on its settings page.

The obscenter-agent command

Command	What it does
obscenter-agent list	The Node services running on this machine.
obscenter-agent selftest	Read the machine and report once, out loud.
obscenter-agent instrument <unit>	Write the drop-in and restart that unit, and only that one. It rolls back if the unit does not stay up.
obscenter-agent uninstrument <unit>	Remove the drop-in and restart the unit.
obscenter-agent nginx --enable / --disable	Add or remove the timed nginx access log.
obscenter-agent rum / rum --disable	List the sites carrying the tag, or remove it.
obscenter-agent allow-instrument / deny-instrument	Give or take back the right to change configuration: drop-ins, nginx, Postgres.
obscenter-agent allow-terminal / deny-terminal	Let the dashboard open a root shell on this machine, or refuse every session.
obscenter-agent status	The agent's own systemd status.
obscenter-agent uninstall	Remove the agent completely (see below).

Updates

Every report is answered with the agent version your install serves and a SHA-256 for each agent file. An agent running something else downloads the new files and refuses any whose checksum does not match. It then checks them with node --check and a self-check, swaps them in, and restarts its own unit.

Before the swap it starts a small guard through systemd, as its own unit, which outlives the restart. If the new version has not reported successfully within four minutes, or restarts on its own three times, the guard puts the previous version back and restarts it. A new version that is running but cannot reach your install is given up to half an hour, so an outage of your install does not roll every machine back.

A version that failed its check or was rolled back is not retried by itself. Update now on the host's settings page tries it once more, on the next report. Why the last update did not take — a download, a checksum, the check, or a rollback — is shown on the host's page and on its agent version. A new in-process agent reaches a traced service only when that service next restarts. Set OBSCENTER_AUTO_UPDATE=0 to stop self-updates.

By default every machine takes a new version within a minute of your install serving it. Settings → Server agent → Roll a new agent version out to lowers that to a share of the machines: the rest are told to keep the version they run until you raise it. The same machines always go first, chosen from their names, and raising the share only adds machines. Update now on a machine always updates it, and its page says when it is being held back.

The checksums protect against a damaged or altered download between your install and the machine. They come from your install, so they do not protect against your install itself being compromised; signing releases with a separate key would, and is not done yet.

Removing it

sudo obscenter-agent uninstall

This undoes the nginx changes, removes every drop-in and restarts those units, disables and deletes the service, and deletes the command and the install directory. Measurements already sent stay on the dashboard. To drop them as well, remove the host on its settings page.

Windows

Windows machines get their own agent, from the box beside the Linux one. See The Windows agent.

Agent keys and install links

The command on the Install page carries an agent key (agt_…). Whatever reports with a key lands in that key's organisation. A machine first reported with your personal key is yours, so a member who installs an agent sees that machine without anyone sharing it. Administrators manage keys under Settings → Agent & keys:

• Agent keys. Rotation is additive. Create a new key, reinstall machines at your own pace, then retire the old key. Every key works until it is retired, and the last active key cannot be retired. The page warns you if a key was used in the last ten minutes.
• Install on a new machine. A one-time link, valid for 4 hours, 1 day or 3 days: curl -fsSL https://your-install/install/<token> | sudo sh. It works once. A machine that first appears within two hours of the link being used is granted to the person who made the link.

Did it work

The machine appears under Backend → Hosts within a minute. See Hosts, processes and technologies. If it does not, run sudo obscenter-agent selftest on the machine. It prints what it read and what your install answered.