Docs Self-hosting

Data, retention and backups

Where Obscenter keeps its data, how long each kind is retained, the storage limits you can set, automatic daily SQLite backups and how to restore them.

The data directory

Everything lives in DATA_DIR:

rum.db, with its -wal and -shm files: the one SQLite database, in WAL mode.
secret: signs sessions and salts the IP hash.
backups/: the automatic backups.
Working folders for the load-test engine and node coordination.

Keep it outside the code directory. Then a fresh unpack or an rsync --delete of the code cannot take the database, the accounts and the secret with it.

Never delete rum.db to clear measurements. It holds the accounts and their sign-ins too. Use node server/cli.js reset, which wipes the measurement tables of every application, or one with reset <app>, and keeps everything else.

How long data is kept

Data	Kept
Raw browser data (sessions, views, resources, requests, errors, events)	Per application, Raw data retention: 30 days by default (`RETENTION_RAW_DAYS`), 1 to 3650
Hourly aggregates, for long-range charts	400 days (`RETENTION_AGG_DAYS`)
Session recordings	Per application, Recording retention: 7 days by default, 1 to 365
Logs	7 days and at most 1024 MB by default, whichever is tighter
IoT readings	The install's raw retention
Request body samples	2 days
Synthetic runs / screenshots	90 days / 7 days
Closed problems	90 days
Sentry data	90 days
Sign-in book	180 days

The rollup runs every minute and always recomputes the last few hours, so late beacons are included. The purge runs on its own thread, so nobody waits for it.

One limit is worth knowing. Inside the raw retention, percentiles are exact. Beyond it, a chart point that spans several hours shows the highest of the hourly p75 values, which is the worst hour rather than a true combined percentile.

Limits

Settings → System health shows what is using the disk, by kind, and how fast each kind is growing. Its Limits card bites only when you set a number. Nothing is deleted until you do. You can set:

a ceiling for the whole database, in GB;
a maximum number of days per kind: visitor sessions, session replays, backend traces, synthetic checks, machines and processes, IoT readings.

Above the ceiling, data is trimmed in this order: replays first, then machine data, traces, synthetic checks, and visitor sessions last. Run rollup now and applying retention are on the same page. On the command line, use node server/cli.js rollup and purge.

How much space

As a rough guide, the demo data is about 8 kB per session: three page loads, seven resources, nine API calls and ten interactions. At 10,000 sessions a day that is about 80 MB a day of raw data, or about 2.4 GB at 30 days' retention. Hourly aggregates are small beside it. If it is too much:

lower the application's retention;
turn off resource timing with data-resources="0", the largest table;
sample visitors;
or set a ceiling.

The collector accepts up to 120 batches a minute per address, 2 MB and 500 events per batch.

Automatic backups

Every install backs itself up once a day, unless BACKUP_OFF=1 is set. The backup:

runs on the jobs' thread;
uses VACUUM INTO, SQLite's online copy, because copying a live WAL database with cp is not safe;
checks the copy with quick_check and row counts against the source;
compresses it with gzip and stores the secret beside it;
keeps the newest three in DATA_DIR/backups/.

It refuses to start when the disk could not hold the copy, and says so. BACKUP_HOURS changes the interval and BACKUP_KEEP the number kept. On an install with several nodes, each backup is also handed to every peer. See Nodes and mirrors. Copy backups off the machine as well. A backup on the same disk does not survive the disk.

Restoring

systemctl stop <watchdog> <collector service>   # however you run it
cd "$DATA_DIR"
mv rum.db rum.db.broken; rm -f rum.db-wal rum.db-shm
gunzip -c backups/rum-YYYYMMDD-HHMMSS.db.gz > rum.db
cp backups/rum-YYYYMMDD-HHMMSS.secret secret && chmod 600 secret
systemctl start <collector service> <watchdog>

Restore the secret that belongs to the backup. With a different secret, everybody is signed out, and visitor counting across the restore point breaks.

ClickHouse (optional)

For heavy log and process-metric volumes, a ClickHouse server can hold those tables. Set CLICKHOUSE_URL (with CLICKHOUSE_USER, CLICKHOUSE_PASSWORD and CLICKHOUSE_DB), and CH_STORE=logs,process_metrics to name what goes there. Nothing changes unless CLICKHOUSE_URL is set. The logs page says which store answered a search.