Skip to main content

Operations

Once the server runs, the work is keeping it healthy: knowing that scans and agents work, keeping the logs, deleting old data, backing up what matters, and upgrading without losing it.

Health checks​

The server has two unauthenticated endpoints, for load balancers and orchestrators:

EndpointMeaning
GET /healthzThe process is alive.
GET /readyzThe server has started and is ready to serve requests.

The Helm chart uses them as probes for the server. A worker has no HTTP port, so it has no probes; a stuck worker shows up in the Running now panel of the Scans and tasks page instead.

What to watch​

Four things tell you whether the system does its job. Look at them after an upgrade and once a week:

  • Agents online on the dashboard. A host that goes offline stops reporting, and its findings age.
  • Last scan on the dashboard. A scan that has not run for hours means the scheduler or a worker is stuck.
  • Assets marked Scan pending for longer than a few minutes.
  • Failed entries in the delivery log of the alerts.

Set up an alert rule for Agent offline so that these show up without anyone looking.

Logs​

The server writes its log to standard output, one line per event, with the component in the line. LOG_LEVEL controls how much it writes: ERROR, WARN, INFO (the default) or DEBUG. Use DEBUG only while you investigate a problem, and set it back afterwards.

Every API request is logged with its method, path, status and duration. Errors of the background jobs are logged as well, with the job that failed.

Retention​

RETENTION_DAYS (90 by default) deletes old scan runs and alert logs once a day, in the retention-cleanup task. Findings are not deleted by age: they belong to the current inventory, and are removed when their asset is deleted or its agent is removed.

Backups​

Back up the state, not the containers. The state is:

WhatWhereNotes
The databaseSQLite file in /data, or your PostgreSQL, MySQL or MSSQL databaseThe inventory, users, roles, settings, mappings and the JWT signing secret. The most important part.
The S3 bucketMinIO or your S3 storeSBOM documents, generated reports and uploaded agent releases.
The secretsYour secret storeINITIAL_ADMIN_PASSWORD, the keys of the S3 store, the database and Redis passwords, the cluster tokens you still need.

Redis holds locks, caches and the job queue. You do not need to back it up: losing it drops queued jobs and the cache, and the schedules run again from the server's configuration. Start any scan you were waiting for again.

The Grype vulnerability database is downloaded again, so it does not need a backup. An air-gapped installation keeps a copy in GRYPE_DB_DIR that it needs to restore from its source.

For SQLite, copy the file while the server is stopped, or use a snapshot of the volume. Back up the database and the bucket at the same time, so that the SBOMs and the findings agree. Test a restore on another machine once.

Upgrades​

  1. Read the release notes of the new version.
  2. Back up the database and the bucket.
  3. Upgrade the server: docker compose up --build -d for compose, or helm upgrade with the new chart version for Kubernetes. Upgrade the agents after the server, with managed updates or by rolling out the new chart. See Agent updates.
  4. Check the health endpoints, the dashboard and the agents.

Migrations run when the server starts with SQLite, and in the migration job before the pods start with PostgreSQL, MySQL or MSSQL. Each migration is applied once and recorded in the database, so a restart never runs one twice. Keep the previous version's image until the new one has run for a day, in case you need to roll back: a migration is not undone by rolling back the image, so restore the database backup if you roll back across a migration.

Air-gapped installations​

Set GRYPE_DB_UPDATE=false. Copy the Grype database into GRYPE_DB_DIR from a machine that can reach the internet, and copy it again whenever you want new vulnerability data. Set RELEASE_MANIFEST_URL to an empty value, and install the agents with their binaries instead of the install command. The agents only need to reach the server; the server does not need to reach the agents.