Operations
Once the server runs, the work is keeping it healthy: knowing that scans and agents work, keeping the logs, deleting old data, backing up what matters, and upgrading without losing it.
Health checks
The server has two unauthenticated endpoints, for load balancers and orchestrators:
| Endpoint | Meaning |
|---|---|
GET /healthz | The process is alive. |
GET /readyz | The server has started and is ready to serve requests. |
The Helm chart uses them as probes for the server. A worker has no HTTP port, so it has no probes; a stuck worker shows up in the Running now panel of the Scans and tasks page instead.
What to watch
Four things tell you whether the system does its job. Look at them after an upgrade and once a week:
- Agents online on the dashboard. A host that goes offline stops reporting, and its findings age.
- Last scan on the dashboard. A scan that has not run for hours means the scheduler or a worker is stuck.
- Assets marked Scan pending for longer than a few minutes.
- Failed entries in the delivery log of the alerts.
Set up an alert rule for Agent offline so that these show up without anyone looking.
Logs
The server writes its log to standard output, one line per event, with the component in the line.
LOG_LEVEL controls how much it writes: ERROR, WARN, INFO (the default) or DEBUG. Use DEBUG
only while you investigate a problem, and set it back afterwards.
Every API request is logged with its method, path, status and duration. Errors of the background jobs are logged as well, with the job that failed.
Retention
RETENTION_DAYS (90 by default) deletes old scan runs and alert logs once a day, in the
retention-cleanup task. Findings are not deleted by age: they belong to the current inventory,
and are removed when their asset is deleted or its agent is removed.
Backups
Back up the state, not the containers. The state is:
| What | Where | Notes |
|---|---|---|
| The database | SQLite file in /data, or your PostgreSQL, MySQL or MSSQL database | The inventory, users, roles, settings, mappings and the JWT signing secret. The most important part. |
| The S3 bucket | MinIO or your S3 store | SBOM documents, generated reports and uploaded agent releases. |
| The secrets | Your secret store | INITIAL_ADMIN_PASSWORD, the keys of the S3 store, the database and Redis passwords, the cluster tokens you still need. |
Redis holds locks, caches and the job queue. You do not need to back it up: losing it drops queued jobs and the cache, and the schedules run again from the server's configuration. Start any scan you were waiting for again.
The Grype vulnerability database is downloaded again, so it does not need a backup. An air-gapped
installation keeps a copy in GRYPE_DB_DIR that it needs to restore from its source.
For SQLite, copy the file while the server is stopped, or use a snapshot of the volume. Back up the database and the bucket at the same time, so that the SBOMs and the findings agree. Test a restore on another machine once.
Upgrades
- Read the release notes of the new version.
- Back up the database and the bucket.
- Upgrade the server:
docker compose up --build -dfor compose, orhelm upgradewith the new chart version for Kubernetes. Upgrade the agents after the server, with managed updates or by rolling out the new chart. See Agent updates. - Check the health endpoints, the dashboard and the agents.
Migrations run when the server starts with SQLite, and in the migration job before the pods start with PostgreSQL, MySQL or MSSQL. Each migration is applied once and recorded in the database, so a restart never runs one twice. Keep the previous version's image until the new one has run for a day, in case you need to roll back: a migration is not undone by rolling back the image, so restore the database backup if you roll back across a migration.
Air-gapped installations
Set GRYPE_DB_UPDATE=false. Copy the Grype database into GRYPE_DB_DIR from a machine that can reach
the internet, and copy it again whenever you want new vulnerability data. Set RELEASE_MANIFEST_URL to
an empty value, and install the agents with their binaries instead of the install command. The agents only need
to reach the server; the server does not need to reach the agents.