Metrics#

Vulnerability-Lookup can expose a Prometheus scrape endpoint at /metrics, reporting the health of its datastores, the state of its derived indexes, and — where the EUVD feeder runs — the integrity of the EUVD identifier space.

Enabling it#

The endpoint is off by default and answers 404 until an operator turns it on. The figures it reports are internal — memory, disk, key counts, index coverage — so on a public instance enabling it is a change of posture rather than a new feature.

Be specific about what that posture change publishes. With no METRICS_TOKEN, enabled and publicly readable are the same state, and anyone who can reach the site can read the Kvrocks and Redis version strings, the memory and disk ceilings those backends are running against and how much of each is used, the RocksDB key estimates, and how far behind the derived indexes are. No user, account or vulnerability data is in the payload — PostgreSQL is never queried and no CVE, EUVD, vendor or product identifier appears, by the cardinality rule below — but the datastore inventory alone is more than most public instances want to hand out. Either keep the endpoint reachable only from inside your network, or set a token.

# config/website.py
WEB_MODULES = {
    "vulnerability_disclosure": False,
    "metrics": True,
}

# Optional. Leave as None when the endpoint is only reachable from inside your
# network; set it when your scraper comes over a path you do not control.
METRICS_TOKEN = None

With a token configured, the scrape needs a bearer header:

# prometheus.yml
scrape_configs:
  - job_name: vulnerability-lookup
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials: <METRICS_TOKEN>
    static_configs:
      - targets: ["vulnerability-lookup.example.org"]

Check it with curl; no Prometheus is needed to develop against it.

curl -s http://127.0.0.1:10001/metrics | head

Two things the endpoint does on its own. It answers Cache-Control: no-store, because a cached scrape freezes every gauge and quietly turns a freshness alert into a statement about a half-hour-old timestamp. And it is rate limited to 30 requests per minute per client address: a scrape is around forty round-trips against the same Kvrocks that serves the site, so an unmetered endpoint is a cheap amplification lever. Thirty a minute is a two-second scrape interval — well below the limit for any real scraper, including an HA pair arriving from one proxy address.

What is reported#

Family	What it answers
`vulnerability_lookup_build_info`	Which version served the scrape.
`vulnerability_lookup_datastore_up`	Whether each backend answered `INFO`.
`vulnerability_lookup_datastore_memory_` / `_disk_` / `_db_size_*`	Resource use of the Kvrocks storage and the Redis cache, labelled `datastore="storage"` / `"cache"`.
`vulnerability_lookup_datastore_keys_estimated`	RocksDB’s per-column-family key estimate.
`vulnerability_lookup_datastore_pending_compaction_bytes`	Bytes awaiting compaction; sustained growth precedes a write stall.
`vulnerability_lookup_index_state_*`	Whether the derived state indexes still account for the corpus — the same check as `flask --app website.app index_health`.
`vulnerability_lookup_source_last_update_timestamp_seconds`	When each source last imported.
`vulnerability_lookup_source_freshness_threshold_seconds`	How old that may get before it counts as stale.
`vulnerability_lookup_source_records`	Records indexed per source; its increase is the ingestion rate.
`vulnerability_lookup_source_feeder_enabled`	Whether a feeder for the source is enabled in `modules.cfg`.
`vulnerability_lookup_http_requests_total`	Requests served, by endpoint and status class.
`vulnerability_lookup_http_request_duration_seconds`	Request duration histogram, by endpoint.
`vulnerability_lookup_euvd_*`	EUVD identifier counts and integrity, only on an instance that mints them.
`vulnerability_lookup_metrics_collection_failed`	1 for any collector that raised on this scrape.

Two things are worth knowing about the datastore figures:

A field a given server does not report is absent rather than zero. Kvrocks has no used_memory and Redis has no disk_capacity; a zero would read as a measurement.
vulnerability_lookup_datastore_keys is only published where the backend counts keys exactly. Kvrocks fills db0:keys from a DBSIZE SCAN and reports zero until one has run, so the RocksDB estimate is exposed instead.

Ingestion freshness#

The alert that matters most is “data stopped arriving”, and it is one expression because the threshold is published next to the measurement:

time() - vulnerability_lookup_source_last_update_timestamp_seconds
  > on(source) vulnerability_lookup_source_freshness_threshold_seconds

The rule for what counts as stale therefore lives in config/website.py (METRICS_FRESHNESS_THRESHOLDS), not in a PromQL expression that nobody updates when a feeder changes cadence. Filter on vulnerability_lookup_source_feeder_enabled == 1 to leave deliberately disabled sources alone.

Everything here is labelled by the name a feeder stores under — the one in last_updates and index:<source> — which is not always the modules.cfg section name. [feeder:cvelist] runs the feeder that stores as cvelistv5, and [feeder:ossf_malicious] the one that stores as ossf_malicious_packages. The source label and the METRICS_FRESHNESS_THRESHOLDS keys both use the stored name; vulnerability_lookup_source_feeder_enabled bridges the two for you, so a threshold written against a section name is the mistake to watch for. redis-cli -p 10002 HKEYS last_updates lists exactly the names to use.

One feeder is not always one source, either. gcve_vl pulls from every GNA the instance federates with and records one entry per GNA — gna-1, gna-1337 and so on, named at run time — so it declares the prefix they share and all of them report source_feeder_enabled = 1 while it is enabled. A source that no configured feeder writes reports 0: the local instance name and any CNA publishing through the API are fed by the site itself rather than by modules.cfg, and the alert leaves them alone deliberately.

For the ingestion rate — the “nothing has come in” signal — use the record count:

delta(vulnerability_lookup_source_records{source="cvelistv5"}[2h]) == 0

Two sources of silence to know about, both reported rather than hidden:

vulnerability_lookup_sources_never_imported counts sources still carrying the never-imported placeholder date. They get no timestamp series, because reporting one as 126 years stale is how an alert gets muted.
vulnerability_lookup_sources_without_timestamp counts sources whose last_updates value is not a time at all. The git-backed feeders store a commit SHA there, so freshness cannot be measured for them by any means — worth knowing rather than discovering during an incident.

Request instrumentation#

Turning WEB_MODULES["metrics"] on also starts timing requests: a duration histogram and a request counter, both labelled by Flask’s endpoint name (home_bp.index), never by path — a path label would mint a series per CVE id.

# p95 latency per endpoint
histogram_quantile(0.95, sum by (endpoint, le) (
  rate(vulnerability_lookup_http_request_duration_seconds_bucket[5m])))

# error rate
sum(rate(vulnerability_lookup_http_requests_total{status="5xx"}[5m]))
  / sum(rate(vulnerability_lookup_http_requests_total[5m]))

The counters live in the cache Redis rather than in the worker process, so they are the same number in every gunicorn worker and survive a restart. Scrapes of /metrics are excluded — otherwise it would be the busiest endpoint on any quiet instance. This measures the application; it does not replace an external prober, which is the only thing that can see the network path in front of it.

EUVD integrity#

The EUVD gauges come from a snapshot rather than from the scrape, because the check walks the whole identifier space. Run it on a schedule:

poetry run euvd --integrity-scan

It counts EUVD records carrying more than one CVE, records carrying none, disagreements between a <cve>:euvd pointer and the EUVD that links it, members of index:euvd that are not EUVD identifiers at all, and published CVEs with no EUVD. It exits non-zero when an invariant is broken — unminted CVEs are reported but do not fail the run, since that is ingestion lag rather than corruption. When that last count does not fall on its own, poetry run euvd --backfill-unminted mints what is missing.

vulnerability_lookup_euvd_integrity_snapshot_age_seconds reports how old the snapshot is, and -1 when no scan has ever run. A check that quietly stops running should not look like a system with nothing to report.

Adding a metric#

Two rules, both load-bearing.

Read shared state; do not accumulate. The website runs under gunicorn with many workers and a scrape reaches exactly one of them, so a process-local counter reports one worker’s share of the truth, chosen at random. Every collector here reads Kvrocks, PostgreSQL or a snapshot, which is the same number in every worker. Anything that has to count events belongs in the cache Redis that all workers share, and is read back like any other shared state.

Respect the cardinality rule. No label whose value space is unbounded by the data: no vendor (61,000 values on a production corpus), product, cve_id (381,385), URL or free-text label. Prometheus keeps a series per label combination for the whole retention window. Safe labels are the ones we control — source, state, datastore, column_family. Distributions belong in histogram buckets, and leaderboards belong in /api/stats/*, which answers them with a single ZREVRANGE.

Exporting only a top N with the name as a label is not a way around this: the names shift, so each change mints new series and abandons old ones, which degrades a Prometheus faster than steady high cardinality does.

Metrics

Contents