GHSA-2XWH-426Q-42C5
Vulnerability from github – Published: 2026-11-09 22:43 – Updated: 2026-11-09 22:43Summary
The legacy Kubeflow Pipelines V1/raw-Argo execution cache does not separate cached results by Kubernetes namespace. In multi-user deployments where tenants share the cache service, a tenant can receive another tenant's cached outputs or seed outputs that are subsequently substituted into another tenant's step.
This affects the Argo-template cache mutating webhook and its cache-server storage path. Standard KFP V2 component pods bypass this webhook cache; the separate V2 cache implementation is outside this advisory's scope.
Affected deployments and impact
The issue requires multiple tenant namespaces sharing the legacy cache service, cache-enabled V1/raw-Argo steps, and cache entries that satisfy the configured staleness policy. The standard multi-user configuration watches pods across namespaces.
A tenant able to submit a step with the same cache-relevant template fields as another tenant's step can:
- Read cached step outputs originally produced in another namespace.
- Populate the shared cache before another tenant's matching step runs, causing that step to report the cached outputs instead of executing its intended containers.
Matching refers to the selected template fields used by the cache key, not merely an identical container image. This is an application-level cache-key collision caused by missing tenant scope, not a cryptographic SHA-256 collision. The finding establishes cross-namespace confidentiality and integrity impact; it does not establish arbitrary code execution.
CVSS rationale
Medium — CVSS 3.1: 5.4
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N
- AV:N / PR:L / UI:N: a tenant with ordinary workload-submission privileges can exercise the shared cache through network-accessible APIs. Reading an existing cached result or poisoning a subsequent automated execution requires no additional victim interaction.
- AC:L: known or shared templates can produce the same cache key deterministically. Cache-enabled execution and matching template fields describe the affected configuration; they do not establish an unavoidable narrow race or secret prerequisite.
- S:U: the established impact is disclosure or substitution of output data managed by the same cache service. Crossing tenant namespaces alone does not establish a change in security authority; execution under a separate victim authority has not been established here.
- C:L / I:L / A:N: the confirmed impact is selective cached-output disclosure and substitution. Arbitrary code execution, complete workload compromise, and a service-availability impact are not established.
Root cause
The vulnerable implementation hashes selected Argo template fields without a namespace. Cache lookups and duplicate checks use the key alone, and stored rows have no namespace ownership field. On a hit, the admission handler restores cached outputs and substitutes a dummy container for the step's containers.
The completion watcher also trusts the mutable pipelines.kubeflow.org/execution_cache_key annotation when publishing outputs. Adding a namespace to the hash alone is therefore insufficient: a tenant able to patch its own pod could supply another namespace's deterministic key. Namespace ownership must be enforced independently on cache writes and reads.
Relevant implementation files are backend/src/cache/server/mutation.go, backend/src/cache/server/watcher.go, backend/src/cache/storage/execution_cache_store.go, and backend/src/cache/model/execution_cache.go.
Fix status
The source fix merged on September 10, 2026 in PR #14359, commit 620d0cc89ae0d042a484a003f8ffe6e9a55d347e.
The merged fix:
- Derives namespaced keys from the API server's admission request namespace and selected template fields.
- Recomputes completed pods' keys from their observed namespace and
ARGO_TEMPLATE, rejecting mismatched or malformed identities rather than trusting the annotation. - Stores the observed namespace separately and scopes normal reads and duplicate detection by namespace and key.
- Adds a namespace column and composite index, leaving historical rows unowned and excluded from normal lookups.
- Rejects structurally malformed template objects without panicking.
The first fixed release is 2.18.0. Releases through 2.17.3 remain affected; the affected-version upper bound is <= 2.17.3 and the patched version is 2.18.0.
Upgrade requirements and legacy fallback
The standard cache-server Deployment supports a normal rolling upgrade from 2.17.2. Apply the upgraded manifests without manually stopping old replicas or pausing submissions. Namespace isolation is fully enforced only after all old cache-server processes, including terminating pods, have exited and every new replica uses enforce. Custom installations sharing the cache database must upgrade every writer. Mixed-version operation does not yet provide the full isolation guarantee.
Expect reduced cache reuse and increased step execution/resource use as tasks next run; there is no startup cache rebuild. Do not infer or backfill namespace ownership for historical rows. Pre-upgrade pods carrying old-format keys do not seed the new namespaced cache. Completing the upgrade does not undo previously disclosed data or reused poisoned outputs. Do not roll back to the vulnerable implementation to recover cache hits.
PR #14427 merged into release-2.18 on October 5, 2026 and standardizes the compatibility setting as KFP_SECURITY_LEGACY_CACHE_MODE=enforce|audit on the cache-server Deployment. This supersedes the original fix's ALLOW_LEGACY_CACHE_FALLBACK interface. Unset, empty, or enforce keeps full namespace isolation; invalid values prevent startup.
Explicit audit permits a genuine namespaced cache miss to reuse a historical template-only entry whose stored namespace is empty. That entry may contain another tenant's outputs, including poisoned outputs: audit accepts residual cross-tenant confidentiality and integrity risk, and is not full remediation. Scoped entries take precedence, known foreign namespaces remain excluded, and a database error does not enable legacy fallback. Legacy hits are not promoted or backfilled into namespaced rows.
The server emits a startup warning and scoped security_audit events for legacy reuse without logging cached outputs or credentials. Keep replicas configured consistently, persist the mode in deployment configuration, and return to enforce after migration. Native V2 caching is outside this control and this advisory.
See the released cache-server guide for rollout completion checks and audit-to-enforce migration. This fix does not establish an integrity boundary between users who can modify workloads within the same namespace.
Mitigation until the fixed release is deployed
Disable legacy cache reuse for affected workloads, or ensure mutually untrusted namespaces do not share the legacy cache service and storage. Restricting the webhook's namespace selector or watcher scope alone is insufficient if mutually untrusted tenants still share cached entries.
Validation
Source review identified the shared key, unscoped storage, and mutable-annotation write path. The public fix includes regressions exercising admission and completion handling, forged foreign-namespace keys, normal same-namespace reuse, malformed templates, legacy-row exclusion, and optional fallback behavior.
During development, the cache test suite and lint passed; populated and repeatable schema migration and scoped storage behavior were verified on SQLite, MySQL 8.4, and PostgreSQL 16. Negative controls removing the watcher identity check or storage namespace filter failed their corresponding regressions. These tests use fake Kubernetes clients; they are not a claim of a live multi-tenant Kubernetes exploit reproduction.
Private patch disposition
The earlier private PR #1 is closed as superseded by public PR #14359 and the released follow-ups described above. The private branch and review history are retained; this closure does not dismiss the report or change reporter credit. The advisory remains a draft pending the publication review.
{
"affected": [
{
"package": {
"ecosystem": "Go",
"name": "github.com/kubeflow/pipelines"
},
"ranges": [
{
"events": [
{
"introduced": "0"
},
{
"fixed": "0.0.0-20260910172610-620d0cc89ae0"
}
],
"type": "ECOSYSTEM"
}
]
}
],
"aliases": [
"CVE-2026-88982"
],
"database_specific": {
"cwe_ids": [
"CWE-345",
"CWE-639"
],
"github_reviewed": true,
"github_reviewed_at": "2026-11-09T22:43:33Z",
"nvd_published_at": "2026-11-05T23:17:01Z",
"severity": "MODERATE"
},
"details": "## Summary\n\nThe legacy Kubeflow Pipelines V1/raw-Argo execution cache does not separate cached results by Kubernetes namespace. In multi-user deployments where tenants share the cache service, a tenant can receive another tenant\u0027s cached outputs or seed outputs that are subsequently substituted into another tenant\u0027s step.\n\nThis affects the Argo-template cache mutating webhook and its cache-server storage path. Standard KFP V2 component pods bypass this webhook cache; the separate V2 cache implementation is outside this advisory\u0027s scope.\n\n## Affected deployments and impact\n\nThe issue requires multiple tenant namespaces sharing the legacy cache service, cache-enabled V1/raw-Argo steps, and cache entries that satisfy the configured staleness policy. The standard multi-user configuration watches pods across namespaces.\n\nA tenant able to submit a step with the same cache-relevant template fields as another tenant\u0027s step can:\n\n- Read cached step outputs originally produced in another namespace.\n- Populate the shared cache before another tenant\u0027s matching step runs, causing that step to report the cached outputs instead of executing its intended containers.\n\nMatching refers to the selected template fields used by the cache key, not merely an identical container image. This is an application-level cache-key collision caused by missing tenant scope, not a cryptographic SHA-256 collision. The finding establishes cross-namespace confidentiality and integrity impact; it does not establish arbitrary code execution.\n\n## CVSS rationale\n\n**Medium \u2014 CVSS 3.1: 5.4**\n\n`CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N`\n\n- **AV:N / PR:L / UI:N:** a tenant with ordinary workload-submission privileges can exercise the shared cache through network-accessible APIs. Reading an existing cached result or poisoning a subsequent automated execution requires no additional victim interaction.\n- **AC:L:** known or shared templates can produce the same cache key deterministically. Cache-enabled execution and matching template fields describe the affected configuration; they do not establish an unavoidable narrow race or secret prerequisite.\n- **S:U:** the established impact is disclosure or substitution of output data managed by the same cache service. Crossing tenant namespaces alone does not establish a change in security authority; execution under a separate victim authority has not been established here.\n- **C:L / I:L / A:N:** the confirmed impact is selective cached-output disclosure and substitution. Arbitrary code execution, complete workload compromise, and a service-availability impact are not established.\n\n## Root cause\n\nThe vulnerable implementation hashes selected Argo template fields without a namespace. Cache lookups and duplicate checks use the key alone, and stored rows have no namespace ownership field. On a hit, the admission handler restores cached outputs and substitutes a dummy container for the step\u0027s containers.\n\nThe completion watcher also trusts the mutable `pipelines.kubeflow.org/execution_cache_key` annotation when publishing outputs. Adding a namespace to the hash alone is therefore insufficient: a tenant able to patch its own pod could supply another namespace\u0027s deterministic key. Namespace ownership must be enforced independently on cache writes and reads.\n\nRelevant implementation files are `backend/src/cache/server/mutation.go`, `backend/src/cache/server/watcher.go`, `backend/src/cache/storage/execution_cache_store.go`, and `backend/src/cache/model/execution_cache.go`.\n\n## Fix status\n\nThe source fix merged on September 10, 2026 in [PR #14359](https://github.com/kubeflow/pipelines/pull/14359), commit [620d0cc89ae0d042a484a003f8ffe6e9a55d347e](https://github.com/kubeflow/pipelines/commit/620d0cc89ae0d042a484a003f8ffe6e9a55d347e).\n\nThe merged fix:\n\n- Derives namespaced keys from the API server\u0027s admission request namespace and selected template fields.\n- Recomputes completed pods\u0027 keys from their observed namespace and `ARGO_TEMPLATE`, rejecting mismatched or malformed identities rather than trusting the annotation.\n- Stores the observed namespace separately and scopes normal reads and duplicate detection by namespace and key.\n- Adds a namespace column and composite index, leaving historical rows unowned and excluded from normal lookups.\n- Rejects structurally malformed template objects without panicking.\n\nThe first fixed release is **[2.18.0](https://github.com/kubeflow/pipelines/releases/tag/2.18.0)**. Releases through **2.17.3** remain affected; the affected-version upper bound is `\u003c= 2.17.3` and the patched version is `2.18.0`.\n\n## Upgrade requirements and legacy fallback\n\nThe standard cache-server Deployment supports a normal rolling upgrade from 2.17.2. Apply the upgraded manifests without manually stopping old replicas or pausing submissions. Namespace isolation is fully enforced only after all old cache-server processes, including terminating pods, have exited and every new replica uses `enforce`. Custom installations sharing the cache database must upgrade every writer. Mixed-version operation does not yet provide the full isolation guarantee.\n\nExpect reduced cache reuse and increased step execution/resource use as tasks next run; there is no startup cache rebuild. Do not infer or backfill namespace ownership for historical rows. Pre-upgrade pods carrying old-format keys do not seed the new namespaced cache. Completing the upgrade does not undo previously disclosed data or reused poisoned outputs. Do not roll back to the vulnerable implementation to recover cache hits.\n\n[PR #14427](https://github.com/kubeflow/pipelines/pull/14427) merged into `release-2.18` on October 5, 2026 and standardizes the compatibility setting as **`KFP_SECURITY_LEGACY_CACHE_MODE=enforce|audit` on the cache-server Deployment**. This supersedes the original fix\u0027s `ALLOW_LEGACY_CACHE_FALLBACK` interface. Unset, empty, or `enforce` keeps full namespace isolation; invalid values prevent startup.\n\nExplicit `audit` permits a genuine namespaced cache miss to reuse a historical template-only entry whose stored namespace is empty. That entry may contain another tenant\u0027s outputs, including poisoned outputs: audit accepts residual cross-tenant confidentiality and integrity risk, and is not full remediation. Scoped entries take precedence, known foreign namespaces remain excluded, and a database error does not enable legacy fallback. Legacy hits are not promoted or backfilled into namespaced rows.\n\nThe server emits a startup warning and scoped `security_audit` events for legacy reuse without logging cached outputs or credentials. Keep replicas configured consistently, persist the mode in deployment configuration, and return to `enforce` after migration. Native V2 caching is outside this control and this advisory.\n\nSee the [released cache-server guide](https://github.com/kubeflow/pipelines/blob/2.18.0/backend/src/cache/README.md#namespace-isolation-and-upgrades) for rollout completion checks and audit-to-enforce migration. This fix does not establish an integrity boundary between users who can modify workloads within the same namespace.\n\n## Mitigation until the fixed release is deployed\n\nDisable legacy cache reuse for affected workloads, or ensure mutually untrusted namespaces do not share the legacy cache service and storage. Restricting the webhook\u0027s namespace selector or watcher scope alone is insufficient if mutually untrusted tenants still share cached entries.\n\n## Validation\n\nSource review identified the shared key, unscoped storage, and mutable-annotation write path. The public fix includes regressions exercising admission and completion handling, forged foreign-namespace keys, normal same-namespace reuse, malformed templates, legacy-row exclusion, and optional fallback behavior.\n\nDuring development, the cache test suite and lint passed; populated and repeatable schema migration and scoped storage behavior were verified on SQLite, MySQL 8.4, and PostgreSQL 16. Negative controls removing the watcher identity check or storage namespace filter failed their corresponding regressions. These tests use fake Kubernetes clients; they are not a claim of a live multi-tenant Kubernetes exploit reproduction.\n\n### Private patch disposition\n\nThe earlier [private PR #1](https://github.com/kubeflow/pipelines-ghsa-2xwh-426q-42c5/pull/1) is closed as superseded by [public PR #14359](https://github.com/kubeflow/pipelines/pull/14359) and the released follow-ups described above. The private branch and review history are retained; this closure does not dismiss the report or change reporter credit. The advisory remains a draft pending the publication review.",
"id": "GHSA-2xwh-426q-42c5",
"modified": "2026-11-09T22:43:33Z",
"published": "2026-11-09T22:43:33Z",
"references": [
{
"type": "WEB",
"url": "https://github.com/kubeflow/pipelines/security/advisories/GHSA-2xwh-426q-42c5"
},
{
"type": "ADVISORY",
"url": "https://nvd.nist.gov/vuln/detail/CVE-2026-88982"
},
{
"type": "WEB",
"url": "https://github.com/kubeflow/pipelines/pull/14359"
},
{
"type": "WEB",
"url": "https://github.com/kubeflow/pipelines/commit/620d0cc89ae0d042a484a003f8ffe6e9a55d347e"
},
{
"type": "PACKAGE",
"url": "https://github.com/kubeflow/pipelines"
},
{
"type": "WEB",
"url": "https://github.com/kubeflow/pipelines/releases/tag/2.18.0"
}
],
"schema_version": "1.4.0",
"severity": [
{
"score": "CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N",
"type": "CVSS_V3"
}
],
"summary": "Kubeflow Pipelines: Cross-tenant execution-cache poisoning / output substitution via a namespace-blind cache key in the cache webhook (multi-user)"
}
Drag nodes to arrange them. Double-click a related vulnerability to open it, or a source to visit its link. The Physics button on the left tunes the layout and the simulation; the legend at the bottom left hides or shows nodes by kind and by sighting type.
Sightings
| Author | Source | Type | Date | Other |
|---|
Nomenclature
- Seen: The vulnerability was mentioned, discussed, or observed by the user.
- Confirmed: The vulnerability has been validated from an analyst's perspective.
- Published Proof of Concept: A public proof of concept is available for this vulnerability.
- Exploited: The vulnerability was observed as exploited by the user who reported the sighting.
- Patched: The vulnerability was observed as successfully patched by the user who reported the sighting.
- Not exploited: The vulnerability was not observed as exploited by the user who reported the sighting.
- Not confirmed: The user expressed doubt about the validity of the vulnerability.
- Not patched: The vulnerability was not observed as successfully patched by the user who reported the sighting.
Vulnerability timeline
Sightings and recorded dates, grouped by day · UTC
How durations are calculated
Following the VTEM framework (Black Hat USA 2015), time to patch starts when the vendor or a coordinator was informed, total time to patch at discovery, and disclosure to patch at public disclosure; vendor response time runs from the report to its acknowledgement. Time to exploit and time unpatched start at the earliest discovery or report, falling back to public disclosure. Unpatched time ends at the first recorded patch, or the calculation time when no patch is recorded. Exploitation to patch shows the interval between those two events; a published proof of concept is a public exploit, not exploitation.
Each result shows its source dates. A negative interval means the end event came first; when date-only inputs leave the order open, the result says so. Date-only inputs are approximate. The earliest evidence wins: KEV report dates and sightings are labelled as such. Records describing another vulnerability are listed but never calculated from, and a PoC alone does not establish exploitation. Missing patch evidence does not prove that no patch exists. Filters below change the event list only.
Loading timeline…
The approach is described in our paper Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion.
Browse all ATT&CK techniques and the vulnerabilities related to each.
Related by attack behaviour
Vulnerabilities whose description is nearest to this one in the vector space of the CIRCL/vulnerability-attack-technique-biencoder model. This is a similarity search over the bi-encoder space (plain cosine), not a classification, and it has no measured accuracy.