Locked Doors, Untrusted Keys: Securing Containers in the Wake of Leaky Vessel Vulnerabilities

Leaky Vessels is four container breakout flaws disclosed on 31 January 2024. The main one, CVE-2024-21626 (CVSS 8.6), affects runc 1.0.0-rc93 through 1.1.11 and is fixed in runc 1.1.12. Patch each node’s runtime, then use the Kyverno policies below to find vulnerable nodes and flag Pods with a /proc/self/fd working directory.

Leaky Vessels CVEs at a glance

CVE	Component	Severity (CVSS 3.1)	Affected	Fixed	Mitigation in this post
CVE-2024-21626	runc	8.6 High	1.0.0-rc93 to 1.1.11	1.1.12 (bundled in containerd 1.6.28 and 1.7.13)	Node runtime version audit; block /proc/self/fd working directories, exec arguments and image layers
CVE-2024-23651	BuildKit	8.7 High	before 0.12.5	0.12.5	Named in the node audit policy message; patch BuildKit
CVE-2024-23652	BuildKit	10.0 Critical	before 0.12.5	0.12.5	Not covered by the policies here; patch BuildKit
CVE-2024-23653	BuildKit	9.8 Critical	before 0.12.5	0.12.5	Not covered by the policies here; patch BuildKit

Scores are the CNA (GitHub) scores from the CVE records; see NVD for CVE-2024-21626 and NVD for CVE-2024-23651. The containerd fix versions come from the containerd 1.7.13 release notes. For other advisories we track, see Kyverno security advisories.

Why do image trust and patching matter for Leaky Vessels?

The recent buzz surrounding the runc vulnerabilities “Leaky Vessels” (CVE-2024-21626) is a timely reminder of two essential container security principles: image trust and comprehensive patching. Here is why both matter.

1. Trustworthy Images are Foundational: It might seem self-evident, but avoiding untrusted images is essential. My experience conducting Kubernetes security reviews highlights this as the most frequent pitfall. In the context of the runc vulnerability, pulling an image from an unreliable source could grant attackers a direct entry point into your Kubernetes cluster. Remember, the attack surface expands with every untrusted image you incorporate.
2. Patching Permeates the Entire Stack: While patching container orchestration platforms like Kubernetes is expected, don’t neglect low-level components like runc. Often overlooked due to their seemingly insignificant nature, these pieces play a critical role. In this case, runc establishes and executes the container sandbox, making its security integral. Leaving it unpatched creates a gaping hole in your defenses, nullifying the effectiveness of security measures applied elsewhere.

Adhering to these fundamental principles can significantly strengthen your container security posture. Scrutinize image sources meticulously and prioritize patching across your container ecosystem, from high-level platforms to foundational components like runc. Remember, a chain is only as strong as its weakest link. Don’t let a seemingly minor vulnerability become your downfall.

How do you find nodes running a vulnerable runc?

Suppose you are a Nirmata customer running Nirmata Control Hub or running Kyverno in production. In that case, you can use the following policy to understand better how impactful this new vulnerability will be.

The policy will check your container runtimes for the vulnerable versions and produce an actionable report:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-container-run-time
spec:
  validationFailureAction: Audit
  background: true
  rules:
  - name: container-run-time-version
    match:
      any:
      - resources:
          kinds:
          - Node
    context:
    - name: cr_version
      variable:
        jmesPath: split(request.object.status.nodeInfo.containerRuntimeVersion, '://')[1]
    - name: cr_runtime
      variable:
        jmesPath: split(request.object.status.nodeInfo.containerRuntimeVersion, '://')[0]
    validate:
      message: "Your container runtime is vulnerable to CVE-2024-21626 & CVE-2024-23651: {{cr_runtime}}:{{cr_version}}"
      deny:
        conditions:
          any:
          - key: |-
              {{ cr_runtime == 'containerd' && semver_compare(cr_version, '<= 1.6.27') || false }}
            operator: Equals
            value: true
          - key: |-
              {{ cr_runtime == 'runc' && semver_compare(cr_version, '<= 1.1.11') || false }}
            operator: Equals
            value: true

Two caveats on this policy. The node reports its container runtime version, such as the containerd version, not the runc binary that runtime calls, so treat a pass as a hint and confirm the runc version on the node. And containerd fixed the bundled runc in both 1.6.28 and 1.7.13, while the condition above only flags containerd 1.6.27 and earlier; nodes on containerd 1.7.0 to 1.7.12 need the condition extended.

How do you detect Leaky Vessels exploit attempts?

The runc advisory describes the core attack: a leaked file descriptor lets a container process start with its working directory set to a path like /proc/self/fd/7/, which points into the host filesystem. A malicious image can hide this behind an innocent-looking path, and runc exec --cwd can be abused the same way from inside a running container.

You can also detect containers trying to exploit the vulnerability by mounting /proc/self/fd/*. Note that it is recommended that you run the policy in Audit mode prior to Enforce mode, as potential false positives are expected.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-proc-self-fd-workingdir
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: no-proc-self-fd-images
      match:
        any:
        - resources:
            kinds:
              - Pod
      validate:
        message: "Mounting /proc/self/fd is not allowed CVE-2024-21626."
        pattern:
          spec:
            containers:
              - =(workingDir): "!/proc/self/fd*"
            =(initContainers):
              - =(workingDir): "!/proc/self/fd*"
            =(ephemeralContainers:
              - =(workingDir): "!/proc/self/fd*"

You can use the following to block persons from blatantly trying to exploit Leaky Vessels. It will not be able to detect a container that has hidden its attempts to abuse the vulnerability deeper in its code; the scope of detection of such an attack is beyond Kyverno.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-proc-self
spec:
  validationFailureAction: Enforce
  background: false
  rules:
  - name: container-exec-block-proc-self
    match:
      any:
      - resources:
          kinds:
          - Pod/exec
    validate:
      message: "Pod {{ request.object.metadata.namespace }}/{{ request.object.metadata.name }} cannot use /proc/self/cwd, CVE-2024-21626"
      deny:
        conditions:
          any:
          - key: "{{ request.object.spec.containers[].args[] | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true
          - key: "{{ request.object.spec.initContainers[].args[] | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true
          - key: "{{ request.object.spec.ephemeralContainers[].args[] | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true
          - key: "{{ request.object.command | contains(@, '/proc/self/cwd') }}"
            operator: Equals
            value: true

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-proc-self-mounting
spec:
  validationFailureAction: Enforce
  rules:
  - name: no-proc-self-images
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Pod {{ request.object.metadata.namespace }}/{{ request.object.metadata.name }} with images having /proc/self/fd/ in their layers are not allowed CVE-2024-21626."
      foreach:
      - list: "request.object.spec.containers"
        context: 
        - name: imageData
          imageRegistry:
            reference: "{{ element.image }}"
        deny:
          conditions:
            any:
              - key: "{{ imageData.configData.history[].created_by | contains(@, '/proc/self/fd/') }}"
                operator: Equals
                value: true
      - list: "request.object.spec.initContainers"
        context: 
        - name: imageData
          imageRegistry:
            reference: "{{ element.image }}"
        deny:
          conditions:
            any:
              - key: "{{ imageData.configData.history[].created_by | contains(@, '/proc/self/fd/') }}"
                operator: Equals
                value: true
      - list: "request.object.spec.ephemeralContainers"
        context: 
        - name: imageData
          imageRegistry:
            reference: "{{ element.image }}"
        deny:
          conditions:
            any:
              - key: "{{ imageData.configData.history[].created_by | contains(@, '/proc/self/fd/') }}"
                operator: Equals
                value: true

Why sign and verify container images after Leaky Vessels?

Container image signing is central to ensuring the authenticity and integrity of the code within your containerized applications. It works like a digital fingerprint, allowing you to verify that the image you’re deploying hasn’t been tampered with or modified. This explicitly blocks potential vulnerabilities such as Leaky Vessel by ensuring you never execute unknown code.

Signing gives you four things:

• Increased security: Signing helps prevent attackers from injecting malicious code into your images, potentially compromising your entire system.
• Enhanced trust: Signing verifies the image’s origin and publisher, ensuring you’re deploying what you intended.
• Improved compliance: Some industries and regulations mandate signed images for security and audit purposes.
• Reduced risk: Identifying and mitigating vulnerabilities becomes easier when you know precisely what code you’re running.

How does container image signing work?

The mechanism has three steps:

• Signing Key Pair: You obtain a signing key pair with a private key (kept secret) and a public key (distributed).
• Signing the Image: The private key is used to sign the container image cryptographically, creating a signature.
• Verifying the Signature: The public key and signature confirm that the image hasn’t been altered on deployment.

Popular tools for this:

• Cosign: An open-source tool based on the Sigstore standard, supporting keyless signing and various signing authorities.
• Notary: An integrated solution from Docker for signing and verifying images within their ecosystem.
• AWS Signer: A managed service from AWS for signing images stored in their Elastic Container Registry (ECR).
• Venafi CodeSign Protect: A managed X.509 and HSM service with enterprise-grade signing and verification.

What should you do first about Leaky Vessels?

If you do nothing else, please consider:

1. Scan your nodes for vulnerable runc versions: Use the provided Kyverno policy to identify nodes running vulnerable versions of containerd or runc (CVE-2024-21626 & CVE-2024-23651). Patch affected nodes immediately.
2. Audit your deployments for suspicious container behavior: Implement the Kyverno policy to detect containers attempting to mount /proc/self/fd/*, the technique used in the Leaky Vessels exploit. Run it in audit mode first to identify potential false positives.
3. Block execution of untrusted images: Enforce the Kyverno policy that checks for images containing /proc/self/fd/ in their layers, which could potentially indicate tampering.

For long-term security, use Nirmata Control Hub and Kyverno together for image security integrations, providing you with the ability to:

1. Implement image signing and verification: This ensures the authenticity and integrity of your container images, preventing vulnerabilities like Leaky Vessels. Consider tools like Cosign, Notary, AWS Signer, or Venafi CodeSign Protect.
2. Prioritize image trust: Only pull images from trusted sources and registries. Avoid untrusted images to minimize the attack surface.
3. Maintain comprehensive patching: Patch not just Kubernetes but also low-level components like runc. A chain is only as strong as its weakest link.
4. Use Nirmata Control Hub: Use these tools to enforce security policies and automate vulnerability detection and mitigation.

The same lesson about patching the whole stack came back two months later with the XZ backdoor; see XZ: a case study in open-source supply chain attacks. For driving image CVEs down over time, read Towards CVE-Free Images, and for the wider picture, software supply chain security.

To manage policies and governance easily across multiple clusters, check out what Nirmata is building at https://nirmata.com/.

Frequently asked questions

Which Docker Engine release fixes Leaky Vessels?

The Docker Engine 25.0.2 release notes list fixes for all four Leaky Vessels CVEs: CVE-2024-21626 through runc 1.1.12, and CVE-2024-23651, CVE-2024-23652 and CVE-2024-23653 through the BuildKit update. Docker Desktop and other distributions of the engine had their own patched releases, so check the version your vendor ships rather than assuming the upstream number.

Are other container runtimes such as crun affected by CVE-2024-21626?

The runc advisory checked several. crun 1.12 and youki 0.3.1 do not leak a useful file descriptor, so the attack is not exploitable on them as far as the runc maintainers could tell, and LXC 5.0.3 does not appear to leak one either. The advisory still warns that none of them verify the working directory after chdir, so a future leak could make them exploitable.

What can I do if I can't upgrade runc right away?

The runc advisory gives two workarounds. For the working-directory attacks, only permit containers and runc exec to use a process.cwd of /, since the root path cannot be swapped for a symlink. For the attacks that start from a malicious image, only let users run trusted images. Both are stopgaps; runc 1.1.12 closes the leaked file descriptors and checks the working directory itself.

Do the BuildKit Leaky Vessels CVEs affect running Kubernetes workloads?

They affect image builds. CVE-2024-23651, CVE-2024-23652 and CVE-2024-23653 need a malicious Dockerfile or BuildKit frontend to reach the build host, through cache mounts, the mount stub cleaner or the interactive containers API. All three are fixed in BuildKit 0.12.5. If you build images inside the cluster, for example in CI pods, patch BuildKit there as well as on developer machines.