← mohdarif.com
Kubernetes Security Series · Part 2 of 3

Supply Chain Security

The images you run are built from code you did not write. Check what is inside them before they reach the cluster.

In Part 1, the Deployment pointed at an image: registry.example.com/web:1.4.2. Kubernetes runs whatever that name points to and asks no questions. This part is about what is inside that image, where it is stored, and how it is deployed.

1. What "supply chain" means

It is everything that ends up in what you run: the libraries your code uses, the base image, the registry that stores the image, and the YAML that deploys it. An attacker only needs one weak link. Your own code can be perfect and you can still run someone else's bug.

Log4Shell (2021) is the best-known example. One logging library had a remote code execution flaw, and it was inside thousands of images, often several layers down. Most teams could not quickly answer "which of our images contain it?" That question is a big part of this post.

STAGEDependenciesBase imageRegistryManifests(YAML)ClusterWHAT GOES WRONGKnown CVEs,typosquatted orhijacked packagesOld, bloatedimage, dev toolsand secrets insidePublic images youdid not check,tags overwrittenRoot, privileged,no limits,unsafe Helm chartAnything thatpasses throughgets runWHAT HELPSLockfiles,small list,Trivy / GrypeSlim base,multi-stage,scan + SBOM (Syft)Private registry,immutable tags,deploy by digestkube-linter,kubesec,trivy configAdmission policy:approved registryonly
Figure 1. Where things can go wrong between your dependencies and the running cluster, and what helps at each stage.

2. Dependencies

Most code in an image is not yours. It comes from npm, pip, Maven, Go modules and OS packages. What goes wrong:

What helps: commit lockfiles and install from them (npm ci, pip install --require-hashes) so you always get the versions you reviewed, keep the dependency list small, and scan it regularly (next sections).

3. The base image

A full OS image carries a shell, a package manager and hundreds of packages your app never uses. Each one is attack surface and a possible CVE. A smaller base image (slim, Alpine, distroless) has fewer.

# stage 1: build and test, with all the tools
FROM node:22 AS build
WORKDIR /app
COPY . .
RUN npm ci && npm test && npm run build

# stage 2: the image you ship, with runtime packages only
FROM node:22-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY package*.json ./
RUN npm ci --omit=dev
USER node
CMD ["node", "dist/server.js"]

Tags can change, digests cannot. web:1.4.2 is only a label, and whoever can push to the registry can point it at different content. A digest is a hash of the image, so it always means one image.

image: registry.example.com/web:1.4.2                 # can change
image: registry.example.com/web@sha256:3f1a...c9e2    # cannot change

Deploy by digest. Never use latest.

4. Tools: Syft, Grype and Trivy

Three tools cover most of what you need for images.

ToolWhat it doesExample
SyftLists everything inside an image: every package and version. This list is the SBOM.syft web:1.4.2 -o cyclonedx-json > sbom.json
GrypeChecks an image, or an SBOM, against vulnerability databases and reports known CVEs.grype sbom:./sbom.json --fail-on high
TrivyAll in one: finds CVEs in images, finds secrets, and checks config files such as Dockerfiles and Kubernetes YAML.trivy image --severity HIGH,CRITICAL web:1.4.2
CHECKS THE IMAGE Imageweb:1.4.2 Syftlists packages SBOMthe list Grypefinds known CVEs Reportfix or fail the build CHECKS THE YAML Manifestor Helm chart kube-linterbest-practice rules kubesecsecurity score Findingsfix before apply Trivy: image CVEs, secrets and YAML checks in one tool
Figure 2. Which tool checks what. Syft and Grype work as a pair. Trivy covers both sides in one tool.

Syft and Grype are made by the same project (Anchore) and are meant to work together: Syft makes the list, Grype checks it. Trivy does both in one command, which is simpler to start with. Many teams use one pair and pick whichever is easier to run.

Scanning has limits. It only finds known problems, and only the ones in the database today. A clean scan does not mean an image is safe. An image that was clean last month can have a critical CVE now, so rescan what is already running, not only new builds. Also agree on how fast you will fix each severity, or the report just becomes a list nobody reads.

5. SBOMs, and the dev and test trap

A Software Bill of Materials (SBOM) is the list Syft produces. Its value shows up when the next Log4Shell lands: you search your SBOMs and know in minutes which images are affected. The common formats are SPDX and CycloneDX.

There are two traps, and both come from development and test dependencies.

Trap 1: an SBOM from source code over-reports

Your project lists libraries for building and testing (test frameworks, linters, bundlers) next to the ones the app needs to run. If you generate the SBOM from the source folder or lockfile, those can show up even though they never reach the final image. You then chase CVEs that cannot affect production, and the real findings get lost in the noise. Whether dev dependencies are included depends on the tool and the language, so check what your output actually contains.

# from source: may include dev and test packages
syft dir:. -o cyclonedx-json > sbom-source.json

# from the built image: only what ships
syft web:1.4.2 -o cyclonedx-json > sbom-image.json

Generate the SBOM from the final image. It describes what is actually deployed.

Trap 2: dev dependencies end up in the image

The reverse problem. If the Dockerfile runs a plain npm install or installs requirements-dev.txt in the final stage, test tools ship to production. They add CVEs and give an attacker more tools. This is what the multi-stage build above prevents: tests run in stage 1, and stage 2 installs runtime packages only (--omit=dev).

A quick check: compare the two SBOMs above. Packages that appear in the source one but not in the image one are your dev and test dependencies. Packages that appear in the image one but should not be there (a test framework, a compiler) mean something leaked into the final stage.

6. Registry: private or public

The registry stores your images until a node pulls them. Where you pull from, and where you push to, both matter.

Public registry (Docker Hub, public repos)Private registry
Who can pushAnyone can publish images, including ones with names that look like official onesOnly people and systems you allow
What you trustWhatever the publisher put in. It can change without noticeOnly what you copied in and checked
Your own imagesAnyone can pull them and read the layers, including any secrets left insideAccess controlled
Scanning and controlNot in your handsScan on push, set retention, use immutable tags
AvailabilityPull limits and outages can break your deploymentsUnder your control

The practical setup:

7. Check your YAML: kube-linter and kubesec

A safe image can still be deployed in an unsafe way. The Part 1 manifest had no securityContext and no resource limits. Two small tools check manifests for this before you apply them. They read YAML and do not look at images.

ToolWhat it doesExample
kube-linterChecks manifests and Helm charts against a list of best practices: running as root, privileged containers, missing resource limits, the latest tag, writable root filesystem, and more.kube-linter lint ./k8s/
kubesecGives a manifest a security score and lists what would improve it, such as setting runAsNonRoot or dropping capabilities.kubesec scan deployment.yaml

Trivy can also check manifests with trivy config ./k8s/, so if you already use it, that may be enough. Run these tools every time a manifest changes, and fix findings before the manifest reaches the cluster. Review Helm charts from the internet the same way. A chart can create privileged Pods and broad RBAC rules, so read what it will create with helm template before installing.

8. Enforce it in the cluster

Scanning only helps if the cluster refuses what fails. Part 1 showed that every request passes through admission before it is stored. A policy engine such as Kyverno or OPA Gatekeeper can use that step to enforce simple rules:

Start in audit mode to see what would be blocked, fix the violations, then switch to enforce.

Where to start

  1. Use a small base image, a multi-stage build, and a non-root user.
  2. Scan every image with Trivy, or with Syft and Grype. Fail the build on critical findings.
  3. Generate the SBOM from the final image and keep it with the release.
  4. Move to a private registry and pull base images through it. Deploy by digest.
  5. Run kube-linter and kubesec on your manifests, then enforce the basics with an admission policy.

Next

Something will still get through: a zero-day, a stolen credential, a policy mistake. Part 3 covers how to see it happen, with audit logs, cluster and application logging, and runtime detection inside containers.

Part 3: Monitoring, logging and runtime security →

References