Container Security Best Practices

Containers have revolutionized how we deploy applications, but they’ve also introduced new security challenges. Today, I’m sharing the container security practices I’ve learned from securing production Kubernetes clusters running hundreds of workloads.

Why Container Security Matters

A single vulnerable container can compromise your entire infrastructure. With containers sharing the host kernel and networking, the attack surface is significant.

Recent statistics:

• 76% of container images have high or critical vulnerabilities
• Average container has 180 vulnerabilities
• 51% of organizations experienced container security incidents

Let’s fix that.

The Container Security Lifecycle

Security must be integrated at every stage:

Build → Scan → Sign → Deploy → Monitor → Update

I’ll cover best practices for each phase.

1. Secure Base Images

Use Minimal Base Images

Bad:

FROM ubuntu:latest
RUN apt-get update && apt-get install -y \
    python3 nodejs npm curl wget vim

Good:

FROM python:3.11-alpine
# Alpine is 5MB vs Ubuntu's 72MB
# Fewer packages = smaller attack surface

Image Selection Hierarchy

1. Distroless - Best security (no shell, package manager)
2. Alpine - Minimal footprint
3. Slim variants - Debian/Ubuntu slim
4. Full images - Use only when necessary

Example: Distroless Python

# Build stage
FROM python:3.11-slim as builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Runtime stage
FROM gcr.io/distroless/python3

COPY --from=builder /root/.local /root/.local
COPY app.py .

ENV PATH=/root/.local/bin:$PATH
CMD ["app.py"]

Benefits:

• No shell (prevents shell-based attacks)
• No package manager (prevents post-deployment modifications)
• Minimal CVE exposure

2. Dockerfile Security

Run as Non-Root User

# Create non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

# Switch to non-root
USER appuser

# All subsequent commands run as appuser
CMD ["python", "app.py"]

Why it matters: Root in container = root on host if container escapes.

Set Read-Only Filesystem

# In Dockerfile
VOLUME /tmp

# In Kubernetes deployment
securityContext:
  readOnlyRootFilesystem: true
volumeMounts:
  - name: tmp
    mountPath: /tmp

Drop Capabilities

securityContext:
  capabilities:
    drop:
      - ALL
    add:
      - NET_BIND_SERVICE  # Only if needed

Complete Secure Dockerfile

FROM python:3.11-alpine as builder

# Install dependencies in build stage
WORKDIR /build
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Runtime stage
FROM python:3.11-alpine

# Create non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

# Copy app and dependencies
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY --chown=appuser:appgroup app.py .

# Security settings
USER appuser
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=3s \
  CMD wget --quiet --tries=1 --spider http://localhost:8080/health || exit 1

CMD ["python", "app.py"]

3. Vulnerability Scanning

Scan Early, Scan Often

Scanning strategy:

# Local development
docker scan myapp:latest

# CI/CD pipeline
trivy image --severity HIGH,CRITICAL myapp:latest

# Registry scanning
# Use Harbor, ECR, or GCR automatic scanning

# Runtime scanning
# Use Falco, Sysdig, or Aqua Security

Trivy Integration Example

# .gitlab-ci.yml
container_scan:
  stage: security
  image: aquasec/trivy:latest
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  allow_failure: false

Exit code 1 = fail build if HIGH/CRITICAL vulnerabilities found.

Vulnerability Remediation Priority

1. Critical - Fix immediately (RCE, privilege escalation)
2. High - Fix within 7 days
3. Medium - Fix within 30 days
4. Low - Fix when convenient

4. Image Signing and Verification

Sign Images with Cosign

# Generate key pair
cosign generate-key-pair

# Sign image
cosign sign --key cosign.key myapp:v1.0.0

# Verify signature
cosign verify --key cosign.pub myapp:v1.0.0

Enforce Signed Images in Kubernetes

Using Kyverno:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: enforce
  rules:
    - name: verify-signature
      match:
        resources:
          kinds:
            - Pod
      verifyImages:
        - imageReferences:
            - "myregistry.io/*"
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      [Your public key]
                      -----END PUBLIC KEY-----

Result: Unsigned images rejected by Kubernetes.

5. Container Runtime Security

Security Contexts

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault

  containers:
    - name: app
      image: myapp:v1.0.0
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL
      resources:
        limits:
          cpu: 500m
          memory: 512Mi
        requests:
          cpu: 250m
          memory: 256Mi

Runtime Threat Detection with Falco

Falco rules (/etc/falco/rules.yaml):

- rule: Unauthorized Process
  desc: Detect processes not in whitelist
  condition: >
    spawned_process and
    container and
    not proc.name in (python, node, java)
  output: >
    Unauthorized process started
    (user=%user.name process=%proc.cmdline container=%container.name)
  priority: WARNING

- rule: Write to Non-Temp Directory
  desc: Detect writes outside /tmp
  condition: >
    open_write and
    container and
    not fd.directory in (/tmp, /var/tmp)
  output: >
    Write to protected directory
    (file=%fd.name container=%container.name)
  priority: ERROR

Alerts sent to Slack, PagerDuty, or SIEM.

6. Network Security

Network Policies

Default deny all traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Allow specific traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Service Mesh for mTLS

Istio automatic mTLS:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT

Result: All pod-to-pod traffic encrypted with mutual TLS.

7. Secrets Management

Never Hardcode Secrets

Bad:

ENV DATABASE_PASSWORD="supersecret123"

Good:

# Use Kubernetes Secrets
env:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Better:

# Use External Secrets Operator with Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: database/prod/password

Encrypt Secrets at Rest

# Enable encryption in Kubernetes
# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: [base64-encoded-key]
      - identity: {}

8. Supply Chain Security

Verify Dependencies

Python example (requirements.txt):

# Pin exact versions
flask==2.3.2
requests==2.31.0

# Include hashes
flask==2.3.2 \
    --hash=sha256:77fd4e1249d8c9923de34907236b747ced06e5467ecac1a419935c1c0870e0c8

Generate hashes:

pip hash flask==2.3.2

SBOM Generation

# Generate Software Bill of Materials
syft packages myapp:v1.0.0 -o spdx-json > sbom.json

# Scan SBOM for vulnerabilities
grype sbom:./sbom.json

Private Registry with Scanning

Harbor configuration:

# Enable automatic scanning
scanner:
  trivy:
    enabled: true
    vulnerability_type: os,library
    severity: CRITICAL,HIGH

# Prevent vulnerable images
prevent_vul:
  enabled: true
  severity: CRITICAL,HIGH

9. Monitoring and Logging

Container Logging Best Practices

# Log to stdout/stderr
CMD ["python", "-u", "app.py"]
# -u = unbuffered output

Why: Kubernetes captures stdout/stderr for centralized logging.

Security Audit Logging

# Kubernetes audit policy
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    verbs: ["create", "update", "delete", "patch"]
    resources:
      - group: ""
        resources: ["pods", "secrets"]

  - level: Metadata
    verbs: ["get", "list", "watch"]

Logs sent to: ELK stack, Splunk, or CloudWatch.

10. Compliance and Benchmarks

CIS Kubernetes Benchmark

# Run kube-bench
kube-bench run --targets master,node

# Example output:
[FAIL] 1.2.1 Ensure that the --anonymous-auth is set to false
[PASS] 1.2.2 Ensure that the --basic-auth-file is not set
[FAIL] 4.2.1 Ensure that the --profiling is set to false

Policy Enforcement with OPA/Gatekeeper

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg}] {
          not input.review.object.metadata.labels.owner
          msg := "Missing required label: owner"
        }

Real-World Example: Securing a Python API

Complete secure deployment:

# Dockerfile
FROM python:3.11-alpine as builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

FROM python:3.11-alpine
RUN addgroup -S api && adduser -S apiuser -G api
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY --chown=apiuser:api app.py .
USER apiuser
EXPOSE 8080
HEALTHCHECK CMD wget -q --spider http://localhost:8080/health || exit 1
CMD ["python", "-u", "app.py"]

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
        owner: platform-team
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
        seccompProfile:
          type: RuntimeDefault

      containers:
        - name: api
          image: myregistry.io/api:v1.0.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 256Mi

          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 30

          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

          volumeMounts:
            - name: tmp
              mountPath: /tmp

      volumes:
        - name: tmp
          emptyDir: {}

# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: database
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: TCP
          port: 53  # DNS

CI/CD Security Pipeline

# .github/workflows/security.yml
name: Container Security

on:
  push:
    branches: [main]

jobs:
  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Scan with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: 1

      - name: Scan with Snyk
        uses: snyk/actions/docker@master
        with:
          image: myapp:${{ github.sha }}
          args: --severity-threshold=high

      - name: Sign image
        run: |
          cosign sign --key ${{ secrets.COSIGN_KEY }} \
            myapp:${{ github.sha }}

      - name: Generate SBOM
        run: syft packages myapp:${{ github.sha }} -o spdx-json > sbom.json

      - name: Push to registry
        run: docker push myapp:${{ github.sha }}

Security Checklist

Build Time

• Use minimal base images (Alpine/Distroless)
• Run as non-root user
• Read-only root filesystem
• Drop all capabilities
• Multi-stage builds
• Pin dependency versions
• No hardcoded secrets

Scan Time

• Vulnerability scanning (Trivy/Snyk)
• Image signing (Cosign)
• SBOM generation
• License compliance check

Deploy Time

• Security contexts configured
• Network policies applied
• Resource limits set
• Secrets from external store
• Image signature verification

Runtime

• Runtime threat detection (Falco)
• Security audit logging
• Network traffic encrypted (mTLS)
• Monitoring and alerting
• Regular vulnerability scanning

Common Pitfalls to Avoid

1. Running as root - Always use non-root users
2. Using :latest tag - Pin specific versions
3. No resource limits - Prevents DoS attacks
4. Privileged containers - Almost never needed
5. No network policies - Default allow is dangerous
6. Secrets in images - Use external secret stores
7. No vulnerability scanning - Scan before deploy

Tools I Use

Scanning:

• Trivy - Fast, accurate vulnerability scanner
• Snyk - Comprehensive security platform
• Clair - Container registry scanner

Runtime Security:

• Falco - Runtime threat detection
• Sysdig - Container security platform
• Aqua Security - Full lifecycle protection

Policy Enforcement:

• Kyverno - Kubernetes native policy engine
• OPA Gatekeeper - Policy as code
• Admission webhooks - Custom validation

Registry:

• Harbor - Open source registry with scanning
• ECR - AWS container registry
• GCR - Google container registry

Key Takeaways

1. Security is a lifecycle concern - Not just deploy time
2. Minimal images reduce attack surface - Use Alpine/Distroless
3. Scan everything, always - Make it part of CI/CD
4. Enforce policies automatically - Use admission controllers
5. Monitor runtime behavior - Detect anomalies early
6. Never trust, always verify - Sign and verify images
7. Defense in depth - Multiple layers of security

Resources

• NIST Application Container Security Guide
• CIS Kubernetes Benchmark
• OWASP Container Security
• Kubernetes Security Best Practices

Conclusion

Container security doesn’t have to be overwhelming. Start with the basics - minimal images, non-root users, vulnerability scanning - and gradually add more sophisticated controls.

The key is making security automatic and enforceable. If your pipeline can deploy vulnerable containers, it will. Build security checks into your CI/CD, enforce policies at runtime, and monitor continuously.

Remember: A chain is only as strong as its weakest link. One vulnerable container can compromise your entire cluster.

---

Published: December 18, 2025