Kubernetes Secrets Management in 2026: External Secrets, Vault and Interview Questions

Kubernetes secrets management remains one of the most critical security challenges in container orchestration. Native Kubernetes Secrets store sensitive data as base64-encoded values, which provides no encryption at rest by default and creates security risks that interviewers frequently probe during DevOps hiring processes.

Why Native Kubernetes Secrets Fall Short

The built-in Secret resource stores data in etcd with base64 encoding. This encoding is reversible with a single command:

# decoding-secret.sh
# Decode any Kubernetes secret value instantly
kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d

Base64 is not encryption. Anyone with cluster access can read secret values. The Kubernetes documentation explicitly warns that Secrets are "not encrypted by default" and recommends enabling encryption at rest.

Three primary limitations affect production deployments:

1. No audit trail: Native Secrets provide no logging of who accessed what value and when
2. No rotation mechanism: Changing a secret requires manual intervention and pod restarts
3. GitOps incompatibility: Storing encrypted secrets in Git still exposes the encryption key management problem

External Secrets Operator Architecture

External Secrets Operator (ESO) bridges Kubernetes with external secret management systems. Released as a CNCF Sandbox project in 2023 and reaching v0.10 in 2026, ESO supports AWS Secrets Manager, HashiCorp Vault, Google Secret Manager, Azure Key Vault, and 15 other backends.

The architecture involves three custom resources:

# secret-store.yaml
# ClusterSecretStore defines the connection to Vault
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets"
            namespace: "external-secrets"

This ClusterSecretStore configures ESO to authenticate with Vault using Kubernetes service account tokens. The kubernetes auth method validates the service account JWT against the cluster's TokenReview API.

# external-secret.yaml
# ExternalSecret fetches and syncs the actual secret data
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-secret
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/production/database
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/production/database
        property: password

ESO polls Vault every hour (refreshInterval: 1h) and updates the Kubernetes Secret automatically. The creationPolicy: Owner ensures the Secret gets deleted when the ExternalSecret is removed.

HashiCorp Vault Integration Patterns

Vault provides dynamic secrets, automatic rotation, and comprehensive audit logging. The Kubernetes auth method, documented in the Vault Kubernetes Auth documentation, enables pods to authenticate without distributing long-lived credentials.

Setting up Vault authentication requires three steps:

# vault-k8s-auth.sh
# Enable Kubernetes auth method in Vault
vault auth enable kubernetes

# Configure Vault to validate tokens against the Kubernetes API
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create a role that binds service accounts to policies
vault write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=readonly-secrets \
  ttl=1h

This configuration binds the external-secrets service account to a Vault policy. The TTL limits token validity, forcing re-authentication and reducing the blast radius of compromised tokens.

For production deployments, the policy should follow least-privilege principles:

# readonly-secrets.hcl
# Vault policy granting read-only access to specific paths
path "secret/data/production/*" {
  capabilities = ["read"]
}

path "secret/metadata/production/*" {
  capabilities = ["list"]
}

This policy grants read access only to the production secrets path. Teams managing different environments receive separate policies with corresponding path restrictions.

Secret Rotation Without Downtime

Automatic rotation prevents secrets from becoming stale attack vectors. ESO handles the Kubernetes side, but the application must reload secrets without restarting.

Two approaches exist for zero-downtime rotation:

Volume-mounted secrets with file watching:

# deployment-volume-secrets.yaml
# Mount secrets as files that update automatically
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  template:
    spec:
      containers:
        - name: api
          image: api:v2.1.0
          volumeMounts:
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: db-secret

Kubernetes updates mounted secret files within the kubelet sync period (default 1 minute). The application reads credentials from /etc/secrets/password on each database connection, picking up new values without restart.

Reloader for environment variable secrets:

Applications using environment variables require pod restarts. Stakater Reloader automates this:

# deployment-with-reloader.yaml
# Annotation triggers rolling restart when secret changes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - name: api
          envFrom:
            - secretRef:
                name: db-secret

Reloader watches for Secret updates and performs rolling restarts, maintaining availability throughout the rotation.

Interview Questions on Kubernetes Secrets

DevOps interviews consistently test secrets management understanding. These questions appear across junior to senior levels:

Q: What encoding do Kubernetes Secrets use, and why is this insufficient for security?

Secrets use base64 encoding, which is a reversible transformation, not encryption. Any user with get permissions on Secrets can decode values. Security requires encryption at rest (enabling etcd encryption) and restricting RBAC permissions. For deeper Kubernetes concepts, review the Kubernetes interview fundamentals.

Q: How would you prevent secrets from appearing in Git repositories when using GitOps?

External Secrets Operator stores only references to secrets in Git, not the values themselves. The ExternalSecret manifest contains the path to the secret in Vault or AWS Secrets Manager, while the actual secret never touches the repository. Sealed Secrets from Bitnami offers an alternative approach using asymmetric encryption.

Q: Explain the difference between SecretStore and ClusterSecretStore.

SecretStore is namespace-scoped: ExternalSecrets in the same namespace can reference it. ClusterSecretStore is cluster-wide: any namespace can reference it. Use ClusterSecretStore when multiple teams share a single Vault instance, and SecretStore when each namespace has isolated secret backends.

Q: A pod cannot access its secrets after deployment. How would you troubleshoot?

Start with the ExternalSecret status:

# troubleshoot-secrets.sh
# Check ExternalSecret sync status and conditions
kubectl get externalsecret database-credentials -o yaml

# Verify the Secret was created
kubectl get secret db-secret -o yaml

# Check ESO controller logs for auth errors
kubectl logs -n external-secrets deployment/external-secrets

Common failures include expired Vault tokens, incorrect secret paths, and RBAC misconfigurations in the ClusterSecretStore.

Q: How do you handle secret rotation for database credentials in production?

Vault's database secrets engine generates dynamic, short-lived credentials. Configure ESO with a refreshInterval shorter than the credential TTL. For static credentials requiring rotation, use Vault's rotation API combined with Reloader to restart pods after updates. The application should handle connection failures gracefully during the rotation window.

AWS Secrets Manager with ESO

AWS Secrets Manager integration requires IAM roles for service accounts (IRSA). This pattern eliminates static credentials entirely:

# aws-secret-store.yaml
# ClusterSecretStore for AWS Secrets Manager with IRSA
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

The service account annotation links to the IAM role:

# service-account-irsa.yaml
# Service account with IAM role annotation
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-sa
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/external-secrets

AWS automatically rotates secrets created through Secrets Manager with a Lambda function. Combined with ESO's refresh interval, secrets propagate to Kubernetes within the configured polling period.

Multi-Cluster Secret Synchronization

Organizations running multiple clusters need consistent secret distribution. Two patterns address this requirement:

Hub-and-spoke with ClusterSecretStore:

All clusters connect to a central Vault instance. Each cluster's ESO installation references the same secrets, and Vault handles access control through namespace-based policies.

Replication with Vault Enterprise:

Vault Enterprise supports performance replication across regions. Each cluster connects to its local Vault replica, reducing latency while maintaining consistency. The Vault replication documentation covers disaster recovery and performance replication modes.

For teams using ArgoCD, the GitOps deployment patterns article covers ApplicationSets that deploy ExternalSecrets across multiple clusters.

RBAC Hardening for Secrets Access

Default cluster roles grant excessive secret access. Harden RBAC by creating minimal roles:

# restricted-role.yaml
# Role allowing secret access only in specific namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames: ["db-secret", "api-key"]

The resourceNames field restricts access to explicitly listed secrets. Without this field, the role grants access to all secrets in the namespace.

Audit logging captures secret access attempts:

# audit-policy.yaml
# Kubernetes audit policy for secret operations
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["get", "list", "watch"]

This policy logs the full request and response for secret operations, enabling security teams to detect unauthorized access patterns.

Production Checklist for Kubernetes Secrets

• Enable etcd encryption at rest using the EncryptionConfiguration API resource
• Deploy External Secrets Operator v0.10+ with ClusterSecretStore pointing to Vault or cloud provider
• Configure refreshInterval below credential TTL to ensure secrets update before expiration
• Use IRSA (AWS), Workload Identity (GCP), or Kubernetes auth (Vault) instead of static credentials
• Restrict RBAC with resourceNames to limit secret access to specific named secrets
• Enable Kubernetes audit logging for secret operations with RequestResponse level
• Deploy Reloader for applications using environment variables to handle secret updates
• Test secret rotation in staging before production deployment
• Document recovery procedures for secret backend outages affecting pod scheduling