Azure Key Vault Secret Management in Kubernetes

Managing sensitive configuration data like API keys, database credentials, and certificates in Kubernetes requires a secure, scalable approach that integrates with enterprise secret management systems. This guide covers best practices for using our custom PlatformApplication operator with Azure Key Vault to securely manage secrets across your Kubernetes clusters.

Overview

Our PlatformApplication operator provides a seamless way to integrate Azure Key Vault with Kubernetes deployments. Instead of manually creating Kubernetes Secret objects or configuring External Secrets Operator, our platform automatically synchronizes secrets from Azure Key Vault and injects them as environment variables.

Why Azure Key Vault + PlatformApplication?

• Centralized Management: Single source of truth for all secrets in Azure Key Vault
• Automatic Sync: Secrets automatically sync from Key Vault to Kubernetes
• Environment Variables: Secrets are automatically injected as environment variables
• Access Control: Leverage Azure AD for fine-grained access control
• Compliance: Meet SOC 2, PCI DSS, HIPAA, and FedRAMP compliance requirements
• Zero Configuration: No need to manually configure secret mounting or environment variables

---

Azure Key Vault Integration

Our platform uses Azure Key Vault as the primary secret management solution, integrated through our custom PlatformApplication operator for seamless Kubernetes deployment.

Azure Key Vault Benefits

• Native Azure Integration: Seamless integration with Azure AD and managed identities
• Enterprise Security: FIPS 140-2 Level 2 validated hardware security modules
• Compliance: SOC 1, SOC 2, PCI DSS, HIPAA, and FedRAMP compliance
• Automatic Rotation: Built-in secret rotation policies and lifecycle management
• Access Control: Fine-grained RBAC with Azure AD integration

PlatformApplication Operator Integration

Our custom operator automatically syncs secrets from Azure Key Vault to Kubernetes, eliminating the need for manual secret management:

apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: web-application
spec:
  # ... other configuration
  secrets:
  - name: "database-credentials"    # Secret name in Azure Key Vault
  - name: "api-keys"               # Another secret from Key Vault
  - name: "tls-certificates"       # TLS certificates stored in Key Vault

How It Works

1. Secret Configuration: Define secret names in your PlatformApplication spec
2. Automatic Setup: PlatformApplication operator creates and configures Azure Key Vault
3. Secret Sync: Operator automatically syncs secrets to Kubernetes
4. Environment Injection: Secrets are automatically injected as environment variables

Important: Since developers don't have direct Kubernetes access, all Key Vault information and management is done through ArgoCD. You'll need the Azure Key Vault Officer role to write secrets to your Key Vault.

---

PlatformApplication Operator Setup

Prerequisites

• Kubernetes cluster with PlatformApplication operator installed
• Access to create PlatformApplication resources

Verify PlatformApplication Operator

# Check if operator is running
kubectl get pods -n platform-system

# Verify PlatformApplication CRD is available
kubectl get crd | grep platformapplication

Finding Your Azure Key Vault

Since our platform restricts direct Kubernetes access to administrators only, developers must use ArgoCD to manage their applications and find their Key Vault information.

Access Your Key Vault Through ArgoCD

1. Navigate to ArgoCD: Access your organization's ArgoCD instance
2. Find Your Application: Locate your PlatformApplication in the ArgoCD dashboard
3. View Resources: Click on your application to see all created resources
4. Locate Key Vault: Look for the Azure Key Vault resource that was automatically created
5. Copy Key Vault URL: The Key Vault URL will be displayed in the resource details

Required Azure Role Assignment

To write secrets to your Key Vault, you need the Azure Key Vault Officer role assigned to your user account:

# This command must be run by an Azure administrator
az role assignment create \
  --role "Key Vault Officer" \
  --assignee "your-email@company.com" \
  --scope "/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.KeyVault/vaults/{keyvault-name}"

Note: Only Azure administrators can assign this role. Contact your platform team if you need access to write secrets to your Key Vault.

Alternative: Request Secret Creation

If you don't have Key Vault Officer access, you can:

1. Submit a request to your platform team to create the secrets
2. Provide the secret values in a secure manner
3. Specify the secret names that match your PlatformApplication configuration

---

Core Concepts

PlatformApplication Operator

Our custom operator automatically manages the synchronization between Azure Key Vault and Kubernetes secrets, eliminating the need for manual External Secrets Operator configuration.

Secret Configuration

Define which secrets to sync from Azure Key Vault in your PlatformApplication:

apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: web-application
  namespace: default
spec:
  # ... other configuration
  secrets:
  - name: "database-credentials"    # Secret name in Azure Key Vault
  - name: "api-keys"               # Another secret from Key Vault
  - name: "tls-certificates"       # TLS certificates stored in Key Vault

How Secrets Are Synced

1. Secret Reference: The name field in the secrets array defines the secret name
2. Automatic Setup: PlatformApplication operator creates and configures Azure Key Vault as needed
3. Secret Sync: Operator automatically syncs secrets to Kubernetes
4. Environment Variables: Each key-value pair in the secret becomes an environment variable
5. Real-time Updates: Changes are automatically reflected in Kubernetes

Secret Structure

The PlatformApplication operator will automatically create the proper secret structure in Azure Key Vault. You just need to define the secret names in your spec:

secrets:
- name: "database-credentials"    # Operator creates this secret in Azure Key Vault
- name: "api-keys"               # Operator creates this secret in Azure Key Vault
- name: "tls-certificates"       # Operator creates this secret in Azure Key Vault

---

Common Use Cases

Database Credentials

Store database credentials in Azure Key Vault and reference them in your PlatformApplication:

apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: postgres-app
spec:
  # ... other configuration
  secrets:
  - name: "postgres-credentials"    # Secret name in Azure Key Vault

Result: The operator automatically creates the secret structure and you'll get environment variables:

• POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, POSTGRES_DB, POSTGRES_PORT

API Keys and Tokens

Manage API credentials securely:

apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: api-service
spec:
  # ... other configuration
  secrets:
  - name: "api-credentials"         # Secret name in Azure Key Vault
  - name: "webhook-tokens"         # Separate secret for webhooks

Result: The operator automatically creates the secret structure and you'll get environment variables:

• API_KEY, API_SECRET, WEBHOOK_TOKEN

TLS Certificates

Store and manage TLS certificates:

apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: secure-web-app
spec:
  # ... other configuration
  secrets:
  - name: "tls-certificates"       # Secret name in Azure Key Vault

Result: The operator automatically creates the secret structure and you'll get environment variables:

• TLS_CRT, TLS_KEY, CA_CRT

Environment-Specific Secrets

Our platform uses a Kustomize-based approach for managing environment-specific configurations. The base directory contains common configuration, while environment-specific patches customize secrets and other settings.

Directory Structure

.platform
└── kubernetes
    ├── base                    # Base configuration for all environments
    │   ├── application_customizations.yaml
    │   ├── application.yaml
    │   └── kustomization.yaml
    ├── dev                     # Development environment
    │   ├── application_patch.yaml
    │   └── kustomization.yaml
    ├── stg                     # Staging environment
    │   ├── application_patch.yaml
    │   └── kustomization.yaml
    └── prd                     # Production environment
        ├── application_patch.yaml
        └── kustomization.yaml

Base Configuration (.platform/kubernetes/base/)

# application.yaml
apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: web-application
spec:
  deployment:
    replicas: 3
    image: my-app:latest
  
  # Common secrets across all environments
  secrets:
  - name: "common-api-keys"           # Shared API keys
  - name: "common-tls-certificates"   # Shared TLS certificates

Environment-Specific Patches

# .platform/kubernetes/dev/application_patch.yaml
apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: web-application
spec:
  secrets:
  - name: "dev-database-credentials"    # Development database
  - name: "dev-api-keys"               # Development API keys
  - name: "common-api-keys"            # Include common secrets
  - name: "common-tls-certificates"    # Include common secrets

# .platform/kubernetes/prd/application_patch.yaml
apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: web-application
spec:
  secrets:
  - name: "prod-database-credentials"  # Production database
  - name: "prod-api-keys"             # Production API keys
  - name: "common-api-keys"           # Include common secrets
  - name: "common-tls-certificates"   # Include common secrets

Kustomization Files

# .platform/kubernetes/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
patches:
- path: application_patch.yaml
  target:
    kind: PlatformApplication
    name: web-application

# .platform/kubernetes/prd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
patches:
- path: application_patch.yaml
  target:
    kind: PlatformApplication
    name: web-application

Secret Naming Convention

• Common Secrets: common-* (shared across all environments)
• Development: dev-* (development-specific secrets)
• Staging: stg-* (staging-specific secrets)
• Production: prod-* (production-specific secrets)

---

Integration with Applications

Automatic Environment Variable Injection

With PlatformApplication, secrets are automatically injected as environment variables - no manual configuration required:

apiVersion: meta.p6m.dev/v1alpha1
kind: PlatformApplication
metadata:
  name: web-application
spec:
  deployment:
    replicas: 3
    image: my-app:latest
  
  # Secrets are automatically synced and injected as environment variables
  secrets:
  - name: "postgres-credentials"    # Creates: POSTGRES_USER, POSTGRES_PASSWORD, etc.
  - name: "api-credentials"        # Creates: API_KEY, API_SECRET, etc.
  - name: "tls-certificates"       # Creates: TLS_CRT, TLS_KEY, etc.
  
  # Optional: Custom environment variable configuration
  config:
    NODE_ENV: "production"
    LOG_LEVEL: "info"

Result: Your application automatically receives these environment variables:

• POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, POSTGRES_DB, POSTGRES_PORT
• API_KEY, API_SECRET, WEBHOOK_TOKEN
• TLS_CRT, TLS_KEY, CA_CRT

Helm Chart Integration

# values.yaml
externalSecrets:
  enabled: true
  secretStore:
    name: vault-backend
    kind: SecretStore
  
  secrets:
    database:
      name: postgres-credentials
      refreshInterval: "24h"
      data:
      - secretKey: POSTGRES_USER
        remoteRef:
          key: "prod/postgres/credentials"
          property: username
      - secretKey: POSTGRES_PASSWORD
        remoteRef:
          key: "prod/postgres/credentials"
          property: password
    
    api:
      name: api-credentials
      refreshInterval: "1h"
      data:
      - secretKey: API_KEY
        remoteRef:
          key: "api/credentials"
          property: key

ArgoCD Integration

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-application
spec:
  source:
    repoURL: https://github.com/company/web-app
    targetRevision: main
    path: k8s
    helm:
      values: |
        externalSecrets:
          enabled: true
          secretStore:
            name: vault-backend
            kind: SecretStore
          secrets:
            database:
              name: postgres-credentials
              refreshInterval: "24h"
              data:
              - secretKey: POSTGRES_USER
                remoteRef:
                  key: "prod/postgres/credentials"
                  property: username
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

---

Troubleshooting

Common Issues

Secret Store Connection Failures

# Check secret store status
kubectl get secretstore vault-backend -o yaml

# Check operator logs
kubectl logs -n external-secrets-system \
  deployment/external-secrets-controller

Secret Synchronization Issues

# Check external secret status
kubectl get externalsecret database-credentials -o yaml

# Verify target secret creation
kubectl get secret database-credentials -o yaml

Authentication Problems

# Check service account tokens
kubectl get serviceaccount external-secrets-sa -o yaml

# Verify RBAC permissions
kubectl auth can-i create secrets --as=system:serviceaccount:default:external-secrets-sa

Debug Commands

# Enable debug logging
kubectl patch deployment external-secrets-controller \
  -n external-secrets-system \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"manager","env":[{"name":"LOG_LEVEL","value":"debug"}]}]}}}}'

# Check operator metrics
kubectl port-forward -n external-secrets-system \
  service/external-secrets-controller 8080:8080

---

Best Practices Summary

Do's

• ✅ Use PlatformApplication operator for all sensitive configuration
• ✅ Implement automatic secret rotation
• ✅ Use RBAC to limit access to external secrets
• ✅ Monitor secret synchronization status
• ✅ Test secret rotation procedures regularly
• ✅ Use secret templates for complex configurations
• ✅ Implement proper backup and recovery procedures

Don'ts

• ❌ Don't store secrets in Git repositories
• ❌ Don't use static Kubernetes Secret objects for sensitive data
• ❌ Don't share secret store credentials across teams
• ❌ Don't skip monitoring and alerting for secret issues
• ❌ Don't forget to test migration procedures
• ❌ Don't ignore audit logs and compliance requirements

Security Checklist

• PlatformApplication operator installed and configured
• Secret store authentication configured securely
• RBAC policies implemented for secret access
• Network policies restrict secret store access
• Monitoring and alerting configured
• Backup and recovery procedures documented
• Secret rotation procedures tested
• Compliance requirements met

---

Support & Resources

Additional Resources

• External Secrets Operator Documentation
• HashiCorp Vault Kubernetes Integration
• AWS Secrets Manager Best Practices
• Kubernetes Secret Management

---

This documentation is maintained by the Platform Engineering team. Last updated: 1/6/2026