External Secret Managers

External secret managers provide centralized, auditable secret storage with automated rotation and fine-grained access control. This guide shows how to integrate Expanso Edge with popular enterprise secret management systems.

Assumption

This guide assumes your secret manager (Vault, AWS Secrets Manager, etc.) is already provisioned and configured. Examples focus on fetching and using secrets, not on secret manager setup.

---

Why Use External Secret Managers?

Benefits

• Automated Rotation: Credentials expire and renew without manual intervention
• Audit Trails: Track who accessed which secrets and when
• Centralized Management: Single source of truth for all credentials
• Dynamic Secrets: Generate short-lived credentials on demand
• Access Control: Fine-grained policies per service/user
• Compliance: Meet SOC 2, HIPAA, PCI-DSS requirements

When to Use

• Production deployments with compliance requirements
• Multi-environment setups (dev/staging/prod)
• Large-scale deployments (10+ edge nodes)
• Regulated industries (finance, healthcare)
• Environments requiring secret rotation

---

Integration Patterns

All external secret managers follow a similar pattern:

1. Fetch secrets at edge agent startup or runtime
2. Inject as environment variables or write to local files
3. Reference in pipeline config using interpolation (${VAR_NAME})

---

HashiCorp Vault

Architecture

┌─────────────────┐
│  Vault Server   │
│  (Pre-existing) │
└────────┬────────┘
         │ Vault Agent (sidecar)
         │ fetches secrets
         ↓
┌─────────────────┐
│  Edge Agent     │
│  (reads from    │
│   env vars)     │
└─────────────────┘

Using Vault Agent Sidecar

Vault Agent runs alongside the edge agent, automatically fetching and refreshing secrets.

Vault Agent config:

vault-agent.hcl

pid_file = "/tmp/vault-agent.pid"

vault {
  address = "https://vault.example.com:8200"
}

auto_auth {
  method {
    type = "kubernetes"  # or "aws", "gcp", "azure"
    config = {
      role = "expanso-edge"
    }
  }

  sink {
    type = "file"
    config = {
      path = "/etc/expanso/secrets.env"
      mode = 0600
    }
  }
}

template {
  source      = "/etc/vault/templates/pipeline.env.tpl"
  destination = "/etc/expanso/pipeline.env"
  perms       = "0600"
}

Template file:

pipeline.env.tpl

{{ with secret "secret/data/expanso/pipeline" }}
KAFKA_USERNAME={{ .Data.data.kafka_username }}
KAFKA_PASSWORD={{ .Data.data.kafka_password }}
API_TOKEN={{ .Data.data.api_token }}
DB_PASSWORD={{ .Data.data.db_password }}
{{ end }}

Docker Compose example:

docker-compose.yml

version: '3.8'
services:
  vault-agent:
    image: hashicorp/vault:latest
    command: agent -config=/vault/config/vault-agent.hcl
    volumes:
      - ./vault-agent.hcl:/vault/config/vault-agent.hcl:ro
      - ./pipeline.env.tpl:/etc/vault/templates/pipeline.env.tpl:ro
      - secrets:/etc/expanso
    environment:
      VAULT_ADDR: https://vault.example.com:8200

  expanso-edge:
    image: ghcr.io/expanso-io/expanso-edge:latest
    command: run --config /etc/expanso/pipeline.yaml
    volumes:
      - ./pipeline.yaml:/etc/expanso/pipeline.yaml:ro
      - secrets:/etc/expanso
    depends_on:
      - vault-agent
    # Reload env vars on change
    env_file:
      - /etc/expanso/pipeline.env

volumes:
  secrets:

Using Vault Init Container (Kubernetes)

Kubernetes example:

Create the ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: expanso-edge
  namespace: expanso-system

Create the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expanso-edge
  namespace: expanso-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: expanso-edge
  template:
    metadata:
      labels:
        app: expanso-edge
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-pipeline.env: "secret/data/expanso/pipeline"
        vault.hashicorp.com/agent-inject-template-pipeline.env: |
          {{- with secret "secret/data/expanso/pipeline" -}}
          export KAFKA_USERNAME="{{ .Data.data.kafka_username }}"
          export KAFKA_PASSWORD="{{ .Data.data.kafka_password }}"
          export API_TOKEN="{{ .Data.data.api_token }}"
          {{- end }}
        vault.hashicorp.com/role: "expanso-edge"
    spec:
      serviceAccountName: expanso-edge
      containers:
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        command:
          - /bin/sh
          - -c
          - |
            source /vault/secrets/pipeline.env
            expanso-edge run --config /etc/expanso/pipeline.yaml
        volumeMounts:
          - name: pipeline-config
            mountPath: /etc/expanso
            readOnly: true
      volumes:
      - name: pipeline-config
        configMap:
          name: expanso-pipeline

---

AWS Secrets Manager

Architecture

┌────────────────────┐
│  AWS Secrets Mgr   │
│  (Pre-existing)    │
└──────────┬─────────┘
           │ IAM Role + SDK/CLI
           │ fetches secrets
           ↓
    ┌──────────────┐
    │  Edge Agent  │
    │  (uses SDK)  │
    └──────────────┘

Using Init Script

Fetch secrets at startup and inject as environment variables.

Startup script:

fetch-secrets.sh

#!/bin/bash
set -e

# Fetch secrets from AWS Secrets Manager
SECRET_JSON=$(aws secretsmanager get-secret-value \
  --secret-id expanso/pipeline/credentials \
  --region us-east-1 \
  --query SecretString \
  --output text)

# Parse JSON and export environment variables
export KAFKA_USERNAME=$(echo $SECRET_JSON | jq -r '.kafka_username')
export KAFKA_PASSWORD=$(echo $SECRET_JSON | jq -r '.kafka_password')
export API_TOKEN=$(echo $SECRET_JSON | jq -r '.api_token')
export DB_PASSWORD=$(echo $SECRET_JSON | jq -r '.db_password')

# Run edge agent with injected secrets
exec expanso-edge run --config /etc/expanso/pipeline.yaml

Docker example:

FROM ghcr.io/expanso-io/expanso-edge:latest

# Install AWS CLI and jq
RUN apt-get update && apt-get install -y awscli jq && rm -rf /var/lib/apt/lists/*

# Copy startup script
COPY fetch-secrets.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/fetch-secrets.sh

ENTRYPOINT ["/usr/local/bin/fetch-secrets.sh"]

Run with IAM role:

docker run -d \
  --name expanso-edge \
  -v ~/.aws:/root/.aws:ro \
  -v $(pwd)/pipeline.yaml:/etc/expanso/pipeline.yaml:ro \
  expanso-edge-aws:latest

Using Kubernetes External Secrets Operator

Install External Secrets Operator:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace

Configure AWS SecretStore:

Create the SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: expanso-system
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: expanso-edge

Create the ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: pipeline-credentials
  namespace: expanso-system
spec:
  refreshInterval: 1h  # Refresh every hour
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: pipeline-credentials
    creationPolicy: Owner
  data:
  - secretKey: KAFKA_USERNAME
    remoteRef:
      key: expanso/pipeline/credentials
      property: kafka_username
  - secretKey: KAFKA_PASSWORD
    remoteRef:
      key: expanso/pipeline/credentials
      property: kafka_password
  - secretKey: API_TOKEN
    remoteRef:
      key: expanso/pipeline/credentials
      property: api_token

The operator creates a Kubernetes Secret named pipeline-credentials that auto-syncs from AWS Secrets Manager.

---

Google Secret Manager

Using Init Container Pattern

Kubernetes example:

Create the ServiceAccount with GCP workload identity:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: expanso-edge
  namespace: expanso-system
  annotations:
    iam.gke.io/gcp-service-account: expanso-[email protected]

Create the Deployment with init container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expanso-edge
  namespace: expanso-system
spec:
  template:
    spec:
      serviceAccountName: expanso-edge
      initContainers:
      - name: fetch-secrets
        image: google/cloud-sdk:slim
        command:
          - /bin/bash
          - -c
          - |
            # Fetch secrets from Google Secret Manager
            gcloud secrets versions access latest \
              --secret=expanso-pipeline-credentials \
              --format=json > /secrets/credentials.json

            # Parse and write environment file
            cat /secrets/credentials.json | jq -r '
              "KAFKA_USERNAME=\(.kafka_username)\n" +
              "KAFKA_PASSWORD=\(.kafka_password)\n" +
              "API_TOKEN=\(.api_token)"
            ' > /secrets/pipeline.env

            chmod 600 /secrets/pipeline.env
        volumeMounts:
          - name: secrets
            mountPath: /secrets
      containers:
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        command:
          - /bin/sh
          - -c
          - |
            set -a
            source /secrets/pipeline.env
            set +a
            expanso-edge run --config /etc/expanso/pipeline.yaml
        volumeMounts:
          - name: secrets
            mountPath: /secrets
            readOnly: true
          - name: pipeline-config
            mountPath: /etc/expanso
            readOnly: true
      volumes:
      - name: secrets
        emptyDir:
          medium: Memory  # Use tmpfs for security
      - name: pipeline-config
        configMap:
          name: expanso-pipeline

---

Azure Key Vault

Using CSI Secret Store Driver

Install Secrets Store CSI Driver:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system

Install Azure Key Vault provider:

helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install azure-provider csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
  --namespace kube-system

Configure SecretProviderClass:

Create the SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: expanso-azure-keyvault
  namespace: expanso-system
spec:
  provider: azure
  parameters:
    usePodIdentity: "true"
    keyvaultName: "expanso-secrets"
    cloudName: "AzurePublicCloud"
    objects: |
      array:
        - |
          objectName: kafka-username
          objectType: secret
        - |
          objectName: kafka-password
          objectType: secret
        - |
          objectName: api-token
          objectType: secret
    tenantId: "your-tenant-id"
  secretObjects:
  - secretName: pipeline-credentials
    type: Opaque
    data:
    - objectName: kafka-username
      key: KAFKA_USERNAME
    - objectName: kafka-password
      key: KAFKA_PASSWORD
    - objectName: api-token
      key: API_TOKEN

Create the Deployment using the secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expanso-edge
  namespace: expanso-system
spec:
  template:
    spec:
      containers:
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        envFrom:
        - secretRef:
            name: pipeline-credentials
        volumeMounts:
        - name: secrets-store
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
      - name: secrets-store
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "expanso-azure-keyvault"

---

Best Practices

1. Use Short-Lived Credentials

Configure secrets to expire and auto-rotate:

# Vault: Generate dynamic database credentials
path "database/creds/expanso-readonly" {
  capabilities = ["read"]
}

2. Implement Least Privilege

Grant only necessary permissions:

// AWS IAM policy
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "secretsmanager:GetSecretValue",
    "Resource": "arn:aws:secretsmanager:*:*:secret:expanso/pipeline/*"
  }]
}

3. Enable Audit Logging

Track all secret access:

# Vault: Enable audit log
vault audit enable file file_path=/vault/logs/audit.log

4. Use Namespaces/Paths

Organize secrets by environment:

vault/
  ├── expanso/
  │   ├── dev/
  │   │   └── pipeline
  │   ├── staging/
  │   │   └── pipeline
  │   └── prod/
  │       └── pipeline

5. Implement Secret Rotation

Automate credential rotation:

# AWS: Enable automatic rotation
aws secretsmanager rotate-secret \
  --secret-id expanso/pipeline/credentials \
  --rotation-lambda-arn arn:aws:lambda:region:account:function:rotate-secret \
  --rotation-rules AutomaticallyAfterDays=30

6. Monitor Secret Access

Set up alerts for:

• Unauthorized access attempts
• Secrets nearing expiration
• Unusual access patterns
• Failed authentication

---

Comparison

Feature	Vault	AWS SM	GCP SM	Azure KV
Dynamic Secrets	✅	❌	❌	❌
Auto-Rotation	✅	✅	✅	✅
Free Tier	✅ (OSS)	❌	✅	❌
Multi-Cloud	✅	❌	❌	❌
K8s Integration	✅	✅ (ESO)	✅ (ESO)	✅ (CSI)
Learning Curve	High	Low	Low	Medium

ESO = External Secrets Operator CSI = Container Storage Interface

---

Troubleshooting

Authentication Failures

Symptom: Error: permission denied fetching secret

Solutions:

• Verify IAM roles/policies are attached
• Check service account annotations (Kubernetes)
• Confirm secret paths are correct
• Review audit logs for specific error

Secrets Not Refreshing

Symptom: Old credentials still in use after rotation

Solutions:

• Check refresh interval configuration
• Verify sidecar/init container restarts
• Restart edge agent to pick up new values
• Monitor rotation logs

High Latency

Symptom: Slow startup due to secret fetching

Solutions:

• Use sidecar pattern instead of init containers
• Cache secrets locally with appropriate TTL
• Use regional endpoints (AWS/GCP/Azure)
• Implement retry logic with exponential backoff

---

Next Steps

• Getting Started? Start with Local Secrets for simpler setups
• Need bootstrap help? See Bootstrapping Secrets
• Production deployment? See Edge Deployment for deployment best practices