External Secret Managers

External secret managers provide centralized, auditable secret storage with automated rotation and fine-grained access control. This guide shows how to integrate Expanso Edge with popular enterprise secret management systems.

:::info Assumption This guide assumes your secret manager (Vault, AWS Secrets Manager, etc.) is already provisioned and configured. Examples focus on fetching and using secrets, not on secret manager setup. :::

---

Why Use External Secret Managers?

Benefits

• Automated Rotation: Credentials expire and renew without manual intervention
• Audit Trails: Track who accessed which secrets and when
• Centralized Management: Single source of truth for all credentials
• Dynamic Secrets: Generate short-lived credentials on demand
• Access Control: Fine-grained policies per service/user
• Compliance: Meet SOC 2, HIPAA, PCI-DSS requirements

When to Use

• Production deployments with compliance requirements
• Multi-environment setups (dev/staging/prod)
• Large-scale deployments (10+ edge nodes)
• Regulated industries (finance, healthcare)
• Environments requiring secret rotation

---

Integration Patterns

All external secret managers follow a similar pattern:

1. Fetch secrets at edge agent startup or runtime
2. Inject as environment variables or write to local files
3. Reference in pipeline config using interpolation (${VAR_NAME})

---

HashiCorp Vault

Architecture

┌─────────────────┐
│  Vault Server   │
│  (Pre-existing) │
└────────┬────────┘
         │ Vault Agent (sidecar)
         │ fetches secrets
         ↓
┌─────────────────┐
│  Edge Agent     │
│  (reads from    │
│   env vars)     │
└─────────────────┘

Using Vault Agent Sidecar

Vault Agent runs alongside the edge agent, automatically fetching and refreshing secrets.

Vault Agent config:

vault-agent.hcl

pid_file = "/tmp/vault-agent.pid"

vault {
  address = "https://vault.example.com:8200"
}

auto_auth {
  method {
    type = "kubernetes"  # or "aws", "gcp", "azure"
    config = {
      role = "expanso-edge"
    }
  }

  sink {
    type = "file"
    config = {
      path = "/etc/expanso/secrets.env"
      mode = 0600
    }
  }
}

template {
  source      = "/etc/vault/templates/pipeline.env.tpl"
  destination = "/etc/expanso/pipeline.env"
  perms       = "0600"
}

Template file:

pipeline.env.tpl

{{ with secret "secret/data/expanso/pipeline" }}
KAFKA_USERNAME={{ .Data.data.kafka_username }}
KAFKA_PASSWORD={{ .Data.data.kafka_password }}
API_TOKEN={{ .Data.data.api_token }}
DB_PASSWORD={{ .Data.data.db_password }}
{{ end }}

Docker Compose example:

docker-compose.yml

version: '3.8'
services:
  vault-agent:
    image: hashicorp/vault:latest
    command: agent -config=/vault/config/vault-agent.hcl
    volumes:
      - ./vault-agent.hcl:/vault/config/vault-agent.hcl:ro
      - ./pipeline.env.tpl:/etc/vault/templates/pipeline.env.tpl:ro
      - secrets:/etc/expanso
    environment:
      VAULT_ADDR: https://vault.example.com:8200

  expanso-edge:
    image: ghcr.io/expanso-io/expanso-edge:latest
    command: run --config /etc/expanso/pipeline.yaml
    volumes:
      - ./pipeline.yaml:/etc/expanso/pipeline.yaml:ro
      - secrets:/etc/expanso
    depends_on:
      - vault-agent
    # Reload env vars on change
    env_file:
      - /etc/expanso/pipeline.env

volumes:
  secrets:

Using Vault Init Container (Kubernetes)

Kubernetes example:

Create the ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: expanso-edge
  namespace: expanso-system

Create the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expanso-edge
  namespace: expanso-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: expanso-edge
  template:
    metadata:
      labels:
        app: expanso-edge
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-pipeline.env: "secret/data/expanso/pipeline"
        vault.hashicorp.com/agent-inject-template-pipeline.env: |
          {{- with secret "secret/data/expanso/pipeline" -}}
          export KAFKA_USERNAME="{{ .Data.data.kafka_username }}"
          export KAFKA_PASSWORD="{{ .Data.data.kafka_password }}"
          export API_TOKEN="{{ .Data.data.api_token }}"
          {{- end }}
        vault.hashicorp.com/role: "expanso-edge"
    spec:
      serviceAccountName: expanso-edge
      containers:
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        command:
          - /bin/sh
          - -c
          - |
            source /vault/secrets/pipeline.env
            expanso-edge run --config /etc/expanso/pipeline.yaml
        volumeMounts:
          - name: pipeline-config
            mountPath: /etc/expanso
            readOnly: true
      volumes:
      - name: pipeline-config
        configMap:
          name: expanso-pipeline

---

AWS Secrets Manager

Architecture

┌────────────────────┐
│  AWS Secrets Mgr   │
│  (Pre-existing)    │
└──────────┬─────────┘
           │ IAM Role + SDK/CLI
           │ fetches secrets
           ↓
    ┌──────────────┐
    │  Edge Agent  │
    │  (uses SDK)  │
    └──────────────┘

Using Init Script

Fetch secrets at startup and inject as environment variables.

Startup script:

fetch-secrets.sh

#!/bin/bash
set -e

# Fetch secrets from AWS Secrets Manager
SECRET_JSON=$(aws secretsmanager get-secret-value \
  --secret-id expanso/pipeline/credentials \
  --region us-east-1 \
  --query SecretString \
  --output text)

# Parse JSON and export environment variables
export KAFKA_USERNAME=$(echo $SECRET_JSON | jq -r '.kafka_username')
export KAFKA_PASSWORD=$(echo $SECRET_JSON | jq -r '.kafka_password')
export API_TOKEN=$(echo $SECRET_JSON | jq -r '.api_token')
export DB_PASSWORD=$(echo $SECRET_JSON | jq -r '.db_password')

# Run edge agent with injected secrets
exec expanso-edge run --config /etc/expanso/pipeline.yaml

Docker example:

FROM ghcr.io/expanso-io/expanso-edge:latest

# Install AWS CLI and jq
RUN apt-get update && apt-get install -y awscli jq && rm -rf /var/lib/apt/lists/*

# Copy startup script
COPY fetch-secrets.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/fetch-secrets.sh

ENTRYPOINT ["/usr/local/bin/fetch-secrets.sh"]

Run with IAM role:

docker run -d \
  --name expanso-edge \
  -v ~/.aws:/root/.aws:ro \
  -v $(pwd)/pipeline.yaml:/etc/expanso/pipeline.yaml:ro \
  expanso-edge-aws:latest

Using Kubernetes External Secrets Operator

Install External Secrets Operator:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace

Configure AWS SecretStore:

Create the SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: expanso-system
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: expanso-edge

Create the ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: pipeline-credentials
  namespace: expanso-system
spec:
  refreshInterval: 1h  # Refresh every hour
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: pipeline-credentials
    creationPolicy: Owner
  data:
  - secretKey: KAFKA_USERNAME
    remoteRef:
      key: expanso/pipeline/credentials
      property: kafka_username
  - secretKey: KAFKA_PASSWORD
    remoteRef:
      key: expanso/pipeline/credentials
      property: kafka_password
  - secretKey: API_TOKEN
    remoteRef:
      key: expanso/pipeline/credentials
      property: api_token

The operator creates a Kubernetes Secret named pipeline-credentials that auto-syncs from AWS Secrets Manager.

---

Google Secret Manager

Using Init Container Pattern

Kubernetes example:

Create the ServiceAccount with GCP workload identity:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: expanso-edge
  namespace: expanso-system
  annotations:
    iam.gke.io/gcp-service-account: expanso-[email protected]

Create the Deployment with init container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expanso-edge
  namespace: expanso-system
spec:
  template:
    spec:
      serviceAccountName: expanso-edge
      initContainers:
      - name: fetch-secrets
        image: google/cloud-sdk:slim
        command:
          - /bin/bash
          - -c
          - |
            # Fetch secrets from Google Secret Manager
            gcloud secrets versions access latest \
              --secret=expanso-pipeline-credentials \
              --format=json > /secrets/credentials.json

            # Parse and write environment file
            cat /secrets/credentials.json | jq -r '
              "KAFKA_USERNAME=\(.kafka_username)\n" +
              "KAFKA_PASSWORD=\(.kafka_password)\n" +
              "API_TOKEN=\(.api_token)"
            ' > /secrets/pipeline.env

            chmod 600 /secrets/pipeline.env
        volumeMounts:
          - name: secrets
            mountPath: /secrets
      containers:
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        command:
          - /bin/sh
          - -c
          - |
            set -a
            source /secrets/pipeline.env
            set +a
            expanso-edge run --config /etc/expanso/pipeline.yaml
        volumeMounts:
          - name: secrets
            mountPath: /secrets
            readOnly: true
          - name: pipeline-config
            mountPath: /etc/expanso
            readOnly: true
      volumes:
      - name: secrets
        emptyDir:
          medium: Memory  # Use tmpfs for security
      - name: pipeline-config
        configMap:
          name: expanso-pipeline

---

Azure Key Vault

Using CSI Secret Store Driver

Install Secrets Store CSI Driver:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system

Install Azure Key Vault provider:

helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install azure-provider csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
  --namespace kube-system

Configure SecretProviderClass:

Create the SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: expanso-azure-keyvault
  namespace: expanso-system
spec:
  provider: azure
  parameters:
    usePodIdentity: "true"
    keyvaultName: "expanso-secrets"
    cloudName: "AzurePublicCloud"
    objects: |
      array:
        - |
          objectName: kafka-username
          objectType: secret
        - |
          objectName: kafka-password
          objectType: secret
        - |
          objectName: api-token
          objectType: secret
    tenantId: "your-tenant-id"
  secretObjects:
  - secretName: pipeline-credentials
    type: Opaque
    data:
    - objectName: kafka-username
      key: KAFKA_USERNAME
    - objectName: kafka-password
      key: KAFKA_PASSWORD
    - objectName: api-token
      key: API_TOKEN

Create the Deployment using the secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expanso-edge
  namespace: expanso-system
spec:
  template:
    spec:
      containers:
      - name: expanso-edge
        image: ghcr.io/expanso-io/expanso-edge:latest
        envFrom:
        - secretRef:
            name: pipeline-credentials
        volumeMounts:
        - name: secrets-store
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
      - name: secrets-store
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "expanso-azure-keyvault"

---

Best Practices

1. Use Short-Lived Credentials

Configure secrets to expire and auto-rotate:

# Vault: Generate dynamic database credentials
path "database/creds/expanso-readonly" {
  capabilities = ["read"]
}

2. Implement Least Privilege

Grant only necessary permissions:

// AWS IAM policy
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "secretsmanager:GetSecretValue",
    "Resource": "arn:aws:secretsmanager:*:*:secret:expanso/pipeline/*"
  }]
}

3. Enable Audit Logging

Track all secret access:

# Vault: Enable audit log
vault audit enable file file_path=/vault/logs/audit.log

4. Use Namespaces/Paths

Organize secrets by environment:

vault/
  ├── expanso/
  │   ├── dev/
  │   │   └── pipeline
  │   ├── staging/
  │   │   └── pipeline
  │   └── prod/
  │       └── pipeline

5. Implement Secret Rotation

Automate credential rotation:

# AWS: Enable automatic rotation
aws secretsmanager rotate-secret \
  --secret-id expanso/pipeline/credentials \
  --rotation-lambda-arn arn:aws:lambda:region:account:function:rotate-secret \
  --rotation-rules AutomaticallyAfterDays=30

6. Monitor Secret Access

Set up alerts for:

• Unauthorized access attempts
• Secrets nearing expiration
• Unusual access patterns
• Failed authentication

---

Comparison

Feature	Vault	AWS SM	GCP SM	Azure KV
Dynamic Secrets	✅	❌	❌	❌
Auto-Rotation	✅	✅	✅	✅
Free Tier	✅ (OSS)	❌	✅	❌
Multi-Cloud	✅	❌	❌	❌
K8s Integration	✅	✅ (ESO)	✅ (ESO)	✅ (CSI)
Learning Curve	High	Low	Low	Medium

ESO = External Secrets Operator CSI = Container Storage Interface

---

Troubleshooting

Authentication Failures

Symptom: Error: permission denied fetching secret

Solutions:

• Verify IAM roles/policies are attached
• Check service account annotations (Kubernetes)
• Confirm secret paths are correct
• Review audit logs for specific error

Secrets Not Refreshing

Symptom: Old credentials still in use after rotation

Solutions:

• Check refresh interval configuration
• Verify sidecar/init container restarts
• Restart edge agent to pick up new values
• Monitor rotation logs

High Latency

Symptom: Slow startup due to secret fetching

Solutions:

• Use sidecar pattern instead of init containers
• Cache secrets locally with appropriate TTL
• Use regional endpoints (AWS/GCP/Azure)
• Implement retry logic with exponential backoff

---

Next Steps

• Getting Started? Start with Local Secrets for simpler setups
• Need bootstrap help? See Bootstrapping Secrets
• Production deployment? See Edge Deployment for deployment best practices