Lobstack/Docs
Deploy Agent
Docs/Security & Infrastructure/Secrets Management
Security

Secrets Management — HashiCorp Vault

Centralized secrets management with HA Raft storage, Kubernetes authentication, Transit encryption, and fine-grained RBAC policies.

💡

Why Vault?

Lobstack manages API keys for multiple AI providers (Anthropic, OpenAI, Google, xAI), cloud providers (Hetzner, DigitalOcean, Vultr), payment processors (Stripe), and per-agent credentials. Vault centralizes all secrets with audit logging, automatic rotation, and zero-trust access control.

Vault Architecture#

Vault Deployment
┌─────────────────────────────────────────────────────┐
│  lobstack-vault namespace                            │
│                                                      │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐          │
│  │ vault-0  │  │ vault-1  │  │ vault-2  │  HA Raft │
│  │ (leader) │◄─┤(follower)│◄─┤(follower)│  Cluster │
│  └────┬─────┘  └──────────┘  └──────────┘          │
│       │                                              │
│  ┌────▼───────────────────────────────────┐         │
│  │  Vault Agent Injector                   │         │
│  │  (Sidecar injection for pods)           │         │
│  └────────────────────────────────────────┘         │
│                                                      │
│  ┌────────────────────────────────────────┐         │
│  │  Secrets Store CSI Driver               │         │
│  │  (Volume-mounted secrets for K8s)       │         │
│  └────────────────────────────────────────┘         │
└─────────────────────────────────────────────────────┘
ComponentReplicasPurpose
Vault Server3 (5 in production)Secret storage, Transit engine, auth
Agent Injector2Sidecar injection for automatic secret delivery
CSI ProviderDaemonSetVolume-mounted secrets for K8s integration
Audit Storage10 GB PVCImmutable audit trail of all operations
Data Storage20 GB PVC (encrypted)Raft integrated storage for all secrets

Secrets Engines#

KV v2 (Key-Value Store)#

The primary secrets engine stores all Lobstack credentials as versioned key-value pairs. KV v2 provides version history, soft delete, and check-and-set for safe concurrent updates.

Secret Paths
# Platform-wide API keys (read by the Lobstack API)
secret/data/lobstack/api-keys
  ├── anthropic_api_key
  ├── openai_api_key
  ├── google_api_key
  ├── xai_api_key
  ├── moonshot_api_key
  ├── hetzner_api_token
  ├── do_api_token
  ├── vultr_api_key
  ├── stripe_secret_key
  ├── supabase_service_key
  └── agent_api_secret

# Per-agent secrets (one path per agent)
secret/data/lobstack/agent/{agent-id}
  ├── ai_api_key
  ├── ai_api_key_env_var
  ├── openclaw_model
  ├── telegram_token
  └── agent_api_secret

Transit Engine (Encryption-as-a-Service)#

The Transit engine provides AES-256-GCM encryption without exposing keys. Applications send plaintext → Vault returns ciphertext. Keys are managed entirely within Vault with 90-day automatic rotation.

Key NameAlgorithmAuto-RotationUsed By
lobstackAES-256-GCM90 daysAPI key encryption, sensitive config
agent-dataAES-256-GCM90 daysAgent workspace data, user tokens

Authentication#

Vault uses the Kubernetes auth method — pods authenticate using their ServiceAccount JWT tokens. No static tokens or passwords are needed. Each service role gets a scoped Vault token with limited TTL.

Auth Roles
# Lobstack API — can read all platform secrets + manage agent secrets
vault write auth/kubernetes/role/lobstack-api \
  bound_service_account_names=lobstack-api \
  bound_service_account_namespaces=lobstack-control-plane \
  policies=lobstack-api \
  ttl=1h \
  max_ttl=4h

# Agent pods — can ONLY read their own agent secret
vault write auth/kubernetes/role/lobstack-agent \
  bound_service_account_names=lobstack-agent \
  bound_service_account_namespaces=lobstack-agents \
  policies=lobstack-agent \
  ttl=1h \
  max_ttl=4h

Access Policies#

Vault policies define exactly what each service can access. The agent policy uses templated paths — each agent can only read secrets for its own agent ID, enforced cryptographically by Vault.

vault-policies.hcl — Agent Policy
# Agent Policy (templated — each agent reads ONLY its own secrets)
path "secret/data/lobstack/agent/{{identity.entity.metadata.agent_id}}" {
  capabilities = ["read"]
}

# Agents can encrypt/decrypt their workspace data
path "transit/encrypt/agent-data" {
  capabilities = ["update"]
}
path "transit/decrypt/agent-data" {
  capabilities = ["update"]
}
vault-policies.hcl — API Policy
# API Policy — full platform secret access + agent management
path "secret/data/lobstack/api-keys" {
  capabilities = ["read"]
}

path "secret/data/lobstack/agent/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "transit/encrypt/lobstack" {
  capabilities = ["update"]
}

path "transit/decrypt/lobstack" {
  capabilities = ["update"]
}

Secret Delivery#

Secrets are delivered to pods via two mechanisms, used together for defense-in-depth:

💉

Vault Agent Injector

A sidecar container is injected into each pod. It authenticates to Vault, fetches secrets, and writes them as environment variable files. Secrets are refreshed every 5 minutes.

📁

Secrets Store CSI Driver

Mounts Vault secrets as files in the pod's filesystem. Syncs to Kubernetes Secrets for envFrom usage. Supports automatic rotation.

Pod Annotations for Vault Injection
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "lobstack-api"
    vault.hashicorp.com/agent-inject-secret-api-keys: "secret/data/lobstack/api-keys"
    vault.hashicorp.com/agent-inject-template-api-keys: |
      {{- with secret "secret/data/lobstack/api-keys" -}}
      export ANTHROPIC_API_KEY="{{ .Data.data.anthropic_api_key }}"
      export OPENAI_API_KEY="{{ .Data.data.openai_api_key }}"
      export STRIPE_SECRET_KEY="{{ .Data.data.stripe_secret_key }}"
      {{- end -}}

Application Client#

The src/lib/vault.ts module provides a TypeScript client that handles authentication, secret retrieval, encryption, and caching with automatic environment variable fallback.

Using the Vault Client
import { getApiKeys, setAgentSecret, deleteAgentSecret, encrypt, decrypt } from "@/lib/vault";

// Get all platform API keys (cached, falls back to env vars)
const keys = await getApiKeys();
keys.anthropicApiKey;  // From Vault or ANTHROPIC_API_KEY env

// Store agent-specific secrets
await setAgentSecret("agent-abc123", {
  ai_api_key: "sk-ant-...",
  telegram_token: "123456:ABC-...",
});

// Clean up on agent teardown
await deleteAgentSecret("agent-abc123");

// Encrypt/decrypt via Transit engine
const cipher = await encrypt("sensitive-data");      // vault:v1:...
const plain = await decrypt(cipher);                  // sensitive-data

Audit Logging#

Every Vault operation is logged — secret reads, writes, encryptions, decryptions, authentication events, and policy changes. Audit logs are written to persistent storage and retained for 2+ years, aligned with SOC 2 requirements.

✅

Initialization

Run infra/kubernetes/vault/vault-init.sh after deploying Vault to configure engines, policies, and Kubernetes auth. See Compliance Readiness for the full audit trail requirements.
Encryption (AES-256)Sandbox Isolation