Deploy on Kubernetes with Helm

The chart in deploy/helm/stampede runs:

• the server (REST API, run manager, web UI) behind a Service with http on 8080 and the worker gRPC port on 8081;
• a worker Deployment whose pods dial the server with the join token;
• optionally a single-node TimescaleDB StatefulSet with a persistent volume, or a connection to an existing PostgreSQL.

It is tested on kind by deploy/kind/smoke.sh (also run in CI by .github/workflows/k8s.yml): install with the bundled database, helm test, then a 10-second distributed run on two in-cluster workers that must pass.

Each release publishes the chart, versioned with the release, as an OCI artifact (oci://ghcr.io/ivan825/charts/stampede) and attaches it to the GitHub release. The chart runs the Stampede image of the same version.

Install

helm install stampede oci://ghcr.io/ivan825/charts/stampede --version 1.0.0 \
  -n stampede --create-namespace
kubectl -n stampede rollout status deploy/stampede-server
helm test stampede -n stampede
kubectl -n stampede port-forward svc/stampede 8080:8080   # then open http://localhost:8080

Create the organisation and the owner account with stampede setup; until then the web UI shows a page saying to run it. Everything else is done with the stampede CLI too; the web UI is for analysis and reporting.

Local image

To run an image built from a clone instead (for a change of your own), build it, make it available to the cluster (for example on kind) and install the chart from the checkout:

git clone https://github.com/Ivan825/Stampede && cd Stampede
docker build -t ghcr.io/ivan825/stampede:dev .
kind load docker-image ghcr.io/ivan825/stampede:dev
helm install stampede deploy/helm/stampede -n stampede --create-namespace \
  --set image.tag=dev --set image.pullPolicy=Never

Secrets

The chart’s Secret (<release>-stampede, or stampede when the release is called stampede) holds:

Key	What	Default
master-key	encrypts secrets stored in the database	32 random bytes, generated
join-token	what workers present to join	48 random characters, generated
postgres-password	bundled TimescaleDB password	32 random characters, generated
database-url	only with database.external.url

Generated values are created on the first install and reused on every upgrade (the chart reads the existing Secret with lookup), so helm upgrade never rotates them. The Secret carries helm.sh/resource-policy: keep and survives helm uninstall; delete it by hand once you no longer need the data.

Bring your own instead with masterKey.existingSecret, joinToken.existingSecret and timescaledb.existingSecret (each with an existingSecretKey), or literal values (masterKey.value, …).

Back up the master key. Without it, secrets stored in Stampede cannot be decrypted:

kubectl -n stampede get secret stampede -o jsonpath='{.data.master-key}' | base64 -d > master.key

Note that helm template has no cluster to look up, so it generates new values every time; use helm install/upgrade, or pass existing secrets, when rendering manifests for GitOps tools.

Database

Bundled (default): timescaledb.enabled=true with an 8Gi volume. It is a single pod without replication or automatic backups: good for evaluation and small teams. See upgrades.md for backups.

External PostgreSQL (TimescaleDB recommended; the schema migration only turns on hypertables and compression when the timescaledb extension is available, so plain PostgreSQL is accepted too):

kubectl -n stampede create secret generic stampede-db \
  --from-literal=url='postgres://stampede:PASSWORD@db.example.com:5432/stampede?sslmode=require'
helm install stampede deploy/helm/stampede -n stampede \
  --set timescaledb.enabled=false \
  --set database.external.existingSecret=stampede-db --set database.external.existingSecretKey=url

Workers

In-cluster workers dial <fullname>:8081 with the join token. Values:

Value	Meaning
workers.replicas	fixed number of workers (default 2)
workers.region	region label used to split load by region
workers.labels	extra --label KEY=VALUE
workers.maxVUs	most virtual users one worker accepts
workers.autoscaling.*	CPU-based HorizontalPodAutoscaler (needs metrics-server; no KEDA)

About autoscaling: a run’s load is split across the workers connected when it starts. The HPA therefore keeps capacity in line with how busy workers have been; it does not reshard a run that is already going. It scales down one pod every two minutes after a ten-minute window, because removing a worker mid-run loses its share of that run. For a fixed fleet per run, the operator’s StampedeRun can scale a worker Deployment up for exactly one run (operator.md); do not combine that with the HPA.

Workers have no probes: they serve no port. A worker that loses the server reconnects by itself, and stops generating load if it is cut off for 10 seconds during a run.

Workers elsewhere (VMs, other clusters, other clouds) need the worker port exposed (for example service.type=LoadBalancer, or a separate Service) and the join token; see terraform.md.

The worker port uses mutual TLS by default (workerMTLS.enabled=true): the server derives a CA from its master key, and every worker enrolls for its own certificate with --mtls. Set workerMTLS.caFingerprint (logged by the server at start) to pin the CA on workers. Without a pin, a worker trusts the CA at its first enrollment, and only a holder of the join token can produce that answer.

To use certificates from your own CA instead, create a TLS Secret with tls.crt, tls.key and ca.crt (the certificate must name the service host), then set workerTLS.enabled=true and workerTLS.secretName. This takes precedence over workerMTLS. These settings render and match the server’s --worker-tls-cert/key and the worker’s --ca flags, but have not been exercised on a cluster yet.

Server replicas

server.ha decides what extra server replicas do.

standby (the default): one replica holds a Postgres advisory lock and serves; the others wait (healthy, not ready) and take over within a few seconds if it goes away. The Deployment uses the Recreate strategy, since a new pod cannot become ready while the old one serves: the old pod stops (finishing its runs’ reports within server.terminationGracePeriodSeconds) before the new one starts, so an upgrade has a short UI/API outage.

active: every replica serves the API and the UI and runs tests.

server:
  replicas: 3
  ha: active

• Each run is owned by the replica that started it. Stop, kill and “kill all” sent to any replica reach the owner through Postgres LISTEN/NOTIFY, and the live view works from any replica (a replica that does not own the run follows it from the database, about a second behind).
• The chart adds a headless service, <release>-stampede-replicas, and workers start with --server dns:<that service>:<worker port>: each worker connects to every replica and re-resolves every 30 seconds. A worker runs load for one replica at a time and tells the others it is busy, so they do not assign it work.
• Replicas send a heartbeat to the database every 5 seconds. When a replica has been silent for 30 seconds, another one marks its unfinished runs and AI jobs as failed. Runs are not handed over to another replica.
• Upgrades roll one pod at a time with no outage; runs on the pod being replaced end when it stops.
• Scheduled runs: every replica looks for due schedules, and each firing is claimed with one conditional update, so it starts one run.

Ingress and TLS for the UI

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: stampede.example.com
      paths: [{ path: /, pathType: Prefix }]
  tls:
    - secretName: stampede-tls
      hosts: [stampede.example.com]
server:
  secureCookies: true
  trustedProxies: ["10.0.0.0/8"]   # the ingress controller's pod network

The live-run view uses server-sent events; ingress controllers that buffer responses need buffering off for /api/v1/runs/*/live (for ingress-nginx: nginx.ingress.kubernetes.io/proxy-buffering: "off").

Security

Every pod runs as a non-root user with a read-only root filesystem, all capabilities dropped, no privilege escalation and the RuntimeDefault seccomp profile; service account tokens are not mounted. The workloads are compatible with the restricted Pod Security Standard.

Monitoring

The server exposes Prometheus metrics on /metrics (port http). With the Prometheus Operator, set serviceMonitor.enabled=true (and serviceMonitor.labels to match your Prometheus selector).

Values reference

All values are documented in values.yaml and checked by values.schema.json: unknown keys and invalid enums are rejected at install time.

Uninstall

helm uninstall stampede -n stampede
# kept on purpose: the Secret with the master key, and the database volume
kubectl -n stampede delete secret stampede
kubectl -n stampede delete pvc data-stampede-timescaledb-0