Zero to Kubernetes Part 4: FastAPI Backend + PostgreSQL in Production

K8s Terraform ArgoCD — Header

K8s Terraform ArgoCD — Header

Series: Zero to Kubernetes — Part 1 · Part 2 · Part 3 · Part 4 · Part 5

We now have a cluster with ingress, TLS, and GitOps. Time to put real workloads on it: a PostgreSQL database managed by an operator, and a FastAPI backend that connects to it, runs migrations on startup, and exposes a REST API with health checks.

This is the part where the tutorial stops being “look how pretty the empty cluster is” and starts being “this actually works in production.”

Components of This Part

┌─────────────────────────────────────────┐
│          api.yourdomain.com              │
│                │                         │
│    ┌───────────▼───────────┐             │
│    │   Ingress             │             │
│    │   (nginx + TLS)       │             │
│    └───────────┬───────────┘             │
│                │                         │
│    ┌───────────▼───────────┐             │
│    │   Backend Service     │             │
│    │   (ClusterIP)         │             │
│    └───────────┬───────────┘             │
│                │                         │
│    ┌───────────▼───────────┐             │
│    │   Backend Deployment  │             │
│    │   2 replicas          │             │
│    │   FastAPI + Uvicorn   │             │
│    └───────────┬───────────┘             │
│                │                         │
│    ┌───────────▼───────────┐             │
│    │   PostgreSQL          │             │
│    │   (CloudNativePG)     │             │
│    │   1 instance          │             │
│    │   PVC 10Gi            │             │
│    └───────────────────────┘             │
└─────────────────────────────────────────┘

Step 1: CloudNativePG — PostgreSQL as a Native Kubernetes Resource

CloudNativePG (CNPG) is an operator that manages PostgreSQL on Kubernetes without manual StatefulSets. You install it and create a Cluster resource. The operator handles replication, backups, failover, and updates.

resource "helm_release" "cnpg" {
  name       = "cloudnative-pg"
  namespace  = kubernetes_namespace.infrastructure.metadata[0].name
  repository = "https://cloudnative-pg.github.io/charts"
  chart      = "cloudnative-pg"
  version    = "0.22.0"
}

Then define the database as code in our GitOps repo:

gitops/apps/database/postgresql-cluster.yaml:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: guayoyo-db
  namespace: production
spec:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4

  bootstrap:
    initdb:
      database: guayoyo
      owner: app
      secret:
        name: db-credentials

  storage:
    size: 10Gi
    storageClass: local-path

  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"
    limits:
      memory: "512Mi"
      cpu: "500m"

  monitoring:
    enablePodMonitor: false

gitops/apps/database/db-credentials-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: kubernetes.io/basic-auth
stringData:
  username: app
  password: "ChangeThisInProductionWithSealedSecrets"

⚠️ In real production: Never put passwords in plain text in git. Use SealedSecrets (encrypts the entire Secret), ExternalSecrets (syncs from Vault), or SOPS + Age.

Step 2: Backend Code (FastAPI)

main.py:

from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
import asyncpg
import os
import time
from contextlib import asynccontextmanager

DB_HOST = os.getenv("DB_HOST", "guayoyo-db-rw")
DB_PORT = os.getenv("DB_PORT", "5432")
DB_NAME = os.getenv("DB_NAME", "guayoyo")
DB_USER = os.getenv("DB_USER", "app")
DB_PASSWORD = os.getenv("DB_PASSWORD", "")

pool = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global pool
    max_retries = 10
    for attempt in range(max_retries):
        try:
            pool = await asyncpg.create_pool(
                host=DB_HOST, port=DB_PORT, database=DB_NAME,
                user=DB_USER, password=DB_PASSWORD,
                min_size=2, max_size=10
            )
            async with pool.acquire() as conn:
                await conn.execute("""
                    CREATE TABLE IF NOT EXISTS items (
                        id SERIAL PRIMARY KEY,
                        name TEXT NOT NULL,
                        description TEXT,
                        created_at TIMESTAMPTZ DEFAULT NOW()
                    )
                """)
            break
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(3)
    yield
    await pool.close()

app = FastAPI(title="Guayoyo API", lifespan=lifespan)

app.add_middleware(
    CORSMiddleware,
    allow_origins=[
        "https://admin.yourdomain.com",
        "https://app.yourdomain.com",
    ],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

class ItemCreate(BaseModel):
    name: str
    description: str = ""

class Item(ItemCreate):
    id: int
    created_at: str

@app.get("/health")
async def health():
    try:
        async with pool.acquire() as conn:
            await conn.fetchval("SELECT 1")
        return {"status": "ok", "db": "connected", "timestamp": time.time()}
    except Exception as e:
        return {"status": "degraded", "db": str(e)}

@app.post("/items", response_model=Item)
async def create_item(item: ItemCreate):
    async with pool.acquire() as conn:
        row = await conn.fetchrow(
            "INSERT INTO items (name, description) VALUES ($1, $2) RETURNING *",
            item.name, item.description
        )
    return Item(id=row["id"], name=row["name"],
               description=row["description"] or "",
               created_at=str(row["created_at"]))

@app.get("/items")
async def list_items():
    async with pool.acquire() as conn:
        rows = await conn.fetch("SELECT * FROM items ORDER BY created_at DESC")
    return [{"id": r["id"], "name": r["name"],
             "description": r["description"] or "",
             "created_at": str(r["created_at"])} for r in rows]

@app.get("/items/{item_id}")
async def get_item(item_id: int):
    async with pool.acquire() as conn:
        row = await conn.fetchrow("SELECT * FROM items WHERE id = $1", item_id)
    if not row:
        raise HTTPException(status_code=404, detail="Item not found")
    return {"id": row["id"], "name": row["name"],
            "description": row["description"] or "",
            "created_at": str(row["created_at"])}

@app.delete("/items/{item_id}")
async def delete_item(item_id: int):
    async with pool.acquire() as conn:
        result = await conn.execute("DELETE FROM items WHERE id = $1", item_id)
    if result == "DELETE 0":
        raise HTTPException(status_code=404, detail="Item not found")
    return {"status": "deleted"}

Dockerfile:

FROM python:3.12-slim AS builder
WORKDIR /app
RUN pip install --no-cache-dir poetry
COPY pyproject.toml poetry.lock ./
RUN poetry config virtualenvs.create false && poetry install --no-dev

FROM python:3.12-slim
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY . .
EXPOSE 8000
HEALTHCHECK --interval=10s --timeout=5s --retries=3 
  CMD curl -f http://localhost:8000/health || exit 1
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Build and push:

docker build -t ghcr.io/your-org/backend:v1.0.0 .
docker push ghcr.io/your-org/backend:v1.0.0

Step 3: Deploy to Kubernetes via ArgoCD

gitops/apps/backend/deployment.yaml (key sections):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      serviceAccountName: apps-sa
      containers:
        - name: backend
          image: ghcr.io/your-org/backend:v1.0.0
          ports:
            - containerPort: 8000
          env:
            - name: DB_HOST
              value: guayoyo-db-rw.production.svc.cluster.local
            - name: DB_PORT
              value: "5432"
            - name: DB_NAME
              value: guayoyo
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10

Key points:

  • guayoyo-db-rw.production.svc.cluster.local — internal DNS that CloudNativePG creates automatically. -rw = read-write (primary). If you had read replicas, they’d be at -ro.
  • secretKeyRef injects username/password from the db-credentials Secret without exposing them in the Deployment
  • livenessProbe and readinessProbe — Kubernetes uses these to know if the pod is alive and if it should receive traffic

gitops/apps/backend/ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backend
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.yourdomain.com
      secretName: api-tls
  rules:
    - host: api.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend
                port:
                  number: 80

Step 4: End-to-End Verification

# Verify ArgoCD synced everything
argocd app list
# backend  Healthy  Synced
# database Healthy  Synced

# Test the API
curl https://api.yourdomain.com:30443/health
# {"status":"ok","db":"connected","timestamp":1716163200.123}

# Create an item
curl -X POST https://api.yourdomain.com:30443/items 
  -H "Content-Type: application/json" 
  -d '{"name":"Test Item","description":"From curl"}'
# {"id":1,"name":"Test Item","description":"From curl",...}

# List items
curl https://api.yourdomain.com:30443/items
# [{"id":1,"name":"Test Item",...}]

# Check that both replicas are running
kubectl get pods -n production -l app=backend
# NAME                      READY   STATUS    RESTARTS   AGE
# backend-xxxxx-xxxxx       1/1     Running   0          1m
# backend-xxxxx-yyyyy       1/1     Running   0          1m

What You Learned in This Part

  • CloudNativePG as a PostgreSQL operator for Kubernetes
  • Kubernetes internal DNS: guayoyo-db-rw.production.svc.cluster.local
  • Secrets for credentials (with serious note about SealedSecrets/ExternalSecrets)
  • FastAPI with connection pooling to PostgreSQL via asyncpg
  • Liveness and readiness probes — the difference and why both matter
  • CORS configured to accept traffic from frontends
  • Multi-stage Docker build optimized for production

In Part 5, we’ll deploy both frontends (React + Vue), connect them to the backend, and run a full integration test where one app creates data and the other consumes it in real time.

At Guayoyo Tech, we design backends that don’t crash under real traffic. FastAPI, Go, Node.js — whatever your project needs, with PostgreSQL, health checks, metrics, and automated Kubernetes deployment. Talk to us free for 15 minutes and we’ll show you how.

Zero to Kubernetes Part 3: ArgoCD, GitOps and App-of-Apps

K8s Terraform ArgoCD — Header

K8s Terraform ArgoCD — Header

Series: Zero to Kubernetes — Part 1 · Part 2 · Part 3 · Part 4 · Part 5

So far we’ve built the platform with Terraform. But Terraform isn’t a CD pipeline — running terraform apply every time a container image changes is torture. This is where ArgoCD comes in.

ArgoCD implements GitOps: your cluster’s desired state lives in a git repository. ArgoCD monitors that repo and ensures the cluster reflects exactly what’s committed. If someone manually runs kubectl edit, ArgoCD reverts it. If a deploy fails, ArgoCD tells you exactly which resource is broken and why.

GitOps in 30 Seconds

┌──────────┐   git push    ┌──────────┐
│  GitHub   │◄──────────────│  You      │
│  (repo)   │               └──────────┘
└────┬─────┘
     │ ArgoCD reads repo every 3 min
     ▼
┌──────────┐    sync    ┌──────────────────┐
│ ArgoCD   │───────────►│  k8s cluster      │
│ (inside  │◄───────────│  (your apps)      │
│ cluster) │   status   └──────────────────┘
     │
     ▼  (optional UI at argocd.yourdomain.com)
┌──────────────┐
│  🌳 App tree  │ → see diff, health, logs
│               │ → manual or auto sync
│               │ → rollback by reverting commit
└──────────────┘

The three properties that differentiate GitOps from a deploy script:

  1. Reconciliation loop: Every 3 minutes ArgoCD compares the cluster against git. If there’s drift, it corrects it.
  2. Self-healing: Someone deleted a Deployment by mistake → ArgoCD recreates it.
  3. Auditability: Every change to the cluster has an associated commit. You know who, what, when, and why.

Step 1: Install ArgoCD with Terraform

resource "helm_release" "argocd" {
  name       = "argocd"
  namespace  = kubernetes_namespace.argocd.metadata[0].name
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = "7.6.7"

  set {
    name  = "server.service.type"
    value = "ClusterIP"
  }

  set {
    name  = "server.ingress.enabled"
    value = "true"
  }

  set {
    name  = "server.ingress.hosts[0]"
    value = "argocd.yourdomain.com"
  }

  set {
    name  = "server.ingress.annotations.nginx\.ingress\.kubernetes\.io/ssl-redirect"
    value = "true"
  }

  set {
    name  = "server.ingress.annotations.cert-manager\.io/cluster-issuer"
    value = "letsencrypt-production"
  }

  set {
    name  = "server.ingress.tls[0].hosts[0]"
    value = "argocd.yourdomain.com"
  }

  set {
    name  = "server.ingress.tls[0].secretName"
    value = "argocd-tls"
  }

  set {
    name  = "server.extraArgs[0]"
    value = "--insecure"
  }

  depends_on = [helm_release.cert_manager]
}

Access the UI at https://argocd.yourdomain.com. Get the initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret 
  -o jsonpath="{.data.password}" | base64 -d

Step 2: GitOps Repository Structure

Create a separate repo (or a directory in the monorepo) with this structure:

gitops/
├── apps/
│   ├── backend/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   ├── ingress.yaml
│   │   ├── configmap.yaml
│   │   └── kustomization.yaml
│   ├── frontend-app1/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   ├── ingress.yaml
│   │   ├── configmap.yaml
│   │   └── kustomization.yaml
│   ├── frontend-app2/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   ├── ingress.yaml
│   │   ├── configmap.yaml
│   │   └── kustomization.yaml
│   └── database/
│       ├── postgresql-cluster.yaml
│       └── kustomization.yaml
└── clusters/
    └── production/
        └── app-of-apps.yaml    ← THE MASTER KEY

Step 3: App-of-Apps — the Pattern That Scales Everything

Instead of creating 15 Applications manually in ArgoCD, you create one root Application that points to apps/. Each subdirectory automatically becomes an Application. Adding a new app is creating a folder. Without touching ArgoCD.

clusters/production/app-of-apps.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: apps
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true      # deletes resources you removed from the repo
      selfHeal: true   # reverts manual changes on the cluster
    syncOptions:
      - CreateNamespace=true

The three flags that matter:

  • prune: true → If you delete apps/backend/deployment.yaml from the repo, ArgoCD deletes the Deployment from the cluster. Without this, the resource stays orphaned forever.
  • selfHeal: true → If someone runs kubectl edit deployment backend and changes replicas from 2 to 5, ArgoCD reverts it on the next reconciliation (≤3 min).
  • CreateNamespace=true → If the namespace doesn’t exist, it creates it. Prevents the “namespace not found” error.

Step 4: Register the Repo in ArgoCD

ArgoCD needs read access to the repo. Use a GitHub token with repo scope:

kubectl create secret generic github-repo-creds 
  -n argocd 
  --from-literal=url=https://github.com/your-org/gitops-repo 
  --from-literal=username=gitops-bot 
  --from-literal=password=ghp_yourGitHubToken 
  --from-literal=type=git 
  --dry-run=client -o yaml | kubectl apply -f -

Label the secret so ArgoCD uses it:

kubectl label secret github-repo-creds -n argocd 
  argocd.argoproj.io/secret-type=repository

Step 5: Create the Root Application

The app-of-apps is created just once. You can do it with Terraform or directly:

resource "kubernetes_manifest" "app_of_apps" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name      = "app-of-apps"
      namespace = "argocd"
    }
    spec = {
      project = "default"
      source = {
        repoURL        = "https://github.com/your-org/gitops-repo"
        targetRevision = "main"
        path           = "apps"
        directory = { recurse = true }
      }
      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = "argocd"
      }
      syncPolicy = {
        automated = { prune = true, selfHeal = true }
        syncOptions = ["CreateNamespace=true"]
      }
    }
  }
}

Step 6: See the App Tree

Within seconds of applying, the ArgoCD UI shows:

🌳 app-of-apps
  ├── 🟢 backend        (Healthy · Synced)
  ├── 🟢 frontend-app1  (Healthy · Synced)
  ├── 🟢 frontend-app2  (Healthy · Synced)
  └── 🟢 database       (Healthy · Synced)

Green circles mean the resource is Healthy (pods running, probes passing) and Synced (cluster matches git).

If something fails:

  • Yellow (Progressing): Pods starting up, health checks not yet passing.
  • Red (Degraded): CrashLoopBackOff, ImagePullBackOff, probes failing.
  • OutOfSync: Someone changed something manually → ArgoCD marks it for correction or auto-corrects it (selfHeal).

Step 7: ArgoCD CLI — for Terminal Fans

brew install argocd
argocd login argocd.yourdomain.com --username admin --password ...
argocd app list
argocd app diff backend
argocd app sync backend         # manual sync if not using auto-sync
argocd app rollback backend      # revert via git commit is better
argocd app logs backend --tail=50

There’s also a VS Code extension that shows ArgoCD status in the editor. Very useful when coding and wanting to see if the deploy passed without switching windows.

The Daily Workflow with GitOps

1. Change something in app code (main.py, App.tsx, etc.)
2. Docker build → new image (ghcr.io/your-org/backend:v1.2.3)
3. Update gitops/apps/backend/deployment.yaml:
      image: ghcr.io/your-org/backend:v1.2.3
4. git add && git commit && git push
5. ArgoCD detects the change → auto sync
6. If something fails → rollback: git revert && git push

The entire deploy history is the git history. All rollbacks are git revert. There’s no “rollback procedure” — it’s the same procedure as any code change.

What You Learned in This Part

  • GitOps as a philosophy: git is the single source of truth for the cluster
  • ArgoCD installed with Terraform, exposed via ingress with TLS
  • The app-of-apps pattern: one root Application managing N applications
  • syncPolicy: prune + selfHeal + auto-sync for automated deployments
  • How ArgoCD detects drift and corrects it without human intervention
  • ArgoCD CLI and UI for debugging
  • The complete flow: git push → ArgoCD sync → deploy in seconds

In Part 4, we’ll deploy the PostgreSQL database and the backend API. With real code, health checks, automatic migrations, and properly managed secrets.

At Guayoyo Tech, we implement GitOps for teams still doing manual deploys. ArgoCD, Flux, CI/CD pipelines — we help you go from “it works on my machine” to “it’s in production and fixes itself.” Talk to us free for 15 minutes and we’ll show you how.

Zero to Kubernetes Part 2: Cluster Bootstrap — Namespaces, Ingress, cert-manager and Storage

K8s Terraform ArgoCD — Header

K8s Terraform ArgoCD — Header

Series: Zero to Kubernetes — Part 1 · Part 2 · Part 3 · Part 4 · Part 5

In Part 1 we built a Kubernetes cluster with Terraform and kubeadm. Three nodes ready but empty. Today we turn it into a usable platform: external traffic with automatic TLS, storage for databases, namespaces that separate environments, and permissions that prevent a staging deploy from destroying production.

All with Terraform. Zero manual kubectl apply.

Why Terraform for Bootstrap and Not ArgoCD?

This is a question we get a lot. The answer lies in the dependency order:

┌─────────────────────────────────────────┐
│  Terraform (infrastructure + bootstrap) │
│  VMs, networks, kubeadm, namespaces,    │
│  ingress, cert-manager, storage, RBAC   │
│  ↓                                      │
│  Terraform installs ArgoCD              │
│  ↓                                      │
│  ArgoCD manages applications            │
│  (backend, frontends, DB)               │
└─────────────────────────────────────────┘

Terraform manages what ArgoCD needs to exist. Once ArgoCD is running, we hand off application control. If you try to use ArgoCD to bootstrap the ingress controller that ArgoCD needs to expose its own UI, you have a circular problem. Terraform breaks that cycle.

What We’re Building Today

┌──────────────────────────────────────────────┐
│                 INTERNET                      │
│                    │                          │
│         ┌──────────▼──────────┐               │
│         │  nginx ingress      │               │
│         │  (NodePort 30080)   │               │
│         │  + cert-manager     │               │
│         │  + Let's Encrypt    │               │
│         └──────────┬──────────┘               │
│                    │                          │
│    ┌───────────────┼───────────────┐          │
│    ▼               ▼               ▼          │
│ ┌────────┐  ┌──────────┐  ┌──────────────┐   │
│ │production│ │ staging  │  │infrastructure│   │
│ │namespace │ │namespace │  │ namespace    │   │
│ └────────┘  └──────────┘  └──────────────┘   │
│                                                │
│ Storage: local-path-provisioner (PVC)          │
│ RBAC: ServiceAccounts + Roles                 │
└──────────────────────────────────────────────┘

Step 1: Connect Terraform to the Cluster

We need the Kubernetes provider pointing to the cluster we created in Part 1:

provider "kubernetes" {
  host                   = "https://${hcloud_server.control_plane.ipv4_address}:6443"
  client_certificate     = base64decode(var.k8s_client_cert)
  client_key             = base64decode(var.k8s_client_key)
  cluster_ca_certificate = base64decode(var.k8s_ca_cert)
}

provider "helm" {
  kubernetes {
    host                   = "https://${hcloud_server.control_plane.ipv4_address}:6443"
    client_certificate     = base64decode(var.k8s_client_cert)
    client_key             = base64decode(var.k8s_client_key)
    cluster_ca_certificate = base64decode(var.k8s_ca_cert)
  }
}

Practical note: For this tutorial, we extract certificates from the kubeconfig. In production, use a secrets backend (Vault, AWS Secrets Manager) and inject them as environment variables.

Step 2: Namespaces — the Foundation of Everything

Namespaces are the logical isolation unit in Kubernetes. Without them, all resources live in default and chaos is guaranteed:

resource "kubernetes_namespace" "production" {
  metadata {
    name = "production"
    labels = {
      environment = "production"
      managed-by  = "terraform"
    }
  }
}

resource "kubernetes_namespace" "staging" {
  metadata {
    name = "staging"
    labels = {
      environment = "staging"
      managed-by  = "terraform"
    }
  }
}

resource "kubernetes_namespace" "infrastructure" {
  metadata {
    name = "infrastructure"
    labels = {
      environment = "infrastructure"
      managed-by  = "terraform"
    }
  }
}

resource "kubernetes_namespace" "argocd" {
  metadata {
    name = "argocd"
    labels = {
      environment = "infrastructure"
      managed-by  = "terraform"
    }
  }
}

The convention: production for real workloads, staging for pre-production testing, infrastructure for system components (ingress, cert-manager, storage), argocd for the GitOps tool.

Step 3: Storage — local-path-provisioner

Kubernetes needs a StorageClass that dynamically creates PersistentVolumes when a pod requests a PVC. On Hetzner we use local-path-provisioner (from Rancher), which creates volumes on the node’s local disk:

resource "helm_release" "local_path_provisioner" {
  name       = "local-path-provisioner"
  namespace  = kubernetes_namespace.infrastructure.metadata[0].name
  repository = "https://rancher.github.io/local-path-provisioner"
  chart      = "local-path-provisioner"
  version    = "0.0.28"

  set {
    name  = "storageClass.defaultClass"
    value = "true"
  }

  set {
    name  = "nodePathMap[0].node"
    value = "DEFAULT_PATH_FOR_NON_LISTED_NODES"
  }

  set {
    name  = "nodePathMap[0].paths[0]"
    value = "/var/local-path-provisioner"
  }
}

Why not Rook/Ceph? For a modest 3-node cluster, Ceph is overengineering. local-path-provisioner is perfect: data lives in /var/local-path-provisioner on the node where the pod runs. The limitation is that if that node dies, the volume is lost. For serious production, you’d later migrate to Longhorn (replication) or use cloud volumes (Hetzner volumes).

Step 4: nginx Ingress Controller — the Front Door

The ingress controller is the reverse proxy that receives external traffic and routes it to the correct Services inside the cluster. We use nginx, the most battle-tested in the ecosystem:

resource "helm_release" "ingress_nginx" {
  name       = "ingress-nginx"
  namespace  = kubernetes_namespace.infrastructure.metadata[0].name
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  version    = "4.11.2"

  set {
    name  = "controller.service.type"
    value = "NodePort"
  }

  set {
    name  = "controller.service.nodePorts.http"
    value = "30080"
  }

  set {
    name  = "controller.service.nodePorts.https"
    value = "30443"
  }

  set {
    name  = "controller.config.proxy-body-size"
    value = "50m"
  }

  set {
    name  = "controller.config.proxy-read-timeout"
    value = "60"
  }

  set {
    name  = "controller.config.proxy-send-timeout"
    value = "60"
  }
}

Important details:

  • We use NodePort because Hetzner doesn’t have a native LoadBalancer. The ingress listens on port 30080 (HTTP) and 30443 (HTTPS) on every node
  • proxy-body-size: 50m allows file uploads
  • Proxy timeouts at 60s prevent long requests (migrations, backups) from being cut off

Step 5: cert-manager — Automatic TLS with Let’s Encrypt

Nobody should be managing TLS certificates manually in 2026. cert-manager creates them, renews 30 days before expiry, and injects them as Secrets:

resource "helm_release" "cert_manager" {
  name       = "cert-manager"
  namespace  = kubernetes_namespace.infrastructure.metadata[0].name
  repository = "https://charts.jetstack.io"
  chart      = "cert-manager"
  version    = "1.16.1"

  set {
    name  = "installCRDs"
    value = "true"
  }

  set {
    name  = "prometheus.enabled"
    value = "false"
  }
}

After installing cert-manager, create the ClusterIssuers:

# Staging — for testing, no strong rate limits
resource "kubernetes_manifest" "cluster_issuer_staging" {
  manifest = {
    apiVersion = "cert-manager.io/v1"
    kind       = "ClusterIssuer"
    metadata = {
      name = "letsencrypt-staging"
    }
    spec = {
      acme = {
        server  = "https://acme-staging-v02.api.letsencrypt.org/directory"
        email   = var.letsencrypt_email
        privateKeySecretRef = {
          name = "letsencrypt-staging-account-key"
        }
        solvers = [{
          http01 = {
            ingress = { class = "nginx" }
          }
        }]
      }
    }
  }
}

# Production — real certificates
resource "kubernetes_manifest" "cluster_issuer_production" {
  manifest = {
    apiVersion = "cert-manager.io/v1"
    kind       = "ClusterIssuer"
    metadata = {
      name = "letsencrypt-production"
    }
    spec = {
      acme = {
        server  = "https://acme-v02.api.letsencrypt.org/directory"
        email   = var.letsencrypt_email
        privateKeySecretRef = {
          name = "letsencrypt-production-account-key"
        }
        solvers = [{
          http01 = {
            ingress = { class = "nginx" }
          }
        }]
      }
    }
  }

  depends_on = [helm_release.cert_manager]
}

Step 6: Minimum Viable RBAC

For this tutorial we need a ServiceAccount that applications will use:

resource "kubernetes_service_account" "apps" {
  metadata {
    name      = "apps-sa"
    namespace = kubernetes_namespace.production.metadata[0].name
  }
}

resource "kubernetes_role" "apps" {
  metadata {
    name      = "apps-role"
    namespace = kubernetes_namespace.production.metadata[0].name
  }

  rule {
    api_groups = [""]
    resources  = ["pods", "services", "configmaps", "secrets"]
    verbs      = ["get", "list", "watch"]
  }

  rule {
    api_groups = ["apps"]
    resources  = ["deployments"]
    verbs      = ["get", "list", "watch"]
  }
}

resource "kubernetes_role_binding" "apps" {
  metadata {
    name      = "apps-rolebinding"
    namespace = kubernetes_namespace.production.metadata[0].name
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.apps.metadata[0].name
  }

  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account.apps.metadata[0].name
    namespace = kubernetes_namespace.production.metadata[0].name
  }
}

This is intentionally restrictive: apps can read resources but not modify or create them. In Part 4 we’ll configure a specific ServiceAccount for the backend with Secret read permissions.

What You Learned in This Part

  • Namespaces as an isolation layer between environments
  • Why Terraform manages bootstrap and ArgoCD manages apps
  • nginx ingress controller exposed via NodePort
  • cert-manager + Let’s Encrypt for automatic TLS with renewal
  • local-path-provisioner as a dynamic StorageClass
  • Minimum viable RBAC with ServiceAccounts, Roles and RoleBindings
  • The ClusterIssuer → Ingress annotation → Secret TLS → ready certificate flow

In Part 3, we’ll deploy ArgoCD, configure GitOps with the app-of-apps pattern, and have our first automated deployment from GitHub.

At Guayoyo Tech, we don’t just write tutorials — we design and deploy architectures like this for real companies. Kubernetes, Terraform, ArgoCD, CI/CD pipelines, and the entire cloud-native ecosystem: we build it with you, no hype, no vendor lock-in. Want your infrastructure to look like what you just read? Talk to us free for 15 minutes. Your cluster, your cloud, your rules — we bring the engineering.

Zero to Kubernetes Part 1: Infrastructure with Terraform and kubeadm Bootstrap

K8s Terraform ArgoCD — Header

K8s Terraform ArgoCD — Header

Series: Zero to Kubernetes — Part 1 · Part 2 · Part 3 · Part 4 · Part 5

This is the first part of a 5-part guide where we’ll build, from absolute scratch, a full Kubernetes cluster with Terraform, deploy a backend with a database, two independent frontends, and manage everything with GitOps using ArgoCD.

This isn’t a “click here and hope” tutorial. We’re going to understand every piece. By the end of the series, you’ll have a real cluster running in the cloud, with automatic TLS, reproducible deployments, and all the code versioned in GitHub.

What we’ll build in Part 1:

  • 3 VMs (1 control-plane + 2 workers) on Hetzner Cloud with Terraform
  • Private network between VMs, firewalls with Kubernetes ports, SSH keys
  • containerd as the container runtime on each node
  • kubeadm init on the control-plane, kubeadm join on the workers
  • Calico as the CNI (Container Network Interface)
  • Everything automated via cloud-init — you’ll have a cluster in 5 minutes

Why kubeadm and not a managed service?

EKS, GKE, and AKS are excellent. But when you use a managed service, you don’t touch the control-plane, you don’t see etcd, you don’t configure the CNI, you don’t understand what happens when a node fails to join the cluster.

kubeadm is Kubernetes’ official tool for bootstrapping clusters. It’s what platform engineers use. By the end of this series, you’ll know exactly what each component does and you’ll be able to reproduce this cluster on any provider: Hetzner, AWS EC2, GCP Compute Engine, or your own bare metal.

Infrastructure Architecture

┌──────────────────────────────────────────────────────┐
│                  Hetzner Cloud                        │
│                                                      │
│  ┌─────────────────────┐  ┌──────────┐  ┌──────────┐ │
│  │   control-plane     │  │ worker-0 │  │ worker-1 │ │
│  │   (CX21, 2vCPU/4GB) │  │(CX21)    │  │(CX21)    │ │
│  │                     │  │          │  │          │ │
│  │ api-server          │  │ kubelet  │  │ kubelet  │ │
│  │ etcd                │  │ containerd│ │ containerd│ │
│  │ controller-manager  │  │ Calico   │  │ Calico   │ │
│  │ scheduler           │  │          │  │          │ │
│  │ kubelet             │  │          │  │          │ │
│  │ containerd          │  │          │  │          │ │
│  │ Calico              │  │          │  │          │ │
│  └────────┬────────────┘  └────┬─────┘  └────┬─────┘ │
│           │                    │             │       │
│           └────────────────────┼─────────────┘       │
│                                │                     │
│                    ┌───────────▼───────────┐         │
│                    │   Private Network     │         │
│                    │   (10.0.0.0/16)      │         │
│                    └───────────────────────┘         │
│                                                      │
│   Firewall: SSH(22) + k8s API(6443) + NodePorts      │
│   + Calico networking (179/tcp, 4789/udp)            │
└──────────────────────────────────────────────────────┘

Terraform Project Structure

Create an infra/ directory in your monorepo:

infra/
├── main.tf              # Providers, main resources
├── variables.tf         # Input variables
├── outputs.tf           # Outputs: IPs, kubeconfig
├── terraform.tfvars     # Values (don't commit secrets)
├── versions.tf          # Provider versions
├── cloud-init/
│   ├── control-plane.yaml
│   └── worker.yaml
└── .gitignore

Step 1: Configure Providers and Variables

versions.tf — Lock versions for reproducibility:

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.47"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }
}

main.tf — The skeleton:

provider "hcloud" {
  token = var.hcloud_token
}

# Private network
resource "hcloud_network" "k8s" {
  name     = "k8s-network"
  ip_range = "10.0.0.0/16"
}

resource "hcloud_network_subnet" "k8s" {
  network_id   = hcloud_network.k8s.id
  type         = "cloud"
  network_zone = "eu-central"
  ip_range     = "10.0.1.0/24"
}

Step 2: SSH Keys

resource "tls_private_key" "k8s" {
  algorithm = "ED25519"
}

resource "hcloud_ssh_key" "k8s" {
  name       = "k8s-cluster-key"
  public_key = tls_private_key.k8s.public_key_openssh
}

Terraform stores the private key in state. Extract it with terraform output -raw ssh_private_key.

Step 3: Firewall with Exact Kubernetes Ports

Don’t open everything. Kubernetes needs specific ports:

resource "hcloud_firewall" "k8s" {
  name = "k8s-firewall"

  # SSH from any IP (use your fixed IP for production)
  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "22"
    source_ips = ["0.0.0.0/0"]
  }

  # Kubernetes API (control-plane)
  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "6443"
    source_ips = ["0.0.0.0/0"]
  }

  # NodePort range (30000-32767 for ingress and services)
  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "30000-32767"
    source_ips = ["0.0.0.0/0"]
  }

  # Calico BGP (179/tcp) and VXLAN (4789/udp)
  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "179"
    source_ips = ["10.0.0.0/16"]
  }
  rule {
    direction  = "in"
    protocol   = "udp"
    port       = "4789"
    source_ips = ["10.0.0.0/16"]
  }

  # Full internal communication between cluster nodes
  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "10250-10256"
    source_ips = ["10.0.0.0/16"]
  }
}

What each port is for:

  • 6443 → API server. kubectl and kubeadm talk to this endpoint
  • 30000-32767 → NodePort range. The nginx ingress exposes itself here until we have a LoadBalancer
  • 179/tcp → Calico BGP to share routes between nodes
  • 4789/udp → Calico VXLAN to encapsulate pod traffic between nodes
  • 10250 → Kubelet API (logs, exec, metrics)
  • 10256 → kube-proxy health check
  • 10257-10259 → Controller manager and scheduler (control-plane only)

Step 4: The VMs — the Heart of the Cluster

# Control-plane
resource "hcloud_server" "control_plane" {
  name        = "k8s-control-plane"
  server_type = "cx21"
  image       = "ubuntu-24.04"
  location    = "nbg1"
  ssh_keys    = [hcloud_ssh_key.k8s.id]
  firewall_ids = [hcloud_firewall.k8s.id]

  network {
    network_id = hcloud_network.k8s.id
    ip         = "10.0.1.10"
  }

  user_data = templatefile("${path.module}/cloud-init/control-plane.yaml", {
    pod_network_cidr = "192.168.0.0/16"
  })

  depends_on = [hcloud_network_subnet.k8s]
}

# Workers
resource "hcloud_server" "worker" {
  count = 2

  name         = "k8s-worker-${count.index}"
  server_type  = "cx21"
  image        = "ubuntu-24.04"
  location     = "nbg1"
  ssh_keys     = [hcloud_ssh_key.k8s.id]
  firewall_ids = [hcloud_firewall.k8s.id]

  network {
    network_id = hcloud_network.k8s.id
    ip         = "10.0.1.${20 + count.index}"
  }

  user_data = templatefile("${path.module}/cloud-init/worker.yaml", {
    control_plane_ip = "10.0.1.10"
  })

  depends_on = [hcloud_server.control_plane]
}

Step 5: cloud-init — The Magic That Automates Everything

This is where the actual Kubernetes installation happens. cloud-init runs these scripts on the first boot of each VM.

cloud-init/control-plane.yaml:

#cloud-config
hostname: control-plane
package_update: true
package_upgrade: true

packages:
  - containerd
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg

write_files:
  - path: /etc/containerd/config.toml
    permissions: '0644'
    content: |
      version = 2
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
        runtime_type = "io.containerd.runc.v2"
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
          SystemdCgroup = true

  - path: /etc/modules-load.d/k8s.conf
    content: |
      overlay
      br_netfilter

  - path: /etc/sysctl.d/k8s.conf
    content: |
      net.bridge.bridge-nf-call-iptables  = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward                 = 1

runcmd:
  - modprobe overlay
  - modprobe br_netfilter
  - sysctl --system
  - swapoff -a
  - sed -i '/swap/d' /etc/fstab
  - systemctl restart containerd
  - systemctl enable containerd

  # Install kubeadm, kubelet, kubectl
  - |
    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" > /etc/apt/sources.list.d/kubernetes.list
    apt-get update
    apt-get install -y kubelet kubeadm kubectl
    apt-mark hold kubelet kubeadm kubectl

  # Initialize the cluster
  - |
    kubeadm init 
      --pod-network-cidr=${pod_network_cidr} 
      --apiserver-advertise-address=10.0.1.10 
      --upload-certs 
      | tee /root/kubeadm-init.log

  # Configure kubeconfig for root
  - mkdir -p /root/.kube
  - cp /etc/kubernetes/admin.conf /root/.kube/config

  # Install Calico
  - |
    kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/tigera-operator.yaml --kubeconfig=/etc/kubernetes/admin.conf
    kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/custom-resources.yaml --kubeconfig=/etc/kubernetes/admin.conf

  # Generate join token for workers
  - kubeadm token create --print-join-command > /root/join-command.sh
  - chmod 644 /root/join-command.sh

final_message: "Control-plane bootstrap completed in $UPTIME seconds"

cloud-init/worker.yaml:

#cloud-config
hostname: worker
package_update: true
package_upgrade: true

packages:
  - containerd
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg

write_files:
  - path: /etc/containerd/config.toml
    permissions: '0644'
    content: |
      version = 2
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
        runtime_type = "io.containerd.runc.v2"
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
          SystemdCgroup = true

  - path: /etc/modules-load.d/k8s.conf
    content: |
      overlay
      br_netfilter

  - path: /etc/sysctl.d/k8s.conf
    content: |
      net.bridge.bridge-nf-call-iptables  = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward                 = 1

runcmd:
  - modprobe overlay
  - modprobe br_netfilter
  - sysctl --system
  - swapoff -a
  - sed -i '/swap/d' /etc/fstab
  - systemctl restart containerd
  - systemctl enable containerd

  # Install kubeadm, kubelet, kubectl
  - |
    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" > /etc/apt/sources.list.d/kubernetes.list
    apt-get update
    apt-get install -y kubelet kubeadm kubectl
    apt-mark hold kubelet kubeadm kubectl

final_message: "Worker bootstrap completed in $UPTIME seconds. Run join command after control-plane init."

Step 6: Joining Workers to the Cluster

Since the join command is generated on the control-plane during bootstrap, workers need to retrieve it after. Use a null_resource in Terraform or do it manually the first time:

# From your local machine, after terraform apply:
scp root@<control-plane-ip>:/root/join-command.sh .
scp join-command.sh root@<worker-0-ip>:/root/
ssh root@<worker-0-ip> "bash /root/join-command.sh"
# Repeat for worker-1

In production, this is automated with Vault or a shared S3 bucket. For the tutorial, this 2-minute manual step is enough and lets you see exactly what happens.

Step 7: Verification

$ terraform apply
# ... (2-3 minutes to create VMs + cloud-init)

$ ssh root@<control-plane-ip>

root@control-plane:~# kubectl get nodes
NAME               STATUS   ROLES           AGE   VERSION
k8s-control-plane  Ready    control-plane   5m    v1.31.0
k8s-worker-0       Ready    <none>          2m    v1.31.0
k8s-worker-1       Ready    <none>          2m    v1.31.0

root@control-plane:~# kubectl get pods -A
NAMESPACE          NAME                                      READY   STATUS    RESTARTS   AGE
calico-system      calico-kube-controllers-xxxxx             1/1     Running   0          4m
calico-system      calico-node-xxxxx                         1/1     Running   0          4m
calico-system      calico-node-xxxxx                         1/1     Running   0          2m
calico-system      calico-node-xxxxx                         1/1     Running   0          2m
kube-system        coredns-xxxxx                             1/1     Running   0          5m
kube-system        coredns-xxxxx                             1/1     Running   0          5m
kube-system        etcd-control-plane                        1/1     Running   0          5m
kube-system        kube-apiserver-control-plane              1/1     Running   0          5m
kube-system        kube-controller-manager-control-plane     1/1     Running   0          5m
kube-system        kube-proxy-xxxxx                          1/1     Running   0          5m
kube-system        kube-scheduler-control-plane              1/1     Running   0          5m

Three nodes Ready, all system pods running. Cluster is live.

Terraform Outputs

output "control_plane_ip" {
  value = hcloud_server.control_plane.ipv4_address
}

output "worker_ips" {
  value = [for w in hcloud_server.worker : w.ipv4_address]
}

output "ssh_private_key" {
  value     = tls_private_key.k8s.private_key_pem
  sensitive = true
}

Real Costs

Resource Monthly Price
3× CX21 (2vCPU, 4GB RAM, 40GB SSD) €11.97
Private network €0.00
Firewall €0.00
Traffic (20TB included) €0.00
Total ≈ €12/month

Under $13 USD per month for a real Kubernetes cluster with a dedicated control-plane. Compare with EKS charging $0.10/hour just for the control-plane ($72/month) plus the VMs separately.

What You Learned in This Part

  • Professional Terraform project structure for Kubernetes
  • Exact ports Kubernetes needs (and why each one)
  • cloud-init as a bootstrap automation tool
  • containerd as the container runtime (not Docker — Kubernetes deprecated Docker in v1.24)
  • kubeadm init and critical flags (pod-network-cidr, advertise-address)
  • Calico as the CNI for pod networking between nodes
  • The control-plane + workers concept and what runs on each

In Part 2, we’ll take this empty cluster and turn it into a production-ready platform: namespaces, ingress controller with automatic TLS, persistent storage, and RBAC. All with Terraform, zero manual kubectl apply.

At Guayoyo Tech, we design and deploy Kubernetes architectures like this for real companies. Terraform, kubeadm, GitOps, CI/CD — we build it with you, no hype, no vendor lock-in. Want your infrastructure to look like what you just read? Talk to us free for 15 minutes. Your cluster, your cloud, your rules — we bring the engineering.

7 Developer Tools I Actually Pay For (And 3 That Are Free)

I’m cheap. I use Arch Linux, Neovim, and open-source everything. But some tools are worth paying for. Here’s my list.

Paid (worth every đồng)

1. DeepSeek API (~$10/month)

My AI coding agent runs on DeepSeek v4 Pro. Cheaper than GitHub Copilot ($10 vs $19/month). Better for Vietnamese context. I use it through Hermes Agent to build boilerplates, write tests, and debug.

DeepSeek Platform — $0.14/M input tokens

2. Railway (~$5/month)

Full-stack hosting that just works. Docker Compose one-click deploy. Built-in MongoDB + Redis. No YAML hell. No Kubernetes.

Railway — free $5 credit

3. Gumroad (free + 10% fee)

I sell my NestJS/Next.js starter kits here. Free to list. They take 10% only on sales. Built-in email marketing to past customers. PayPal + card payments handled automatically.

Gumroad — free to start

4. Namecheap (~$12/year)

Domains. .dev, .app, .com — pick whatever. I use .gumroad.com for now but will switch when there’s more revenue.

Namecheap — domains from $1/year

Free (surprisingly good)

5. Hermes Agent (free, open-source)

AI agent that runs in my terminal. Builds projects, manages Obsidian, schedules cron jobs. It wrote 80% of my starter kits while I was AFK. Works with DeepSeek, OpenAI, Anthropic, local models.

Hermes Agent

6. Obsidian (free)

My second brain. PARA method. Everything I know about NestJS/Next.js/Go lives here. Free. Offline. No vendor lock-in.

Obsidian

7. Neovim (free)

0.12 “Stale Earth” with native LSP. No VS Code. No JetBrains. Instant startup. Full control.

Neovim

Not worth it (tried, cancelled)

Tool Why I cancelled
GitHub Copilot $19/month for worse code than DeepSeek
Notion Slow, online-only, bloated
Vercel Pro Hobby tier is enough for my scale
Adobe Creative Cloud GIMP + Inkscape do the job
Grammarly Premium Free tier catches 90%

Total monthly cost

DeepSeek API:  ~$10
Railway:        ~$5
Domain:         ~$1/month (annual)
Gumroad:        free until sale
─────────────────────
Total:         ~$16/month

$16/month to run a full dev setup with AI coding, hosting, and a storefront. Not bad.

Building a NestJS project? Save 7 hours of setup with my pre-configured starter kit.

CI Failures Aren’t the Bottleneck — The Debugging After Them Is

The Build Is Red. Now What?

CI pipelines exist to catch problems early. And they do — they just don’t tell you much about what actually went wrong.

When a build fails, developers don’t spend their time fixing the problem. They spend it finding the problem. Expanding collapsed log sections, scrolling past irrelevant output, trying to identify whether the first error caused everything else or if there are multiple independent issues. That’s not engineering. That’s archaeology.

Industry surveys consistently show that development teams spend 25-30% of their time dealing with CI/CD issues. Research conducted in collaboration with Cambridge Judge Business School puts a finer point on it: 26% of developer time goes to reproducing and fixing failing tests — roughly 620 million developer hours per year across the industry.

That number should make engineering leaders uncomfortable.

The Tooling Gap Is Real

The experience of defining CI pipelines has improved dramatically. GitHub Actions and GitLab CI are flexible, well-documented, and widely adopted. But the experience after a failure hasn’t kept pace.

When a build breaks, the developer needs to answer a simple question: did my change cause this, and if so, which part? Getting to that answer usually means manually cross-referencing log output with your diff, checking if the failure existed on main before your branch, and ruling out flaky tests.

Speaking of flaky tests — recent production benchmarks show that roughly a third of CI failures have no underlying code change at all. They’re triggered by infrastructure noise or timing issues. Teams rerun entire suites to work around them, wasting compute and developer focus.

The Cost of Log Archaeology

For a team of 20 developers, CI pipeline failures can add up to roughly $1 million in lost productivity per year. Beyond the dollar figure, there’s a cultural cost. When debugging CI is painful, teams stop investigating intermittent failures. They hit rerun and move on. Flakiness becomes background noise, and “the build is red” stops being a useful signal.

This creates what one analysis called “learned helplessness around test failures.” Nobody owns CI quality. Nobody tracks flake rates. What starts as a one-off rerun becomes standard practice.

Close the Gap

The fix isn’t a single tool — it’s treating the post-failure experience as seriously as the pipeline definition itself. Better log formatting. Automatic failure categorization. Mapping errors back to the specific lines changed in a PR.

This is one of the reasons we built CI Failure Intelligence into Code Board — AI-driven analysis that takes failing CI logs, maps errors to your diff, and identifies root causes with suggested fixes. But regardless of tooling, the principle holds: the gap between “build failed” and “here’s what to fix” is where engineering hours go to die.

CI should surface signal, not create busywork. If your developers are spending more time reading logs than writing code, the pipeline isn’t serving its purpose.

I built a cinematic dark luxury coffee shop template with Next.js, GSAP & Three.js — here’s how (and it’s on Gumroad)

Why I built this

Most coffee shop websites look the same — a hero image, a menu, a contact form.

I wanted to build something different. Something that feels like walking into a dimly lit, jazz-filled espresso bar at midnight.

So I built Noir Espresso — a cinematic, dark-luxury coffee shop template using Next.js, Tailwind CSS, GSAP, and Three.js.

🔗 Live demo: https://noir-espresso.vercel.app
🛒 Get it on Gumroad ($29): https://4701123641931.gumroad.com/l/fkrwq

What’s inside the template

Here’s what makes Noir Espresso different from a generic coffee shop template:

1. Three.js animated background

Instead of a static hero image, the homepage features a subtle 3D scene built with Three.js. It adds depth and atmosphere without killing performance.

// Minimal Three.js setup used in the hero section
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer({ alpha: true, antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);

2. GSAP scroll animations

Every section reveals with smooth, cinematic GSAP animations triggered on scroll. No janky CSS transitions — everything feels intentional.

gsap.from(".hero-text", {
  scrollTrigger: ".hero-text",
  y: 60,
  opacity: 0,
  duration: 1.2,
  ease: "power3.out"
});

3. Next.js App Router + Tailwind CSS

The entire project uses the Next.js App Router for clean routing and Tailwind CSS for rapid, utility-first styling. The dark palette is built around custom Tailwind config.

// tailwind.config.js (excerpt)
extend: {
  colors: {
    noir: {
      bg: "#0a0a0a",
      accent: "#c9a96e",
      text: "#f5f0e8",
    }
  }
}

4. Fully responsive

Tested on mobile, tablet, and desktop. The cinematic feel holds even on small screens.

Tech stack

Tool Purpose
Next.js 14 Framework (App Router)
Tailwind CSS Styling
GSAP + ScrollTrigger Scroll animations
Three.js 3D hero background
Vercel Deployment

Who is this for?

This template is perfect for:

  • Freelancers building sites for coffee shops, bars, or luxury brands
  • Developers who want a premium-looking starter for client projects
  • Anyone learning how to combine GSAP + Three.js with Next.js in a real project

What’s included ($29)

  • Full Next.js source code
  • GSAP + Three.js integration
  • Custom Tailwind dark theme
  • App Router structure
  • Responsive design
  • MIT License — use it in client projects

What I learned building this

Combining GSAP and Three.js inside a Next.js app requires some care:

  1. Three.js needs "use client" — the renderer depends on window, so keep it in a Client Component.
  2. GSAP ScrollTrigger needs cleanup — always return a cleanup function in useEffect to avoid duplicate triggers.
  3. Tailwind + dark themes — extend the config with semantic color names (like noir-bg) so you’re not writing bg-zinc-950 everywhere.

Links

Also launching today on Product Hunt — would love your support! 🚀

Drop any questions in the comments — happy to help you customise it for your next project.

Why we built Replay before everything else

A note on building reliability infrastructure for AI agents — and why post-incident debugging matters more than pre-flight validation.
A few weeks ago I started building SafeRun — inline reliability infrastructure for AI agents in production. The temptation, when you’re building something in the agent reliability space, is to lead with validation. Block the bad action before it happens. Stop the runaway loop. Enforce the policy.
These are real features. SafeRun ships all of them. But they’re not the first thing we built. The first thing we built was Replay.
Here’s why.
The failure mode no one talks about
Most teams shipping AI agents into production discover the same problem after their first bad incident. The agent did something it shouldn’t have. They go to investigate. And they find that they can’t reproduce what happened.
The traces are flat. The logs don’t show the model’s reasoning between tool calls. The arguments to the failed call aren’t fully captured. The retrieved context that informed the decision is missing. The agent’s plan, if it had one, isn’t anywhere.
So the engineer does what engineers do. They start rerunning the agent, trying to recreate the conditions that led to the failure. The agent is non-deterministic. The conditions change. They spend a weekend trying to reproduce one bad action.
This is the universal pain. I’ve talked to maybe twenty engineers shipping agents in production, and every single one of them has lived this. Not “heard about it.” Lived it.
Why observability tools don’t solve this
LangSmith, Langfuse, Helicone, Arize, and the broader observability category do something genuinely useful: they tell you what happened. But “what happened” is a description, not a reproduction. You can read a trace. You can’t re-execute it.
Replay is different. Replay means capturing the complete state of an agent run with enough fidelity to step through it frame by frame after the fact, see the exact arguments to each tool call, see the model’s reasoning between calls, see the retrieved context at each decision point, see the policy that evaluated each action, see the decision that was returned.
This is a different engineering problem than logging. It requires deterministic state capture. It requires decision-time context snapshotting separately from outcome context. It requires versioning every policy and every rule and every classifier that participated in a decision. We built this first because everything else depends on it.
The four-step loop, and why Replay is the foundation
SafeRun’s product loop is Replay → Understand → Create Rule → Prevent.
You can’t understand a failure you can’t reproduce.
You can’t create a rule to prevent a failure you don’t understand.
You can’t prevent a category of failure if your rule was created against an incomplete picture of what happened.
The order matters. Build Replay first, and everything else compounds. Build prevention first, and your rules will be flat patches against failures you don’t fully see.
The Stripe boolean problem
Here’s the failure that taught me Replay matters more than any other layer.
An agent issues a Stripe refund instead of a charge because a single boolean flipped in the agent’s planning step. The call shape is correct. The schema passes. Type-checking passes. Most observability tools log a successful refund and move on.
The engineer notices the next morning when the customer complains. They go to investigate. They have a trace. The trace tells them “Stripe refund issued, amount $4,500, customer cus_9281.” That’s true. It tells them nothing about why.
With Replay, they can step back through the agent’s decision frame by frame. See the user’s request was actually a charge. See the agent’s planning step had is_refund: false. See that somewhere between the plan and the tool call, the boolean flipped. See whether it was a model hallucination, a prompt injection, a code bug, or a retrieved-context misinterpretation.
Now they know what to do. They can write a prevention rule. They can fix the upstream cause. They can ship a fix that actually prevents recurrence, instead of patching the symptom.
This is what Replay enables. None of the rest of the product matters without it.
What we shipped, in order:

Phase 0: Working prototype with six failure simulations, including the Stripe boolean problem.
Phase 1: Persistent backend on Supabase. Replays survive page reload, browser close, account switch.
Phase 2: POST /v1/check-action API with sub-50ms p95 latency. Decision-time context snapshotting (inputs, retrieved context, external state, policy version, evaluator model version) captured synchronously, persisted asynchronously. The replay is built from the decision, not assembled after.
Phase 3: Python and TypeScript SDKs. Three-line install. @guard decorator wraps any tool call.
Phase 4: Intent Guard — catches valid-shape, wrong-intent tool calls. The Stripe boolean problem from above. Visible confidence scores, threshold calibration as a product surface, feedback loop closes back into recalibration.
Phase 5: Multi-tenant, project-scoped API keys, environment separation (dev logs, staging warns, production blocks), replay redaction, audit log, rule versioning.
Phase 6: Design partner onboarding, Prevention Impact Dashboard.
Phase 7: Self-hosted/VPC, SSO/SAML, audit log export, SOC 2 readiness, SafeRun as an MCP-callable tool.

The whole roadmap exists in service of the Replay layer. Every phase compounds on the previous one. Every feature ladders to Replay → Understand → Create Rule → Prevent.
What’s next
We’re onboarding the first design partners now. Engineering teams shipping AI agents into production — agents that move real money, modify real customer data, talk to real customers. Free during the partnership in exchange for honest feedback.
If you’re shipping agents and want to be one of the first teams running SafeRun in production, get in touch. saferun.dev.
If you’re shipping agents and don’t want to be a design partner but want to try the SDK, it’s pip install saferun and three lines.
Either way, the bet is this: replay the failure, prevent the next one. The first one always happens. The second one is the company’s choice.

What is Gemini 3.5 Flash? Google’s New Fast Frontier Model Explained

Gemini 3.5 Flash overview

Google shipped Gemini 3.5 Flash on May 19, 2026. It is the fast, low-cost model in the Gemini 3.5 family and the only 3.5 variant available today. Gemini 3.5 Pro is announced for June 2026, but Flash is the model you can actually build with right now.

Try Apidog today

Gemini 3.5 Flash targets production workloads that need speed, long context, and reliable tool use: agent loops, terminal automation, multi-file coding, multimodal document analysis, and streaming chat. Google positions it as roughly 4× faster on output tokens than other frontier models and less than half the cost per task.

This guide covers what changed, where Flash performs well, how to access it, and how to test Gemini API endpoints with Apidog.

Quick facts about Gemini 3.5 Flash

Item Details
Release date May 19, 2026
Variant Gemini 3.5 Flash
Pro availability Gemini 3.5 Pro announced for June 2026
Context window 1M input tokens, 64K output tokens
Modalities Text, images, code, graphics generation
API model name gemini-3.5-flash
Speed ~4× faster output tokens/second than other frontier models
Cost Less than half the cost of comparable frontier models for agentic tasks
Access Gemini app, AI Mode in Search, Google Antigravity, Gemini API, AI Studio, Android Studio, Gemini Enterprise

Headline benchmark numbers from Google:

  • Terminal-Bench 2.1: 76.2%
  • CharXiv Reasoning: 84.2%
  • MCP Atlas: 83.6%
  • GDPval-AA: 1656 Elo

For cost details, free-tier limits, and usage scenarios, see the Gemini 3.5 Flash pricing guide.

Gemini 3.5 Flash overview

What changed from Gemini 3 and 3.1

Gemini 3.5 Flash builds on Gemini 3 Flash and Gemini 3.1 Pro with five practical upgrades.

1. Better long-running agent execution

Flash is designed for longer chains of work. In practice, that matters when your agent needs to:

  • Call tools in the correct order
  • Recover from tool errors
  • Dispatch work to subagents
  • Keep state across a long task

2. Stronger coding output

The useful upgrade is not only code completion. Flash is better suited for workflows like:

  • Multi-file refactors
  • CLI-based development tasks
  • Long-horizon debugging
  • Repository-level code changes

3. Built-in graphics generation

Flash can generate inline diagrams, SVG, and interactive web UI output directly. That reduces the need to route every visual task through a separate image model.

4. Faster streaming output

Google claims roughly 4× the output-token speed of other frontier models. If you are building a streaming UI, this affects implementation details:

  • Render partial output incrementally
  • Add backpressure if your frontend cannot render fast enough
  • Measure time-to-first-token and full completion latency separately

5. Wider safety guardrails

Google describes stronger safeguards for cyber and CBRN misuse, plus interpretability tools that explain why the model refused or rerouted a request.

The overall direction is clear: Flash is optimized for production agent workloads, similar to how OpenAI and Anthropic position GPT-5.5 and Claude Opus 4.7.

Gemini 3.5 Flash comparison

Gemini 3.5 Flash benchmarks

Google’s published benchmark table shows Flash performing especially well on long-context, tool-heavy, and multimodal reasoning tasks.

Benchmark What it tests Gemini 3.5 Flash
Terminal-Bench 2.1 Long-horizon CLI workflows 76.2%
MCP Atlas Multi-tool coordination 83.6%
CharXiv Reasoning Chart and diagram interpretation 84.2%
GDPval-AA General agentic value 1656 Elo
MRCR v2, 1M context Long-context retrieval Top of Google’s table

Flash is strongest in:

  • Chart reasoning
  • Agentic multi-tool workflows
  • Long-context retrieval
  • Cost-sensitive long-running tasks

It does not dominate every category. For pure SWE-Bench Verified-style bug fixing, Opus 4.7 and GPT-5.5 remain competitive. If your workload is single-shot bug fixing, benchmark before switching. If your workload is long agent execution at lower cost, Flash is more compelling.

For a workload-by-workload comparison, see Gemini 3.5 Flash vs GPT-5.5 vs Opus 4.7.

The Gemini 3.5 model family

Gemini 3.5 Flash

Flash is available now through:

  • AI Studio
  • Gemini API
  • Gemini app
  • AI Mode in Search
  • Google Antigravity
  • Android Studio
  • Gemini Enterprise

Reported launch-day pricing is around:

  • $1.50 per 1M input tokens
  • $9.00 per 1M output tokens

That is higher than last year’s 3.1 Flash-Lite, but still cheaper than Pro-tier competitors. For batch mode, cached input, and Vertex rates, see the full pricing guide.

Use Flash when you need:

  • High-throughput agent loops
  • Vision-heavy chart or document understanding
  • Low-latency calls from Apidog test scripts
  • Streaming chat responses
  • 1M-token document analysis without chunking

Gemini 3.5 Pro

Gemini 3.5 Pro is announced but not yet shipping. Google positions it as the flagship model for deeper agentic work, multi-hour autonomous tasks, and top leaderboard performance.

Until Pro becomes available, Flash is the practical default for developers who want to start building on the 3.5 family now.

Gemini Nano

Google did not ship a Gemini 3.5 Nano variant. On-device inference still uses the 3.1 Flash-Lite line. A 3.5 Nano announcement may arrive closer to the next Pixel cycle.

Where you can use Gemini 3.5 Flash

Six launch surfaces are available:

  1. Gemini app: global rollout for free and paid tiers
  2. AI Mode in Google Search: answers and follow-ups
  3. Google Antigravity: Google’s agent platform for end-user automation
  4. Gemini API: developer access through AI Studio
  5. Android Studio: IDE-level coding assistance for Android developers
  6. Gemini Enterprise + Agent Platform: managed agent runtime for organizations

The newest surface is Gemini Spark, a personal agent that runs continuously on your account. Spark uses Flash and connects to Gmail, Calendar, and Drive context.

Gemini Spark

Search also adds information agents: small autonomous helpers that track topics and pull updates without requiring you to re-query.

How to start using Gemini 3.5 Flash

You have four main paths.

1. Use the Gemini app for manual testing

Go to gemini.google.com, select 3.5 Flash from the model selector, and test prompts manually.

This is useful for:

  • Prompt exploration
  • Research
  • Writing tasks
  • Code sketching
  • Image analysis

It is not enough for production validation because you cannot reliably test API payload shape, streaming behavior, or tool-call schemas from the chat UI alone.

2. Use Google AI Studio for free API access

Go to ai.google.dev, sign in, and create an API key. Flash is available on the free tier at roughly 1,500 requests per day at launch.

The basic implementation flow is:

export GEMINI_API_KEY="YOUR_API_KEY"
export GEMINI_MODEL="gemini-3.5-flash"

Then call the model from your SDK or REST client.

If you have used the Google Gemini API before, the pattern is the same:

  1. Create an API key
  2. Set GEMINI_API_KEY
  3. Use the model name gemini-3.5-flash
  4. Send your request
  5. Capture latency, token usage, and response shape

For setup details, see the free Gemini API key guide or the Flash-specific free guide.

3. Use the Gemini API in production

For production, use the same model name with a billed account:

gemini-3.5-flash

A minimal request shape looks like this:

{
  "model": "gemini-3.5-flash",
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Summarize this API error log and suggest the next debugging step."
        }
      ]
    }
  ]
}

When moving from prototype to production, validate:

  • Request payload shape
  • Response schema
  • Streaming chunks
  • Tool-call arguments
  • Error responses
  • Token usage
  • Latency under load

For complete Python, Node, curl, streaming, tool-use, and multimodal examples, see How to Use the Gemini 3.5 Flash API.

When you wire Flash into your stack, test the API like any other dependency. Apidog can help you inspect the full request/response cycle for Flash REST and streaming endpoints in one workspace, including tool calls and multimodal payloads.

4. Use Gemini Enterprise for managed deployment

For organizations, Gemini Enterprise Agent Platform packages Flash with enterprise controls such as audit logs, data residency, and managed runtime features.

A practical rollout path is:

  1. Prototype in AI Studio
  2. Validate API behavior in a test workspace
  3. Build an evaluation suite
  4. Run Flash against production-like workloads
  5. Move governed workloads to Gemini Enterprise if required

What Gemini 3.5 Flash is good at

Long agent loops at lower cost

Flash is designed for multi-step tasks with tool calls. The MCP Atlas score of 83.6% is the strongest signal here.

Use it for workflows like:

User request
  -> plan task
  -> call search/tool
  -> inspect result
  -> call second tool
  -> update plan
  -> produce final answer

When testing this class of workload, measure:

  • Tool selection accuracy
  • Tool-call order
  • Recovery from failed tool calls
  • Repetition or loop behavior
  • Total cost per completed task

Chart and document reasoning

CharXiv Reasoning at 84.2% makes Flash a strong fit for reports, PDFs, charts, and diagrams.

Example workloads:

  • Extract values from charts
  • Summarize financial reports
  • Compare tables across documents
  • Explain diagrams
  • Turn visual data into structured JSON

Interactive UI generation

Flash can generate HTML, SVG, diagrams, and interactive UI output directly. For developers, the useful workflow is:

  1. Ask for a component or dashboard
  2. Request HTML/CSS/JS output
  3. Run it locally
  4. Feed errors back into the model
  5. Repeat until usable

The graphics quality is a visible upgrade over 3.1 Flash-Lite.

Cost-sensitive production workloads

Google frames Flash as less than half the cost of other frontier models for agentic tasks. For long-running agents, per-task cost matters more than simple per-token cost.

Compare models using:

total_task_cost =
  input_tokens_cost
  + output_tokens_cost
  + retry_cost
  + tool_call_cost
  + failed_task_cost

Flash may be attractive compared with Opus 4.7 or GPT-5.5 when the workload is agent-heavy. See the pricing breakdown before shifting traffic.

What Flash is still not great at

No model is a universal default. Watch these areas.

  • Pure SWE-Bench Verified-style work: Opus 4.7’s 87.6% still leads on isolated bug-fix benchmarks. If your KPI is single-issue resolution, test carefully.
  • Voice: Gemini’s voice stack is separate. For that workload, compare with Grok Voice vs GPT-Realtime.
  • Tool ecosystem maturity: OpenAI and Anthropic have a head start in third-party adapters. Google is catching up with Antigravity, but the ecosystem is younger.

How to test Gemini 3.5 Flash properly

Before putting Flash into production, test two things:

  1. Response shape stability
  2. Tool-call correctness

A small evaluation harness is enough to start.

Step 1: Pin representative prompts

Create a fixed prompt set that reflects real production traffic.

Example categories:

- Short user questions
- Long-context document tasks
- Tool-calling tasks
- Multimodal tasks
- Coding tasks
- Refusal/safety edge cases
- Streaming responses

Step 2: Run Flash and your current model side by side

For every prompt, store:

{
  "prompt_id": "tool_call_001",
  "model": "gemini-3.5-flash",
  "latency_ms": 0,
  "input_tokens": 0,
  "output_tokens": 0,
  "status": "pass",
  "notes": ""
}

Step 3: Score task success

Do not score only on “good-looking output.” Score on downstream success.

Useful dimensions:

  • Did the model call the right tool?
  • Were arguments valid?
  • Did the response match your schema?
  • Did the task complete?
  • How many retries were needed?
  • What was the total cost?

Step 4: Watch for schema drift

If you use function calling, validate tool arguments strictly.

Example checklist:

- Required fields present
- Field types correct
- Enum values valid
- No unexpected nested structures
- No missing tool call when one is required

For steps 1 and 3, Apidog can provide a recorded test suite for Flash API endpoints, including streaming. You can replay the same prompts across model versions and diff outputs. Download Apidog if you want to set this up locally.

Migration tips from Gemini 3.1 to 3.5 Flash

If you already use Gemini 3.1, migration is usually a one-line model string change:

- gemini-3.1-flash
+ gemini-3.5-flash

Before routing production traffic, check the following.

Area What to verify
Token budgets 1M input / 64K output stays the same
Tool schemas Existing function definitions should carry over
Streaming UI Faster output may require frontend throttling
Pricing Re-baseline using the Flash pricing guide
Safety behavior Rerun red-team and refusal-pattern tests

For deeper SDK notes, see the Google Gemini 3 API guide.

FAQ

When is Gemini 3.5 Pro available?

Google announced “rolling out next month” on May 19, 2026. Expect general availability in June 2026 across AI Studio, Gemini API, and Gemini Enterprise. Until then, Flash is the only Gemini 3.5 variant you can call.

Is Gemini 3.5 Flash free to use?

Yes, with daily quotas. The Gemini app standard tier and AI Studio API keys both provide Flash access without payment. See the Flash free guide and Get Free Unlimited Gemini API for available free paths.

Does Gemini 3.5 Flash support function calling?

Yes. Tool calling and subagent dispatch are first-class features. Google’s MCP Atlas score of 83.6% is the benchmark signal.

How does Flash compare to Opus 4.7 and GPT-5.5?

Flash leads on cost, output speed, and chart reasoning. Opus 4.7 still edges ahead on SWE-Bench Pro and long-form writing. GPT-5.5 wins on token efficiency. See the three-way comparison for details.

Can I run Gemini 3.5 Flash locally?

No. There is no open-weights release. For local inference, see the best local LLMs of 2026.

Does Gemini 3.5 Flash work with Cursor?

Yes, through the standard Gemini API. The setup pattern is the same as Gemini 3.0 Pro with Cursor.

What is the API model name for Flash?

Use:

gemini-3.5-flash

What this means for your stack

Use this decision path:

  • Already on Gemini 3.1 Flash? Run a side-by-side test this week. Streaming UIs may benefit immediately from faster output.
  • Already on Opus 4.7 or GPT-5.5? Run a cost-and-quality evaluation. Agent-heavy workloads may justify routing some traffic to Flash.
  • Building a new agent loop? Start with Flash if cost and speed matter.
  • Handling multimodal documents or charts? Test Flash early. CharXiv Reasoning at 84.2% is meaningful.

Treat Gemini 3.5 Flash as one component in a larger production pipeline. You still need prompt design, tool wiring, retries, eval scripts, and endpoint testing. Apidog covers the Gemini API testing side; the rest of the loop is your application logic.

Webhook Verification: How to Validate Every Incoming Request (and Why You Must)

Webhook Verification: How to Validate Every Incoming Request (and Why You Must)

Webhooks are the backbone of event-driven integrations — Stripe charges, GitHub pushes, Slack commands, Shopify orders. They’re simple on the surface: a third-party service sends an HTTP POST to your endpoint when something happens.

The problem? Anyone on the internet can POST to your webhook endpoint. Without verification, a malicious actor can send fake events and trick your system into shipping an order, granting permissions, or triggering a refund. Webhook signature verification is non-negotiable — yet it’s often the last thing developers add (if they add it at all).

Here’s how to do it right, with real examples.

How Webhook Signatures Work

Most services that send webhooks sign their payloads using HMAC-SHA256. The flow is:

  1. You register your endpoint URL with the service.
  2. The service gives you a secret key (or you generate one).
  3. When the service sends a webhook, it computes HMAC-SHA256(secret, raw_body) and includes the result in a request header (e.g. X-Hub-Signature-256, Stripe-Signature, X-Shopify-Hmac-SHA256).
  4. Your server recomputes the same HMAC and compares it to the header. If they match, the request is authentic.

The critical rule: always compute the HMAC against the raw request body — before any JSON parsing. Parsing and re-serializing can change whitespace and key order, breaking the signature check.

Verifying a Stripe Webhook (Node.js)

Stripe adds a timestamp and a signature to the Stripe-Signature header to protect against replay attacks.

import express from 'express';
import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);
const webhookSecret = process.env.STRIPE_WEBHOOK_SECRET;
const app = express();

// IMPORTANT: use raw body parser for webhook routes
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), (req, res) => {
  const sig = req.headers['stripe-signature'];

  let event;
  try {
    event = stripe.webhooks.constructEvent(req.body, sig, webhookSecret);
  } catch (err) {
    console.error(`Signature verification failed: ${err.message}`);
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  // Handle the verified event
  switch (event.type) {
    case 'payment_intent.succeeded':
      console.log('Payment succeeded:', event.data.object.id);
      break;
    default:
      console.log(`Unhandled event type: ${event.type}`);
  }

  res.json({ received: true });
});

Stripe’s SDK handles replay-attack protection by rejecting events with a timestamp older than 5 minutes by default.

Verifying a GitHub Webhook (Python)

GitHub uses the X-Hub-Signature-256 header with HMAC-SHA256.

import hashlib
import hmac
import os
from flask import Flask, request, abort

app = Flask(__name__)
GITHUB_SECRET = os.environ['GITHUB_WEBHOOK_SECRET'].encode()

def verify_github_signature(payload_body: bytes, signature_header: str) -> bool:
    if not signature_header or not signature_header.startswith('sha256='):
        return False
    expected = 'sha256=' + hmac.new(
        GITHUB_SECRET, payload_body, hashlib.sha256
    ).hexdigest()
    # Use compare_digest to prevent timing attacks
    return hmac.compare_digest(expected, signature_header)

@app.route('/webhooks/github', methods=['POST'])
def github_webhook():
    signature = request.headers.get('X-Hub-Signature-256', '')
    if not verify_github_signature(request.data, signature):
        abort(400, 'Invalid signature')

    payload = request.get_json()
    event = request.headers.get('X-GitHub-Event')
    print(f'Received {event} event for {payload.get("repository", {}).get("full_name")}')
    return '', 204

Note the use of hmac.compare_digest — this is essential to prevent timing attacks, where an attacker could infer the correct signature by measuring how long comparisons take.

A Generic HMAC Verifier

If you’re working with a service that doesn’t have an official SDK, here’s a reusable helper:

import { createHmac, timingSafeEqual } from 'crypto';

function verifyWebhookSignature(
  rawBody: Buffer,
  receivedSig: string,
  secret: string,
  algorithm: 'sha256' | 'sha1' = 'sha256'
): boolean {
  const expectedSig = createHmac(algorithm, secret)
    .update(rawBody)
    .digest('hex');

  const a = Buffer.from(receivedSig.replace(/^shad+=/, ''), 'hex');
  const b = Buffer.from(expectedSig, 'hex');

  if (a.length !== b.length) return false;
  return timingSafeEqual(a, b);
}

Common Mistakes to Avoid

1. Parsing the body before verifying. JSON parsers can normalize whitespace or reorder keys. Always verify against the raw bytes.

2. Using === for signature comparison. This leaks timing information. Always use timingSafeEqual / hmac.compare_digest or equivalent.

3. No replay protection. If your service doesn’t handle this automatically (like Stripe does), store and check event IDs or validate timestamps.

4. Hardcoding secrets. Store webhook secrets in environment variables or a secrets manager, never in source code.

Testing Webhook Verification Locally

Webhook verification is notoriously hard to test because you need a public URL. Tools like ngrok or smee.io can tunnel traffic to your local machine during development.

This is where having a proper API client environment pays off. APIKumo lets you replay saved webhook payloads against your local endpoint — including the original headers and signatures — so you can iterate on your verification logic without needing to trigger real events from the third-party service every time.

Webhook verification is one of those things that takes 20 minutes to add and can save you from a catastrophic security incident. Add it at day one, not after your first production incident.