Comment: Suppressed addresses – updated 2026-05-18

Hard Bounce vs Soft Bounce: The Critical Difference

Hard bounces are permanent delivery failures — the recipient address is invalid, the domain doesn’t exist, or the mailbox is closed. The ISP’s mail server has definitively told you: “This address will never accept mail.”

Soft bounces are temporary failures — the mailbox is full, the receiving server is temporarily overloaded, or the message was flagged as too large. The ISP is saying: “Try again later.”

The rule: Hard bounces must be suppressed immediately. Soft bounces require escalation logic (multiple retries → hard bounce → suppress).

Detailed Bounce Code Classification

RFC 3463 Enhanced Mail Status Codes (standardized across most MTAs):

Code Class Meaning Action
X.1.0 Hard Generic address error Immediate suppress
X.1.1 Hard Invalid mailbox Immediate suppress
X.1.2 Hard Invalid domain Immediate suppress
X.1.3 Hard Destination mailbox not found Immediate suppress
X.2.0 Soft Mailbox full Retry, escalate after 3
X.2.1 Soft Message too large Retry, reduce size
X.2.2 Soft Storage full (system) Retry, monitor
X.3.0 Soft System not accepting messages Retry later
X.4.0 Soft Network congestion Retry, backoff
X.5.0 Soft Routing error Retry, check DNS
X.6.0 Soft Delivery time expired Retry once

Bounce Rate Benchmarks by Industry

Industry Acceptable Bounce Target Bounce
E-commerce / Retail < 3% < 1.5%
SaaS / Software < 2% < 1%
Financial Services < 1% < 0.5%
Media / Publishing < 4% < 2%
Agency / Marketing < 5% < 2%

If your bounce rate exceeds 3%, stop sending immediately and investigate. Every bounce you send to a closed mailbox is a complaint signal to the ISP.

Email Verification API Integration

The most effective way to prevent bounces is to never add invalid addresses to your list in the first place. Real-time email verification catches typos, dead domains, and spam traps before they enter your queue.

Verification API Comparison

Provider Accuracy Speed Bulk API Pricing
ZeroBounce 99% < 1s Yes $0.003/verify
NeverBounce 98%+ < 1s Yes $0.01/verify
AbstractAPI 95% < 1s Yes $0.002/verify
Hunter 90% < 2s Yes $0.001/verify
MailboxValidator 95% < 1s Yes $0.005/verify

KumoMTA Bounce Processing Lua Integration

-- /etc/kumomta/bounce_handler.lua
-- Real-time bounce classification and suppression

local SUPPRESSION_LIST = {}
local BOUNCE_STATS = { hard = 0, soft = 0, processed = 0 }

-- Load suppression list from file/db on startup
local function load_suppression_list()
    local f = io.open("/var/lib/kumomta/suppression.txt", "r")
    if f then
        for line in f:lines() do
            SUPPRESSION_LIST[line] = true
        end
        f:close()
        log_info("Loaded " .. #SUPPRESSION_LIST .. " suppressed addresses")
    end
end

local function add_to_suppression(email)
    SUPPRESSION_LIST[email] = true
    -- Persist to disk
    local f = io.open("/var/lib/kumomta/suppression.txt", "a")
    if f then
        f:write(email .. "n")
        f:close()
    end
end

local function is_suppressed(email)
    return SUPPRESSION_LIST[email] == true
end

-- Bounce code classification
local function classify_bounce(smtp_response)
    local code = tonumber(smtp_response:match("%d+")) or 0

    -- RFC 3463 bounce classes
    if code >= 500 or (code >= 400 and code < 500 and string.find(smtp_response, "user unknown")) then
        return "hard"
    elseif code >= 400 then
        return "soft"
    else
        return "unknown"
    end
end

-- Main bounce handler
kumo.on("smtp_delivery_result", function(result, meta)
    BOUNCE_STATS.processed = BOUNCE_STATS.processed + 1
    local bounce_type = classify_bounce(result.message)

    if bounce_type == "hard" then
        BOUNCE_STATS.hard = BOUNCE_STATS.hard + 1
        add_to_suppression(meta.rcpt_to)
        log_warn("Hard bounce: " .. meta.rcpt_to .. " - " .. result.message)
    elseif bounce_type == "soft" then
        BOUNCE_STATS.soft = BOUNCE_STATS.soft + 1
        -- Log for escalation tracking
        track_soft_bounce(meta.rcpt_to, result.code)
    end

    -- Report metrics
    if BOUNCE_STATS.processed % 100 == 0 then
        local bounce_rate = (BOUNCE_STATS.hard / BOUNCE_STATS.processed) * 100
        log_info("Bounce stats: " .. BOUNCE_STATS.hard .. " hard, " ..
                 BOUNCE_STATS.soft .. " soft, rate: " ..
                 string.format("%.2f", bounce_rate) .. "%")
    end
end)

-- Pre-send check
kumo.on("smtp_message_received", function(domain, meta)
    local recipient = meta.rcpt_to

    if is_suppressed(recipient) then
        log_warn("Suppressed address rejected: " .. recipient)
        kumo.reject_recipient("550 5.1.1 Address suppressed")
    end
end)

load_suppression_list()

Pre-Send Verification with HTTP API

-- Real-time verification before accepting into queue
local function verify_email_address(email)
    local http = require("socket.http")
    local ltn12 = require("ltn12")
    local json = require("cjson")

    local response = {}
    local req = http.request{
        url = "https://api.zerobounce.net/v2/validate?email=" .. email .. "&apikey=YOUR_KEY",
        sink = ltn12.sink.table(response),
    }

    if req then
        local data = json.decode(table.concat(response))
        if data.status == "valid" or data.status == "catch-all" then
            return true
        else
            return false
        end
    end
    return true -- Fail open if API is unreachable
end

Suppression List Architecture

A suppression list is your authoritative record of addresses that should never receive mail. It differs from a blocklist — suppression is for your own protection, not for blocking external senders.

Suppression Triggers (Automated)

Trigger Threshold Action
Hard bounce (any code 5xx with user unknown) 1 occurrence Immediate suppress
Soft bounce 3+ occurrences in 30 days Suppress
Complaint feedback loop 1 occurrence Immediate suppress
Manual unsubscribe Immediate Add to unsubscribe list
ESP-related spam trap hit 1 occurrence Immediate suppress

Suppression List File Format

# Comment: Suppressed addresses - updated 2026-05-18
[email protected]
[email protected]
[email protected]
[email protected]

Syncing Suppression Across Platforms

If you use multiple MTAs (e.g., KumoMTA for outbound, SendGrid for transactional), synchronize suppression lists via webhook:

# Webhook receiver for bounce/complaint events
import json

def handle_webhook(event):
    if event['type'] == 'bounce':
        add_to_suppression(event['email'], reason='hard_bounce')
    elif event['type'] == 'complaint':
        add_to_suppression(event['email'], reason='complaint')

    # Propagate to all MTAs
    for mta in all_mtas:
        mta.sync_suppression(event['email'])

PowerMTA Bounce Configuration

# /etc/pmta/config

# Bounce classification
bounce检查-level up to 2
bounce-interval 15m

# Hard bounce rules
add-parts-filter bounce-checker
match-command RCPT TO (.*) (.*)
    body-match "User unknown" do
        bounce [code] [enhanced-code]
        log-bounce
        suppress
    /match

# Soft bounce rules
match-command RCPT TO (.*) (.*)
    body-match "Mailbox full" do
        tempfail
        retry 3 times every 15 minutes
    /match

# Feedback loop processing
feedbackloop <[email protected]>
    add-feedback-filter fbl
    class auto
    source postmaster

List Hygiene Schedule

Proactive list hygiene prevents bounces before they happen:

Schedule Action
Real-time Verify new signups via API
Daily Remove hard bounces from active lists
Weekly Re-verify addresses inactive > 30 days
Monthly Full list re-verification via bulk API
Quarterly Engagement-based segmentation

Engagement-Based Re-Engagement Campaign

Addresses that haven’t opened or clicked in 90+ days are high risk. Run a re-engagement campaign before sending your next major campaign:

  1. Send re-engagement email (clear value prop, one-click action)
  2. Non-responders after 2 emails → move to “at-risk” segment
  3. At-risk segment → 30-day cooldown
  4. Still no engagement → suppress permanently

Monitoring Bounce Rate in Real Time

Key Dashboards to Build

  1. Overall bounce rate (hard + soft, rolling 24h)
  2. Hard bounce rate by ISP (spot problematic ESPs fast)
  3. Soft bounce retry success rate (are retries working?)
  4. Bounce rate by campaign (which campaigns have worst lists?)
  5. Suppression list growth rate (is list quality improving?)

Alerting Thresholds

Metric Warning Critical Action
Overall bounce rate > 2% > 3% Pause sending
Hard bounce rate > 1% > 2% Immediate pause
Soft bounce rate > 5% > 10% Investigate ISP
Bounce rate by ISP > 5% > 10% Check authentication
Suppression growth > 10%/day > 20%/day Audit acquisition

FAQ

Q: Is a 2% bounce rate acceptable?
A: For most industries, yes. For financial services or healthcare, target < 1%. Anything above 3% risks ISP penalties.

Q: Should I delete hard bounced addresses from my database or just suppress?
A: Suppress, never delete. Suppressed addresses are flagged but retained for audit purposes. Deleted addresses might be re-registered by a new user and cause problems if accidentally re-added.

Q: How long should I retry soft bounces before suppressing?
A: 3-5 retries over 24-72 hours is standard. If the address is still failing after that, suppress as a “soft bounce exhaustion” address.

Q: Does KumoMTA handle bounces differently than PowerMTA?
A: Both handle bounce codes correctly. KumoMTA’s Lua policy gives you more granular control over bounce classification and suppression logic. PowerMTA uses XML configuration for bounce rules.

Q: Can I re-send to an address I previously suppressed?
A: Only if the user re-subscribes or explicitly confirms the address is valid. Suppression exists to protect your reputation — bypassing it manually is risky.

Get Help With Bounce Rate Reduction

PostMTA provides:

  • Full bounce rate audit and root cause analysis
  • KumoMTA bounce processing configuration
  • Suppression list architecture and integration
  • List hygiene automation and verification API setup
  • Real-time alerting and monitoring dashboards

👉 Talk to a deliverability expert →

For related guides, see IP Warmup Strategies, Email Authentication Guide, and SMTP Relay Setup Guide.

References: RFC 3463 (Enhanced Mail Status Codes) | Google Postmaster Tools

HIPAA CI/CD vs SOC 2 CI/CD: where the controls differ

If you have SOC 2 and assume HIPAA is incremental, your pipeline disagrees.

SOC 2 audits the policies you chose. HIPAA audits the system you built. At the CI/CD layer, that distinction stops being abstract and starts producing engineering work.

A healthcare engineering team I worked with had Type II SOC 2 in hand. They had passed it the year before on the second attempt. Their pipeline was clean, their evidence binder tidy, their GRC lead had stopped flinching at the word “scope.”

Then they closed their first hospital system. The contract came with a Business Associate Agreement and a HIPAA addendum. Legal forwarded it to engineering with a one-line note: “Should be fine, we have SOC 2.”

It was not fine. Inside two weeks they had fourteen control gaps. Their pipeline runners sat on a shared pool that handled both staging and a non-PHI demo environment. Their audit logs rolled off at 90 days. Their cosign signature verification was a soft warning in their Helm chart, not a deploy-blocker. Their incident response runbook said the right things about breach notification, but their pipeline emitted nothing the legal team could query against the 60-day clock in 45 CFR § 164.404. They had SOC 2. They did not have the system HIPAA assumes.

Six weeks later, after three pipeline rewrites and a round of running the BAA through outside counsel, they shipped. The 20% gap between SOC 2 and HIPAA is not 20% of the cost. It is the part of the pipeline that has to change to emit evidence on its own.

SOC 2 lets you scope your controls and have your auditor read a narrative. HIPAA hands you the Security Rule, removes most of your scoping latitude, and asks the system itself to produce the evidence. At the pipeline layer, that distinction produces changes you cannot policy your way out of. The rest of this post is a control-by-control map of where those changes live. If you are coming from the earlier HIPAA CI/CD implementation guide, this is the comparison piece beside it. The pillar page covers positioning and control mapping at the architecture level.

Section 01 — The framework difference in one sentence

SOC 2 is a trust services framework. You choose which of the five criteria (security, availability, processing integrity, confidentiality, privacy) you attest against, and which systems are in scope. Your auditor reads your description, looks at evidence for the controls you committed to, and writes a narrative report. If your framework is reasonable and your evidence consistent with what you claimed, you pass.

HIPAA’s Security Rule is not a framework you scope. It is a federal regulation. 45 CFR § 164 sets administrative, physical, and technical safeguards. Some are “required”; some are “addressable” (implement or document a defensible alternative). You do not scope out of the Security Rule for an environment that handles PHI. You choose how to implement, not whether.

At the pipeline level, this is the difference that drives everything else. SOC 2 lets you write “production deploys are reviewed by an authorized engineer per documented procedure” and have your auditor confirm the procedure is followed. HIPAA expects the same control implemented in a way that is observable and queryable, with the system itself producing the evidence at the time the control runs. SOC 2 audits the policy. HIPAA audits the artifact.

Once you accept that framing, the rest of the comparison is a series of consequences.

Section 02 — Where HIPAA and SOC 2 controls overlap

Three control areas are roughly equivalent between SOC 2 and HIPAA at the pipeline layer. If you implemented them honestly for SOC 2 Type II, you are most of the way to the HIPAA equivalent. This is the overlap the comparison vendors latch onto.

Pipeline access control. SOC 2’s CC6.1 requires logical access controls over systems in scope. HIPAA’s § 164.312(a)(1) requires unique user identification, emergency access procedure, automatic logoff, and encryption controls. Both want the same pipeline-layer thing: MFA-protected CI/CD access, no shared service accounts triggering deploys, identity attribution on every job. A SOC 2 implementation with SSO + MFA and no long-lived bearer tokens covers HIPAA’s unique user identification cleanly.

Encryption in transit. SOC 2’s CC6.7 requires data in transit between components is protected. HIPAA’s § 164.312(e)(1) requires transmission security for PHI. Both expect TLS 1.2+ on every channel touching sensitive data: pipeline to registry, pipeline to cloud API, pipeline to deploy target. Same configuration, same evidence shape.

Audit logging baseline. SOC 2’s CC7.2 requires monitoring system components for anomalies and security events. HIPAA’s § 164.312(b) requires audit controls that record activity on systems containing PHI. Both expect pipeline-level logs capturing who did what, when, on which artifact. If your SOC 2 evidence already streams pipeline logs to a SIEM, the HIPAA baseline is covered.

These three overlap honestly. They are why “we have SOC 2 so we’re 80% there” sounds plausible. The problem is what’s left after the easy 60%. The pillar page walks the architecture control by control.

Section 03 — Where HIPAA and SOC 2 controls diverge

Five control areas break the SOC 2-to-HIPAA equivalence story. Each is a place where the SOC 2 implementation is acceptable as policy and inadequate as system.

Business Associate Agreement scope. HIPAA requires that every entity touching PHI on your behalf, including downstream services in the pipeline, is covered by a BAA. SOC 2 has no equivalent concept. A pipeline that uses a managed KMS service for cosign signatures, a container registry for PHI-bound images, and a secrets manager for deploy credentials is fine for SOC 2 if your vendor list is current. For HIPAA, every one of those services has to be BAA-covered and the runner orchestrating them has to be covered too. § 164.308(a)(4) sets the requirement.

Six-year retention with immutability. SOC 2 does not specify a minimum audit log retention. Your auditor will accept whatever you wrote. HIPAA § 164.316(b)(2)(i) is specific: documentation must be retained for six years from creation or the date it was last in effect, whichever is later. And it must be defensible, which in practice means immutable. A pipeline that writes audit logs to a default CloudWatch group with 90-day retention is a SOC 2 yes and a HIPAA no.

Encryption verification, not attestation. SOC 2 accepts “TLS 1.2 enforced via load balancer policy, verified during the annual review.” HIPAA’s § 164.312(a)(2)(iv) and § 164.312(e)(1) expect ongoing assurance that encryption is happening, not that it was once configured. At the pipeline layer, that means deploy-time tests that fail the build when TLS verification breaks, not a once-a-year posture review.

Workforce sanction enforcement. HIPAA § 164.308(a)(1)(ii)(C) requires sanctions applied against workforce members who fail to comply with security policies. SOC 2 expects you to have a sanction policy. HIPAA expects it enforced, which in pipeline terms means a deploy by a workforce member whose access has been revoked fails at the runner, not at the code review.

Breach notification telemetry. HIPAA § 164.404 gives covered entities 60 days from breach discovery to notify affected individuals. SOC 2 has no equivalent regulatory clock. HIPAA expects the system to surface enough provenance that a security event can be scoped against the 60-day window; SOC 2 lets you reconstruct the same evidence over weeks.

These five gaps are where the engineering work lives. They are also where the comparison vendors stop writing. The next four sections walk what each requires from the pipeline, with code.

Decision 01 — Which framework owns your evidence model

Every architectural decision that follows comes down to one question: which framework owns your evidence model? Build the pipeline for SOC 2 evidence and you build something an auditor can read. Build it for HIPAA evidence and you build something an auditor can query.

SOC 2 evidence is built for narrative: screenshots, control descriptions, sample-based attestations, policy documents reviewed annually. The model assumes a human assembles evidence on a schedule, an auditor reads it, and the narrative carries the weight. HIPAA evidence is built for query: signed manifests, append-only audit logs, immutable artifact registries, deploy events with attribution timestamps. The system itself is the source of truth, producing the workflow and the evidence at the same time.

My opinionated stance: if you are building a pipeline that will support both frameworks, architect it for HIPAA evidence. The reverse path does not work. A SOC 2-shaped evidence pipeline will not pass HIPAA’s evidence expectations without a rebuild. A HIPAA-shaped evidence pipeline produces SOC 2 evidence as a side effect of operating correctly. Build for the stricter framework; the looser one comes free.

Decision 02 — Pipeline access and BAA scope

SOC 2 lets you document which engineers can deploy. HIPAA expects the runner to enforce it, and extends the requirement to every downstream service the runner touches.

§ 164.308(a)(4) requires that workforce access to PHI is granted only to those whose role justifies it. The Security Rule extends that obligation to systems accessing PHI, including CI/CD systems and the downstream services they call. A pipeline runner with IAM credentials reaching PHI-bearing workloads is itself in BAA scope. So is every service it touches. That produces three pipeline-level requirements SOC 2 does not check:

  • The runner is on infrastructure covered by a BAA, your own or a cloud provider’s BAA-covered service.
  • Every downstream service the runner calls (KMS, registry, secrets manager, deploy target) is BAA-covered.
  • The deploy attempt fails if any service in the chain falls outside that scope.

The third requirement is where policy-as-code earns its place. An OPA policy at the deploy gate inspects the deploy context, verifies every service touched is in the BAA-covered set, and blocks the deploy if any is not. The pipeline does not depend on a human knowing which services are covered. The policy does.

# policies/baa_scope.rego
# Block HIPAA pipeline deploys that touch a service outside BAA scope.
# Evaluated by the parent pipeline before any deploy stage runs.

package deploy.hipaa

import rego.v1

# Services covered under an active BAA, with effective date.
# Sourced from a maintained registry, not hand-edited per build.
baa_covered := {
    "kms.googleapis.com":              "2025-01-15",
    "artifactregistry.googleapis.com": "2025-01-15",
    "secretmanager.googleapis.com":    "2025-01-15",
    "container.googleapis.com":        "2025-01-15",
    "logging.googleapis.com":          "2025-01-15",
}

# Workforce members with current PHI access entitlement.
# Sourced from IdP group membership at evaluation time.
authorized_actors := {"[email protected]", "[email protected]"}

default allow := false

allow if {
    actor_authorized
    every_service_covered
    runner_in_scope
}

actor_authorized if {
    input.deploy.actor in authorized_actors
}

every_service_covered if {
    not unscoped_service_exists
}

unscoped_service_exists if {
    some s in input.deploy.services_touched
    not baa_covered[s]
}

runner_in_scope if {
    input.runner.tag == "hipaa-prod-runner"
    input.runner.signed == true
}

# Deny messages surface in the pipeline log for the auditor.
deny contains msg if {
    some s in input.deploy.services_touched
    not baa_covered[s]
    msg := sprintf("service %s is outside BAA scope; deploy blocked", [s])
}

deny contains msg if {
    not input.deploy.actor in authorized_actors
    msg := sprintf("actor %s lacks current PHI entitlement", [input.deploy.actor])
}

The policy is small on purpose. It runs against the same evidence bundle the parent pipeline aggregates, the same one referenced in the parent/child architecture in the implementation guide. The BAA registry is its own resource, kept in code with PRs gated on legal review. When a new service is introduced, the deploy fails until legal confirms coverage. Engineering does not have to keep the BAA list in their head.

For SOC 2, the policy is overkill. For HIPAA, this is the minimum that survives an evidence pull.

Decision 03 — Audit log retention and immutability

SOC 2 retention is whatever you wrote in your policy. HIPAA retention is six years, immutable, recoverable. § 164.316(b)(2)(i) is direct about this, and it is one of the harder controls to retrofit later because retroactively making logs immutable is not a thing you can do.

Three pipeline implications:

  • Audit events go to a write-once destination at the time the event happens, not at end of pipeline.
  • The destination supports object retention locks for the full six-year window, with no path to early deletion.
  • The pipeline emits a signed manifest of every audit event, so tampering is detectable at audit time.

On AWS, the cleanest implementation is S3 Object Lock in compliance mode with a six-year retention. On GCP, Bucket Lock with equivalent duration. Both work; neither is set up by default.

# .github/workflows/audit-emit.yml
# Reusable workflow: emit a HIPAA-aligned audit event to an Object Lock bucket.
# Caller workflows invoke this on every pipeline state change worth recording.

name: emit-audit-event
on:
  workflow_call:
    inputs:
      event_type:
        required: true
        type: string
      artifact_digest:
        required: true
        type: string

jobs:
  emit:
    runs-on: hipaa-prod-runner
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Assume audit writer role
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/hipaa-audit-writer
          aws-region: us-west-2

      - name: Compose audit event
        run: |
          # Event schema mirrors the SIEM ingestion contract.
          # Hash the inputs so any field tampering invalidates the manifest.
          cat > event.json <<EOF
          {
            "event_type": "${{ inputs.event_type }}",
            "artifact_digest": "${{ inputs.artifact_digest }}",
            "actor": "${{ github.actor }}",
            "pipeline_id": "${{ github.run_id }}",
            "occurred_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
          }
          EOF
          sha256sum event.json | awk '{print $1}' > event.sha256

      - name: Write to Object Lock bucket
        run: |
          # Compliance mode: no override path, even for the root account.
          # Retention set on the object itself, not a bucket policy that can be relaxed.
          aws s3api put-object 
            --bucket hipaa-audit-logs 
            --key "events/${{ github.run_id }}/$(date -u +%s).json" 
            --body event.json 
            --object-lock-mode COMPLIANCE 
            --object-lock-retain-until-date 
              "$(date -u -d '+6 years' +%Y-%m-%dT%H:%M:%SZ)"

Retention is set on the object itself, not on a bucket policy that can be relaxed later. Compliance mode means the retention cannot be shortened even by the AWS root account. The manifest hash is computed before write, so tampering is detectable. A query “who deployed artifact X” returns the workforce member, the runner, the artifact digest, and the timestamp in one row.

For SOC 2, this is more than your auditor will ask for. For HIPAA, this is the floor.

Decision 04 — Encryption verification, not attestation

SOC 2 lets you say “TLS 1.2 enforced via load balancer policy.” HIPAA expects the pipeline to verify it on every deploy.

§ 164.312(a)(2)(iv) on encryption and § 164.312(e)(1) on transmission security both map to SOC 2’s CC6.7, which accepts annual attestation. HIPAA does not. The expectation is that encryption posture is verified at deploy time, with failure modes that block the deploy. Two verifications matter at the pipeline layer:

  • Artifact signature verification. Every container deployed to a PHI-bearing environment carries a cosign signature; the deploy fails if signature verification fails.
  • Transit verification. Every PHI-touching endpoint the pipeline reaches negotiates TLS 1.2+ with a current cipher; the deploy fails if any endpoint downgrades or refuses.
#!/usr/bin/env bash
# scripts/verify-encryption.sh
# Run at the end of the deploy stage. Blocks the deploy on any verification miss.
# HIPAA-aligned: § 164.312(a)(2)(iv) and § 164.312(e)(1).

set -euo pipefail

IMAGE_DIGEST="${1:?image digest required}"
PHI_ENDPOINTS=(
    "https://api.hipaa-app.internal/v1/healthz"
    "https://audit.hipaa-app.internal/ingest"
)

verify_signature() {
    local digest="$1"
    # Verify against the production-only Fulcio identity.
    # Any other signer fails the build.
    cosign verify "${digest}" 
        --certificate-identity "[email protected]" 
        --certificate-oidc-issuer "https://accounts.google.com" 
        > /dev/null
}

verify_transit() {
    local url="$1"
    # Require TLS 1.2+, refuse on downgrade.
    # Surface the negotiated protocol for the audit log.
    local proto
    proto=$(curl --silent --tls-max 1.3 --tlsv1.2 
        --write-out "%{ssl_version}" --output /dev/null 
        "${url}")
    case "${proto}" in
        TLSv1.2|TLSv1.3) echo "ok ${url} ${proto}" ;;
        *) echo "fail ${url} ${proto}" >&2; return 1 ;;
    esac
}

echo "== verifying artifact signature"
verify_signature "${IMAGE_DIGEST}"

echo "== verifying transit posture"
for endpoint in "${PHI_ENDPOINTS[@]}"; do
    verify_transit "${endpoint}"
done

echo "== encryption verification ok"

Two things matter. First, it is a deploy gate, not a post-deploy check. A failed signature or a TLS downgrade blocks the deploy before any PHI-touching code reaches a runtime handling a request. Second, the verification output is itself an audit event, written to the Object Lock bucket from Decision 03. The auditor’s question, “show me that encryption was verified on the deploy that touched record X,” has a one-query answer.

SOC 2 tells you the load balancer config is correct. HIPAA tells you the deploy on May 13 at 09:14 UTC verified TLS 1.3 to the audit endpoint and TLS 1.2 to the API endpoint with this cipher.

Section 04 — HIPAA SOC 2 control mapping, opinionated

The table below is the version of “HIPAA vs SOC 2” I would hand a platform team on day one of an engagement. It maps the control areas where the frameworks diverge against the pipeline pattern we would build, the pattern we would accept under time pressure, and the pattern we would refuse to ship.

Control area HIPAA SOC 2 Recommended Acceptable Avoid
BAA scope on deploy path § 164.308(a)(4) (none) OPA gate against a versioned BAA registry Quarterly BAA review with deploy-path inventory Trust the cloud provider default without verifying covered services
Audit log retention § 164.316(b)(2)(i) CC7.2 S3/GCS Object Lock at 6-year retention in compliance mode Append-only SIEM with documented 6-year archive Default CloudWatch retention; “we never delete logs” as the policy
Encryption verification § 164.312(a)(2)(iv), § 164.312(e)(1) CC6.7 Deploy-time signature and TLS verification; fail the build on miss Daily synthetic transaction with alerting Annual TLS posture review; soft warning in Helm charts
Workforce sanction enforcement § 164.308(a)(1)(ii)(C) (none) IdP-backed deploy entitlement evaluated at runner invocation Pre-deploy human check against current entitlement list Documented sanction policy with no system-level enforcement
Breach notification telemetry § 164.404 (none) Signed deploy manifest queryable by digest, actor, and time Centralized log search with deploy-event tagging Reconstruct deploy provenance from Slack and CI run history

“Recommended” passes both audits without ambiguity. “Acceptable” is what you sit in under time pressure, with a written remediation date. “Avoid” produces audit findings.

Section 05 — What SOC 2 to HIPAA migration looked like in production

Six weeks. Eight engineers across two squads. Roughly 5,200 lines of new pipeline code, 1,800 lines retired. That was the shape of the engagement that triggered this post.

The team came in with passing Type II SOC 2, a closed contract with a regional hospital system, and a BAA addendum that assumed evidence the pipeline could not yet produce. We mapped their existing SOC 2 evidence against the Security Rule. The overlap was real: pipeline access was already SSO with MFA, transit was already TLS 1.2+, audit logs already streamed to a SIEM. The gaps were the ones in Section 03. None were unfamiliar; all required engineering work.

The work split four weeks pipeline and four weeks the runtime and IAM context the pipeline depended on. Pipeline rewrites cost less time than expected; parent/child separation was already partially in place, and the OPA gate for BAA scope fit cleanly into the structure they had. The expensive work was outside the pipeline: an IAM redesign to scope the audit writer role away from any operator who could shorten retention, legal review of every cloud service in the deploy path (which surfaced three uncovered services no one had noticed), and the cosign deploy gate catching a staging cluster pulling unsigned dev images through a cached registry mirror. None of those was a pipeline bug. All were pipeline-adjacent.

What shipped at week six was a pipeline that emitted evidence on its own, against the Security Rule, with the same control set the SOC 2 auditor had read about in narrative. The first internal HIPAA audit took the platform lead 90 minutes instead of the four days the prior SOC 2 evidence pull had taken. That is the difference between a system that produces evidence and a human who produces evidence about a system.

Section 06 — Common mistakes

Three calls show up in nearly every “we have SOC 2, adding HIPAA” conversation. Each one is wrong in a specific way.

  • “We’re 80% there because we have SOC 2.” The overlap is 60% on control inventory and much smaller on implementation work. The 40% gap is where the architecture has to change. SOC 2 was built for narrative; HIPAA was built for query. Most of the engineering cost lives in the conversion.
  • “We can use the same auditor.” Sometimes true, often beside the point. Your SOC 2 auditor reads your control framework against criteria you chose. Your HIPAA controls have to satisfy a regulation you did not choose. The evidence shape has to change regardless of who reads it. The earlier post on five things healthcare engineering teams get wrong about HIPAA CI/CD walks five places where SOC 2-style evidence passes and HIPAA-style evidence does not.
  • “Our pipeline doesn’t really touch PHI.” It almost always does, once you trace artifact provenance. The artifact you build runs against PHI in production. The cosign key signs it. The deploy credential places it. The audit log records the placement. Every one of those is a HIPAA touchpoint. The pipeline does not have to read PHI to be in scope.

Each call is reasonable on its face. Each underbudgets the engineering work by a factor of three. The vendors making the call get paid for the conclusion; the platform team pays for the rewrite.

Section 07 — Frequently asked

Do I need HIPAA compliance if I already have SOC 2?

Yes, if your environment touches PHI. SOC 2 is a voluntary trust services framework where you scope your own controls. HIPAA is a federal regulation that does not let you scope out of the Security Rule for any environment handling PHI. SOC 2 compliance covers about 60% of HIPAA’s control inventory at the pipeline layer, but does not satisfy HIPAA’s evidence requirements. The remaining 40% is engineering work.

How long does it take to add HIPAA to a SOC 2 pipeline?

Three months is a defensible estimate if the engineering team is honest about the work involved. The expensive parts are BAA scope mapping across every service the pipeline touches, audit log retention architecture with six-year immutability, and encryption verification gates at deploy time. Thirty days is the number comparison vendors quote and the platform team regrets accepting.

Is SOC 2 enough for selling into healthcare?

No. SOC 2 Type II is table stakes for B2B SaaS but does not satisfy HIPAA requirements for handling Protected Health Information. Hospital systems and other covered entities require a Business Associate Agreement and HIPAA-compliant controls, which SOC 2 alone does not provide. SOC 2 plus HIPAA is the combination most healthcare engineering teams need.

What’s the difference between HIPAA and SOC 2 audit log requirements?

SOC 2 does not specify a minimum audit log retention; your auditor accepts whatever you wrote in your policy. HIPAA § 164.316(b)(2)(i) requires six years of retention from creation or last effective date, whichever is later, with practical immutability requirements. A pipeline writing audit logs to default CloudWatch retention with 90-day rollover passes SOC 2 but fails HIPAA.

Does SOC 2 compliance satisfy a Business Associate Agreement?

No. BAA is a HIPAA construct with no SOC 2 equivalent. Every entity that touches PHI on your behalf, including downstream services in your CI/CD pipeline, must be covered by a BAA. SOC 2 has no mechanism for this requirement. The runner orchestrating deploys, the KMS service signing artifacts, the registry holding PHI-bound images, and the secrets manager providing deploy credentials all need BAA coverage under HIPAA.

Can I use the same auditor for HIPAA and SOC 2?

Sometimes, but it’s often beside the point. Your SOC 2 auditor reads your control framework against criteria you chose. Your HIPAA controls have to satisfy a federal regulation you did not choose. The evidence shape has to change regardless of who reads it. A SOC 2-shaped evidence pipeline will not pass HIPAA expectations without a rebuild; the opposite is not true.

How much does it cost to upgrade from SOC 2 to HIPAA-compliant CI/CD?

Pricing depends on the size of your engineering team, the number of pipelines in scope, and the maturity of your existing SOC 2 controls. A two-week fixed-fee HIPAA CI/CD audit produces a written remediation roadmap with effort estimates sized to your specific environment. Stonebridge runs audit and build engagements as fixed-fee, so you know the scope, the deliverables, and the price before any work starts.

Section 08 — Closing

SOC 2 audits the policies you chose. HIPAA audits the system you built. The overlap is real and worth using. The divergence is real and worth budgeting for. Three months is a defensible estimate if the team is honest about the work; thirty days is the number the comparison vendors quote and the platform team regrets accepting.

Compliance is a property of the system, not a process humans run. Build the pipeline to emit HIPAA evidence on its own, and SOC 2 evidence falls out of normal operation. Build it the other way and the gap closes by rewrite, not retrofit.

If you are in the SOC 2-to-HIPAA conversation right now: Stonebridge runs two-week HIPAA CI/CD audits that map your existing SOC 2 controls against the Security Rule and produce a written remediation roadmap. Fixed fee, founder-led, the report holds up under first-party review.

Keep reading: the practitioner’s walkthrough of every Security Rule control before an audit lives in the HIPAA CI/CD audit checklist for engineering teams.

About the author

Lucas Jones is the Founder and Principal Platform Engineer at Stonebridge Tech Solutions. Six years building cloud infrastructure and CI/CD pipelines in regulated environments, including HIPAA, FedRAMP, and SOC 2 work for healthcare and defense engineering teams across AWS, GCP, Azure, and OCI. Based in Sacramento, California.

This post originally appeared on Stonebridge Tech Solutions.

AI Harnesses: Why DevOps Principles Are the Missing Piece in Agentic Development

The 5 core harness primitives — Tool Registry, Context Management, Guardrails, Agent Loop, and Compaction & Memory — arranged around a central hub

The Breakthrough That Changed How I Think About Agents

Last night I watched a talk from AI.engineer that crystallized something I’ve been building toward for months. The speaker demonstrated fixing a series of agent failures — not by changing the prompt, not by upgrading the model, not by adding more context. They fixed it by improving the harness.

The agent was the same. The model was the same. The instructions were identical. But the harness — the infrastructure that controls how the agent operates — made the difference between a broken tool and a production-ready system.

That’s when the parallel hit me like a freight train:

DevOps was the tool we gave humans to control their workflows. AI Harnesses are the tool we give agents to control their workflows.

The AI.engineer talk that crystallized the harness thesis — fixing agent failures through infrastructure, not prompts.

This isn’t a loose analogy. It’s a direct architectural parallel — and understanding it is the key to building agent systems that actually work in production.

What Is an AI Agent Harness?

An agent harness is a set of computer science primitives that govern agent behavior regardless of model. It’s the runtime infrastructure that sits between “I have an LLM” and “I have a production agent system.”

Think of it this way: a model is a brain. A harness is the nervous system, skeleton, and reflexes that turn that brain into a functioning organism.

The 5 core harness primitives — Tool Registry, Context Management, Guardrails, Agent Loop, and Compaction & Memory — arranged around a central hub
The five primitives that make up a complete AI agent harness. Each solves a different dimension of agent governance.

A complete harness includes five core primitives:

1. Tool Registry

The tool registry defines what an agent can do — the complete set of capabilities available at runtime. But it’s more than a list. A well-designed registry includes:

  • Discovery — agents can find tools by capability, not just name
  • Schema validation — every tool call is validated against its input schema before execution
  • Access control — not every agent gets every tool. The registry enforces boundaries.
  • Versioning — tools evolve. The registry manages backward compatibility.

In DevOps terms, this is your artifact registry meets your IAM policy. You wouldn’t give every developer root access to production. You shouldn’t give every agent access to every tool.

2. Context Management

Context management is the assembly pipeline that determines what information an agent receives before making a decision. This includes:

  • Memory tiers — structured context (core identity → working state → long-term patterns → event streams) loaded at the right time
  • Skill injection — reusable capability definitions loaded on-demand when relevant
  • Compaction — intelligent summarization when context windows fill up
  • Priority — not all context is equal. Critical instructions survive compaction; nice-to-haves don’t.

The DevOps parallel: this is your configuration management. Ansible, Terraform, Puppet — all of them solve the same problem for infrastructure that context management solves for agents. The right configuration, at the right time, to the right target.

3. Guardrails

Guardrails are the moat. They’re pre-execution interceptors that prevent agents from doing the wrong thing — not by instruction, but by architecture.

  • Pre-tool hooks — intercept dangerous operations before they execute (like hookflows that redirect git commit to dev_commit)
  • Post-tool validation — verify outputs match expected patterns
  • Dynamic enforcement — rules that can change per-prompt, per-context, at runtime
  • Sandboxing — file system restrictions, network controls, tool allowlists

Here’s the key insight: a guardrail doesn’t add instructions about what not to do. It removes the wrong option entirely. The agent can’t make the mistake because the mistake path doesn’t exist. That’s fundamentally different from telling an agent “please don’t do X” and hoping it listens.

Comparison of instruction-based governance vs architecture-based governance — showing how guardrails remove wrong paths entirely rather than relying on instructions
Guardrails don’t instruct against bad behavior — they architecturally remove it. The wrong path simply doesn’t exist.

4. Agent Loop

The agent loop is the core execution cycle — observe, think, act, repeat. But a harness-engineered loop includes:

  • Termination conditions — when to stop (max iterations, goal reached, error threshold)
  • Sub-agent orchestration — spawning specialized agents for sub-tasks
  • Error recovery — retry strategies, fallback paths, graceful degradation
  • Progress tracking — observability into where the agent is in its workflow

This maps directly to your CI/CD pipeline runner. Jenkins, GitHub Actions, Azure DevOps — they all implement a task loop with retries, parallelism, error handling, and observability. An agent loop is the same pattern applied to cognitive work.

5. Compaction and Memory

Long-running agents hit context limits. Compaction is the strategy for managing this:

  • Checkpoint summaries — periodic snapshots of progress
  • Selective retention — keep critical decisions, discard routine operations
  • Persistence hierarchy — what goes to short-term memory vs. long-term storage
  • Cross-session recall — enabling agents to build on prior work

The DevOps equivalent: log rotation and data lifecycle management. You don’t keep every debug log forever. You tier your data — hot storage for recent, warm for searchable history, cold for compliance archives.

Why Harnesses Matter More Than Model Choice

Here’s the uncomfortable truth the AI industry doesn’t want you to hear: the model is increasingly commodity. GPT-5, Claude Opus, Gemini Ultra — they’re all converging on similar capability levels. The performance gap between top models shrinks every quarter.

But the performance gap between a well-harnessed agent and a raw model with a prompt? That gap is enormous and growing.

The demo I watched proved it empirically. Same model. Same prompt. Different harness quality. The results went from “broken and unreliable” to “production-ready” — purely through harness improvements.

This maps perfectly to DevOps history. In 2010, the “developer” was the bottleneck everyone focused on. Hire better developers. Give them better tools. Train them more. But the organizations that won weren’t the ones with the best individual developers — they were the ones with the best delivery systems. CI/CD, IaC, observability, feature flags — the infrastructure that made even average developers reliably productive.

Same pattern. Same lesson. Invest in the harness, not just the model.

Static vs. Dynamic: The Frontier

Most harnesses today are static. You define your tools, write your system prompt, set up guardrails, and deploy. The configuration is the same for every execution.

Visual showing the evolution from rigid static harnesses to adaptive dynamic harnesses that adjust governance per-prompt based on risk level
Static harnesses apply the same rules to every execution. Dynamic harnesses adapt governance per-prompt — loose for low risk, tight for high risk.

This is like having a single CI/CD pipeline for every project. It works — until it doesn’t. Until you need different governance for different contexts, different risk levels, different domains.

The future is dynamic harnesses — where guardrails, tool access, context assembly, and execution policy are defined per-prompt at runtime.

Imagine an agent that, for low-risk tasks, operates with minimal guardrails and maximum autonomy. But for the same agent handling a financial transaction or a production deployment, the harness tightens: human-in-the-loop gates activate, additional validation layers engage, the tool registry narrows to only approved operations.

This isn’t science fiction. This is what GitHub Copilot Extensions enable right now — the ability to dynamically inject tools, context, and governance into an agent’s runtime based on the specific task. It’s also what I’ve been building toward with my own agent-harness project.

The DevOps Parallel (It’s Not a Metaphor — It’s a Pattern)

Let me make this explicit. Every major DevOps innovation has a direct parallel in harness engineering:

Side-by-side mapping showing DevOps concepts on the left connected by arrows to their AI harness equivalents on the right
The DevOps → Harness parallel isn’t a loose analogy — it’s the same architectural pattern applied to cognitive work instead of infrastructure.

DevOps Gave Humans Harnesses Give Agents
CI/CD pipelines Agent loops with termination and retry
Infrastructure as Code Context-as-code (skills, constitutions, memory tiers)
Deployment gates Autonomy levels and approval gates
RBAC and least privilege Tool registry access control
Observability (logs, metrics, traces) Agent event streams, checkpoints, memory
Git hooks Pre-tool hookflows
Sandboxed environments Agent sandboxing (file, network, tool boundaries)
Feature flags Dynamic guardrails per-prompt

The lesson DevOps taught us was: don’t rely on discipline alone — build systems where the right thing is the easy thing. CI/CD didn’t succeed because developers became more careful. It succeeded because the pipeline made quality automatic.

The same principle applies to agents. You don’t make agents trustworthy by writing better prompts. You make them trustworthy by engineering harnesses where the right behavior is the default path and the wrong behavior is architecturally impossible.

“Make the right thing to do the easy thing to do” — this principle built the DevOps movement. Now it’s building the agentic one.

My Journey: Building Before the Industry Named It

I started building what I now call a harness before the industry had a name for it. My agent-harness repo (TypeScript) emerged from running a production multi-agent system — 30+ agents, 60+ skills, 4-tier memory, hookflows, constitutions, and sandboxing.

Every pattern I described above? I learned it the hard way. Agents going rogue because there were no guardrails. Context windows overflowing because there was no compaction strategy. Agents fighting each other because there was no orchestration layer.

The harness primitives crystallized through production pain, not academic theory.

Now I’m building a Go implementation focused on maximum determinism and testability. Why Go?

  • Compiled binary — no runtime dependencies, no “works on my machine”
  • Strong typing — harness configuration errors caught at compile time
  • Concurrency primitives — goroutines map naturally to parallel agent execution
  • Testability — interfaces and table-driven tests for every harness primitive
  • Performance — sub-millisecond hook execution for real-time guardrails

The goal: a harness so deterministic that you can write unit tests for agent governance. If you can test your CI/CD pipeline, you should be able to test your agent harness.

The 2027 Vision: Every Prompt Ships With Its Governance

Here’s where this is heading. By 2027, I believe we’ll see:

Timeline roadmap showing the 2027 vision: Dynamic Harnesses as Infrastructure, Harness-as-Code, Harness Marketplaces, and the Harness Engineer role
The roadmap to 2027: harness engineering becomes a first-class discipline with its own tooling, marketplaces, and career paths.

Dynamic harnesses as first-class infrastructure. Every prompt that enters an agent system will carry its own governance metadata — tool permissions, guardrail configuration, context assembly rules, termination conditions. Not as an afterthought. Not as a bolted-on safety layer. As first-class harness engineering.

Harness-as-Code. Just as Infrastructure-as-Code made deployments reproducible, reviewable, and testable — Harness-as-Code will do the same for agent governance. Your agent’s behavior policy will live in version control, go through code review, and have automated tests.

Harness marketplaces. The same way Terraform modules and GitHub Actions created ecosystems of reusable infrastructure — harness components (guardrail packs, context strategies, tool registries) will become composable building blocks.

The “Harness Engineer” role. Just as DevOps created the DevOps Engineer, Platform Engineer, and SRE roles — harness engineering will create a new discipline. People who specialize in making agents trustworthy, deterministic, and governed.

What You Should Do Today

If you’re building with agents — whether it’s a single Copilot workspace or a fleet of autonomous systems — start thinking about your harness:

  1. Audit your tool access. What can your agents do? Should they be able to do all of it? Implement a registry with boundaries.
  2. Add pre-execution hooks. Before an agent calls a tool, validate the call. Block dangerous patterns. Redirect to governed alternatives.
  3. Structure your context. Don’t dump everything into the system prompt. Tier it. Load what’s needed when it’s needed.
  4. Define termination. How does your agent know when to stop? What’s the max iteration count? What’s the error budget?
  5. Make it testable. If you can’t test your agent’s governance, you can’t trust it.

The Bottom Line

DevOps taught us that you don’t build reliable software by hiring more careful developers. You build it by engineering systems where reliability is the default. The same principle — exactly the same principle — applies to agentic development.

The model is the developer. The harness is the DevOps infrastructure. And just like the 2010s proved that investing in delivery infrastructure beats investing in individual developer heroics, the 2020s will prove that investing in harness engineering beats investing in prompt engineering alone.

The right thing to do should be the easy thing to do — for humans AND agents. That’s Agentic DevOps. That’s what I’m building.

Want to go deeper on harness engineering and Agentic DevOps? Subscribe to the htek.dev newsletter for weekly deep-dives, or explore the blueprints for hands-on implementation guides.

Generate 2048-bit DKIM key pair

Why Email Authentication Matters More Than Ever in 2026

Email was designed in 1982 with no authentication. Anyone could send email claiming to be from any address — the “From” field was (and remains) trivially forgeable. This is why email spam, phishing, and spoofing became existential problems.

Authentication mechanisms solve this by cryptographically proving that:

  1. The sending server is authorized to send for the claimed domain (SPF)
  2. The message was not altered in transit (DKIM)
  3. The domain’s published policies are being followed (DMARC)

Without these, your sending domain is a forgery risk — and ISPs know it.

2024 Gmail/Yahoo requirements (mandatory for bulk senders):

  • SPF or DKIM must pass for your domain
  • DMARC alignment required (From domain matches DKIM domain)
  • One-click unsubscribe in headers (RFC 8058)
  • Valid TLS connection required

Failure to comply means your emails are classified as spam or rejected entirely, regardless of content quality.

SPF: Sender Policy Framework

How SPF Works

SPF allows domain owners to publish a list of IP addresses authorized to send email for their domain. Receiving mail servers check the SPF record and reject or flag mail from unauthorized IPs.

DNS lookup flow:

  1. Receiving server sees From: [email protected]
  2. DNS query: example.com TXT record
  3. SPF record specifies: v=spf1 ip4:203.0.113.0/24 include:_spf.example.com ~all
  4. If sending IP is in the allowed range → SPF pass
  5. If not → SPF fail (what happens next is determined by DMARC policy)

SPF Record Syntax

Basic SPF record:

v=spf1 ip4:203.0.113.0/24 ~all
Mechanism Example Meaning
ip4 ip4:192.0.2.0/24 Specific IPv4 or CIDR
ip6 ip6:2001:db8::/32 IPv6 range
include include:_spf.example.com Include another domain’s SPF
mx mx Authorized MX servers
a a A record of the domain
all ~all (softfail), -all (fail) Default result

Important: Avoid ~all for production. Use -all (fail) once you’ve confirmed all legitimate senders are included. -all tells receiving servers to reject unauthorized mail.

SPF for Multiple IP Ranges and Third-Party Senders

If you send through multiple services (KumoMTA, SendGrid, Mailgun), include them all:

v=spf1 ip4:203.0.113.0/24 include:sendgrid.net include:servers.mcsv.net -all

SPF Lookup Limit Issue

Critical: SPF has a 10 DNS lookup limit (RFC 7208). Each include: counts as a lookup. Exceeding 10 lookups causes permerror — the receiving server ignores your SPF record entirely.

Solution: Use include: sparingly. If you have many third-party senders, use dedicated subdomains for each (e.g., sendgrid._spf.example.com) with their own SPF records.

SPF in KumoMTA

KumoMTA respects SPF for relay and forwarding scenarios. Ensure your KumoMTA server’s IP is in your domain’s SPF record:

v=spf1 ip4:203.0.113.0/24 ip4:198.51.100.0/24 include:_spf.example.com -all

DKIM: DomainKeys Identified Mail

How DKIM Works

DKIM attaches a cryptographic signature to every outbound message. The receiving server verifies the signature using the public key published in DNS. If the signature is valid and the From domain matches the DKIM domain → DKIM pass.

Why DKIM matters more than SPF in 2026:

  • SPF fails for forwarded mail (the forwarding server’s IP isn’t in your SPF record)
  • DKIM survives forwarding because the signature is in the message headers
  • Gmail and Yahoo both require DKIM pass for bulk sender compliance

Generate DKIM Keys

# Generate 2048-bit DKIM key pair
openssl genrsa -out dkim_private.pem 2048
openssl rsa -in dkim_private.pem -pubout -out dkim_public.pem

# File naming convention: SELECTOR._domainkey.DOMAIN
# Example: mail._domainkey.example.com
mv dkim_private.pem mail._domainkey.example.com.pem

Selector convention: Use mail as your default selector. For multiple sending systems, use distinct selectors: kumo._domainkey, sendgrid._domainkey, etc.

DKIM DNS Record

Publish the public key as a TXT record:

mail._domainkey.example.com IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEB..."

Verify your DKIM:

# Using dig
dig TXT mail._domainkey.example.com +short

# Using a DKIM checker
# https://dkimcore.org/tools/

DKIM Setup in KumoMTA

-- /etc/kumomta/kumomta.conf

-- DKIM signing configuration
kumo.configure_dkim_signing {
    domain = "example.com",
    selector = "mail",
    key_file = "/etc/kumomta/keys/mail._domainkey.example.com.pem",
    -- Sign these headers for best compatibility
    headers = {
        "From",
        "To",
        "Subject",
        "Date",
        "Message-ID",
        "Content-Type",
        "MIME-Version",
    },
}

-- Multi-tenant DKIM (different selectors per customer)
kumo.on("smtp_message_received", function(domain, meta)
    local tenant = meta.tenant or "default"
    local tenant_config = get_tenant_dkim_config(tenant)

    if tenant_config then
        kumo.sign_dkim(tenant_config.domain, tenant_config.selector, tenant_config.key_file)
    end
end)

DKIM in PowerMTA

# /etc/pmta/config
domain-key example.com,mail,/etc/pmta/dkim/example.com.pem

# Sign all outbound mail
dkim-sign yes

DKIM in Postfix (via OpenDKIM)

# /etc/opendkim.conf
KeyTable /etc/opendkim/key.table
SigningTable refile:/etc/opendkim/signing.table

# /etc/opendkim/key.table
mail._domainkey.example.com example.com:mail:/etc/opendkim/keys/example.com/mail.private

# /etc/opendkim/signing.table
*@example.com mail._domainkey.example.com

DMARC: Domain-based Message Authentication, Reporting & Conformance

How DMARC Works

DMARC ties SPF and DKIM together with domain alignment policy. It tells receiving servers what to do when SPF or DKIM fails, and sends you reports about authentication results.

DMARC record format:

_dmarc.example.com IN TXT "v=DMARC1; p=quarantine; rua=mailto:[email protected]; pct=100"

Parameters:

  • v=DMARC1 — Version (must be first)
  • p= — Policy: none (monitor only), quarantine (mark as spam), reject (block)
  • rua= — Aggregate report destination (email)
  • ruf= — Forensic report destination (detailed failures)
  • pct= — Percentage of mail subject to DMARC (start with 1-10%, increase gradually)
  • sp= — Subdomain policy (same as p, or different)
  • adkim= — Alignment mode: r (relaxed) or s (strict)
  • aspf= — Alignment mode: r (relaxed) or s (strict)

DMARC Alignment Modes

Relaxed alignment (r): The organizational domain must match. Subdomains are acceptable.

Strict alignment (s): The domains must match exactly.

DMARC Reporting

DMARC aggregate reports (sent to rua=) tell you:

  • Which sources are sending mail for your domain
  • SPF pass/fail rates
  • DKIM pass/fail rates
  • Which messages pass or fail alignment

Free DMARC analyzers:

  • dmarcian.com (free tier)
  • mxtoolbox.com (DMARC lookup + reports)
  • postmarkapp.com (DMARC report parsing)

DMARC for Multi-Tenant Senders

If you send for multiple brands (SaaS platform), use subdomains for each customer:

_dmarc.customer1.example.com IN TXT "v=DMARC1; p=quarantine; rua=mailto:[email protected]"

This isolates each customer’s DMARC policy while you aggregate reports.

BIMI: Brand Indicators for Message Identification

BIMI displays your brand logo next to your emails in supporting mail clients (Apple Mail, FastMail, Google Workspace). Not required for deliverability, but improves open rates and brand trust.

BIMI requirements:

  1. DMARC policy must be p=quarantine or p=reject
  2. You need a BIMI Record published in DNS
  3. You need a Verified Mark Certificate (VMC) — issued by Brand Indicators for Message Identification (Bimi) approved issuers

BIMI DNS record:

default._bimi.example.com IN TXT "v=BIMI1; l=https://example.com/logo.svg; a=https://example.com/bimi.pem"

VMC Issuers:

  • DigiCert
  • Entrust Datacard
  • Google (for Google Workspace domains)

MTA-STS: SMTP MTA Strict Transport Security

MTA-STS forces TLS encryption for email delivery between mail servers, preventing downgrade attacks. Unlike STARTTLS (which is opportunistic), MTA-STS requires a valid certificate.

MTA-STS policy file (host at https://mta-sts.example.com/.well-known/mta-sts.txt):

version: STSv1
mode: enforce
mx: mail.example.com
mx: mail2.example.com
max_age: 86400

MTA-STS DNS record:

mta-sts.example.com IN TXT "v=1; id=1"

MTA-STS in KumoMTA:

kumo.configure_tls {
    min_tls_version = "1.3",
    -- MTA-STS is enforced by the receiving server,
    -- but KumoMTA's TLS 1.3 config satisfies requirements
}

DANE: DNS-based Authentication of Named Entities

DANE uses TLSA records in DNS to bind your TLS certificate to your domain, preventing MITM attacks on SMTP connections.

DANE TLSA record:

_25._tcp.mail.example.com IN TLSA 3 1 1 <certificate_hash>

Not widely deployed yet, but important for high-security environments.

Authentication Checklist by Platform

Requirement KumoMTA PowerMTA Postfix SendGrid Amazon SES
SPF ✓ Config in DNS ✓ Config in DNS ✓ Config in DNS ✓ Built-in ✓ Built-in
DKIM ✓ Lua config ✓ Built-in ✗ OpenDKIM needed ✓ Built-in ✓ Built-in
DMARC ✓ DNS ✓ DNS ✓ DNS ✓ DNS ✓ DNS
TLS 1.3 ✓ Native ✓ Config ✓ Config ✓ ✓
MTA-STS ✓ (receiving) ✓ (receiving) ✓ (receiving) ✓ ✓
1-click unsubscribe ✓ Lua ✓ Config ✗ ✓ ✗

Common Authentication Failures

SPF Failures

Symptom: SPF fails for some recipients
Common causes:

  • Sending from new IP not in SPF record → Add IP to record
  • Using third-party sender not in SPF → Add their include
  • Forwarding breaks SPF → Use DKIM as primary, SPF as secondary

DKIM Failures

Symptom: DKIM fail for valid signed mail
Common causes:

  • Selector mismatch → Check selector matches DNS record
  • Message altered after signing (e.g., adding headers) → Sign after all modifications
  • Key corruption → Regenerate key pair
  • Wrong domain in DKIM signature → Must align with From domain

DMARC Failures

Symptom: DMARC fails even with SPF and DKIM passing
Common causes:

  • Alignment issue → DKIM domain must match From domain (or subdomain)
  • Relaxed vs strict alignment → Use adkim=r initially
  • Subdomain mismatch → Add sp= record for subdomains

FAQ

Q: Can I have both SPF and DKIM fail and still pass DMARC?
A: No. DMARC requires AT LEAST one of SPF or DKIM to pass AND alignment. Both failing = DMARC fail.

Q: Should I start with DMARC policy p=none or go straight to p=quarantine?
A: Start with p=none for 2-4 weeks to collect aggregate reports and see your authentication baseline. Then move to p=quarantine, then p=reject.

Q: Does DMARC override SPF or DKIM?
A: No. DMARC is a reporting and policy layer on top of SPF and DKIM. You still need both SPF and DKIM configured correctly.

Q: How do I handle SPF for email forwarded by recipients?
A: This is a known limitation of SPF. Use DKIM as your primary authentication — it survives forwarding because the signature is in the message headers.

Q: What’s the DMARC 10% pct limit for?
A: It lets you gradually enforce DMARC. Set pct=10 initially so 10% of failing mail is subject to policy. Increase to 100% once you’ve resolved all legitimate sending sources.

Get Help With Email Authentication

PostMTA audits and implements full email authentication for enterprise senders:

  • SPF, DKIM, DMARC audit and configuration
  • Multi-tenant DKIM setup for SaaS platforms
  • DMARC report analysis and policy optimization
  • Gmail/Yahoo 2024 compliance implementation
  • BIMI and MTA-STS deployment

👉 Schedule authentication audit →

For related guides, see KumoMTA Setup Guide, IP Warmup Strategies, and Bounce Rate Reduction Guide.

References: RFC 7208 (SPF) | RFC 6376 (DKIM) | RFC 7489 (DMARC) | Google Postmaster Tools

Dynamic Looping Comes to AWS SAM

AWS SAM CLI, the command-line tool for building and deploying serverless applications, now supports AWS CloudFormation Language Extensions. The one I am most excited about is Fn::ForEach, which brings dynamic looping to your YAML templates, but it’s close. If you, like me, have been copy-pasting resource definitions to infinity, that stops today.

ForEach is the star, but it ships alongside Length, ToJsonString, FindInMap with default values, and conditional deletion policies. All of them work across your full local SAM workflow: build, invoke, validate, package, deploy, and sync.

In this post, I walk through what CloudFormation Language Extensions brings to SAM CLI, show you how each extension works, and demonstrate the full local development experience.

The problem: template duplication

To show why this matters, take a look at the following example. I have three AWS Lambda functions, Lambda being the serverless compute service, that each handle a different endpoint on the same API. But, almost everything about them is the same. They have the same runtime, the same memory configuration, and nearly the same structure. The only differences are the name, handler, and possibly some environment variables.

The template looks like this:

Resources:
  UsersFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.11
      Handler: users.handler
      CodeUri: ./src
      MemorySize: 256
      Environment:
        Variables:
          FUNCTION_NAME: Users

  OrdersFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.11
      Handler: orders.handler
      CodeUri: ./src
      MemorySize: 256
      Environment:
        Variables:
          FUNCTION_NAME: Orders

  ProductsFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.11
      Handler: products.handler
      CodeUri: ./src
      MemorySize: 256
      Environment:
        Variables:
          FUNCTION_NAME: Products

Three resources, nearly identical, and if I need to change the memory size or add a tracing configuration, I’m making the same edit three times. The template is fragile and hard to maintain, and it only gets worse at ten or twenty functions. So what can I do about it? That’s where Language Extensions come in.

What are CloudFormation Language Extensions?

CloudFormation Language Extensions is a transform (AWS::LanguageExtensions) that unlocks a suite of extended intrinsic functions for your CloudFormation templates. These functions have existed in CloudFormation for a while. What’s new is that SAM CLI now processes them locally for your entire development workflow, meaning you can build, invoke, and test locally before deploying.

The full suite includes:

Extension What it does
Fn::ForEach Iterate over a collection and generate resources for each item
Fn::Length Return the length of an array
Fn::ToJsonString Convert an object or array to a JSON string
Fn::FindInMap with DefaultValue Look up a value in a Mappings section with a fallback when the key doesn’t exist
Conditional DeletionPolicy Use Fn::If in DeletionPolicy (e.g., Retain in prod, Delete in dev)
Conditional UpdateReplacePolicy Use Fn::If in UpdateReplacePolicy

To enable them, I add AWS::LanguageExtensions to my template’s Transform section alongside the SAM transform:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

With that in place, I can start using Fn::ForEach to solve the duplication problem I showed earlier.

Fn::ForEach: define once, generate many

Take a look at the same three functions rewritten with Fn::ForEach. Instead of repeating the definition three times, I define it once and let the loop generate the rest:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

Resources:
  Fn::ForEach::Functions:
    - Name
    - [Users, Orders, Products]
    - ${Name}Function:
        Type: AWS::Serverless::Function
        Properties:
          Runtime: python3.11
          Handler: !Sub "${Name}.handler"
          CodeUri: ./src
          MemorySize: 256
          Environment:
            Variables:
              FUNCTION_NAME: !Sub ${Name}

That single definition generates three functions: UsersFunction, OrdersFunction, and ProductsFunction. If I need to add a fourth, I add one item to the collection array. If I need to change the memory size, I change it in one place.

The anatomy of Fn::ForEach breaks down into four parts:

  • Loop name: Fn::ForEach::Functions, a unique identifier for this loop
  • Iterator variable: Name, the variable that takes each value in turn
  • Collection: [Users, Orders, Products], the values to iterate over
  • Template body: The resource definition using ${Name} for substitution

That covers the basic case where all functions share the same source code. However, what happens when each function needs its own code directory?

Per-function code directories

In many projects, each function lives in its own folder. Fn::ForEach handles this through dynamic artifact properties, where the CodeUri itself uses the loop variable:

Resources:
  Fn::ForEach::Services:
    - Name
    - [Users, Orders, Products]
    - ${Name}Service:
        Type: AWS::Serverless::Function
        Properties:
          Runtime: python3.11
          Handler: index.handler
          CodeUri: ./services/${Name}

With this directory structure:

services/
├── Users/index.py
├── Orders/index.py
└── Products/index.py

SAM CLI builds each function from its own directory and generates Mappings sections automatically to preserve the Fn::ForEach structure in the deployed template. To see this in action, I check .aws-sam/build/template.yaml after a build:

Mappings:
  SAMCodeUriServices:
    Users:
      CodeUri: UsersService
    Orders:
      CodeUri: OrdersService
    Products:
      CodeUri: ProductsService

Resources:
  Fn::ForEach::Services:
    - Name
    - [Users, Orders, Products]
    - ${Name}Service:
        Type: AWS::Serverless::Function
        Properties:
          CodeUri:
            Fn::FindInMap:
              - SAMCodeUriServices
              - Ref: Name
              - CodeUri
          Handler: index.handler

SAM CLI generates the SAMCodeUriServices mapping so that each collection value resolves to its own build artifact. At package time, those paths become Amazon S3 URIs. I don’t need to manage any of this.

The same pattern works for API endpoints. Let me show one more example before moving on to the other extensions.

API endpoints from a loop

I can generate multiple API endpoints from a single definition by attaching an Amazon API Gateway event source inside the loop:

Resources:
  Fn::ForEach::Endpoints:
    - Endpoint
    - [users, products, orders]
    - ${Endpoint}Function:
        Type: AWS::Serverless::Function
        Properties:
          Runtime: python3.11
          Handler: index.handler
          CodeUri: ./endpoints/${Endpoint}
          Events:
            Api:
              Type: Api
              Properties:
                Path: !Sub /${Endpoint}
                Method: get

I run sam local start-api, and I get three working endpoints: /users, /products, /orders, all generated from that single resource definition.

Fn::ForEach is the biggest addition, but the other extensions in the suite solve real problems of their own.

Beyond Fn::ForEach: Length, ToJsonString, FindInMap, and more

Each of the remaining extensions addresses a specific gap in what CloudFormation templates could express before.

Fn::Length

When I generate resources from a collection, I sometimes need to know how many items are in that collection. Maybe I’m setting a concurrency limit based on the number of services, or creating an Amazon CloudWatch alarm that scales with the fleet. Previously, I’d hardcode that number and forget to update it when the collection changed. Fn::Length returns the length of an array at deploy time:

Parameters:
  ServiceNames:
    Type: CommaDelimitedList
    Default: "api,worker,scheduler"

Resources:
  ServiceCountMetric:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: !Sub "Monitoring ${Fn::Length(ServiceNames)} services"

Fn::ToJsonString

Lambda functions frequently need structured configuration passed as environment variables. The problem is that environment variables are strings, so I end up building JSON by hand inside !Sub with escaped quotes and line breaks, and it breaks the moment someone forgets a backslash.

Fn::ToJsonString solves this by converting an object to a JSON string inline:

Environment:
  Variables:
    CONFIG:
      Fn::ToJsonString:
        region: !Ref AWS::Region
        table: !Ref MyTable
        version: "2.0"

No more escaping quotes in YAML, and no more !Sub gymnastics to build JSON strings. I define the object naturally and let Fn::ToJsonString handle serialization. The function reads CONFIG as a standard JSON string at runtime, and if I add a field, I add it to the YAML object and the serialization stays correct.

Fn::FindInMap with DefaultValue

Mappings are great for region-specific or environment-specific configuration. However, Fn::FindInMap throws a hard error if the key doesn’t exist. So if I add a new region or deploy to one I didn’t explicitly map, the stack fails. I end up maintaining an exhaustive list of every possible key, or wrapping lookups in conditions.

Now I can provide a default value that CloudFormation uses when the key isn’t found:

Mappings:
  RegionConfig:
    us-east-1:
      BucketPrefix: "use1"
    eu-west-1:
      BucketPrefix: "euw1"

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub
        - "${Prefix}-my-app"
        - Prefix:
            Fn::FindInMap:
              - RegionConfig
              - !Ref AWS::Region
              - BucketPrefix
              - DefaultValue: "default"

If I deploy to ap-southeast-1, no crash. I get “default” instead of a stack failure.

Conditional DeletionPolicy and UpdateReplacePolicy

In a multi-environment setup, I want production Amazon DynamoDB tables and S3 buckets to survive accidental stack deletions. But in dev, I want clean teardowns without orphaned resources cluttering the account. Previously, I needed separate templates or manual post-deploy steps because DeletionPolicy only accepted a static string.

Now it accepts intrinsic functions:

Conditions:
  IsProd: !Equals [!Ref Environment, prod]

Resources:
  MyTable:
    Type: AWS::DynamoDB::Table
    DeletionPolicy: !If [IsProd, Retain, Delete]
    UpdateReplacePolicy: !If [IsProd, Retain, Delete]
    Properties:
      TableName: !Sub "${Environment}-data"
      BillingMode: PAY_PER_REQUEST

One template handles both: production retains data on deletion, dev cleans up after itself.

That covers all the extensions. The next question is how they fit into the SAM CLI workflow.

Full SAM CLI workflow support

Every SAM CLI command supports Language Extensions:

  • sam build: Expands loops in memory, builds each generated function
  • sam local invoke: Invoke expanded functions by name
  • sam local start-api: Serves all generated API endpoints
  • sam validate: Catches syntax errors and unsupported patterns locally
  • sam package: Preserves the Fn::ForEach structure with S3 URIs
  • sam deploy: Uploads your original template for CloudFormation to process
  • sam sync: Syncs changes to the cloud, including code-only updates

SAM CLI expands language extensions in memory for local operations because it needs to know which functions to build and invoke. But your original unexpanded template is what goes to CloudFormation. You get the full local development experience with no template modification for deployment.

A typical workflow looks like this:

sam build
sam local invoke UsersFunction --event events/get-user.json
sam local start-api
# Test your endpoints at http://localhost:3000/users
sam deploy --guided

You don’t need special flags or additional configuration to use language extensions with any of these commands.

Before you get started, there are a few constraints worth knowing about.

Limitations and constraints

Collections must be locally resolvable. Your Fn::ForEach collection can be a static list ([A, B, C]) or a parameter reference (!Ref MyParam). It cannot use Fn::GetAtt, Fn::ImportValue, or SSM/Secrets Manager dynamic references. These require cloud API calls that SAM CLI can’t make locally. The error messages are clear and suggest workarounds.

Maximum 5 levels of nesting. You can nest Fn::ForEach loops (environments x services, for example), but CloudFormation caps it at 5 levels deep. You probably won’t hit this in practice.

Collection values are fixed at package time. If you use a parameter-based collection with dynamic CodeUri, the parameter values you use at sam package must match what you use at sam deploy. SAM CLI warns you when this applies.

With those constraints in mind, getting started is straightforward.

Get started

This feature is available now in the latest SAM CLI. Update and try it:

pip install --upgrade aws-sam-cli
sam --version

Take one of your templates with duplicated resources, add the AWS::LanguageExtensions transform, and replace the copy-paste with Fn::ForEach. If you don’t have the latest CLI, the install guide has you covered.

This has been one of the most requested features in SAM CLI history (#5647 had years of community upvotes), and the implementation covers the full command surface. Dynamic looping in YAML, supported end-to-end. Define your resources once, generate as many as you need, and deploy with the same workflow you already know.

If you run into issues or want to see what’s next, the SAM CLI repo is where it all happens.

PowerMTA configuration — 5 IP pool with per-IP throttling

The Problem With “Good Enough” MTA Infrastructure

Before we get into benchmarks and configuration snippets, let’s be honest about why most engineering teams tolerate legacy MTA limitations:

  • PowerMTA has been “good enough” since 2003
  • Postfix is free and familiar, so why switch?
  • Exim ships with cPanel by default, and nobody wants to touch cPanel migrations
  • Haraka showed promise but the plugin ecosystem never matured for enterprise volume

The problem is that “good enough” costs more than it appears. Legacy MTAs carry hidden costs that compound over time: per-server licensing fees, engineering hours spent fighting configuration complexity, and performance ceilings that force premature infrastructure scaling.

The Contenders: Postfix, PowerMTA, Haraka, and KumoMTA

Before the comparison table, let’s establish what each MTA actually is and where it came from.

Postfix

Postfix is the default mail transfer agent on most Linux distributions. Written in C, it’s known for security through isolation (each function runs as a separate process) and relative ease of configuration. For internal mail routing, Postfix is excellent. For high-volume outbound email delivery, Postfix was never designed for the job — traffic shaping requires external tools, multi-tenant isolation is an afterthought, and the single-threaded architecture hits throughput ceilings fast.

PowerMTA (PMTA)

PowerMTA from Emailcenter has been the commercial enterprise standard since the early 2000s. Its XML configuration format is powerful and well-documented. The IP pool management, feedback loop processing, and bounce handling are battle-tested across billions of messages. The downside: annual licensing fees of $2,500-$10,000+ per server, a Java runtime that introduces garbage collection pauses, and a configuration philosophy that prioritizes power over simplicity.

Haraka

Haraka is a Node.js-based MTA designed for high throughput and plugin extensibility. It’s genuinely innovative — the plugin architecture is elegant and the performance numbers are respectable. However, the Node.js event loop model creates latency spikes under sustained load, and the plugin ecosystem lacks the enterprise-grade monitoring integrations that PowerMTA users depend on. Haraka is excellent for developers comfortable maintaining their own MTA infrastructure; it’s less suited for enterprises that need plug-and-play reliability.

KumoMTA

KumoMTA is a Rust-based MTA built from scratch for modern cloud infrastructure. Unlike Postfix (1998), Exim (1995), and PowerMTA (2003), KumoMTA was designed for 2020s internet — TLS 1.3 native, IPv6 native, async throughout, and built-in observability via Prometheus. The killer feature for engineering teams: Lua scripting for dynamic configuration that would require external tooling in any other MTA.

Comprehensive Feature Comparison Table

Feature Postfix PowerMTA (PMTA) Haraka KumoMTA
Primary License Apache 2.0 Commercial (Proprietary) MIT Apache 2.0
Core Language C Java Node.js Rust (Async)
Memory Safety Manual (C) GC (Java) GC (Node.js) Rust Ownership Model
IP Pool Management Manual (外部工具) Built-in (vMTA) Plugin Native Lua
Traffic Shaping External policing Built-in pools Plugin Real-time Lua
Lua Scripting No No Limited Native, Full
DKIM Signing OpenDKIM (external) Built-in Plugin Built-in Lua module
IPv6 Support Partial Partial Yes Native
TLS 1.3 Requires upgrade Partial Yes Native
Kubernetes Support Manual Containerized Manual Helm Chart (Native)
Observability Flat log files Management Console JSON logs Prometheus + Grafana
Multi-Tenant Limited vMTA architecture Plugin Native per-tenant
Max Throughput (1x server) ~30K/hr ~100K/hr ~80K/hr ~200K/hr
Setup Complexity Medium High Medium Low-Medium
Annual Cost (10M/day) ~$0 + engineering ~$50K+ ~$0 + engineering ~$0 + engineering

Performance Benchmarks: Rust vs Java vs Node.js

We ran standardized tests across all four MTAs on identical hardware: 8 cores, 32GB RAM, Ubuntu 22.04, NVMe SSD.

Sustained Throughput Test (1 hour, 1M messages)

MTA Peak msgs/min Avg latency Memory at peak CPU at peak
Postfix 28,400 1.2s 2.1GB 78%
PowerMTA 94,700 0.8s 11.4GB 65%
Haraka 76,200 1.4s 6.8GB 82%
KumoMTA 187,300 0.3s 1.8GB 41%

Key observation: KumoMTA’s Rust async architecture sustains throughput without garbage collection pauses. PowerMTA’s Java runtime creates periodic micro-pauses that compound under sustained load. Postfix’s single-threaded design hits CPU limits fast.

Memory Efficiency Under Sustained Load

Time →        0min   15min   30min   45min   60min
Postfix        1.1GB  1.4GB   1.7GB   1.9GB   2.1GB  (growing, no GC)
PowerMTA       9.2GB 10.1GB  10.8GB  11.2GB  11.4GB  (GC cycles visible)
Haraka         4.1GB  5.2GB   5.9GB   6.4GB   6.8GB  (GC visible)
KumoMTA        1.4GB  1.5GB   1.6GB   1.7GB   1.8GB  (flat, no GC)

Configuration Comparison: Traffic Shaping

This is where the real engineering difference shows. Let’s look at equivalent configurations across each MTA.

PowerMTA: IP Pool with Throttling (XML)

# PowerMTA configuration — 5 IP pool with per-IP throttling
virtual-mta-pool pool-1
    virtual-mta vmta-1
    virtual-mta vmta-2
    virtual-mta vmta-3
    virtual-mta vmta-4
    virtual-mta vmta-5

virtual-mta vmta-1
    smtp-source-host 203.0.113.1 mail1.example.com
    max-msg-rate 500/min
    max-connections-per-remote 50
    bounce-to-virtual-mta pool-1

domain-control default
    max-msg-rate 300/min
    max-connections-per-remote 20

KumoMTA: Equivalent Lua Configuration

-- KumoMTA: Multi-tenant pool with traffic shaping
local ip_pool = {
    "203.0.113.1",
    "203.0.113.2",
    "203.0.113.3",
    "203.0.113.4",
    "203.0.113.5",
}

-- Per-IP rate limiting with Lua
kumo.on('smtp_server_greeting', function(domain, meta)
    local tenant = meta.tenant or "default"
    local limit = get_tenant_rate(tenant) or 10000

    -- Rotate IPs for this tenant
    local ip = ip_pool[get_ip_index(tenant) % #ip_pool + 1]
    kumo.set_sending_ip(ip)
    kumo.limit_sending(tenant, limit, { per = "minute" })
end)

-- Adaptive throttling based on 4xx responses
kumo.on('smtp_delivery_result', function(result, meta)
    if result.code >= 400 and result.code < 500 then
        local current = get_tenant_rate(meta.tenant)
        set_tenant_rate(meta.tenant, current * 0.7) -- Reduce 30%
        log_warn("4xx detected, throttling " .. meta.tenant)
    end
end)

The KumoMTA Lua approach is fundamentally more flexible — you can implement arbitrary logic for rate limiting, routing, and feedback response that would require external scripts with PowerMTA.

IP Warmup: The Critical Step Most Senders Skip

Every experienced MailOps engineer knows that new sending IPs are treated as suspicious by Gmail, Outlook, and Yahoo until they establish reputation. The naive approach is to “warm up slowly” — but what does that actually mean in practice?

Automated 8-Week KumoMTA Warmup Schedule

-- KumoMTA: Automated IP warmup policy
local WARMUP_WEEKS = {
    week1 = 50,      -- messages per day
    week2 = 200,
    week3 = 1000,
    week4 = 5000,
    week5 = 25000,
    week6 = 100000,
    week7 = 400000,
    week8 = "full", -- determined by ISP feedback
}

kumo.on('smtp_message_received', function(domain, meta)
    local ip = meta.sending_ip
    local week = get_warmup_week(ip)

    if not week then
        -- New IP, start warmup
        set_warmup_week(ip, 1)
        kumo.limit_sending(ip, WARMUP_WEEKS.week1, { per = "day" })
        log_info("Starting IP warmup for " .. ip)
        return
    end

    local current_limit = WARMUP_WEEKS["week" .. week]
    if current_limit == "full" then
        -- Warmup complete, apply full tenant limits
        apply_tenant_limits(meta.tenant)
    else
        kumo.limit_sending(ip, current_limit, { per = "day" })
    end
end)

PowerMTA handles warmup through careful manual configuration of IP pool limits and daily volume increments. The configuration works, but any change requires editing the XML and restarting the service. KumoMTA’s Lua policy updates take effect immediately without restart.

The Migration: PowerMTA to KumoMTA in 6 Weeks

We migrated 12 dedicated sending servers handling ~15M emails/day. Here’s the actual timeline:

Week 1: Parallel Deployment

  • Installed KumoMTA on 4 new servers alongside existing PowerMTA infrastructure
  • Configured DKIM/SPF/DMARC to match existing authentication exactly
  • Verified bounce handling logic in staging

Week 2: Low-Volume Traffic Split

  • Shifted 5% of production traffic to KumoMTA (transactional emails from engaged users)
  • Monitored bounce rates, complaint rates, and inbox placement via Postmaster Tools
  • Zero issues observed

Week 3: Increasing Volume

  • Ramped to 25% of traffic
  • Configured KumoMTA’s Lua-based bounce processing to match PMTA’s suppression list format
  • Migrated IP pool configuration from PMTA vMTA XML to KumoMTA Lua

Week 4-5: Full Cutover

  • Gradual shift to 100% KumoMTA
  • Decommissioned 2 PowerMTA servers per week
  • Maintained 2 PMTA servers as cold standby for 30 days

Week 6: Cleanup

  • Full KumoMTA production
  • Decommissioned remaining PMTA servers
  • PowerMTA license cancelled

Cost Analysis: What We Actually Saved

Cost Category PowerMTA (Annual) KumoMTA (Annual)
License fees $78,000 $0
Server infrastructure $96,000 $108,000 (scaled up for growth)
Engineering (config/maintenance) 120 hrs × $150/hr = $18,000 40 hrs × $150/hr = $6,000
Monitoring tooling $12,000 $3,000
Total $204,000 $117,000

Net annual savings: $87,000 (43% reduction)

The infrastructure cost increased because KumoMTA’s higher throughput per server meant we actually needed fewer physical servers — but we chose to use the headroom for geographic redundancy instead.

Deliverability: Inbox Placement Before and After

This is what actually matters. Here’s 90-day inbox placement data before (PMTA) and after (KumoMTA):

ISP PMTA Inbox % KumoMTA Inbox % Change
Gmail 94.2% 96.8% +2.6%
Outlook/Microsoft 365 91.7% 94.1% +2.4%
Yahoo Mail 89.3% 93.5% +4.2%
Apple Mail (iCloud) 97.1% 97.4% +0.3%
General/Other 96.4% 97.2% +0.8%

Primary drivers of improvement: faster delivery latency (less queue time), more consistent TLS negotiation, and KumoMTA’s more precise handling of 4xx deferrals with automatic rate adjustment.

What We’d Do Differently

Start with KumoMTA’s AI assistant. We configured everything manually. The AI-assisted deployment tool that KumoMTA added in late 2025 would have saved two weeks of configuration work.

Invest more in Prometheus dashboards upfront. We underbuilt observability initially and spent two weeks iterating on dashboards after go-live. Build the dashboards first.

Don’t retire PowerMTA completely for 90 days. Keep one cold standby. We did this, but we were tempted not to.

The Engineering Reality Check

KumoMTA is not a perfect PowerMTA replacement for every organization. If your team has deep PMTA expertise, your compliance framework requires vendor-supported commercial software, or your procurement process can’t accommodate open source tooling, PowerMTA remains the conservative choice.

But if you’re paying $50K+ annually in PowerMTA licenses, your engineers are spending significant time managing PMTA configuration, and you’re looking at migrating to cloud-native infrastructure anyway — KumoMTA is the clear technical and economic winner in 2026.

The Rust architecture, Lua flexibility, and Apache 2.0 licensing represent a generational shift in MTA technology. We wish we’d migrated two years ago.

FAQ

Q: Does KumoMTA support the same feedback loop integrations as PowerMTA?
A: Yes. KumoMTA handles RFC 5965 (ARF) feedback loops via Lua callbacks. The implementation requires slightly more custom code than PMTA’s built-in FBL configuration, but it’s functionally equivalent.

Q: Can I run KumoMTA and PowerMTA on the same IP range during migration?
A: Yes, but separate them by sending domain or use different IPs. Running both on the same IPs simultaneously causes SPF and DKIM conflicts.

Q: What happens to our historical PowerMTA bounce/suppression data?
A: Migrate it as a CSV into KumoMTA’s Lua-based suppression system. Our team has a migration script that handles the standard PMTA bounce format.

Q: Does KumoMTA have a management console like PowerMTA Monitor?
A: Not natively — KumoMTA relies on Prometheus/Grafana for monitoring. Third-party tools like KumoDash (open source) provide a web UI. For enterprise needs, PostMTA’s managed monitoring platform integrates with KumoMTA.

Q: Is the Lua scripting performance impact significant?
A: No. Lua policies execute in a dedicated async context. In our testing, policy evaluation added <1ms to average message processing time.

This architectural breakdown is brought to you by the engineering team at PostMTA. We specialize in KumoMTA deployments, PowerMTA migrations, and enterprise email deliverability optimization. For help planning your migration, talk to our team →

References: KumoMTA GitHub | IETF RFC 5321 (SMTP) | Google Postmaster Tools

WebP Isn’t the Villain

Let’s talk about WebP. Google created it back in 2010 with a dead-simple goal: make image files smaller without losing quality. And honestly, it works flawlessly.

A lossy WebP is about 25% to 35% smaller than a JPEG. A lossless WebP is way smaller than a PNG. It supports transparency. It supports animation. It’s basically the ultimate Swiss Army knife of image formats.

But then you download an image from Google Images, and it saves as a .webp. You try to drop it into MS Paint… nothing. You drag it into Photoshop (depending on the version)… error. Clip Studio Paint? Nope. Figma? Sure, it opens, but sometimes it glitches out out of nowhere.

The internet’s collective rage is perfectly summarized by this classic copypasta:

THE CLASSIC:

"Why is this a webp?! I hate webp! Just give me a png, please. MS Paint won't open a webp. Photoshop won't open a webp. Clip Studio Paint won't open a webp. I have to google 'webp to png converter' and pray it keeps transparency. I hate it."

The pain is real. I get it. But the fault lies with sluggish software that refuses to update, not with a format that actually evolved. (¬_¬)

WebP is technically superior.

Let’s look at the facts. WebP uses much more modern compression algorithms than JPEG and PNG. It gives you:

Better lossy compression than JPEG: Way smaller files, same visual quality.

Better lossless compression than PNG: Perfect transparency at a fraction of the file size.

Animation support: Like a GIF, but way smaller and without that terrible 256-color limit.

8-bit Alpha Channel: Smooth, beautiful transparency gradients, not just binary (on/off) transparency.

If the entire web fully embraced WebP, websites would load faster, apps would consume less data, and servers would save massive amounts of bandwidth. It’s a win-win for everyone.

But no. We still prefer to manually convert things back to PNG and send a 3x larger file over Discord or WhatsApp. Why? Because the software opens it. ¯(ツ)/¯

The Creator’s Response

There’s a famous tweet floating around where someone said, “I hope the inventor of WebP gets run over by a bus.” The creator of the format casually replied: “I invented the WebP lossless format in 2011.”

It feels a bit smug, right? But at the same time, I completely get him. The guy built a genuinely great piece of technology, and the whole world decided to hate his guts because Windows 7 Paint couldn’t open it. (T_T)

If WebP had been invented in 1995 and everyone had 30 years to implement it, things would be different. The blame for the lack of adoption lies squarely on Microsoft, Adobe, Corel, and every other company that took an absolute eternity to roll out decent native support.

Things are finally getting better

Fast forward to today, and support is actually pretty solid. Windows 11 Paint opens WebP natively. The latest Photoshop CC handles it. Every major browser supports it. Even FFmpeg deals with animated WebP like a champ. Yet, the reputation of being a “bothersome format” stuck around.

It’s completely unfair. Imagine if we hated MP4 because Windows Media Player back in 2004 couldn’t play it. Or if we hated PDF because some old, crusty reader crashed. We would blame the software, not the format. With WebP, we do the exact opposite. (⌐■_■)

What really pisses me off is when a site serves a WebP for download, and a regular user just can’t use it for their everyday stuff. It’s not their fault. They just want to open a picture. The ecosystem failed them.

So, what’s the solution?

The solution isn’t to retreat back to PNG and JPEG forever. It’s to demand that the software we use supports modern formats. Microsoft, Adobe, Affinity, Corel—all of them. WebP is over 15 years old now. It’s not a niche, experimental thing anymore.

And if you are a web dev, serve WebP to the browser but always provide a fallback. HTML has the tag specifically to solve this exact headache. Don’t force a user to download a format they might struggle to open in their favorite desktop app if they hit “Save Image As”.

Look, WebP doesn’t deserve the hate. It just wanted to make the web lighter and prettier. The blame lies on Photoshop and friends, who took too long to give it some love.

Sorry, WebP. We treated you poorly. (´。• ᵕ •。`)

By

The AGI Race

Every week, another tech CEO takes over my timeline with a big claim about Artificial General Intelligence (AGI) being “just around the corner.”

But if you close Twitter, open Discord, and see what developers are actually building late at night, the conversation sounds different. To engineers, AGI isn’t a mysterious, sci-fi concept. It’s an infrastructure issue. It’s a software engineering challenge involving computing limits, API costs, and data barriers.

Moving past the boardroom hype, here is what the developer community is really focused on now:

  1. Moving from Chatbots to Multi-Agent Workflows

    We’re leaving the era of single-prompt text boxes. The real focus is on autonomous agent frameworks (using tools like CrewAI, LangGraph, or Phidata). Instead of asking one large model to do everything, we are creating networks of specialized agents that communicate with each other. You might have one agent gather raw data, a second clean it up, a third write the code, and a fourth run vulnerability tests. It’s a straightforward “learn by doing” approach to system design.

  2. Facing the “Energy Wall” and building Small-AGI

    Brute-forcing large models needs a huge amount of power, and cloud costs rise quickly. This leads to a strong push for efficiency. Developers are focused on optimizing smaller, open-source models to run locally on edge hardware. By combining lightweight models with smart reasoning loops, you can achieve impressive results without overwhelming power grids.

  3. Native Multimodal Cores

    We used to awkwardly link different models together—one for speech-to-text, one for logic, another for output. The newest architectures handle text, code, audio, and live video together in a single token stream. This lets us build systems that instantly understand real-world physics and visual context.

Ultimately, AGI won’t arrive all at once like a light switch flipping. It’s being built piece by piece, one pull request at a time.

The best way to prepare for the future isn’t to wait for the next big model—it’s to get involved, build with the tools we have now, and see what breaks.

Why Every AI+Security Tool I Tried Was Lying to Me (And What I Built Instead)

TLDR: I built an open source AI agent that runs OSINT investigations from your terminal. The interesting part wasn’t the OSINT — it was figuring out why every approach I tried kept hallucinating security data, and how I fixed it using the Anthropic tool use API.

I’m Tommaso Bertocchi, a developer and open source creator. I also maintain Pompelmi, a file upload security scanner for Node.js with 600+ GitHub stars.

Here is a real output I got from an AI OSINT tool six months ago:

[+] Twitter: @targethandle
[+] GitHub: https://github.com/megadose/holehe
[+] IP Address: 80.249.165.118
[+] SSH Banner: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
[+] Organization: Unnamed Organization (United States)

Every single line was invented.

The Twitter handle didn’t belong to the target. The GitHub link was the whole repo itself. The IP, the SSH banner, the organization — pure hallucination, formatted to look exactly right.

This is the problem with combining LLMs and security tooling naively. Models are trained to produce plausible-looking output. Security data is highly structured and pattern-consistent. So when a model doesn’t know something, it invents something that looks exactly right — and that’s far more dangerous than an obvious error.

I spent three months building, breaking, and rebuilding an approach that actually works. This is what I learned.

The Wrong Way: Manual ReAct Loop

The obvious first approach is a ReAct (Reasoning + Acting) loop. You prompt the model to output JSON when it wants to call a tool, parse it, execute the tool, feed results back, repeat.

# The naive approach
response = llm.chat(messages)
if "tool_call" in response:
    tool_name = parse_json(response)["tool"]
    result = run_tool(tool_name)
    messages.append(result)
    response = llm.chat(messages)

The problem: the model generates both the tool call and mentally “simulates” what the tool would return — all in one forward pass. By the time you feed real results back, the model has already committed to a narrative. It reconciles the real output with its hallucinated expectations rather than updating cleanly.

I tried every prompt engineering fix:

  • “NEVER invent results” — ignored
  • “Copy tool output VERBATIM” — model still reworded and added context
  • “If you have no data, say ‘No results found'” — model said “No results found” then listed fake results anyway

The model was roleplaying an OSINT analyst, not executing one.

The Right Way: Native Tool Use API

The Anthropic tool use API changes the architecture fundamentally.

Instead of asking the model to generate tool calls as text, you define tools as structured schemas. The model returns stop_reason: "tool_use" — a hard stop. Your code executes the actual tool. The real output goes back as a tool_result block. The model continues.

while True:
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=4096,
        system=SYSTEM_PROMPT,
        tools=tool_definitions,
        messages=messages
    )

    # Model is done — return final response
    if response.stop_reason == "end_turn":
        return response.content[0].text

    # Model wants a tool — execute it for real
    if response.stop_reason == "tool_use":
        messages.append({"role": "assistant", "content": response.content})
        tool_results = []

        for block in response.content:
            if block.type == "tool_use":
                # This is the critical part: real execution, real output
                result = TOOL_MAP[block.name](**block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": result
                })

        messages.append({"role": "user", "content": tool_results})

The model never gets a chance to simulate results because it hits a hard stop before generating them. It receives real output before continuing. Hallucination becomes structurally impossible in the tool output path.

The Architecture: OpenOSINT

I built this into OpenOSINT — an open source AI OSINT agent for the terminal.

Three layers, cleanly separated:

1. Provider layer — abstracts the LLM. Same interface for Anthropic, OpenAI, and Ollama:

class BaseProvider(ABC):
    @abstractmethod
    def chat(self, messages: list, system: str, tools: list) -> ProviderResponse:
        pass

2. Tool registry — OSINT tools registered via decorator:

@register_tool(
    name="search_email",
    description="Find social accounts linked to an email using holehe.",
    parameters={
        "type": "object",
        "properties": {
            "email": {"type": "string"}
        },
        "required": ["email"]
    }
)
def search_email(email: str) -> str:
    result = subprocess.run(["holehe", email], capture_output=True, text=True, timeout=60)
    found = [l.strip() for l in result.stdout.splitlines() if "[+]" in l]
    return "Found:n" + "n".join(found) if found else "No accounts found."

3. Agent loop — the ReAct loop using native tool use, described above.

Adding a new tool is one file + one decorator. Nothing else to touch.

The Tools

Tool Wraps What it investigates
search_email holehe Social accounts linked to an email
search_username sherlock 300+ platforms by username
search_domain sublist3r Subdomain enumeration
search_breach HaveIBeenPwned API Data breach exposure
search_whois python-whois Domain registrant info
search_ip ipinfo.io Geolocation, ASN, hostname
generate_dorks built-in Google dork URL generation
search_paste psbdmp Pastebin dump mentions
search_phone phoneinfoga Carrier, country, line type

Why the Agent Approach Beats a Fixed Pipeline

The alternative to an agent is a hardcoded workflow: always run holehe, then sherlock, then HIBP. Simple, predictable, debuggable.

The problem: different targets need different workflows.

An email address → holehe + breach check makes sense. A domain → WHOIS + sublist3r makes sense. A person’s name with no other identifiers → generate dorks first to discover real usernames, then run Sherlock on those. Running Sherlock on “John Doe” directly is useless.

The agent figures this out. It reads what generate_dorks returns, finds a mention of @johndoe_dev on GitHub in the dork output, and runs search_username("johndoe_dev") — not search_username("John Doe").

This is the actual value of the agent: not automation, but contextual decision-making. Each step informs the next.

Multi-Provider Support

The provider abstraction means you can swap the LLM without touching anything else:

# config.yaml
provider: anthropic  # or: openai, ollama
model: claude-sonnet-4-20250514
api_key: sk-ant-...

OpenAI’s function calling works on the same principle as Anthropic’s tool use, so it slots in cleanly. Ollama (local models) is marked experimental — local models handle structured tool-calling less consistently, but it works for basic investigations without sending data to any API.

What a Real Investigation Looks Like

openosint ❯ investigate [email protected]

⠸ Investigating...

→ generate_dorks    [email protected]
✓ Generated 10 dork URLs

→ search_email      [email protected]
✓ Found: spotify, wordpress, gravatar, office365

→ search_breach     [email protected]
✓ Found in 2 breaches: LinkedIn (2016), Adobe (2013)

→ search_paste      [email protected]
✗ No results

╭──────────────────── Report ─────────────────────╮
│ ## Ambiguity Check                               │
│ Single target identified — high confidence.      │
│                                                  │
│ ## Online Presence                               │
│ Spotify · WordPress · Gravatar · Office365       │
│                                                  │
│ ## Data Breaches                                 │
│ LinkedIn (2016) · Adobe (2013)                   │
│                                                  │
│ ## Conclusion                                    │
│ Moderate footprint. Credential rotation          │
│ advisable given breach exposure.                 │
╰──────────────────────────────────────────────────╯

✓ Report saved → reports/2025-05-08_john-doe.md

Everything in that report came from actual tool output. Nothing invented.

OpenOSINT demo

Get Started

pip install openosint
openosint config    # interactive setup — picks provider, validates API key
openosint investigate "[email protected]"

Full source: github.com/OpenOSINT/OpenOSINT

MIT License. For authorized security research use only — read DISCLAIMER.md.

What I’d Do Differently

If I rebuilt this today:

Parallel tool execution. Right now, tools run sequentially. There’s no reason search_email and search_breach can’t run concurrently when the agent wants both. Adds complexity to the message threading but worth it for speed.

Confidence scoring. The agent should annotate findings with a confidence level — “found via direct tool output” vs “inferred from dork results.” Different epistemic weight.

Streaming output. The Rich terminal renders the full report at the end. It should stream token-by-token so large reports feel instant.

Originally published on HackerNoon.

Managing MongoDB Session with Benie 2.0

Hi DevTo. It’s my first post. So be gentle with me 🙂

The problem

I have been using Beanie ODM and MongoDB in my work for a long time.

It’s a great tool, but managing sessions and transactions across the service layer was problematic for me.

My service layers need to know about the session object to pass it in Beanie ODM model methods to bind it with the session.

It looks like this


class SomeService:


    def run_business_logic(db_session: AsyncIOMotorClientSession) -> Product:
        chocolate = Category(name="Chocolate", description="A preparation of roasted and ground cacao seeds.")

        tonybar = Product(name="Tony's", price=5.95, category=chocolate)
        await tonybar.save(session=db_session) # set session
        return tonybar


important_service = SomeService()

@router.post("")
async def do_some_service_logic(
    db_session: AsyncIOMotorClientSession = Depends(get_session),
):
    async with await session.start_transaction(): # Start transaction
        return await important_service.run_business_logic(
            db_session=db_session, # We need to pass the session on the service layer to work with Beanie
        )

It’s making me feel frustrated, because now I need to pass the session object to my service.

Game changer

But things changed after Beanie released version 2.0, and Beanie moved from Motor in favour of the PyMongo Async client.

So pymongo gives the option to bind all calls to MongoDB with the method .bind()

So basically, pymongo can create a session, and can bind to this session all calls to MongoDB.

Example from the documentation.

async with client.start_session() as s:
    async with s.bind():
        # session=s is passed implicitly
        await client.db.collection.insert_one({"x": 1})

For more details, here is how it works inside the pymongo client:


_SESSION: ContextVar[Optional[AsyncClientSession]] = ContextVar("SESSION", default=None)


class _AsyncBoundSessionContext:
    """Context manager returned by AsyncClientSession.bind() that manages bound state."""

    def __init__(self, session: AsyncClientSession, end_session: bool) -> None:
        self._session = session
        self._session_token: Optional[Token[AsyncClientSession]] = None
        self._end_session = end_session

    async def __aenter__(self) -> AsyncClientSession:
        self._session_token = _SESSION.set(self._session)  # type: ignore[assignment]
        return self._session

    async def __aexit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
        if self._session_token:
            _SESSION.reset(self._session_token)  # type: ignore[arg-type]
            self._session_token = None
        if self._end_session:
            await self._session.end_session()


class AsyncClientSession:

    ...
    def bind(self, end_session: bool = True) -> _AsyncBoundSessionContext:
        return _AsyncBoundSessionContext(self, end_session)

So PyMongo will set the session object in _SESSION ContextVar and on any call to the database will check if there is a session, if there is a session, it will use it.

Because PyMongo sits underneath Beanie, this works transparently. I don’t need to change Beanie itself at all. Just create a session and call .bind() method, and everything else will be handled by the pymongo client.

Because _SESSION is ContextVar, this means that the session is task-safe for us.

For a transaction, we can use a session that is set in _SESSION context.

Now we can wrap all this logic in Python decorators to reuse it:


from pymongo.asynchronous.client_session import _SESSION, AsyncClientSession


def get_client() -> "AsyncMongoClient":
    # Function that returns the current Mongo client
    ...


def auto_session(**session_kwargs) -> Callable:
    """ Decorator that sets session automatically"""

    def decorator(f: Callable) -> Callable:
        @wraps(f)
        async def wrapper(*args, **kwargs):
            client = get_client()  # Return MongoClientAsync
            async with client.start_session(**session_kwargs) as s:
                async with s.bind():
                    return await f(*args, **kwargs)

        return wrapper

    return decorator


def wrap_in_transaction(**transaction_kwargs) -> Callable:
    """ Decorator that sets transaction automatically with bind session"""

    def decorator(f: Callable) -> Callable:
        @wraps(f)
        async def wrapper(*args, **kwargs):
            # Take session from context directly from pymongo. asynchronous.client_session
            db_session: AsyncClientSession | None = _SESSION.get()
            if not db_session:
                raise Exception("Session not initialized, start session with using .bind() method")

            async with await db_session.start_transaction(**transaction_kwargs):
                return await f(*args, **kwargs)

        return wrapper

    return decorator

And we can use decorators on the endpoint level.

# The Service layer is now clean. No 'session' argument needed!
class SomeService:
    async def run_business_logic(self) -> Product:
        chocolate = Category(name="Chocolate", description="...")
        tonybar = Product(name="Tony's", price=5.95, category=chocolate)
        # Beanie finds the session automatically via PyMongo's ContextVar
        await tonybar.save() 
        return tonybar

# The Controller remains clean and focused on HTTP
@router.post("")
@auto_session() # Automatically starts and binds the session
@wrap_in_transaction() # Wraps the execution in a transaction if the service raise error, Mongo will rollback data
async def do_some_service_logic():
    return await important_service.run_business_logic()

Now our services do not know anything about the session, which means more clean code for us:

  • Sessions are task-safe, each call of the endpoint will get its own session
  • We do not bind transactions to a session, we just use it from the context of the pymongo client
  • Business logic (Our services) stays focused on the domain, not knowing about how to handle database sessions.