generated my first solana keypair from scratch and stared at the terminal for a solid 10 seconds 😅
no username. no email. no password. just a public key staring back at me — and that IS my identity on-chain. that’s the shift. in web2 you log in. In web3 you prove you exist with cryptography.
went from zero setup to sending actual SOL between two wallets on devnet in 5 days. the moment sendAndConfirmTransaction resolved and both balances updated — that’s when it stopped being theory.
what surprised me most: a wallet isn’t a container for your tokens. the tokens don’t “live” in your wallet. your wallet is just a keypair that proves you have authority over them.
next up — going deeper on on-chain identity and program interactions. day 6 and beyond 👀
screenshot of the terminal drop incoming 👇

100DaysOfSolana @solana_devs

This is a submission for the Hermes Agent Challenge

A practical deep-dive for engineers, founders, and curious builders.
Date: 2026-05-18

Hermes Agent is the agent framework that, in roughly twelve weeks since its February 2026 release, has gone from a niche Nous Research project to 140,000+ GitHub stars and the most-used agent on OpenRouter. That growth is not just hype — it reflects a meaningful design shift away from “agents as orchestrated prompt graphs” toward agents as long-lived, self-improving processes that own their own learning artifacts.

Companion reads: 🔮 Hermes Agent 🤖 — Deep Dive & Build-Your-Own Guide 📘 and 🏗️ Building High-Quality AI Agents 🤖 — A Comprehensive, Actionable Field Guide 📚.

This article is a working engineer’s tour:

1. 🧠 What Hermes is, and what genuinely separates it from LangGraph / CrewAI / AutoGen
2. 🏗️ Its core architecture
3. ⚔️ How it compares with two adjacent open-source projects: OpenClaw and GoClaw — when to pick which
4. 🌍 Real-world and personal use cases
5. 🔌 Integration patterns into existing apps and SaaS
6. 🛠️ A setup / extend / customize playbook
7. 💭 An opinion on what open, capable agent systems mean for the future of AI development

1. 🧠 What Hermes Agent Actually Is

Hermes is an open-source, model-agnostic, long-running AI agent built by Nous Research. The tagline — “the agent that grows with you” — is technically literal: Hermes is the only mainstream agent framework with a built-in learning loop that creates, edits, and improves its own skills during normal use.

It ships as:

• A CLI / TUI you run locally (hermes).
• A messaging gateway that turns Telegram / Discord / Slack / WhatsApp / Signal / Email / Matrix into agent surfaces.
• A web UI and an Agent Client Protocol (ACP) endpoint for AI-native editors.
• A cron scheduler for unattended work.
• A pluggable terminal backend layer: local, Docker, SSH, Singularity, Modal, Daytona, Vercel Sandbox — including serverless backends that hibernate when idle, so a 24/7 agent can cost essentially nothing.

It supports 200+ models through Nous Portal, OpenRouter, OpenAI, Anthropic, NVIDIA NIM, Hugging Face, NovitaAI, z.ai/GLM, Kimi, MiniMax, xAI Grok, and any OpenAI-compatible endpoint. Switching providers is hermes model — no code change.

✨ What separates it from LangGraph, CrewAI, AutoGen

The popular frameworks treat an agent as a graph or crew you define ahead of time. You design nodes, you wire edges, you ship. The agent’s capability is bounded by what you prompted into it.

Hermes treats an agent as a process that accumulates capability over time. Concretely:

Said differently: LangGraph is a build-time framework. Hermes is a run-time being. The two are not competitors so much as different scales of the same problem — LangGraph is excellent for building a deterministic flow inside an enterprise app; Hermes is excellent when you want an agent that lives somewhere, hears you across channels, and gets better at you over months.

2. 🏗️ Core Architecture

Hermes’ architecture is deceptively simple — almost every “feature” is a thin layer over a single, stable agent loop.

                            ┌─────────────────────────────────┐
                            │         User Surfaces           │
                            │  CLI · TUI · Gateway · Web ·    │
                            │     ACP · Cron · Subagents      │
                            └────────────────┬────────────────┘
                                             │
                            ┌────────────────▼────────────────┐
                            │          Agent Loop             │
                            │  prompt → think → tool → obs →  │
                            │   memory write → continue       │
                            └──┬──────────────┬───────────┬───┘
                               │              │           │
        ┌──────────────────────▼─┐  ┌─────────▼────┐  ┌───▼─────────────────┐
        │     System Prompt      │  │    Tools     │  │   Skills (Markdown) │
        │  (cache-stable header) │  │  70+ builtin │  │ ~/.hermes/skills/   │
        │                        │  │  + MCP + you │  │ self-edited         │
        └────────────────────────┘  └──────┬───────┘  └─────────────────────┘
                                           │
                              ┌────────────▼─────────────┐
                              │  Execution Environment   │
                              │ local · Docker · SSH ·   │
                              │ Modal · Daytona · Vercel │
                              └──────────────────────────┘
                                           │
                              ┌────────────▼─────────────┐
                              │         Memory           │
                              │ Frozen-snapshot · FTS5   │
                              │  SessionDB · Honcho      │
                              └──────────────────────────┘

The pieces worth understanding in depth:

2.1 🔄 The Agent Loop

A textbook think → act → observe loop, but with two non-obvious decisions baked in:

• Cache-friendly prompt layout. The system prompt header is deliberately stable across turns so provider-side prompt caching (especially Anthropic’s) hits 80–95% of the time. This is the single biggest cost lever — on Hermes’ default Claude config, prompt caching alone yields up to ~90% input-token savings on long sessions.
• Skill nudges. The loop periodically prompts itself to reflect on whether the current trajectory should be captured as a reusable skill — that is what gives it the “self-improving” property.

2.2 🧰 Tools

70+ built-in tools across filesystem, shell, browser, search, fetch, code execution, image/audio/video generation, and orchestration (spawnable subagents). Tools are self-registering: drop a Python module into tools/, the registry picks it up. You can also wire any MCP server; tool filters let you allow-list per-session.

2.3 📚 Skills — the killer feature

A skill is a markdown file with optional YAML frontmatter that the agent stores under ~/.hermes/skills/<skill-name>/SKILL.md. The agent invokes them by reference, sometimes nested. Three reasons this is bigger than it looks:

• Procedural memory. The agent doesn’t just remember facts — it remembers how to do things you’ve taught it.
• Progressive disclosure. Skills can have multiple disclosure levels — a one-line description for retrieval, an expanded body when triggered, and deep references loaded on demand. This keeps the context window tight.
• Self-improvement loop. Via the skill_manage tool, the agent can edit, fork, or retire its own skills based on what worked. v0.10.0 ships 118 bundled skills; the community Skills Hub (agentskills.io) tracks thousands more.

2.4 🗂️ Memory

Three independent mechanisms, intentionally layered:

1. Frozen-snapshot persistent memory — a stable, append-only log inserted into the cache-friendly portion of the prompt.
2. SessionDB — FTS5-indexed full-text store of every past session; recall is “search + LLM summarize the hits”.
3. Pluggable provider — Honcho (dialectic user-model framework), mem0, or supermemory if you want fancier semantics.

2.5 🌐 Surfaces

Hermes treats “how the user reaches the agent” as a separate concern from the loop:

• TUI — the most polished terminal UI in the open-source agent space, with streaming, slash-command autocomplete, and multimodal output.
• Gateway — bridges 20+ messaging platforms. This is what makes Hermes feel like a person you message rather than a tool you launch.
• Cron — ~/.hermes/cron/ schedules unattended runs.
• Subagents — spawnable, isolated peers for parallel workstreams (e.g., one searches, one drafts, one critiques).

2.6 🧪 RL & self-evolution

The companion project hermes-agent-self-evolution (ICLR 2026 Oral) uses DSPy + GEPA to optimize Hermes’ skills, prompts, and even agent code against benchmarks. This is the research substrate behind “the agent improves itself” — and it is open.

3. ⚔️ Hermes vs OpenClaw vs GoClaw

These three projects rhyme, but they target different builders. Quick orientation:

• Hermes — research-grade, Python/TS, self-improving, model-agnostic, ships as “the agent itself.”
• OpenClaw — TypeScript / Node, messaging-first, “your personal assistant on every channel you use,” local-first daemon.
• GoClaw — Go reimplementation of OpenClaw aimed at multi-tenant production: row-level isolation, 5-layer security, single ~25 MB binary, PostgreSQL + pgvector. CC BY-NC license.

3.1 📊 Feature matrix

3.2 🎯 When to pick which

Pick Hermes if:

• You want the strongest learning loop in the open-source space — skills, memory, self-improvement are the headline.
• You want a single agent that grows with you over months and years.
• You want to swap models freely (200+ supported) or run on serverless backends with near-zero idle cost.
• You’re building on top of an agent platform and want active research velocity (Nous Research is shipping fast, ICLR-grade work).
• You’re comfortable with Python.

Pick OpenClaw if:

• Your dominant requirement is “I want the assistant to live where I already chat” — every messenger, every device.
• You want first-class voice and Canvas rendering on Mac / iOS / Android.
• You prefer TypeScript and the npm ecosystem; you want an installable daemon (openclaw onboard --install-daemon).
• The agent’s job is “respond reliably across channels” more than “plan autonomously over hours.”

Pick GoClaw if:

• You’re shipping a product or SaaS that runs many agents for many users — multi-tenancy, row-level isolation, encrypted per-user API keys, and audit-friendly security matter.
• You want enterprise operational characteristics: 25 MB single binary, sub-second startup, native concurrency, OTLP tracing, PostgreSQL durability.
• You’re a Go shop, or you want a runtime your platform/ops team can love.
• ⚠️ Note the CC BY-NC 4.0 license — commercial use requires a separate arrangement. If your business is for-profit SaaS, do due diligence before committing.

Pick more than one:

• Hermes + OpenClaw is a credible pairing: Hermes as the brain (learning, skills, planning) routed into OpenClaw’s channel/device surfaces.
• Hermes for personal + GoClaw for product is a common split — your team learns one stack twice, once as the user, once as the operator.

4. 🌍 Real-World Use Cases

4.1 🏢 Common production use cases

• Always-on engineering operator. Wired to GitHub + Slack + your CI: triages issues, summarizes PRs, runs flaky-test bisection, files draft fixes, reports back in-channel.
• Customer-facing support copilot. Behind a WhatsApp or Telegram gateway, handling Tier-1 support with sandboxed tool access to your knowledge base + ticket system.
• Internal ops bot. Cron-driven: every morning pulls metrics dashboards, summarizes anomalies, drops a note in the team channel; runs ad-hoc investigations on demand.
• Research assistant. Long-running, scopes literature reviews, maintains a personal knowledge base of summaries, and notices when new papers contradict prior ones.
• Sales/CRM concierge. Watches inbound channels, drafts replies in your voice, schedules follow-ups via cron, hands hot leads to humans with a packaged brief.
• Devrel / community manager. Across Discord + Twitter/X + GitHub, drafts responses, escalates real issues, maintains FAQ skills that improve every week.

4.2 👤 Personal / “agent for one” use cases

• A second brain that talks back. Journals, recalls past projects via FTS5 SessionDB, surfaces patterns (“you’ve burned out the last three Aprils — want to lighten this week?”).
• Calendar / inbox triage. Connect Email + Telegram. The agent ingests, classifies, drafts replies, never sends without approval until you trust it.
• Personal trainer / coach. Skills like weekly-review, progressive-overload-plan, recovery-check accumulate over months — literally a coach that learns you.
• Home automation brain. Webhook / MCP into Home Assistant. Natural-language schedules, anomaly alerts (“there’s been a leak sensor spike, do you want me to close the main valve?”).
• Travel concierge. Pulls fare data, drafts itineraries, books via tool calls behind your confirmation, files receipts to a notes app.
• Writing / creative partner. A long-running collaborator that remembers your style and last 80,000 words of context; skills can encode editing rules (“never use the word ‘leverage'”).
• Tax / finance helper. Skills capture your accounting policies; one cron runs monthly reconciliations against bank exports; nothing leaves your machine.
• Family group assistant. Sit Hermes (or OpenClaw) in a family Signal group: shared lists, reminders, photo organization, vacation planning.

5. 🔌 Integration Patterns for Existing Systems / SaaS

Hermes is intentionally open at every seam. Five integration shapes you’ll likely use:

5.1 📥 Inbound integrations — letting the agent reach into your systems

• MCP servers (recommended default). Wrap your internal APIs as MCP tools — your stack stays untouched and any agent (Hermes, Claude Desktop, Cursor, etc.) can consume it. Hermes filters MCP tools per session.
• Custom Hermes tools (Python). Drop a module into tools/, declare a schema, the registry picks it up. Use this when you want first-class tool ergonomics, streaming, or tool-side caching.
• Webhooks via the cron / event bus. Schedule pulls (every 10 min, fetch open tickets) or expose webhook endpoints that drop an event onto the agent’s queue.
• The Gateway as inbox. Treat Telegram/Slack/Email as the input plane — your existing messaging surface becomes the agent’s UI without you building one.

5.2 📤 Outbound — embedding the agent into your product

• ACP (Agent Client Protocol). Hermes speaks ACP, so AI-native editors (Cursor-style) and any ACP client can drive it. This is the cleanest way to embed an agent into a desktop or editor product.
• Web UI iframe / API. hermes web exposes a usable UI; for deeper integration, wrap the agent process and proxy I/O.
• Subagents as microservices. Spawn a subagent per request from your backend; let it run isolated in a Daytona/Modal sandbox; collect the trajectory.
• Trajectory export → fine-tuning. Hermes ships batch trajectory generation; you can use real production runs to fine-tune cheaper local models for your domain.

5.3 🧱 Architecture sketch for a SaaS

A pragmatic three-tier embedding:

[Your SaaS]
   │
   ├── /api/* (your existing app)
   │
   └── /agent/* ── proxy ──► [Hermes process]
                              │
                              ├── MCP ──► your internal API (Stripe, Postgres, S3, etc.)
                              ├── Sandbox: Modal / Daytona (per-tenant)
                              └── Memory: Postgres + pgvector (per-tenant namespace)

For multi-tenant scenarios specifically (one agent per customer), this is where GoClaw earns its keep: it gives you tenant isolation, encrypted per-user keys, and row-level DB security out of the box, so you don’t have to build them.

5.4 ⚠️ Common gotchas

• Cache invalidation. Anything that mutates the cache-stable prompt header (timestamps, dynamic counters) tanks prompt-cache hit rate. Keep volatile content below the cache boundary.
• Skill explosion. Without grooming, an agent will accumulate 500 mediocre skills. Periodic skill_manage review (or a cron that runs it) is worth its weight.
• Tool approval UX. In a user-facing product, “agent wants to run X” prompts need real product thought — don’t paper over with auto-approve.
• Cost. Skills + memory + long sessions = many tokens. Lean hard on prompt caching, and consider mixing a small local model for routine turns.

6. 🛠️ Setup / Run / Customize / Extend

6.1 🚀 Install (Linux / macOS / WSL2 / Termux)

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
source ~/.bashrc
hermes

Windows native: a PowerShell one-liner installs uv, Python 3.11, Node, ripgrep, ffmpeg, and a bundled MinGit.

For contributors:

git clone https://github.com/NousResearch/hermes-agent.git
cd hermes-agent
./setup-hermes.sh

6.2 ⌨️ Day-1 commands

6.3 ✍️ Writing a skill

Skills are just markdown. The smallest useful one:

---
name: weekly-review
description: Run a Friday weekly review with the user
triggers: ["weekly review", "friday review"]
---

When triggered:
1. Pull the last 7 days of journal entries from SessionDB.
2. Group by theme; surface 3 wins, 3 frictions, 1 pattern.
3. Ask the user one sharp question, then propose next week's top 3.

Drop it into ~/.hermes/skills/weekly-review/SKILL.md. The agent will discover it via progressive disclosure (description first; full body when relevant). To share, publish to the Skills Hub.

6.4 🔧 Writing a custom tool

A tool is a Python module that the self-registering registry picks up. Pattern:

# tools/jira_search.py
from hermes.tools import tool

@tool(name="jira_search", description="Search Jira issues by JQL.")
def jira_search(jql: str, limit: int = 20) -> list[dict]:
    """JQL → list of issues."""
    return jira_client.search(jql=jql, limit=limit)

Reload tools (hermes tools) and the agent can call it. For shared/installable tools, prefer MCP.

6.5 🎭 Customizing personality & context

• Personalities: ~/.hermes/personalities/<name>.md — slot in via /personality <name>.
• Context files: project-level markdown that becomes part of every conversation in that project (think CLAUDE.md, but Hermes-native).
• Cron: ~/.hermes/cron/ — drop YAML/markdown schedules; the daemon runs the agent unattended.

6.6 🧩 Extending the runtime itself

• Memory provider. Swap to Honcho, mem0, or supermemory via config.
• Execution backend. Switch from local → Docker → Modal/Daytona with a config change; no code rewrite.
• Surface. Add an ACP client, expose /v1/agent over your own HTTP layer, or write a new gateway adapter (the gateway is a clean adapter pattern).
• Plugins. The plugin system + COMMAND_REGISTRY pattern lets you add slash commands and entirely new subsystems without forking core.

6.7 ✅ Production checklist

• Pin a specific Hermes version; don’t ride main in production.
• Run in Docker (or Modal/Daytona) — never local backend for shared agents.
• Set explicit tool allow-lists per session/profile.
• Turn on prompt caching at the provider level; verify cache hit rate > 80%.
• Cron a skill-grooming run weekly.
• Log trajectories (cheap) — they become training data and audit trail.
• Wrap external API tools with rate limits & circuit breakers; agents will hammer broken endpoints harder than humans.

7. 💭 Opinion — What an Open, Capable Agent System Means for AI Development

Three years ago, “agent framework” meant “fancy retry loop around a chat completion.” Hermes — and the OpenClaw/GoClaw lineage — represent something genuinely different, and it’s worth naming:

1. The unit of software is shifting from “app” to “agent.”
An app is a UI + business logic + persistence. An agent is a process + tools + memory + a way to be reached. Hermes treats every surface (CLI, messaging, web, ACP, cron) as interchangeable adapters to the same underlying being. Once you internalize that, building “an app” and “an agent that does the app’s job” stop being separate disciplines — and the agent wins almost every time, because it composes with everything else the user has.

2. Self-improvement, when it’s just markdown, is real.
The deepest insight in Hermes’ design is unglamorous: skills are markdown files the agent writes. No vector store gymnastics, no opaque fine-tunes — just a folder of text files that the loop edits. That’s enough for a closed learning loop, because LLMs are extraordinarily good at reading and writing their own instructions. The implication is that a long-lived open agent will, in practice, become as capable as proprietary ones — not by matching their base model, but by accumulating thousands of small procedural wins their stateless competitors can’t.

3. Openness changes the economics.
With serverless backends like Modal/Daytona that idle at near-zero, plus 200+ provider support, plus an MIT license — the marginal cost of running a personal Hermes is approaching nothing. We are roughly one user-experience cycle away from the world where running your own agent is more natural than using a hosted one, the same way self-hosting a wiki briefly was, before it wasn’t, and then was again with Obsidian. The companies that bet exclusively on hosted agent moats are going to have to find a different moat.

4. The interesting frontier moves from models to artifacts.
The model is becoming a commodity input. What differentiates one user’s agent from another is the artifact graph that accumulates around it — their skills, their memories, their personalities, their tool wiring, their channel presence. That graph is portable, exportable, forkable, gift-able. It is the part that’s yours. Hermes is the first major framework to take that seriously by design.

5. The risks compound the same way.
A self-editing agent with tool access is exactly as much of a security problem as it sounds. The trio of agent runs tools + agent edits its own instructions + agent persists across sessions is genuinely new threat surface. GoClaw’s 5-layer model — rate limits, prompt-injection detection, SSRF guards, AES-256-GCM, RBAC, row-level DB isolation — is the floor, not the ceiling, for anyone running this for other people. Expect “agent security” to become a discipline with its own conferences within 18 months.

6. The community wins.
The agentskills.io standard is the part of this story I’d watch closest. A portable, vendor-neutral skill format means a skill someone wrote for Hermes can run inside OpenClaw, can run inside your in-house framework, can be inspected and forked. Compare to the alternative — every vendor’s “GPTs / Agents / Assistants” being a walled garden. The open-skill bet is the same bet HTTP made against AOL: more chaotic in the short run, structurally inevitable in the long.

The bottom line. Hermes is not “the best agent framework” the way React is “the best UI framework.” It’s the first credible attempt at a living agent — a piece of software that runs continuously, reaches you where you already are, edits itself, and gets noticeably better at you over time. That’s a different product category, and the next five years of personal/professional AI use are going to be defined by whoever masters it. If you build software for a living, spend a weekend with Hermes — not because you’ll necessarily adopt it, but because the shape of what you’re building is changing, and this is one of the clearest views of the new shape that exists today.

📎 Sources

• GitHub — NousResearch/hermes-agent
• Hermes Agent Documentation
• Hermes Agent — The Agent That Grows With You
• Hermes Unlocks Self-Improving AI Agents, Powered by NVIDIA RTX PCs and DGX Spark — NVIDIA Blog
• Hermes Agent Review — TokenMix Blog
• Hermes Agent from Nous Research — i-scoop
• Hermes Agent Review — Medium / kisztof
• hermes-agent-docs (mudrii)
• Nous Research Hermes Agent Integrates xAI Grok — Technetbook
• GitHub — openclaw/openclaw
• The Ultimate Guide to OpenClaw AI Assistant — Skywork
• awesome-openclaw-skills (VoltAgent)
• explain-openclaw (centminmod)
• GitHub — nextlevelbuilder/goclaw
• GoClaw — Enterprise AI Agent Platform
• GoClaw Deep Dive — Viblo
• 10 Best OpenClaw Alternatives in 2026 — Vellum
• Personal AI Agents in 2026: The Complete Landscape — Ry Walker
• agentskills.io

If you found this helpful, let me know by leaving a 👍 or a comment!, or if you think this post could help someone, feel free to share it! Thank you very much! 😃

10 Things You Can Do With Logs Using Garudust Agent 🦅

Most developers treat logs the same way: something breaks, you grep for errors, squint at timestamps, and piece together what happened. That’s table stakes.

Garudust Agent is a self-hostable AI agent runtime written in Rust. Pair it with the log-analyst skill from garudust-hub, and your logs become something you can talk to — and that can act on your behalf.

Here are 10 things you can do that go well beyond grepping for ERROR.

Setup in 60 Seconds

# Install
wget -qO- https://github.com/garudust-org/garudust-agent/releases/latest/download/garudust-$(uname -m)-unknown-linux-musl.tar.gz | tar -xz
sudo mv garudust garudust-server /usr/local/bin/

# Configure (picks your LLM provider)
garudust setup

# Install the log skill
garudust skill install log-analyst

Local GPU? Skip the API entirely:

export VLLM_BASE_URL=http://localhost:8000/v1
export GARUDUST_MODEL=Qwen/Qwen3-8B-AWQ

1. 🕵️ Explain Why an Incident Happened

grep finds what. Garudust explains why.

garudust "read /var/log/app.log around 2025-05-14 03:22:00 ±10 minutes.
Reconstruct the chain of events that led to the service crash.
What was the root cause?"

The agent reads the surrounding context — not just the error line — and builds a timeline:

Timeline reconstruction:
  03:14:22 — Memory usage crossed 80% threshold (first warning)
  03:19:05 — DB connection pool exhausted (upstream pressure)
  03:21:58 — Request queue backed up, workers stalled
  03:22:11 — OOM killer terminated process (root cause)

Root cause: sustained traffic spike + connection pool too small for load
Recommendation: increase pool size from 10 → 25, add circuit breaker

2. 📊 Generate a Weekly Incident Report

Stop writing incident summaries by hand.

garudust "read /var/log/app.log for the past 7 days.
Generate a weekly incident report with:
- Total error count by day
- Top 5 recurring error types
- Mean time between incidents
- Any worsening trends
Format as Markdown."

Pipe it to a file, commit it, send it to Slack — whatever fits your workflow:

garudust "weekly incident report from /var/log/app.log" > reports/week-$(date +%V).md

3. 🔁 Detect Crash Loops Before They Become Outages

A service that restarts every 4 minutes isn’t down — your uptime monitor won’t catch it. But Garudust will.

garudust "scan /var/log/syslog for the last 2 hours.
Detect any process that has started and stopped more than 3 times.
Flag it as a crash loop candidate with restart intervals."

⚠️  CRASH LOOP DETECTED: api-worker
    Restarts in last 2h: 7
    Average interval: 16 minutes
    Last restart: 10 minutes ago
    Pattern: always crashes ~2 min after start (initialization failure likely)

4. 🌡️ Baseline “Normal” and Alert on Deviations

What if you don’t know what an anomaly looks like — only that something feels off?

garudust "compare /var/log/nginx/access.log from last Monday vs this Monday,
same time window 09:00–12:00.
What's statistically different? Flag anything that deviates by more than 2x."

Garudust doesn’t need pre-defined rules. It compares distributions and reasons about what changed:

Deviations vs. last Monday baseline:
  • 4xx error rate: 0.3% → 4.1%  (13.7x increase) ⚠️
  • Avg response time: 120ms → 380ms  (3.2x increase) ⚠️
  • /api/search endpoint: 2% → 31% of traffic  (unusual spike)

Hypothesis: new client or bot hitting /api/search aggressively

5. 🔒 Audit Security Events

Spot brute force attempts, suspicious IPs, and unusual access patterns.

garudust "analyze /var/log/auth.log from the last 24 hours.
Find:
- Failed login attempts > 5 from the same IP
- Successful logins from IPs with prior failures
- Any logins at unusual hours (outside 08:00–20:00 local time)
- New user accounts created
Summarize as a security audit."

🔴 Brute Force Attempt:
   IP: 185.234.xx.xx — 847 failed SSH attempts in 3 hours
   ⚠️  This IP succeeded once at 03:41 — INVESTIGATE IMMEDIATELY

🟡 Off-hours Logins:
   user: deploy  — login at 02:14 from 10.0.1.55 (internal, but unusual)

6. 📈 Build a Performance Degradation Timeline

Know exactly when things started slowing down — not just that they’re slow now.

garudust "read /var/log/app.log for the past 30 days.
Plot response time trends by week.
Identify the exact date when p95 latency started increasing.
What changed around that time in the logs?"

This is invaluable for debugging regressions that crept in silently over weeks.

7. 🤖 Auto-Remediate Known Issues

Garudust has a terminal tool. Combine log analysis with action.

garudust "check /var/log/app.log — if you see more than 20 'disk quota exceeded'
errors in the last hour, run: find /tmp -mtime +1 -delete
then report how many files were removed and current disk usage."

Or for a service restart:

garudust "monitor /var/log/api.log — if the error rate exceeds 15% over 5 minutes,
run: sudo systemctl restart api-service
and report what you did and why."

Note: Garudust’s GARUDUST_APPROVAL_MODE defaults to smart — it will ask before running destructive commands. Set to auto only in controlled environments.

8. 🔍 Trace a Single Request Across Services

Distributed systems scatter a single user request across multiple log files. Garudust can stitch them back together.

garudust "find all log entries related to request_id=a3f9c2b1 across these files:
/var/log/gateway.log
/var/log/auth-service.log
/var/log/api.log
/var/log/db-proxy.log
Build a complete trace timeline with latency at each hop."

Request trace: a3f9c2b1
  00ms  gateway.log      — request received, routed to auth
  12ms  auth-service.log — token validated
  13ms  api.log          — handler invoked
  891ms api.log          — ⚠️ waiting on DB (unusually long)
  903ms db-proxy.log     — query executed (full table scan detected)
  905ms gateway.log      — response returned  total: 905ms

9. 📝 Generate a CHANGELOG from Deploy Logs

If your deploy pipeline writes structured logs, you can generate changelogs automatically.

garudust "read /var/log/deploy.log for this month.
Extract all deployments: service name, version, timestamp, who deployed.
Format as a CHANGELOG grouped by week, in Keep a Changelog style."

## [Week 20] — 2025-05-12 to 2025-05-18

### Deployed
- **api-service** v2.4.1 — 2025-05-14 10:32 (alice)
- **worker** v1.9.0 — 2025-05-16 14:15 (bob)
- **gateway** v3.1.2 — 2025-05-17 09:00 (alice)

10. ⏰ Schedule All of the Above

Everything above can run automatically on a cron schedule — no extra infrastructure needed.

TELEGRAM_TOKEN=your_token 
GARUDUST_CRON_JOBS="
0 9 * * 1=Read /var/log/app.log last 7 days, generate weekly incident report, send to telegram;
*/15 * * * *=Check /var/log/app.log last 15 minutes for crash loops or error spikes, alert telegram if found;
0 8 * * *=Audit /var/log/auth.log last 24 hours for security anomalies, send daily security digest to telegram
" 
garudust-server --anthropic-key sk-ant-...

Three jobs. Zero extra services. No Prometheus, no Grafana, no ELK cluster.

The Mental Model Shift

Traditional log tooling asks you to define rules upfront:

“Alert me when error rate > 5%”

Garudust lets you ask questions in plain language — after something happens, or proactively via cron — and get reasoning, not just matching:

“What went wrong this week, and is it getting worse?”

The difference is the jump from pattern matching to understanding.

What’s Next

• Install the log-analyst skill and try #1 on your own logs right now
• Write a custom SKILL.md for your specific log format in ~/.garudust/skills/
• Contribute your own log tools to garudust-hub

🔗 Links:

• GitHub: garudust-org/garudust-agent
• Hub: garudust-org/garudust-hub
• 🇹🇭 Thai version: (link after publishing)

Which of these 10 use cases would save you the most time? Drop a comment — I read every one. 🙌

Originally published on sendmsg.io. Cross-posted with a canonical-URL tag so Google attributes SEO authority back to the source.

TL;DR

Email anomaly detection is a real-time monitoring layer inside modern sending platforms that flags abnormal patterns — bounce spikes, ISP block events, complaint surges, throttle deferrals — within minutes, not hours. Detection runs at multiple levels (sender, domain, IP, campaign) and triggers automated protective responses: speed throttling, sending pauses, gradual recovery. The result is fewer reputation incidents, less manual firefighting, and predictable deliverability. This article explains how email anomaly detection works, what signals it watches, and why minute-level latency matters more than daily reports.

Email reputation damage happens fast and recovers slowly. A bounce rate spike that runs unchecked for two hours can take weeks to recover from. An ISP block that persists because nobody noticed can spread to other ISPs through shared blocklists. Anomaly detection in email platforms is the practice of automatically identifying unusual sending patterns — and crucially, taking protective action before a human needs to intervene. It is the difference between catching a problem at the ‘unusual trend’ stage versus the ‘reputation emergency’ stage.

What Counts as a Sending Anomaly

Bounce Rate Spikes — A sudden increase in bounce rate — hard or soft — indicates something has changed. It could be list quality degradation, a DNS misconfiguration, or an ISP-side issue. The anomaly system detects when bounce rates exceed historical norms for this sender.

Block Clusters — When multiple ISPs start blocking your email in a short time window, that’s a block cluster. This is more urgent than individual blocks because it suggests a widespread reputation issue rather than a single ISP’s policy.

Complaint Surges — A sudden increase in spam complaints indicates recipients are unhappy with your email. Complaint surges are particularly dangerous because ISPs weight complaints heavily in reputation calculations.

Deferral Patterns — ISPs use deferrals to signal congestion or suspicion. A pattern of increasing deferrals from a specific ISP is an early warning that the ISP is losing trust in your sending.

Engagement Drops — While less urgent than blocks or complaints, a significant drop in open or click rates can indicate deliverability problems — emails may be landing in spam folders even though they’re not bouncing.

Real-time Detection vs Daily Reports

The most common approach to monitoring email sending is daily reports — a summary of yesterday’s metrics delivered each morning. For low-volume senders, this is adequate. For high-volume senders processing thousands or millions of emails per day, daily reports are dangerously slow. A lot of damage can happen in 24 hours. Live detection processes every delivery signal as it arrives. A bounce, a block, a complaint — each event is evaluated against current baselines and historical patterns. When the system detects that metrics are deviating from normal patterns, it can trigger protective measures within minutes.

Cascade Detection

Some anomalies are isolated — a single campaign hitting a bad list segment, or a single ISP having a temporary issue. Others are systemic — problems that affect multiple campaigns, multiple domains, or multiple ISPs simultaneously. Cascade detection identifies when isolated problems are spreading. If one campaign’s bounce rate spikes, that’s a campaign-level issue. If multiple campaigns from the same domain start showing elevated bounces, that’s a domain-level issue. If multiple domains from the same account are affected, that’s an account-level issue requiring account-level response. Each escalation level triggers proportionally stronger protective measures.

Automated Response Patterns

1. Alert and Intensify Monitoring — At the first sign of anomaly, the system intensifies monitoring frequency. Metrics that are normally evaluated periodically are now evaluated continuously. This gives the system maximum visibility to determine if the anomaly is transient or persistent.
2. Reduce Sending Speed — If the anomaly persists or worsens, sending speed is automatically reduced. This limits the rate at which problematic emails are sent, giving ISPs time to process the existing volume and giving the system time to evaluate trends.
3. Isolate the Problem — The system identifies which entity is causing the anomaly — a specific campaign, domain, or sending pattern — and applies controls specifically to that entity. Unaffected sending continues normally.
4. Temporary Protection — If reduction and isolation don’t resolve the anomaly, the affected entity’s sending is temporarily paused. This prevents further damage while preserving the sender’s ability to investigate and fix the root cause.

The Speed of Detection Matters

Consider a scenario where a campaign starts sending to a list with a 15% bounce rate (well above the safe threshold). At 1,000 emails per minute, that’s 150 bounces per minute accumulating against your domain’s reputation. After 30 minutes, you have 4,500 bounces. After 2 hours, you have 18,000 bounces. A system that detects anomalies within minutes catches this at 150-300 bounces — easily recoverable. A daily report catches it at 18,000+ bounces — potentially causing weeks of reputation impact. Detection speed is not a nice-to-have feature; it directly determines how much damage an incident can cause.

Notification and Transparency

Automated detection and response must be paired with clear notification. When the system takes protective action, senders need to know what happened, why, and what they can do about it. Effective notification includes: the specific anomaly detected, the protective action taken, the affected entity, the metrics that triggered the response, and guidance on investigating the root cause. Transparency builds trust — senders are more accepting of automated interventions when they understand the logic behind them.

Core capabilities of an anomaly-detection layer

Early Warning System — Detects sending anomalies — bounce spikes, blocks, complaint surges — within minutes of occurrence rather than hours.

Cascade Detection — Identifies when problems spread from campaign to domain to account level, triggering proportional protective responses.

FAQ

What is email anomaly detection?

Email anomaly detection is the practice of monitoring sending data in real time to identify abnormal patterns — sudden bounce spikes, ISP block events, deferral surges, complaint clusters — that indicate a deliverability problem before it cascades. Modern platforms run this at multiple levels (sender, domain, IP, campaign) so an issue on one slice doesn’t drag the whole sending operation down.

How fast should email anomaly detection respond?

Useful detection happens at minute-level granularity, not hourly or daily. ISPs apply throttling and blocks within seconds of detecting bad sending behavior, so a daily report is too late — by then the damage to sender reputation is already locked in. Production-grade systems respond within 30–60 seconds of an anomalous signal.

What signals does email anomaly detection watch?

The core signals are bounce rate, soft-bounce/deferral rate, ISP block codes, complaint rate, and engagement collapse. Mature systems also watch derived signals like rate-of-change (acceleration of bounces), per-ISP divergence (one ISP blocking but others fine), and per-domain or per-campaign correlation that flags issues before any single absolute threshold trips.

How is anomaly detection different from bounce monitoring?

Bounce monitoring tells you the absolute number — “you had 47 bounces today.” Anomaly detection tells you the meaning — “your bounce rate to Gmail just jumped 8× in 10 minutes, which historically precedes a block.” It’s the difference between recording weather and forecasting it. Anomaly detection requires baseline comparison, multi-level aggregation, and statistical significance, not just counting.

Can email anomaly detection prevent reputation damage automatically?

Yes. The typical pattern is a graduated automated response: warn → throttle → pause → freeze → recover. When the system detects an anomaly above threshold, it automatically slows or stops sending on the affected slice (a campaign, a domain, an IP), waits for signals to normalize, then ramps back up. This prevents the cascade of bad sending leading to blocks leading to reputation drop leading to more bad sending.

Related reading

• For why anomaly detection needs to run at multiple aggregation layers — not just per-IP — see Multi-Level Email Reputation: Why Tracking at Campaign, Domain, and Account Matters.
• For the control-loop side of how detected anomalies translate into automatic sending adjustments, see Adaptive Rate Limiting in Email: How ISP Feedback Controls Sending Speed.

We built this kind of detection layer into sendmsg.io’s Anomaly Cortex — happy to answer questions in the comments.

We run a Next.js 16 site behind nginx on a single VPS. Recently Google Search Console reported a single 500 on one of our locale-prefixed pages. The page was working fine by the time I clicked through. I almost ignored it. I’m glad I didn’t. The trail led to a bug that fires on every deploy, and the fix is short.

Here’s the story and what the fix cost us.

The single 500

Search Console flagged a locale-prefixed product route. The URL returned a clean 200 when I curled it. So either the indexer hit a transient blip, or something in our deploy flow occasionally leaks a 500 to whichever request happens to be in flight at the wrong second.

The nginx access log made it concrete. One 500 for that URL, single timestamp, never before or after:

[06:58:05]  GET /es/products/details  500

Now the matching journalctl -u frontend for the same second:

06:58:04  Error [ChunkLoadError]: Failed to load chunk
          server/chunks/ssr/messages_es_json_[json]_cjs_xxxxxxxx._.js
          from module 83578
   [cause]: Error: Cannot find module
            '/opt/app/frontend/.next/standalone/.next/server/chunks/ssr/...'
06:58:04  Error [ChunkLoadError]: Failed to load chunk
          server/chunks/ssr/[root-of-the-server]__xxxxxxx._.js ...
06:58:04  ⨯ unhandledRejection: ChunkLoadError ...

Hundreds of these in a five-second window, then silence. That five-second window matched the deploy run from earlier that morning to the second. A later deploy left a bigger spread of 500s across other locale-prefixed routes. Same root cause, same five seconds, more URLs simply because more requests landed in the window.

What the rebuild was doing

Our deploy on master push was:

cd /opt/app/frontend
npm ci --prefer-offline
npm run build              # writes .next/standalone/ + .next/static/ + .next/server/
cp -r .next/static .next/standalone/.next/static
cp -r public        .next/standalone/public
systemctl restart frontend

The WorkingDirectory of the systemd unit was .next/standalone/. next build overwrites that directory in place. So during a 3-minute rebuild, the running Node process held a CPU full of in-memory references to chunk filenames (say, server/chunks/ssr/messages_es_json_[json]_cjs_xxxxxxxx._.js) that the new build had just deleted and replaced with a different hash. Then systemctl restart finally killed the old process and started a new one.

Any SSR request that hit the old process during that ~5-second window between “files replaced” and “process restarted” tried to lazy-load a chunk by its old filename. Node went to disk, didn’t find it, threw ChunkLoadError. Next.js doesn’t handle that in the SSR path. It bubbles up as a 500.

In-memory code that pre-loaded its chunks at boot kept working. Anything that touched a route that lazy-loaded (a different locale, an MDX-rendered page, a dynamic import) was a coin flip.

This isn’t a Next.js bug. It’s the cost of in-place rebuild deploys for any Node.js process that uses dynamic imports. We had lots of them: one per locale message bundle, one per MDX route, one per locale-prefixed page.

What we considered

Four options, in increasing order of “actually adequate”:

1. Stop, then build, then start. systemctl stop before npm run build. The running process never sees mismatched chunks because it isn’t running. Cost: nginx returns 502 for 30–60 seconds while the build runs. 502 is “service unavailable, retry later”, which Google treats as transient. Much friendlier than 500. Users still see a maintenance-ish page for a minute.

2. Atomic directory swap. Build into a sibling directory, then mv .next/standalone .next/standalone-old && mv .next/standalone-new .next/standalone && systemctl restart. The running process keeps reading its old (now-renamed) directory until restart. Window shrinks from 30 seconds of 502 to 3–5 seconds of 502. Still some downtime, no 500s.

3. proxy_next_upstream with a backup server. Tell nginx to retry on a backup if the primary returns 500. Requires keeping two upstream instances in sync forever, including during deploys. That sync is exactly the problem we were trying to solve, so this just relocates it.

4. Blue-green at the systemd + nginx layer. Two long-running pools on different ports. Build into the idle one. Health-check it. Atomically swap nginx upstream. Drain. Stop the old. Zero failed requests during deploy.

We chose 4. The first three each shave a different chunk off the failure window; 4 closes it entirely. And it costs almost nothing on a 16 GB box (more on this below).

The pieces

Two systemd instances from one template unit

# /etc/systemd/system/[email protected]
[Unit]
Description=Frontend (Next.js standalone, %i pool)
After=network.target
ConditionPathExists=/opt/app/frontend/pools/%i/server.js

[Service]
Type=simple
User=app
WorkingDirectory=/opt/app/frontend/pools/%i
EnvironmentFile=/etc/frontend-%i.env
Environment=NODE_ENV=production
Environment=HOSTNAME=127.0.0.1
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=5

%i is the instance name. frontend@blue runs from pools/blue/, frontend@green from pools/green/. The per-color env files supply PORT=3000 and PORT=3001 respectively, kept VPS-local because they don’t belong in git.

ConditionPathExists is doing real work. Without it, an empty pool slot (fresh install, partial deploy) would loop on Restart=always. With it, systemd just doesn’t start the unit until the path appears.

Nginx upstream as an include file

# /etc/nginx/conf.d/frontend-upstream.conf
upstream frontend {
    include /etc/nginx/frontend-upstream-active.inc;
}

# /etc/nginx/frontend-upstream-active.inc
server 127.0.0.1:3001;

The deploy script never edits frontend-upstream.conf. It writes a new frontend-upstream-active.inc via temp-file + mv (which is atomic on a single filesystem), then sends nginx -s reload. mv(2) flips the upstream pointer in one instruction; reload graceful-rotates the workers.

One trap: name the include file with an extension that isn’t .conf, or put it outside /etc/nginx/conf.d/. Otherwise the top-level include /etc/nginx/conf.d/*.conf will try to load it as a standalone config and choke on the bare server directive. We used .inc.

Deploy script flow

ACTIVE=$(cat pools/active-color 2>/dev/null || echo blue)
if [ "$ACTIVE" = "blue" ]; then
  IDLE=green; IDLE_PORT=3001; ACTIVE_PORT=3000
else
  IDLE=blue;  IDLE_PORT=3000; ACTIVE_PORT=3001
fi

# Sanity check: abort if the world is inconsistent.
NGINX_PORT=$(grep -oE '127.0.0.1:[0-9]+' "$NGINX_UPSTREAM" | cut -d: -f2)
[ "$NGINX_PORT" = "$ACTIVE_PORT" ] || { echo "FAIL: marker mismatch"; exit 1; }

# Build (only writes inside .next/, not pools/).
rm -rf "pools/$IDLE"
npm run build

# Stage build into idle pool.
mv .next/standalone "pools/$IDLE"

# Bring idle online and prove it works.
systemctl restart "frontend@$IDLE"
for i in $(seq 1 60); do
  curl -sf -o /dev/null --max-time 2 
    -H 'Host: example.com' 
    "http://127.0.0.1:$IDLE_PORT/" && break
  sleep 1
done

# Multi-route smoke (locale-prefixed + MDX + dynamic) before cutover.
for route in / /es/ /products/details /docs/guide /blog; do
  curl -s -L -o /dev/null --max-time 5 -w '%{http_code}' 
    -H 'Host: example.com' "http://127.0.0.1:$IDLE_PORT$route" | grep -qE '^[23]'
done

# Atomic upstream swap + reload.
printf 'server 127.0.0.1:%s;n' "$IDLE_PORT" > "${NGINX_UPSTREAM}.new"
mv "${NGINX_UPSTREAM}.new" "$NGINX_UPSTREAM"
nginx -t && nginx -s reload

# Mark, drain, retire.
echo "$IDLE" > pools/active-color
systemctl enable  "frontend@$IDLE"
systemctl disable "frontend@$ACTIVE"
sleep 30                              # drain in-flight requests on the old pool
systemctl stop "frontend@$ACTIVE"

The order matters more than it looks. Two specifics.

Write the marker file immediately after the nginx reload, before the drain. If the script crashes during the sleep or the systemctl stop, the marker reflects what nginx is doing right now. The next deploy reads truth, not stale state.

Sanity-check before destructive ops. rm -rf pools/$IDLE is fine if $IDLE really is idle. If the marker file lies (say a previous rollback was incomplete), $IDLE could be the pool that’s serving traffic. The pre-flight check compares the marker against nginx’s upstream port and refuses to proceed on a mismatch.

What it costs us

Measured on the live VPS. Your absolute numbers will vary with bundle size and traffic; the ratios won’t:

Against a 16 GB / 150 GB box where Redis already eats 4–5 GB resident, this is rounding error. The build itself is the expensive part of any deploy and it didn’t change.

What it bought us:

• Zero 500s during deploy. The old pool keeps serving its own (unchanged) chunks until it’s gracefully stopped. The new pool starts from a complete on-disk build before nginx ever sends it a request.
• Zero 502s during deploy. No restart window. nginx -s reload is graceful and doesn’t drop in-flight connections.
• Cheap rollback. The previous pool’s directory is retained until the next deploy. To revert, the rollback script starts the old pool, writes the include file back, reloads nginx. No rebuild needed. About 10 seconds end-to-end.
• Honest failure mode. If the build fails, the script aborts before touching nginx; the old pool is still serving. If the new pool fails health-check, the script stops it and exits non-zero; the old pool is still serving. There’s no state in which the deploy can take the site down mid-flight.

Gotchas

next build cleans .next/ at start

The first cut of this had pool directories at .next/standalone-blue/ and .next/standalone-green/. They got wiped on every rebuild. next build does a recursive clean of .next/ before running. If you want anything to survive across builds, keep it outside .next/. We moved pools to pools/<color>/ (sibling of .next/).

Not Next.js-specific. Most build tools assume their output dir is theirs to own. Don’t squat in it.

mv is safe under a running Linux process

While migrating prod to the new layout I had to move pools/blue/ while frontend@blue was actively serving from inside it. Linux inode semantics make this fine: a process holds inode references through its open file descriptors and CWD, not path strings. mv within a single filesystem is just a rename(2); the inodes don’t move. The running pool kept serving without noticing.

Same reason tail -f keeps working when you rotate a log file by renaming it. Useful primitive once you remember it.

Don’t put backup files in sites-enabled/

I made a backup of /etc/nginx/sites-enabled/default next to the original, then nginx -t started warning about “conflicting server name” entries. The top-level include /etc/nginx/sites-enabled/* was loading my .bak as a config. Move backups elsewhere or rename them so the glob misses.

systemd templates aren’t auto-enabled by enable --now

Our CI workflow has a generic loop that auto-enables newly-installed singleton units. Templates ([email protected]) are explicitly skipped because they need an instance name. That’s the right behavior for our case: we want exactly one of blue/green enabled at a time, and the deploy script decides which.

Health-check should match production conditions

A bare curl http://127.0.0.1:$PORT/ will succeed in a lot of cases where production is broken. Add -H 'Host: example.com' if you’re behind a reverse proxy, follow redirects with -L, and probe routes that exercise the middleware / SSR / MDX paths that you care about. We had a Next.js + Cloudflare + nginx interaction bug that only surfaced when the request Host header didn’t match 127.0.0.1. A localhost-only health check wouldn’t have caught it.

When this isn’t worth it

This pattern is small enough to recommend for any single-VPS deploy that uses systemd + a reverse proxy. It scales to multiple boxes the same way. Replace “two systemd instances on one box” with “two server fleets behind a load balancer” and the swap mechanic is identical.

It is not worth it if:

• Your app boots in under a second and has no in-flight state that matters across restarts. A plain restart is simpler and the failure window is too short to care.
• You already use a real orchestrator. Kubernetes, Nomad, ECS: all of them do this for you as a rolling deploy. If you have it, use it.
• You’re on a serverless platform where the runtime owns the deploy lifecycle. Same reason.

For a single-VPS Node.js process behind nginx, though, blue-green is the proportionate fix. Half a day of work, no new dependencies.

What changed for us, concretely

• The five-second ChunkLoadError window is gone by construction. The old pool’s chunks never get touched; the new pool starts from a complete build before nginx ever sends it a request.
• Rollback is a 10-second nginx-upstream rewrite, not a rebuild.
• The next time next build evicts a critical chunk filename (and it will), nobody outside our journal will know.

The hour we spent figuring out the original bug was longer than the hour we spent implementing the fix. If you’re running anything stateful behind nginx and your deploy is git pull && build && restart, look at what your single-500 window looks like.

I’ve been building a construction safety inspection app — GroundCheck — and from day one I made a decision that surprised a few people: no cloud AI. Every hazard detection runs on-device, offline, in under 50 MB.

Here’s why that wasn’t just a cost call — it was an engineering call.

The problem with cloud AI on a construction site

Construction sites are not Silicon Valley offices. Mid-size commercial builds — the beachhead market I’m targeting — often have patchy LTE at best, and active floors can be dead zones. A safety inspector can’t pause a walkthrough because their hazard detection app is waiting on a round-trip to an API.

When I looked at the alternatives:

• Cloud vision APIs — fast to build, $0.001–0.003/image at scale, but useless offline and creates a real liability question around who holds footage of an active construction site
• On-device ML — more upfront work, but deterministic latency, zero connectivity dependency, and no data leaves the device

For safety tooling, determinism matters. If an inspector gets a false negative because the API timed out, that’s not a UX bug — it’s potentially a serious incident.

The model stack: keeping it under 50 MB

The target is YOLOv8s + MobileNetV3, combined under 50 MB. Here’s why each choice:

YOLOv8s — the small variant sits around 22 MB as a CoreML model. Fast enough to run on-device without throttling the camera feed. The ‘s’ variant trades some mAP against the nano (which would be faster but misses smaller objects — a real problem when you’re detecting things like exposed rebar or missing PPE at distance).

MobileNetV3 — classification backbone for finer-grained scene understanding. Once YOLOv8s has found a detection region, MobileNetV3 does the heavy lifting on “is this person wearing a hard hat” vs. “is this person holding a hard hat.” Two-stage pipeline, both on-device.

The total bundle target is <50 MB because that’s the threshold where App Store cellular auto-download kicks in. Installers and safety managers shouldn’t have to think about it.

Offline-first all the way down

The inspection flow is built on Drizzle ORM over expo-sqlite. Everything captures locally first — photos, GPS coordinates, hazard detections, inspection notes. A sync queue handles Supabase replication when connectivity returns.

This means the app works identically with zero bars as it does on a solid connection. Photo uploads are deferred, sync state is visible in the UI, and no part of the core inspection loop has a network dependency.

It’s more upfront architecture work. But for a product that’s supposed to replace $600/month SafetyCulture contracts, “it doesn’t work if you’re underground” isn’t acceptable.

What I’ve learned so far

On-device AI is actually quite accessible now. CoreML tooling has matured significantly. Converting a YOLOv8 model to CoreML is mostly straightforward via coremltools; the sharp edges are around input preprocessing and getting confidence thresholds tuned for your domain (construction hazard detection ≠ COCO defaults).

Offline-first is a discipline, not a feature. It touches every layer — schema design, UI state, sync conflict resolution. You can’t bolt it on after the fact. I scaffolded the sync queue before I wrote a single inspection screen.

The 50 MB budget forces good decisions. Model quantization, INT8 where it matters, careful layer pruning. Constraints produce better models than unlimited compute budgets.

If you’re building for any domain where connectivity isn’t guaranteed — field work, logistics, healthcare, agriculture — the on-device AI stack has never been more viable. The cloud-first assumption deserves to be challenged.

Visual Testing vs Functional Testing: Two Quality Dimensions You Can’t Afford to Ignore

Functional testing verifies that an application behaves according to its specifications — buttons trigger the right actions, forms validate data, APIs return expected responses. Visual testing verifies that an application looks the way it should — elements are positioned correctly, colors are accurate, and layout is intact.

Here’s an uncomfortable truth: the vast majority of development teams invest heavily in functional testing and almost entirely ignore visual testing. They have hundreds of unit tests, dozens of integration tests, respectable code coverage — and yet visual bugs make it to production regularly.

This isn’t a minor oversight. It’s a systemic blind spot. This article explores the fundamental difference between these two types of testing, why they are complementary and not interchangeable, and why ignoring visual testing is a risk you’re probably underestimating.

The Fundamental Distinction: It Works vs It Looks Right

Let’s take a concrete example. You have an “Add to Cart” button on your e-commerce site.

Functional testing verifies: when a user clicks this button, the product is added to the cart, the counter increments, and the API receives the correct request. The test passes. Everything works.

Visual testing verifies: this button is visible, it has the right color, the right size, the right position, the right text, and it isn’t hidden behind another element. If the button is technically present in the DOM but visually invisible (opacity set to 0, positioned off-screen, covered by an overlay), the functional test passes. The visual test fails.

That’s the fundamental distinction. Functional testing verifies behavior. Visual testing verifies appearance. One does not replace the other.

Why Functional Testing Isn’t Enough

If your functional tests pass and your application looks correct, everything is fine. The problem is the “looks correct” part. Who’s verifying that?

CSS is not covered by functional tests

Your unit tests don’t check CSS. Your integration tests don’t either. A change in a CSS file can break the layout of twenty pages without a single test raising a flag. This is the reality of most test suites: they are blind to the visual layer.

Think about it: you have a global CSS file. A developer modifies an overly broad selector. The padding on every .card element drops from 16px to 0px. Visually, it’s a disaster. Functionally, everything passes green.

Dependency updates silently break the visual layer

You update a UI component library. The new version subtly changes how a dropdown renders, the spacing in a form, or the size of an icon. Your functional tests verify that the dropdown opens and closes. They don’t verify that it no longer overlaps the adjacent button.

Responsive design is an invisible minefield

Your application works on mobile — functional tests pass at a 375px viewport. But the hamburger menu covers the main content. The submit button is off-screen. The login form is unreadable. Functionally, everything is there. Visually, it’s unusable.

Browsers render differently

A component that displays perfectly in Chrome can have a broken layout in Safari or Firefox. CSS rendering differences between browsers are well documented but rarely tested — certainly not by functional tests running in a single browser.

Why Visual Testing Doesn’t Replace Functional Testing Either

Let’s be fair. Visual testing has its own limitations.

Visual testing doesn’t verify business logic. A registration form can look perfect — all fields aligned, the right colors, the right layout — but send data to the wrong endpoint. Visual testing won’t catch that.

Visual testing doesn’t verify complex interactions. A multi-step workflow (cart, address, payment, confirmation) has business logic that only functional tests can validate. Visual testing verifies that each step looks the way it should, not that the transition between steps actually works.

Visual testing doesn’t verify data. A dashboard can display completely wrong data while having a flawless layout. Visual testing says “it looks like it should.” Functional testing says “the data is correct.”

This is exactly why the two are complementary. They cover orthogonal dimensions of quality.

The Dangerous Blind Spot: What Nobody Tests

Here are real-world scenarios where the absence of visual testing causes production issues. These aren’t theoretical — they are situations every web team eventually encounters.

The z-index chaos

A developer adds a component with z-index: 9999 to make sure it appears on top of everything. Two months later, another developer does the same with z-index: 99999. Elements overlap unpredictably. Functional tests detect nothing — every element is present in the DOM. Visually, the interface is a battlefield.

The forgotten dark mode

Your team launches a dark mode. The main components are adapted. But a secondary page uses hardcoded colors: black text on a black background. Functionally, the content is there — a getByText() finds it. Visually, the user sees a black screen.

The fallback font

Your custom font fails to load (CDN down, network issue, incompatible browser). The browser uses a fallback font — Arial instead of your carefully chosen Inter. The text is wider, lines break differently, the layout shifts. Functional tests don’t check fonts. Your trusted AI could have warned you, but it was too busy debating the best way to center a div.

The invisible overflow

A component contains text longer than expected. The text overflows its container and overlaps the next element. Or it gets clipped without an ellipsis, making the information unreadable. The functional test checks that the text is rendered. The visual test checks that it’s readable.

The spacing regression

A spacing token is modified in the design system. Every component using it sees its spacing change. The modification was intentional for one component, but it affects fifty others unexpectedly. Functional tests don’t test margins and paddings.

Complementarity in Practice: What to Test and How

Functional testing excels at

• Form validation (validation rules, error messages)
• User flows (sign-up, purchase, onboarding)
• API calls and responses
• Error handling and edge cases
• Authentication and permissions
• Complex business logic

Visual testing excels at

• Design system compliance (colors, typography, spacing)
• Layout and element positioning
• Responsive design (behavior across different viewports)
• Cross-browser rendering (rendering differences between browsers)
• Unintended CSS regressions
• Impact of dependency updates on appearance
• Visual states (hover, focus, disabled, error, loading)

The complementary strategy

A mature testing strategy covers both dimensions:

Layer 1 — Unit tests (functional). Fast, numerous, focused on logic.

Layer 2 — Integration tests (functional). Verify that components interact correctly.

Layer 3 — Visual tests. Capture the appearance of your pages and components. The visual safety net.

Layer 4 — End-to-end tests (functional + visual). Critical scenarios tested from start to finish.

Visual testing isn’t at the top of the pyramid. It’s a parallel dimension that should exist alongside your functional tests — not after them.

Why Most Teams Ignore Visual Testing

If visual testing is so important, why don’t most teams practice it? The reasons are many, and none of them are truly valid.

“Our functional tests cover that”

No. We just demonstrated they don’t. But this is the most common belief. When your code coverage shows 85%, it’s tempting to believe everything is tested. Code coverage only measures code that was executed, not what the user sees.

“Visual testing produces too many false positives”

That was true five years ago. Raw pixel-by-pixel comparison did generate a lot of noise. Modern tools — including Delta-QA — use perceptual comparison algorithms that tolerate micro-rendering differences while detecting significant changes. The technology has caught up with the problem, but the reputation lingers.

“We don’t have the budget for another tool”

Visual testing doesn’t necessarily require additional budget. Playwright is free. BackstopJS is free. Delta-QA offers an accessible entry point. The cost of not doing visual testing — visual bugs in production, manual review time, regressions discovered by users — is often far greater than the cost of the tool.

“We do visual review in pull requests”

Manual visual review depends on human vigilance — and humans are terrible at spotting subtle differences after the fifteenth CSS file in a PR. The reviewer sees the code, not the rendering. Even your favorite AI, despite its talent for creative hallucination, can’t guess what your page looks like from a Git diff.

“It’s too complicated to set up”

That was true when the only option was to manually configure screenshot capture scripts, manage baselines in Git, and build your own comparison system. Today, tools like Delta-QA make visual testing accessible without writing a single line of test code. The complexity excuse no longer holds.

The Real Costs of Not Doing Visual Testing

Visual bugs have a cost, even if it’s less visible than a functional bug.

Impact on perceived quality. A misaligned button, overflowing text, an inconsistent color — these details signal a lack of attention to your users. Perceived quality makes the difference between a user who stays and one who leaves for a competitor.

The cost of late detection. A visual bug discovered in production costs infinitely more than one caught in CI. The detection, reporting, triage, fix, deploy cycle takes days. Automated detection reduces it to minutes.

Erosion of trust. When visual bugs reach production, developers become reluctant to touch CSS, designers complain, and visual debt accumulates.

Manual review time. Without automated visual testing, someone has to visually verify every change — human time spent on a task that a tool does better and faster.

How Delta-QA Combines Both Dimensions

Delta-QA positions itself in the visual dimension — that’s its specialty. But its approach naturally complements your existing functional tests.

No replacement. Delta-QA doesn’t claim to replace your unit tests, your Cypress tests, or your Playwright tests. It covers what they don’t: the actual appearance of your application.

Integration in the same pipeline. Delta-QA runs in your CI, alongside your functional tests. Your functional tests validate behavior. Delta-QA validates appearance. Both dimensions are covered in the same workflow.

Accessible to the whole team. Functional tests are a developer’s domain. Visual testing with Delta-QA is accessible to the entire team — developers, QA, designers. Reviewing visual changes doesn’t require coding skills.

FAQ

Can visual testing detect functional bugs?

Indirectly, yes. If a functional bug has a visual manifestation — an error message appearing when it shouldn’t, a missing element, an incorrect state — visual testing will catch it. But it can’t detect a functional bug with no visual impact (a miscalculated value displayed in the correct format, for example).

Should you start with functional testing or visual testing?

If you have neither, start with functional testing — it covers the most critical risks (bugs that prevent usage). Add visual testing as soon as your functional tests are in place. If you already have functional tests but no visual testing, now is the time to act: you have a significant blind spot.

Is visual testing relevant for backend applications or APIs?

No. Visual testing is specific to user interfaces — web, mobile, desktop. If your application has no visual interface, visual testing isn’t relevant. For APIs, functional tests and contract tests are the right approaches.

How long does it take to add visual testing to an existing project?

With a no-code tool like Delta-QA, a few hours are enough to cover your critical pages. With Playwright, plan on a few days to write the tests, configure baselines, and integrate into your CI. The initial investment is modest compared to the risk coverage gained.

Does visual testing work with mobile applications?

Web visual testing tools (Delta-QA, Percy, Playwright) target web interfaces, including PWAs and responsive layouts. For native mobile applications, specific tools exist. Web visual testing already covers a large portion of cases if your mobile app uses a webview or cross-platform technology.

Does visual testing slow down development?

On the contrary. It accelerates the feedback cycle by catching visual regressions before they reach production. The time “lost” setting up visual testing is recovered the moment the first visual bug is caught automatically instead of being reported by a user two weeks later.

Conclusion

Visual testing and functional testing aren’t competing. They’re complementary, like the structure and appearance of a building. You don’t choose between a solid floor and a straight wall — you need both.

If you have functional tests but no visual testing, you have a blind spot. Your tests tell you everything works, but nobody verifies that everything looks right. That’s a risk you carry with every deployment.

The best time to add visual testing to your testing strategy was yesterday. The second best time is now.

We build Delta-QA, a visual regression testing tool. Always open to feedback from the community!