I Tried to Run Symphony for Real on Rentello. It Broke in Exactly the Right Place.

Linear issue created

A few weeks ago, I published a build diary about porting OpenAI’s Symphony to Claude Code.

That piece was about getting the machine to exist.

This one is about what happened when I tried to trust it.

Not in a toy repo. Not with a fake ticket. Not with a carefully staged demo that only exercised the happy path. I wired the workflow into a real project, Rentello, pointed it at a real Linear board, and tried to make Codex, Claude, and Symphony-family orchestrators coexist on the same execution model.

That is when the most useful bug of the whole project appeared.

It wasn’t a crash inside the orchestration loop.
It wasn’t a retry bug.
It wasn’t even a Linear label bug.

It was a hidden architectural assumption inside one innocent file: WORKFLOW.md.

Phase two of the same story

The first article was the infrastructure story.

I had taken OpenAI’s Symphony architecture, ported the orchestration ideas to work with Claude Code, and documented the engineering journey: configuration, polling, workspaces, retries, MCP tooling, the terminal dashboard, the CLI quirks, the Linear GraphQL edges. It was the “can I build this?” phase.

This sequel starts after that.

Once the port existed, the obvious next question was: can this become a reliable workflow across agent surfaces?

I didn’t just want a Claude-only orchestrator anymore.

I wanted a system where:

  • planning could happen in Codex App or Codex web
  • work could be split into Linear issues that were safe for parallel or sequential execution
  • exec:agent work could be handled by generic agent surfaces
  • exec:symphony work could be handled by Symphony-family orchestrators
  • the same repository could remain deterministic whether the active surface was Codex or Claude

That sounds tidy when you say it fast.

In practice, it meant turning “AI can work on tickets” into an actual contract.

The experiment: make the workflow explicit

The first big change was conceptual, not technical.

I stopped treating the workflow as an informal set of conventions and turned it into a real repository contract:

  • shared execution-owner labels: exec:agent and exec:symphony
  • wave labels like wave:1, wave:2
  • a stable issue template
  • a single persistent ## Workpad comment per issue
  • a shared machine-readable contract plus shared docs/templates

The important detail here is that I did not want separate “Codex rules” and “Claude rules” at the planning layer.

The workflow itself needed to be generic.

The orchestrator choice should affect how the work is run, not how the work is defined.

So I introduced a shared contract and kept runtime-specific prompts only where they belonged: under .codex/ and .claude/, as adapters to the same underlying rules.

That gave me a clean execution model:

  • exec:agent means “owned by the non-Symphony agent surface”
  • exec:symphony means “owned by a Symphony-compatible orchestrator”
  • blockedBy remains authoritative
  • wave labels are scheduling metadata, not permission to ignore dependencies

That was the theory.

Then I decided to test it for real.

The smoke test: two issues, one dependency, no excuses

I created a new Linear project called Agent Workflow Smoke Test.

Then I used the repo’s planning contract to generate the smallest meaningful DAG I could think of:

  • MAS-26 — exec:agent, wave:1, research, docs
  • MAS-27 — exec:symphony, wave:2, research, docs, blockedBy MAS-26

The goal was deliberately low-risk: a docs-only smoke test with a real sequencing edge.

If the contract was sound, I should be able to verify:

  1. planning against the real repo
  2. issue creation in Linear with the right labels and dependency edges
  3. Codex App dispatch respecting exec:agent
  4. Symphony respecting exec:symphony
  5. a single persistent workpad model across the whole system

The first pass went well.

The repo-level checks passed.
The Linear issues were created correctly.
The labels were right.
The blockedBy relation was correct.

Then I moved MAS-26 into Todo and let Codex App handle the bootstrap.

That part worked exactly the way I wanted.

Codex App picked up only the exec:agent issue, moved it to In Progress, and created a single ## Workpad comment. More importantly, the next run reused that same comment and updated it in place instead of spamming the issue with milestone chatter.

That detail matters more than it sounds.

A lot of agent workflows die the death of a thousand comments. If every automation pass creates another “status update,” the issue becomes unreadable. Reusing one persistent workpad is the difference between an automation system that looks operationally credible and one that looks like a bot farm.

Linear issue created

Notes in the issue

At that point, the first conclusion felt obvious:

the workflow contract was working.

That conclusion was wrong.

The first false conclusion

Once the Codex App leg worked, I moved to the Symphony leg.

I launched openai/symphony against the repo’s WORKFLOW.md.

And immediately noticed something odd: it was looking at the real Rentello project instead of the smoke-test one.

The reason was simple in hindsight.

WORKFLOW.md was doing two very different jobs at once:

  1. it was the shared execution contract
  2. it was the runtime entrypoint for a specific orchestrator, with a specific project slug and a specific launcher configuration

That meant my “shared workflow file” wasn’t really shared at all.

It was secretly carrying environment-specific assumptions:

  • which Linear project to poll
  • which states count as active
  • which command to spawn for the agent runtime
  • what approval policy and sandbox mode to use

So before I even got to the deeper integration problem, I had already hit a design smell:

I was treating a runtime wrapper as if it were a universal contract.

Wrong project slug in symphony

I worked around it temporarily with a copied workflow file that targeted the smoke-test project.

That got me to the real failure.

The real failure: not labels, not blockers, not Linear

The next visible symptom was noisy and misleading.

Symphony would pick up the smoke-test issue, then back off with timeouts and failed runs.

At first glance, it looked like the kind of thing that sends you down the wrong rabbit hole:

  • maybe the new exec:* label routing was wrong
  • maybe the issue state transitions were inconsistent
  • maybe the blockedBy logic had a bug

None of that was the root cause.

The actual problem was much deeper and much simpler:

openai/symphony expected a Codex-compatible app-server runtime, while the repo’s WORKFLOW.md was still configured for Claude-style execution.

In other words, the orchestrator and the launcher contract disagreed about what “the agent” even was.

The workflow file still contained a Claude-oriented runtime block. openai/symphony expected something more like:

  • a Codex app-server command
  • a compatible approval policy
  • the sandbox settings the OpenAI implementation expects

So the visible failure was timeout and retry noise.
The real failure was a runtime mismatch.

That distinction matters.

If I had blamed the new workflow model, I would have “fixed” the wrong layer.

The issue was not exec:agent vs exec:symphony.
The issue was not the shared issue template.
The issue was not the workpad model.

The issue was that one literal WORKFLOW.md cannot honestly be the runtime entrypoint for both openai/symphony and a Claude-oriented Symphony port.

The fix that changed the architecture

This was the moment the architecture got better.

The correct lesson was not “make WORKFLOW.md more clever.”

It was: stop asking one file to represent two incompatible runtime contracts.

So I split the model into three layers:

  1. a shared execution contract
  2. a shared Symphony instruction body
  3. runtime-specific workflow wrappers

Concretely, the repo now has:

  • a shared machine-readable contract for labels, states, and workpad expectations
  • shared docs/templates for planning and issue execution
  • a shared Symphony body template
  • WORKFLOW.openai.md
  • WORKFLOW.claude.md
  • a small render script to generate the wrappers from the shared body

That was the missing separation all along.

The shared workflow body defines how the issue should be executed.

The runtime wrapper defines how the orchestrator should launch the agent.

Those are not the same concern.

Trying to force them into one file had worked only because I was not yet testing both runtimes seriously enough.

Architecture diagram

Once I made that split, the system became easier to reason about immediately.

The repository now says, in effect:

  • the execution rules are shared
  • the runtime launcher is not

That is a much healthier contract.

The second failure: the GraphQL ghost was still waiting

Of course, fixing the runtime mismatch didn’t magically make the full run clean.

After the split, Symphony could start correctly.
It could pick up MAS-27.
It could begin doing real work.

And then it hit another boundary failure: stale assumptions in the Linear GraphQL layer.

The logs told the story:

  • Field "identifier" is not defined by type "IssueFilter".
  • Cannot query field "blockedByIssues" on type "Issue".

This was a different class of bug entirely.

Now the runtime was correct, but the orchestrator’s GraphQL expectations were out of sync with the actual Linear schema. The result was a partial execution: the issue moved into In Progress, but the run did not complete cleanly, and no workpad comment was created on MAS-27.

That was important to verify because from the outside, a half-working orchestration system can look deceptively healthy.

The dashboard shows activity.
The issue state changes.
The agent session starts.

But the operational contract is still broken if the workpad never appears and the run dies on a schema edge.

This is a schema/client mismatch, not a workflow-contract bug.

What actually worked

This is the part I care about most, because I don’t want the story to sound more broken than it really was.

A lot did work:

  • the shared execution-owner model with exec:agent and exec:symphony
  • the planning contract based on shared docs/templates
  • the Linear issue generation for MAS-26 and MAS-27
  • blocker gating via blockedBy
  • Codex App automation dispatching only exec:agent
  • in-place reuse of a single ## Workpad comment on Linear
  • Symphony respecting the exec:symphony lane
  • the runtime split into WORKFLOW.openai.md and WORKFLOW.claude.md

That is not a small list.

In fact, the test did exactly what a good smoke test should do:

it validated the architectural core, then exposed the next real boundary failure.

First the runtime wrapper problem.
Then the stale GraphQL problem.

That is progress.

The actual lesson

If you want cross-agent orchestration, the wrong abstraction is “one workflow file for everything.”

What you actually need is:

  • one shared execution contract
  • one shared behavioral prompt body, if the execution model is the same
  • separate runtime launch wrappers for each orchestrator/runtime pair

The reason is simple.

Codex and Claude are not just different models. In this kind of system, they are different runtime surfaces with different launcher assumptions, different session protocols, and different orchestration expectations.

The workflow itself should stay stable across those surfaces.

The launcher should not.

That is the architectural correction this test forced me to make.

And in hindsight, it’s exactly the kind of correction you only get from trying to run the system for real.

The happy path rarely teaches you where your abstractions are lying.
Live orchestration does.

Same workflow rules, different launchers.

Where this goes next

The runtime split is in.
The generic exec:agent / exec:symphony workflow is in.
The Codex App leg is validated.

The next cleanup is clear:

  • patch the stale Linear GraphQL assumptions in the Symphony runtime
  • rerun the MAS-27 path cleanly
  • verify the full end-to-end chain with both executor lanes

That will be the point where the system stops being “an interesting orchestration prototype” and starts becoming something I would trust against a real backlog.

And honestly, that is the part I find most interesting now.

Porting the orchestrator was fun.

Finding the seam between shared execution contracts and runtime-specific launch contracts was the part that actually made it usable.

I got tired of juggling terminal windows for every AI agent, so I built an all-in-one IDE

https://dqy38fnwh4fqs.cloudfront.net/UHNN7ND8LGD6NEPIJJND6NGQ7LKO/blog/blog-content-image-e9d7dbb5-1e3f-49d5-a3cf-08caf79aa6f3.webp

If you use AI coding agents like Claude Code, Codex CLI, Gemini CLI, or Amp, you probably have a mess of terminal windows scattered across your screen.

Switching between projects means losing context, and there’s no easy way to see what all your agents are doing at once. I kept closing everything at the end of a session and spending the first 10 minutes of the next one just getting back to where I was.

So I built something to fix that.

What is 1DevTool?

1DevTool is an Electron app that gives you a single persistent workspace per project — multiple real terminals, a file explorer, code editor, HTTP client, database client, and an embedded browser, all in one window.

The core idea: your AI agents shouldn’t live in isolated terminal tabs. They should coexist in a layout you control, with full context preserved across sessions.

https://dqy38fnwh4fqs.cloudfront.net/UHNN7ND8LGD6NEPIJJND6NGQ7LKO/blog/blog-content-image-e9d7dbb5-1e3f-49d5-a3cf-08caf79aa6f3.webp

https://dqy38fnwh4fqs.cloudfront.net/UHNN7ND8LGD6NEPIJJND6NGQ7LKO/blog/blog-content-image-90a8e372-5d0d-4f32-a2a3-580d2b8306eb.webp

Key features

🔲 Multi-agent layout
Run multiple AI agents simultaneously in a 2×2 grid, columns, or single-focus layout. See everything at once.

💾 Terminal persistence
Sessions survive app restarts via tmux. Your scrollback is always there — no more re-running setup scripts every morning.

🔔 AI Activity Feed
Live notifications when any agent finishes a task or generates files, across all your active projects.

✂️ Send code to AI
Select code in the editor, right-click, and send it to any running agent — with file path and line numbers included automatically.

🔄 Resume AI sessions
Browse and continue past conversations from Claude Code, Codex, Gemini CLI, and Amp directly from the sidebar.

🧠 Skills Manager
Browse, install, and manage AI agent skills (CLAUDE.md, agent rules, etc.) with built-in security scanning.

🎨 VS Code themes
Import any .vsix theme file and it applies across the entire workspace — editor, terminal, panels.

🗄️ Database client
Connect to Postgres, MySQL, MongoDB, Redis, ClickHouse, and 8 more engines without leaving the window.

📂 Drag & drop projects
Drop a folder from Finder (or Explorer) onto the sidebar to add it instantly.

🧩 Customizable layouts
Drag panels between sidebars, save layout presets, and show/hide header items to match your workflow.

Availability & pricing

Available on macOS, Windows, and Linux.

Free tier: 1 project, 4 terminals — enough to try it with a real workflow.

🔗 1devtool.com

Happy to answer any questions or hear feedback in the comments. If you’ve run into the same terminal chaos with AI agents, I’d love to know how you’re currently handling it.

How to Deploy Next.js to Flux Cloud: Decentralized, High-Uptime, and Budget-Friendly

If you are a Next.js developer, you are probably used to the “Vercel way” of doing things. It is great, but as projects grow, the costs can scale quickly, and you are tied into a single provider’s ecosystem.

Flux Cloud recently released their new Deploy with Git feature. It is a decentralized alternative that offers a “push to deploy” experience but runs your app across a global network of independent nodes.

Here is why it is a solid choice for Next.js projects and how to get set up.

Why Next.js on Flux Cloud?

  • Drastically Lower Costs: Because Flux is a decentralized network, you aren’t paying the “convenience tax” of major corporate clouds. It is often a fraction of the price for the same CPU and RAM.
  • Automatic Global Redundancy: Your Next.js app isn’t just on one server. It is deployed across multiple nodes globally. If one node goes offline, the network keeps your site reachable.
  • Built-in Failover: If a deployment fails, Flux automatically rolls back to the last working version.
  • Full SSR Support: Unlike static-only hosts, Flux handles full-stack Next.js features, including Server-Side Rendering (SSR) and API routes, since it manages the runtime environment for you.

Step-by-Step Deployment

1. Prep your Repository

Flux makes it easy by detecting your framework. Ensure your package.json has the standard Next.js scripts:

  • "build": "next build"
  • "start": "next start"

2. Configure the Deployment

You can use the official Flux boilerplate to ensure your environment variables and ports are mapped correctly.

View the Next.js Deployment Guide here:
https://github.com/RunOnFlux/deploy-with-git/tree/master/deploy-nextjs

3. Connect to Flux Cloud

  1. Go to the Flux Cloud Dashboard.
  2. Select Deploy with Git.
  3. Enter your GitHub/GitLab repo URL.
  4. Flux will build your image and distribute it across the network.

4. Push to Update

Just like you are used to, any push to your main branch triggers a fresh build. Your global URL will update automatically once the new version is ready.

Useful Links

Let me know if you try this out!

Before You Rely on AI for Coding, Fix Your Docs First

AI tools are everywhere now ChatGPT, Claude, Gemini, Copilot, Cursor, and Zed.

They can write code, refactor files, and explain complex snippets in seconds.

But there’s a hidden problem we are all starting to notice:

If your project has weak or no documentation, AI will add more confusion, not more clarity.

This post is for every developer who uses AI daily and wants their codebase to stay understandable and maintainable.

What AI Is Actually Good At

AI is incredibly good at:

  • Writing boilerplate and repetitive code.
  • Translating code between languages or frameworks.
  • Cleaning up or refactoring isolated pieces of code.
  • Explaining a single function or file.

If you give it a small, tightly-scoped problem, it usually excels.

Where AI Fails (Without Docs)

AI struggles when:

  • Only a few people know “how things actually work” in the system.
  • There is no single source of truth for APIs or data models.
  • System behavior differs from what the function names suggest.
  • Business rules live exclusively in Slack threads, Zoom calls, or people’s memories.

In these cases, AI doesn’t know your real context. It guesses.

Sometimes the guesses look smart, but they are structurally wrong.

That’s how you end up with:

  • Hallucinated libraries or outdated syntax (e.g., using React 16 patterns in a Next.js 14 app).
  • Duplicate logic.
  • Wrong assumptions about edge cases.
  • Code that “works locally” but breaks real production flows.

Good documentation reduces this guessing.

The Basic Docs Every Project Should Have

You don’t need massive wikis. You just need a few simple, reliable documents.

1. The “Agent Rules” File (AGENTS.md or .cursorrules)

This is the modern developer’s secret weapon.
Instead of hoping the AI guesses your tech stack, write a simple markdown file in your root directory (like AGENTS.md) that tells the AI exactly how to behave.

  • What UI framework are you using? (e.g., “Use MUI v5, never v4”)
  • What state management? (e.g., “Only use Redux Toolkit”)
  • What are your logging standards? (e.g., “Use Pino, never console.log”)

When you feed this file to your AI Assistant, it instantly writes code that matches your exact project architecture.

2. A Short README Per Service or Module

For each main service or module, the README should explain in plain English:

  • What does this do?
  • How do I run it locally?
  • What are the main inputs and outputs?
  • Which other services or modules does it call?

This is useful for new devs joining the project, your future self, and AI tools when you paste it into the prompt as context.

3. Clear API or Data Contracts

If your project exposes APIs or uses shared data models, you need:

  • An OpenAPI / Swagger file, or a GraphQL schema.
  • Example requests and responses.

With this contract in place, AI can flawlessly generate clients, SDKs, and unit tests. Without it, AI only sees random route handlers and guesses the payload.

4. Simple Decision Notes (ADRs)

When you make an important architectural decision, write a small note:

  • What was the problem?
  • What did we choose and why?
  • What are the trade-offs?

This stops others from repeating old discussions and gives AI the context it needs when you ask it to help plan future changes.

How AI and Docs Work Together

Docs and AI are not enemies. They are a feedback loop.

Here are some easy workflows:

  • Use AI to format your docs: Paste a rough brain-dump and ask, “Turn this into a clean README section.”
  • Use docs to ground your AI: Paste the API spec and ask, “Create unit tests for these endpoints.”
  • Use AI to find doc gaps: Paste a git diff and ask, “Which parts of our README might be outdated now based on these code changes?”

Docs give AI real context. AI then gives you speed.

My Simple Take as a Developer

Here’s how I think about it now:

  • Code is cheap to generate. Understanding is expensive.
  • Clean, small docs save a massive amount of time: fewer Slack pings, fewer “quick calls,” fewer “what does this do?” moments.
  • If important knowledge lives only in your head, your team (and your future self) will pay the tax later.
  • You don’t need perfect English or long pages. A clear bulleted list is enough.

Final Thoughts

AI is powerful, but it’s not magic. It doesn’t automatically know your system, your business rules, or your past decisions.

Before you lean harder on AI, fix the basics: READMEs, API contracts, simple decision notes, and an AGENTS.md file.

Start very small:
Pick one feature you are working on today. Improve its README. Next time you use AI on that part of the codebase, give it this context.

You’ll immediately notice that the answers get more accurate, the code suggestions get more useful, and your project feels less like a black box.

HBO’nun Harry Potter ve Felsefe Taşı dizisinden ilk fragman paylaşıldı

Harry Potter ve Felsefe Taşı dizisinden ilk fragman paylaşıldı

Harry Potter ve Felsefe Taşı dizisinden ilk fragman paylaşıldıHBO Max’in, “Büyücülük dünyasının her noktasını keşfeden bu televizyon dizisi, Harry Potter’ın inanılmaz maceralarını yeni ve mevcut izleyicilere sunuyor.” açıklamasını yaptığı Harry Potter ve Felsefe Taşı dizisinden ilk resmi fragman paylaşıldı. Aralık ayında ekranlara gelmeye başlayacak dizinin başrollerinde Alastair Stout (Ron Weasley rolüğnde), Dominic McLaughlin (Harry Potter rolünde) ve Arabella Stanton (Hermione Granger rolünde) yer alıyor.

HBO’nun Harry Potter ve Felsefe Taşı dizisinden ilk fragman paylaşıldı yazısı ilk olarak Teknoloji haberleri – LOG üzerinde yayınlandı.

Your MCP Server Has No Tests. Here Are 4 Patterns to Fix That.

Most MCP servers ship with exactly one test: a developer typing a prompt into Claude and checking if the output looks right.

That is not testing. That is hoping. And it breaks the moment you change a tool signature, add a parameter, or update your database schema.

Why MCP Servers Are Hard to Test

MCP servers sit between deterministic code and non-deterministic LLMs. Your tools are pure functions — they take inputs, return outputs. But the consumers of those tools are language models that interpret schemas, pick tools based on descriptions, and pass arguments based on inference.

This creates a testing gap. Traditional unit tests cover your business logic. LLM integration tests are slow, expensive, and non-deterministic. The four patterns below close that gap using real MCP protocol interactions — without calling an LLM.

Pattern 1: In-Memory Unit Tests With FastMCP Client

FastMCP 2.x includes a Client class that connects directly to your server in-memory. No subprocess. No network. No LLM. You test your actual server logic through the real MCP protocol — in milliseconds.

Install the testing dependency:

pip install pytest-asyncio

Configure pytest to handle async automatically in pyproject.toml:

[tool.pytest.ini_options]
asyncio_mode = "auto"

Here is a server with two tools:

# server.py
from fastmcp import FastMCP

mcp = FastMCP("WeatherServer")

@mcp.tool()
def get_forecast(city: str, days: int = 3) -> dict:
    """Get weather forecast for a city."""
    if days < 1 or days > 14:
        raise ValueError(f"Days must be 1-14, got {days}")
    return {
        "city": city,
        "days": days,
        "forecast": [{"day": i + 1, "temp_c": 20 + i} for i in range(days)],
    }

@mcp.tool()
def get_alerts(region: str) -> list[str]:
    """Get active weather alerts for a region."""
    alerts_db = {"northwest": ["Wind advisory until 6 PM"]}
    return alerts_db.get(region.lower(), [])

Now write the tests. Create a pytest fixture that wraps your server in a Client:

# test_server.py
import pytest
from fastmcp import Client
from server import mcp  # your FastMCP server instance

@pytest.fixture
async def client():
    async with Client(transport=mcp) as c:
        yield c

async def test_forecast_returns_correct_days(client):
    result = await client.call_tool("get_forecast", {"city": "Seattle", "days": 5})
    data = result.data
    assert data["city"] == "Seattle"
    assert len(data["forecast"]) == 5

async def test_forecast_default_days(client):
    result = await client.call_tool("get_forecast", {"city": "Portland"})
    assert len(result.data["forecast"]) == 3

async def test_forecast_invalid_days_raises(client):
    with pytest.raises(Exception):
        await client.call_tool("get_forecast", {"city": "Seattle", "days": 30})

async def test_alerts_existing_region(client):
    result = await client.call_tool("get_alerts", {"region": "Northwest"})
    assert len(result.data) == 1
    assert "Wind advisory" in result.data[0]

async def test_alerts_unknown_region(client):
    result = await client.call_tool("get_alerts", {"region": "Antarctica"})
    assert result.data == []

Run with pytest -v. Every test executes in milliseconds because it uses in-memory transport. No HTTP, no subprocess, no LLM calls.

This pattern catches three categories of bugs immediately: broken tool logic, wrong return types, and missing error handling.

Pattern 2: Schema Validation Tests

Your MCP tools expose JSON schemas to LLMs. If the schema drifts from your implementation — a renamed parameter, a changed type, a missing description — the LLM picks the wrong tool or passes wrong arguments. Schema tests lock this contract down.

async def test_tool_registry_completeness(client):
    """Verify all expected tools are registered."""
    tools = await client.list_tools()
    tool_names = {t.name for t in tools}
    assert tool_names == {"get_forecast", "get_alerts"}

async def test_forecast_schema_has_required_params(client):
    """Verify the forecast tool schema matches expectations."""
    tools = await client.list_tools()
    forecast = next(t for t in tools if t.name == "get_forecast")

    schema = forecast.inputSchema
    assert "city" in schema["properties"]
    assert schema["properties"]["city"]["type"] == "string"
    assert "city" in schema.get("required", [])

async def test_all_tools_have_descriptions(client):
    """LLMs select tools based on descriptions. Missing = broken routing."""
    tools = await client.list_tools()
    for tool in tools:
        assert tool.description, f"Tool '{tool.name}' has no description"
        assert len(tool.description) > 10, (
            f"Tool '{tool.name}' description too short: '{tool.description}'"
        )

Schema tests catch a specific class of failure that unit tests miss entirely: your code works, but the LLM cannot use it. This happens more often than you think. A developer renames a parameter from city to location. The tool still works. The schema updates automatically. But every prompt template, every LLM workflow, and every agent that was trained on the old schema now sends city and gets a validation error.

Schema tests make this failure loud and immediate. When the parameter name changes, test_forecast_schema_has_required_params fails in CI. The developer sees the break before it ships.

If your server exposes resources alongside tools, test those registrations too:

async def test_resources_are_registered(client):
    """Verify static resources are accessible."""
    resources = await client.list_resources()
    assert len(resources) > 0, "Server exposes no resources"

Run schema tests in CI on every commit. Schema drift is silent and devastating — these tests make it visible.

Pattern 3: Parameterized Edge Case Testing

MCP tools receive arguments from LLMs. LLMs send unexpected inputs — empty strings, extreme values, wrong types. Parameterized tests cover these systematically.

@pytest.mark.parametrize(
    "city, days, expected_count",
    [
        ("Seattle", 1, 1),
        ("Tokyo", 7, 7),
        ("São Paulo", 14, 14),
        ("New York", 3, 3),
    ],
)
async def test_forecast_valid_ranges(client, city, days, expected_count):
    result = await client.call_tool("get_forecast", {"city": city, "days": days})
    assert len(result.data["forecast"]) == expected_count

@pytest.mark.parametrize(
    "invalid_days",
    [0, -1, 15, 100],
)
async def test_forecast_rejects_invalid_days(client, invalid_days):
    with pytest.raises(Exception):
        await client.call_tool(
            "get_forecast", {"city": "Seattle", "days": invalid_days}
        )

@pytest.mark.parametrize(
    "region, has_alerts",
    [
        ("Northwest", True),
        ("northwest", True),
        ("NORTHWEST", True),
        ("southeast", False),
        ("", False),
    ],
)
async def test_alerts_case_insensitive(client, region, has_alerts):
    result = await client.call_tool("get_alerts", {"region": region})
    if has_alerts:
        assert len(result.data) > 0
    else:
        assert len(result.data) == 0

For complex return structures, the inline-snapshot library eliminates manual assertion writing. Install it with pip install inline-snapshot, then write:

from inline_snapshot import snapshot

async def test_forecast_structure(client):
    result = await client.call_tool("get_forecast", {"city": "Seattle", "days": 2})
    assert result.data == snapshot()

Run pytest --inline-snapshot=fix,create once. The library fills in the expected value automatically from the actual output. On subsequent runs, it asserts against the stored snapshot. When your output changes intentionally, run --inline-snapshot=fix to update.

Pattern 4: Interactive Testing With MCP Inspector

Unit tests verify code paths. But sometimes you need to see what the LLM sees — the exact schemas, the raw responses, the protocol messages. MCP Inspector is the official visual testing tool for this.

Launch it against your server:

# For a Python MCP server
npx @modelcontextprotocol/inspector uv --directory ./my-server run my-server

# For a published PyPI package
npx @modelcontextprotocol/inspector uvx mcp-server-git --repository ~/code/repo.git

This opens a web UI at http://localhost:6274 that connects to your MCP server through a local proxy. From the Inspector, you can:

  1. Browse all tools — see the JSON schema exactly as an LLM receives it
  2. Call any tool — fill in parameters through a form, see the raw response
  3. Inspect resources — view static context your server exposes
  4. Test prompts — verify prompt templates render correctly

MCP Inspector serves a different purpose than pytest. Use it for:

  • Exploratory testing during development — try edge cases manually
  • Schema review — verify descriptions are clear enough for LLM tool selection
  • Debugging failures — reproduce exact inputs that caused production issues
  • Demo and documentation — show stakeholders what your server exposes

The Inspector does not replace automated tests. It complements them. Write pytest tests for regression coverage. Use Inspector for exploration and debugging.

A Practical Test Strategy

Combine all four patterns into a layered strategy:

Layer 1: In-memory unit tests (Pattern 1)
  → Run on every save. Sub-second feedback. Catch logic bugs.

Layer 2: Schema validation tests (Pattern 2)
  → Run in CI on every commit. Catch contract drift.

Layer 3: Parameterized edge cases (Pattern 3)
  → Run in CI. Catch boundary failures and type handling.

Layer 4: MCP Inspector (Pattern 4)
  → Use during development. Manual exploration and debugging.

Your pyproject.toml test configuration:

[tool.pytest.ini_options]
asyncio_mode = "auto"
markers = [
    "schema: schema validation tests",
    "edge: edge case and boundary tests",
]

Run fast tests during development:

pytest -v -m "not schema and not edge"

Run everything in CI:

pytest -v --tb=short

What This Costs You

Setting up in-memory MCP tests takes about 30 minutes for an existing server. The fixture is 5 lines. Each test is 3-6 lines. You get sub-second feedback on every change.

Compare that to the alternative: a user discovers your tool returns the wrong type, the LLM hallucinates a workaround, and you spend two hours debugging a production trace.

Thirty minutes of test setup prevents hours of production debugging. That trade is worth it every time.

The MCP ecosystem is growing fast. As of March 2026, thousands of MCP servers exist on npm and PyPI, and the number is accelerating. Most of them have zero automated tests. If you ship yours with a proper test suite, you are already ahead of 90% of the ecosystem. More importantly, your users will trust your server because it actually works when they upgrade.

Follow @klement_gunndu for more MCP and AI engineering content. We’re building in public.

Scaling ID Generation with Redis

The Bottleneck

It Started with a Simple Counter

I work on a cloud-based document management platform used by large construction and engineering firms. Every document uploaded drawings, RFIs, approvals gets an unique ID following a tenant-defined schema:

PROJ-9012-1001
  │      │     │
  │      │     └── Sequence number (auto-incremented)
  │      └──────── Document type identifier
  └──────────────── Project code

An internal microservice called id-generator handled this. It worked fine for years, a closed system behind our portal, moderate traffic, no drama.

Then we opened Public APIs so customers could automate their workflows. And the first large customer tried to migrate 200,000 documents in a single batch.

The id-generator was called once per document, sequentially. Each call was a network hop. The migration ran for over six hours. The customer was not pleased.

Why Sequential Doesn’t Scale: The Math

A single ID generation involves a network round-trip to the id-generator (~100ms) plus the generator’s own processing and sequence counter commit (~50ms). Call it 150ms per ID.

For 200,000 documents, sequential processing means:

200,000 × 150ms = 30,000 seconds ≈ 8.3 hours

“Just add more app instances” doesn’t help. We can spread requests across three instances behind a load balancer, but all three call the same id-generator. The generator processes one request at a time to guarantee sequential numbering. The bottleneck isn’t the app layer, it’s the single-threaded sequence generation.

Even if I made the id-generator handle *10 concurrent requests *(with database level locking on the counter):

200,000 ÷ 10 concurrent × 150ms = 3,000 seconds ≈ 50 minutes

Better, but fragile. The id-generator becomes a high contention hotspot, and any slowdown cascades to every project uploading at the same time.

I needed to decouple ID consumption from ID generation entirely, serve IDs without waiting for the generator.

The Bottleneck

The Architecture

This is a classic producer-consumer problem. Thousands of clients across different projects and organizations fire upload requests through an API gateway. The gateway distributes traffic across multiple app instances. But all instances need IDs from the same sequence space per (project, documentType), and a single producer (the id-generator) feeds that space.

The insight: if I pre-generate IDs in bulk and stash them in a shared store, the app instances become consumers popping from a ready made pool, while the id-generator becomes a background producer that refills the pool asynchronously. Consumers never wait for the producer. The pool is the buffer that decouples them.
I chose Redis as that shared store.

Producer Consumer Architecture

The Fix: Pre-populated ID Pools in Redis

The idea is straightforward. Instead of generating an ID when a request arrives, generate them ahead of time and stash them in Redis.

When a request comes in, pop one off the list. No waiting.
Each (project, documentType) combination gets its own Redis List:

Key:   id-pool:PROJ:9012
Value: ["1001", "1002", "1003", ..., "2000"]   ← 1000 pre-generated IDs

LPOP gives us an atomic, O(1) retrieval. One pop, one ID, sub-millisecond.

But a pool drains. I needed a way to refill it before it runs dry.

Threshold-based Replenishment

I used a watermark pattern, borrowed from stream processing systems like Kafka and Flink. You define a threshold level on a resource, and when usage crosses that mark, you trigger an action before the resource is exhausted.

Think of a water tank with a sensor at the 25% mark: when water drops below it, you automatically reorder before the tank runs dry.

In my case: each time an ID is served, I check the pool size. When 75% of the pool has been consumed (250 or fewer remaining out of 1000), I kick off an async task that calls the id-generator for a fresh batch and pushes them into the Redis list.

public String fetchNextId(String projectCode, String docTypeId) {
    String poolKey = "id-pool:" + projectCode + ":" + docTypeId;
    // Atomic pop - O(1), sub-ms
    String sequence = redisTemplate.opsForList().leftPop(poolKey);
    if (sequence == null) {
        // Pool empty - synchronous fallback (discussed later)
        sequence = idGenerator.generateSingle(projectCode, docTypeId);
    }
    checkAndReplenish(poolKey, projectCode, docTypeId);
    return projectCode + "-" + docTypeId + "-" + sequence;
}

The replenishment runs on a bounded thread pool with CallerRunsPolicy. If the pool and queue are saturated, the request thread itself does the refill. This applies natural backpressure instead of silently dropping work. Or so I thought.

Pool Architecture — Happy Path

This worked beautifully in testing. Sub millisecond ID retrieval. Invisible background replenishment. We shipped it.

The Production Outage: OOM from a Thundering Herd

Within a week of going live with the large customer migration, the service started crashing with java.lang.OutOfMemoryError: Java heap space. Repeatedly.

I pulled a heap dump and started analysing it. The heap was full of hundreds of ArrayList<String> instances, each holding a thousand generated IDs. They were all alive simultaneously.

The root cause was a race condition in the replenishment logic.

Here’s what happened: during a bulk upload, thousands of concurrent requests hit the service. Multiple requests for the same (project, docType) would check the pool level at nearly the same instant, all see “below 75%”, and all independently trigger replenishAsync(). Multiply this across hundreds of (project, docType) combinations, and you get hundreds of async tasks, each generating and holding a 1000 element list in memory.

The heap couldn’t take it.

Counterintuitively, the CallerRunsPolicy I’d chosen for backpressure made things worse. It was supposed to prevent RejectedExecutionException when the executor’s queue filled up. It did by making the request-handling threads also run generation tasks, adding even more large lists to the heap. The policy solved thread pool rejection but amplified the memory problem.

The Thundering Herd

The Fix: Redis Distributed Locks

The problem was clear: nothing prevented duplicate replenishment for the same pool key. I needed a mutex, but a Java ReentrantLock or synchronized block only protects a single JVM. Our service runs on multiple instances behind a load balancer. Instance A’s lock means nothing to Instance B.

I needed a distributed lock. Redis gives you one with a single command:

SET lock:id-pool:PROJ:9012 "a1b2c3-uuid" NX EX 120

NX means “only set if it doesn’t exist”, atomic check-and-acquire. EX 120 means “auto-expire after 120 seconds”, prevents deadlocks if the holder crashes.

The Subtle Bug: Safe Lock Release

My first implementation used a simple DELETE in the finally block. This has a nasty race condition:

t=0s    Instance A acquires lock (TTL=60s)
t=65s   Lock auto-expires (A's generation was slow)
t=66s   Instance B acquires the lock
t=70s   Instance A finishes → DELETE → deletes B's lock
t=71s   Instance C sees no lock → acquires → duplicate work

The fix: store a UUID as the lock value, and use a Lua script for atomic compare-and-delete. An instance only deletes the lock if the value still matches its own UUID.

Why Lua? Redis doesn’t offer a native “delete if value equals X” command. A GET followed by a conditional DELETE in application code has a race window another instance could acquire the lock between the GET and DELETE. Lua scripts execute atomically inside Redis, eliminating that gap.

@Async("idGenExecutor")
public void replenishAsync(String projectCode, String docTypeId) {
    String lockKey = "lock:id-pool:" + projectCode + ":" + docTypeId;
    String lockValue = UUID.randomUUID().toString();
    // Atomic acquire: SET NX EX
    Boolean acquired = redisTemplate.opsForValue()
        .setIfAbsent(lockKey, lockValue, Duration.ofSeconds(120));
    if (Boolean.FALSE.equals(acquired)) {
        return;  // Someone else is handling it
    }
    try {
        // Double-check pool level under lock - another instance
        // may have already refilled between our check and acquire
        Long currentSize = redisTemplate.opsForList().size(poolKey);
        if (currentSize > POOL_SIZE * 0.5) {
            return;  // Already refilled
        }
        List<String> newIds = idGenerator.generateBatch(
            projectCode, docTypeId, POOL_SIZE - currentSize.intValue());
        redisTemplate.opsForList().rightPushAll(poolKey, newIds);
    } finally {
        // Lua script: delete ONLY if value matches our UUID
        String lua = 
            "if redis.call('get',KEYS[1]) == ARGV[1] then " +
            "  return redis.call('del',KEYS[1]) " +
            "else return 0 end";
        redisTemplate.execute(
            new DefaultRedisScript<>(lua, Long.class),
            List.of(lockKey), lockValue);
    }
}

The double-check after acquiring the lock is the distributed equivalent of double-checked locking between the time I decided to replenish and the time I got the lock, another instance may have already done the work.

A note on Redisson’s RLock: it offers automatic lock renewal via a watchdog thread and reentrancy, which sounds appealing. I chose raw SET NX EX + Lua because my replenishment task has a predictable duration (200-500ms) a 120-second TTL gives 200x headroom. If generation ever takes longer than that, something is seriously wrong with the id-generator and I want the lock to expire. Adding Redisson for a single lock pattern felt like pulling in a heavy dependency for a problem I didn’t have.

Distributed Lock Flow

After this fix, OOM crashes dropped to zero. Each pool key gets exactly one concurrent replenishment, regardless of how many instances or threads are contending.

The Problem I Didn’t See Coming: Lost IDs

With performance solved, I hit a subtler issue during failure testing.

When a request pops an ID from the pool but the downstream document persistence fails network timeout, storage error, app exception that ID is gone. It was consumed from the pool, never attached to a document, and on retry the client gets a different ID. This creates sequence gaps and, worse, potential duplicates if the original write eventually succeeds after a timeout.

I considered several approaches: client-supplied idempotency keys (requires client cooperation unreliable with external API consumers), a reservation-commit pattern (extra Redis round-trips per request), and pre-assigned batch reservations (good for migrations, overkill for single uploads).

My Choice: The Outbox Pattern

I went with the Outbox pattern because it solves the problem entirely on the server side. No API contract changes. No client cooperation.

The idea: instead of popping an ID and then separately persisting the document, I pop the ID and write it along with the document metadata to an outbox table in a single database transaction. If the transaction fails, neither the ID assignment nor the record exists atomic.

A separate background processor handles the actual storage. The client gets their ID back immediately; the real persistence happens seconds later.

@Transactional
public String assignId(String projectCode, String docTypeId,
                        DocumentMetadata metadata) {
    String docId = idPoolService.fetchNextId(projectCode, docTypeId);
    // Single transaction: ID + record exist together or not at all
    outboxRepository.save(OutboxEntry.builder()
        .documentId(docId)
        .tenantCode(projectCode)
        .entityTypeId(docTypeId)
        .payload(metadata)
        .status(OutboxStatus.PENDING)
        .retryCount(0)
        .build());
    return docId;  // Client gets the ID immediately
}

The Outbox Processor: Retries, Backoff, and Dead Letters

The processor is a @Scheduled method that polls the outbox table every 2 seconds, picks up PENDING entries, and tries to persist each document to the target store.

On the happy path, it persists the document and marks the entry COMMITTED. But when storage fails with network timeout, S3 returning 500, disk full things get interesting.

A naive approach retries the entry on the next poll, 2 seconds later. If storage is down, I’m hammering it every 2 seconds and blocking the processor from making progress on other entries.

I used exponential backoff instead. Each outbox entry has a next_retry_at timestamp:

next_retry_at = NOW() + (2 ^ retry_count) seconds

Retry 1 waits 2 seconds. Retry 2 waits 4. Retry 3 waits 8. Retry 5 waits 32. Failing entries naturally “sink to the bottom” while fresh entries get processed promptly. The processor’s query becomes:

SELECT * FROM id_outbox 
WHERE status = 'PENDING' AND next_retry_at <= NOW()
ORDER BY created_at LIMIT 100

After a maximum number of retries (I used 5), the entry moves to DEAD_LETTER status. This means the system has given up on automatic recovery and the document payload might be corrupted, the target storage bucket might not exist for this tenant, or there’s a permission issue that no amount of retrying will fix.

Dead-lettered entries become a to-do list for the ops team: investigate, fix the root cause, and either reprocess manually or mark as abandoned.

Why not retry forever? A poisoned entry like corrupted payload, invalid schema will never succeed. Infinite retries waste processing capacity and mask the real bug. The dead letter queue acts as a circuit breaker for individual entries.

The outbox table:

CREATE TABLE id_outbox (
    id              BIGSERIAL PRIMARY KEY,
    document_id     VARCHAR(64) NOT NULL UNIQUE,
    tenant_code     VARCHAR(32) NOT NULL,
    entity_type_id  VARCHAR(32) NOT NULL,
    payload         JSONB       NOT NULL,
    status          VARCHAR(16) NOT NULL DEFAULT 'PENDING',
    retry_count     INT         NOT NULL DEFAULT 0,
    next_retry_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    processed_at    TIMESTAMPTZ
);
-- Partial index: only scans PENDING rows ready for processing
CREATE INDEX idx_outbox_pending
    ON id_outbox (next_retry_at ASC) WHERE status = 'PENDING';

Outbox Pattern Flow

The tradeoff is eventual consistency. The document isn’t in the target store the instant the client gets the ID. There’s a 2–5 second delay while the processor runs. For bulk document uploads, this was perfectly acceptable.

When Redis Goes Down

My fallback is straightforward: if LPOP fails with a RedisConnectionFailureException, fall through to synchronous id-generator calls.

try {
    sequence = redisTemplate.opsForList().leftPop(poolKey);
} catch (RedisConnectionFailureException e) {
    sequence = idGenerator.generateSingle(projectCode, docTypeId);
}

This re-introduces latency during a Redis outage (sub-ms jumps to ~200ms per ID), but the system stays available.

One important detail: the id-generator's sequence counter must live independently of Redis. If the generator also relies on Redis for its counter (INCRBY), then a Redis outage takes down both the pool and the fallback. I backed the counter with a database sequence so the fallback path has no Redis dependency.

The other concern is consistency after Redis recovers. During the outage, the sync fallback generates IDs using the database counter. The Redis pool still holds stale pre-generated IDs from before the crash. If you naively resume popping from the pool, you could issue duplicates.

My approach: on Redis reconnection, invalidate all pool keys and re-seed from the current database counter value. Simple but safe.

Redis Recovery

Results

Remember the back-of-envelope math? With the pool, a single ID retrieval drops from ~150ms (network hop to id-generator) to ~0.5ms (Redis LPOP). Across 3 app instances handling 20 concurrent requests each, the theoretical throughput for 200,000 documents is about 2 minutes. In practice, accounting for API gateway overhead, outbox commits, and occasional sync fallbacks, the observed results:

Results

The gap between theoretical (2 min) and observed (~1 hour) is mainly the outbox processor’s polling interval, document storage latency, and the client’s own upload pacing. The 85% improvement is on the end-to-end migration, not just ID generation but ID generation was the bottleneck that unlocked everything else.

What I’d Improve

Gap Reconciliation

IDs still get lost. An instance can crash between popping from the pool and writing to the outbox. The outbox processor might exhaust retries and send entries to the dead letter queue. These are rare and a handful per million, but they create sequence gaps that confuse customers expecting contiguous numbering.

I’d address this with a periodic reconciliation job that compares three sources: the sequence counter (highest ID ever generated), the document table (which IDs have committed documents), and the outbox table (which IDs are still pending or dead-lettered). Everything in the counter range that doesn’t appear in any of these is a gap. The job writes these to a lightweight id_gaps table with the detection timestamp and inferred reason — outbox_dlq for dead-lettered entries, untracked_loss for IDs that never reached the outbox at all.

An endpoint like GET /api/v1/ids/gaps?project=PROJ&type=9012 would let customers distinguish “this ID was skipped due to infrastructure” from “this document is actually missing.” Gaps become explained rather than mysterious.

I’d avoid trying to reclaim and reuse lost IDs. Reuse sounds clean but risks collision with late-arriving writes from the original assignment the kind of bug that’s nearly impossible to reproduce and debug. Better to waste a few numbers and track them than to reintroduce them into circulation.

Key Takeaways

  1. Pre-populate, don’t generate on-the-fly. When generation is expensive or serial, trade storage for latency.

  2. Watermark replenishment, not reactive. Refill at 75% consumed, not when empty. No request should ever wait on generation.

  3. Distributed systems need distributed locks. Java’s ReentrantLock doesn’t cross JVM boundaries. Redis SET NX EX gives you a lightweight mutex.

  4. Lock ownership matters. UUID lock values + Lua atomic compare-and-delete. Never blindly DELETE a lock key.

  5. Double-check after acquiring the lock. The distributed equivalent of double-checked locking prevents redundant work.

  6. Solve idempotency server-side. The Outbox pattern gives you atomicity without burdening API clients. Exponential backoff and dead letter queues turn “retry until it works” into something manageable.

  7. Don’t reuse lost IDs. Track gaps, explain them, move on. Reuse introduces collision risks that are far worse than a missing number.

UNDERSTANDING CLEAN ARCHITECTURE IN SOFTWARE DEVELOPMENT

hexagonal architecture

Origins and Evolution of Clean Architecture

The concept of Clean Architecture was formalized by Robert C. Martin (“Uncle Bob”) in 2012. It was not a discovery of new principles, but a synthesis of several existing architectural patterns designed to address the increasing entanglement of business logic with framework-specific code.

1. The Precursor Patterns

Clean Architecture integrates ideas from four primary architectural movements:

  • Hexagonal Architecture (Ports and Adapters): Developed by Alistair Cockburn around 2005. It introduced the idea that an application should be equally driven by users, programs, automated tests, or batch scripts, and be developed and tested in isolation from its eventual run-time devices and databases.

hexagonal architecture
Above image is extracted from a blog on about Hexagonal Architecture and Clean Architecture (with examples) by Dyarlen Iber

  • Onion Architecture: Proposed by Jeffrey Palermo in 2008. This pattern placed the Domain Model at the center and established the rule that all coupling is toward the center. It emphasized that the application core should not depend on the database or any infrastructure concerns.

onion architecture
Above image is extracted from a medium blog on Onion Architecture by Ritesh Kapoor

  • Screaming Architecture: A concept emphasizing that the folder structure and organization of a project should communicate the intent of the system (e.g., “Healthcare System”) rather than the frameworks used (e.g., “Ruby on Rails” or “Spring Boot”).
  • DCI (Data, Context, and Interaction) and BCE (Boundary-Control-Entity): These patterns contributed to the definition of how objects interact within use cases and how boundaries between the system and the external world are managed.

2. The Synthesis (2012)

Robert C. Martin unified these concepts into a single actionable framework called “Clean Architecture.” The goal was to provide a standardized approach to building systems that are:

  • Independent of Frameworks: The architecture does not rely on the existence of some library of feature-laden software. This allows you to use frameworks as tools, rather than cramming your system into their limited constraints.
  • Testable: The business rules can be tested without the UI, Database, Web Server, or any other external element.
  • Independent of UI: The UI can change easily, without changing the rest of the system. A Web UI could be replaced with a console UI, for example, without changing the business rules.
  • Independent of Database: You can swap out SQL Server or Oracle for MongoDB, BigTable, CouchDB, or something else. Your business rules are not bound to the database.

3. The Shift Toward Domain-Driven Design (DDD)

The evolution of Clean Architecture is closely tied to the rise of Domain-Driven Design (DDD). It adopted the DDD emphasis on the “Domain” as the most important part of the software. This led to the structural requirement that the innermost circle—the Entities—contains the most general and high-level rules, while the outer circles contain the mechanisms (UI, DB, etc.). This ensures that the most stable part of the application (the business logic) does not depend on the most volatile parts (the technology stack).

The Core Reasoning

The fundamental objective of Clean Architecture is the isolation of business logic from technical implementation details. This separation ensures that the software remains adaptable, maintainable, and testable throughout its lifecycle.

1. Separation of Concerns and Framework Independence

Software systems often become rigid when business rules are tightly coupled to specific frameworks (e.g., Spring, Express, .NET) or libraries. Clean Architecture treats these as “details.” By decoupling the core logic, the organization can upgrade or replace frameworks without rewriting the underlying business rules. This protects the enterprise’s intellectual property from the volatility of the technology market.

2. Protection of the Domain (Domain Integrity)

The most critical reasoning behind this architecture is the preservation of the “Domain.” The domain contains the rules that define how the business functions.

To ensure the domain remains valid and consistent, Clean Architecture utilizes Value Objects. Unlike Entities, which have a unique identity, Value Objects are defined by their attributes. Their core reasoning includes:

  • Self-Validation: A Value Object is responsible for its own validity. It cannot be instantiated in an invalid state. For example, an Email value object will validate the format upon creation.
  • Immediate Error Handling: If the input data fails business constraints, the Value Object throws a domain-specific error (e.g., InvalidEmailException). This prevents “garbage data” from propagating into the deeper layers of the application.
  • Immutability: Once created, a Value Object cannot be changed. Any modification results in a new instance. This eliminates side effects and makes the system’s behavior predictable.

3. Testability Without Side Effects

When business logic is isolated, unit testing does not require a database connection, a web server, or any external infrastructure. Tests run faster and are more reliable because they focus strictly on logic. By using Value Objects that validate themselves, the testing suite can verify that the system correctly rejects invalid data at the boundary of the domain layer, ensuring the “Golden Rule” of the domain is never violated.

4. Dependency Inversion

The reasoning for the Dependency Rule (dependencies point only inward) is to ensure that the high-level policy (Business Rules) does not depend on low-level detail (Data Access, UI).

  • Stable Abstractions: The inner layers define interfaces (Ports).
  • Volatile Implementations: The outer layers implement these interfaces (Adapters).
    This inversion allows developers to defer decisions about databases or external APIs until they are absolutely necessary, or change them later with minimal impact on the core code.

5. Long-term Velocity

While Clean Architecture requires more boilerplate and initial setup (due to the creation of DTOs, Mappers, and Value Objects), the reasoning is to prevent “Software Rot.” By maintaining strict boundaries, the cost of adding new features or changing existing ones remains relatively constant over time, rather than increasing exponentially as the codebase grows.

Architectural Layers

The architecture is organized into four concentric layers, each representing a different level of abstraction and isolation.

Clean architecture by uncle bob

1. Entities (The Domain Layer)

This is the innermost circle, representing the core business rules of the enterprise. It contains the most general and high-level rules that are least likely to change when external factors shift.

  • Entities: Objects that have a unique identity and a lifecycle. They encapsulate the fundamental business logic of the system.
  • Value Objects: These represent descriptive aspects of the domain with no conceptual identity. They are defined by their attributes and are immutable.
    • Validation and Error Handling: Value Objects are responsible for enforcing domain invariants. They validate their own state during instantiation.
    • Logic Enforcement: If a Password Value Object requires a minimum of eight characters, the constructor must validate this constraint. If the input is invalid, it must immediately throw a domain-specific exception (e.g., WeakPasswordException). This ensures that no invalid data can ever penetrate the domain layer.
    • Independence: This layer must have zero dependencies on any external library, framework, or database driver.

2. Use Cases (The Application Layer)

This layer contains application-specific business rules. it orchestrates the flow of data to and from the entities and directs those entities to use their enterprise-wide business rules to achieve the goals of the use case.

  • Functionality: It implements all the “features” of the system (e.g., ProcessOrder, RegisterUser, GenerateReport).
  • Decoupling: Use cases do not know how data is stored or how it is presented to the user. They interact with the outer layers through interfaces (Input and Output Ports).
  • Boundary Crossing: This layer defines the contracts (interfaces) that the infrastructure layer must implement.

3. Interface Adapters

This layer is a set of adapters that convert data from the format most convenient for the use cases and entities to the format most convenient for some external agency like the Database or the Web.

  • Controllers: Convert incoming requests (e.g., HTTP requests) into simple data structures that the use cases can understand.
  • Presenters: Convert the results from a use case back into a format suitable for the UI or API response.
  • Gateways: Interfaces used by the application layer to communicate with external systems (like a database or an external mail service).
  • Mappers: This layer contains the logic to map Domain Entities/Value Objects into Data Transfer Objects (DTOs) or Persistence Models, and vice-versa.

4. Frameworks and Drivers (The Infrastructure Layer)

The outermost layer is composed of frameworks and tools such as the Database, the Web Framework, the GUI, and the configuration.

  • External Mechanisms: This is where the actual implementation of the database (e.g., SQL scripts, MongoDB clients) and the web server (e.g., Express, Spring Boot controllers) resides.
  • Volatility: This layer is the most volatile. Because it is kept separate from the inner layers, replacing a database or switching from a REST API to a message queue involves changing code only in this layer.
  • Dependency: This layer depends on the inner layers by implementing the interfaces defined in the Use Case or Domain layers. It is the “plug-in” area of the architecture.

Proposed File Structure (Layer-Wise)

This structure organizes the project strictly by architectural layers. It separates technical concerns across the entire application, which is suitable for smaller to mid-sized projects where a feature-based split is not yet required.

src/
├── domain/                         # Enterprise-wide business logic and rules
│   ├── entities/                   # Core objects with identity (e.g., User.ts, Order.ts)
│   ├── value-objects/              # Immutable objects with self-validation logic
│   │   ├── Email.ts                # Validates format; throws InvalidEmailError
│   │   ├── Password.ts             # Validates strength; throws WeakPasswordError
│   │   └── Money.ts                # Validates non-negative; throws InvalidAmountError
│   ├── repositories/               # Interfaces/Contracts for data persistence
│   ├── exceptions/                 # Domain-specific error classes
│   └── services/                   # Domain services for logic involving multiple entities
│
├── application/                    # Application-specific business rules (Use Cases)
│   ├── use-cases/                  # Orchestrators (e.g., CreateUser.ts, GetOrder.ts)
│   ├── ports/                      # Interfaces for external services (e.g., IMailer.ts)
│   └── dtos/                       # Data Transfer Objects for Use Case input/output
│
├── infrastructure/                 # External implementations and technical details
│   ├── persistence/                # Database-specific logic
│   │   ├── models/                 # Database schemas or ORM entities
│   │   ├── repositories/           # Concrete implementations of Domain repositories
│   │   └── mappers/                # Maps Persistence Models <-> Domain Entities
│   ├── external-services/          # Concrete implementations of Application ports
│   │   ├── mailer/                 # e.g., SendGrid or SMTP implementation
│   │   └── storage/                # e.g., AWS S3 or LocalStorage implementation
│   └── config/                     # Environment and framework-specific configurations
│
├── presentation/                   # Entry points and delivery mechanisms
│   ├── http/                       # REST/GraphQL API implementation
│   │   ├── controllers/            # Handles requests and invokes Use Cases
│   │   ├── middlewares/            # Auth, validation, and error handling
│   │   └── routes/                 # Endpoint definitions
│   ├── cli/                        # Terminal/Command-line interface logic
│   └── view-models/                # Formatting logic for the final response/UI
│
└── main/                           # Composition Root
    ├── di/                         # Dependency Injection wiring and containers
    ├── factories/                  # Logic for assembling complex object graphs
    └── index.ts                    # Application entry point

Layer-Wise Component Details

Domain Layer: Value Objects and Validation

In this layer-based approach, all Value Objects reside in domain/value-objects/.

  • Validation Logic: Every Value Object contains a private constructor and a static create method (or logic within the constructor) that performs validation.
  • Error Handling: If an input violates a business rule (e.g., a ZipCode that is not 5 digits), the Value Object throws a specific exception from domain/exceptions/. This ensures that an invalid object can never exist within the application or infrastructure layers.

Application Layer: Use Case Orchestration

The application/use-cases/ directory contains the logic that bridges the UI and the Domain.

  • Process: It receives a DTO from the presentation layer, uses the domain/value-objects/ to validate the data, interacts with domain/repositories/ (interfaces), and returns a result.
  • Isolation: This layer knows “what” needs to happen but has no knowledge of “how” the data is stored or “how” the response is rendered.

Infrastructure Layer: The Implementation Detail

The infrastructure/ directory contains the “low-level” code.

  • Mappers: Crucial in this structure to prevent database-specific logic from leaking. infrastructure/persistence/mappers/ convert raw database rows into the rich Domain Entities and Value Objects defined in the center.
  • Adapters: If the application needs to send an email, the concrete implementation (e.g., NodemailerAdapter) stays here, satisfying the interface defined in application/ports/.

Presentation Layer: Data Transformation

This layer is responsible for the “delivery” of the application.

  • Controllers: In presentation/http/controllers/, the code handles the web-specific details like HTTP status codes and headers. It maps the incoming request body into an Application DTO.
  • Error Mapping: This layer is also responsible for catching the domain-specific exceptions (thrown by Value Objects) and converting them into user-friendly error messages or appropriate HTTP status codes (e.g., 400 Bad Request).

Native Android (Kotlin) Clean Architecture File Structure

In standard Android development, Clean Architecture is typically organized into three primary layers: Domain (pure Kotlin), Data (Infrastructure), and Presentation (Android UI). These can be implemented as separate Gradle modules or as packages within a single app module.

Below is the package structure for a single-module Android application utilizing modern components like Jetpack Compose, ViewModels, Room, and Retrofit.

app/src/main/java/com/example/app/
├── domain/                             # Pure Kotlin/Java (No Android dependencies)
│   ├── model/                          # Entities and Value Objects
│   │   ├── User.kt                     # Entity with identity
│   │   └── valueobject/
│   │       ├── Email.kt                # Kotlin class/value class with validation
│   │       ├── Password.kt             # Validates rules and throws Domain Exceptions
│   │       └── Money.kt
│   ├── repository/                     # Interface definitions for data operations
│   │   └── UserRepository.kt
│   ├── usecase/                        # Single-purpose business rules
│   │   ├── CreateUserUseCase.kt
│   │   └── GetUserUseCase.kt
│   └── exception/                      # Custom Domain Exceptions
│       └── DomainExceptions.kt         # e.g., InvalidEmailException, WeakPasswordException
│
├── data/                               # Data management and Infrastructure
│   ├── repository/                     # Implementations of Domain repositories
│   │   └── UserRepositoryImpl.kt
│   ├── mapper/                         # Maps Data Entities <-> Domain Models
│   │   └── UserMapper.kt
│   ├── local/                          # Room Database, SQLite, DataStore
│   │   ├── database/ AppDatabase.kt
│   │   ├── dao/ UserDao.kt
│   │   └── entity/ UserEntity.kt       # Room Database Entity
│   └── remote/                         # Retrofit, Ktor, API Clients
│       ├── api/ UserApiService.kt
│       └── dto/ UserResponseDto.kt     # Network response objects
│
├── presentation/                       # Android UI and ViewModels
│   ├── common/                         # Reusable UI components, themes
│   ├── user/                           # Grouped by screen/flow
│   │   ├── UserViewModel.kt            # Orchestrates Use Cases and manages UI state
│   │   ├── UserUiState.kt              # Data class representing the screen state
│   │   └── UserScreen.kt               # Jetpack Compose UI (or Fragment/Activity)
│   └── mapper/                         # UI Mappers (Domain -> UI State)
│
└── di/                                 # Dependency Injection (Hilt, Koin, or Dagger)
    └── AppModule.kt                    # Provides DB, API, and Repository singletons

Domain Layer: Code Samples (Kotlin/Android)

The Domain Layer is pure Kotlin and contains zero dependencies on Android frameworks, databases, or network libraries.

1. Custom Domain Exceptions

Define specific exceptions to represent business rule violations.

sealed class DomainException(message: String) : Exception(message)

class InvalidEmailException(email: String) : 
    DomainException("The provided email '$email' is not a valid format.")

class WeakPasswordException(reason: String) : 
    DomainException("Password does not meet security requirements: $reason")

class NegativeAmountException : 
    DomainException("Monetary amounts cannot be negative.")

2. Value Objects with Validation

Value Objects encapsulate validation logic. They are immutable and ensure that the system never processes “garbage” data.

@JvmInline
value class Email(val value: String) {
    init {
        val emailRegex = "^[A-Za-z0-9+_.-]+@(.+)$".toRegex()
        if (!value.matches(emailRegex)) {
            throw InvalidEmailException(value)
        }
    }
}

data class Password(val value: String) {
    init {
        if (value.length < 8) {
            throw WeakPasswordException("Must be at least 8 characters.")
        }
        if (!value.any { it.isDigit() }) {
            throw WeakPasswordException("Must contain at least one digit.")
        }
    }
}

3. Entities

Entities have a unique identity (ID) and encapsulate enterprise-wide business rules.

data class User(
    val id: String,         // Unique Identity
    val email: Email,       // Value Object
    val password: Password, // Value Object
    val isActive: Boolean = true
) {
    // Business logic within the entity
    fun deactivate(): User {
        return this.copy(isActive = false)
    }
}

4. Repository Interfaces (Ports)

These interfaces define the contracts for data operations. The implementation resides in the Infrastructure/Data layer.

interface UserRepository {
    suspend fun save(user: User)
    suspend fun findByEmail(email: Email): User?
    suspend fun exists(email: Email): Boolean
}

5. Use Cases (Application-Specific Rules)

Use Cases orchestrate the flow of data. They use the Repository interfaces and Domain Entities to perform a specific task.

class RegisterUserUseCase(
    private val userRepository: UserRepository
) {
    suspend operator fun invoke(emailInput: String, passwordInput: String) {
        // 1. Validation happens automatically during Value Object instantiation
        val email = Email(emailInput)
        val password = Password(passwordInput)

        // 2. Business Rule Check
        if (userRepository.exists(email)) {
            throw DomainException("A user with this email already exists.")
        }

        // 3. Entity Creation
        val newUser = User(
            id = java.util.UUID.randomUUID().toString(),
            email = email,
            password = password
        )

        // 4. Persistence via Interface
        userRepository.save(newUser)
    }
}

Key Implementation Principles in this Layer

  • Validation at the Gate: The Email and Password objects cannot be created in an invalid state. If the RegisterUserUseCase receives bad strings, the code crashes (with a caught DomainException) before any business logic is executed.
  • No Frameworks: Notice the absence of Context, Parcelable, JSONObject, or Room annotations (@Entity, @PrimaryKey). This ensures the core logic is 100% unit-testable on a standard JVM without an emulator.
  • Immutability: Data classes and value classes are used to ensure that once a domain object is validated and created, its state cannot be modified unpredictably.

Data Layer: Code Samples (Kotlin/Android with API)

The Data Layer (Infrastructure) implements the interfaces defined in the Domain Layer. It handles the technical details of communicating with external APIs using libraries like Retrofit and mapping external data formats into Domain Entities and Value Objects.

1. Data Transfer Objects (DTOs)

These classes represent the structure of the JSON data returned by the API. They are annotated with serialization metadata and remain separate from Domain Entities.

import com.google.gson.annotations.SerializedName

data class UserResponseDto(
    @SerializedName("id") val id: String,
    @SerializedName("email_address") val email: String,
    @SerializedName("is_active") val isActive: Boolean,
    @SerializedName("created_at") val createdAt: String
)

data class UserRequestDto(
    @SerializedName("email") val email: String,
    @SerializedName("password_hash") val passwordHash: String
)

2. Retrofit API Service

This interface defines the HTTP endpoints. It uses the DTOs for requests and responses.

import retrofit2.http.Body
import retrofit2.http.GET
import retrofit2.http.POST
import retrofit2.http.Path

interface UserApiService {
    @GET("users/{id}")
    suspend fun getUserById(@Path("id") id: String): UserResponseDto

    @POST("users/register")
    suspend fun registerUser(@Body request: UserRequestDto): UserResponseDto
}

3. Data Mappers

Mappers are responsible for converting DTOs into Domain Entities. During this conversion, strings from the API are passed into Value Objects, which triggers the domain-level validation.

// Extension functions for mapping
fun UserResponseDto.toDomain(): User {
    return User(
        id = this.id,
        // Passing raw API string into the Value Object constructor
        // If the API returns an invalid email, the Value Object throws an exception here
        email = Email(this.email), 
        password = Password("dummy_api_password"), // Passwords typically aren't returned by APIs
        isActive = this.isActive
    )
}

fun User.toRequestDto(): UserRequestDto {
    return UserRequestDto(
        email = this.email.value,
        passwordHash = this.password.value // Simplified for example
    )
}

4. Repository Implementation

This class implements the UserRepository interface from the Domain Layer. It uses the UserApiService to fetch data and the mappers to return domain objects to the application layer.

class UserRepositoryImpl(
    private val apiService: UserApiService
) : UserRepository {

    override suspend fun findByEmail(email: Email): User? {
        return try {
            // Note: In a real scenario, you would have an endpoint to find by email
            val response = apiService.getUserById(email.value)
            response.toDomain()
        } catch (e: Exception) {
            // Log error or handle network-specific exceptions
            null
        }
    }

    override suspend fun save(user: User) {
        val requestDto = user.toRequestDto()
        apiService.registerUser(requestDto)
    }

    override suspend fun exists(email: Email): Boolean {
        return try {
            apiService.getUserById(email.value)
            true
        } catch (e: Exception) {
            false
        }
    }
}

Key Implementation Principles in this Layer

  • Decoupling: The UserResponseDto can change its field names (e.g., from email_address to contact_email) without affecting the User entity in the Domain Layer. Only the Mapper needs to be updated.
  • Infrastructure Isolation: Retrofit annotations and JSON logic are confined to this layer. The Domain Layer remains unaware of the networking library being used.
  • Boundary Validation: When response.toDomain() is called, the Email(this.email) constructor is invoked. If the server sends an invalid email format, the system throws a DomainException at the boundary of the Data Layer, preventing invalid data from entering the application core.
  • Dependency Inversion: This layer depends on the UserRepository interface (Domain) to know what methods to implement, but the Domain does not depend on this implementation.

Presentation Layer: Code Samples (Kotlin/Android)

The Presentation Layer is responsible for rendering data to the screen and capturing user interactions. In modern Android development, this involves a ViewModel to manage state and a UI Component (Jetpack Compose or Fragments) to display it.

1. UI State Representation

The UI state is a data class or sealed class that represents exactly what the user sees on the screen at any given moment.

sealed class RegistrationUiState {
    object Idle : RegistrationUiState()
    object Loading : RegistrationUiState()
    object Success : RegistrationUiState()
    data class Error(val message: String) : RegistrationUiState()
}

2. ViewModel Implementation

The ViewModel interacts with the Use Cases. It is responsible for catching DomainException thrown by Value Objects or Use Cases and converting them into human-readable error messages for the UI.

import androidx.lifecycle.ViewModel
import androidx.lifecycle.viewModelScope
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.launch

class RegistrationViewModel(
    private val registerUserUseCase: RegisterUserUseCase
) : ViewModel() {

    private val _uiState = MutableStateFlow<RegistrationUiState>(RegistrationUiState.Idle)
    val uiState: StateFlow<RegistrationUiState> = _uiState

    fun onRegisterClicked(emailInput: String, passwordInput: String) {
        viewModelScope.launch {
            _uiState.value = RegistrationUiState.Loading

            try {
                // The Use Case is called with raw strings.
                // Internally, the Use Case creates Value Objects (Email, Password).
                // If validation fails, a DomainException is thrown immediately.
                registerUserUseCase(emailInput, passwordInput)

                _uiState.value = RegistrationUiState.Success
            } catch (e: DomainException) {
                // Catch specific domain errors and update the UI state
                _uiState.value = RegistrationUiState.Error(e.message ?: "Invalid Input")
            } catch (e: Exception) {
                // Catch unexpected system/network errors
                _uiState.value = RegistrationUiState.Error("An unexpected error occurred")
            }
        }
    }
}

3. UI Component (Jetpack Compose)

The UI observes the uiState from the ViewModel and reacts to changes. It does not contain any business logic or validation logic; it simply passes strings to the ViewModel.

@Composable
fun RegistrationScreen(viewModel: RegistrationViewModel) {
    val state by viewModel.uiState.collectAsState()
    var email by remember { mutableStateOf("") }
    var password by remember { mutableStateOf("") }

    Column {
        TextField(
            value = email,
            onValueChange = { email = it },
            label = { Text("Email") }
        )

        TextField(
            value = password,
            onValueChange = { password = it },
            label = { Text("Password") },
            visualTransformation = PasswordVisualTransformation()
        )

        Button(onClick = { viewModel.onRegisterClicked(email, password) }) {
            Text("Register")
        }

        // Display state-specific UI
        when (state) {
            is RegistrationUiState.Loading -> CircularProgressIndicator()
            is RegistrationUiState.Success -> Text("Account Created Successfully!", color = Color.Green)
            is RegistrationUiState.Error -> Text("Error: ${(state as RegistrationUiState.Error).message}", color = Color.Red)
            else -> {}
        }
    }
}

Key Implementation Principles in this Layer

  • Logic-Free Views: The RegistrationScreen does not check if the email is valid. It delegates that responsibility to the Domain layer (via the ViewModel and Use Case). This ensures that validation rules are never duplicated in the UI code.
  • State-Driven UI: The UI is a “dumb” reflection of the RegistrationUiState. This makes the UI highly predictable and easy to test with Compose UI tests.
  • Exception Translation: The ViewModel acts as a translator. It catches technical or domain-specific exceptions (like WeakPasswordException) and transforms them into a state that the UI can render (an error message string).
  • Decoupling from Domain: The UI layer depends on the ViewModel, and the ViewModel depends on the Use Case interface. The UI never interacts directly with the UserRepository or the UserApiService.