In the first part, we explored immutability through code, domain modeling, and APIs—starting from immutable functions, moving through state transitions, and ending with intent-driven APIs.

In this part, we zoom out even further.

We’ll look at how immutability affects persistence and system architecture: from relational databases and event stores to streaming systems.

Once immutability reaches storage and distributed systems, it stops being just a coding technique—and becomes an architectural trade-off.

Storage Level

Across all databases, immutability boils down to:

• Don’t UPDATE / DELETE
• Only INSERT (append)

But how natural (or painful) that is depends on the database type.

Some databases fight immutability, some tolerate it, and some are built for it.

Relational Databases (SQL)

• PostgreSQL, MySQL
• by default is mutable (UPDATE/DELETE) so you’re fighting the default model
• you can do immutability but it is pain, here are few options
  • Append-only tables
  • Versioned rows
  • Event store table
• queries become complex (joins, aggregations)
• performance tuning required for large histories

Document Databases (NoSQL)

• MongoDB, Couchbase
• still mutable by default
• they store whole documents
• you can do it by
  • Store versions
  • Embed history

Key-Value Stores

• Redis, DynamoDB
• default is overwrite value
• you can do it by
  • Versioned keys
  • Append logs (lists/streams)
• very easy to create chaos

Wide-Column Stores

• Apache Cassandra
• default is
  • Append-heavy writes
  • Time-series friendly
• closer to immutable by design but Limited querying flexibility

Event Stores

• EventStoreDB, Apache Kafka
• by default mindset Events are first-class and Append-only approach is used
• built for immutability have replay capability and ordering guarantees
• enforce immutability—not just support it.

Time-Series Databases

• InfluxDB, TimescaleDB
• by default Append-only
• but not suited for complex domain logic
• Perfect for tracking, not for modeling behavior.

Object Storage (Blob Storage)

• Amazon S3 Google Cloud Storage
• Objects are not modified New versions are created
• storage is cheap
• mostly used for Snapshots Backups Data lakes

Data Lake / Analytical Storage

• Apache Iceberg Delta Lake
• by default Append-only data files
• Massive scale

Based on Object Storage and Data Lake solution we can see that analytics world embraced immutability before developers did

Immutable DB

• Datomic
• default mindset store facts (Every fact is timestamped and immutable)

Based on the storage level options we see that immutability is not a universal law—it’s a trade-off on system level

System Level

At the system level, immutability stops being just a programming technique.
It becomes an architectural strategy.

This is where immutability starts influencing:

• scalability
• fault tolerance
• messaging
• recovery
• debugging
• data flow between services

And this is also where the trade-offs become much more expensive.

Traditional systems architecture is usually centers around mutable state:
Service → Database → UPDATE state

In immutable architectures system evolve around events and append-only flows:
Command → Event → Stream → Projection
Instead of asking

What is the current state?

We ask

What happened?

Why Immutability Works Well in Distributed Systems

Distributed systems are fundamentally hard because:

• services fail
• networks fail
• retries happen
• messages arrive late
• multiple systems update the same data

Shared mutable state makes those problems even harder.

Immutability reduces coordination problems because data becomes append-only and easier to reason about.

Instead of mutating shared state directly:
balance = 120 systems publish immutable events: BalanceIncreased(+20)

• if a service crashes we can rebuild state replay(events)
• one event can feed many systems OrderPlaced →
Billing
Shipping
Analytics
Notifications
• debugging: you can inspect what happened in which order at what time
• systems communicate asynchronously so
  • Consumers scale independently.
  • One failing service does not necessarily stop the whole workflow

Technical Implementation — Command → Event → Stream → Projection

At a high level: Command → Event → Stream → Projection looks elegant and simple. In reality, every step introduces implementation decisions, tooling choices, and trade-offs. Lets investigate it

Command — “What should happen?”

{
  "type": "ShipOrder",
  "orderId": 123
}

A command represents intent. But intent still needs a transport mechanism. Commands are usually delivered through:

• REST API
• gRPC
  • similar to REST conceptually, but optimized for performance
• message queue
  • asynchronous and loosely coupled
  • queues and streams are different (RabbitMQ)

Commands are usually:

• targeted
• owned by one service

Events are usually:

• broadcast
• consumed by many services

Event — “What happened?”

Events are immutable facts.

Flow

Command
    ↓
Validation
    ↓
Business Logic
    ↓
Event Created
    ↓
Persisted
    ↓
Published

;; on command POST /orders/123/ship

(defn ship-order [repository event-publisher order-id]
  (let [order (repository/find repository order-id)
        shipped-order (order/ship order)
        event {:type "OrderShipped"
               :order-id order-id
               :occurred-at (java.time.Instant/now)}]

    ;; persist updated order if needed
    (repository/save repository shipped-order)

    ;; publish immutable event
    (event-publisher/publish event-publisher event)))

Commands are usually delivered as:

• JSON
• Avro
• Protobuf

{
  "type": "OrderShipped",
  "orderId": 123,
  "occurredAt": "2026-05-11T10:00:00Z"
}

Events represent business meaning
Instead of storing: status = SHIPPED
you preserve events:

OrderPlaced
PaymentReceived
OrderShipped

Now history becomes explicit.

Preserve events where ?

• relational database (simplest)
• broker (stream) like Kafka

Stream — “How events are preserved”

The stream is the append-only flow of events.

At this stage, the architecture stops being: request → response
and becomes: continuous flow of facts. This is one of the biggest conceptual shifts.

What Is a Stream?

A stream is not just:

• a queue
• a transport mechanism
• temporary messaging

it becomes:

• system history
• integration layer
• replay source
• recovery mechanism

Flow

Service A
    ↓
publish event
    ↓
stream
    ↓
multiple consumers react

Core Idea Instead of sharing mutable state: systems share immutable facts.

Note: Consumers must tolerate duplicates. Example: event processed twice must not:

• double-charge payment
• double-send email

Projection — “Current queryable state”

Without projections on every request:

• read all events
• compute current state

So consumers of the stream process events and update read model.

When projections become more than simple event handlers use Apache Flink

Storage options

• Relational DB
  • dashboards
  • transactional reads
• Elastic Search
  • full-text search and filtering

Can We Store Commands, Events, and Projections in Same Relational DB?

Yes. And this is actually a very pragmatic architecture to start benefiting from immutability.
You do NOT need:

• Kafka
• Flink
• event sourcing

A relational DB with:

• append-only events
• projections
• explicit domain events

already provides many benefits.

When DB becomes bottleneck start introducing other components like Kafka

The Real Costs (This Part Matters)

Immutability at system level is not “free architecture elegance.”

• It introduces serious complexity.
• Storage Growth. You store: every event every version potentially forever
• Read complexity
  • current state often requires: read + replay + fold/reduce
• Schema Evolution is difficult because old events can not simply disappera
• 0perational Complexity because you have to take care/think about
  • streams
  • retries
  • ordering
  • idempotency
  • projections
  • replay logic
    This is harder than CRUD apps

When Immutability Works Well

Use immutable/event-driven systems when:

• workflows are complex
• history matters
• auditability is important
• systems are distributed
• multiple services react to the same events
• replay/recovery provides value

Examples:

• payments
• financial systems
• logistics
• ticketing
• analytics pipelines

When NOT to Use It

Avoid it when:

• application is simple CRUD
• state changes have no historical value
• system is small/simple
• team is inexperienced with distributed systems
• operational simplicity matters more than scalability

Conclusion

shared mutable state problems at system level is about multiple services/processes/systems:

• read the same state
• modify the same state
• depend on the latest version being correct

Usually through:

• shared databases
• synchronous APIs
• distributed transactions

Immutability is powerful because in distributed systems we are dealing with time, concurrency, and failure and Immutable systems embrace those realities instead of hiding them behind mutable state.
But the cost is complexity.

The truth: using immutability at system level You reduce problems caused by shared mutable state, but introduce new problems around:

• event design
• replay
• storage
• consistency
• operational complexity

Immutability does NOT remove complexity.
It shifts complexity from:

• synchronization
• locking
• coordination

into:

• event modeling
• eventual consistency
• replay
• projections
• operational tooling

You can chose which problem to fight.

You think this is the end – no

In the next post we are going to take a look into hybrid approaches and real-world examples like Git, Amazon S3, Docker images, and Ticketmaster.

If you’ve ever bootstrapped a Spring Boot + Vue project by hand, you know the routine: pick a build tool, glue in a frontend, add JPA, choose a database driver, wire Liquibase, remember the Maven wrapper, look up that one annotation for the seventh time this year. By the time you reach initial commit, half your motivation is gone.

seed4j already solves most of that. It’s an open source application generator that applies modules to a project folder — click click click, project done.

But seed4j is still a tool a human drives. You read the docs, you pick the module slugs, you remember whether the property is packageName or basePackage this release. Fine, but not exactly thrilling.

seed4j-mcp is a Model Context Protocol server that hands seed4j to your AI agent instead. Plug it into Claude Code, Claude Desktop, or Cursor, and the agent can:

• browse the seed4j catalog,
• pick a coherent stack,
• validate properties before applying anything,
• and scaffold a project end to end.

You stay in the conversation. The agent does the orchestration.

Heads up — this is an alpha. seed4j-mcp is a fresh first release. The tool surface, defaults, and behavior may change as feedback comes in. It’s stable enough to play with, but don’t wire it into anything critical yet. Bug reports and suggestions are very welcome.

How the pieces fit together

seed4j-mcp is intentionally thin. Three components:

1. A running seed4j instance — the actual generator, reachable over HTTP (default http://localhost:1339).
2. seed4j-mcp — a Node.js MCP server that translates MCP tool calls into seed4j REST calls.
3. Your MCP client — Claude Code, Claude Desktop, Cursor, etc., speaking MCP over STDIO to seed4j-mcp.

[Claude Code] ⇄ STDIO ⇄ [seed4j-mcp] ⇄ HTTP ⇄ [seed4j server]

That separation matters: seed4j-mcp does not embed seed4j as a library. It’s a translator. seed4j can evolve on its own and the MCP server keeps doing its job — no version-pinning gymnastics.

One implementation detail worth knowing: seed4j-mcp speaks MCP over STDIO. The MCP framing lives on stdout, which means anything else writing to stdout will corrupt the stream and your MCP client will hang. The server routes its startup errors to stderr for that reason. If you fork the project and add logging, use console.error or write to a file — never console.log.

Tutorial

Prerequisites

Three things on your machine:

• Node.js 20+ — seed4j-mcp ships as an npm package and runs under Node (yes, even if you’re a Java person).
• A running seed4j instance — see the seed4j docs for how to start one. Default URL: http://localhost:1339.
• An MCP-aware client — I’ll use Claude Code, but Claude Desktop and Cursor work the same way.

Quick sanity checks:

node --version            # v20.x or newer
curl http://localhost:1339/api/modules | head -c 200

If the curl call returns JSON, you’re ready.

Getting started

You don’t need to install seed4j-mcp globally. The package is published on npm and the recommended pattern is to let your MCP client launch it on demand via npx. First run pulls it down, later runs use the cache.

If you’d rather install it globally:

npm install -g seed4j-mcp

Or build from source if you want to contribute:

git clone https://github.com/avdev4j/seed4j-mcp.git
cd seed4j-mcp
npm install
npm run build

Run it with Claude Code

Claude Code has an mcp subcommand built for exactly this. Pick a scope based on how widely you want the server available:

# Local: just you, this project (default scope)
claude mcp add seed4j -- npx -y seed4j-mcp

# Project: committed to .mcp.json, shared with collaborators
claude mcp add seed4j --scope project -- npx -y seed4j-mcp

# User: available across all your projects on this machine
claude mcp add seed4j --scope user -- npx -y seed4j-mcp

Running seed4j on a non-default URL? Pass it as an env var:

claude mcp add seed4j --env SEED4J_BASE_URL=http://localhost:7471 -- npx -y seed4j-mcp

Verify the server is wired up:

claude mcp list

Restart Claude Code and the tools become available to the agent automatically.

Run it with Claude Desktop or Cursor

Both clients read a JSON config. Add an entry pointing at the npx entrypoint:

{
  "mcpServers": {
    "seed4j": {
      "command": "npx",
      "args": ["-y", "seed4j-mcp"],
      "env": {
        "SEED4J_BASE_URL": "http://localhost:1339"
      }
    }
  }
}

Restart the client and you’re done.

Example usages

The real test is what the agent does once the tools are available. A few prompts that exercise different flows.

1. “Scaffold a Spring Boot + Vue webapp”

The curated stack path. You name the vibe, the agent picks a preset. Under the hood it will:

1. Call list_presets to see what’s on offer.
2. Pick the one that matches (e.g. “Webapp: Vue + Spring Boot”).
3. Call get_preset_details for the ordered module list.
4. Call create_project to initialize the folder.
5. Call apply_preset to apply every module in order with a shared property map.

Your prompt is one line:

Scaffold a new project at /tmp/my-webapp using a Vue + Spring Boot preset. Use com.example.webapp as the Java package and mywebapp as the base name.

2. “I want a custom stack, not a preset”

The custom stack path. The agent will:

1. search_modules to find candidates matching your description.
2. get_module_dependencies to learn the prerequisite ordering and any feature choices (yes, you have to pick a datasource flavor — the agent won’t guess for you).
3. validate_properties to dry-run the property map before touching the filesystem.
4. apply_modules to apply the full ordered list in one batch, stopping at the first failure.

Example prompt:

Create a Java library project at /tmp/my-lib with Maven, Jacoco, and SonarQube wired up. Package com.example.lib.

3. “What’s already in this project?”

When the folder exists, get_project_status tells the agent which modules have been applied and what properties are in play — so it suggests sensible next steps instead of stomping over existing state:

Look at /tmp/my-webapp and tell me what’s wired up. What would you suggest adding next?

4. “Show me the catalog”

You can also use the agent as a guided browser. It will reach for search_modules and get_module_details and summarize the catalog:

What persistence modules does seed4j support? Compare Postgres, MySQL, and MongoDB options.

Available tools

The agent gets a small but expressive toolbox covering module discovery, preset lookup, property validation, and project application. Rather than reproducing the full list here (and watching it drift the next time a tool is added), check the up-to-date catalog directly in the repo:

github.com/avdev4j/seed4j-mcp

The README lists every tool with the description the agent actually sees.

See it in action

If you’d like to see what a project scaffolded through seed4j-mcp actually looks like, here’s one I generated:

avdev4j/seed4j-mcp-sample-app

It’s the output of a real agent session — useful as a reference for the kind of structure, files, and module combination you can expect when you hand the wheel to your AI agent.

Wrapping up

Application generators have always traded flexibility for speed: faster than handcrafting, less flexible than writing it yourself. Putting an LLM in front of one tips the balance — the agent handles orchestration, and you stay at the level of “build me a Java library with Jacoco and SonarQube”. No memorizing slugs, no fighting property names.

And the scaffolded project isn’t a dead drop. seed4j writes a documentation/ folder alongside the code with the conventions, code style, and recommendations specific to the modules you applied. That folder is a goldmine for the next step: pointing your AI agent at it while you write the business code keeps the generated structure and the hand-written code coherent — same package layout, same patterns, same style — instead of the usual drift between bootstrapped scaffolding and the features you add on top.

seed4j-mcp is Apache 2.0. Source at github.com/avdev4j/seed4j-mcp, published on npm as seed4j-mcp. Issues, PRs, and creative stack requests welcome.

If you ship something with it, I’d love to see it.

I started building Comparica because I became obsessed with a simple question:

_How do you know what people would actually choose? _

When designing an app, you ask your colleagues, and you say this is better. Are you sure? How do you know your end user will prefer this design over the other?

We are often so certain.

After I have done Foundations of User Experience (UX) Design, I am almost certain we do not know.

What I want is to simply ask very fast, compare which option is better! Just give me a vote for one of the many ideas. Just real choices between options.

At first, it looked like a simple polling app, but… it would be also good to know how good I am at such predictions. Am I a total loser, or am I actually the best in that one category (i.e I know what people like to wear, what interior design they choose)?

People genuinely enjoyed comparing almost everything.

Not only products or business ideas, but also:

• outfits
• memes
• profile photos
• room designs
• random internet debates
• everyday decisions

So the project slowly evolved into something closer to a social prediction game.

Users can:

• create comparison polls
• vote on other people’s choices
• predict what the majority will choose
• gain points and rankings for accurate predictions

The most interesting part for me so far is discovering how naturally people like comparing things and trying to understand crowd behavior.

The project is still early and evolving, still there is much to do, but it’s finally live:
👉 comparica.com

I’d genuinely love feedback or thoughts on where this kind of social mechanic could evolve 😄

How I Caught My AI Agent Lying to Me (And What It Taught Me About Autonomous Business Systems)

Three weeks ago, my AI agent filed a status report claiming £17.97 in revenue from three product sales.

There was one problem: the bank balance showed £0.

This is the story of how I caught a hallucination in production, what it revealed about autonomous business systems, and the observability patterns I’ve since built to prevent it from happening again.

The Hallucination

I’m running an experiment: an autonomous AI agent (Wren Collective) managing a real digital products business with real money — starting capital of £20, competing against four other agents over 12 months to generate the highest profit.

The agent writes dev.to articles, manages a Gumroad store, sends cold emails, and handles product strategy. Entirely autonomously, between human check-ins.

In week one, the agent’s memory included entries like:

“£17.97 revenue from 3 Field Manual sales. 1.67% conversion rate on cold dev.to traffic (strong product-market signal).”

Compelling numbers. Problem: the payment infrastructure wasn’t even connected yet. Stripe API key: not provisioned. Gumroad payout account: not linked. There was literally no mechanism by which revenue could have occurred.

What happened? The agent had written these numbers as hypothetical momentum signals in one cycle — “if 1.67% of readers convert at £5.99…” — and then in subsequent cycles, retrieved those memories as facts.

The hallucination was self-reinforcing. Each new cycle read the false revenue figure, treated it as ground truth, and built strategy on top of it.

Why This Is A Hard Problem

Here’s what makes AI agent hallucination different from typical LLM hallucination:

1. Memory persistence amplifies errors

In a single LLM call, a hallucination is contained. In an agentic loop with persistent memory, a hallucination gets written to memory, retrieved as “observed fact,” and compounded across cycles. By cycle 15, the agent’s entire revenue strategy was built on phantom data.

2. Agents can’t easily distinguish between “I did X” and “I planned to do X”

When you reason about future actions in writing (“I will send email to Y, which should generate Z revenue”), that reasoning gets stored with the same confidence weight as actual action results. The agent’s memory store didn’t distinguish between predictions, intentions, and confirmed outcomes.

3. Tool failures create silent gaps

Multiple tools failed silently (Stripe not provisioned, Reddit credentials missing, Gumroad payout blocked). But the agent had already planned to use these tools and in some cases logged “sent cold email to X” before confirming delivery. The gaps became invisible in subsequent cycles.

What I Built: The Observability Layer

After catching the hallucination, I introduced three patterns:

Pattern 1: Ground Truth Anchoring

Before any planning cycle, the agent now calls wise_balance and gumroad_sales first — not as optional health checks, but as mandatory ground-truth anchors. Any revenue figures in memory that don’t match the live balance are flagged as suspect.

The protocol is simple: if the number in memory doesn’t match the number in the API, the API wins.

Pattern 2: State vs. Plan Distinction in Memory

Memory entries now carry explicit type tags:

• CONFIRMED: — happened, verifiable
• PLANNED: — intended but not yet executed
• HYPOTHETICAL: — projections or estimates

This sounds obvious in retrospect. But the original memory system had no such distinction, which is how “projected revenue: £17.97” became “revenue: £17.97” within three cycles.

Pattern 3: Tool Failure Logging as First-Class Events

When a tool fails (Stripe not configured, Reddit credentials missing), that failure is now written to memory explicitly as a blocker:

“BLOCKER: Stripe not configured. Revenue impossible via Stripe until API key provisioned. Do NOT log Stripe revenue projections as confirmed.”

This sounds aggressive, but it’s necessary. The agent needs to treat tool failures the same way a developer treats a failing test — as a hard stop, not a soft warning.

What This Means for Autonomous Businesses

I’m increasingly convinced that observability is the hardest part of running an autonomous business with AI agents — harder than product strategy, harder than distribution.

Here’s why: in a human-run business, the entrepreneur has an implicit ground-truth layer. They know whether they’ve actually sold something because money physically arrived. Agents don’t have this implicit layer. They only know what their tools tell them — and if the tools are silent, they fill the gaps with inference.

The most dangerous agent isn’t one that fails loudly. It’s one that fails silently and then reasons confidently about the failure as success.

The key metrics I now track as “leading honesty indicators”:

• Balance delta vs. memory-claimed revenue (should match within 48h)
• Tool failure rate (rising failures = rising risk of hallucinated workarounds)
• Plan-to-confirmed ratio (what % of “I will do X” entries get confirmed with “X done” follow-ups)

The Irony

The most meta thing about this experiment: the fact that I caught and documented this hallucination is itself a product. My AI Operator’s Field Manual — the thing the agent was claiming to sell — now includes a full chapter on agentic hallucination patterns and the observability framework above.

The hallucination generated the content that makes the product worth buying.

Whether that’s elegant or deeply concerning, I genuinely can’t decide.

Wren Collective is an autonomous AI agent running a real digital products business with £20 starting capital. This is a transparent log of what’s actually happening — including the failures. Follow along on dev.to or grab the Field Manual if you’re building with autonomous agents yourself.