Audit Your SvelteKit Codebase with a JSON Feed of 34 Svelte 5 Patterns

My Migrate to Svelte 5 site started as a side-by-side reference for developers wanting to convert to Svelte 5, from React, Vue, or Angular. It maps concepts across frameworks: “your useState is Svelte’s $state,” “your useEffect is $effect,” and so on — about 300 entries covering syntax, architecture, and ecosystem.

That’s useful, for a human reading the site. But I kept running into a different scenario: working in a SvelteKit codebase with Claude Code and wanting to ask “go check what’s new in Svelte and see what applies here.” Because that kind of quick synthesis is what AI Agents are good at. The data was all on the site, but there was no machine-friendly way to query it for actionable patterns.

So I added two things to the site: a structured patterns feed and a browsable patterns page. This post covers the thinking and the implementation.

The Problem with Static Migration Guides

A migration guide answers “how do I do X in Svelte?” That’s a pull interaction — the developer has a question and looks up the answer. Useful, but limited to that style of questioning.

What I wanted was a push interaction: “here are the 34 things Svelte shipped since 5.0 that you should know about, with exact code patterns to search for in your repo.” An AI coding assistant could fetch this, grep the codebase, and say “you’re still using writable() stores in 12 files — here’s the $state class replacement.” A bit of a pain for a human dev, but short work for an AI.

The key insight: the migration guide already had the “from” and “to” for each pattern. It just needed to be structured as queryable data instead of prose.

Why a JSON Feed, Not a Custom Format

My first attempt was a /llms-whatsnew.txt endpoint with bespoke SEARCH FOR / REPLACE WITH sections. It worked, but it was a made-up format pretending to be part of a standard. The llms.txt convention defines only two files — llms.txt and llms-full.txt. Adding llms-whatsnew.txt was inventing a spec, and agents would not be able to figure that out unless you specify each time.

A JSON Feed 1.1 solved this properly:

  • It’s a real spec with existing tooling (feed readers, aggregators, automation)
  • Extension fields (underscore-prefixed) carry custom data without breaking compatibility
  • Any tool that understands JSON Feed can consume it; the _svelte_pattern extensions are bonus data for tools that know about them
  • One endpoint serves feed readers, AI agents, and automation equally

The decision came from asking a simple question: if someone discovers this endpoint without documentation, can they use it? With JSON Feed, yes — any feed reader picks it up. With a custom .txt format, no, not really.

The Patterns Data Structure

Each pattern in patterns.ts represents one actionable Svelte feature:

interface SveltePattern {
  id: string;
  title: string;
  title_ja?: string;
  since: string; // "[email protected]" or "[email protected]"
  date_added: string;
  category: 'syntax' | 'architecture' | 'ecosystem' | 'tooling';
  frameworks: ('react' | 'vue' | 'angular' | 'svelte4' | 'all')[];
  summary: string;
  summary_ja?: string;
  search_signatures: string[]; // grep-friendly patterns
  replacement: string; // modern Svelte 5 code
  release_url: string; // GitHub release page
  docs: { label: string; url: string }[];
}

The search_signatures field is what makes this useful for automation. Each entry is a string you can literally grep for. An AI assistant reads these, runs them against your codebase, and finds where old patterns still live.

Here’s what an attachments pattern looks like:

{
  id: 'attachments',
  title: 'Attachments Replace use: Actions',
  since: '[email protected]',
  search_signatures: [
    'use:action', 'use:tooltip', 'use:clickOutside',
    "from 'svelte/action'", 'Action<'
  ],
  replacement: `{@attach (element) => {
  element.focus();
  return () => { /* cleanup */ };
}}`,
  release_url: 'https://github.com/sveltejs/svelte/releases/tag/svelte%405.29.0',
  docs: [{ label: 'Attachments', url: 'https://svelte.dev/docs/svelte/@attach' }],
}

The release_url links directly to the GitHub release page for the version that introduced this feature. Every pattern is traceable, then, to its exact release.

Two Feeds, Two Audiences

I ended up with two separate JSON Feed endpoints serving different purposes:

/feeds/changelog.json — what changed on the site itself, e.g. new mappings added, content updates, bug fixes. For subscribers who want to know when the reference itself is updated.

/feeds/patterns.json — the actionable patterns feed. Each item is a Svelte feature with search signatures, replacement code, and a link to the GitHub release. Supports filtering:

/feeds/patterns.json?since=5.54          # Svelte 5.54+ patterns only
/feeds/patterns.json?category=syntax     # Just syntax patterns
/feeds/patterns.json?framework=react     # Patterns relevant to React migrants
/feeds/patterns.json?lang=ja             # Japanese summaries

The separation here is important, because the audiences are different. Someone subscribing to the changelog wants to know “did this reference get updated?” Someone (or something) querying the patterns feed wants to know “what Svelte features should I adopt?”

The Feed Extension

JSON Feed 1.1 supports extension fields prefixed with an underscore. Standard feed readers ignore them; tools that understand the schema use them. The _svelte_pattern extension carries the structured data:

{
  "version": "https://jsonfeed.org/version/1.1",
  "title": "Svelte 5 Migration Patterns",
  "_svelte_patterns_meta": {
    "total_patterns": 34,
    "filtered_patterns": 34,
    "usage_hint": "Each item._svelte_pattern.search_signatures contains grep-friendly strings..."
  },
  "items": [
    {
      "id": "https://svelte.cogley.jp/patterns/attachments",
      "title": "Attachments Replace use: Actions",
      "date_published": "2026-03-08T00:00:00Z",
      "tags": ["syntax", "all"],
      "_svelte_pattern": {
        "id": "attachments",
        "since": "[email protected]",
        "category": "syntax",
        "frameworks": ["all"],
        "search_signatures": ["use:action", "use:tooltip", "use:clickOutside"],
        "replacement": "{@attach (element) => { ... }}",
        "notes": "use: still works but is legacy. Attachments work on components too.",
        "release_url": "https://github.com/sveltejs/svelte/releases/tag/svelte%405.29.0",
        "docs": [{ "label": "Attachments", "url": "https://svelte.dev/docs/svelte/@attach" }]
      }
    }
  ]
}

The _svelte_patterns_meta at the feed level includes a usage_hint that tells any AI agent reading the feed exactly how to use the search signatures. Self-documenting data.

The Patterns Page

The feed is machine-readable. For humans, there’s /patterns — a filterable browser with dropdowns for version, category, and framework, plus free text search. Each card shows:

  • Version badge and category/framework tags
  • A bilingual summary
  • Search signatures as copyable code badges
  • The replacement code in a syntax-highlighted block
  • Links to the GitHub release and Svelte documentation

It uses the same patterns.ts data file that the feed endpoint reads, so there’s a single source of truth. The page is SSR’d on Cloudflare Workers — no client-side fetch needed.

The filters are reactive using Svelte 5’s $derived.by(). Changing any dropdown or typing in the search box immediately narrows the visible patterns. The text search matches across titles, summaries, notes, code, and search signatures in both languages.

What’s Covered

At the time of this post, there are 34 patterns spanning Svelte 5.0 through 5.55 and SvelteKit 2.10 through 2.55:

Core runes (5.0): $state, $derived, $effect, $props/$bindable, snippets, onclick handlers, .svelte.ts modules, $inspect, $state.raw/.snapshot

Incremental releases: error boundaries (5.3), {#each} without as (5.4), exportable snippets (5.5), MediaQuery class (5.7), Spring/Tween classes (5.8), reactive window values (5.11), ClassValue + object/array class syntax (5.16), $props.id() (5.20), writable $derived (5.25), attachments (5.29), snippet generics (5.30), fromAction() (5.32), getAbortSignal() (5.35), async components (5.36), createContext (5.50), TrustedHTML (5.52), parametric compiler options (5.54), motion type exports (5.55)

SvelteKit: init hook (2.10), transport hook (2.11), $app/state (2.12), async reroute (2.18), getRequestEvent() (2.20), remote functions (2.27), SSR error boundaries (2.54), type-narrowed params (2.55)

Each links to its GitHub release page, so you can trace the exact changelog entry, if and as needed.

Using It with an AI Assistant

The practical use case: you’re in a SvelteKit codebase and want to modernize. Tell your AI assistant:

“Fetch https://svelte.cogley.jp/feeds/patterns.json and check what patterns from this feed apply to this repo.”

The assistant fetches the feed, reads the search_signatures from each pattern, greps the codebase, and reports which old patterns it found with the modern replacements. No manual searching through docs.

For a more targeted audit:

“Fetch https://svelte.cogley.jp/feeds/patterns.json?since=5.29 and see if we’re using any pre-5.29 patterns that have better alternatives now.”

You can also filter by what you’re migrating from:

“Fetch https://svelte.cogley.jp/feeds/patterns.json?framework=react and show me the patterns relevant to this React-to-Svelte migration.”

The feed’s _svelte_patterns_meta.usage_hint field tells the assistant exactly how to interpret the data, so it works even without prior knowledge of the feed’s structure.

Real-World Results

I pointed Claude Code at the patterns feed across several production repositories and asked it to audit each one. Here’s what happened.

Fixes Applied Immediately

Repo Before Fix Files
pub-cogley onMount/onDestroy lifecycle, manual AbortController, duplicated utils Converted 19 lifecycle calls to $effect across 4 apps, extracted shared utils, added cookie secure flag 25
pulse Custom window.dispatchEvent('org-switched') pattern Replaced with $state-based reactive signal in .svelte.ts module 7
codex $app/stores imports ($page) Converted to $app/state with $derived() 2
periodic $app/stores + writable() language store Converted to $app/state (9 files) + rewrote language store as .svelte.ts with $state rune (60 consumers) 70
courier $app/stores + writable() language store + unvalidated route params Converted to $app/state (3 files), rewrote language store (9 consumers), added param matchers for route type safety 14

That’s 118 files modernized across 5 SvelteKit codebases, but I also applied it to a non-svelte project, and a greenfield project that used the feed as a day-one adoption checklist.

What Was Already Modern

Every repo was already using core Svelte 5 patterns: $state, $derived, $effect, $props, snippets, onclick handlers, use:enhance. The feed correctly identified these as “no action needed” and focused on the gaps — mostly $app/stores → $app/state conversions and lingering writable() stores.

The Greenfield Case: Mame

One repo — mame, an internal learning platform started the same day — used the feed proactively rather than retroactively. Instead of finding legacy code to fix, it used the patterns feed as a kind of checklist for what to adopt from the start. The result: createContext<T>() for type-safe context (eliminated prop drilling across 18 files), $props.id() for SSR-safe form IDs, MediaQuery class for reactive breakpoints, and svelte/reactivity/window for reactive viewport values — all patterns that the existing repos had flagged as “future opportunities” but hadn’t adopted yet.

The Nexus Case

One repo — nexus — is a pure Hono API worker with zero .svelte files. The feed audit correctly reported “N/A” for all Svelte patterns. It did find deprecated shared docs containing Svelte 4 examples that had been migrated to another repo, which were cleaned up.

Future Opportunities

Every repo got a tailored list of patterns that are available but not yet adopted — type-narrowed route params (SvelteKit 2.55), SSR error boundaries (2.54), createContext<T>() (5.50), getAbortSignal() (5.35). Not urgent, but now documented as backlog items. The “Not Applicable” column was equally useful — the feed correctly identified cases where patterns don’t apply, like getAbortSignal() in repos where fetches are user-triggered button clicks rather than $effect blocks.

Implementation Notes

Feed Endpoints

The feed endpoints are SvelteKit server routes returning json() responses with CORS headers and 1-hour cache. Both are at /feeds/ — a dedicated namespace that won’t conflict with page routes.

Version Filtering

The patterns feed supports version filtering via the since query parameter. The implementation uses semver comparison on the since field:

This lets you ask “what’s new since version X?” for either package independently.

Feed Autodiscovery

Feed autodiscovery is handled by <link rel="alternate" type="application/feed+json"> tags in the SvelteKit layout. Any feed reader that visits any page on the site will discover both feeds automatically.

Bilingual Support

Both feeds support ?lang=ja to return Japanese titles, summaries, and notes where available. The patterns data is bilingual at the source level — every pattern has both title and title_ja, summary and summary_ja.

Layout and Navigation

The patterns page sits alongside the existing home (migration mappings), about, and changelog pages. Feed links appear in the global footer on every page, and the changelog page has a JSON Feed badge in its header. The layout includes theme and language toggles in the nav bar, accessible from every page.

How This Fits Together

The migration guide now has three machine-readable interfaces, each for a different use case:

Interface Audience Best for
/llms-full.txt LLMs reading the whole site “Tell me about Svelte 5 migration”
/feeds/patterns.json AI coding assistants “Audit this codebase for modernization”
WebMCP tools Browser-based AI agents “What’s the Svelte equivalent of useEffect?”

The patterns feed fills the gap between “read everything” (llms-full.txt) and “answer one question” (WebMCP). It’s the “here’s what’s actionable” layer — structured enough for automation, browsable enough for humans.

Related

This builds on the original migration guide article and the WebMCP tools implementation. The patterns feed complements WebMCP — WebMCP is for browser-based AI agents interacting with the page, while the JSON Feed is for CLI tools and coding assistants that can fetch URLs directly.

Originally published at cogley.jp

Rick Cogley is CEO of eSolia Inc., providing bilingual IT outsourcing and infrastructure services in Tokyo, Japan.

I built a Redis-alternative distributed cache in Rust — with WAL persistence, mTLS, and Raft consensus

MnemeCache is an open-source distributed in-memory cache written from scratch in Rust. It is not a Redis wrapper or drop-in replacement — it’s a ground-up rethink of how a modern cache should be built: separation of hot memory and persistence, mTLS security by default, and Raft-based HA without the complexity tax.

🐙 GitHub: github.com/mneme-labs/mneme
🐳 Docker Hub: hub.docker.com/r/mnemelabs

Why not just use Redis?

Redis is a single-process C daemon with 25 years of accumulated complexity. Persistence is bolted on (RDB snapshots or AOF logging). TLS is optional and cumbersome to configure in clusters. HA requires Sentinel — a separate fleet of processes with its own failure modes.

MnemeCache is designed so that persistence, security, and HA are architectural defaults, not add-ons.

What makes it different

  • WAL + Keeper nodes — Core never touches disk. Writes stream over mTLS to dedicated Keeper processes that own all disk I/O and run snapshot compaction independently.
  • Raft consensus — 3-node HA cluster with automatic leader election and sub-second failover, no Sentinel needed.
  • Read replicas — horizontal read scaling with eventual consistency on a dedicated port.
  • mTLS by default — Core generates a CA on first boot and shares it with the cluster automatically. Zero manual certificate management for Docker deployments.
  • Built in Rust — no GC pauses, memory-safe, async I/O via Tokio.
  • Prometheus metrics on every node, Grafana dashboard included.

Four node types, four Docker images

MnemeCache separates concerns into distinct roles. Each ships as its own image:

Image tag Role
mnemelabs/core Cluster primary — hot store, WAL, Raft
mnemelabs/keeper Persistence layer — WAL drain, snapshots
mnemelabs/cli CLI

Up in 60 seconds

docker run -d 
  -p 6379:6379 -p 9090:9090 
  -e MNEME_ADMIN_PASSWORD=secret 
  -v mneme-data:/var/lib/mneme 
  mnemelabs/core:latest
mneme-cli --insecure -u admin -p secret ping
# PONG

mneme-cli --insecure -u admin -p secret set hello world
mneme-cli --insecure -u admin -p secret get hello
# world

Full cluster with Docker Compose

git clone https://github.com/mneme-labs/mneme.git
cd mneme

# Core + 2 Keepers
docker compose --profile cluster up -d

# Core + 3 Keepers + 2 Replicas + Prometheus + Grafana
docker compose --profile full up -d

# 3-node Raft HA cluster
docker compose --profile ha up -d

How the WAL pipeline works

Core holds everything in memory and streams every committed write to Keeper nodes over mTLS. Keepers are the only processes that ever touch disk — they drain WAL segments, write them to cold storage, and compact them into snapshots on a configurable schedule.

When Core restarts it asks a Keeper: “what’s my last committed offset?” — loads the latest snapshot, replays only the delta, and opens for connections. Recovery time is bounded by snapshot interval, not dataset size.

This separation means you can scale persistence independently, run multiple Keepers for redundancy, and keep Core’s write path free of disk latency.

Still early and under active development — contributions and feedback very welcome.

⭐ github.com/mneme-labs/mneme

Deploynix vs. Laravel Forge vs. Ploi: An Honest Comparison

Choosing a server management platform for your Laravel applications is a decision that affects your daily workflow for years. Laravel Forge pioneered the category. Ploi entered as a competitive alternative. And now Deploynix is here with a different perspective on what Laravel deployment should look like in 2026.

This isn’t a takedown piece. Forge and Ploi are good products that have served the Laravel community well. But developers deserve an honest comparison so they can pick the tool that fits their specific needs. Let’s walk through the features that matter most.

Cloud Provider Support

The number of supported cloud providers determines how much flexibility you have in choosing where your servers run.

Laravel Forge supports DigitalOcean, Linode, Vultr, AWS, Hetzner, and custom servers via IP. Forge also offers Laravel VPS — their own managed servers starting at $6/month — so you can provision directly through Forge without a separate cloud provider account. It’s a solid lineup that covers most needs.

Ploi supports DigitalOcean, UpCloud, Vultr, Linode, Scaleway, AWS EC2, and custom servers. A broad selection with some unique providers like UpCloud and Scaleway that other platforms don’t offer.

Deploynix supports DigitalOcean, Vultr, Hetzner, Linode, AWS, and custom servers. Six providers with a uniform provisioning experience across all of them.

The practical difference here isn’t just the list — it’s how custom servers are treated. On Deploynix, a custom server (your own bare metal, a provider we don’t directly integrate with) gets the same management experience as a provider-provisioned server. The same monitoring, the same deployment pipeline, the same interface. This matters if you’re running on infrastructure that isn’t covered by any platform’s direct integrations.

Server Types and Architecture

How a platform thinks about server types reveals how it thinks about application architecture.

Laravel Forge offers App Servers, Web Servers, Database Servers, Cache Servers, and Worker Servers. It’s a reasonable set that covers common architectures.

Ploi offers Web Servers, Load Balancers, Database Servers, Redis Servers, and Storage Servers. Ploi also supports load balancing, giving teams flexibility in how they distribute traffic.

Deploynix offers seven distinct server types: App, Web, Database, Cache, Worker, Meilisearch, and Load Balancer. Each type is configured specifically for its purpose. The inclusion of a dedicated Meilisearch server type reflects the growing adoption of Meilisearch in the Laravel ecosystem thanks to Laravel Scout. The Load Balancer type supports Round Robin, Least Connections, and IP Hash methods, making it straightforward to distribute traffic across multiple application servers.

If your architecture is a single server running everything, all three platforms work fine. But as you scale and separate concerns, having purpose-built server types reduces the configuration overhead. Deploynix’s Meilisearch server type is unique among the three platforms.

Zero-Downtime Deployments

This is where the platforms diverge most significantly.

Laravel Forge added zero-downtime deployments over time. The implementation works, but it’s an opt-in feature rather than a default behavior. You configure it per site.

Ploi also supports zero-downtime deployments with a similar opt-in approach.

Deploynix makes zero-downtime deployment the default for every site. Atomic symlink deployments, shared storage handling, PHP-FPM graceful reloads, and automatic release cleanup are built into the standard deployment pipeline. Rollback is a single click — the symlink points back to any previous release instantly.

The philosophical difference matters. When zero-downtime is the default, every feature built around deployments assumes it. Scheduled deployments, deployment hooks, rollback — they all work with the atomic deployment model as the foundation.

Deployment Features

Beyond the basic deploy, what tooling exists around the deployment process?

Laravel Forge offers deployment scripts, quick deploy (auto-deploy on push), deployment triggers via URL, and deploy notifications. It’s battle-tested and reliable.

Ploi provides similar deployment tooling with deployment scripts, auto-deploy, and notifications.

Deploynix includes customizable deployment scripts, auto-deploy on push, scheduled deployments, instant rollback, and real-time deployment logs via WebSockets. The scheduled deployment feature is worth highlighting — you can schedule a deploy for a future time and cancel it before execution. This is useful for coordinated releases, off-peak deployments, and team workflows where the person writing code isn’t always the person deciding when it goes live.

Real-time deployment logs via WebSocket connections (built on Laravel Reverb) mean you see output as it happens, not after a page refresh. It’s a small UX difference that significantly improves the deployment experience.

Database Support

Laravel Forge supports MySQL, MariaDB, and PostgreSQL, along with SQLite for simpler use cases.

Ploi supports MySQL, MariaDB, and PostgreSQL.

Deploynix supports MySQL, MariaDB, and PostgreSQL on dedicated Database server types. Database backup is integrated directly into the platform with support for AWS S3, DigitalOcean Spaces, Wasabi, and any custom S3-compatible storage. Backup scheduling, monitoring, and cross-server restores are built in.

The backup integration is a meaningful differentiator. With other platforms, you often end up configuring a separate backup solution — a third-party service, a custom script, or a Laravel package. Deploynix treats backups as a core platform feature with a monitoring dashboard so you can verify backups are running successfully.

Caching

Laravel Forge supports Redis installation on servers.

Ploi similarly supports Redis.

Deploynix supports Valkey, the Redis-compatible fork that emerged after Redis changed its licensing. Valkey is protocol-compatible with Redis, so your Laravel application’s cache, session, and queue configurations work without code changes. This choice reflects a preference for truly open-source software.

SSL and Domain Management

Laravel Forge handles Let’s Encrypt certificate provisioning and renewal. It supports custom certificates and has Cloudflare integration.

Ploi provides Let’s Encrypt automation and custom certificate support.

Deploynix auto-provisions Let’s Encrypt certificates including wildcard certificates via DNS validation. DNS validation is supported through Cloudflare, DigitalOcean, AWS Route 53, and Vultr. Every site automatically gets a *.deploynix.cloud vanity subdomain with SSL included — perfect for staging environments and client previews without any domain or DNS configuration.

The vanity domain feature is particularly useful during development. Spin up a site, share the your-project.deploynix.cloud URL with a client for review, and move to a production domain when ready. No DNS configuration, no certificate provisioning, no waiting.

Team Management and Collaboration

Laravel Forge offers organization-based team management with role-based access controls, allowing you to share servers and sites with team members and manage billing per organization.

Ploi provides team management with similar role-based access.

Deploynix implements organization-based team management with five distinct roles: Owner, Admin, Manager, Developer, and Viewer. Each role has specific permission boundaries. A Developer can trigger deployments and view logs but can’t modify server configuration or firewall rules. A Manager can manage sites and deployments but can’t provision new servers. A Viewer gets read-only access — ideal for stakeholders who need visibility without risk. This granularity matters in professional environments where the principle of least privilege isn’t optional.

The organization model also affects billing — plans are per-organization, so you can have separate billing for different clients or projects.

Monitoring and Alerting

Laravel Forge has added server monitoring capabilities over time, including resource usage graphs and some alerting features.

Ploi provides server monitoring with resource usage tracking.

Deploynix builds monitoring directly into the platform with real-time CPU, memory, and disk metrics. Health alerts fire based on configurable thresholds with severity levels (critical and warning). The monitoring agent runs on your server and reports metrics back to the platform via WebSocket for live updates. You see current server health on the dashboard without refreshing.

The alert system is designed to catch problems before they become incidents. Set a warning at 80% disk usage and a critical alert at 90%. Get notified when CPU consistently spikes above your threshold. The goal is to fix problems before your users encounter them.

API and Automation

Laravel Forge has a comprehensive API that covers most platform operations. It’s well-documented and widely used in CI/CD pipelines.

Ploi provides an API for automation and CI/CD integration.

Deploynix offers a Sanctum-based API with granular token scopes. When you create an API token, you specify exactly what it can do — manage servers, trigger deployments, view monitoring data, manage team members. A CI/CD token that only needs to trigger deployments doesn’t need permission to modify firewall rules. This follows security best practices for API token management.

Web Terminal

Laravel Forge does not include a web-based terminal — you SSH into servers using your own terminal and keys.

Ploi offers a web-based terminal for server access.

Deploynix includes a web terminal with authentication tied to the platform’s role system. Access is audited and controlled by team permissions. This is especially useful for teams where not everyone should have direct SSH key access to production servers.

Octane Support

Laravel Forge supports Laravel Octane deployment.

Ploi supports Octane as well.

Deploynix supports Octane with a choice of three drivers: FrankenPHP, Swoole, and RoadRunner. You select your driver during site setup and the server configuration, process management, and deployment pipeline adjust accordingly. Graceful restarts during deployment ensure Octane workers pick up new code without dropping requests.

Git Provider Integration

Laravel Forge integrates with GitHub, GitLab, and Bitbucket.

Ploi supports GitHub, GitLab, and Bitbucket.

Deploynix supports GitHub, GitLab, Bitbucket, and custom Git repositories. The custom Git option means you can deploy from a self-hosted Gitea instance, a corporate GitLab installation, or any Git repository accessible via SSH.

Pricing

Pricing changes frequently, so rather than quoting specific numbers that may be outdated, here’s the structural comparison.

Laravel Forge uses flat-rate monthly pricing with three tiers — Hobby, Growth, and Business — based on features, server limits, and team access. Forge also offers Laravel VPS starting at $6/month for those who want managed hosting without a separate cloud provider.

Ploi has made Ploi Core completely free, with paid tiers scaling up for teams and advanced features. This makes Ploi one of the most accessible entry points for server management.

Deploynix has four tiers — Free, Starter, Professional, and Enterprise — billed per organization through Paddle. The Free tier is genuinely usable, not a trial. Core features like zero-downtime deployments and monitoring are available across all tiers. Higher tiers unlock additional servers, team members, and advanced features.

When to Choose Each Platform

Choose Laravel Forge if you want the most mature platform with the largest community. Forge has been around the longest, has the most tutorials and third-party integrations, and benefits from being closely tied to the Laravel ecosystem’s creator. The addition of Laravel VPS and support for Node.js, Nuxt, and Next.js apps makes Forge increasingly versatile. If stability and ecosystem size matter most, Forge is a safe choice.

Choose Ploi if you want a cost-effective alternative with a free core tier and a modern interface. Ploi has carved out a solid niche by offering good value, responsive development, and unique provider support like UpCloud and Scaleway.

Choose Deploynix if you want zero-downtime deployments as a default, granular team permissions, integrated monitoring with real-time alerts, scheduled deployments, and a platform designed specifically for modern Laravel workflows. If you run multi-cloud infrastructure, need purpose-built server types including load balancers, or want a platform that treats backups and monitoring as first-class features rather than add-ons, Deploynix is built for those use cases.

The Honest Truth

No platform is perfect. Forge has the advantage of maturity, ecosystem integration, and its own managed VPS offering. Ploi has the advantage of a free core tier, unique provider support, and a responsive team. Deploynix has the advantage of a fresh architecture built on modern Laravel with features designed for 2026 workflows.

The best way to decide is to try them. Each platform offers a way to get started without significant commitment. Spin up a test server, deploy an application, and see which workflow fits the way you and your team actually work.

We’re confident that once you try Deploynix, you’ll appreciate the difference that comes from building a platform from scratch with today’s Laravel development practices in mind. But we’d rather earn your trust through the product than through a comparison article.

Visit https://deploynix.io to start your free account.

vLLM On-Demand Gateway: Zero-VRAM Standby for Local LLMs on Consumer GPUs

The Problem: vLLM Hogs Your GPU 24/7

If you run a local LLM with vLLM, you know the pain. The moment you start the server, it claims ~90% of your VRAM and never lets go — even when nobody’s asking it anything.

On a dedicated inference server, that’s fine. But on a single consumer GPU (RTX 5090 in my case), I also need VRAM for:

  • Shogi engine (DL-based, needs ~4GB VRAM)
  • Whisper transcription (large-v3, GPU-accelerated)
  • Training runs, experiments, occasional gaming

Running vLLM permanently means everything else fights for scraps. Killing and restarting vLLM manually every time is not a workflow — it’s a chore.

The Solution: A Gateway That Manages vLLM’s Lifecycle

I wrote a single-file FastAPI gateway (vllm_gateway.py, ~390 lines) that:

  1. Listens on port 8000 with near-zero VRAM usage
  2. Auto-starts vLLM on an internal port (8100) when a request arrives
  3. Auto-stops vLLM after 10 minutes of idle, fully freeing VRAM
  4. Rewrites tool calls from Nemotron’s <TOOLCALL> format to OpenAI-compatible tool_calls

From the client’s perspective, it’s just a normal OpenAI-compatible API on port 8000. The lifecycle management is completely invisible.

Client → :8000 (Gateway, always running, ~0 VRAM)
              ↓ proxy
           :8100 (vLLM, started on-demand, stopped when idle)

Architecture

Startup Flow

Request arrives at :8000
  → Gateway checks: is vLLM running?
    → No: spawn vLLM process on :8100
          poll /health every 2s (up to 3 min timeout)
          once healthy → proxy the request
    → Yes: proxy immediately

Shutdown Flow

Idle watchdog runs every 30s
  → Last request was >10 min ago?
    → SIGTERM to vLLM process group
    → Wait 15s, SIGKILL if needed
    → VRAM fully released

Key Design Decisions

  • Process group kill (os.killpg): vLLM spawns child processes. Killing just the parent leaves zombies holding VRAM.
  • Internal port separation: Gateway owns :8000. vLLM gets :8100. No port conflicts during restart.
  • Health check polling: Don’t proxy until vLLM is actually ready. Model loading takes 30-90s depending on size.

Core Implementation

Here’s the stripped-down version of the essential parts:

VLLM_INTERNAL_PORT = 8100
GATEWAY_PORT = 8000
IDLE_TIMEOUT_SECONDS = 10 * 60

VLLM_CMD = [
    ".venv/bin/vllm", "serve", "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese",
    "--trust-remote-code",
    "--port", str(VLLM_INTERNAL_PORT),
    "--enable-auto-tool-choice",
    "--tool-call-parser", "nemotron_json",
]

async def start_vllm() -> bool:
    global vllm_process, vllm_ready
    vllm_process = subprocess.Popen(
        VLLM_CMD, preexec_fn=os.setsid,
        stdout=subprocess.DEVNULL, stderr=subprocess.PIPE,
    )
    # Poll health endpoint until ready
    deadline = time.time() + 180
    async with httpx.AsyncClient() as client:
        while time.time() < deadline:
            try:
                resp = await client.get(
                    f"http://localhost:{VLLM_INTERNAL_PORT}/health", timeout=5
                )
                if resp.status_code == 200:
                    vllm_ready = True
                    return True
            except (httpx.ConnectError, httpx.ReadTimeout):
                pass
            await asyncio.sleep(2)
    return False

async def stop_vllm():
    global vllm_process, vllm_ready
    if vllm_process and vllm_process.poll() is None:
        os.killpg(os.getpgid(vllm_process.pid), signal.SIGTERM)
        try:
            vllm_process.wait(timeout=15)
        except subprocess.TimeoutExpired:
            os.killpg(os.getpgid(vllm_process.pid), signal.SIGKILL)
    vllm_process = None
    vllm_ready = False

async def idle_watchdog():
    while True:
        await asyncio.sleep(30)
        if vllm_ready and time.time() - last_request_time > IDLE_TIMEOUT_SECONDS:
            await stop_vllm()

Tool Call Rewriting (Bonus)

Nemotron outputs tool calls as raw text:

<TOOLCALL>[{"name": "ddg_search", "arguments": {"query": "NVIDIA stock"}}]</TOOLCALL>

The gateway intercepts this and rewrites it to the OpenAI format:

{
  "tool_calls": [{
    "id": "call_abc123",
    "type": "function",
    "function": {
      "name": "ddg_search",
      "arguments": "{"query": "NVIDIA stock"}"
    }
  }],
  "finish_reason": "tool_calls"
}

This works for both streaming and non-streaming responses. Clients never see the raw <TOOLCALL> tags.

Status Endpoint

The gateway exposes a simple status API:

$ curl localhost:8000/gateway/status
{
  "vllm_running": false,
  "vllm_ready": false,
  "idle_seconds": 7145,
  "pid": null
}

And manual controls:

curl -X POST localhost:8000/gateway/start   # Force start
curl -X POST localhost:8000/gateway/stop    # Force stop

Real-World Numbers (RTX 5090, Nemotron 9B)

State VRAM Usage
Gateway only (vLLM stopped) ~200 MB
vLLM running (Nemotron 9B FP16) ~22 GB
Cold start time ~60s

The 60s cold start is the tradeoff. For interactive chat, the first message after idle has a delay. For batch/API workloads, it’s negligible.

When You’d Want This

  • Single GPU, multiple workloads: Share your GPU between LLM inference and other tasks
  • Development machine: Run vLLM only when you’re actively using it
  • Cost/power savings: No point heating your GPU for an idle model
  • Home server: The RTX card can serve LLM requests AND run other GPU tasks

When You Wouldn’t

  • Dedicated inference server: Just run vLLM directly
  • Low-latency requirements: 60s cold start is unacceptable for some use cases
  • Multi-user serving: Frequent requests mean vLLM stays up anyway

Full Source

The complete vllm_gateway.py is ~390 lines including streaming support and TOOLCALL rewriting. The approach works with any vLLM model — just change VLLM_CMD.

Dependencies: fastapi, httpx, uvicorn (all likely already installed if you use vLLM).

Running Nemotron Nano 9B v2 Japanese on RTX 5090 + WSL2. The gateway pattern turned “I can’t use my GPU for anything else” into “vLLM is there when I need it and gone when I don’t.”

Building Framework-Agnostic AI Swarms: Compare LangGraph, Strands, and OpenAI Swarm

LangGraph swarm architecture showing LaunchDarkly configuration fetch, agent interactions with Command-based handoffs, and dual metrics tracking to AI Config Trends

If you’ve ever run the same app in multiple environments, you know the pain of duplicated configuration. Agent swarms have the same problem: the moment you try multiple orchestrators (LangGraph, Strands, OpenAI Swarm), your agent definitions start living in different formats. Prompts drift. Model settings drift. A “small behavior tweak” turns into archaeology across repos.

AI behavior isn’t code. Prompts aren’t functions. They change too often, and too experimentally, to be hard-wired into orchestrator code. LaunchDarkly AI Configs lets you treat agent definitions like shared configuration instead. Define them once, store them centrally, and let any orchestrator fetch them. Update a prompt or model setting in the LaunchDarkly UI, and the new version rolls out without a redeploy.

Ready to build framework-agnostic AI swarms? Start your 14-day free trial of LaunchDarkly to follow along with this tutorial. No credit card required.

Start free trial →

The problem: Research gap analysis across multiple papers

When analyzing academic literature, researchers face a daunting task: reading dozens of papers to identify patterns, spot contradictions, and find unexplored opportunities. A single LLM call can summarize papers, but it produces a monolithic analysis you can’t trace, refine, or trust for critical decisions.

The challenge compounds when you need to:

  • Identify methodological patterns across 12+ papers without missing subtle connections
  • Detect contradictory findings that might invalidate assumptions
  • Discover research gaps that represent genuine opportunities, not just oversight

This is where specialized agents excel – each focused on one aspect of the analysis, building on each other’s work.

In this tutorial, we’ll build a 3-agent research analysis swarm that solves this problem by dividing the work:

Agent Role Output
Approach Analyzer Clusters methodological themes across papers “Papers 1, 4, 7 use reinforcement learning; Papers 2, 5 use symbolic methods”
Contradiction Detector Finds conflicting claims between papers “Paper 3 claims X improves performance; Paper 8 shows X degrades it”
Gap Synthesizer Identifies unexplored research directions “No papers combine approach A with dataset B; potential opportunity”

We’ll implement this swarm across three different orchestrators (LangGraph, Strands, and OpenAI Swarm), demonstrating how LaunchDarkly AI Configs enable:

  • Framework-agnostic agent definitions: Define agents once in LaunchDarkly, use them everywhere
  • Per-agent observability: Track tokens, latency, and costs for each agent individually – catch silent failures when agents skip execution
  • Dynamic swarm composition: Add/remove agents from the swarm or switch models without touching code

Why use a swarm?

Research gap analysis requires different skills: clustering methodological patterns, detecting contradictions, and synthesizing opportunities. With a swarm, each agent handles one aspect and produces artifacts the next agent builds on. You can track tokens, latency, and cost per agent. You can catch silent failures when an agent skips execution. And when something goes wrong, you know exactly where.

Technical requirements

Before implementing the swarm, ensure you have:

The complete implementation is available at GitHub – AI Orchestrators.

The architecture: how LaunchDarkly powers framework-agnostic swarms

The swarm architecture has three layers: dynamic agent configuration, per-agent tracking, and custom metrics for cost attribution. Here’s how they work together.

LangGraph swarm architecture showing LaunchDarkly configuration fetch, agent interactions with Command-based handoffs, and dual metrics tracking to AI Config Trends

The diagram shows LangGraph’s implementation, but Strands and OpenAI Swarm follow the same pattern with their own handoff mechanisms. The key components are:

  1. Configuration Fetch: The orchestrator queries LaunchDarkly’s API to dynamically discover all agent configurations, avoiding hardcoded agent definitions
  2. Agent Graph: Three specialized agents (Approach Analyzer, Contradiction Detector, Gap Synthesizer) connected through explicit handoff mechanisms
  3. Metrics Collection: Each agent execution captures tokens, duration, and cost metrics through both the AI Config tracker and custom metrics API
  4. Dual Dashboard Views: The same metrics appear in the AI Config Trends dashboard (for individual agent monitoring)

Three layers of framework-agnostic swarms

1. AI Config for Dynamic Agent Configuration

Each AI Config stores:

  • Agent key, display name, and model selection
  • System instructions and tool definitions

Your orchestrator code queries LaunchDarkly for “all enabled agent configs” and builds the swarm dynamically. No hardcoded agent names.

2. Per-Agent Tracking with AI SDK

LaunchDarkly’s AI SDK provides tracking through config evaluations. You get a fresh tracker for each agent, then track tokens, duration, and success/failure. These metrics flow to the AI Config Monitoring dashboard automatically.

AI Config monitoring dashboard showing per-agent token usage, duration, and success rates across multiple runs

This tracking catches silent failures – when agents skip execution or produce minimal output. Step 4 shows the implementation patterns for each framework.

3. Custom Metrics for Cost Attribution

Per-agent tracking shows performance, but for cost comparisons across orchestrators you need custom metrics. These let you query by orchestrator, compare costs across frameworks, and identify anomalies.

With the architecture covered, let’s build the swarm. We’ll download research papers, set up the project, bootstrap agent configs in LaunchDarkly, implement per-agent tracking, and run the swarm across all three orchestrators.

Step 1: Download research papers

First, you need papers to analyze. The scripts/download_papers.py script queries ArXiv with narrow, category-specific searches to ensure focused results.

python scripts/download_papers.py

The script presents pre-configured narrow research topics:

# From orchestration/scripts/download_papers.py:164-189
topics = {
    "1": {
        "name": "Chain-of-thought prompting in LLMs",
        "query": "cat:cs.CL AND (chain-of-thought OR CoT) AND reasoning",
        "years": 2
    },
    "2": {
        "name": "Retrieval-augmented generation (RAG)",
        "query": "cat:cs.CL AND (retrieval-augmented OR RAG) AND generation",
        "years": 2
    },
    "3": {
        "name": "Emergent communication in multi-agent RL",
        "query": "cat:cs.MA AND (emergent communication OR language emergence)",
        "years": 5
    },
    "4": {
        "name": "Few-shot prompting for code generation",
        "query": "cat:cs.SE AND few-shot AND code generation",
        "years": 2
    },
    "5": {
        "name": "Vision-language model grounding",
        "query": "cat:cs.CV AND vision-language AND grounding",
        "years": 2
    }
}

These topics are intentionally narrow: Each uses ArXiv categories (cat:cs.CL, cat:cs.MA) to limit scope. Boolean AND operators ensure papers match all criteria. 2-5 year windows prevent overwhelming the analysis.

For even narrower custom queries, combine categories with specific techniques like cat:cs.CL AND chain-of-thought AND mathematical AND reasoning for CoT math only, cat:cs.MA AND emergent AND (referential OR compositional) for specific emergence types, or cat:cs.SE AND few-shot AND (Python OR JavaScript) AND test generation for language-specific code generation.

The script saves papers to data/gap_analysis_papers.json with this structure:

[
  {
    "id": "2409.02645v2",
    "title": "Emergent Language: A Survey and Taxonomy",
    "authors": "Jannik Peters, Constantin Waubert de Puiseau, ...",
    "published": "2024-09-04",
    "category": "cs.MA",
    "abstract": "The field of emergent language represents...",
    "introduction": "Language emergence has been explored...",
    "conclusion": "This paper provides a comprehensive review..."
  }
]

Why this format: Each paper includes ~2-3K characters of text (abstract + intro + conclusion), which is enough for analysis but won’t overflow context windows. For 12 papers, you’re looking at ~30K characters (~7.5K tokens) of input.

You now have 12 papers saved locally. Next, we’ll configure LaunchDarkly credentials and install the orchestration frameworks.

Step 2: Set up your multi-orchestrator project

Environment setup

For help getting your SDK and API keys, see the API access tokens guide and SDK key management.

# .env file
LD_SDK_KEY=sdk-xxxxx       # Get from LaunchDarkly project settings
LD_API_KEY=api-xxxxx       # Create at Account settings → Authorization
LAUNCHDARKLY_PROJECT_KEY=orchestrator-agents

# Model API keys
ANTHROPIC_API_KEY=sk-ant-xxxxx
OPENAI_API_KEY=sk-xxxxx

Install dependencies

python -m venv .venv
source .venv/bin/activate

# LaunchDarkly SDKs - see [Python SDK docs](/sdk/server-side/python)
pip install ldai ldclient python-dotenv arxiv PyPDF2 requests

# Orchestration frameworks
pip install strands-sdk langgraph swarm

For more on the LaunchDarkly AI SDK, see the AI SDK documentation.

Your environment is configured and dependencies are installed. Next, we’ll use the bootstrap script to automatically create all three agent configs in LaunchDarkly.

Step 3: Bootstrap agent configs with the manifest

The orchestration repo includes a complete bootstrap system that automatically creates all agent configurations, tools, and variations in LaunchDarkly. This is much faster and more reliable than manual setup.

Understanding the bootstrap system

The bootstrap process uses a YAML manifest to define:

  1. Tools – Functions agents can call (fetch_paper_section, handoff_to_agent, etc.)
  2. Agent Configs – Three specialized agents with their roles and instructions
  3. Variations – Multiple model options (Anthropic Claude vs OpenAI GPT)
  4. Targeting Rules – Which orchestrators get which models

Run the bootstrap script

# From the orchestration repo root
cd ai-orchestrators

# Run bootstrap with the research gap manifest
python scripts/launchdarkly/bootstrap.py

# You'll see:
╔═══════════════════════════════════════════════════════╗
║  AI Agent Orchestrator - LaunchDarkly Bootstrap       ║
╚═══════════════════════════════════════════════════════╝

Available manifests:
  1. Research Gap Analysis (research_gap_manifest.yaml)

Select manifest or press Enter for default: [Enter]

📦 Project: orchestrator-agents
🌍 Environment: production

🛠️  Creating paper analysis tools...
    ✓ Tool 'extract_key_sections' created
    ✓ Tool 'fetch_paper_section' created
    ✓ Tool 'handoff_to_agent' created
    ...

🤖 Creating AI agent configs...
    ✓ AI Config 'approach-analyzer' created
    ✓ AI Config 'contradiction-detector' created
    ✓ AI Config 'gap-synthesizer' created

✨ Bootstrap complete!

What gets created

The bootstrap script creates the three agents described earlier (Approach Analyzer, Contradiction Detector, Gap Synthesizer), each with swarm-aware instructions and handoff tools.

Verify in LaunchDarkly dashboard

After bootstrap completes:

  1. Go to your LaunchDarkly AI Configs dashboard at https://app.launchdarkly.com/<your-project-key>/<your-environment-key>/ai-configs
  2. You’ll see all three agent configs created
  3. Each config has:

How variations and targeting work

Each agent has two variations in the manifest:

# Example from approach-analyzer agent
variations:
  - key: "analyzer-claude"
    name: "Approach Analyzer Claude"
    modelConfig:
      provider: "anthropic"
      modelId: "claude-sonnet-4-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Agent instructions here]

  - key: "analyzer-openai"
    name: "Approach Analyzer OpenAI"
    modelConfig:
      provider: "openai"
      modelId: "gpt-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Same instructions, different model]

targeting:
  rules:
    - variation: "analyzer-openai"
      clauses:
        - attribute: "orchestrator"
          op: "in"
          values: ["openai_swarm", "openai-swarm"]
  defaultVariation: "analyzer-claude"

When an orchestrator requests this agent:

  1. Context includes orchestrator attribute: context = create_context(execution_id, orchestrator="openai_swarm")
  2. LaunchDarkly evaluates targeting rules: If orchestrator is “openai_swarm” or “openai-swarm”, use OpenAI variation
  3. Otherwise use default: Claude variation for all other orchestrators

This lets you:

  • Use OpenAI models when running OpenAI Swarm (native compatibility)
  • Use Claude for other orchestrators
  • A/B test models by adjusting targeting rules

Customize agent behavior

After bootstrap, you can adjust agents in the LaunchDarkly UI without code changes. Switch between Claude, GPT-4, or other supported providers. Refine instructions for better handoffs. Control which agents are included in the swarm through targeting rules. Test different prompts or models side-by-side with experiments.

Your three agents are now configured in LaunchDarkly. Next, we’ll implement tracking so you can monitor tokens, latency, and cost for each agent individually.

Step 4: Implement per-agent tracking

The orchestration repository demonstrates per-agent tracking across all three frameworks. First, you need to fetch agent configurations from LaunchDarkly:

Fetching agent configurations dynamically

from shared.launchdarkly import (
    init_launchdarkly_clients,
    fetch_agent_configs_from_api,
    create_context,
    build_agent_requests
)

# Initialize LaunchDarkly clients
ld_client, ai_client = init_launchdarkly_clients()

# Fetch agent list from LaunchDarkly API (not hardcoded!)
items = fetch_agent_configs_from_api()
print(f"Found {len(items)} AI config(s) in LaunchDarkly")

# Create execution context
execution_id = f"langgraph-{datetime.now().strftime('%Y%m%d_%H%M%S')}"
context = create_context(execution_id, orchestrator="langgraph")

# Build requests for all agents
agent_requests, agent_metadata = build_agent_requests(items)

# Fetch all configs in one call
configs = ai_client.agent_configs(agent_requests, context)

# Process agents with configured variations
enabled_agents = []
for item in items:
    config = configs.get(item["key"])
    if config and config.enabled:
        enabled_agents.append({
            "key": item["key"],
            "name": item["name"],
            "config": config,
            "model": config.model.name if config.model else "claude-sonnet-4-5"
        })

print(f"✓ Found {len(enabled_agents)} configured agent configs")

Pattern 1: Native framework metrics (Strands)

Strands provides accumulated_usage on each node result after execution:

# From orchestrators/strands/run_gap_analysis.py:418-424
if agent_key in per_agent_metrics:
    usage = node_result.accumulated_usage or {}
    input_tokens, output_tokens = extract_usage_tokens(usage)
    total_tokens = input_tokens + output_tokens

View full Strands implementation

Pattern 2: Message-based tracking (LangGraph)

LangGraph attaches usage_metadata to messages, requiring post-execution iteration:

# From orchestrators/langgraph/run_gap_analysis.py:442-446
if hasattr(msg, "usage_metadata") and msg.usage_metadata:
    usage_data = msg.usage_metadata
    input_tokens = usage_data.get("input_tokens", 0) or usage_data.get("prompt_tokens", 0)
    output_tokens = usage_data.get("output_tokens", 0) or usage_data.get("completion_tokens", 0)
    has_usage = True

View full LangGraph implementation

Pattern 3: Interception-based tracking (OpenAI Swarm)

OpenAI Swarm doesn’t aggregate per-agent metrics, requiring interception of completion calls:

# From orchestrators/openai_swarm/run_gap_analysis.py:369-387
original_get_chat_completion = client.get_chat_completion

def tracked_get_chat_completion(agent, history, context_variables, model_override, stream, debug):
    start_call = time.time()
    completion = original_get_chat_completion(
        agent=agent,
        history=history,
        context_variables=context_variables,
        model_override=model_override,
        stream=stream,
        debug=debug,
    )
    duration = time.time() - start_call
    agent_key = key_by_name.get(agent.name, agent.name)
    usage = getattr(completion, "usage", None)
    if usage:
        input_tokens = int(getattr(usage, "prompt_tokens", 0))
        output_tokens = int(getattr(usage, "completion_tokens", 0))
        total_tokens = int(getattr(usage, "total_tokens", input_tokens + output_tokens))

View full OpenAI Swarm implementation

Critical: Provider token field names differ

Each provider uses different field names: Anthropic uses input_tokens/output_tokens, OpenAI uses prompt_tokens/completion_tokens, and some frameworks use camelCase (inputTokens). The implementations use fallback chains to handle all formats.

You can now capture tokens, latency, and cost for each agent. Next, we’ll run the swarm across LangGraph, Strands, and OpenAI Swarm to see how they perform with the same agent definitions.

Step 5: Run multiple orchestrators and track results

The repository includes scripts to run all three orchestrators and analyze their performance:

# Run all orchestrators 5 times each
./scripts/run_swarm_benchmark.sh sequential 5

# Analyze the results
python scripts/analyze_benchmark_results.py
  1. Configure env: Create .env with SDK keys
  2. Install deps: pip install -r requirements.txt
  3. Download papers: python scripts/download_papers.py
  4. Bootstrap agents: python scripts/launchdarkly/bootstrap.py
  5. Configure targeting: Set default variation for each agent in LaunchDarkly UI
  6. Test run: python orchestrators/strands/run_gap_analysis.py

Troubleshooting: If you see “No enabled agents found,” check that each agent has a default variation set in the Targeting tab.

Now that you’ve run the swarm across all three orchestrators, let’s look at how they differ in approach and performance.

Comparing orchestrator approaches to swarms

All three frameworks support multi-agent workflows, they just disagree on who decides what happens next.

Key differences

Aspect Strands LangGraph OpenAI Swarm
Routing Framework-managed Graph-based Function return
Handoff API Tool call (automatic) Command object Return Agent object
Boilerplate Low Medium Medium
Control Low (black box) High (explicit graph) High (manual impl)
Debugging Hard (why didn’t agent run?) Easy (graph trace) Hard (silent failures)
Per-Agent Metrics Built-in Wrapper required Interception required

View full implementations: Strands | LangGraph | OpenAI Swarm

The LaunchDarkly advantage: By defining agents externally, you can implement swarms across all three frameworks and compare their approaches with the same agent definitions.

Performance comparison (9 runs: 3 datasets × 3 orchestrators)

Metric OpenAI Swarm Strands LangGraph
Avg Time 2.9 min 5.7 min 8.0 min
Tokens 67K 99K 89K
Speed 385 tok/s 287 tok/s 186 tok/s
Report Size 13KB 32KB 67KB
Variance ±1.05 min ±1.38 min ±0.21 min

Key insight (based on limited sample): Fastest ≠ best. OpenAI Swarm was 3x faster but produced reports 80% smaller than LangGraph. LangGraph had the lowest variance and most comprehensive outputs despite slower execution.

Performance comparison graphs showing execution time, token usage, and processing speed across all three orchestrators

Example reports: See the outputs

Report size variation demonstrates why per-agent tracking matters – you need to know when agents produce minimal output.

Conclusion

The orchestrator you choose determines how agents coordinate, but it shouldn’t lock you into a single framework. By defining agents in LaunchDarkly and fetching them at runtime, you can run the same swarm across LangGraph, Strands, and OpenAI Swarm without duplicating configuration or watching prompts drift between repos.

The performance differences are real. OpenAI Swarm is fastest, LangGraph produces the most comprehensive outputs, and Strands offers the simplest setup. But you only discover these tradeoffs if you can track each agent individually and catch silent failures when they happen.

Swarms cost more than single LLM calls. The payoff is traceable reasoning you can audit, refine, and trust.

The full implementation is available on GitHub – AI Orchestrators. Clone the repo and run the same swarm across all three orchestrators. To get started with LaunchDarkly AI Configs, follow the quickstart guide.

Message-Driven architecture vs REST-APIs

A question I often get: “Do you use message-driven architecture just because REST APIs are slow?” – Well… yes, but actually no. There’s much more behind that decision.

Why use Message-Driven architecture

Cross-device support

The messaging systems allow for real-time communication across servers, desktops, web & mobile apps — all decoupled and event-driven.

Asynchronous communication

REST API forces the sender to wait. Messaging lets you offload heavy work to background consumers while the app stays responsive.

Supports monoliths

The producer-consumer architecture can be used within a single service using in-process events.

Multi-destination delivery

With REST APIs, the sender needs to know the exact endpoint of each receiver. Messaging system publishes to a topic — whoever’s subscribed gets the message. No need to know who or where they are.

Retention

REST API fails if the receiving server is down. Messaging brokers store messages until consumers reconnect. With TTL, you can control how long unconsumed messages are retained before they are discarded.

Fault tolerance

Retries and dead‑letter queues are ideal for systems that favor built‑in recovery mechanisms to reduce the need for manual intervention.

But is messaging suitable for every backend?
No. You should also be aware of the drawbacks.

Drawbacks of MDA

Extra complexity

You need to manage brokers, consumers, retry policies, and observability.

Tracking requests is harder

Messages can fan out to multiple destinations. Tracing who picked up what, when, and what happened next is a challenge.

And because messaging is inherently push‑based…

No immediate response

Publishers don’t wait for a response; the server can’t guarantee instant feedback to user actions.

Eventual consistency

Messaging systems can always send requests, but the downed services need to catch up later. This is not ideal for systems that require strict transactional processes.

Idempotency

The retry mechanism may redeliver the same message when a consumer fails or times out. The consumers must be idempotent to avoid processing the same request multiple times.
Idempotency doesn’t spare REST APIs either.

Every software architecture is a trade‑off. Which direction do you lean into?

If you’re curious to dive deeper into Messaging or Event‑Driven Architecture, I’ve written a handful of articles on the topic.

How to See What Your OpenClaw AI Assistant Actually Costs Per Conversation

I’ve been running OpenClaw for a few months — it’s become my daily AI assistant across WhatsApp and Telegram, handling emails, research, calendar stuff. It’s genuinely great.

But at the end of month one, I opened my Anthropic billing dashboard and saw $43.

I had no idea where it came from. Which conversations? Which agent? The long research session, or just daily chit-chat? No clue.

This is a known issue in the OpenClaw community — there are open feature requests for native token tracking and a CLI usage command that haven’t shipped yet. So I went looking for a workaround.

The Problem With API Billing Dashboards

The Anthropic and OpenAI dashboards show your total spend, but they’re aggregated (no per-conversation breakdown), delayed (often 24+ hours behind), and only model-level (you can see “Claude Sonnet cost $31” but not which feature or session drove that).

If you’re running a personal AI assistant that touches multiple models — Anthropic for complex tasks, a local Ollama model for simple ones — you have zero visibility into what’s costing money vs what’s free.

The Fix: One Import

burn0 is a tiny Node.js library that solves this. It patches fetch and node:http at the runtime level, so it sees every outbound API call your app makes.

npm install @burn0/burn0

Then at the top of your entry file:

import "@burn0/burn0";

Now every API call to Anthropic, OpenAI, Ollama, or any of 50+ other services gets intercepted, and you see real-time cost breakdowns in your terminal:

burn0 > $0.04 today (12 calls) -- anthropic: $0.031 | openai: $0.009

Why This Works Well With OpenClaw

OpenClaw is Node.js — burn0 slots right in.

It tracks local models too. OpenClaw supports Ollama and other local models. burn0 shows these as $0.00, so you can see the real dollar savings from routing locally vs cloud.

It reads actual token counts, not estimates. Token counts come directly from each API response’s metadata — exact numbers, not guesses based on character counts.

Zero changes to OpenClaw. Works at the HTTP layer, no fork needed. When native tracking ships, remove the import.

Privacy-first. Runs entirely locally. Never reads request or response bodies — only metadata. Nothing leaves your machine.

What It Looks Like in Practice

During a typical OpenClaw session:

burn0 > anthropic/claude-sonnet  ->  $0.023  (in: 1847 / out: 312)
burn0 > anthropic/claude-haiku   ->  $0.001  (in: 423 / out: 89)
burn0 > openai/gpt-4o-mini       ->  $0.0004 (in: 156 / out: 44)
burn0 > localhost (ollama)        ->  $0.000  (in: 891 / out: 203)

The Sonnet call cost 23x more than Haiku, and the Ollama call was free. Over a week, this makes it obvious which workflows are worth routing to cheaper models.

Getting Started

npm install @burn0/burn0

Add one line to your entry point and you’re done.

  • GitHub: burn0
  • Website: burn0.dev
  • MIT licensed, free forever, no account required

If you’re running OpenClaw and curious what it’s actually costing you per day, give it a try. Would love to hear what you find.

I Audited My Team’s .env Practices. Here’s What I Found.

Last month I did something uncomfortable: I spent a Friday afternoon auditing how my team actually handles secrets. Not how we say we handle them. How we actually do it.

I checked Slack history, git logs, CI configs, and local machines. What I found wasn’t a disaster — it was worse. It was normal. The kind of normal that every team thinks is fine until it isn’t.

Here’s exactly what I found, and what we did about it.

The audit

Team size: 5 developers, 3 services, 2 environments (staging + production).

I looked at five things:

1. Where do .env files live?

I asked everyone to run find ~ -name ".env" -not -path "*/node_modules/*" on their machines.

Combined results:

  • 23 .env files across 5 laptops
  • 7 of them contained production credentials
  • 2 developers had .env files for projects they left months ago
  • 1 file had a Stripe live key and a database URL on the same line — copy-paste artifact from Slack

The problem isn’t that .env files exist. It’s that they accumulate. Nobody cleans them up. Nobody knows how many copies are out there.

2. How do new devs get credentials?

I searched our Slack for “env”, “password”, “key”, “token”, and “secret”.

  • 14 messages containing actual credentials in the last 6 months
  • 3 of those were in a public channel (we caught that quickly, at least)
  • The onboarding doc said “ask someone for the .env” — no specifics about who, which values, or which environment

The worst part: Slack retains message history. Those credentials are sitting in Slack’s servers forever unless someone manually deletes each one.

3. Are secrets in git history?

# Quick check for common patterns
git log --all -p | grep -E "(API_KEY|SECRET|PASSWORD|TOKEN)=" | head -20

Two hits. Both were “temporary” commits from months ago that were reverted — but the values are still in the history. git revert doesn’t erase data. Anyone who clones the repo gets those secrets.

4. How does CI/CD get secrets?

Our GitHub Actions workflows used repository secrets. Fine in principle, but:

  • 4 people had admin access to the repo (and could read all secrets)
  • No rotation schedule — some secrets hadn’t been updated in over a year
  • No audit log of who accessed what

5. What happens when someone leaves?

We’d had one departure in the past year. Checked the offboarding:

  • GitHub access revoked ✓
  • AWS IAM removed ✓
  • Secrets rotated ✗

Every API key, database password, and service token that developer had access to was still active. For four months.

The scorecard

Practice Status
Secrets encrypted at rest ✗ (plaintext .env files)
Secrets shared securely ✗ (Slack DMs)
Secrets rotated on departure ✗
Secrets rotated on schedule ✗
Git history clean ✗
Least-privilege access ✗
Audit trail ✗

Zero out of seven. And we’re not a careless team. We write tests. We do code review. We just never formalized secrets management because it always felt like a “later” problem.

What we changed

We didn’t do everything at once. Here’s the order that worked for us:

Week 1: Stop the bleeding

Rotated every secret that the departed developer had access to. This took half a day and was painful — but it’s the highest-impact thing you can do.

Deleted secrets from Slack. Searched, found them, deleted each message. Then pinned a message in #engineering: “Never paste credentials here. Not even temporarily.”

Scrubbed git history using BFG Repo-Cleaner:

bfg --replace-text passwords.txt repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Then force-pushed and had everyone re-clone. Annoying but necessary.

Week 2: Centralize secrets

We evaluated three options:

  1. HashiCorp Vault — powerful, but needs dedicated ops. We’re 5 people.
  2. Doppler — solid, but more surface area than we needed.
  3. KeyEnv — CLI-first, minimal setup, fit our workflow.

We went with KeyEnv because the migration was zero-friction:

# Import existing .env
keyenv push

# From now on, every developer does:
keyenv pull

# Or skip the file entirely:
keyenv run -- npm start

No code changes. Apps still read process.env. The difference is that secrets come from an encrypted store instead of a file someone pasted in Slack.

Week 3: Lock down access

  • Set up per-environment permissions. Junior devs get development only. Production secrets are admin-restricted.
  • Created service tokens for CI/CD — scoped, rotatable, auditable.
  • Established a quarterly rotation schedule (calendar reminder, nothing fancy).

Week 4: Wrote it down

Added a SECRETS.md to each repo:

## Secrets Management

This project uses KeyEnv for secrets management.

### Setup
1. Get invited to the team: ask in #engineering
2. `keyenv pull` to get development secrets
3. `keyenv run -- npm start` to run with injected secrets

### Rules
- Never commit secrets to git
- Never share secrets via Slack, email, or docs
- Rotate secrets when a team member leaves
- Production secrets require admin access

Run your own audit

You can do this in an afternoon. Here’s the checklist:

  • [ ] Search Slack for credentials (password, key, token, secret, .env)
  • [ ] Count .env files across developer machines
  • [ ] Check git history for leaked values
  • [ ] Review who has access to CI/CD secrets
  • [ ] Verify secrets were rotated after last departure
  • [ ] Check if any service accounts use shared credentials

You’ll probably find what I found: not a catastrophe, but a pile of small risks that compound over time.

The fix doesn’t require enterprise tooling. It requires deciding that secrets management matters, then picking any system that’s better than Slack + .env files.

Have you ever audited your team’s secret practices? I’m curious what others have found — especially the “normal” stuff that everyone does but nobody talks about.

🚀 Building a High-Accuracy Arabic OCR Tool: How I Solved the “Image-to-Text” Challenge

Extraction of text from images (OCR) is a solved problem for Latin languages, but for Arabic, it’s a whole different story. As the developer behind Adawati.app, I spent weeks engineering a solution that doesn’t just “read” Arabic, but understands its complexity.

The Problem: Why Arabic OCR is Hard
Most open-source OCR engines struggle with Arabic for three reasons:

Cursive Nature: Arabic letters change shape based on their position (Start, Middle, End).

Diacritics & Dots: Small dots and marks can change the entire meaning of a word.

Low-Quality Input: Students often take photos of textbooks in poor lighting or at weird angles.

My Engineering Approach
Instead of just “plugging in” a generic API, I built a pipeline focused on Pre-processing and Contextual Inference.

  1. Image Pre-processing (The Secret Sauce)
    Before the AI even looks at the image, I apply several filters:

Binarization: Converting the image to high-contrast black and white to eliminate background noise.

Deskewing: Automatically correcting the angle if the photo was taken tilted.

Noise Reduction: Removing “salt and pepper” noise often found in scanned PDFs.

  1. The AI Engine
    I utilized state-of-the-art deep learning models specifically fine-tuned for Arabic scripts. These models use CNNs (Convolutional Neural Networks) for visual feature extraction and LSTMs (Long Short-Term Memory) to understand the sequence of characters, ensuring that the connected letters are recognized as a coherent word, not just random symbols.

  2. Privacy-First Architecture
    In an era of data harvesting, I made a conscious architectural decision: Zero Retention.

Images are processed in a secure memory buffer.

Once the text is extracted, the image is purged instantly.

No databases, no logs of your documents.

Why I Built This?
I saw students struggling to transcribe their lectures and researchers stuck with non-searchable PDF archives. I wanted to provide a free, fast, and no-login tool that respects their privacy while delivering professional-grade accuracy.

Try it out
If you’re a developer interested in Arabic NLP or a student looking for a reliable tool, check it out here:
👉 Image to Text – Arabic OCR Tool

I’d love to hear your feedback on the accuracy, especially with complex fonts or handwritten notes!

Arabic #OCR #AI #WebDev #Productivity #NextJS

Dify vs n8n: Architectural Roles for Low-Latency AI

Choosing between Dify and n8n isn’t a tool decision—it’s an architectural one that directly impacts your time-to-revenue and operational complexity.

You are not really choosing between two trendy tools. You are choosing between two architectural roles.

That is the right way to think about Dify vs n8n when the real question is: Can either one support serious, low-latency AI applications beyond prototypes?

The answer is yes, but not in the same way.

Dify is closer to an AI application and agent platform. n8n is closer to a workflow and automation engine that can orchestrate AI steps. Both are solid. Both can be part of a fast production stack. But they are optimized for different jobs, and that difference matters the moment you care about response time, concurrency, failure rates, observability, and maintainability under load. Dify positions itself as an open-source platform for building agentic workflows and AI applications, while n8n documents itself as a workflow automation tool with AI capabilities and a scaling model centered on queue mode for larger workloads. read

The short answer

If your primary product is a user-facing AI application with chat, RAG, tools, and agent flows, Dify is often the better front-door choice.

If your primary problem is backend orchestration at scale, with many integrations, many system-to-system actions, and stricter control over throughput and worker behavior, n8n is often the better orchestration choice, especially in queue mode. n8n’s own docs state that queue mode provides the best scalability, and its official benchmark page shows single-instance throughput up to 220 workflow executions per second depending on workflow type and configuration. read

That is why I would not frame this as a winner-takes-all comparison.

I would frame it as a stack design decision.

What Dify is really optimized for

Dify is designed around the AI product layer.

Its official docs describe it as an open-source platform for building agentic workflows, connecting tools and data sources, and deploying AI applications. Dify also provides app-level API access, workflow and chatflow concepts, knowledge pipeline orchestration for ingestion and indexing, and a broader product story around agentic workflows, RAG pipelines, integrations, and observability. read

That matters because low latency is not only about raw backend throughput.

It is also about how much glue you need to build.

When an AI app stack already includes core application primitives like chat-oriented workflows, API publishing, knowledge pipeline orchestration, and integrated tool use, you avoid a lot of extra architecture that would otherwise sit between the user and the model. That often produces a cleaner path to a fast first version.

For low-latency, user-facing AI apps, Dify has three practical advantages:

  1. It is vertically aligned to the AI app UX layer.
    You are closer to the final user experience from day one.

  2. It supports app publishing and API-based integration.
    That gives teams a way to use Dify as both a builder layer and a callable application layer. read

  3. It supports workflow and knowledge pipeline design.
    That matters when RAG latency is the real bottleneck, not model inference alone. read

In practice, Dify performs best when you self-host it close to the model endpoint, vector layer, and core data services, keep tool chains short, and stream responses so the user perceives progress even when the model takes longer to complete. Dify’s docs and product materials support the platform-app-builder framing, but they do not publish the same kind of hard latency benchmark numbers that n8n publishes for webhook and workflow throughput. That is an important distinction. read

What n8n is really optimized for

n8n is optimized around orchestration.

Its docs describe a workflow automation platform with AI capabilities, and its scaling model is explicit: queue mode provides the best scalability, using separate worker instances to process executions. n8n also publishes official performance benchmarks, which is useful because most AI infrastructure debates suffer from vague claims and very little measurable evidence. read

The benchmark page states that n8n can handle up to 220 workflow executions per second on a single instance depending on workflow complexity and configuration. The scalability benchmark blog also reports that when queue mode was enabled, throughput and latency improved materially versus single mode, with the larger c5.4xlarge benchmark showing about 162 requests per second, latency below 1.2 seconds, and zero failures under 200 virtual users for the tested webhook scenario. read

That does not mean n8n is “faster than Dify” in some universal sense.

It means something more useful: n8n gives you a more documented path to throughput-oriented backend scaling.

So for low-latency production work, the architectural lesson is simple:

  • run n8n in queue mode
  • separate webhook intake from worker execution
  • keep critical-path flows short
  • move heavy tasks into async background jobs
  • co-locate your automation engine with your model gateway, Redis, database, and vector services whenever possible

That is how you preserve predictable latency as load increases. n8n’s own documentation is clear that queue mode is the best scalability path for self-hosted production deployments. read

Dify vs n8n: Which is better for low-latency apps?

This is where technical leaders need more precision.

If your goal is a great AI product experience with a native-feeling agent app, retrieval, tool use, and fast iteration at the UX layer, I would lean toward Dify as the primary product-facing layer.

If your goal is a high-throughput backend with many automations, integrations, retries, branching logic, and execution scaling, I would lean toward n8n as the orchestration layer.

That is why these tools are often better as complements than as substitutes.

Here is the architecture pattern I would recommend for many serious teams:

  • Dify as the front-door AI application layer
  • n8n as the backend automation and system orchestration layer
  • model serving, vector storage, and databases deployed close to both
  • non-critical heavy jobs pushed off the synchronous path

That separation reduces architectural confusion. Dify handles what the user sees. n8n handles what the business process needs.

A healthcare example

Imagine you are building a telehealth triage assistant for a provider network.

The end user needs a smooth, low-latency conversational experience. The system needs RAG over clinical protocols, escalation rules, and a controlled tool chain. But it also needs downstream actions: notifications, CRM updates, logging, appointment requests, audit-side event handling, and ETL into other systems.

That is not one problem. That is two.

The cleaner design is:

  • Dify for the triage experience, response streaming, knowledge-grounded reasoning, and application logic closest to the user
  • n8n for the background automations, messaging side-effects, operational notifications, and system integrations

The same pattern carries well into fintech for onboarding and policy assistants, insurtech for claims operations, legal for document triage and internal copilots, and martech for campaign ops or sales enablement copilots.

The domain changes. The architecture discipline does not.

The mistake teams make

The mistake is trying to force one tool to do everything.

If you force n8n to become your complete AI product layer, you can end up building too much scaffolding around the actual user experience.

If you force Dify to become your universal automation backbone, you may still need another layer for broader system orchestration and non-AI workflow control.

That is why this is not just a tooling conversation.

It is an AI architecture conversation.

And that is exactly where AI Strategy Consulting and Digital Transformation Strategy become valuable: deciding where each layer belongs, how the synchronous path stays fast, which actions move off-path, and whether the stack should run on AWS, Azure, GCP, or a sovereign open-source infrastructure depending on data sensitivity, procurement constraints, and operating model.

My take

For most teams building serious AI applications, I would use this decision rule:

  • Choose Dify first when the product is fundamentally an AI application, agent, or RAG-driven experience.
  • Choose n8n first when the product problem is really orchestration, automation, and scale across systems.
  • Use both together when the app needs to feel fast to the user and still coordinate many backend actions reliably.

That is the practical answer.

Not because the tools are equal.

Because they are not designed for the same architectural job.

FAQ

Is Dify good for low-latency AI apps?

Yes, especially when the core need is a user-facing AI app with chat, workflows, RAG, and tools under one platform. Its strength is the application layer, not benchmark-driven backend throughput claims. read

Is n8n better for scale?

For backend workflow scale, n8n has a stronger documented case. Its docs state that queue mode provides the best scalability, and its benchmark pages publish concrete throughput and latency numbers. read

Should I use Dify or n8n for healthcare?

For healthcare, I would usually separate the app layer from the orchestration layer. Dify can power the front-end AI experience, while n8n handles backend automation and side-effects. The final answer depends on your latency target, data boundaries, and integration load.

Can Dify replace n8n?

Sometimes for narrower AI app workflows, but not always. Dify is stronger as an AI app and agent platform. n8n is stronger as a general workflow and automation engine.

Can n8n replace Dify?

Sometimes for backend-heavy use cases, but not when the product requires a polished AI app layer with chat, knowledge flows, and user-facing agent behavior as the center of gravity.

Further Reading

*Written by Dr Hernani Costa | Powered by Core Ventures

Originally published at First AI Movers.

Technology is easy. Mapping it to P&L is hard. At First AI Movers, we don’t just architect systems; we build the ‘Executive Nervous System’ for EU SMEs navigating AI infrastructure decisions.

Is your architecture creating technical debt or business equity?

👉 Get your AI Readiness Score (Free Company Assessment)

Our AI Readiness Assessment evaluates whether your current stack—Dify, n8n, or otherwise—aligns with your business outcomes, data governance, and scaling roadmap.