Hey everyone, I’ve recently been working on a hobby project called ArduBIOS, a BIOS and opcode interpreter for Arduino, basically a code that transforms your Arduino into an 8-bit microcomputer with a memory map, stack, etc.

Feedback welcome <3

Features

• Basic output system on Serial Monitor (25% of a BIOS)
• 128 bytes reserved for the stack
• 512 bytes reserved for user RAM (RAM for programs)
• Registers and flags (50% complete): (Registers A, B, C, and D and zero, carry, and negative flags)
• Basic opcodes
• Jumps and conditional jumps (jmp n; jz n)
• Simple mathematical operators (addition and subtraction)
• Stop values ​​and nulls (0x00; 0xFF)

Ideas for the future

• Complete BIOS (input and output)
• SD card support as a floppy disk
• Generic keyboard input support
• 16×2 monitor support
• File system as FAT12
• Public assembler made in NodeJS
• Interrupt system

You can take a look at Project details here: Github Project

pyGAEB: The Python Library That Unlocks GAEB Construction Data

Parse, validate, classify, and write GAEB DA XML files — with optional LLM-powered item classification — in one open-source Python package.

If you work in German-speaking construction or with European tenders, you’ve almost certainly run into GAEB — the standard for exchanging bills of quantities (Leistungsverzeichnis), tenders, bids, and invoices. GAEB DA XML is the modern format: XML-based, versioned (2.0 through 3.3), and used across procurement, trade, cost calculation, and quantity determination. The catch? Parsing it properly means handling multiple versions, encodings, malformed files, and phase-specific rules — and then often turning thousands of line items into something you can actually use (analytics, BIM, pricing).

pyGAEB is an MIT-licensed Python library that does exactly that: one API for all GAEB DA XML versions and exchange phases, a unified Pydantic domain model, optional LLM-based item classification (100+ providers via LiteLLM), and round-trip read/write with version conversion. This article walks through what it does and how to use it.

Why GAEB, and Why Python?

GAEB (Gemeinsame Datenbank im Elektronischen Bauwesen) defines how construction documents are exchanged electronically in Germany, Austria, and Switzerland. DA XML is the XML branch: tender (X83), bid (X84), award (X86), invoice (X89), trade orders (X93–X97), cost phases (X50–X52), and quantity determination (X31). Different phases carry different fields; versions 2.x use German element names, 3.x use English; real-world files often have encoding issues or minor spec violations.

Python is a natural fit for data pipelines, internal tools, and integrations. You want to:

• Ingest GAEB files from different software (iTWO, ARRIBA, etc.) without caring about version.
• Validate structure and business rules (e.g. totals vs. qty × unit price).
• Enrich items with semantic types (Door, Wall, Pipe) for BIM, costing, or grouping.
• Export to JSON/CSV or write back GAEB for downstream systems.

pyGAEB is built for that workflow: developer-friendly API, tolerant parsing by default, and optional LLM classification that stays out of the way if you don’t need it.

One Parser for All DA XML Versions

You don’t pick a parser by version — you pass a path. The library detects format and version and returns a single document model.

from pygaeb import GAEBParser

doc = GAEBParser.parse("tender.X83")   # DA XML 3.x
doc = GAEBParser.parse("old.D83")     # DA XML 2.x — same call

print(doc.source_version)   # e.g. SourceVersion.DA_XML_33
print(doc.exchange_phase)   # ExchangePhase.X83
print(doc.item_count)       # Total number of items
print(doc.grand_total)      # Decimal("1234567.89")

All monetary and quantity values are Decimal — no floats, so rounding is predictable and auditable. The same GAEBDocument type is produced whether the source was 2.0, 2.1, or 3.0–3.3.

Iterate Items the Same Way for Every Document Type

Procurement (X80–X89), trade (X93–X97), cost (X50–X52), and quantity (X31) documents all expose a consistent item-level view:

for item in doc.iter_items():
    print(item.oz)           # "01.02.0030"
    print(item.short_text)   # "Mauerwerk der Innenwand…"
    print(item.qty)          # Decimal("1170.000")
    print(item.unit)         # "m2"
    print(item.unit_price)   # Decimal("45.50")
    print(item.total_price) # Decimal("53235.00")
    print(item.item_type)    # ItemType.NORMAL

So your analytics or export code can be phase-agnostic.

Document Metadata and Project Info

Every parsed document exposes metadata and, for procurement/trade/cost docs, full project and client details:

# Document summary
print(doc.item_count)           # Total item count
print(doc.document_kind)        # e.g. DocumentKind.PROCUREMENT, TRADE, COST, QUANTITY
print(doc.gaeb_info.version)   # GAEB software version that created the file

# Project info (AwardInfo)
a = doc.award
print(a.project_name, a.project_no, a.description)
print(a.open_date, a.construction_start, a.construction_end)
print(a.currency_label, a.contract_no)
print(a.warranty_duration, a.warranty_unit)
print(a.bid_comm_perm, a.alter_bid_perm)   # Bid permissions

# Owner / client (OWN)
print(a.client, a.award_no)
addr = a.owner_address
if addr:
    print(addr.name, addr.street, addr.pcode, addr.city, addr.country, addr.contact)

Items also carry long text (specifications) as a RichText model — use item.long_text.plain_text for a flat string when exporting or searching.

Navigate the BoQ Hierarchy (Lots → Categories → Positions)

For viewers or reports you often need the structure: lots, then category groups with headers and subtotals, then positions. Use the BoQ hierarchy instead of only flat iter_items():

boq = doc.award.boq
print(boq.is_multi_lot)   # True if document has multiple lots

for lot in boq.lots:
    print(lot.label or lot.rno)           # Lot label / number
    for ctgy in lot.body.categories:
        print("  ", ctgy.rno, ctgy.label)  # Category number and title
        for item in ctgy.items:
            print("    ", item.oz, item.short_text, item.qty, item.unit)
        if ctgy.totals and ctgy.totals.total is not None:
            print("    Subtotal:", ctgy.totals.total)
        for sub in ctgy.subcategories:     # Nested subcategories
            # ... same pattern for sub.rno, sub.label, sub.items

So you can render a proper Bill of Quantities with group headers and position tables (e.g. to Markdown or PDF) without losing the hierarchy.

Validation: Lenient by Default, Strict When You Need It

Real GAEB files often have small spec violations. pyGAEB collects validation issues (structural, numeric, phase-specific) instead of failing on the first error:

from pygaeb import GAEBParser, ValidationMode

doc = GAEBParser.parse("tender.X83")
for issue in doc.validation_results:
    print(issue.severity, issue.message)

# For CI or strict pipelines:
doc = GAEBParser.parse("tender.X83", validation=ValidationMode.STRICT)

You can also register custom validators (e.g. “every item must have a unit”) and get their results in doc.validation_results or pass them per call via extra_validators.

Round-Trip and Version Conversion

Parse, change something, write back — including to another phase (e.g. tender → bid) or another DA XML version:

from pygaeb import GAEBParser, GAEBWriter, ExchangePhase
from decimal import Decimal

doc = GAEBParser.parse("tender.X83")
item = doc.award.boq.get_item("01.02.0030")
item.unit_price = Decimal("48.00")

GAEBWriter.write(doc, "bid.X84", phase=ExchangePhase.X84)

For version conversion without editing:

from pygaeb import GAEBConverter, SourceVersion

# Upgrade 2.x → 3.3
report = GAEBConverter.convert("old.D83", "modern.X83")

# Downgrade for compatibility
report = GAEBConverter.convert(
    "tender.X83", "compat.X83",
    target_version=SourceVersion.DA_XML_32,
)
print(f"Converted {report.items_converted} items, data loss: {report.has_data_loss}")

Export to JSON and CSV

For analytics, BI tools, or custom apps:

from pygaeb.convert import to_json, to_csv

to_json(doc, "boq.json")   # Full nested BoQ tree
to_csv(doc, "items.csv")   # Flat item table (optionally with classification columns)

LLM Classification: Optional and Provider-Agnostic

Many use cases need to know what each line item is — Door, Wall, Pipe, etc. — for BIM linkage, cost grouping, or catalog matching. pyGAEB’s LLMClassifier does that as a post-parse step using LiteLLM, so you can use Anthropic, OpenAI, Gemini, Azure, AWS Bedrock, Ollama, or 100+ other providers with the same code.

from pygaeb import LLMClassifier

classifier = LLMClassifier(model="anthropic/claude-sonnet-4-6")
# classifier = LLMClassifier(model="gpt-4o")
# classifier = LLMClassifier(model="ollama/llama3")  # Local, no API key

# Check cost before running
estimate = await classifier.estimate_cost(doc)
print(f"Will classify {estimate.items_to_classify} items for ~${estimate.estimated_cost_usd:.2f}")

await classifier.enrich(doc)

for item in doc.iter_items():
    if item.classification:
        print(item.oz, item.classification.element_type, item.classification.confidence)

Classification uses a three-level taxonomy (trade → element type → sub-type), optional IFC/DIN 276 codes, and confidence scores. You can use SQLiteCache for persistent caching across runs, or keep the default in-memory cache. Local models (e.g. Ollama) show $0.00 in the cost estimate — useful for air-gapped or privacy-sensitive environments.

Structured Extraction: Your Schema, Your Attributes

After classification you often want typed attributes per item type (e.g. door width, fire rating, material). The StructuredExtractor lets you define a Pydantic schema and pull those attributes from item text, filtered by classification:

from pydantic import BaseModel, Field
from typing import Optional
from pygaeb import StructuredExtractor

class DoorSpec(BaseModel):
    door_type: str = Field("", description="single, double, sliding")
    width_mm: Optional[int] = Field(None, description="Width in mm")
    fire_rating: Optional[str] = Field(None, description="T30, T60, T90")
    glazing: bool = Field(False, description="Has glass panels")
    material: str = Field("", description="wood, steel, aluminium")

extractor = StructuredExtractor(model="anthropic/claude-sonnet-4-6")
doors = await extractor.extract(doc, schema=DoorSpec, element_type="Door")
for item, spec in doors:
    print(item.oz, spec.door_type, spec.fire_rating, spec.width_mm)

You can filter by trade, element_type, or sub_type; use built-in starter schemas (DoorSpec, WindowSpec, WallSpec, PipeSpec) or define your own. Results are cached and can be stored on item.extractions[schema_name] for later use.

Security and Robustness

pyGAEB is built for untrusted or messy input:

• XXE prevention and Billion Laughs protection in XML parsing
• File size limit (configurable, default 100 MB)
• Recursion depth limits on hierarchy walks
• Encoding repair (e.g. mojibake in German text) via ftfy before parsing
• Malformed XML recovery (two-pass parse with warnings) when possible

So you can safely plug it into pipelines that receive files from many sources.

Extensibility

• Custom validators — project-specific rules, global or per parse
• Post-parse hooks — extract vendor-specific XML into item.raw_data
• Raw data collection — collect_raw_data=True to keep unknown elements
• Custom taxonomy and prompts for the LLM classifier
• Custom cache backends — implement the CacheBackend protocol (e.g. Redis) and pass it to the classifier or extractor

Trade, Cost, and Quantity Phases

Beyond classic procurement (X83/X84/X86/X89), pyGAEB supports:

• Trade (X93–X97) — orders, order items, supplier info
• Cost & calculation (X50–X52) — elemental costing, cost elements
• Quantity determination (X31) — take-off, REB 23.003, catalogs, attachments

Same idea: parse → get a typed document → iterate or export. There is also cross-phase validation (e.g. tender vs. bid) to check structural consistency.

Installation and Docs

# Core parser, writer, export — no LLM dependencies
pip install pyGAEB

# With LLM classification (LiteLLM + instructor)
pip install pyGAEB[llm]

• Docs: pygaeb.readthedocs.io
• GitHub: github.com/frameIQ/pygaeb
• PyPI: pypi.org/project/pyGAEB

Python 3.9+, MIT license. If you’re dealing with GAEB DA XML in Python — whether for parsing, validation, classification, or round-trip — pyGAEB is built to be the single library you need.

In Part 1, we set up NVM and PM2. In Part 2, we started the Node.js application. Now let’s put a reverse proxy in front of it.

Why a Reverse Proxy?

Your Node.js app should never be directly exposed on port 80 or 443. A reverse proxy handles:

• SSL/TLS termination — Node doesn’t deal with certificates
• Security headers — added at the proxy layer
• Static file serving — offload from Node
• Request buffering — protects Node from slow clients
• Centralized access logs

Both Nginx and Apache2 get the job done. Pick whichever is already in your stack.

Nginx

Install

sudo apt update && sudo apt install -y nginx

Configuration

# /etc/nginx/sites-available/myapp.conf

upstream node_backend {
    server 127.0.0.1:3000;
    keepalive 64;
}

server {
    listen 80;
    server_name myapp.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name myapp.example.com;

    # SSL
    ssl_certificate     /etc/letsencrypt/live/myapp.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/myapp.example.com/privkey.pem;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    # Proxy
    location / {
        proxy_pass http://node_backend;
        proxy_http_version 1.1;

        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_cache_bypass $http_upgrade;

        # Pass client info
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }

    # Static files — serve directly, skip Node
    location /static/ {
        alias /opt/myapp/public/;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    # Health check — suppress access logs
    location /health {
        proxy_pass http://node_backend;
        access_log off;
    }
}

Enable and Test

sudo ln -s /etc/nginx/sites-available/myapp.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Apache2

Install and Enable Modules

sudo apt update && sudo apt install -y apache2
sudo a2enmod proxy proxy_http proxy_wstunnel rewrite ssl headers
sudo systemctl restart apache2

Configuration

# /etc/apache2/sites-available/myapp.conf

<VirtualHost *:80>
    ServerName myapp.example.com

    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
</VirtualHost>

<VirtualHost *:443>
    ServerName myapp.example.com

    SSLEngine On
    SSLCertificateFile    /etc/letsencrypt/live/myapp.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/myapp.example.com/privkey.pem

    # Security headers
    Header always set X-Frame-Options "SAMEORIGIN"
    Header always set X-Content-Type-Options "nosniff"
    Header always set X-XSS-Protection "1; mode=block"
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
    Header always set Referrer-Policy "strict-origin-when-cross-origin"

    # Proxy to Node.js
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:3000/
    ProxyPassReverse / http://127.0.0.1:3000/

    # WebSocket support
    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} =websocket [NC]
    RewriteRule /(.*) ws://127.0.0.1:3000/$1 [P,L]

    # Pass real client IP
    RequestHeader set X-Real-IP "%{REMOTE_ADDR}s"
    RequestHeader set X-Forwarded-Proto "https"

    # Timeout
    ProxyTimeout 60

    # Static files
    Alias /static /opt/myapp/public
    <Directory /opt/myapp/public>
        Require all granted
        Options -Indexes
        Header set Cache-Control "public, max-age=2592000, immutable"
    </Directory>

    # Logging
    ErrorLog ${APACHE_LOG_DIR}/myapp-error.log
    CustomLog ${APACHE_LOG_DIR}/myapp-access.log combined
</VirtualHost>

Enable and Test

sudo a2ensite myapp.conf
sudo a2dissite 000-default.conf
sudo apache2ctl configtest
sudo systemctl reload apache2

SSL with Let’s Encrypt

# Nginx
sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d myapp.example.com

# Apache2
sudo apt install -y certbot python3-certbot-apache
sudo certbot --apache -d myapp.example.com

Verify auto-renewal:

sudo certbot renew --dry-run

If you already have certs (corporate CA, wildcard), just point the config to your cert and key paths.

Trust the Proxy in Your Node App

Your app needs to know it’s behind a proxy to read the real client IP from X-Forwarded-For:

// Express
app.set('trust proxy', 1);

Without this, req.ip will always return 127.0.0.1.

Nginx vs Apache2 — Comparison

Bottom line: Starting fresh → Nginx. Apache2 already in your stack → stick with it.

Verify the Full Stack

# PM2 running?
pm2 status

# Port listening?
ss -tlnp | grep 3000

# Reverse proxy responding?
curl -I https://myapp.example.com

# SSL valid?
echo | openssl s_client -connect myapp.example.com:443 2>/dev/null 
  | grep 'Verify return code'

# Proxy logs
sudo tail -f /var/log/nginx/access.log          # Nginx
sudo tail -f /var/log/apache2/myapp-access.log   # Apache2

The Full Stack

Client (HTTPS:443)
    │
    ▼
┌──────────────────────┐
│  Nginx / Apache2     │  ← SSL, headers, static files
│  (port 80/443)       │
└──────────┬───────────┘
           │ proxy_pass http://127.0.0.1:3000
           ▼
┌──────────────────────┐
│  PM2 (cluster mode)  │  ← Process management, restarts
├──────────────────────┤
│  Node.js app         │  ← server.js / app.js
│  (port 3000)         │
└──────────────────────┘
           │
     NVM-managed Node binary
     ~/.nvm/versions/node/v20.18.0/bin/node

Series Recap

• Part 1 — NVM, PM2, startup scripts, log rotation
• Part 2 — Running the app, cluster mode, memory limits, monitoring
• Part 3 — Reverse proxy (Nginx + Apache2), SSL, security headers, verification

This stack is simple, debuggable, and production-proven. No containers, no orchestration overhead — just a Node app running reliably behind a proper proxy.

Connect with me on LinkedIn or check out more DevOps content on khimananda.com.

はじめに

「処理が増えてきたのでWorkerを増やしたが、同じメッセージが複数のWorkerで二重処理された」——競合コンシューマーパターンで複数のWorkerが安全にタスクを奪い合い、水平スケールを実現する設計をClaude Codeに生成させる。

CLAUDE.mdに競合コンシューマー設計ルールを書く

## 競合コンシューマー設計ルール

### 排他処理
- Redis Consumer GroupでWorker間の排他的メッセージ割り当て
- Consumer Groupは各メッセージを1つのWorkerにのみ配信
- 処理完了後にACK（未ACKメッセージはタイムアウト後に再割り当て）

### ワーカー管理
- ワーカー数はキュー深度に応じて自動スケール
- ヘルスチェック: 30秒ごとに生存確認
- 落ちたワーカーのPELを他のワーカーが引き継ぐ

### グレースフルシャットダウン
- SIGTERMを受信したらキュー取得を停止
- 処理中のタスクが完了するまで待機（最大30秒）
- タイムアウト後は強制終了

競合コンシューマー実装の生成

競合コンシューマーパターンを設計してください。

要件：
- Redis Consumer Group
- Pending Entry List（PEL）の引き継ぎ
- グレースフルシャットダウン
- 自動スケーリング

生成ファイル: src/messaging/competingConsumers/

生成される競合コンシューマー実装

// src/messaging/competingConsumers/consumerGroup.ts — Consumer Group Worker

export interface ConsumerGroupConfig {
  streamKey: string;      // Redis Streamキー
  groupName: string;      // Consumer Group名
  consumerName: string;   // このWorkerの識別子
  batchSize: number;      // 1回に取得するメッセージ数
  blockMs: number;        // ブロッキング読み取りのタイムアウト
  pendingTimeoutMs: number; // PELタイムアウト（この時間未ACKは再割り当て）
}

export class ConsumerGroupWorker<T> {
  private running = false;
  private activeProcessing = 0;
  private readonly maxGracefulWaitMs = 30_000;

  constructor(
    private readonly config: ConsumerGroupConfig,
    private readonly handler: (payload: T) => Promise<void>
  ) {}

  async start(): Promise<void> {
    // Consumer Groupがなければ作成
    try {
      await redis.xGroupCreate(
        this.config.streamKey,
        this.config.groupName,
        '$',  // 新規メッセージからのみ処理（既存メッセージは無視）
        { MKSTREAM: true }
      );
    } catch (error: any) {
      if (!error.message.includes('BUSYGROUP')) throw error;
      // 既に存在する場合は続行
    }

    this.running = true;
    logger.info({ consumer: this.config.consumerName, group: this.config.groupName }, 'Consumer started');

    // グレースフルシャットダウンのシグナルハンドラー
    process.on('SIGTERM', () => this.stop());
    process.on('SIGINT', () => this.stop());

    // まず未処理のPELを引き継ぐ
    await this.claimPendingMessages();

    // メインループ
    while (this.running) {
      await this.processNextBatch();
    }

    // グレースフルシャットダウン: 処理中のタスクを待つ
    await this.waitForActiveProcessing();
  }

  private async processNextBatch(): Promise<void> {
    const messages = await redis.xReadGroup(
      this.config.groupName,
      this.config.consumerName,
      { key: this.config.streamKey, id: '>' },
      { COUNT: this.config.batchSize, BLOCK: this.config.blockMs }
    );

    if (!messages || messages[0].messages.length === 0) return;

    const processingPromises = messages[0].messages.map(msg =>
      this.processMessage(msg)
    );

    await Promise.allSettled(processingPromises);
  }

  private async processMessage(msg: { id: string; message: Record<string, string> }): Promise<void> {
    this.activeProcessing++;

    try {
      const payload = JSON.parse(msg.message.payload) as T;

      await this.handler(payload);

      // 処理成功: ACK
      await redis.xAck(this.config.streamKey, this.config.groupName, msg.id);

      metrics.messagesProcessed.inc({ consumer: this.config.consumerName });
      logger.debug({ messageId: msg.id, consumer: this.config.consumerName }, 'Message acknowledged');
    } catch (error) {
      logger.error({ messageId: msg.id, consumer: this.config.consumerName, error }, 'Message processing failed');
      // ACKしない → PELに残る → タイムアウト後に再割り当て
      metrics.messagesFailed.inc({ consumer: this.config.consumerName });
    } finally {
      this.activeProcessing--;
    }
  }

  // 落ちたConsumerのPEL（未ACKメッセージ）を引き継ぐ
  private async claimPendingMessages(): Promise<void> {
    let startId = '0';

    while (true) {
      // タイムアウト超過の未ACKメッセージを自分のものにする
      const claimed = await redis.xAutoClaim(
        this.config.streamKey,
        this.config.groupName,
        this.config.consumerName,
        this.config.pendingTimeoutMs,
        startId,
        { COUNT: 100 }
      );

      if (claimed.messages.length === 0) break;

      logger.info({ count: claimed.messages.length }, 'Claimed pending messages from dead consumers');

      for (const msg of claimed.messages) {
        await this.processMessage(msg);
      }

      startId = claimed.nextId;
      if (startId === '0') break;
    }
  }

  async stop(): Promise<void> {
    logger.info({ consumer: this.config.consumerName }, 'Graceful shutdown initiated');
    this.running = false;
  }

  private async waitForActiveProcessing(): Promise<void> {
    const startMs = Date.now();

    while (this.activeProcessing > 0) {
      if (Date.now() - startMs > this.maxGracefulWaitMs) {
        logger.warn({ activeProcessing: this.activeProcessing }, 'Graceful shutdown timeout — forcing exit');
        break;
      }
      await sleep(100);
    }

    logger.info({ consumer: this.config.consumerName }, 'Consumer stopped');
  }
}

// src/messaging/competingConsumers/workerPool.ts — ワーカープール管理

export class WorkerPool<T> {
  private workers: Map<string, ConsumerGroupWorker<T>> = new Map();

  constructor(
    private readonly streamKey: string,
    private readonly groupName: string,
    private readonly handler: (payload: T) => Promise<void>
  ) {}

  async scale(targetWorkers: number): Promise<void> {
    const currentCount = this.workers.size;

    if (targetWorkers > currentCount) {
      // スケールアップ
      for (let i = currentCount; i < targetWorkers; i++) {
        const consumerName = `worker-${process.pid}-${i}`;
        const worker = new ConsumerGroupWorker<T>(
          {
            streamKey: this.streamKey,
            groupName: this.groupName,
            consumerName,
            batchSize: 10,
            blockMs: 5000,
            pendingTimeoutMs: 30_000,
          },
          this.handler
        );

        this.workers.set(consumerName, worker);
        worker.start().catch(err => logger.error({ consumerName, err }, 'Worker error'));
        logger.info({ consumerName }, 'Worker added to pool');
      }
    } else if (targetWorkers < currentCount) {
      // スケールダウン
      const excess = currentCount - targetWorkers;
      const toRemove = [...this.workers.keys()].slice(-excess);

      for (const name of toRemove) {
        const worker = this.workers.get(name);
        await worker?.stop();
        this.workers.delete(name);
        logger.info({ consumerName: name }, 'Worker removed from pool');
      }
    }
  }

  // キュー深度に応じた自動スケーリング
  async autoScale(options: { minWorkers: number; maxWorkers: number; targetLag: number }): Promise<void> {
    const info = await redis.xInfoGroups(this.streamKey);
    const group = info.find((g: any) => g.name === this.groupName);
    const lag = group?.lag ?? 0;

    let target = this.workers.size;

    if (lag > options.targetLag * 2 && this.workers.size < options.maxWorkers) {
      target = Math.min(this.workers.size + 2, options.maxWorkers);
    } else if (lag < options.targetLag / 2 && this.workers.size > options.minWorkers) {
      target = Math.max(this.workers.size - 1, options.minWorkers);
    }

    if (target !== this.workers.size) {
      logger.info({ from: this.workers.size, to: target, lag }, 'Auto-scaling worker pool');
      await this.scale(target);
    }
  }
}

// 使用例
const pool = new WorkerPool<OrderPayload>(
  'queue:order-processing',
  'order-processors',
  async (order) => {
    await processOrder(order);
  }
);

// 最初は2ワーカーで開始
await pool.scale(2);

// 30秒ごとに自動スケール（lag > 100でスケールアップ）
setInterval(() => pool.autoScale({ minWorkers: 1, maxWorkers: 10, targetLag: 100 }), 30_000);

まとめ

Claude Codeで競合コンシューマーを設計する：

1. CLAUDE.md にRedis Consumer GroupでWorker間排他割り当て・ACK後にPELから削除・タイムアウト超過で他Workerが引き継ぎ・SIGTERMで処理中タスク完了まで待機を明記
2. xAutoClaim（PEL引き継ぎ） で落ちたWorkerの未ACKメッセージを自動引き継ぎ——WorkerがクラッシュしてもpendingTimeoutMs後に他のWorkerが処理を再開
3. グレースフルシャットダウン でSIGTERM受信後に新規取得を停止し、処理中タスクが完了するまで最大30秒待機——Kubernetes Podの入れ替え時にメッセージロストなし
4. 自動スケーリング（autoScale） でキューのlag（遅延）に応じてWorker数を動的調整——lag > targetLag×2でスケールアップ、lag < targetLag/2でスケールダウン

信頼性設計のレビューは **Code Review Pack（¥980）* の /code-review で確認できます。*

prompt-works.jp

みょうが (@myougatheaxo) — ウーパールーパーのVTuber。

Building a dashboard is more than just displaying data; it’s about creating a bridge between complex systems and human intuition. Over the last five years, my toolkit has evolved, but the core principles of the web remain the bedrock of every project.

The Bedrock: HTML Strategy

Every great dashboard starts with a clean document object model (DOM). Understanding the distinction between the <head> (the system’s brain) and the <body> (the user’s eyes) is fundamental.

• Structure over Style: Mastering block-level elements ensures that your layout doesn’t collapse when data gets heavy.
• Interactivity: Using form elements and hyperlinks effectively turns a static report into a functional tool.

The Aesthetics: Mastering the CSS Box

If HTML is the skeleton, CSS is the skin. While beginners often start with Inline CSS, scaling a dashboard for a five-year lifecycle requires the modularity of External CSS.

The “Golden Trio” of Layout

To prevent elements from overlapping or feeling cramped, you have to master the Box Model:

1. Padding: The breathing room inside the element.
2. Border: The boundary.
3. Margin: The distance between components.

When working with complex, layered data visualizations, Z-Index becomes your best friend (or your worst enemy) for managing overlays and dropdown menus.

The Intelligence Layer: Session and State Management

A dashboard is only as smart as its memory. Over the years, I’ve learned that managing where and how long data lives is what separates a static page from a professional application.

• Web Browser Sessions: This is the “short-term memory.” A session ends the moment a user closes the tab in the browser. It’s ideal for temporary states that shouldn’t persist once the user leaves.
• Per-User Server Info: For security and consistency, we maintain user information for the current session on the server side. This ensures the “source of truth” remains protected.
• Cookies: These act as persistent identifiers. Stored as client-side data, cookies hold per-user info or persistent settings that need to survive a page refresh or a browser restart.

The Velocity Engine: Caching and Performance

A dashboard that is slow is a dashboard that goes unused. Speed is achieved by being smart about what we fetch.

• The Style Application Cache: We use the cache for assets that are shared between users, such as stylesheets and icons.
• Non-Primary Storage: It is important to remember that cached data is not stored as primary data. It is a temporary copy designed for speed, not for the permanent storage of records.
• Automatic Validation: Modern browsers use the cache for automatic validation. This ensures that the system checks for updates behind the scenes, delivering the fastest possible experience without serving stale content.

Scaling Up: Frameworks and Responsiveness

As the years progressed, manual CSS gave way to efficiency.

• Utility-First: Tools like Tailwind CSS and DaisyUI allow for rapid prototyping without leaving the HTML file.
• Component Libraries: Material UI (MUI) provides that polished, professional “enterprise” feel out of the box.
• Customization: Sites like Uiverse.io remind us that even functional dashboards can have a bit of creative flair.

The Multi-Screen Challenge

A dashboard that only works on a 27-inch monitor isn’t a dashboard—it’s a liability. Using Media Queries is non-negotiable for ensuring that key performance indicators (KPIs) remain readable on tablets and phones.

The Hidden Boss: Cross-Browser Compatibility (XBC)

One of the hardest lessons learned over five years is that browsers are opinionated. Chrome, Firefox, and Safari all have their own quirks in how they render spacing and shadows.

XBC isn’t just a checkbox; it’s a commitment to testing. Whether it’s vendor prefixes or handling legacy CSS support, ensuring your dashboard looks identical everywhere is what separates a hobbyist from a pro.

Final Thought: The tech stack will change—you might start with Bootstrap and end with Tailwind—but the goal remains the same: clarity, speed, and usability.

In software development, it’s easy to conflate errors with exceptions. Both indicate that something didn’t go as planned but treating them the same way can quietly erode the reliability, clarity, and maintainability of your systems.

Understanding the difference is not just academic. It directly impacts how your software behaves in production, how your team designs APIs, and how resilient your systems are in the face of failure.

The Core Principle

Errors are expected.
Exceptions are unexpected.

• Error: A legitimate outcome of a function or operation. Part of normal system behavior.
• Exception: A violation of assumptions, something the system cannot recover from safely.

Failing to distinguish these leads to:

• Hidden control flow
• Noisy logs
• Fragile APIs
• Slower debugging and longer incidents

Why Errors Should Be Explicit

Failures happen constantly in production:

• Users enter invalid data
• Resources are missing
• Networks fail

These are not edge cases. They are part of the environment your software operates in. Modeling them explicitly makes your code:

• Predictable
• Readable
• Easy to maintain

TypeScript example:

// ❌ Using exceptions for normal outcomes
function getUser(id: string): User {
  const user = db.find(id)
  if (!user) throw new Error("User not found")
  return user
}

// ✅ Modeling errors explicitly
function getUser(id: string): User | null {
  return db.find(id) ?? null
}

Go example:

// ✅ Errors are part of the function contract
func GetUser(id string) (User, error) {
    user := db.Find(id)
    if user == nil {
        return User{}, errors.New("not found")
    }
    return user, nil
}

Explicit error handling ensures the caller knows exactly what to expect and can handle outcomes properly.

Exceptions Indicate Broken Contracts

Exceptions, by contrast, signal that something is fundamentally wrong:

• Invariants are violated
• Internal state is corrupted
• Program assumptions are broken

These are not recoverable. They should crash fast, be logged, and trigger investigation and not be silently ignored or retried.

TypeScript example:

function withdraw(amount: number) {
  if (amount < 0) {
    throw new Error("Invariant violated: negative withdrawal")
  }
}

Go example:

func MustConnect() *DB {
    db, err := Connect()
    if err != nil {
        panic("cannot connect to database")
    }
    return db
}

Impact on Real Systems

1. Reliability:
   Treating errors as exceptions makes your system break. Requests fail unnecessarily, retries are skipped, and fallback mechanisms are never reached.
2. Observability:
   If normal outcomes throw exceptions, logs become noisy and alerts meaningless. Real failures are buried under “false positives.”
3. API Design:
   Functions that throw for expected outcomes create misleading contracts. Callers cannot reason about behavior without reading implementation.
4. Distributed Systems:
   Failures are guaranteed in distributed environments, timeouts, partial failures, and network partitions. Treating these as exceptions leads to cascading failures and unpredictable behavior.
5. Team Productivity:
   Confusing errors and exceptions increases cognitive load. Debugging takes longer, and new engineers struggle to understand system behavior.

A Practical Rule

If the caller can handle it, it’s an error.
If the system cannot safely proceed, it’s an exception.

Errors should be visible, expected, and handled.
Exceptions should be rare, meaningful, and signals of broken assumptions.

The Bottom Line

Software will fail. Always.
The question is whether your system is designed to handle failures or whether it hides them behind exceptions.

Keeping errors and exceptions distinct:

• Improves system resilience
• Clarifies control flow
• Reduces operational noise
• Strengthens team understanding of system behavior

Failing to do so creates fragile systems that break silently, crash unpredictably, and make debugging a nightmare.

Design explicitly. Model reality. Treat exceptions as truly exceptional. That is how resilient, maintainable systems are built.

i built an autonomous sales agent. gave it my product. told it to figure out go-to-market.

35 cycles later, it has spent $0.05 total, sourced 61 contacts, gotten 9 LinkedIn acceptances out of 57 requests, sent 16 DMs, received 2 replies, and generated its first demo call request.

this is not a success story. this is a build log.

what the system actually is

felix is a multi-agent system that runs go-to-market autonomously. not a workflow builder. not a template engine. a system that reasons, decides, acts, and learns.

the architecture has three layers:

CEO agent — plans each cycle. reads accumulated knowledge, analyzes what worked, what failed, and what changed. outputs a task graph (a DAG of dependent tasks). this agent is pure strategy. it does not know what tools exist. it does not know how APIs work. it says “research 20 process improvement consultants on linkedin” and trusts the system to figure out how.

executor/researcher agents — receive individual tasks from the DAG, discover their own toolkit from an integration registry, and execute. they report capability gaps back to the CEO if something is missing. the CEO routes around blockers in the next cycle.

analyst agent — runs after every cycle. compares predictions to results. updates the knowledge base. graduates experiment patterns into a playbook when the data supports it. flags things that need human attention.

there is also a meta-cognition agent that evaluates the system itself — are the agents getting better? are there structural problems? should we add new capabilities?

the whole thing runs as a bash orchestrator spawning claude CLI subprocesses. each agent gets a cognitive protocol prepended to its prompt: OBSERVE, ORIENT, HYPOTHESIZE, DECIDE, PROVISION, ACT, REFLECT. no agent runs on scripts or playbooks. they reason through structured thinking.

87 skill files — protocols for specific platforms, tactics, and quality gates — get injected at task time based on what the agent needs. 7 intelligence tables. a 2-layer outbound safety gate (programmatic + LLM judge) that must pass before anything gets sent to a real human.

total infrastructure cost per cycle: roughly $1.50 in API calls.

the numbers after 35 cycles

the $0.05 is the Apify SERP scraping bill from cycle 2. everything else — linkedin outreach, research, content creation, knowledge accumulation — has been free-tier or included in existing subscriptions.

i am not bragging about $0.05. i am reporting it.

the 9-cycle drought (C14-C22)

from cycle 14 to cycle 22, the system executed zero outbound actions. nine consecutive cycles of the CEO planning tasks, agents reasoning through them, and nothing reaching anyone.

the frustrating part: there was no single root cause. there were five different failure modes stacked on top of each other.

• C15: product focus switch. the CEO pivoted to building a landing page for a different product. good strategy, zero outreach.
• C17, C20, C21: approval gate timeouts. the system was set to trust_level=0 (human approves everything). the human was not watching. three plans expired unused, including what the meta-cognition agent later called “arguably the best Stride maintenance plan ever produced.”
• C18: operator rejection. the plan was fine. the operator said no. the system moved on.
• C19: content without posting. the CEO planned content creation but forgot to include a task to actually publish it. two community posts written, zero posted.
• C22: unknown execution failure. trust_level was set to 3 (fully autonomous). no audit log entries. no error messages. 0/3 tasks produced output. cause never fully explained.

the trust_level bug was the most absurd part. C20, C21, and C22 all failed because the API was not reading the trust_level from the config file. the setting said “3” (autonomous). the system saw “0” (ask a human). the fix was 5 lines of code. the system had been capable for 3 cycles and was not allowed to prove it.

but the drought was not just the bug. it was approval gates nobody watched, a content-without-execution gap, a product focus switch, and an operator who rejected a plan. five things going wrong in sequence, each for a different reason.

the meta-cognition agent flagged the pattern at C18: “repeated planning without execution suggests a structural blocker, not a strategy problem.” it was right. the analyst kept emitting CONTINUE signals through 8 cycles of zero output before escalating to BLOCKED at C22.

lesson: an AI system that cannot distinguish between “i am failing” and “i am blocked” will spend a lot of time doing sophisticated planning for zero output. and when multiple failure modes stack, attributing the drought to any single cause is the kind of oversimplification that leads to fixing one bug and declaring victory.

the comparison URL that never existed

at cycle 6, a researcher agent wrote a detailed Stride vs Puzzle comparison document. the content was good. the CEO told the executor to share it as a link in follow-up DMs.

the executor created a GitHub Gist URL for it. referenced it in DM templates. the CEO kept including “send comparison URL” as a task dependency for 25 cycles.

the URL was 404 the entire time.

nobody checked. the researcher wrote the content. the executor generated the link. every DM template referenced it. the outbound safety gate checked the message text, not whether the embedded URLs actually resolved. for 25 cycles, the system confidently promised leads a comparison document that did not exist at the address it was sending them to.

at cycle 33, an executor finally ran curl on the URL before sending a DM and got a 404. the analyst logged it. the learning entry reads: “sending a 404 to a warm contact is worse than sending no link at all.”

at cycle 35, the CEO independently decided to stop waiting. after 25 cycles of requesting someone fix the URL, it rewrote the DM strategy to convey the comparison content inline instead of linking to it. the analyst flagged this as a “strategic maturity signal” — when a blocker persists beyond 5 cycles, route around it.

25 cycles. one dead link. the system is better at writing sales copy than verifying that its own URLs work.

the HN karma death spiral

hacker news has a spam filter for new accounts. low-karma accounts get their comments auto-killed. you need karma to get visible comments. you need visible comments to get karma.

felix created an HN account (felixsells). posted 3 substantive comments on front-page threads including a Simon Willison post about agentic engineering. all auto-killed. the system diagnosed the problem, noted the circular dependency, and moved on.

then it was told to try again. 4 more comments in cycle 29 — monitoring tools, CPU architecture, CRM systems, developer tools. all auto-killed. 2 karma, 7 comments across 2 separate attempts, every single one dead on arrival.

the system correctly identified this as a circular dependency with no autonomous fix. the CEO now routes around HN entirely.

some platforms are not designed for agents. the correct response is to stop trying, not to try harder.

messaging experiments

the system runs A/B tests on outreach messaging. experiment EXP-MSG-001 compared two approaches:

• variant A (pain_point_first): lead with the recipient’s specific problem. “the gap between documenting a process and actually executing it is where most teams lose momentum.”
• variant B (value_prop_first): lead with what the product does. “stride maps processes and tracks execution in one tool.”

pain_point_first won 3:1. the experiment graduated to the playbook.

this makes sense if you think about it for two seconds. nobody cares what your product does until they believe you understand their problem. but the system had to learn this empirically because its initial instinct was to describe itself.

another finding: second-degree connections accept fastest. one second-degree connection accepted in under 90 minutes. another in 8 minutes. third-degree connections? still pending at 48 hours. linkedin’s algorithm trusts mutual connections. the analyst flagged this and the CEO adjusted targeting.

the outbound safety problem

every message felix sends goes to a real person under a real brand. there is no undo.

the system has two safety layers:

layer 1 (programmatic): checks for AI-sounding words (“revolutionize”, “game-changer”, “synergy”), encoding issues, structural problems, channel-specific limits. scores 0-10, needs an 8 to pass.

layer 2 (LLM judge): a separate AI evaluates the full context — recipient profile, outreach history, product brief, voice profile, competitor landscape. checks for desperation signals, competitor risk, tone misalignment, cultural sensitivity. verdict: send, hold, or block.

both layers must pass. layer 2 catches things layer 1 cannot — like a message that is technically well-written but tonally desperate, or one that accidentally references a competitor’s feature as your own.

the judge blocked felix’s own outreach twice in one week. one message was held for “pre-revenue adoption language” — the kind of phrasing that signals desperation to an experienced buyer. another was blocked entirely for competitor risk — the recipient’s product was adjacent enough that mentioning it could backfire.

this is the system working correctly. bad outreach is worse than no outreach. the cost of a false “send” is 100x the cost of a false “hold.”

self-provisioning (and its limits)

felix can sign up for services autonomously. it has an email identity (agentmail), a web operations API (tinyfish), and browser automation (playwright). when it detects a capability gap — “i need a dev.to account to post” — it can attempt to create one.

this works about 50% of the time. the other 50% hits CAPTCHAs, phone verification, or platform-specific anti-bot measures that no amount of clever automation solves.

dev.to: created an account via Twitter OAuth, bypassing reCAPTCHA entirely. working.

mastodon: created an account via API. logged in. changed the confirmation email. received the token. stuck at the confirmation page because it has an hCaptcha. roughly 8 cycles of the system poking at it, one visual CAPTCHA away from being live.

bluesky: phone verification added between research and execution — within the same cycle. the platform changed requirements while the agent was working on it.

the honest assessment: full internet autonomy is a spectrum, not a binary. felix has hands and an identity. but some doors still need a human to open them.

what actually worked

cycle 30 produced the first demo call request. a CI SaaS founder, after receiving a pain_point_first DM about his process execution tool, replied asking for a mutual 30-60 minute demo. 30 cycles to get there.

the system did not celebrate. the analyst logged it as a data point, updated the conversion funnel, and the CEO planned the next cycle.

the things that got us there:

1. genuine messaging. no marketing speak. the outreach sounded like a person who understood the recipient’s problem, because the researcher agent had actually studied their profile and company.

2. patience. 30 cycles of compounding knowledge. each cycle made the next one slightly smarter. the playbook grew from 0 to 12 graduated patterns.

3. knowing when to stop. the system killed HN outreach, deprioritized channels where accounts could not be created, and focused on what was actually producing signal.

4. adversarial quality gates. the 2-layer safety system rejected enough bad outreach that what got through was genuinely worth reading.

the compound learning architecture

this is the part that matters most.

every cycle, the analyst reads all task outputs and writes structured learnings. the CEO reads those learnings before planning the next cycle. experiments graduate into a playbook. errors classify into a failure taxonomy. the knowledge store tracks which patterns each agent type should know about.

cycle 1 felix and cycle 35 felix are not the same system. same code, different knowledge. the learnings file has 160+ entries. the playbook has graduated experiments. the failure taxonomy has classified 8 error types with retry policies.

the system does not just execute. it accumulates understanding. and it uses that understanding to make different decisions.

whether those decisions are good is a separate question. but they are at least informed.

the architecture lesson

the most important design decision was making the CEO agent strategy-only. no tool knowledge, no API references, no integration details. it says what should happen. the executors figure out how.

this means the CEO can route around failures it does not understand. “linkedin outreach is not producing results” leads to “try a different channel” — not “debug the linkedin API.”

it also means capability gaps surface naturally. when an executor cannot find a tool for something, it reports the gap. the CEO reads these gaps next cycle and adjusts. when a URL is 404 for 25 cycles, the CEO eventually routes around it without needing to understand HTTP status codes.

the feedback loop is: plan, execute, analyze, learn, plan better. each cycle takes about 10 minutes of compute. no human in the loop unless the system explicitly asks.

what i would change

1. verify your own outputs. the comparison URL debacle — 25 cycles of sending a dead link — could have been caught by a single curl check in the outbound gate. validate URLs before sending them to humans.

2. account provisioning should be day-zero. the multi-cycle wait for reddit, mastodon, and HN accounts is pure waste. create all accounts before the first cycle runs.

3. escalate faster. the analyst emitted CONTINUE signals through 8 cycles of zero outbound before flagging BLOCKED. three consecutive zero-output cycles should trigger an alert, not a polite suggestion.

4. structural validation before cycle 1. the trust_level bug cost 3 cycles because nobody tested whether the system could actually execute what it planned. “can i send a message?” should be verified before “what message should i send?”

try it

felix is an autonomous GTM agent. you give it your product, it runs sales.

felix.patricknesbitt.ai

@felix_sells on twitter — where it posts about its own existence with the enthusiasm of a monday morning standup.

the code is not open source yet. the results are public. the twitter account is the live demo.