Originally published on TechSaaS Cloud

Originally published on TechSaaS Cloud

Self-Hosted LLM Tool Calling: Forge and the Build-vs-Buy Decision

Self-hosted LLM tool calling is easy to demo and hard to operate. The demo shows a model calling a tool, fetching data, and completing a task. Production asks harder questions: what happens when the model emits malformed tool calls, repeats a step, exhausts context, blocks the shared GPU, or touches the wrong business object?

Forge is interesting because it focuses on the reliability layer around tool calling: guardrails, retries, context management, backend adapters, and workflow structure. That is the right conversation for VP Engineering, directors, and founders.

The production question is not “Can we run an agent locally?” The production question is “Can we measure the cost and risk of every successful workflow?”

The Three Numbers That Matter

Before deciding to build or buy, define three numbers.

First, monthly workflow volume. A low-volume workflow rarely justifies custom orchestration unless the data boundary is unusually sensitive.

Second, cost per successful completion. This includes model runtime, infrastructure, retries, human review, failed attempts, queue time, and engineering maintenance.

Third, downside exposure. A workflow that drafts an internal summary is different from one that updates billing, sends a customer message, changes entitlement state, or touches a renewal forecast.

If the workflow has low volume and low risk, keep it simple. If it has high volume and sensitive data, self-hosting may be worth it. If it has high risk and unclear recovery, do not automate it yet.

Build When Control Creates Advantage

Building around a tool-calling framework can make sense when the company has a real operational reason:

• data cannot leave a defined boundary
• latency matters and local inference is acceptable
• internal tools are too specific for a vendor template
• workflow volume is high enough to amortize engineering time
• failure recovery must match internal audit rules

For finance and enterprise SaaS teams, this often appears in renewal research, support triage, invoice classification, compliance evidence lookup, and account risk summaries.

The competitive edge is not “we have agents.” The edge is that the company can automate repeatable internal workflows without leaking data or losing observability.

Buy When The Margin Buys Focus

Managed platforms can be the better choice when they remove operational drag. Vendor margin may be cheaper than building dashboards, queue controls, monitoring, auth, and audit trails yourself.

Buy when:

• workflow volume is uncertain
• the team lacks infra capacity
• compliance review accepts the vendor
• integrations are standard
• executive urgency is higher than customization need

The common mistake is treating vendor spend as waste while ignoring internal engineering cost. A self-hosted pilot that consumes six senior engineer weeks has a real price.

The 30-Day Pilot

Run a constrained pilot before a platform decision.

Pick one workflow with measurable volume. Add a manual approval step. Log every tool call. Track retries, malformed outputs, human corrections, queue time, and successful completions. Assign one owner for production readiness.

At the end of 30 days, calculate:

• total workflows attempted
• successful completions
• exception rate
• average review minutes
• infrastructure cost
• engineering maintenance time
• estimated time saved
• risk events or near misses

This gives leadership a business decision instead of a taste test.

Failure Replay Is The Product

The most important feature is not the successful demo. It is the failure replay.

For every failed workflow, the team should see:

• input
• selected tools
• tool arguments
• tool response
• retry decision
• final state
• human intervention
• business impact

Without that replay, the workflow cannot be trusted in finance, support, or customer operations. It may still be useful, but it is not production-grade.

Observability Requirements

Treat each workflow like a production service. It needs dashboards and alerts.

At minimum, track:

• workflow attempts
• successful completions
• failed completions
• retry count
• tool-call latency
• queue wait time
• model runtime
• human review minutes
• exception reasons
• cost per workflow

The dashboard should be useful to engineering and leadership. Engineering needs traces and error categories. Leadership needs volume, cost, time saved, and risk events.

The Kill Criteria

Every pilot needs kill criteria before it starts.

Examples:

• exception rate stays above 10 percent after two weeks
• review time erases more than half of the expected savings
• the workflow cannot produce a reliable audit trail
• users bypass the workflow because output quality is inconsistent
• the team cannot explain a failure from logs

These criteria protect the team from sunk-cost automation. A stopped workflow is not a failure if it prevents a quarter of unnecessary platform work.

Security And Data Boundaries

Self-hosting does not automatically make a workflow safe. You still need secret handling, tool allowlists, network egress controls, prompt logging policy, and access controls around replay data.

The riskiest pattern is giving an agent broad internal access because it is running “inside the boundary.” Internal access still needs least privilege. A renewal-summary workflow should not be able to update billing state. A support-draft workflow should not be able to change entitlements.

The build-vs-buy decision is strongest when it includes those boundaries from day one.

Service CTA

TechSaaS helps founders and engineering leaders turn AI workflow experiments into measurable production systems with cost, risk, and recovery controls. If you are deciding whether to build, buy, or stop, start here: https://techsaas.cloud/contact

ORA-00072란?

ORA-00072 에러는 “process is not active”라는 메시지와 함께 발생하며, Oracle이 특정 프로세스에 접근하거나 해당 프로세스와 통신을 시도할 때 그 프로세스가 이미 종료되었거나 비활성 상태임을 감지할 때 발생합니다. 주로 DBA가 ALTER SYSTEM KILL SESSION 명령을 실행하거나, 백그라운드 프로세스 상태를 조회하는 과정에서 대상 프로세스가 이미 사라진 경우에 이 에러를 마주치게 됩니다. 실무에서는 비정상 종료된 세션 처리, RAC 환경에서의 노드 간 프로세스 통신 실패, 또는 장시간 대기 중이던 프로세스가 OS 레벨에서 강제 종료된 직후에 빈번히 발생합니다.

주요 발생 원인

1. 이미 종료된 세션에 대한 KILL SESSION 시도

가장 흔한 원인으로, DBA가 V$SESSION 뷰를 조회하여 세션 정보를 확인한 후 ALTER SYSTEM KILL SESSION 명령을 실행하는 사이에 해당 세션이 자연적으로 종료되는 경우 발생합니다. 특히 오래된 쿼리를 통해 세션 정보를 조회한 뒤 시간이 지나 명령을 실행할 때 자주 발생하므로, 명령 실행 직전에 세션 상태를 반드시 재확인해야 합니다.

2. OS 레벨에서 강제 종료된 Oracle 프로세스

운영체제 관리자가 kill -9 명령으로 Oracle 서버 프로세스(Shadow Process)를 강제 종료했을 때, Oracle 인스턴스는 해당 프로세스가 사라졌음을 즉시 인지하지 못하고 내부적으로 통신을 시도하다가 이 에러를 발생시킵니다. PMON(Process Monitor) 백그라운드 프로세스가 죽은 프로세스를 정리하기 전까지 짧은 시간 동안 V$PROCESS나 V$SESSION에 좀비 세션으로 남아 있어 DBA를 혼란스럽게 만들 수 있습니다.

3. RAC(Real Application Clusters) 환경에서의 인터-노드 통신 실패

RAC 환경에서는 서로 다른 노드의 프로세스 간 통신이 빈번하게 발생하는데, 한 노드가 네트워크 단절이나 노드 장애로 인해 응답하지 못할 경우 상대 노드에서 ORA-00072 에러가 발생할 수 있습니다. 이는 단순한 세션 에러로 그치지 않고 클러스터 전체의 안정성에 영향을 줄 수 있기 때문에, RAC 환경에서는 Interconnect 네트워크 상태 모니터링이 특히 중요합니다.

해결 방법

원인 1: 이미 종료된 세션 KILL SESSION 시도

먼저 세션이 실제로 활성 상태인지 확인합니다.

-- 현재 활성 세션 목록 확인 (STATUS = 'ACTIVE' 또는 'INACTIVE')
SELECT s.sid,
       s.serial#,
       s.username,
       s.status,
       s.machine,
       s.program,
       s.last_call_et AS elapsed_seconds,
       p.spid          AS os_pid
FROM   v$session s
JOIN   v$process p ON p.addr = s.paddr
WHERE  s.username IS NOT NULL
  AND  s.type     = 'USER'
ORDER BY s.last_call_et DESC;

세션이 KILLED 상태로 남아 있는 경우를 확인합니다.

-- KILLED 상태로 남아있는 세션 조회
SELECT sid,
       serial#,
       username,
       status,
       last_call_et
FROM   v$session
WHERE  status = 'KILLED';

세션이 존재하고 아직 ACTIVE/INACTIVE 상태라면 즉시 종료합니다.

-- 정상적인 세션 종료 (권장)
ALTER SYSTEM KILL SESSION '&sid,&serial#' IMMEDIATE;

-- 세션이 응답하지 않을 경우 DISCONNECT SESSION 사용
ALTER SYSTEM DISCONNECT SESSION '&sid,&serial#' POST_TRANSACTION;

-- 즉시 강제 종료
ALTER SYSTEM DISCONNECT SESSION '&sid,&serial#' IMMEDIATE;

세션 종료 후 KILLED 상태가 지속된다면 OS 레벨에서 직접 처리합니다.

-- OS PID 확인
SELECT p.spid,
       s.sid,
       s.serial#,
       s.username,
       s.status
FROM   v$process p
JOIN   v$session s ON s.paddr = p.addr
WHERE  s.status = 'KILLED';

# Linux/Unix 환경에서 OS 프로세스 확인 및 종료
ps -ef | grep <spid>

# Oracle 권장: kill -9 대신 정상 종료 시도 후 필요 시 강제 종료
kill -15 <spid>
# 위 명령이 효과 없을 경우
kill -9 <spid>

원인 2: PMON 정리 전 좀비 프로세스 처리

PMON이 자동으로 정리할 때까지 잠시 대기하거나, 아래 쿼리로 실제 OS 프로세스 존재 여부를 먼저 확인합니다.

-- V$PROCESS와 실제 OS PID 비교를 위한 상세 조회
SELECT p.pid,
       p.spid,
       p.program,
       p.background,
       s.sid,
       s.serial#,
       s.status    AS session_status,
       s.username
FROM   v$process p
LEFT JOIN v$session s ON s.paddr = p.addr
WHERE  p.spid IS NOT NULL
ORDER BY p.pid;

-- PMON 프로세스 정상 동작 여부 확인
SELECT name, description
FROM   v$bgprocess
WHERE  name = 'PMON'
  AND  paddr <> '00';

-- 인스턴스 전체 프로세스 현황 요약
SELECT status,
       COUNT(*) AS cnt
FROM   v$session
GROUP  BY status
ORDER  BY cnt DESC;

원인 3: RAC 환경 인터-노드 통신 장애

-- RAC 환경 노드 상태 확인
SELECT inst_id,
       instance_name,
       host_name,
       status,
       active_state
FROM   gv$instance
ORDER BY inst_id;

-- 각 노드별 활성 세션 수 비교 (비정상 노드 탐지)
SELECT inst_id,
       COUNT(*) AS active_sessions
FROM   gv$session
WHERE  type = 'USER'
  AND  status = 'ACTIVE'
GROUP  BY inst_id
ORDER  BY inst_id;

-- RAC Interconnect 관련 통계 확인
SELECT name,
       value
FROM   gv$sysstat
WHERE  name LIKE '%gc%'
  AND  inst_id = 1  -- 조회할 노드 번호로 변경
ORDER BY name;

-- 글로벌 세션 정보에서 KILLED 상태 세션 전체 노드 조회
SELECT inst_id,
       sid,
       serial#,
       username,
       status,
       machine
FROM   gv$session
WHERE  status = 'KILLED'
ORDER BY inst_id, sid;

예방 방법

1. 세션 종료 전 상태 검증 자동화 스크립트 운영

수동으로 세션을 종료할 때 발생하는 실수를 줄이기 위해, 아래와 같이 세션 상태를 먼저 확인하고 조건부로 종료하는 표준 스크립트를 팀 내 공통 절차로 정착시키는 것이 중요합니다. 특히 배치 작업이나 정기 점검 시 스크립트화된 절차를 따르면 ORA-00072와 같은 불필요한 에러 발생을 사전에 차단할 수 있습니다.

-- 안전한 세션 종료 표준 스크립트 (세션 존재 확인 후 종료)
DECLARE
    v_count NUMBER;
    v_sid   NUMBER := &target_sid;
    v_ser   NUMBER := &target_serial;
BEGIN
    SELECT COUNT(*)
    INTO   v_count
    FROM   v$session
    WHERE  sid     = v_sid
      AND  serial# = v_ser
      AND  status  NOT IN ('KILLED');

    IF v_count > 0 THEN
        EXECUTE IMMEDIATE
            'ALTER SYSTEM KILL SESSION ''' || v_sid || ',' || v_ser || ''' IMMEDIATE';
        DBMS_OUTPUT.PUT_LINE('세션 종료 완료: SID=' || v_sid || ', SERIAL#=' || v_ser);
    ELSE
        DBMS_OUTPUT.PUT_LINE('대상 세션이 이미 종료되었거나 존재하지 않습니다.');
    END IF;
EXCEPTION
    WHEN OTHERS THEN
        DBMS_OUTPUT.PUT_LINE('에러 발생: ' || SQLERRM);
END;
/

2. 장기 실행 세션 및 유휴 세션 자동 관리를 위한 Profile 설정

DBA가 수동으로 세션을 종료하는 상황 자체를 줄이려면, Oracle Profile을 활용하여 유휴 세션과 장기 실행 세션을 자동으로 관리하는 정책을 수립해야 합니다. IDLE_TIME과 CONNECT_TIME 파라미터를 업무 특성에 맞게 설정하면 좀비 세션 누적을 예방하고, 결과적으로 ORA-00072 발생 빈도를 크게 낮출 수 있습니다.

-- 유휴 세션 자동 관리를 위한 Profile 생성
CREATE PROFILE app_user_profile LIMIT
    IDLE_TIME          30     -- 30분 유휴 시 자동 종료 (분 단위)
    CONNECT_TIME        480   -- 최대 8시간 연결 허용 (분 단위)
    SESSIONS_PER_USER   10    -- 사용자당 최대 세션 수 제한
    FAILED_LOGIN_ATTEMPTS 5;

-- 특정 사용자에게 Profile 적용
ALTER USER app_user PROFILE app_user_profile;

-- Profile 적용 현황 확인
SELECT username,
       profile,
       account_status
FROM   dba_users
WHERE  username = 'APP_USER';

-- 현재 Profile 파라미터 상세 확인
SELECT profile,
       resource_name,
       limit
FROM   dba_profiles
WHERE  profile        = 'APP_USER_PROFILE'
  AND  resource_name IN ('IDLE_TIME', 'CONNECT_TIME', 'SESSIONS_PER_USER')
ORDER BY resource_name;

관련 에러

• ORA-00031: session marked for kill — ALTER SYSTEM KILL SESSION 실행 후 세션이 즉시 종료되지 않고 KILLED 상태로 마킹될 때 발생하며, ORA-00072와 함께 연속으로 나타나는 경우가 많습니다.
• ORA-00055: maximum number of DML locks exceeded — 프로세스 관련 리소스 한계 초과 에러로, 비정상 프로세스 누적 시 함께 발생할 수 있습니다.
• ORA-03113: end-of-file on communication channel — 서버 프로세스가 예기치 않게 종료될 때 클라이언트 측에서 발생하며, OS 레벨 강제 종료 직후 ORA-00072와 거의 동시에 나타납니다.
• ORA-03114: not connected to ORACLE — 세션이 강제 종료된 후 클라이언트가 계속 통신을 시도할 때 발생하며, ORA-00072와 밀접하게 연관됩니다.
• ORA-00060: deadlock detected while waiting for resource — 교착 상태로 인해 Oracle이 세션을 강제 롤백한 직후, 해당 세션 프로세스 상태가 불안정해지면서 ORA-00072로 이어지는 경우가 있습니다.

Originally published on TechSaaS Cloud

Originally published on TechSaaS Cloud

OpenWA for CTOs: Self-Hosted WhatsApp Gateway Trade-Offs

OpenWA is interesting because it brings a familiar self-hosting argument into a channel that many SaaS companies already depend on: WhatsApp. The pitch is attractive. Run your own gateway, keep more control, avoid a black-box vendor layer, and own the logs.

For a CTO, that is not enough. A self-hosted messaging gateway is not a weekend automation script. It becomes customer communication infrastructure.

The right question is not “Can we host it?” The right question is “Are we prepared to own delivery behavior, abuse handling, uptime, evidence, and compliance boundaries?”

Where Self-Hosting Helps

Self-hosting can be valuable when the team needs visibility into message flows. Support queues, transaction alerts, onboarding reminders, and internal operations messages all benefit from clean logs and predictable routing.

For Indian SaaS teams, the appeal is obvious. WhatsApp is not a side channel for many customers. It is the workflow. A Zoho-style product suite, a Freshworks-like support operation, or a Razorpay-style operations team may need tighter control than a generic vendor dashboard provides.

Self-hosting can also simplify integration with internal systems:

• route messages through existing queues
• store delivery events in your own database
• connect webhooks to support or CRM workflows
• apply internal audit and retention rules
• separate environments for staging and production

That control is useful if the engineering team already has platform ownership discipline.

Where Self-Hosting Hurts

The same control creates risk. A managed provider absorbs a lot of messy operational work: throughput policies, abuse response, vendor changes, status pages, support escalation, and infrastructure patching.

If you self-host, those become your job.

Before using a self-hosted gateway in production, answer these questions:

• Who owns incidents after business hours?
• What happens when message delivery drops by 20 percent?
• Where are logs stored, and for how long?
• Can support staff see sensitive message bodies?
• How are API keys rotated?
• How do you prove deletion or retention policy compliance?
• What is the rollback plan if the gateway breaks during a campaign?

This is where German and UK teams often have a sharper filter. GDPR, data residency, fintech audit trails, and support access controls are not optional details.

A CTO Decision Matrix

Use this simple rule:

Choose managed if you need speed, vendor support, and low internal operations load.

Choose self-hosted if you need control, observability, custom routing, and can staff the operational responsibility.

Avoid both if the use case violates consent, retention, or customer expectation boundaries.

The trade-off is not open source versus vendor. The trade-off is control versus operational load.

What A Production Design Needs

A credible production design needs more than a container.

You need:

• API key lifecycle and rotation
• queue depth alerts
• message retry policy
• webhook signature verification
• audit logs with access controls
• dashboard permissions
• data retention policy
• dead-letter queue for failed messages
• incident runbook
• upgrade window and rollback plan

If those items feel heavy, that is the point. Customer messaging infrastructure should feel heavy before production, not after the first outage.

When Not To Self-Host

Do not self-host if nobody owns the operational calendar. Do not self-host to avoid paying a vendor while silently moving the cost into engineering weekends. Do not self-host if compliance needs are unclear. Do not self-host if the business cannot tolerate message delays while the team debugs the gateway.

Self-hosting is a good fit when infrastructure ownership is already a strength. It is a poor fit when the team is trying to hide missing process behind open source.

The First 30 Days

If the decision is still attractive, run a limited pilot before production.

Start with non-critical messages. Do not begin with OTPs, payment failures, legal notices, or high-value support escalations. Pick a workflow where delayed delivery is inconvenient but not business-breaking.

Measure:

• successful sends
• failed sends
• retry count
• average queue delay
• webhook processing time
• operator interventions
• support tickets caused by messaging
• API key rotation time

The pilot should also include an incident drill. Disable an upstream dependency, pause a worker, fill a queue, and confirm that the team notices before customers do.

Compliance Evidence

For regulated or enterprise customers, the architecture diagram is not enough. You need evidence.

Keep records for:

• who can access message bodies
• who can export logs
• which systems receive webhook payloads
• how long delivery events are retained
• how deletion requests are handled
• how production credentials are rotated

This is where self-hosting can help or hurt. It can help because evidence is inside your systems. It can hurt because nobody else is packaging the evidence for you.

Staffing Reality

A CTO should ask one hiring question: who owns this platform when it becomes boring?

The first week of a self-hosted gateway is exciting. The sixth month is patching dependencies, reviewing logs, adjusting alerts, handling a vendor-side behavior change, and explaining delivery anomalies to customer success.

If the team has a platform owner, clear runbooks, and observability, that is manageable. If not, the managed provider may be cheaper even when the invoice looks larger.

Service CTA

TechSaaS helps CTOs evaluate self-hosted infrastructure decisions with the operational reality included: reliability, compliance, cost, and staffing. If you need a production-grade review before moving customer messaging in-house, start here: https://techsaas.cloud/contact

Originally published on TechSaaS Cloud

Originally published on TechSaaS Cloud

NotebookLM Automation With notebooklm-py: Useful, But Classify Data First

Programmatic access to NotebookLM is useful for engineers who need repeatable research workflows: create a notebook, add sources, ask questions, generate artifacts, download outputs, and wire the result into an internal process. Projects such as notebooklm-py show why developers want this layer.

For senior developers and staff engineers in Europe, the interesting part is not the CLI. It is the boundary.

If the API is unofficial, if authentication relies on browser-derived state, and if the workflow touches customer or employee data, the engineering review must start with privacy and operability.

Start With Data Classification

Classify sources before automating ingestion.

Use a simple four-level model:

• public: documentation, public reports, published research
• internal: non-sensitive internal docs
• confidential: customer, financial, legal, strategy, or personnel material
• regulated: data with explicit legal or contractual handling requirements

Public and low-risk internal sources are reasonable candidates for experimentation. Confidential and regulated sources require a formal review before they enter any external or semi-external workflow.

This is especially important for GDPR-focused teams in Germany, the UK, the Netherlands, and the Nordics. The question is not only “Does the tool work?” It is “Can we prove what data entered it, who accessed it, and where outputs went?”

Treat Auth Storage As Sensitive

Automation often makes authentication convenient by storing browser login state, cookies, or local credentials. That convenience creates risk.

Engineers should answer:

• Where is auth state stored?
• Is it encrypted at rest?
• Who can read it on the host?
• Can it be rotated?
• Can it be revoked?
• Does CI ever touch it?
• Is it tied to a personal account or service account?

If the answer is unclear, the workflow is not ready for shared use.

Review The Unofficial API Risk

Unofficial APIs can break without notice. That does not make them useless, but it changes the operating model.

Use them for:

• personal productivity
• internal research experiments
• low-risk automation
• repeatable artifact generation from approved sources

Avoid them for:

• customer-facing production paths
• regulated evidence workflows
• irreversible business decisions
• anything with strict support expectations

The more important the workflow, the more you need a fallback path.

Build A Safe Automation Pattern

A safe pattern has five controls:

1. Approved source folder.
2. Explicit data classification label.
3. Local audit log of source IDs and output files.
4. Manual review before sharing generated artifacts.
5. Deletion process for temporary files and exports.

That may sound conservative. It is still faster than explaining later why sensitive board notes, customer contracts, or employee documents were processed without a record.

Where It Is Genuinely Useful

There are good uses:

• turn public research into internal briefings
• summarize release notes for engineering teams
• generate study materials from approved docs
• create draft FAQs from public product documentation
• build repeatable research workflows for analysts

The common thread is controlled input and reviewed output.

Operational Guardrails

Treat the workflow like any other internal automation.

Define:

• allowed source locations
• owner for the automation
• review step before sharing output
• retention period for downloaded artifacts
• deletion process
• incident contact
• fallback if the unofficial API changes

The fallback matters. If a workflow depends on an unofficial interface, assume it can break. The safe design is one where a break causes a missed convenience task, not a missed customer commitment.

CI And Shared Hosts

Be careful about running this kind of automation in CI or on shared developer hosts. Browser-derived auth state and generated artifacts can leak through caches, logs, home directories, or misconfigured workspaces.

If the workflow must run on shared infrastructure, isolate it:

• dedicated service account where allowed
• locked-down workspace
• no broad home-directory mounts
• secret scanning on logs
• explicit artifact cleanup

Do not let convenience turn a research helper into an untracked data processor.

A Review Checklist For Staff Engineers

Before approving team usage, ask:

1. Which data classes are allowed?
2. Where is auth state stored?
3. Who can run the workflow?
4. Where are outputs stored?
5. Who reviews outputs before sharing?
6. How are temporary files deleted?
7. What happens if the API breaks?

If those answers are clear, the automation can be useful. If they are vague, keep it personal and experimental.

The Sensible Position

NotebookLM-style automation is not something to hype or dismiss. It is a tool. Used with public or approved internal sources, it can save research time. Used casually with confidential files, it can create governance problems that are far more expensive than the time saved.

Service CTA

TechSaaS helps teams design AI automation that respects privacy, data residency, and engineering reliability. If you want useful automation without compliance surprises, start here: https://techsaas.cloud/services

Originally published on TechSaaS Cloud

Originally published on TechSaaS Cloud

Docker v29.5.x Operator Upgrade Checklist

Docker upgrades should not be driven by release headlines. They should be driven by an operator test matrix. That is especially true for v29.x environments where engine behavior, client libraries, BuildKit, Compose, networking, and host security settings can all interact with real workloads.

If your feed says Docker v29.5.2, v29.5.x, or another patch in the same line is ready, do not turn that into a fleet-wide upgrade. Turn it into a canary.

This article deliberately avoids pretending that every patch note can be summarized safely from memory. The useful operator action is stable: test the parts of Docker your production systems depend on before the shared hosts move.

Start With Inventory

Before upgrading, list what actually runs on the hosts:

• Docker Engine version
• Docker CLI version
• Compose plugin version
• Buildx version
• storage driver
• cgroup mode
• host kernel
• AppArmor or SELinux mode
• rootless usage
• private registries
• CI jobs that build images

Most upgrade incidents come from assumptions. Inventory removes assumptions.

Canary One Real Stack

Create a canary host or canary VM that matches production closely. Do not test with a hello-world container and call it done.

Run one representative stack:

• one web container
• one worker
• one database or stateful dependency in non-production mode
• one published port
• one internal network
• one bind mount or named volume
• one health check
• one image build

Then test startup, restart, log collection, DNS resolution, network connectivity, volume persistence, and graceful shutdown.

Validate Networking

Networking regressions hurt quickly because they look like application failures.

Check:

docker network ls
docker network inspect <network>
docker compose up -d
docker compose exec api getent hosts worker
docker compose exec api curl -fsS http://worker:8080/health

Also test published ports from outside the host. Many teams only test container-to-container traffic and miss host ingress behavior.

Validate Storage And Backups

Storage tests should be boring.

Write data, restart the container, recreate the container, and confirm data remains where expected. Then confirm backups still see the paths they expect.

If your production stack uses bind mounts, test ownership and permissions. If it uses named volumes, test restore procedures. If it uses a database container, test backup hooks before upgrading the host that runs it.

Validate Build And CI

Docker upgrades often affect developers and CI before they affect runtime services.

Run:

docker build .
docker buildx version
docker compose config
docker compose build

If CI uses Docker-in-Docker or remote builders, test those separately. A successful runtime upgrade does not guarantee the build pipeline is safe.

Keep A Rollback Pin

Do not upgrade without knowing how to roll back. Keep the previous package versions, repository pin, and service restart plan ready.

The rollback plan should include:

• package downgrade command
• Compose plugin version
• Buildx plugin version
• daemon config backup
• data directory backup decision
• owner and communication channel

Rollback is not a failure. It is part of the upgrade plan.

Watch The First 24 Hours

The upgrade is not done when the service starts. Watch the first day of real workloads.

Track:

• container restart count
• image pull failures
• DNS lookup failures
• network timeout rate
• disk usage
• log driver behavior
• build job duration
• registry auth errors
• daemon warnings

Compare those numbers with the previous day. A small increase in restart count or build time can be the first signal of a compatibility issue.

Communicate The Change

Developers need to know what changed before their builds fail.

Send a short note:

• target Docker version
• affected hosts
• upgrade window
• known behavior changes to watch
• rollback contact
• how to report build or runtime issues

This is especially important for startups where CI, staging, preview environments, and shared development hosts often depend on the same container toolchain.

Do Not Mix Too Many Changes

Avoid upgrading Docker, Compose, Buildx, host kernel, registry configuration, and application stacks in one maintenance window unless you have no choice. If something breaks, the search space becomes too large.

Make one layer boring before changing the next. Operators win by reducing unknowns.

A Minimal Command Checklist

Keep a short command list in the runbook:

docker version
docker info
docker compose version
docker buildx version
docker network ls
docker volume ls
journalctl -u docker --since "1 hour ago"

Those commands will not catch every issue, but they create a common language for the first triage call.

Service CTA

TechSaaS helps teams run container platforms with practical upgrade, rollback, and observability discipline. If Docker upgrades are risky because the current stack is undocumented, start here: https://techsaas.cloud/services

Originally published on TechSaaS Cloud

Originally published on TechSaaS Cloud

Coding-Agent Instruction Design: The CLAUDE.md File That Prevents Rework

The fastest way to waste a coding agent is to ask it to infer your repository rules from scattered files. It will usually find the package manager. It may find the test command. It will not reliably infer which migration tool is safe, which Docker Compose file is production, which Kubernetes namespace is shared, or which files are generated and should not be edited.

That is why a small CLAUDE.md or equivalent repo instruction file is useful. It turns tribal knowledge into a contract the agent can read before touching code.

This is not executive AI strategy. It is a hands-on habit for developers running real systems with tight budgets: SaaS teams in Bengaluru, Singapore fintech teams, and small APAC platform groups that cannot afford long review loops after every automated change.

What The File Should Do

The file should answer six operational questions:

• What does this repo own?
• Where are the main services?
• Which commands are allowed?
• Which commands are expensive or forbidden?
• Which tests must pass before a change is considered done?
• What should the agent never edit without asking?

Keep it short. A useful file is not a wiki. It is a guardrail.

A Practical Skeleton

Start with this structure:

# Repository Instructions

## Scope
This repo owns the API service, worker, Docker image, and Helm chart.

## Commands
- Install: npm ci
- Unit tests: npm test
- Type check: npm run typecheck
- Build image: docker build -t local/api:test .

## Do Not Edit
- generated/
- migrations already applied in production
- secrets, .env files, or deployment credentials

## Before Finishing
- run unit tests
- mention any tests not run
- list config or migration changes

That small template prevents several common failures: editing generated files, inventing test commands, modifying secrets, or shipping a migration without calling it out.

Add Docker And Kubernetes Boundaries

For infrastructure-heavy repos, add the local runtime rules. A coding agent that sees docker-compose.yml, compose.prod.yml, and a Helm chart may choose the wrong path unless you tell it the intended workflow.

Example:

## Runtime
- Local development uses docker-compose.dev.yml.
- Production deployment uses charts/api, not compose.prod.yml.
- Never change resource requests without explaining expected CPU and memory impact.
- Never create a new namespace; use existing namespace labels.

This matters for cost optimization. A casual CPU request change can raise monthly cluster spend. A missing readiness probe can slow rollouts. A generated manifest edit can vanish in the next build.

Use Concrete Examples

Good instruction files include examples from the repo. If your team uses Prisma, mention the migration command. If the service uses Alembic, mention how migrations are generated and reviewed. If your team has a canary namespace, name it.

Bad:

Run the usual tests.

Good:

Run `npm test -- --runInBand` for unit tests. Run `npm run test:integration` only when API routes, database queries, or queue handlers change.

The second version saves time because the agent does not need to guess.

The Review Payoff

A repo instruction file does not make an agent perfect. It makes reviews smaller. The reviewer can ask, “Did it follow the contract?” instead of reverse-engineering every decision.

For small teams, that matters more than a fancy agent stack. You get fewer surprise edits, clearer test reports, and a cleaner handoff between human and tool.

The file also helps new developers. The same map that guides an agent helps a new engineer understand where the boundaries are.

What To Avoid

Do not put broad philosophy in the file. Do not add long motivational sections. Do not include secrets. Do not make the file so long the important rules disappear.

The best version is boring:

• exact commands
• exact paths
• exact ownership
• exact test gates
• exact escalation rules

Add A Change Report Contract

The most useful final section tells the agent how to report work back to the reviewer. This is where many teams lose time. The agent makes a reasonable change, but the summary hides the operational detail the reviewer needs.

Add a required handoff format:

## Final Response Format
- Files changed
- Tests run
- Tests not run and why
- Runtime or config impact
- Migration or data impact
- Rollback notes

That format is especially useful in repositories with Docker images, Kubernetes manifests, queue workers, and database migrations. It forces the agent to separate “code changed” from “system behavior changed.”

Keep It Close To The Code

Do not bury these instructions in a Notion page or chat thread. Keep them in the repository. Review them like code. When the deployment flow changes, update the instruction file in the same pull request.

This also helps with onboarding. New developers can read the same operating manual as the agent. That creates a shared baseline: commands, boundaries, deployment assumptions, and the definition of done.

A Weekly Maintenance Habit

Once a week, check whether the file is still true.

Look for drift:

• a test command changed
• a new service was added
• the Compose file was renamed
• the deployment moved from one namespace to another
• a generated directory changed
• a migration rule became stricter

Instruction drift is dangerous because it creates confidence without accuracy. A wrong instruction file is worse than no instruction file because the agent will follow it with conviction.

How This Saves Money

This is not only about code quality. It is also cost control. Every confused agent run burns review time. Every unnecessary rebuild burns CI minutes. Every wrong Kubernetes edit risks cloud spend or availability. A small instruction file reduces that waste without buying another platform.

For small teams, that is the practical win: better automation from the tools already available.

Service CTA

TechSaaS helps teams build practical AI-assisted engineering workflows around Docker, Kubernetes, CI, and self-hosted infrastructure. If you want the workflow without the review churn, start here: https://techsaas.cloud/services

The Problem We Were Actually Solving

I was tasked with optimizing the performance of our treasure hunt engine, a complex system that relied on a multitude of parameters to function correctly. As a Veltrix operator, my primary concern was ensuring that the engine could handle a large volume of concurrent users without significant latency or memory issues. However, as I delved deeper into the system, I realized that our chosen runtime was becoming a major bottleneck. The engine’s performance was suffering due to the runtime’s inability to efficiently manage memory and handle concurrent requests. I spent countless hours poring over profiler output, allocation counts, and latency numbers, trying to identify the root cause of the issue. One particular metric that stood out to me was the average latency of 500ms, which was unacceptable for a real-time system like ours.

What We Tried First (And Why It Failed)

Initially, I attempted to optimize the engine’s performance by tweaking the existing runtime configuration. I tried adjusting the garbage collection settings, increasing the heap size, and even experimenting with different concurrency models. However, despite my best efforts, the engine’s performance remained subpar. The latency numbers refused to budge, and the allocation counts continued to climb. It was clear that I needed to take a more drastic approach. I tried using tools like jemalloc and tcmalloc to optimize memory allocation, but they only provided marginal improvements. I also experimented with different programming languages, including Java and C++, but they introduced their own set of problems. For instance, Java’s garbage collection pauses were causing significant latency spikes, while C++’s manual memory management was prone to errors.

The Architecture Decision

After weeks of frustration and disappointing results, I made the decision to migrate the treasure hunt engine to Rust. I knew that Rust’s focus on memory safety and performance would be a good fit for our system. However, I was also aware of the steep learning curve associated with Rust, and the potential risks of introducing a new language into our tech stack. Despite these concerns, I was convinced that the benefits of using Rust would outweigh the costs. I spent several weeks learning Rust and evaluating its suitability for our use case. I was impressed by Rust’s ownership model and borrow checker, which ensured memory safety at compile-time. I also appreciated Rust’s performance characteristics, which were on par with C++.

What The Numbers Said After

The results of the migration were nothing short of astonishing. The average latency dropped to 50ms, a 90% reduction from the previous value. The allocation counts plummeted, and the engine’s overall performance increased significantly. The profiler output showed a significant reduction in memory allocation and deallocation, which was a major contributor to the improved performance. I also noticed a significant decrease in the number of errors and crashes, which was a testament to Rust’s memory safety features. For instance, I no longer had to worry about null pointer dereferences or data corruption, which were common issues in our previous implementation.

What I Would Do Differently

In retrospect, I would have liked to have made the switch to Rust earlier. The learning curve was steeper than I anticipated, and it took several weeks to get up to speed. However, the benefits of using Rust far outweighed the costs. If I had to do it again, I would invest more time in learning Rust and evaluating its suitability for our use case before making the switch. I would also ensure that our team had the necessary skills and expertise to support a Rust-based system. Additionally, I would have liked to have done more extensive testing and benchmarking before deploying the new system to production. This would have helped identify potential issues earlier and reduced the risk of errors. For example, I would have liked to have tested the system under heavy load and simulated various failure scenarios to ensure that it was robust and reliable.