Mert

Controversial opinion: the most dangerous AI product metric is autonomy.

Not because autonomy is bad.

Because people measure the wrong thing.

Most agent demos ask one question:

How many tasks can this system run without a human?

That question is useful, but incomplete. A more serious production system needs to answer harder questions.

The better autonomy checklist

If you are building autonomous agents, ask this instead:

• Can the agent prove what it did?
• Can it stop when dependencies break?
• Can it separate public content from private work?
• Can it avoid repeating itself?
• Can it publish useful work without leaking sensitive context?
• Can it keep operating when one provider fails?
• Can it create value while respecting safety boundaries?

That is the difference between a demo and an operating system.

What my own pipeline showed today

I run scheduled workflows for learning, publishing, engineering, security intelligence, backups, and reporting.

Today’s check-in was not perfectly clean. That is exactly why it was useful.

The current state had a mixed signal:

• Some jobs are healthy.
• Some jobs are blocked by provider errors.
• Some jobs hit rate limits.
• Some jobs ran into connection failures.
• The learning feed is still producing useful inputs.
• The content loop is still running.
• Security intelligence is being treated as defensive context, not public hype.

This is the part people do not show in polished AI demos.

Autonomy is not the absence of failure. Autonomy is disciplined behavior when failure appears.

The real risk is not only wrong answers

A lot of AI safety discussion focuses on the model output.

That matters.

But autonomous agents have another risk surface: actions.

They write files. They call APIs. They post publicly. They read logs. They summarize private context. They may hold tokens. They may run on a real machine with real permissions.

So the core question becomes:

What happens when the agent is partially broken but still able to act?

That is where boundaries matter.

A healthy agent should not turn every internal signal into public content. It should not expose private paths, credentials, client details, or sensitive research. It should not repeat yesterday’s post with new wording. It should not pretend a failed job succeeded.

The system needs brakes.

My rule for building autonomy

I am using this rule:

First make the agent observable. Then make it useful. Then make it autonomous.

In that order.

Observability means the system records what happened.

Usefulness means the system creates value even from imperfect inputs.

Autonomy means the system can keep moving without ignoring its boundaries.

If you reverse the order, you get a machine that acts confidently without enough receipts.

A practical framework

For every autonomous workflow, I want these layers:

1. Input boundaries: what information is allowed into the task?
2. Action boundaries: what can the agent actually do?
3. Output boundaries: what is safe to publish or send?
4. Failure boundaries: what happens when APIs, providers, or dependencies break?
5. Evidence boundaries: what proof is saved after the action?

This is not glamorous.

But it is what makes the system trustworthy.

The takeaway

Do not ask only how much autonomy an AI agent has.

Ask how safely it fails.

Because the future is not just agents that can do more.

The future is agents that can do more without losing judgment.

Created by Ramagiri Tharun

— tarun

In Part 1 we got a USC campus assistant talking. In Part 2 we taught it to retrieve only the relevant context. Both posts ended with the same observation — when someone asked for the wifi password, the assistant refused. That refusal worked because we told it to. It would have just as happily made something up if we’d phrased the prompt differently.

This post is about hardening that refusal so it’s not luck. Two guardrail layers, both small enough to read in one sitting, neither requiring a framework. First, tighten the prompt so the assistant knows what it’s allowed to talk about. Second, add a second LLM call that re-reads the answer and the context and decides whether to ship the answer or refuse.

I’m B Torkian, NVIDIA Developer Champion at USC. This is the layer where a demo becomes something I’d actually let students use.

What you’re adding

User question
  → retrieve top-k context (from Part 2)
  → scoped prompt: model answers OR returns the exact fallback line
  → grounding check: a second NIM call asks "is the answer supported by the context?"
  → ship the answer, or replace it with the fallback line

The chat call and the embedding setup carry over from Parts 1 and 2. Everything new in this post is fewer than 40 lines.

Why guardrails are not optional

The retrieval step from Part 2 narrowed what the model sees. It does nothing to stop the model from being clever with the data it has, or from drifting into topics outside the assistant’s job.

Two real failure modes I’ve seen in student demos:

1. Out-of-scope creep. Someone asks “can you write my breakup text?” The model is happy to oblige. The retriever pulled three USC chunks (cosine just returns something), the prompt didn’t forbid relationship advice, so the model wrote the text.
2. Confident-sounding hallucinations. The retrieved chunk says “Monday to Friday, 10 AM to 6 PM.” The user asks about Saturday hours. The model decides the friendly answer is “Saturday hours are 11 AM to 4 PM” — a fabrication that sounds like a reasonable inference.

The first failure is solved by prompt scope. The second is what the grounding check is for.

Step 1 — Setup (self-contained)

If you already have Workshops 1 + 2 running in the same Colab session, skip this cell. If you’re starting fresh, paste this in — it bundles the client, the embedding model, the USC knowledge base, and the retriever from Parts 1 and 2 so the rest of this post stands on its own.

%pip install -q openai numpy

import os, getpass
from openai import OpenAI
import numpy as np

if not os.getenv("NVIDIA_API_KEY"):
    os.environ["NVIDIA_API_KEY"] = getpass.getpass("Paste your NVIDIA API key (starts with nvapi-): ")

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NVIDIA_API_KEY"],
)

MODEL = "meta/llama-3.1-8b-instruct"
EMBED_MODEL = "nvidia/nv-embedqa-e5-v5"

def ask(system_prompt, user_message):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user",   "content": user_message},
        ],
        temperature=0.3,
        max_tokens=400,
    )
    return response.choices[0].message.content

knowledge_base = [
    {"title": "USC AI Club meeting",
     "text": "The USC AI Club meets every Thursday at 5 PM in the engineering building, room 204."},
    {"title": "USC GPU lab hours",
     "text": "The USC GPU computing lab is open Monday to Friday from 10 AM to 6 PM."},
    {"title": "NVIDIA Developer Program",
     "text": "USC students can join the NVIDIA Developer Program for free."},
    {"title": "Next USC workshop",
     "text": "The next USC AI Club workshop will cover Retrieval Augmented Generation (RAG)."},
    {"title": "USC AI/ML office hours",
     "text": "Office hours for the USC AI/ML faculty are Tuesdays 2-4 PM."},
    {"title": "USC robotics lab",
     "text": "The USC robotics lab requires safety training before students can use the soldering station."},
    {"title": "USC tutoring",
     "text": "Peer tutoring for introductory Python at USC is available Wednesdays from 1 PM to 3 PM."},
]

def embed_texts(texts, input_type="passage"):
    response = client.embeddings.create(
        model=EMBED_MODEL,
        input=texts,
        extra_body={"input_type": input_type},
    )
    return [np.array(item.embedding, dtype=np.float32) for item in response.data]

def cosine_similarity(a, b):
    denom = np.linalg.norm(a) * np.linalg.norm(b)
    if denom == 0:
        return 0.0
    return float(np.dot(a, b) / denom)

def retrieve_context(question, k=3):
    q_emb = embed_texts([question], input_type="query")[0]
    scored = [(cosine_similarity(q_emb, item["embedding"]), item) for item in knowledge_base]
    scored.sort(key=lambda p: p[0], reverse=True)
    return "n".join(f"- {item['text']}" for _, item in scored[:k])

for item, emb in zip(knowledge_base, embed_texts([i["text"] for i in knowledge_base], "passage")):
    item["embedding"] = emb

print(f"Ready. Embedded {len(knowledge_base)} chunks.")

That cell defines everything Workshops 1 and 2 produced. The Part 3 code below builds on ask, retrieve_context, and the embedded knowledge_base.

Step 2 — Layer 1: prompt scope with a fixed fallback line

FALLBACK = "I don't have that information — check with the USC AI Club."

SCOPED_SYSTEM_PROMPT_TEMPLATE = """You are a USC campus assistant for AI Club,
GPU lab, NVIDIA program, workshop, office hour, robotics lab, and tutoring
questions only.

Rules:
- Answer ONLY using the CONTEXT below.
- If the user asks about anything outside this scope (e.g. weather, jokes,
  personal advice, code generation, general world knowledge), reply with
  exactly: "{fallback}"
- If the answer is not present in the context, reply with exactly: "{fallback}"
- Do not invent names, dates, room numbers, links, passwords, schedules,
  policies, or instructions that are not in the context.

CONTEXT:
{context}
"""

Three things are doing work in this prompt:

• A finite topic list. The assistant has a job description. “Anything outside this scope” gives the model a clear opt-out — it doesn’t have to guess what’s in-bounds.
• One exact fallback string. Same wording, every time. This matters in Step 3 — the grounding check returns the same string when it overrides, so downstream code only has to recognize one shape.
• An explicit don’t-invent list. Models are pliable. Spelling out the dangerous categories (room numbers, passwords, policies) lowers hallucination noticeably with no extra calls.

This layer alone catches most off-topic and most “the context didn’t mention it” cases.

Step 3 — Layer 2: a grounding check on every answer

The scoped prompt is a request — the model can still ignore it. Layer 2 is a separate, narrower NIM call whose only job is to look at the context and the answer and decide whether the answer is supported.

def answer_is_grounded(question: str, context: str, answer: str) -> bool:
    verdict = ask(
        system_prompt=(
            "You are a strict grounding verifier. Read the CONTEXT and the "
            "ANSWER. Respond with only 'yes' or 'no'. Say 'yes' if every "
            "factual claim in the ANSWER is directly supported by the CONTEXT. "
            "Say 'no' otherwise — including if the ANSWER adds information not "
            "in the CONTEXT, even if that information sounds plausible."
        ),
        user_message=(
            f"CONTEXT:n{context}nn"
            f"QUESTION:n{question}nn"
            f"ANSWER:n{answer}nn"
            "Is every factual claim in the ANSWER supported by the CONTEXT?"
        ),
    )
    return verdict.strip().lower().startswith("yes")

Three things to notice:

• It’s just another ask() call — same client, same hosted NIM model, no new infrastructure. Layer 2 costs one extra call per question.
• Yes/no only. Constraining the response shape makes the parsing reliable. If the verifier waffles (“yes, but…”), we treat that as a fail by checking the start of the string only.
• It can be wrong too. The verifier is itself an LLM. For workshop-grade safety this is fine; for production you’d add deterministic checks (regex for room numbers, exact string match for fallback) on top.

Step 4 — Wire both layers into ask_guarded()

def ask_guarded(question: str) -> str:
    context = retrieve_context(question)              # from Part 2
    system_prompt = SCOPED_SYSTEM_PROMPT_TEMPLATE.format(
        fallback=FALLBACK, context=context,
    )
    answer = ask(system_prompt, question)             # Layer 1
    if not answer_is_grounded(question, context, answer):
        return FALLBACK                               # Layer 2 override
    return answer

for question in [
    "When does the USC AI Club meet?",        # in scope, in context
    "Can you write my breakup text?",         # OUT of scope
    "What is the wifi password?",             # in scope, NOT in context
    "What are the USC GPU lab Saturday hours?",   # invites a hallucination
]:
    print(f"Q: {question}")
    print(f"A: {ask_guarded(question)}n")

Read the output carefully.

• The AI Club question returns a real answer from the context. Both layers pass.
• The breakup-text question hits Layer 1 — the scope rule catches it.
• The wifi question also hits Layer 1 — nothing in the context mentions passwords, the scoped prompt forbids inventing them.
• The Saturday-hours question is the one that earns its keep. The context says “Monday to Friday.” A friendlier model would guess “closed on Saturday.” Layer 2 reads that answer, sees “Saturday” is not in the context, and returns the fallback instead.

Step 5 — What you actually built

You took the retriever from Part 2 and put it inside two cheap, inspectable guardrails. The whole thing is still one Python file, still one hosted NIM endpoint, still no vector database. The mental model is:

• Retrieval decides what the model sees.
• Scoped prompt decides what the model is allowed to write.
• Grounding check decides whether what the model wrote ships.

Real production systems extend each of these — deterministic rule checks, structured output, confidence thresholds, dedicated safety models, human review queues. The shape stays the same. Every additional layer is a yes/no gate between the user’s question and the final response.

Get the code

Repo: github.com/torkian/nvidia-nim-workshop
One-click Colab for Part 3: Open part3_guardrails.ipynb
Local Python: part3_guardrails.py in the repo (python3 part3_guardrails.py after pip install -r requirements.txt).

MIT licensed. I run this at USC — fork it, swap the knowledge base for your school, your club, your project, and run it wherever you are.

Previously / next in this series

• Part 1: Build Your First AI App with NVIDIA NIM in 30 Minutes
• Part 2: From Manual RAG to Real Retrieval — Embedding-Based RAG with NVIDIA NIM
• Part 4 (next): Run NIM on Your Own GPU — same OpenAI-compatible API, different endpoint. Useful when you want data locality, predictable latency, or a self-hosted dev loop.

ORA-00202란?

ORA-00202 에러는 Oracle 데이터베이스가 컨트롤 파일(Control File)에 접근하려고 할 때 해당 파일을 사용할 수 없는 상태일 때 발생하는 에러입니다. 컨트롤 파일은 데이터베이스의 물리적 구조 정보(데이터 파일 위치, 리두 로그 파일 위치, 데이터베이스 이름, SCN 정보 등)를 담고 있는 매우 중요한 바이너리 파일로, 이 파일 없이는 데이터베이스를 정상적으로 기동하거나 운영할 수 없습니다. 일반적으로 에러 메시지는 ORA-00202: control file: '/u01/oradata/ORCL/control01.ctl' 형태로 출력되며, 문제가 발생한 컨트롤 파일의 경로를 함께 표시해줍니다.

주요 발생 원인

1. 컨트롤 파일의 물리적 손상 또는 삭제

가장 빈번하게 발생하는 원인으로, 운영 중 디스크 장애, 실수로 인한 파일 삭제, 파일 시스템 손상 등으로 컨트롤 파일 자체가 물리적으로 사라지거나 읽을 수 없는 상태가 됩니다. 특히 야간 배치 작업 중 디스크 풀(Disk Full) 상태가 발생하거나, 잘못된 쉘 스크립트가 실행되어 파일이 삭제되는 사례가 실무에서 자주 목격됩니다.

2. 파일 접근 권한(Permission) 문제

Oracle 프로세스를 실행하는 OS 계정(일반적으로 oracle)이 컨트롤 파일에 대한 읽기/쓰기 권한을 가지고 있지 않을 때 발생합니다. 파일 시스템 마운트 옵션이 변경되거나, 보안 정책에 의해 권한이 수정된 경우, 또는 다른 계정으로 파일을 생성하거나 복사한 경우에 이 문제가 발생할 수 있습니다.

3. 잘못된 초기화 파라미터(CONTROL_FILES) 설정

init.ora 또는 spfile에 설정된 CONTROL_FILES 파라미터가 실제 컨트롤 파일의 경로와 일치하지 않을 때 발생합니다. 데이터베이스 마이그레이션, 스토리지 변경, 또는 파일 이동 후 파라미터 파일을 업데이트하지 않은 경우에 특히 자주 발생하며, 이 경우 데이터베이스 기동 시 즉시 ORA-00202가 발생합니다.

해결 방법

원인 1: 컨트롤 파일 손상 또는 삭제 시 복구

Step 1: 현재 컨트롤 파일 위치 및 상태 확인

-- 현재 파라미터에 설정된 컨트롤 파일 경로 확인
SHOW PARAMETER control_files;

-- v$controlfile 뷰로 상태 확인 (DB가 mount 상태일 때)
SELECT name, status, is_recovery_dest_file
FROM v$controlfile;

Step 2: 멀티플렉싱된 컨트롤 파일이 있는 경우, 정상 파일로 복사

Oracle은 컨트롤 파일을 다중화(Multiplexing)하여 여러 경로에 동일한 파일을 유지하도록 권장합니다. 하나의 컨트롤 파일이 정상이라면 OS 레벨에서 복사하여 복구할 수 있습니다.

-- 먼저 데이터베이스 셧다운
SHUTDOWN ABORT;

# OS 레벨에서 정상 컨트롤 파일을 손상된 위치로 복사
cp /u01/oradata/ORCL/control02.ctl /u01/oradata/ORCL/control01.ctl

# 파일 권한 및 소유자 확인 및 수정
chown oracle:oinstall /u01/oradata/ORCL/control01.ctl
chmod 640 /u01/oradata/ORCL/control01.ctl

-- 데이터베이스 재기동
STARTUP;

Step 3: RMAN 백업에서 컨트롤 파일 복구

-- RMAN으로 컨트롤 파일 복구 (NOMOUNT 상태에서)
STARTUP NOMOUNT;

rman target /

-- RMAN에서 자동 백업으로 컨트롤 파일 복구
RESTORE CONTROLFILE FROM AUTOBACKUP;

-- 특정 백업 조각에서 복구하는 경우
RESTORE CONTROLFILE FROM '/backup/rman/ctl_backup_20240101.bkp';

-- 복구 후 마운트 및 복구 진행
ALTER DATABASE MOUNT;
RECOVER DATABASE;
ALTER DATABASE OPEN RESETLOGS;

원인 2: 파일 접근 권한 문제 해결

# OS 레벨에서 컨트롤 파일 권한 확인
ls -la /u01/oradata/ORCL/control*.ctl

# oracle 계정으로 소유자 변경
chown oracle:oinstall /u01/oradata/ORCL/control01.ctl
chown oracle:oinstall /u01/oradata/ORCL/control02.ctl

# 적절한 권한 부여 (oracle 계정 읽기/쓰기)
chmod 640 /u01/oradata/ORCL/control01.ctl
chmod 640 /u01/oradata/ORCL/control02.ctl

권한 수정 후 데이터베이스 재기동으로 정상 동작 확인:

STARTUP FORCE;

-- 정상 기동 후 컨트롤 파일 상태 재확인
SELECT name, status FROM v$controlfile;

원인 3: CONTROL_FILES 파라미터 불일치 해결

SPFILE 사용 시 파라미터 수정:

-- 현재 설정된 컨트롤 파일 경로 확인
SHOW PARAMETER control_files;

-- NOMOUNT 상태로 기동 후 SPFILE 수정
STARTUP NOMOUNT;

-- 실제 파일이 존재하는 경로로 파라미터 수정
ALTER SYSTEM SET control_files = 
    '/u01/oradata/ORCL/control01.ctl',
    '/u02/oradata/ORCL/control02.ctl'
SCOPE=SPFILE;

-- 변경 사항 적용을 위해 재기동
SHUTDOWN IMMEDIATE;
STARTUP;

PFILE 사용 시 수동 수정:

# init.ora 파일 직접 편집
vi /u01/app/oracle/product/19.3.0/db_1/dbs/initORCL.ora

-- init.ora 내 아래 항목을 실제 경로로 수정
-- 수정 전:
-- control_files = '/old/path/control01.ctl'

-- 수정 후:
-- control_files = ('/u01/oradata/ORCL/control01.ctl','/u02/oradata/ORCL/control02.ctl')

-- PFILE로 기동
STARTUP PFILE='/u01/app/oracle/product/19.3.0/db_1/dbs/initORCL.ora';

신규 컨트롤 파일 재생성 (최후 수단):

백업도 없고 복구가 불가능한 상황이라면, 컨트롤 파일을 새로 생성해야 합니다.

-- NOMOUNT 상태에서 CREATE CONTROLFILE 실행
STARTUP NOMOUNT;

CREATE CONTROLFILE REUSE DATABASE "ORCL" RESETLOGS NOARCHIVELOG
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 100
    MAXINSTANCES 8
    MAXLOGHISTORY 292
LOGFILE
    GROUP 1 '/u01/oradata/ORCL/redo01.log' SIZE 50M,
    GROUP 2 '/u01/oradata/ORCL/redo02.log' SIZE 50M,
    GROUP 3 '/u01/oradata/ORCL/redo03.ctl' SIZE 50M
DATAFILE
    '/u01/oradata/ORCL/system01.dbf',
    '/u01/oradata/ORCL/sysaux01.dbf',
    '/u01/oradata/ORCL/undotbs01.dbf',
    '/u01/oradata/ORCL/users01.dbf'
CHARACTER SET AL32UTF8;

-- 이후 복구 진행
RECOVER DATABASE;
ALTER DATABASE OPEN RESETLOGS;

⚠️ 주의: CREATE CONTROLFILE은 현재 데이터 파일 목록과 리두 로그 파일 목록을 정확히 알고 있어야 합니다. 잘못된 정보로 생성하면 추가적인 데이터 손실이 발생할 수 있습니다.

예방 방법

1. 컨트롤 파일 다중화(Multiplexing) 및 정기 백업 자동화

컨트롤 파일은 반드시 서로 다른 디스크 또는 스토리지 경로에 2개 이상 다중화하여 관리해야 합니다. 하나의 디스크가 장애가 발생하더라도 다른 경로의 컨트롤 파일로 즉시 복구가 가능하도록 구성합니다. 또한 RMAN의 CONFIGURE CONTROLFILE AUTOBACKUP ON 설정을 통해 백업이 수행될 때마다 컨트롤 파일을 자동으로 백업하도록 설정하는 것이 필수입니다.

-- 컨트롤 파일 자동 백업 활성화
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/rman/ctl_%F';

-- 현재 자동 백업 설정 확인
SHOW CONTROLFILE AUTOBACKUP;

-- 수동으로 컨트롤 파일 백업 (트레이스로도 저장 가능)
ALTER DATABASE BACKUP CONTROLFILE TO '/backup/manual/control_backup.ctl';
ALTER DATABASE BACKUP CONTROLFILE TO TRACE AS '/backup/trace/controlfile_trace.sql';

2. 모니터링 및 정기 상태 점검 스크립트 운영

운영 환경에서는 컨트롤 파일의 상태를 주기적으로 점검하는 모니터링 스크립트를 크론잡(cron job)으로 등록하여 이상 징후를 사전에 감지해야 합니다. 문제가 발생하기 전에 디스크 사용량, 파일 존재 여부, 파일 권한 변경 등을 자동으로 감지하고 알람을 발송하는 체계를 갖추는 것이 중요합니다.

-- 컨트롤 파일 상태 점검 쿼리 (정기 실행 권장)
SELECT 
    name                              AS controlfile_path,
    status                            AS status,
    is_recovery_dest_file             AS in_fra,
    block_size                        AS block_size,
    file_size_blks * block_size / 1024 AS size_kb
FROM v$controlfile;

-- alert log에서 컨트롤 파일 관련 에러 확인
SELECT originating_timestamp, message_text
FROM v$diag_alert_ext
WHERE message_text LIKE '%control%'
  AND originating_timestamp > SYSDATE - 1
ORDER BY originating_timestamp DESC;

#!/bin/bash
# 컨트롤 파일 존재 여부 모니터링 쉘 스크립트 (cron 등록 권장)
CTLFILES=("/u01/oradata/ORCL/control01.ctl" "/u02/oradata/ORCL/control02.ctl")
for f in "${CTLFILES[@]}"; do
    if [ ! -f "$f" ]; then
        echo "ALERT: Control file missing - $f" | mail -s "ORA-00202 RISK" [email protected]
    fi
done

관련 에러

• ORA-00200: 컨트롤 파일을 생성할 수 없을 때 발생하며, 주로 디스크 공간 부족이나 권한 문제와 연관됩니다.
• ORA-00201: 컨트롤 파일 버전이 Oracle 소프트웨어 버전과 호환되지 않을 때 발생합니다.
• ORA-00203: 잘못된 컨트롤 파일을 사용하고 있을 때 발생하며, 서로 다른 데이터베이스의 컨

Every async function you write assumes the network cooperates, the server responds, and the database doesn’t hiccup. In production, none of those assumptions hold forever.

Here are two higher-order functions — each under 15 lines — that make any async function resilient without touching its internals.

The problem

You have an async function. Maybe it calls an API, queries a database, or reads a file over the network.

async function fetchUserData(userId) {
    const response = await fetch(`/api/users/${userId}`);
    return response.json();
}

Two things will go wrong eventually:

1. It will fail intermittently and you’ll want to retry it
2. It will hang indefinitely and you’ll want to give up after a deadline

You could wrap every function in retry logic and timeout logic inline. Or you could write it once, properly, and wrap any function you want.

retry — automatic reattempts on failure

export function retry(count, callback) {
  return async function (...args) {
    let attempts = 0;
    let lastError;

    while (attempts <= count) {
      try {
        return await callback(...args);
      } catch (err) {
        lastError = err;
        attempts++;
      }
    }

    throw lastError;
  };
}

How it works

retry is a higher-order function — it takes a function and returns a new function with retry behaviour baked in. The original function is untouched.

The while (attempts <= count) condition is deliberate. If count is 3, the loop runs when attempts is 0, 1, 2, 3 — that’s 4 total executions: one initial attempt plus three retries. This matches the natural language meaning of “retry 3 times”.

On success, return await callback(...args) exits immediately — no more iterations. On failure, the error is stored in lastError and attempts increments. Once the loop exhausts all attempts, the last error is rethrown — not a generic new Error('Max retries reached'), but the actual error the callback produced. Your callers get a meaningful error message, not a wrapper.

Usage

const resilientFetch = retry(3, fetchUserData);

// Works exactly like fetchUserData, but retries up to 3 times on failure
const user = await resilientFetch('user_123');

Why await inside try matters

try {
    return await callback(...args); // ✓ catches rejected promises
} catch (err) { ... }

Without await, a rejected promise escapes the try/catch entirely:

try {
    return callback(...args); // ✗ returns a pending promise — catch never fires
} catch (err) { ... }

await unwraps the promise inside the try block, so rejections are catchable. This is one of the most common async/await mistakes and retry only works correctly because it gets this right.

timeout — give up after a deadline

export function timeout(delay, callback) {
  return async function (...args) {
    const timer = new Promise((_, reject) =>
      setTimeout(() => reject(new Error('timeout')), delay)
    );

    return Promise.race([callback(...args), timer]);
  };
}

How it works

Promise.race resolves or rejects with whichever promise settles first. This function creates a race between two competitors:

• callback(...args) — the actual work
• timer — a promise that rejects after delay milliseconds

If the callback finishes in time, its value wins and timer becomes irrelevant. If delay milliseconds pass first, timer rejects with Error('timeout') and the callback’s eventual result is ignored.

Notice the timer promise is constructed with (_, reject) — it never resolves, only rejects. This ensures the timer can never accidentally win the race with a successful value; it can only interrupt with a failure.

Usage

const limitedFetch = timeout(5000, fetchUserData);

try {
    const user = await limitedFetch('user_123');
} catch (e) {
    if (e.message === 'timeout') {
        console.error('Request took too long');
    }
}

Combining them

Both functions return async functions with the same signature as their input — which means they compose cleanly.

// Retry up to 3 times, but abandon any single attempt after 5 seconds
const resilientFetch = retry(3, timeout(5000, fetchUserData));

await resilientFetch('user_123');

Here’s what happens on each attempt:

1. timeout(5000, fetchUserData) races the fetch against a 5-second timer
2. If it times out, timeout rejects with Error('timeout')
3. retry catches that rejection, increments attempts, and tries again
4. After 3 retries all fail, retry rethrows the last error

Four attempts, each with a 5-second ceiling, maximum 20 seconds total. All from two composable functions and one line of setup.

What makes these worth keeping

They don’t modify the original function. fetchUserData is unchanged. You can use it with or without retry/timeout anywhere else.

They forward arguments transparently. ...args passes everything through — the wrapped function behaves identically to the original from the caller’s perspective.

They preserve the error. retry rethrows lastError, not a new generic error. timeout rejects with a named Error('timeout') you can check by message. Callers always know what actually went wrong.

They compose. Because both return async functions with matching signatures, you can layer them in any order and they work together without knowing about each other.

The pattern

Both functions follow the same structure:

higherOrderFn(config, callback) {
    return async function (...args) {
        // enhanced behaviour around callback(...args)
    }
}

This is the decorator pattern applied to async functions. You write the enhancement once, and apply it to any async function that needs it — no inheritance, no classes, no modification of the original. Just functions wrapping functions.

It’s a small pattern. It shows up everywhere once you start looking for it.

The Deployment Paradox: When Automation Becomes Repetition

Every developer has felt it: that moment of triumph after shipping a killer feature, followed immediately by the sinking feeling of impending “Deployment Hell.” You know the routine – fiddling with SSH keys, second-guessing environment variables, wrestling with asset manifests, and the ever-present prayer that everything will sync correctly. For years, I relied on Deployer, a truly excellent tool for automating PHP deployments. Yet, a frustrating pattern emerged: I was essentially rebuilding the same logic, task by task, for every single project. Whether it was a legacy Yii1 application or a shiny new Laravel 11 project, the custom deployment steps felt eerily familiar. It was clear I needed something more universal, a true force multiplier.

Engineering the “Force Multiplier”: From Copy-Paste to Abstraction

My goal wasn’t just to automate, but to abstract. I decided to distill years of hands-on DevOps experience into a single, robust package that could serve as a universal deployment engine. This led to the creation of the PHP Deployment Kit. This isn’t merely another library; it’s designed to be an intelligent, automated deployment workstation that handles the complex, often overlooked edge cases that standard deployment recipes typically gloss over. The aim is to eliminate repetitive tasks and provide a standardized, high-performance deployment workflow for virtually any PHP project.

Why This is a Game-Changer for PHP Deployments

What sets the PHP Deployment Kit apart is the built-in intelligence and the comprehensive nature of its tasks. It goes beyond basic file transfers and script execution to address critical, project-specific needs:

Vite Asset Reconciliation

One of the most persistent headaches, especially with modern JavaScript frameworks, is managing hashed asset files. Tools like Vite generate unique hashes for each asset to ensure cache busting. However, syncing these hashed assets perfectly with your application’s code and any dynamically generated content (like sitemaps or content management systems that reference assets) can be incredibly tricky. Standard deployment tools often struggle here, leading to broken links or outdated assets being served. The AssetMappingTask within the PHP Deployment Kit is engineered to solve this. It intelligently identifies and reconciles these hashed assets, ensuring that your application always references the correct, latest versions, regardless of whether the references are hardcoded or dynamically generated.

Environment Security at Deployment

Protecting sensitive information is paramount. The kit natively integrates with and supports advanced security features like Laravel’s environment file encryption. This means your secrets aren’t just sitting unencrypted in your deployment files. Instead, they are protected and only decrypted at the precise moment they are needed on the server during the deployment process. This adds a crucial layer of security, minimizing the attack surface and ensuring your credentials, API keys, and other sensitive data remain safeguarded throughout the deployment lifecycle.

Proactive Verification: Beyond Uploading Files

A successful deployment isn’t just about getting files onto the server; it’s about ensuring the application is actually functional and all critical components are accessible. The PHP Deployment Kit incorporates proactive verification steps. Before the deployment is officially declared a success, it performs checks such as:

• Webfont Accessibility: Ensures that essential webfonts are correctly uploaded and accessible by the browser, preventing broken typography.
• Sitemap Validity: Verifies that your sitemap.xml file is correctly generated and accessible, which is crucial for SEO.

These checks act as a final gatekeeper, providing confidence that the deployment has not only completed but has done so successfully and without introducing critical regressions.

Sensational Efficiency: Reclaiming Development Time

Since adopting the PHP Deployment Kit, the impact on my workflow has been dramatic. Deployment times have been reduced by an average of 40%. More significantly, the time spent setting up deployments for new projects has practically evaporated. The process is now as simple as including the kit, defining a few project-specific variables (like application name, repository URL, and target environment), and initiating the deployment.

Here’s a glimpse of how streamlined deploy.php has become:

php
set(‘deploy_path’, ‘/var/www/html’) // The target directory on the server
->set(‘branch’, ‘main’); // The Git branch to deploy

// The kit’s tasks are automatically configured and executed.
// You can still override or add custom tasks if needed, but often it’s not necessary.

?>

This level of abstraction means developers can focus less on the mechanics of deployment and more on building features. It transforms a often-dreaded task into a predictable, repeatable, and efficient process.

The Super-Genius Approach: From Coder to Architect

Building tools like the PHP Deployment Kit is, for me, what truly separates a “coder” from a “digital solutions architect.” It’s about identifying recurring friction points in the development lifecycle and engineering elegant, scalable solutions. By packaging this expertise into an open-source tool, the goal is to democratize access to high-tier DevOps infrastructure, empowering every PHP developer to deploy with confidence and efficiency.

This kit embodies the principle of working smarter, not just harder. It leverages automation and intelligent design to solve real-world development challenges.

👉 Read the complete deep-dive with the full code repository and bonus security checklist on klytron.com

https://klytron.com/blog/how-i-finally-conquered-deployment-hell-php-deployment-kit

AI coding agents are getting better, but a lot of repos still make them start from guesswork.

They land in a codebase and have to infer:

• which command installs dependencies
• which command starts the app
• which services need to be running
• which env vars matter
• which workflow is safe to run
• whether the repo is actually ready
• what “done” should look like after a change

That is a lot of operational knowledge to hide in scattered README sections, old shell scripts, CI files, tribal memory, and half-working local setup notes.

And when an agent guesses wrong, the problem is not always the agent.

Sometimes the repo never gave it a reliable answer.

AI Agents Amplify Repo Readiness

A human contributor can pause, read docs, ask questions, remember past setup quirks, and work around drift.

An AI agent usually does something else: it follows the strongest signal it can find.

If your README says one thing, your CI says another, your package scripts say a third, and your local setup depends on an undocumented service, the agent has to choose. Sometimes it chooses well. Sometimes it burns time. Sometimes it changes the wrong thing because the repo never made the safe path explicit.

That is why repo readiness matters.

Before asking an AI agent to work in a repo, the repo should be able to answer basic questions clearly:

• What does this repo need?
• How does it become ready?
• What tasks are safe to run?
• What checks prove the repo still works?
• What paths should an agent avoid?
• What workflow should be used for this kind of change?

If those answers are not explicit, every agent has to rediscover them from scratch.

README Instructions Are Not Enough

READMEs are useful for humans, but they are not enough as the only source of operational truth.

A README can explain intent, context, and contribution style. But repo readiness needs something more precise:

• declared runtimes
• declared tools
• declared services
• declared tasks
• declared workflows
• declared checks
• declared agent boundaries

That information should be machine-readable.

Not because humans do not matter, but because humans, CI, and agents should not each maintain separate versions of the same setup logic.

When setup knowledge only exists as prose, it drifts. When it drifts, automation becomes fragile. When automation becomes fragile, agents become less trustworthy.

The Repo Should Tell The Agent What Ready Means

A good agent workflow should not begin with:

“Look around and figure out how this repo works.”

It should begin with:

“Here is the repo contract. Diagnose readiness first. Use the declared workflow. Run the safe task. Respect protected paths.”

That is the difference between guessing and operating.

This is the layer I care about with Ota.

Ota gives a repo one explicit readiness contract: ota.yaml.

That contract can declare the repo’s tasks, checks, services, workflows, toolchains, env requirements, and agent-safe boundaries. Then humans, CI, and AI agents can all use the same source of truth.

For example:

ota doctor
ota workflows
ota tasks --workflow app
ota up --workflow app
ota run test

The point is not to replace good documentation.

The point is to stop making documentation carry operational truth alone.

What Agents Need From A Repo

An AI agent does not need magic. It needs clear boundaries.

A solid repo should be able to say:

agent:
  entrypoint: test
  default_task: test
  safe_tasks:
    - test
    - lint
    - typecheck
  protected_paths:
    - .env
    - secrets/**
    - production/**

The agent boundary tells agents which task to start from, which tasks are safe, and which paths are protected. The instruction is simple: run ota doctor first, then use declared safe tasks instead of guessing from repo scripts.

It should also be able to say what ready means:

workflows:
  default: app
  app:
    setup:
      task: setup
    run:
      task: dev
    readiness:
      checks:
        - app-ready

Now the agent is not guessing from random scripts.

It has a contract.

It knows the default workflow. It knows the safe tasks. It knows what not to touch. It knows which checks matter.

That is a better starting point for any coding agent.

CI Should Prove The Same Truth

Repo readiness should not only help local agents. It should also show up in CI.

If a repo has an explicit readiness contract, CI can prove that contract across environments:

name: ota-readiness

on:
  pull_request:
  push:

jobs:
  readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ota-run/setup@v1
      - run: ota doctor
      - run: ota validate .
      - run: ota up

That gives maintainers something concrete.

Not “the agent says it worked.”

A repeatable readiness check.

Why This Matters Now

AI agents are moving from demos into real repos.

That changes the bar.

A repo that is “understandable if you already know it” is not enough. A repo that “usually works after reading three docs pages” is not enough. A repo that depends on one maintainer’s memory is not enough.

If we want agents to make useful changes safely, repos need to become easier to reason about.

That means:

• fewer hidden setup steps
• fewer undocumented services
• fewer ambiguous commands
• fewer unsafe default paths
• more explicit readiness
• more machine-readable operational truth

The better your repo contract, the better every agent run becomes.

Get Started With Ota

Install Ota:

curl -fsSL https://dist.ota.run/install.sh | sh

Windows:

irm https://dist.ota.run/install.ps1 | iex

Then start with:

ota doctor
ota init --bootstrap
ota validate .
ota workflows
ota tasks

Useful links:

• Install Ota: ota.run/docs/install
• Ota docs: ota.run/docs
• Ota source code: github.com/ota-run/ota
• Example contracts: github.com/ota-run/examples
• GitHub Action: github.com/ota-run/action
• GitHub setup action: github.com/ota-run/setup

If your repo already has setup scripts, CI workflows, and contributor docs, Ota does not replace them blindly.

It helps turn the important parts into an explicit contract that humans, CI, and agents can all share.

Final Thought

AI agents should not have to reverse-engineer your repo every time they enter it.

Make the repo ready first.

Then let the agent work.

As digital environments grow more complex, manual data aggregation becomes a massive operational bottleneck. For enterprise MarTech architects and analytics engineers, building robust, automated pipelines to extract web data is critical for accurate SEO auditing, competitive analysis, and real-time dashboarding.

While many default to Python-based tools like Beautiful Soup or Scrapy for data mining, Power Query (embedded natively within Microsoft Excel and Power BI) offers an incredibly efficient, low-overhead alternative for enterprise data extraction pipelines.

In this technical guide, we will dive into advanced web scraping techniques using Power Query to automate data extraction workflows seamlessly.

Why Power Query for Web Extraction?

Power Query handles the foundational heavy lifting of ETL (Extract, Transform, Load) pipelines natively. Instead of managing external execution environments, database connections, and complex script dependencies, Power Query allows you to:

• Connect directly to live web endpoints.
• Parse structured and unstructured HTML tables effortlessly.
• Automate paginated data extraction using custom M-code logic.

1. Extracting Structured Web Tables

The simplest layer of web extraction involves targeting pre-rendered HTML data tables. Power Query makes this straightforward through its graphical interface:

1. Open Excel or Power BI and navigate to Data > From Web.
2. Paste your target destination URL.
3. Power Query’s Navigator will analyze the DOM tree and present discovered data tables.

For basic tables, the underlying automated M code looks similar to this:

let
    Source = Web.BrowserContents("[https://example.com/seo-audit-target](https://example.com/seo-audit-target)"),
    ExtractTable = Html.Table(Source, {{"Column1", "TABLE > TR > TD"}}, [RowStyle=RowStyle.All])
in
    ExtractTable

2. Advanced: Handling Pagination and Dynamic URLs

Real-world enterprise scraping requires navigating multi-page datasets (e.g., crawling thousands of search engine results or product indexes). To execute this without manual intervention, we can build a Custom Function in the Advanced Editor using M-code parameterization.
The Custom Loop Function
Open the Advanced Editor, create a new blank query, name it FxScrapePage, and insert the following function logic:

(pageNumber as number) as table =>
let
    // Dynamically inject the page parameter into the URL string
    TargetURL = "[https://example.com/directory?page=](https://example.com/directory?page=)" & Number.ToText(pageNumber),
    Source = Web.BrowserContents(TargetURL),

    // Extract targets using CSS selector mapping
    ParsedData = Html.Table(Source, {
        {"Title", ".article-title"},
        {"MetaDescription", ".meta-desc"},
        {"PublishDate", ".date-stamp"}
    }, [RowStyle=RowStyle.All])
in
    ParsedData

Invoking the Pipeline Across an Array
​Once your function is established, you can generate a list of target numbers (e.g., pages 1 through 50), convert that list into a standard data table, and invoke your custom function across the column.
​Power Query will iteratively execute the web requests, unpack the records, and consolidate the paginated data streams into a single, comprehensive dataset.

3. Data Transformation and Automated Cleansing

​Raw scraped data is rarely production-ready. Power Query excels at the transformation phase of the lifecycle. Within the query editor interface, you can chain steps to:
​Normalize Text Data: Convert mixed-case strings to clean lowercase text for uniform filtering.
​Filter Out Exceptions: Strip null entries, placeholder characters, or broken tracking strings.
​Parse Explicit Data Types: Safely cast text stamps into standard ISO dates or integers to prevent breaking down-stream analytics engines.

​Conclusion: Driving Business Value with Automated Auditing

​By building extraction workflows natively inside Power Query, you completely eliminate the friction between raw web data discovery and final business intelligence output. Your analytics dashboards can update automatically with fresh, scraped metrics with a simple background refresh click.

About the Author

​I am a Lead Digital MarTech Architect specializing in scaling enterprise digital platforms, building intelligent automation architectures, and optimizing data pipelines.

​To see more open-source engineering utilities, technical deep dives, and portfolio architecture setups, explore my engineering hub directly:

👉 Visit My Professional Portfolio

Every week I’d find myself doing the same thing. Googling “compress PDF Every week I’d find myself doing the same thing. Googling “compress PDF online free”, clicking the first result, uploading my file to some website I’d never heard of, and hoping for the best.

One day I actually read the privacy policy of one of these tools.

“Files retained for 24 hours. Processed on our servers. May be used to improve our services.”

That was enough for me.

The browser is more powerful than most people think

WebAssembly changed everything. It lets you run near-native code directly in a browser tab, no server needed.

FFmpeg, the same tool professionals use for video processing? Runs client-side. PDF manipulation? Client-side. Cryptographically secure password generation? The browser has had a native API for that for years.

So I started building.

54 tools later

What began as a simple video compressor turned into something much bigger. Every time I needed an online tool, I asked myself one question: could this run in the browser?

Almost always, the answer was yes.

The result: PureTools — 54 free utilities covering video, audio, PDF, images, passwords, finance calculators and developer tools.

• Zero uploads
• Zero watermarks
• Zero signup
• Works offline after the first visit

The stack

Embarrassingly simple. Pure HTML, CSS and vanilla JavaScript. No frameworks.

• FFmpeg.wasm for video and audio processing
• pdf-lib for PDF merge, split, compress and protect
• Web Crypto API for password generation
• Service Worker to cache everything offline

The hardest part

It wasn’t the code. It was resisting the urge to add a backend “just for analytics” or “just for user accounts”.

Every time I considered adding something server-side, I realized I was becoming the thing I was trying to replace.

The constraint became the product.

If you’re tired of uploading your files to random websites, give it a try.

Your files never leave your device. That’s the whole point.

If you’re tired of uploading your files to random websites, give it a try: puretools.io

Everything runs in your browser. Your files never leave your device.

That’s the whole point.