Written by an AI Pipeline, Verified by Three Models. Is It Slop?

How This Article Was Built (And Why I’m Showing You the Kitchen)

Disclaimer up front: I’m not entering the Hermes Agent challenge. I noticed the challenge and realized I could use my AI pipeline to write an article about Hermes Agent architecture. So I did. And thought, why not share both the result and the process that created it? What I actually want is your honest criticism.

Who Is The Author?

For the past several months I’ve been building Bestaiweb, navigating the shift from traditional development to AI. The site runs on Hugo, and the content is generated through what I call an AI content pipeline. The pipeline itself is built in TypeScript, orchestrated through Claude Code, and runs on Anthropic’s Claude models. Still in progress.

That phrase — “AI content pipeline” — probably triggered your slop detector. Fair. Let me explain why I think this case is different, and then let you judge.

The Pipeline

BestAIweb currently has 450+ technical articles across 45 topic clusters. Every article goes through a multi-phase pipeline:

  1. Market scanning — an LLM agent surveys the current tool and framework landscape for each topic, identifying what’s leading, what’s declining, and what’s emerging
  2. Query fan-out — the pipeline generates the questions a developer would actually search for, not the questions that sound good as headlines
  3. Research — a dedicated research agent gathers facts, version numbers, benchmark data, and source URLs. Everything gets a structured fact sheet
  4. Writing — here’s where personas come in. The pipeline has four author personas, each with a distinct voice and content type specialization:

    • MAX — the engineer. Writes step-by-step guides. Pragmatic, implementation-focused, opinionated about tool choices
    • MONA — the explainer. Breaks down concepts. Thinks in diagrams and mental models
    • DAN — the reporter. Covers news, market shifts, and what just shipped
    • ALAN — the critic. Writes opinion pieces and ethical assessments
  5. Claim verification — a separate agent cross-checks every factual claim against the research fact sheet. Unsupported claims get flagged
  6. Deterministic validation — a Python script runs 30+ structural and quality checks: word count, link integrity, frontmatter completeness, source coverage
  7. Hugo integration — the article lands in the static site with schema.org markup, generated images, and internal links

The Hermes Agent guide below was written by MAX using his guide template. His tone of voice is direct, specification-oriented, and allergic to hand-waving. The template enforces a fixed structure: prerequisites, numbered steps, pitfalls table, FAQ, and a deployable artifact at the end.

The Multi-Model Judging Layer

Pipeline generation was step one. Then came “a manual judging round”. I paste the draft into ChatGPT, Gemini, and DeepSeek and ask each to evaluate it as a technical reviewer — checking factual accuracy, logical gaps, tone inconsistencies, and whether the advice would actually work if someone followed it.I then reviewed their feedback together with Claude Code and incorporated the changes that held up under scrutiny.

The AI Slop Question

Here’s the question I keep circling back to: Is everything AI-generated inherently slop?

The reflexive answer in 2026 is “yes, obviously.” And for most AI-generated content, that’s correct. GPT-powered blog farms, SEO filler, those LinkedIn posts prompted with “write a thought leadership post about AI” — that is slop. Generated without specification, without sourcing, without verification, and without a quality gate.

But what about content where:

  • Every factual claim traces to a documented source (GitHub issues, official docs, arxiv papers)
  • A claim verification agent flags unsupported statements before publication
  • A deterministic validator enforces structural quality independent of the LLM
  • The voice and structure come from a multi-page specification, not a one-line prompt
  • Multiple independent models review the output for different failure modes

Is that still slop? Or is it closer to what a well-managed editorial team produces — except the heavy lifting is done by LLMs under human direction?

I genuinely don’t know the answer. That’s why I’m sharing this.

What I’d Like From You

Criticism. Specifically:

  • Does the article below read like AI slop? If yes, what gives it away — the sentence rhythm, the structure, the depth, or something else?
  • Is the technical content accurate? If you’ve deployed Hermes Agent or any persistent agent framework, does the three-layer model match your experience? Did I miss a critical failure mode?
  • Does the pipeline approach change anything? Is multi-phase generation with claim verification and multi-model judging enough to produce content worth reading? Or is it just expensive slop with better sourcing?

I’m not looking for “great article!” responses. I’m looking for the engineer who says “this is wrong because…” or “you missed the part where…” That feedback makes the next pipeline iteration better.

More Guides From the Same Pipeline

If you want to judge more output from the same pipeline and the same MAX persona, the full library has 95+ implementation guides from him, covering agents, RAG, training, inference, evaluation, and image generation guides.

What follows is the article as the pipeline produced it, after multi-model review. Judge for yourself.

How to Architect Always-On AI Agents with Hermes: Decompose, Specify, Deploy

TL;DR

  • Persistent agents need three specs your chatbot never did: memory policy, tool boundaries, and session recovery
  • Hermes Agent is model-agnostic — the model choice matters less than how you specify context, tools, and failure handling
  • Always-on means always-failing-somewhere — build validation into the deployment spec, not as an afterthought

You spun up Hermes Agent on a Friday evening. Gave it access to Slack, a web scraper, and your project database. Told it to “keep the team updated on competitor releases.” Monday morning: 47 Slack messages, three of them citing products that don’t exist, and a web scraper loop that burned through your OpenRouter credits overnight. The agent ran exactly as specified. The specification was the problem.

Before You Start

You’ll need:

  • A Linux or macOS server (even a $5 VPS works — Hermes Agent runs on minimal hardware)
  • An LLM provider account (OpenRouter, Anthropic, OpenAI, or a local runtime like Ollama)
  • Understanding of function calling — how models invoke external tools
  • A clear picture of what your agent should do when you’re not watching

This guide teaches you: How to decompose a persistent agent deployment into specifiable components so Hermes Agent does what you intended — not what you literally typed.

What this guide does NOT cover:

  • Production security hardening (firewall rules, secrets management, network isolation)
  • Enterprise compliance (SOC 2, GDPR data residency, audit certification)
  • Full evaluation frameworks (systematic benchmarking, regression test suites)
  • Model fine-tuning or training (Hermes models are pre-trained; this guide covers the agent framework)

The Agent That Worked Until It Didn’t

Here’s the pattern. Developer discovers Hermes Agent. Reads that it has persistent memory, self-improving skills, 20+ platform integrations. Installs it. Connects everything. Types a system prompt. Walks away.

Two things happen next. Either the agent does nothing useful because the specification was too vague. Or it does too much because the boundaries were never set.

According to Hermes Agent GitHub Issues, long sessions exceeding 700K tokens trigger environment hallucination — the agent confuses tool descriptions with actual environment state. It starts acting on what it thinks is true rather than what is true. This isn’t a bug in the traditional sense. It’s a specification gap. You never told the agent when to stop, reset, or ask for help.

Step 1: Map the Three Layers

Hermes Agent is not a single system. It’s three systems wearing a trench coat.

Your deployment has these parts:

  • The runtime layer — where the agent executes (Docker, SSH, Modal, local terminal). This determines resource limits, restart behavior, and isolation
  • The intelligence layer — the LLM provider and model. This determines reasoning quality, context window size, and cost per token
  • The integration layer — platform connections (Slack, Telegram, web tools) and the tools the agent can invoke. This determines what the agent can touch in the real world

The Architect’s Rule: If you can’t draw a clear line between what the agent thinks, where it runs, and what it touches — your spec is incomplete.

According to Hermes Agent Docs, the framework supports 30+ providers and 7 terminal backends. That flexibility is the point — and the trap. Every combination has different failure modes. A Modal serverless backend hibernates when idle. An Ollama local model defaults to 4K context tokens. An SSH backend loses the agent if the connection drops. You need to specify which combination you’re using and what happens at each boundary.

One thing the “always-on” framing obscures: what happens when the LLM provider goes down? OpenRouter has outages. API rate limits hit. Local models crash. An always-on agent needs a fallback plan — a secondary provider, a circuit breaker that pauses tool execution after N consecutive failures, or at minimum a notification that the agent is degraded. Specify this in the runtime layer, not as an afterthought.

Step 2: Lock Down the Context Contract

The intelligence layer needs a specification before it sees a single user message. This is where most deployments fail — not in the tools, not in the platform, but in the context that frames every decision the agent makes.

Context checklist:

  • System prompt with explicit role boundaries (what the agent does and does NOT do)
  • Memory policy: what gets persisted, what gets discarded, and when
  • Tool authorization with risk classification (see table below)
  • Access control: which platforms and channels can trigger the agent (not every DM deserves a response)
  • Session limits: when to compress or reset (Hermes Agent Docs default to auto-compression at 50% of the model’s context window, plus a hard ceiling of 400 messages)
  • Output format contracts: how the agent reports results on each platform
  • Rate limits: maximum messages per minute per platform (an agent with no rate limit is a spam bot waiting to happen)

Tool Risk Classification

An always-on agent with database access and Slack permissions is making autonomous decisions about your data and your team’s attention. Classify every tool before you enable it.

Risk Class Description Example Tools Authorization
read-only Observes, never modifies web_search, database_query (SELECT), file_read Auto-approved
reversible-write Creates or modifies, can be undone file_write, note_create, draft_message Auto-approved with audit log
irreversible-write Deletes or overwrites permanently file_delete, database_delete, channel_archive Requires human confirmation
external-send Sends to humans or external systems slack_post, email_send, webhook_trigger Rate-limited + audit log
billing-sensitive Incurs direct cost api_call (paid), image_generate, compute_spawn Budget ceiling + alert

The Spec Test: If your system prompt doesn’t mention what happens at 3 AM when the agent encounters an error and no human is online — you’ve specified a supervised agent and deployed it as unsupervised. If it doesn’t classify tool risk levels, the agent treats database_delete and web_search as equally safe. If it doesn’t set a compression trigger, the default (50% context window) may or may not match your workload.

Here’s what a minimal context contract looks like in practice. This is the MEMORY.md the agent reads on every session start:

# MEMORY.md — Agent Operating Contract
role: "Monitor competitor AI product releases for the engineering team"
boundaries:
  - "NEVER post to channels outside #competitor-monitoring"
  - "NEVER summarize or forward internal company data"
  - "NEVER execute irreversible-write tools without human confirmation"
  - "Maximum 3 Slack messages per hour"
tools:
  auto_approved: [web_search, file_read]
  rate_limited: [slack_post]  # max 3/hour
  requires_confirmation: [file_delete, database_write]
  forbidden: [email_send, channel_archive]
memory_policy:
  persist: "confirmed competitor releases, product names, dates"
  discard: "intermediate search results, draft summaries"
  compress_after: "50%"  # of context window
escalation: "If uncertain about any action, post to #agent-review instead"

A critical distinction: A memory or system-prompt policy is not a security boundary. Writing “NEVER execute irreversible-write tools” in MEMORY.md is a behavioral instruction to the model, not a technical lock. The model can ignore it — especially under long-context degradation or adversarial input. Destructive tools should be blocked or approval-gated at the runtime level (process permissions, API middleware, webhook filters), not merely discouraged in instructions. Treat the YAML above as the agent’s intent. Build enforcement outside the model.

According to Hermes Agent GitHub Issues, the persistent notes layer has a limit of roughly 2,200 characters. That’s the manually curated knowledge — not the agent’s entire memory. Hermes also maintains a full-text search index over past sessions and a per-person user model that evolves automatically. So the agent isn’t blind between sessions. But the notes layer is where you store hard constraints and project-critical context, and 2,200 characters fills up fast across three projects. You still need a compression strategy for notes — what gets stored verbatim, what moves to session history, what gets dropped.

Step 3: Wire the Components in Order

Deployment order matters. Each layer depends on the one below it.

Build order:

  1. Runtime first — because everything else crashes without a stable execution environment. Choose your backend, set resource limits, configure restart-on-failure
  2. Intelligence layer next — because tool and platform behavior depends on the model’s capabilities. According to Hermes Agent Docs, vLLM requires explicit --enable-auto-tool-choice and --tool-call-parser flags. Without them, the model outputs tool calls as plain text instead of executing them
  3. Integration layer last — because platform connections should only activate after the agent can reason and recover from errors. Connect Slack after the agent handles tool failures gracefully, not before

For each component, your specification must cover:

  • What it receives (inputs and triggers)
  • What it returns (outputs and side effects)
  • What it must NOT do (boundaries and prohibitions)
  • How it handles failure (retry logic, fallback behavior, human escalation)

The self-improving skills feature is powerful — Hermes Agent automatically creates workflow documents from successful task completions and refines them over time. But the skill creation itself needs a boundary spec. Without one, the agent writes skills for one-off tasks, cluttering the skill library with noise.

Skill boundary example — add this to your system prompt:

skills_policy:
  auto_create: ["competitor-monitoring", "weekly-summary", "data-formatting"]
  never_create: ["one-off-queries", "debugging-sessions", "ad-hoc-searches"]
  review_before_use: ["any skill not used in 14+ days"]
  max_skills: 20  # force deduplication when library exceeds this

Without this, the agent treats every successful task as a reusable pattern. Three months in, you have 200 skills — most of them variations of the same web search with slightly different parameters.

One more thing about skills: they can regress. A skill written for Hermes-3-8B may produce wrong tool calls after switching to a different model. A skill that relies on a specific API endpoint breaks when that endpoint changes. Skills older than 30 days should be re-validated or archived. The review_before_use field above is your safety net — but only if you actually review them.

Step 4: Prove It’s Actually Working

Running the agent is not validation. Validation means you know what “correct” looks like and can detect when the agent drifts from it.

Validation checklist:

  • Memory consistency — after 24 hours, does the agent’s memory reflect reality? Failure looks like: agent references a “completed” task that was never finished, or forgets a constraint you set yesterday
  • Tool call accuracy — are tool invocations well-formed and targeted? Failure looks like: invalid function names, malformed arguments, or calls to tools that aren’t registered. This is a general problem with LLM-driven tool use, not Hermes-specific — any agent framework that delegates tool selection to a model will hit it. Hermes Agent GitHub Issues documents concrete examples like todo:list calls that don’t match any schema
  • Platform output quality — are messages to Slack/Telegram/Discord useful and accurate? Failure looks like: hallucinated product names, duplicate messages, or empty responses
  • Cost trajectory — is daily token usage stable or growing? Failure looks like: runaway context accumulation driving costs up 10x within a week

Common Pitfalls

What You Did Why the Agent Failed The Fix
One-shot system prompt: “monitor competitors” No boundaries — agent decides scope, frequency, and format Decompose into: what to monitor, how often, where to report, what format
Connected all tools on day one Agent uses tools in unexpected combinations Enable tools incrementally, validate each before adding the next
Chose a 4K-context local model Tool schemas + system prompt + memory exceed context Use minimum 16K–32K context for tool-calling workloads
No session hygiene policy 700K+ token sessions trigger hallucination loops Use Hermes built-in compression (default: 50% context window) and set a hard message ceiling. Monitor context growth.
Skipped memory policy Agent stores everything, including noise Specify what gets persisted: decisions, outcomes, blockers. Not intermediate reasoning

Pro Tip

The specification you write for Hermes Agent is not a prompt. It’s an operating manual for an unsupervised system. The same decomposition — runtime, intelligence, integration — works for any persistent agent, regardless of framework. The tools change. The layers don’t.

Frequently Asked Questions

Q: How does Hermes Agent’s persistent memory differ from conversation history?
A: Conversation history is a raw log that grows until it hits the context window limit. Hermes uses three structured layers: persistent notes you curate manually, a full-text search index over past sessions, and a user model that evolves per-person. The practical difference — session history gets summarized and compressed, while persistent notes survive indefinitely. Watch for the 2,200-character limit on notes: it forces disciplined compression.

Q: Can I run Hermes Agent with local models instead of cloud API providers?
A: Yes — Ollama, vLLM, SGLang, llama.cpp, and LM Studio all work as backends. The catch is context window configuration. Ollama defaults to 4K tokens, which isn’t enough once you add tool schemas and system prompts. Set the context window explicitly to at least 16K on the server side. For vLLM, you also need the --enable-auto-tool-choice flag or tool calls render as text.

Q: What context window size does Hermes Agent need for reliable tool calling?
A: According to Hermes Agent Docs, minimum 16K–32K tokens for agent workloads with tools. The system prompt, tool schemas, memory context, and conversation history all compete for the same window. With 5+ tools registered, 32K is the safer starting point. Below that, the model starts dropping tool definitions mid-session.

Q: How do I prevent hallucination loops in long-running Hermes Agent sessions?
A: Hermes has built-in session compression — by default it triggers at 50% of the model’s context window, with a hard ceiling of 400 messages. According to Hermes Agent Docs, these thresholds are configurable. The documented failure zone is 700K+ tokens, where environment hallucination has been observed. Keep compression active, tune the trigger percentage for your workload, and monitor for repeated identical tool calls — that’s the earliest signal of a loop forming. Store critical state in persistent notes before any forced reset.

Your Spec Artifact

By the end of this guide, you should have:

  • A three-layer deployment map — runtime, intelligence, and integration with explicit boundaries between each
  • A context contract with tool risk classification — system prompt, memory policy, tool authorization by risk class, access control, rate limits, and output format per platform
  • A security baseline — tool isolation, rate limiting, audit logging, and escalation paths
  • A validation checklist — memory consistency, tool call accuracy, output quality, and cost trajectory checks you run daily

Your Deployment Spec Prompt

This prompt generates a first draft of your agent specification — not a production-ready deployment. Paste it into Claude Code, Cursor, or your preferred AI coding tool. Fill in every bracketed placeholder with your specific values from Steps 1-4.

I'm specifying a Hermes Agent deployment. Generate a first-draft specification
based on these inputs. I will review and harden it before production use.

RUNTIME LAYER:
- Backend: [Docker / SSH / Modal / local — pick one]
- Resource limits: [RAM, CPU cores, disk]
- Restart policy: [on-failure / always / manual]
- Server: [OS, VPS provider, specs]

INTELLIGENCE LAYER:
- LLM provider: [OpenRouter / Anthropic / Ollama / vLLM — pick one]
- Model: [model name and size]
- Context window: [minimum 16K — specify exact value]
- Provider-specific flags: [e.g., --enable-auto-tool-choice for vLLM]

INTEGRATION LAYER:
- Platforms: [Slack / Telegram / Discord — list all]
- Allowed trigger channels: [e.g., only #competitor-monitoring, not DMs]
- Tools by risk class:
  - read-only (auto-approved): [web_search, file_read, database SELECT]
  - reversible-write (auto + audit): [file_write, note_create]
  - irreversible-write (human approval): [file_delete, database DELETE]
  - external-send (rate-limited): [slack_post — max messages/hour]
  - billing-sensitive (budget ceiling): [paid API calls — max $/day]
- Tools forbidden: [list tools the agent must never invoke]

CONTEXT CONTRACT:
- Agent role: [one sentence — what this agent does]
- Explicit boundaries: [what the agent must NOT do, stated as prohibitions]
- Memory policy: [what gets persisted, what gets discarded, compression rules]
- Compression trigger: [percentage of context window — default 50%]
- Hard message ceiling: [number — default 400]
- Output format per platform: [e.g., Slack = bullet points, email = report]
- Skill boundary: [which task categories auto-generate skills, which don't]

SECURITY & PERMISSIONS:
- Access control: [which platforms/channels can trigger the agent]
- Rate limits per platform: [messages per minute/hour]
- Destructive action policy: [never auto-approve / require confirmation / forbidden]
- Audit log location: [where tool calls + results are logged]

OBSERVABILITY:
- Log format: [timestamp, tool name, input summary, output status, cost estimate]
- Loop detection: [alert on N repeated identical tool calls within M minutes]
- Cost alerts: [alert when daily spend exceeds $X]
- Error spike alerts: [alert when tool error rate exceeds X% in Y minutes]

DRY RUN:
- Generate a dry-run mode where all external-send and write tools are simulated
- Include 5 test scenarios that exercise each risk class

VALIDATION:
- How to verify memory consistency after [24h / 48h / 7d]
- Expected daily token usage range: [min–max tokens]
- Escalation trigger: [what condition sends an alert to a human]

RULES FOR GENERATION:
- Do not invent Hermes-specific configuration fields. If Hermes does not
  support a field natively, label it as "external wrapper / policy layer
  required".
- For every generated config field, mark one of:
  [native] — Hermes Agent built-in setting
  [prompt] — system prompt / MEMORY.md behavioral instruction
  [external] — requires runtime middleware, API gateway, or wrapper script
  [manual] — operational checklist item, not automatable
- Before generating final output, separate policy from enforcement:
  - What the model is instructed to do (behavioral, can be ignored)
  - What the runtime technically prevents (enforced, cannot be bypassed)
  - What requires human approval (gated)
  - What is only monitored after the fact (observable but not blocked)

Generate:
1. The MEMORY.md agent operating contract (see article for format example)
2. The tool authorization config with risk classifications (each field tagged
   as native / prompt / external / manual)
3. A daily validation checklist
4. Cost and error monitoring alert thresholds
5. A dry-run test plan with 5 scenarios

Ship It

You now have a framework for specifying persistent agents that doesn’t depend on Hermes Agent specifically — the three-layer model works for any long-running AI system. The difference between an agent that helps and one that burns your credits at 3 AM is never the model. It’s the spec.

Different Perspectives

From the architecture side: The three-layer decomposition maps cleanly to isolation boundaries in distributed systems. Runtime is the execution substrate. Intelligence is the reasoning process. Integration is the I/O surface. What makes persistent agents architecturally distinct from request-response chatbots is that all three layers maintain state across invocations — and state synchronization between layers is where failure modes cluster. The memory limit finding is telling: the notes layer caps at 2,200 characters while session search and user modeling compensate, but the degradation curve of each layer matters more than the initial capability.

From the market side: The adoption velocity here is real — 157K GitHub stars in under four months signals a market that was waiting for open-source persistent agents. The competitive positioning against Claude Code and OpenAI Agents SDK is smart: Hermes doesn’t compete on code quality or API simplicity, it competes on uptime and learning. The $5-80/month self-hosted cost structure undercuts every managed alternative. Watch for the enterprise play — the moment Nous Research ships team memory sharing, this becomes an infrastructure layer, not a developer tool.

From the governance side: The specification gap described above is a governance gap by another name. An always-on agent with tool access and persistent memory is making autonomous decisions on behalf of someone — and the specification determines whose values it encodes. The hallucination loop at 700K tokens is not just a technical failure. It’s an agent acting on a reality that doesn’t exist, with real-world consequences on the platforms it’s connected to. Who reviews the specification before deployment? Who monitors drift between what was specified and what the agent learned? The self-improving skills feature means the agent’s behavior changes over time without human approval. At what scale does that become a problem?

Sources

Part1 Vulkan: Drawing Triangle 1

This is not a guide, just me writing down my notes
This fallows the Vulkan Tutorial link

Some details I learned along the way

1.Base Code section

  • This is more related with Cpp than Vulkan but I’m just gonna write it as a reminder to myself. Members in cpp or oop means the components in the class which are (filed/variable) and (method/function)

  • When objects are created it uses vkCreateXXX or allocate using vkAllocateXXX. However when these objects are no longer needed, these are destroyed using vkDestroyXXX and vkFreeXXX

  • pAllocator. This is an optional parameter that allows you to specify callbacks for a custom memory allocator. We will ignore this parameter in the tutorial and always pass nullptr as argument.(cause we are beginners)

2.Instance section

  • Instance is the connection between “Your Application” and “The Vulkan Library”. Also it initializes the Vulkan Library

  • VkApplicationInfo –> this just specify application information. This code is not really required but it provides some useful info to dirver.

//not really required but still provide the info needed for the dirver.
void createInstance() {
    VkApplicationInfo appInfo{};
    appInfo.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO;
    appInfo.pApplicationName = "Hello Triangle";
    appInfo.applicationVersion = VK_MAKE_VERSION(1, 0, 0);
    appInfo.pEngineName = "No Engine";
    appInfo.engineVersion = VK_MAKE_VERSION(1, 0, 0);
    appInfo.apiVersion = VK_API_VERSION_1_0;
}
  • VkInstanceCreateInfo –> this is not optional and tells the Vulkan driver which global extensions and validation layers we want to use. You don’t have to know what global extension and validation layers are because they will be mentioned in next chapter.
//this creates the info full detail is in the end of the code.
//for now don't think to much about this.

VkInstanceCreateInfo createInfo{};
createInfo.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO;
createInfo.pApplicationInfo = &appInfo;

-vkCreateInstance –> now the important part! we are going to store the “createInfo” inside the “vkInstance instance variable”. Not complicated!!

//What is "nullptr" inside the second parameter???
//that is the callback which we will not use cause... I mentioned this before
VkResult result = vkCreateInstance(&createInfo, nullptr, &instance);
  • vkEnumerateInstanceExtensionProperties –> this just retrieves the supported extension before actually creating an instance(not really required but cool to have/checks which support is missing)
    uint32_t extensionCount = 0;
    //you can just think of this like an array
    //where inside the extensions vector the information about supported extensions will be stored annnnd 
    //it could be accesed by using index named extension count
    std::vector<VkExtensionProperties> extensions(extensionCount);
    //this like mentioned above the actual supported extensions details will bbe stored in extensions vector
    //------explaination of the code-------
    //vkEnumerateInstanceExtensionProperties(the callback which we will not be using, the address of "number of extensions", the actual vector data);
    vkEnumerateInstanceExtensionProperties(nullptr, &extensionCount, extensions.data());

    //now actually printing
    std::cout << "available extensions:n";
    //this is not realted with vulkan but still gonna write it down
    //(const auto& extension : extensions)
    // ":" --> bring the first value in extensions vecotr and put it inside extension variable
    // auto --> let computer choose what data type will the extension be
    // & point to the actual value of extension variable do not make a copy of extension varaible
    for (const auto& extension : extensions) {
        std::cout << 't' << extension.extensionName << 'n';
    }

The Full Code

#include <vector>
#define GLFW_INCLUDE_VULKAN
#include <GLFW/glfw3.h>
//the vulkan header has been replaced becuase the GLFW will call it automatically
#include <iostream>
#include <cstdlib>

//this is a constant definition
//uint32_t is 32bit integer
const uint32_t WIDTH = 800;
const uint32_t HEIGHT = 600;

class HelloTriangleApplication {
public:
    void run() {
    //add this method initWindow() to run
    initWindow();
        initVulkan();
        mainLoop();
        cleanup();
    }

private:
    //write a private member that creates the actual window
    GLFWwindow* window;
    //a private member that hold the instance
    VkInstance instance;

    void initWindow(){
    //the very first call is glfwInit();
    //why?? you may ask because we have to tell the glfw it is not an OpenGL and should tell glfw to NOT create OpenGL Context
    glfwInit();
    //not sure what these 2 does
    glfwWindowHint(GLFW_CLIENT_API, GLFW_NO_API);
    glfwWindowHint(GLFW_RESIZABLE, GLFW_FALSE);
    //this as you know creates an actual window
    //detail of this line
    //window = glfwCreateWindow(the width, the height, the title, which monitor to open the window, this is only relevent to OpenGL);
    window = glfwCreateWindow(WIDTH, HEIGHT, "Vulkan", nullptr, nullptr);
    }

    void initVulkan() {
    //a self made function that creates an instance
    createInstance();
    }

    void mainLoop() {
    //run the application untill the window is closed or an error occurs
        while (!glfwWindowShouldClose(window)) {
        glfwPollEvents();
    }
    }
    //we need to clean up resources
    void cleanup() {
    //destroyes the instance
    vkDestroyInstance(instance, nullptr );
    //this destroyes the window
    glfwDestroyWindow(window);

    glfwTerminate();
    }

    void createInstance() {
    //VkApplicationInfo-Structure specifiying application information
    //this whole code is kind of optional but it provieds some useful information to the driver
    VkApplicationInfo appInfo{};
    appInfo.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO;
    appInfo.pApplicationName = "Hello Triangle";
    appInfo.applicationVersion = VK_MAKE_VERSION(1, 0, 0);
    appInfo.pEngineName = "No Engine";
    appInfo.engineVersion = VK_MAKE_VERSION(1, 0, 0);
    appInfo.apiVersion = VK_API_VERSION_1_0;

    //the code below is not optional and tells the Vulkan driver which global extensions and validation layers we want to use. 
    //You don't have to know what global extension and validation layers are 
    //because they will be mentioned in next chapter

    VkInstanceCreateInfo createInfo{};
    createInfo.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO;
    createInfo.pApplicationInfo = &appInfo;

    //---------EXTENSION PART---------

    //how many extension do we need??
    uint32_t glfwExtensionCount = 0;
    //a storage for extension names
    const char** glfwExtensions;

    //this basically fills in two variable we mentioned
    //1. glfwExtensions
    //2. glfwExtensionCount
    //Asks GLFW: "How many extensions do we need??? and which one??"
    glfwExtensions = glfwGetRequiredInstanceExtensions(&glfwExtensionCount);

    //we put the "count" and the "name tag" inside the VkInstanceCreateInfo createInfo we mentioned earlier
    createInfo.enabledExtensionCount = glfwExtensionCount;
    createInfo.ppEnabledExtensionNames = glfwExtensions;

    //we are not touching this for now
    createInfo.enabledLayerCount = 0;

    //this is like giving information "Which stuff I am going to use to make Vulkan Project!!!"  
    //1.the first parameter is the pointer to the createInfo. See the code we have been putting information about "Extensions" in the createInfo.
    //2.the second paramter is the callback which we don't have to know cause it will always stay nullptr.
    //3.remember the "Vkinstance instance;" we mentioned earlier outside private and public??? we are going to put the created instance inside that variable.
    VkResult result = vkCreateInstance(&createInfo, nullptr, &instance);

    //the error checker
    if (vkCreateInstance(&createInfo, nullptr, &instance) != VK_SUCCESS) {
        throw std::runtime_error("failed to create instance!");
    }

    uint32_t extensionCount = 0;
    //you can just think of this like an array
    //where inside the extensions vector the information about supported extensions will be stored annnnd 
    //it could be accesed by using index named extension count
    std::vector<VkExtensionProperties> extensions(extensionCount);
    //this like mentioned above the actual supported extensions details will bbe stored in extensions vector
    //------explaination of the code-------
    //vkEnumerateInstanceExtensionProperties(the callback which we will not be using, the address of "number of extensions", the actual vector data);
    vkEnumerateInstanceExtensionProperties(nullptr, &extensionCount, extensions.data());

    //now actually printing
    std::cout << "available extensions:n";
    //this is not realted with vulkan but still gonna write it down
    //(const auto& extension : extensions)
    // ":" --> bring the first value in extensions vecotr and put it inside extension variable
    // auto --> let computer choose what data type will the extension be
    // & point to the actual value of extension variable do not make a copy of extension varaible
    for (const auto& extension : extensions) {
        std::cout << 't' << extension.extensionName << 'n';
    }
    }
};

int main() {
    HelloTriangleApplication app;

    try {
        app.run();
    } catch (const std::exception& e) {
        std::cerr << e.what() << std::endl;
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

part2 not here yet

Why I Stopped Using useEffect to Sync State — and What I Use Instead

Why I Stopped Using useEffect to Sync State — and What I Use Instead

I made a mistake that, I suspect, most teams working with React are still making today: I used useEffect as a general-purpose state synchronization tool. If something changed and I wanted to react to it, there was the effect. Clean, familiar, and — I figured this out way too late — completely wrong for most of those cases.

I’m not telling you this to flagellate myself. I’m telling you because when I systematically audited the effects in a React 19 codebase, I found exactly four categories of misuse repeated over and over. Each one has a better solution. And none of the four requires hacks or external libraries.

My thesis: useEffect isn’t broken. What’s broken is the mental model we teach alongside it — as if it were the natural place to “do things when something changes.” React 19 puts better tools closer to the surface, but you still need the judgment to choose between them. Without that, React 19 just gives you new places to put the same old problems.

useEffect for state synchronization: the antipattern nobody names

The official React documentation has an entire page called “You Might Not Need an Effect” that anyone should read before writing their first effect. It’s not an opinionated blog post — it’s official documentation from the React team. And it says explicitly that using effects to transform data during render is wrong.

The core problem is that useEffect runs after the render. When you use it to derive or sync state from props or existing state, you’re forcing an extra cycle: render → effect → setState → render again. That’s visual noise, momentary inconsistencies, and a dependency graph that becomes impossible to follow.

Look at this classic pattern I found repeated everywhere:

// ❌ useEffect to derive state — documented antipattern by React
const [items, setItems] = useState<Item[]>([]);
const [filteredItems, setFilteredItems] = useState<Item[]>([]);

useEffect(() => {
  // Every time items changes, we filter
  // This generates an unnecessary extra render
  setFilteredItems(items.filter(item => item.active));
}, [items]);

This effect looks reasonable. It isn’t. It generates two renders when one is enough. And if items comes from an async fetch, the intermediate state where filteredItems is stale is completely visible to the user.

The solution is to not have filteredItems as state at all:

// ✅ Derived state — calculated during render, no effect needed
const [items, setItems] = useState<Item[]>([]);

// Recalculated on every render that involves items
// If the calculation is expensive, useMemo is the tool — not useEffect
const filteredItems = items.filter(item => item.active);

If the filter is computationally expensive, useMemo with the right dependency. If it’s cheap — and most are — not even that. Just a direct calculation during render.

The 4 concrete categories and what replaces them

1. State derived from props or existing state → calculation in render or useMemo

We already saw this above. The practical rule: if you can calculate something from the state or props you already have, it’s not new state. It’s a function of what already exists.

// ❌ Version with effect
const [user, setUser] = useState<User | null>(null);
const [displayName, setDisplayName] = useState('');

useEffect(() => {
  setDisplayName(user ? `${user.firstName} ${user.lastName}` : 'Guest');
}, [user]);

// ✅ Correct version: derived inline
const displayName = user ? `${user.firstName} ${user.lastName}` : 'Guest';

Looks trivial. In a real codebase with 30 components doing this, the cumulative impact on re-renders is anything but trivial.

2. Post-event synchronization → direct event handlers

Another pattern I kept finding: an effect that watches a piece of state to “do something when it changes,” but that change always comes from a user interaction.

// ❌ useEffect reacting to a change that only ever comes from a click
const [selectedId, setSelectedId] = useState<string | null>(null);

useEffect(() => {
  if (selectedId) {
    analytics.track('item_selected', { id: selectedId });
    loadDetails(selectedId);
  }
}, [selectedId]);

// ✅ The event handler already knows everything it needs to know
const handleSelect = (id: string) => {
  setSelectedId(id);
  // The logic that reacts to the event goes HERE, not in an effect
  analytics.track('item_selected', { id });
  loadDetails(id);
};

The difference isn’t just aesthetic. With the effect, any change to selectedId — even a programmatic one, even from another effect — triggers the logic. With the handler, the intent is explicit: this happens when the user selects something. Fewer surprises, more control.

The React docs put it plainly: if something happens because the user did something, it goes in the event handler. If something happens because the component was shown, it goes in an effect. The confusion between these two cases is the root of most useEffect bugs I see.

3. Data fetching → use() with Suspense in React 19, or Server Components

This is the category where React 19 changes the game the most. Fetching with useEffect is the most copy-pasted pattern on the internet and one of the most problematic:

// ❌ The classic useEffect fetch — race conditions waiting to happen
const [data, setData] = useState(null);
const [loading, setLoading] = useState(true);

useEffect(() => {
  // Without cleanup, this can set state on an unmounted component
  fetch(`/api/items/${id}`)
    .then(r => r.json())
    .then(setData)
    .finally(() => setLoading(false));
}, [id]);

This code has a classic race condition: if id changes quickly, two fetches run in parallel and the result can arrive in any order. You can mitigate it with cleanup, but that’s boilerplate most people skip.

React 19 brings use() which integrates with Suspense:

// ✅ React 19: use() + Suspense — no useEffect, no manual loading state
import { use, Suspense } from 'react';

// The promise is created outside the component or passed as a prop
function ItemDetail({ itemPromise }: { itemPromise: Promise<Item> }) {
  // use() suspends the component until the promise resolves
  const item = use(itemPromise);

  return <div>{item.name}</div>;
}

// In the parent component:
function Page({ id }: { id: string }) {
  // The promise is created here, React manages the lifecycle
  const itemPromise = fetchItem(id); // function that returns Promise<Item>

  return (
    <Suspense fallback={<Skeleton />}>
      <ItemDetail itemPromise={itemPromise} />
    </Suspense>
  );
}

But if you’re working with App Router in Next.js 16, the most honest answer is: use Server Components for the fetch. The data arrives serialized to the client, no loading state, no race conditions, no useEffect. For more complex cases where the fetch depends on user interaction, use() with Suspense is the way.

This connects to something I documented in the post about Prisma Server Actions in Next.js 16 — the boundary between what runs on the server and what runs on the client significantly shifts which patterns actually make sense.

4. Post-submit transformations → Server Actions

The last category: the effect that listens to the result of a submit to update derived state, show messages, or redirect.

// ❌ useEffect watching the result of a submit
const [submitResult, setSubmitResult] = useState<Result | null>(null);
const [errorMessage, setErrorMessage] = useState('');

useEffect(() => {
  if (submitResult?.error) {
    setErrorMessage(submitResult.error.message);
  }
}, [submitResult]);

With Server Actions in React 19 and the useActionState hook (previously useFormState), this collapses into a much more direct pattern:

// ✅ useActionState — form state and action result, integrated
import { useActionState } from 'react';

async function submitForm(prevState: State, formData: FormData): Promise<State> {
  'use server';
  // The action runs on the server, returns the new state
  const result = await processForm(formData);
  if (!result.ok) {
    return { error: result.message };
  }
  return { success: true };
}

function MyForm() {
  const [state, action, isPending] = useActionState(submitForm, { error: null });

  return (
    <form action={action}>
      {/* No useEffect, no extra state, no manual synchronization */}
      {state.error && <p className="error">{state.error}</p>}
      <button disabled={isPending}>Save</button>
    </form>
  );
}

Form state, error feedback, and loading are all integrated without a single useEffect. It’s not magic — it’s that the responsibility is in the right place.

The errors that persist and the ones that disappear

There are cases where useEffect is the right tool: subscriptions to external stores, synchronization with DOM APIs you don’t control (a third-party map, a canvas library), or setup/teardown of resources that exist outside of React’s model.

What disappears with these patterns:

  • Race conditions in fetch — use() and Server Components eliminate them structurally
  • Inconsistent intermediate renders — derived state has no inconsistent state because it isn’t state
  • Chained effect graphs — when one effect sets state that fires another effect, debugging is hell. Event handlers cut that off at the root
  • Forgotten cleanup — if there’s no effect, there’s no cleanup to forget

What doesn’t disappear:

  • Thinking about dependencies — useMemo also has a dependency array. use() requires understanding how React handles promises. The judgment is still necessary.
  • Suspense complexity — badly placed boundaries break UX in non-obvious ways. It’s not an automatic replacement.

FAQ

Did useEffect become obsolete in React 19?

No. The React team didn’t deprecate it or mark it as legacy. What changed is that React 19 puts better tools within reach for the cases where useEffect was previously the only available option. For subscriptions, integration with external DOM APIs, and synchronization with systems outside of React’s model, useEffect is still correct.

Does use() replace useEffect for all fetches?

For fetches in client components that depend on user interaction, use() with Suspense is a concrete alternative. For initial data fetches in App Router, Server Components are the most direct answer — neither use() nor useEffect. The choice depends on whether the data can be resolved on the server or needs to wait for the client.

When to use useMemo vs direct calculation for derived state?

Direct calculation first. useMemo only when the calculation is measurably expensive (sort/filter over large arrays, complex transformations) and the profiler confirms there’s an actual problem. Adding useMemo preemptively is just another form of over-engineering — it has a readability cost and the dependencies can also be wrong.

Does useActionState work without Server Actions?

Yes. useActionState accepts any async function, not just Server Actions. If you prefer to handle the submit on the client side, you can pass it a client-side function. The integration with Server Actions is the cleanest in Next.js with App Router, but it’s not a requirement.

How do I migrate an existing codebase? Is there an incremental path?

The safest path is by category, not by file. Start by identifying all the useEffect calls that only derive state — they’re the safest to migrate and give the most immediate benefit. Then the ones that react to user events. Fetches last, because they require decisions about Suspense boundaries that affect UX.

Do these patterns change if I’m using Zustand, Jotai, or Redux?

Partially. External stores solve the global state problem, but the antipattern of deriving state inside a component with useEffect shows up just the same. The question “can I calculate this during render?” applies regardless of which state management system you’re using.

The judgment that doesn’t come from the framework

What frustrated me most when I audited this codebase wasn’t finding the antipatterns — that was expected. It was realizing they were there because at the time nobody had a clear rule for deciding when to use useEffect. We used it as the default tool for “do something when something changes,” and that mental model is wrong from the start.

React 19 makes it easier to do the right thing: use() exists, useActionState exists, Server Components are more integrated. But without the judgment of when each tool applies, what happens is that the same problems just migrate to the new APIs.

My concrete take: before writing a useEffect, ask yourself whether what you want to do is (a) calculate something from existing state, (b) react to a user action, (c) load data, or (d) sync with something external to React. Only the last case justifies an effect. The first three have better solutions in React 19, and the official React documentation says so explicitly.

If you’re auditing a codebase and don’t know where to start, the same audit exercise applies to other patterns: the N+1 that shows up when you least expect it in Prisma has the same structure — a pattern that seems reasonable until you look at it with the right question.

Original source:

Por qué dejé de usar useEffect para sincronizar estado y qué uso ahora

Por qué dejé de usar useEffect para sincronizar estado y qué uso ahora

Cometí un error que, sospecho, la mayoría de los equipos que trabajan con React sigue cometiendo hoy: usé useEffect como una herramienta de sincronización de estado de propósito general. Si algo cambiaba y yo quería reaccionar, ahí estaba el efecto. Limpio, familiar, y — lo entendí tarde — completamente equivocado para la mayoría de esos casos.

No lo cuento para flagelante. Lo cuento porque cuando revisé sistemáticamente los efectos de una codebase en React 19, encontré exactamente cuatro categorías de mal uso repetidas. Cada una tiene una solución mejor. Y ninguna de las cuatro requiere hacks ni librerías externas.

Mi tesis: useEffect no está roto. Lo que está roto es el modelo mental con el que lo enseñamos — como si fuera el lugar natural para “hacer cosas cuando algo cambia”. React 19 pone herramientas mejores más cerca de la superficie, pero el criterio para elegir entre ellas sigue siendo necesario. Sin eso, React 19 solo te da nuevos lugares donde meter los mismos problemas.

useEffect para sincronizar estado: el antipatrón que nadie nombra

La documentación oficial de React tiene una página entera llamada “You Might Not Need an Effect” que debería leer cualquiera antes de escribir su primer efecto. No es un post de blog opinado — es documentación oficial del equipo de React. Y dice explícitamente que usar efectos para transformar datos durante el render es incorrecto.

El problema de fondo es que useEffect corre después del render. Cuando lo usás para derivar o sincronizar estado a partir de props o estado existente, estás provocando un ciclo extra: render → efecto → setState → render de nuevo. Eso es ruido visual, inconsistencias momentáneas y un grafo de dependencias que se vuelve imposible de seguir.

Mirá este patrón clásico que encontré repetido:

// ❌ useEffect para derivar estado — antipatrón documentado por React
const [items, setItems] = useState<Item[]>([]);
const [filteredItems, setFilteredItems] = useState<Item[]>([]);

useEffect(() => {
  // Cada vez que items cambia, filtramos
  // Esto genera un render extra innecesario
  setFilteredItems(items.filter(item => item.active));
}, [items]);

Este efecto parece razonable. No lo es. Genera dos renders cuando uno alcanza. Y si items viene de un fetch asíncrono, el estado intermedio donde filteredItems está desactualizado es completamente visible.

La solución es no tener filteredItems como estado en absoluto:

// ✅ Estado derivado — calculado durante el render, sin efecto
const [items, setItems] = useState<Item[]>([]);

// Se recalcula en cada render que involucra items
// Si el cálculo es costoso, useMemo es la herramienta, no useEffect
const filteredItems = items.filter(item => item.active);

Si el filtro es computacionalmente caro, useMemo con la dependencia correcta. Si es barato — y la mayoría lo son — ni eso. Cálculo directo durante el render.

Las 4 categorías concretas y qué las reemplaza

1. Estado derivado de props o estado existente → cálculo en render o useMemo

Ya lo vimos arriba. La regla práctica: si podés calcular algo a partir del estado o las props que ya tenés, no es estado nuevo. Es una función de lo que ya existe.

// ❌ Versión con efecto
const [user, setUser] = useState<User | null>(null);
const [displayName, setDisplayName] = useState('');

useEffect(() => {
  setDisplayName(user ? `${user.firstName} ${user.lastName}` : 'Invitado');
}, [user]);

// ✅ Versión correcta: derivado en línea
const displayName = user ? `${user.firstName} ${user.lastName}` : 'Invitado';

Parece trivial. En una codebase real con 30 componentes que hacen esto, el impacto acumulado en re-renders no es trivial.

2. Sincronización post-evento → event handlers directos

Otro patrón que encontré seguido: el efecto que observa un estado para “hacer algo cuando cambia”, pero ese cambio siempre viene de una interacción del usuario.

// ❌ useEffect que reacciona a un cambio que solo viene de un click
const [selectedId, setSelectedId] = useState<string | null>(null);

useEffect(() => {
  if (selectedId) {
    analytics.track('item_selected', { id: selectedId });
    loadDetails(selectedId);
  }
}, [selectedId]);

// ✅ El handler del evento sabe todo lo que necesita saber
const handleSelect = (id: string) => {
  setSelectedId(id);
  // La lógica que reacciona al evento va ACÁ, no en un efecto
  analytics.track('item_selected', { id });
  loadDetails(id);
};

La diferencia no es solo estética. Con el efecto, cualquier cambio en selectedId (incluso programático, incluso desde otro efecto) dispara la lógica. Con el handler, la intención es explícita: esto pasa cuando el usuario selecciona algo. Menos sorpresas, más control.

La documentación de React lo pone directo: si algo ocurre porque el usuario hizo algo, va en el event handler. Si algo ocurre porque el componente se mostró, va en un efecto. La confusión entre estos dos casos es la raíz de la mayoría de los bugs de useEffect que veo.

3. Fetch de datos → use() con Suspense en React 19, o Server Components

Esta es la categoría donde React 19 cambia más el juego. El fetch con useEffect es el patrón más copiado de internet y uno de los más problemáticos:

// ❌ El clásico useEffect para fetch — race conditions esperando suceder
const [data, setData] = useState(null);
const [loading, setLoading] = useState(true);

useEffect(() => {
  // Sin cleanup, esto puede setear estado en un componente desmontado
  fetch(`/api/items/${id}`)
    .then(r => r.json())
    .then(setData)
    .finally(() => setLoading(false));
}, [id]);

Este código tiene una race condition clásica: si id cambia rápido, dos fetches corren en paralelo y el resultado puede llegar en cualquier orden. Se puede mitigar con cleanup, pero es boilerplate que la mayoría omite.

React 19 trae use() que integra con Suspense:

// ✅ React 19: use() + Suspense — sin useEffect, sin estado de loading manual
import { use, Suspense } from 'react';

// La promesa se crea fuera del componente o se pasa como prop
function ItemDetail({ itemPromise }: { itemPromise: Promise<Item> }) {
  // use() suspende el componente hasta que la promesa resuelve
  const item = use(itemPromise);

  return <div>{item.name}</div>;
}

// En el componente padre:
function Page({ id }: { id: string }) {
  // La promesa se crea aquí, React maneja el ciclo de vida
  const itemPromise = fetchItem(id); // función que devuelve Promise<Item>

  return (
    <Suspense fallback={<Skeleton />}>
      <ItemDetail itemPromise={itemPromise} />
    </Suspense>
  );
}

Pero si trabajás con App Router en Next.js 16, la respuesta más honesta es: usá Server Components para el fetch. El dato llega serializado al cliente, sin estado de carga, sin race conditions, sin useEffect. Para casos más complejos donde el fetch es dependiente de interacción del usuario, use() con Suspense es el camino.

Esto conecta con algo que documenté en el post sobre Prisma Server Actions en Next.js 16 — la frontera entre lo que corre en servidor y lo que corre en cliente cambia bastante qué patrones tienen sentido.

4. Transformaciones post-submit → Server Actions

La última categoría: el efecto que escucha el resultado de un submit para actualizar estado derivado, mostrar mensajes, o redireccionar.

// ❌ useEffect que observa el resultado de un submit
const [submitResult, setSubmitResult] = useState<Result | null>(null);
const [errorMessage, setErrorMessage] = useState('');

useEffect(() => {
  if (submitResult?.error) {
    setErrorMessage(submitResult.error.message);
  }
}, [submitResult]);

Con Server Actions en React 19 y el hook useActionState (anteriormente useFormState), esto colapsa en un patrón mucho más directo:

// ✅ useActionState — el estado del form y el resultado de la acción, integrados
import { useActionState } from 'react';

async function submitForm(prevState: State, formData: FormData): Promise<State> {
  'use server';
  // La acción corre en el servidor, devuelve el nuevo estado
  const result = await processForm(formData);
  if (!result.ok) {
    return { error: result.message };
  }
  return { success: true };
}

function MyForm() {
  const [state, action, isPending] = useActionState(submitForm, { error: null });

  return (
    <form action={action}>
      {/* Sin useEffect, sin estado extra, sin sincronización manual */}
      {state.error && <p className="error">{state.error}</p>}
      <button disabled={isPending}>Guardar</button>
    </form>
  );
}

El estado del form, el feedback de error y el loading están integrados sin un solo useEffect. No es magia — es que la responsabilidad está en el lugar correcto.

Los errores que persisten y los que desaparecen

Hay casos donde useEffect sí corresponde: suscripciones a stores externos, sincronización con APIs del DOM que no tenés control (un mapa de terceros, una librería de canvas), o setup/teardown de recursos que existen fuera del modelo de React.

Lo que desaparece con estos patrones:

  • Race conditions en fetch — use() y Server Components los eliminan estructuralmente
  • Renders intermedios inconsistentes — el estado derivado no tiene estado inconsistente porque no es estado
  • Grafos de efectos encadenados — cuando un efecto setea estado que dispara otro efecto, el debug es un infierno. Los event handlers lo cortan de raíz
  • Cleanup olvidado — si no hay efecto, no hay cleanup que olvidar

Lo que no desaparece:

  • Pensamiento sobre dependencias — useMemo también tiene un array de dependencias. use() requiere entender cómo React maneja las promesas. El criterio sigue siendo necesario.
  • Complejidad de Suspense — boundaries mal ubicados rompen la UX de maneras no obvias. No es un reemplazo automático.

FAQ

¿useEffect quedó obsoleto en React 19?

No. El equipo de React no lo deprecó ni lo marcó como legacy. Lo que cambió es que React 19 pone herramientas mejores más a mano para los casos donde useEffect era la única opción disponible. Para suscripciones, integración con APIs del DOM externas, y sincronización con sistemas fuera del modelo de React, useEffect sigue siendo correcto.

¿use() reemplaza a useEffect para todos los fetches?

Para fetches en componentes cliente que dependen de interacción del usuario, use() con Suspense es una alternativa concreta. Para fetches de datos iniciales en App Router, Server Components son la respuesta más directa — ni use() ni useEffect. La elección depende de si el dato puede resolverse en servidor o necesita esperar al cliente.

¿Cuándo usar useMemo vs cálculo directo para estado derivado?

Cálculo directo primero. useMemo solo cuando el cálculo es mediblemente costoso (sort/filter sobre arrays grandes, transformaciones complejas) y el profiler confirma que hay un problema. Agregar useMemo preventivamente es otra forma de sobre-ingeniería — tiene un costo de legibilidad y las dependencias también se pueden equivocar.

¿useActionState funciona sin Server Actions?

Sí. useActionState acepta cualquier función asíncrona, no solo Server Actions. Si preferís manejar el submit del lado del cliente, podés pasarle una función client-side. La integración con Server Actions es la más limpia en Next.js con App Router, pero no es un requisito.

¿Cómo migro una codebase existente? ¿Hay un path incremental?

El path más seguro es por categoría, no por archivo. Empezá identificando todos los useEffect que solo derivan estado — son los más seguros de migrar y los que dan más beneficio inmediato. Después los que reaccionan a eventos del usuario. Los fetches últimos, porque requieren decisiones sobre Suspense boundaries que afectan la UX.

¿Estos patrones cambian si uso Zustand, Jotai o Redux?

Parcialmente. Los stores externos resuelven el problema del estado global, pero el antipatrón de derivar estado dentro de un componente con useEffect aparece igual. La pregunta “¿puedo calcular esto durante el render?” aplica independientemente del sistema de estado que usés.

Conclusión: el criterio que no viene del framework

Lo que más me frustró cuando revisé esta codebase no fue encontrar los antipatrones — eso era esperable. Fue darme cuenta de que estaban ahí porque en su momento nadie tenía una regla clara para decidir cuándo usar useEffect. Lo usábamos como herramienta por defecto para “hacer algo cuando algo cambia”, y ese modelo mental es incorrecto desde el principio.

React 19 hace más fácil hacer lo correcto: use() existe, useActionState existe, los Server Components están más integrados. Pero sin el criterio de cuándo cada herramienta aplica, lo que pasa es que los mismos problemas migran a las nuevas APIs.

Mi postura concreta: antes de escribir un useEffect, preguntate si lo que querés hacer es (a) calcular algo a partir del estado existente, (b) reaccionar a una acción del usuario, (c) cargar datos, o (d) sincronizar con algo externo a React. Solo el último caso justifica un efecto. Los tres primeros tienen soluciones mejores en React 19, y la documentación oficial de React lo dice explícitamente.

Si estás revisando una codebase y no sabés por dónde empezar, el mismo ejercicio de auditoría sirve para otros patrones: el N+1 que aparece cuando menos lo esperás en Prisma tiene la misma estructura — un patrón que parece razonable hasta que lo mirás con la pregunta correcta.

Fuente original:

Migrating a Long-Running WordPress Site to Payload CMS (And All The Chaos That Came With It)

TL;DR for the skimmers:

  • Design the Payload schema before writing any migration code
  • Use a wpId field on every collection for idempotent upserts
  • Expect multiple media migration passes : the XML export won’t contain everything
  • The HTML-to-Lexical parser will dominate the timeline, budget accordingly
  • Separate migration scripts per content type is fine. Don’t build abstractions that are going to be deleted.

If you’ve ever opened a WordPress XML export, scrolled for a while, and quietly wondered who let this happen, this post is for you.

Content migrations from WordPress to a headless CMS look simple on paper. Export. Transform. Load. Three words, absolutely not innocent in practice. Most of the work isn’t moving data, it’s figuring out what the data means and where and how it should live in the new system.

Recently i had to migrate a long-running Saas Documnetaion WordPress site to Payload CMS. A site that had been through a commercial page builder like Divi, SEO plugin, a custom FAQ plugin, and at least three different image upload mechanisms over the years.

This post is about what worked, what didn’t, and the specific patterns that did it correctly when the script ran for the 30th time at 1 AM and finally produced clean output.

1. Design the Payload Schema Before Writing Any Migration Code

The migration script is throwaway code. The Payload collection is what the editorial team lives inside for the next several years. Spending a week on schema decisions before writing a single line of migration code is what keeps the project from becoming a rewrite of WordPress with extra steps.

Three decisions in this project carried more weight than everything else combined:

Model hierarchy with a self-referencing relationship, then derive the path from it. The documentation pages, policy pages, and integration catalog all had parent-child relationships. Some three or four levels deep. Each collection has a parent field pointing at the same collection, plus a read-only fullSlug and fullUrl that get recomputed in a beforeChange hook by walking up the parent chain:

// policy-pages collection — beforeChange hook
async ({ data, req }) => {
  if (!data?.slug) return data
  const fullSlug = await generateFullSlug(req.payload, data.slug, data.parent)
  data.fullSlug = fullSlug
  data.fullUrl = `/${fullSlug}`
  return data
}

async function generateFullSlug(payload, slug, parentId) {
  if (!parentId) return `policies/${slug}`
  const parent = await payload.findByID({
    collection: 'policy-pages', id: parentId, depth: 0,
  })
  if (parent?.fullSlug) return `${parent.fullSlug}/${slug}`
  return `policies/${slug}`
}

This does two things. First, renaming a parent page doesn’t break any of its children and the next save rewrites the full path. Second, the unique constraint goes on fullSlug (like docs/integrations/[provider]/setup), not on the individual slug. Multiple pages called “setup” under different parents is fine, because each resolves to a unique full path. WordPress handles this with friendly URL aliases. Payload handles it with a derived path. Both work, but the model needs to be committed to before the migration starts.

Stamp every migrated document with its WordPress ID. A wpId number field, hidden in the sidebar, read-only. Not for the editors but the migration script. This is what makes re-running the script safe instead of catastrophic. Section 2 covers the mechanics, but the schema decision needs to happen here: if the field doesn’t exist on the collection, none of the idempotency logic works.

Decide which WordPress structures become their own collections vs. blocks within a collection. The source site had landing pages, policy pages, documentation, an integration catalog, FAQs, and blog posts. These all lived in WordPress’s posts and pages tables, differentiated by parent IDs, custom post types, and template names. In Payload they became separate collections because each had fundamentally different fields: integration pages have a category relationship and platform-specific metadata. Documentation pages have a breadcrumbs array and cross-linking to related articles. Policy pages have none of that. FAQs are just a question/answer pair. If these had been shoe-horned into one pages collection with conditional fields, the admin panel would have been unusable.

2. Use a wpId Field for Idempotent Migrations

The migration script runs more than once. A lot more than once. Different subsets, different flags, different content types, recovery attempts after something breaks. In this project the documentation migration script alone ran dozens of times before everything settled.

Without a way to identify “is this WordPress post already in Payload?”, duplicates pile up on every run. Once is annoying. Thirty times is a cleanup project of its own.

The fix is a small in-memory map populated at the start of every run:

const wpIdToPayloadId = new Map<number, number>()

async function loadWpIdMap(payload: any) {
  const result = await payload.find({
    collection: 'docs',
    limit: 500,
    select: { id: true, wpId: true },
  })
  result.docs.forEach((doc) => {
    if (doc.wpId) wpIdToPayloadId.set(doc.wpId, Number(doc.id))
  })
}

Then during import:

const existingId = wpIdToPayloadId.get(wpPost.id)
if (existingId) {
  await payload.update({ collection: 'docs', id: existingId, data })
} else {
  await payload.create({
    collection: 'docs',
    data: { ...data, wpId: wpPost.id },
  })
}

The script is now idempotent. Same input produces the same result. This unlocks everything else: staging refreshes, retry logic, partial reruns when a parsing bug surfaces halfway through a batch.

Seven collections in this project ended up with wpId fields. Every migration script uses the same map-and-upsert pattern. It’s boring, and that’s the point.

3. Media Migration Won’t Be One Script

This is where the clean narrative “write a media resolver” falls apart.

WordPress had a complicated relationship with its own media library. The same image could appear in body HTML as a full CDN URL, a WordPress-generated size variant (screenshot-300x200.png), an artifact from a half-completed format migration (screenshot.png.webp), a protocol-less reference, or a relative path with no domain.

And that’s just the images that were in the media library. This particular site also had:

  • 60+ images uploaded directly to the server via FTP years ago, never registered in WordPress’s media library at all. They lived on disk but had no database entry. The XML export didn’t mention them.
  • ~100 theme images from the child theme’s CSS such as retina variants at 2x and 3x, icon sprites, background patterns. These were referenced in stylesheets, not in post content, but the new site needed them in Payload’s media collection so the storage adapter could serve them from S3.

That’s three separate media migration scripts. The first handles the XML export. The second is a hardcoded list of FTP URLs discovered by crawling the staging server. The third is a hardcoded list of theme asset paths. None of them shared code with each other beyond the Payload create call.

The first script also had a subtle bug that took weeks to surface. Retina images like [email protected] and [email protected] were all failing silently. The WordPress XML stored their attachment URLs with %40 (URL-encoded @) in the filename: icon%402x.png. The migration script extracted the filename with path.basename(new URL(...).pathname) which doesn’t decode URL encoding. Payload received icon%402x.png instead of [email protected]. Thirteen images affected. The fix was a one-liner (decodeURIComponent), but finding it required inspecting the XML by hand and wondering why only images with @ in the name were broken. This is the kind of bug that only shows up when you stop looking at the script and start looking at the data.

In this project the content-body media resolver (used during HTML-to-Lexical conversion) was a lookup function, not a migration script.

function findMediaId(src: string): number | null {
  const filename = extractFilename(src)
  const baseName = getBaseFilename(filename) // strips .png.webp → base

  // 1. Exact base match
  if (mediaFilenameToId.has(baseName)) return mediaFilenameToId.get(baseName)!

  // 2. Full lowercase filename
  if (mediaFilenameToId.has(filename.toLowerCase())) return mediaFilenameToId.get(filename.toLowerCase())!

  // 3. Partial match (handles WP size variants like -300x200)
  for (const [key, id] of mediaFilenameToId) {
    if (key.startsWith(baseName + '-') || baseName.startsWith(key + '-')) return id
  }

  // 4. Substring match (catches double-extension edge cases)
  for (const [key, id] of mediaFilenameToId) {
    if (key.startsWith(baseName + '.') || key === baseName) return id
  }

  missingMediaSrcs.add(`${filename}::${currentPageSlug}`)
  return null
}

A tiered lookup: exact match, then lowercase, then strip the WordPress size suffix, then substring fallback. Anything that still misses goes into a missingMediaSrcs set that prints at the end of the run, grouped by page slug.

An important decision: don’t fail the document write when a media reference can’t be resolved. Some source HTML will point at images that aren’t in the export at all. If the script throws on missing media, the whole migration stops on the first batch. Logging and continuing produces a clean triage list at the end instead of a hard halt at the start.

4. The HTML-to-Lexical Body Parser Is Where Most of the Time Goes

WordPress stores rich content as HTML. Payload wants a structured Lexical JSON tree. The documentation parser alone ended up at ~900 lines. Here’s why:

Page builder wrappers everywhere. The source HTML wasn’t clean semantic markup. It was wrapped in multiple layers of page builder containers like section > row > column > module > text-inner > actual content. Sometimes there were additional wrapper divs with custom CSS classes. The parser had to recognize and recurse through all of them without treating them as content.

HTML entity decoding. Smart quotes, &nbsp; sequences, numeric entities, etc. What’s worth noting is that two separate implementations of decodeHtmlEntities() ended up in the codebase because policy pages and documentation were migrated as separate scripts with slightly different entity handling. Copy-paste creeps in when scripts are written independently.

Page-builder shortcodes that need to become Payload blocks. A toggle module becomes a CollapsibleBlock. A styled button with a specific CSS class becomes a CustomLink block. A row of grouped tabbed links for different integration methods becomes a multi-link block. Each of these has different Lexical node shapes.

Inline images that need to become MediaBlock references. WordPress drops <img> tags anywhere: inside paragraphs, inside list items, inside blockquotes. Payload’s Lexical editor represents images as block-level nodes with a relationship to a media document. The parser has to extract inline images, resolve them through the media lookup, and emit them as separate nodes.

The pattern that works: use JSDOM to walk the source HTML, classify each node, emit the corresponding Lexical structure. Be explicit and boring. Don’t try to be clever.

The central processor ended up looking like this:

function processBlockNodes(container: Element, isNested: boolean = false): any[] {
  const children: any[] = []
  let inlineBuffer: any[] = []

  function flushBuffer() {
    if (inlineBuffer.length === 0) return
    children.push(makeParagraph(inlineBuffer))
    inlineBuffer = []
  }

  function processNode(node: ChildNode) {
    // ... classify and emit
    // Headings → createHeading()
    // Paragraphs → extract images first, then processInlineNodes()
    // Lists → createList() (with nested list and trailing content handling)
    // Blockquotes → createQuote() + extract trailing images
    // Page builder wrappers → recurse
    // FAQ accordions → processFaqList()
    // CTA buttons → createCustomLinkSingle()
    // Bare text → accumulate in inlineBuffer
  }

  container.childNodes.forEach(processNode)
  flushBuffer()
  return children
}

The inlineBuffer pattern is worth calling out. WordPress HTML sometimes has bare text nodes between block elements. Lexical requires all text to live inside a paragraph node. The buffer collects loose text and flushes it into a paragraph whenever a real block element comes along.

The classification rules are specific to the source site. In this project, the documentation parser handled things the policy page parser didn’t (FAQ accordions, tabbed link blocks) and vice versa (tables, back-navigation links). Each script handles its own content type.

One thing the code above doesn’t convey: the parser wasn’t designed correctly on the first pass, or the second, or the third. The first dry-run produced 9 media blocks across 80 pages. That was clearly wrong. The source HTML had images everywhere. The fix for images inside <p> tags bumped it to 160. Then it turned out blocks nested inside <li> elements were being silently skipped. Then support buttons inside <p> tags weren’t becoming CustomLink blocks. Then querySelector was only grabbing the first page builder text block on pages that had several. Each dry-run revealed a new class of missed content.

The parser wasn’t designed but discovered, one bug at a time.

For example, one bug from this phase didn’t surface until much later, which is the lesson in itself. The parser generated Lexical node IDs with Date.now() plus a short random hex suffix. On fast iterations, Date.now() returns the same millisecond twice, and the random suffix occasionally collided producing duplicate node IDs across a document. The migrated content rendered fine on the front-end, so nothing looked wrong. But opening those documents in the Payload admin editor crashed it with a React duplicate-key error. A parser bug, surfacing as a CMS crash. The fix was a proper unique ID generator; finding it meant tracing a React error in the admin panel all the way back to a migration script that had already been deleted.

Speaking of which.

5. Accept That You’ll Have Separate Scripts Per Content Type

This is something I chose not to gloss over, and it’s worth being honest about. The project didn’t end up with one unified migration system. I ended up with twelve separate scripts:

  • migrate-media.ts: XML export media
  • migrate-media-ftp.ts: 60+ FTP images not in WordPress
  • migrate-theme-media.ts: ~100 theme images from the child theme
  • migrate-clients.ts
  • migrate-categories.ts
  • migrate-platforms.ts: taxonomy terms for the integration catalog
  • migrate-pages.ts: root landing pages (page builder hero detection)
  • migrate-policy-pages.ts: 5 policy pages with table support
  • migrate-docs.ts: ~65 hierarchical documentation pages (the big one)
  • migrate-integrations.ts: ~100 integration catalog pages
  • migrate-faqs.ts: FAQ custom post type from XML
  • migrate-bespoke.ts: one hand-crafted page that didn’t fit the general parser

Each has its own CLI flags (--dry-run, --limit, --slug, --verbose, --force-update), its own HTML-to-Lexical converter (ranging from “strip all tags and split by newlines” for landing pages to the full 900-line parser for documentation), and its own idempotency logic.

A planned unified CLI entry point (scripts/index.ts) never got built. Running individual scripts worked fine and less time consuming as well.

The fragmentation wasn’t a failure of planning. It was a consequence of the “design the schema first” advice. Each content type had different fields, different block types, different HTML structures, and different parent/child relationships. The documentation needed hierarchy sorting and related-articles linking. The policy pages needed back-navigation stripping and table support. The integration catalog needed category taxonomy mapping. The FAQ custom post type had to be parsed from XML, not JSON.

Didn’t want to optimize for dry. Feels like a trap!

6. Lexical Block Constraints Need a Sanitization Pass

This one caught me off guard, and it cost a day of mysterious validation errors.

Lexical blocks have rules about what they can contain. A CollapsibleBlock might allow h2 through h6, paragraphs, lists, media, and horizontal rules but not h1, blockquotes, or nested collapsibles. Real source HTML doesn’t always follow those rules.

The documentation parser hits this constantly because the WordPress FAQ plugin generates accordion sections, and those sections contain whatever HTML the editor put in them including other FAQ sections. Nest an accordion inside an accordion and Payload rejects the write.

The fix is a sanitization pass that runs between parsing and writing:

function sanitizeForCollapsible(lexical: any): any {
  if (!lexical?.root?.children) return lexical

  const sanitized: any[] = []
  for (const node of lexical.root.children) {
    // h1 → h2
    if (node.type === 'heading' && node.tag === 'h1') {
      sanitized.push({ ...node, tag: 'h2' })
      continue
    }
    // blockquote → paragraphs
    if (node.type === 'quote') {
      const paras = (node.children || []).filter((c) => c.type === 'paragraph')
      sanitized.push(...(paras.length > 0 ? paras : [makeParagraph(extractText(node))]))
      continue
    }
    // nested collapsible → flatten content into parent
    if (node.type === 'block' && node.fields?.blockType === 'collapsible') {
      sanitized.push(...(node.fields?.content?.root?.children || []))
      continue
    }
    sanitized.push(node)
  }

  return { root: { ...lexical.root, children: sanitized } }
}

Without this, roughly a quarter of documents can fail validation at write time. The error messages don’t indicate which nested rule was violated, which is most of what makes this expensive to debug the first time.

7. WordPress URL Aliases Need Their Own Mapping…sometimes

This took a full afternoon to figure out and it’s not obvious.

WordPress has a concept of “friendly” URL aliases. A documentation page might live at /docs/integrations/provider-a/setup-guide-full but also be reachable at /docs/provider-a/setup. The short URL is what appears in navigation links and “related articles” sections within other pages.

Payload doesn’t have this concept. A document has one slug, one parent chain, one fullSlug. So when the source HTML contains a related-articles section with links like /docs/provider-a/setup, that path can’t just be looked up in Payload. It doesn’t exist.

The fix was a hardcoded lookup table mapping WordPress friendly URLs to the actual Payload fullSlug:

const wpFriendlyToFullSlug: Record<string, string> = {
  'docs/provider-a/setup': 'docs/integrations/provider-a/setup-guide-full',
  'docs/provider-b/webhooks': 'docs/integrations/provider-b/configuring-webhook-endpoints',
  'docs/api/v1': 'docs/api/v1/getting-started',
  'docs/api/v2': 'docs/api/v2/schema-reference',
  // ... 30 more entries
}

This table was built by manually inspecting which WordPress page each friendly URL pointed to. It’s not glamorous work. But without it, related-articles links would silently fail to resolve, and the migrated site would have broken cross-linking.

The SEO plugin’s breadcrumb schema data was useful here as it was a structured BreadcrumbList JSON with absolute URLs and titles, which helped verify that the mapping was correct.

8. Build Dry-Run Mode From Day One

The highest-leverage decision is adding proper CLI flags at the start of the project, not at the end:

tsx migrate-docs.ts --dry-run
tsx migrate-docs.ts --dry-run --limit=5
tsx migrate-docs.ts --slug=getting-started
tsx migrate-docs.ts --force-update

These take an hour to add per script. They save days of cleanup over the course of the project. Once they exist, make --dry-run the default. Running the script accidentally should be harmless.

The --slug flag is particularly valuable when debugging a single problematic page. Running the parser against one document and inspecting the full Lexical output without touching the database saves 20 minutes of debugging.

Running the migration in dry-run mode with verbose logging also turned out to be useful beyond catching script bugs. Inspecting every document on its way through the parser surfaced things worth flagging:

  • Slug collisions across different parents, usually legitimate. Three pages named “setup” under different parent integrations is fine when the unique constraint is on fullSlug.
  • Orphaned drafts with no slug, reachable only by ?page_id=12345 URLs.
  • Media references without matching files in the export.
  • One page whose HTML was unique enough that it warranted its own bespoke migration script rather than trying to bend the general parser to handle it.

The dry-run output ended up serving as the definitive content inventory.

9. Sort by Hierarchy Before Migrating

When content is hierarchical, parents need to be migrated before children.

The documentation pages formed a tree. The wpIdToPayloadId map needs the parent’s Payload ID before setting the child’s parent relationship field. Process children first and the lookup fails, leaving orphaned documents.

The fix was a topological sort that guarantees parents come before children:

function sortPagesByHierarchy(pages: WordPressPage[]): WordPressPage[] {
  const sorted: WordPressPage[] = []
  const remaining = [...pages]
  const added = new Set<number>()
  const allPageIds = new Set(pages.map((p) => p.id))

  while (remaining.length > 0) {
    const beforeLength = remaining.length

    for (let i = remaining.length - 1; i >= 0; i--) {
      const page = remaining[i]
      const parentInSet = allPageIds.has(page.parent)
      const parentMigrated = added.has(page.parent)

      if (!parentInSet || parentMigrated) {
        sorted.push(page)
        added.add(page.id)
        remaining.splice(i, 1)
      }
    }

    // Circular dependency guard
    if (remaining.length === beforeLength) {
      console.warn(`Circular dependency, force-adding ${remaining.length} pages`)
      sorted.push(...remaining)
      break
    }
  }

  return sorted
}

The circular dependency guard is there because one page in the export referenced a parent that didn’t exist in the set. Without it, the sort loops forever.

Closing Thoughts

The chaos isn’t in the code. It’s in the decisions and the modeling. Which content types become their own collections? Which URL conventions carry over? What counts as a “block” versus a “field”? The migration script just translates those decisions one at a time.

PS – I didn’t write all of this code alone, either. Claude and GLM were on the team for most of it, especially through the parser layer where pattern-matching twelve variations of the same shortcode is exactly the tedious work AI is good at. The architectural calls stayed mine; the boilerplate didn’t. If a deeper post on that workflow would be useful consisting what I prompted, where it failed, where it shaved off days versus quietly added them back, say so in the comments and I’ll write it up.

Keep your dry-runs greedy and your logs verbose.

Hidden Partitioning: How Iceberg Eliminates Accidental Full Table Scans

Exposed partitioning in Hive versus hidden partitioning in Iceberg showing the same pruning with different user experience

This is Part 5 of a 15-part Apache Iceberg Masterclass. Part 4 covered partition evolution. This article covers hidden partitioning, the feature that ensures users never need to know how their data is physically organized.

The most expensive mistake in data lake querying is the accidental full table scan: a query that reads every file because the user did not correctly reference the partition columns. In Hive, this happens constantly. In Iceberg, it is structurally impossible because users never reference partition columns at all.

Table of Contents

  1. What Are Table Formats and Why Were They Needed?
  2. The Metadata Structure of Current Table Formats
  3. Performance and Apache Iceberg’s Metadata
  4. Technical Deep Dive on Partition Evolution
  5. Technical Deep Dive on Hidden Partitioning
  6. Writing to an Apache Iceberg Table
  7. What Are Lakehouse Catalogs?
  8. Embedded Catalogs: S3 Tables and MinIO AI Stor
  9. How Iceberg Table Storage Degrades Over Time
  10. Maintaining Apache Iceberg Tables
  11. Apache Iceberg Metadata Tables
  12. Using Iceberg with Python and MPP Engines
  13. Streaming Data into Apache Iceberg Tables
  14. Hands-On with Iceberg Using Dremio Cloud
  15. Migrating to Apache Iceberg

The Accidental Full Scan Problem

Exposed partitioning in Hive versus hidden partitioning in Iceberg showing the same pruning with different user experience

In Hive, a table partitioned by year, month, and day requires queries to filter on those exact columns:

-- Hive: This prunes correctly
SELECT * FROM orders WHERE year = 2024 AND month = 3 AND day = 15

-- Hive: This scans EVERYTHING (no pruning)
SELECT * FROM orders WHERE order_date = '2024-03-15'

The second query reads every partition because Hive does not know that order_date maps to the year, month, and day partition columns. There is no error, no warning. The query simply runs 100x slower than it should.

This happens because Hive partitioning is “exposed.” The physical partition columns (year, month, day) are separate from the source column (order_date). Users must understand this mapping and construct their filters accordingly.

How Iceberg Hides Partitioning

Iceberg flips this model. Users filter on the source column (order_date), and the engine automatically maps the filter to the partition values using transform functions.

-- Iceberg: This prunes correctly. Always.
SELECT * FROM orders WHERE order_date = '2024-03-15'

The table’s partition spec declares: PARTITIONED BY (day(order_date)). When the engine processes this query, it:

  1. Reads the partition spec from the table metadata
  2. Applies the day() transform to the filter value: day('2024-03-15') = 2024-03-15
  3. Checks manifest entries for files with matching partition values
  4. Skips every file whose partition value is not 2024-03-15

The user writes natural SQL against the source columns. The engine handles the physical-to-logical mapping. This is why it is called “hidden” partitioning: the partition structure is invisible to the user.

The Six Transform Functions

Iceberg's six partition transform functions showing how each maps source values to partition values

Iceberg defines six partition transforms that map source column values to partition values:

Temporal Transforms

Temporal Transforms

The temporal transforms are hierarchical. If a table is partitioned by day(ts) and a user filters WHERE ts >= '2024-03-01' AND ts < '2024-04-01', the engine recognizes this as a range of days and prunes to only the 31 matching partitions. Engines like Dremio handle this mapping automatically for equality, range, and IN-list predicates.

Value Transforms

Value Transforms

truncate(N, col) takes the first N characters of a string (or truncates a number to a width). This is useful when you want to group data by a string prefix without creating one partition per unique value.

bucket(N, col) applies a hash function and mod N to produce a bucket number from 0 to N-1. This distributes data evenly across a fixed number of buckets, regardless of the column’s value distribution. It is the go-to transform for high-cardinality columns like user_id or order_id where identity partitioning would create millions of tiny partitions.

The Identity Transform

The identity transform (identity(col)) uses the raw column value as the partition value. This is equivalent to Hive-style partitioning, but the column is still “hidden” because the engine handles the mapping. It is useful for low-cardinality columns like region or status where each unique value should be its own partition.

How Pruning Works Under the Hood

Step-by-step flow showing how the engine maps a user query through the partition spec to prune files

The pruning process works in three phases:

Phase 1: Predicate translation. The engine examines each WHERE clause predicate and checks if the filtered column is part of the partition spec. If order_date is the source column for day(order_date), the engine can translate order_date = '2024-03-15' into a partition filter.

Phase 2: Manifest list evaluation. The manifest list stores partition value ranges per manifest. The engine checks if each manifest’s range includes the target partition value. Manifests whose range does not overlap are skipped entirely.

Phase 3: Manifest entry evaluation. For each surviving manifest, the engine checks individual file entries. Only files whose partition value matches 2024-03-15 are included in the scan plan.

This is the same pruning cascade described in Part 3, but now the partition values were derived automatically from the user’s filter on a source column.

Choosing the Right Transform

The choice of partition transform depends on data volume and query patterns:

Choosing the Right Transform

The goal is to create partitions that are large enough to contain optimally-sized files (128-512 MB each) but small enough that partition pruning eliminates most files for typical queries.

Over-partitioning (too many small partitions) creates the small file problem: thousands of tiny files that bloat metadata and slow query planning. Under-partitioning (too few large partitions) reduces pruning effectiveness because each partition contains too much data.

Combining Transforms

Iceberg supports multi-column partition specs:

CREATE TABLE events (
  event_id BIGINT,
  event_time TIMESTAMP,
  user_id BIGINT,
  event_type STRING
) PARTITIONED BY (day(event_time), bucket(32, user_id))

This creates a two-dimensional partition space: each combination of day and user bucket is a separate partition. Queries filtering on event_time get day-level pruning. Queries filtering on user_id get bucket-level pruning. Queries filtering on both get pruning from both dimensions.

Dremio supports all Iceberg transform functions and automatically applies pruning for any combination of partition columns in the query’s WHERE clause.

Why This Matters for Teams

Hidden partitioning changes the operational model for data teams:

Data engineers define the partition strategy once in the table’s partition spec. They can change it later through partition evolution without breaking anything.

Analysts and data scientists write natural SQL against the business columns they understand. They never need to know whether the table is partitioned by day, month, or bucket. Their queries are automatically optimized.

BI tools and dashboards connect to Iceberg tables and issue standard SQL. The tools do not need to understand Iceberg’s partitioning; the engine handles the optimization. This is why hidden partitioning is essential for self-service analytics platforms like Dremio.

The net result: no accidental full table scans, no partition-aware query patterns required from users, and the ability to change the physical layout without impacting any downstream consumer. Part 6 covers what happens when data is written to an Iceberg table.

Books to Go Deeper

Free Resources

Frameworks Are No Longer Being Designed Only for Humans

This is a submission for the Google I/O Writing Challenge

TL;DR
After watching the Google I/O 2026 sessions, I started noticing the same pattern across Flutter, Search, Chrome DevTools, and Google’s agent workflows.

AI is not only becoming part of applications.

Frameworks, tooling, and even the web itself are slowly starting to evolve around AI systems as active builders too.

Estimated read time: ~8 minutes

Table of Contents

  • I thought this year would be about models
  • The Flutter update that changed how I looked at the rest of I/O
  • We used to build interfaces for people
  • Search is starting to behave more like software generation
  • The Human Clipboard Problem
  • Websites are changing too
  • The tension underneath all of this
  • What developers may actually need to adapt to
  • Final thoughts
  • References
  • 🤝 Stay in Touch

I thought this year would be about models

Every Google I/O eventually returns to the same themes.

Better models.

Faster responses.

More capable AI systems.

So when I started watching the Google I/O 2026 sessions, I expected another version of that story.

And there was plenty of it.

But after watching:

another pattern slowly started showing up underneath everything else.

Not through one major announcement.

Not through one headline feature.

It appeared through smaller architectural decisions spread across different sessions.

At first, none of them seemed connected.

Then the Flutter session started talking about separating Material and Cupertino from the core framework.

After that, I started noticing the same idea everywhere else too.

The Flutter update that changed how I looked at the rest of I/O

On the surface, the Flutter announcements sounded like framework maintenance.

Material and Cupertino becoming separate packages.

A more style-neutral Flutter core.

The GenUI SDK.

The A2UI protocol.

But the more I listened, the less it sounded like ordinary framework work.

Because UI frameworks have always been opinionated by design.

That is their job.

They guide developers toward structure and consistency.

For years, frontend development has mostly followed the same model:

developers design interfaces first

users interact with them later

The screens are already decided.

The flows are already decided.

Even dynamic applications are usually built from predefined layouts assembled by humans.

But many of the announcements at Google I/O 2026 seemed to move in another direction.

Interfaces that adapt:

  • to context
  • to conversation
  • to intent
  • to environment
  • to the task happening in that moment

Not fixed screens.

Generated ones.

And that changes something underneath the framework itself.

Because if software is expected to generate interfaces dynamically, frameworks cannot remain tightly structured around predefined human layouts.

They need to become flexible enough for software to assemble software.

That was the moment the rest of I/O started looking different to me.

Most of that realization came from watching What’s New in Flutter.

The session spends a good amount of time talking about GenUI, A2UI, and Flutter becoming more style-neutral instead of tightly coupled to predefined design systems.

The more I watched it, the more it sounded less like a normal framework update and more like Flutter preparing for dynamically generated interfaces.

We used to build interfaces for people

What started becoming more obvious across the sessions was how deeply human-centered most software tooling has always been.

Component systems.

Design systems.

Navigation patterns.

Accessibility flows.

Everything was built around how humans:

  • read
  • navigate
  • interpret
  • scan
  • and move through interfaces visually

But now there is another participant inside the system.

Agents.

And they interact with software differently.

An AI agent does not care whether a button feels visually balanced on a screen.

It cares about:

  • structure
  • interaction points
  • capabilities
  • semantic meaning
  • machine-readable workflows
  • accessible pathways

That changes what frameworks need to optimize for.

Not replacing humans.

But supporting both humans and agents at the same time.

And I think that may be one of the biggest shifts underneath this year’s I/O announcements.

For the first time, frameworks no longer felt entirely human-first.

Google was not only showing how AI could build software.

The more sessions I watched, the more it started feeling like software ecosystems themselves may slowly reorganize as AI systems become active participants inside them.

Search is starting to behave more like software generation

One of the moments I kept returning to from the Google I/O ’26 Keynote was watching Search generate interactive software experiences directly inside the results.

Not just summaries.

Not just responses.

Actual usable interfaces.

A planner.

A simulator.

A generated workflow.

The important part was not the interface itself.

It was what produced it.

Traditionally, developers built applications and search engines helped users discover them.

Now the platform itself is beginning to generate software experiences dynamically for specific situations.

Not permanent applications.

Temporary ones.

Software created for the exact moment it is needed.

The examples shown during the Google I/O ’26 Keynote are probably the clearest way to understand this direction.

Watching Search generate interactive planners and visual simulators in real time felt very different from traditional search demos.

It felt closer to software being assembled during the interaction itself.

And once software starts behaving like this, the frameworks underneath it have to change too.

The Human Clipboard Problem

Another moment I kept thinking about came from Chrome DevTools for Agents.

Mostly because it addressed a workflow almost every developer using AI tools already knows.

You ask AI to generate code.

Something breaks.

You copy the error.

Paste it back into the assistant.

Wait for another response.

Then repeat the cycle again.

The developer becomes the connection between:

  • the runtime
  • the browser
  • the tooling
  • and the AI system

But now agents can:

  • inspect the DOM
  • run Lighthouse audits
  • read error logs
  • attempt fixes
  • validate changes themselves

That changes the role of the developer too.

Because once AI systems can directly interact with tooling, they stop behaving like passive assistants.

They start participating inside the workflow itself.

This part came from the Developer Keynote (Google I/O ’26), especially the Chrome DevTools for Agents demonstrations.

That session introduced one of the most practical shifts from the entire event for me, mostly because it changes how AI interacts with debugging and runtime tooling itself.

Websites are changing too

The same pattern appeared again with WebMCP.

For most of the web’s history, websites were designed almost entirely for human interaction.

Humans:

  • navigate menus
  • press buttons
  • read layouts
  • move through interfaces visually

But WebMCP introduces the idea that websites should expose capabilities directly to agents too.

Not visually.

Structurally.

The Developer Keynote referred to this as improving “Agent Experience.”

And the more I thought about it, the more important that phrase started feeling.

Because once software ecosystems begin optimizing for agents too, the internet itself starts becoming something different.

Not only human-readable.

Machine-readable in a much deeper way.

The conversations around WebMCP and “Agent Experience” also came from the Developer Keynote (Google I/O ’26).

I found myself returning to that part of the session because it changes how the web itself is expected to behave when agents become part of normal software workflows.

The tension underneath all of this

I do not think any of this means:

  • designers disappear
  • frontend developers disappear
  • or interfaces become unpredictable AI-generated chaos

Actually, one of the most interesting parts across these announcements was the tension between:

  • flexibility
  • and consistency

Generated interfaces are powerful because they can adapt dynamically.

But people still rely on:

  • familiarity
  • stable navigation
  • predictable interaction patterns
  • consistent visual systems

That tension matters.

Because completely unrestricted generated interfaces would become exhausting very quickly.

And Google seemed aware of that throughout the sessions.

The direction did not feel like:
“remove structure completely.”

It felt more like:
“allow software to adapt within carefully defined boundaries.”

That is a very different idea.

And probably a much more practical one too.

What developers may actually need to adapt to

I do not think the biggest shift here is that AI will generate more code.

That conversation already exists everywhere.

The more interesting shift is that software ecosystems themselves are beginning to evolve around AI participation.

Frameworks are becoming more flexible.

Developer tools are becoming agent-aware.

Web standards are becoming machine-readable.

Interfaces are becoming dynamic.

And developers may slowly spend less time:

  • manually assembling static screens
  • wiring repetitive workflows
  • acting as the bridge between tools and AI systems

And more time:

  • defining boundaries
  • designing primitives
  • shaping behavior
  • reviewing generated systems
  • guiding adaptive workflows

That feels like a very different model of software development.

Not fully automated.

But definitely different from the one most of us learned.

Because the biggest change may not be AI replacing software development.

It may be software ecosystems slowly reorganizing themselves around AI participation.

Final thoughts

Going into Google I/O 2026, I expected better AI systems.

What I did not expect was how many sessions were pointing toward the same deeper shift underneath them.

Flutter becoming more style-neutral.

Search generating interfaces dynamically.

Web standards exposing capabilities directly to agents.

Developer tooling evolving around autonomous workflows.

Individually, these announcements seemed unrelated.

But together, they started feeling like pieces of the same transition.

For years, frameworks were designed almost entirely around human developers building interfaces for human users.

Now software ecosystems are beginning to evolve around AI systems too.

And I think future developers may look back at this moment less as:
“the year AI became smarter”

and more as:

“the year software ecosystems started reshaping themselves around AI participation.”

References

These sessions helped shape most of the ideas and observations in this article:

I also found these sessions helpful while thinking through the broader direction around AI workflows, tooling, and interface generation:

🤝 Stay in Touch

Place Find me here
GitHub building things → hemapriya-kanagala
LinkedIn resources & updates → hemapriya-kanagala
X random dev thoughts → @KanagalaHema

And seriously, if something here made sense (or didn’t), drop a comment.

Azure DevOps Structure Explained: Organizations, Projects, and Repos Without the Mess

Azure DevOps editorial cover

Azure DevOps editorial cover

A lot of Azure DevOps confusion starts before the first pipeline even runs.

Teams open the portal, create a few repos, maybe a couple of boards, and keep moving. That works for a while. Then a few months later, nobody is fully sure which project owns what, where the shared code should live, or whether a new team should create another project or just add one more repo somewhere random.

That is usually not a tooling problem.

It is a structure problem.

If your Azure DevOps layout is clean, permissions stay easier to manage, pipelines stay easier to reason about, and new developers can understand the system faster.

In this post, I will break down the practical hierarchy in simple words:

  • Organization
  • Project
  • Repository

And more importantly, I will explain how I think about the boundaries between them.

The basic hierarchy

Azure DevOps usually works like this:

  1. Organization is the top-level container
  2. Projects sit inside the organization
  3. Repos sit inside projects
  4. Pipelines, boards, artifacts, and permissions usually attach at the project level and then get refined further

Here is the structure visually:

Azure DevOps organization project repo diagram

What an organization should represent

An Azure DevOps organization is the highest-level workspace.

In most real setups, one organization maps to one company, one startup, one internal platform group, or one major business unit.

Think of it as the outer shell that owns:

  • users
  • billing
  • global access patterns
  • the set of projects your teams work in

A good default is:

  • one company = one organization

You usually do not want a new organization for every product or every team unless there is a strong reason such as:

  • legal separation
  • isolated billing
  • hard security boundaries
  • completely different admin ownership

If you split organizations too early, you create extra admin work and make cross-team visibility worse.

What a project should represent

A project is where most teams should make their main structural decisions.

This is the level where Azure DevOps starts to feel like a working delivery environment instead of just a container.

A project commonly groups together:

  • boards
  • repos
  • pipelines
  • service connections
  • artifacts
  • team-specific permissions
  • delivery workflows

The clean mental model is:

  • one project = one product, platform, or delivery stream

That means a project should represent a chunk of work that has enough internal connection to belong together.

Examples:

  • customer web platform
  • internal data platform
  • mobile app ecosystem
  • infrastructure automation platform

That does not always mean one project per microservice.

If you create too many projects, you make navigation, permissions, reporting, and pipeline management more annoying than they need to be.

What a repo should represent

A repo is where the actual code lives.

Inside a project, you can have one repo or many repos depending on how your system is built.

A repo might represent:

  • one application
  • one backend service
  • one frontend
  • one infrastructure codebase
  • one shared SDK or library

A healthy repo split usually follows ownership and deployment boundaries.

Good repo examples inside a project:

customer-platform/
  frontend-app
  backend-api
  infra-terraform
  shared-components

That split makes sense when those codebases:

  • change at different speeds
  • deploy independently
  • have different owners
  • need different pipeline logic

If everything always ships together and is maintained by the same small team, a monorepo can also be totally fine.

A practical way to decide the boundary

Here is the shortcut I use.

Use a new organization when:

  • admin ownership is fully different
  • billing or compliance must be isolated
  • the systems should not share visibility by default

Use a new project when:

  • a product or platform has its own delivery lifecycle
  • the team wants separate boards, pipelines, and permissions
  • the work is important enough to deserve its own operating space

Use a new repo when:

  • code should version separately
  • deployment should happen separately
  • ownership is different
  • the codebase is becoming noisy or hard to review as one unit

That is the cleanest rule set for most teams.

Example setup that scales well

Say your company has three main delivery areas:

  • customer-facing web product
  • mobile app stack
  • data platform

A clean Azure DevOps layout could look like this:

Organization: nimay-tech

Project: Customer Platform
  Repo: frontend-app
  Repo: backend-api
  Repo: infra-terraform

Project: Mobile Platform
  Repo: mobile-app
  Repo: shared-sdk

Project: Data Platform
  Repo: data-platform

That structure works because each project has a clear mission, and each repo has a clear code boundary.

It is easy to answer:

  • where the mobile shared code lives
  • where the web infra code lives
  • which project owns the related pipelines
  • which team should get access

That clarity saves time every single week.

The mistake people make too early

The most common mistake is either:

  • throwing everything into one giant project forever
  • splitting into too many projects too early

Both create pain.

If everything is in one giant project

You usually get:

  • messy repo lists
  • unclear board ownership
  • too many pipelines in one place
  • broad permissions that are harder to manage
  • shared noise across unrelated teams

If you create too many projects too early

You usually get:

  • duplicated setup work
  • scattered dashboards
  • more service connection management
  • harder cross-team reporting
  • random inconsistency in how teams configure things

The win is not maximum separation.

The win is useful separation.

How pipelines fit into this

Pipelines usually follow the repo and project structure.

A common pattern is:

  • repo-specific build pipeline
  • environment-specific release or deploy stages
  • project-level visibility for delivery flow

For example:

trigger:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

steps:
  - script: npm ci
  - script: npm test
  - script: npm run build

That pipeline belongs naturally with the repo that owns the code. But it still benefits from project-level organization because related teams can find the rest of the delivery assets in one place.

Monorepo vs multi-repo in Azure DevOps

This question comes up a lot.

There is no universal winner.

Monorepo is usually better when:

  • one team owns the whole system
  • components change together often
  • shared tooling matters more than separation
  • the deployment model is still simple

Multi-repo is usually better when:

  • teams own different services
  • services deploy independently
  • access should differ across codebases
  • the platform is growing and review boundaries matter more

Do not force multi-repo because it sounds more advanced.
Do not force monorepo because it feels simpler today.
Pick the one that matches ownership and deployment reality.

The rule that keeps the whole thing sane

If you are stuck, ask one question:

Will this boundary make ownership clearer or more confusing six months from now?

That question cuts through a lot of overthinking.

Good Azure DevOps structure is really about:

  • clean ownership
  • clean visibility
  • clean delivery flow
  • clean permission boundaries

If your structure improves those four things, you are probably making the right call.

Final takeaway

The easiest model to remember is this:

  • Organization = company or top-level admin space
  • Project = product, platform, or delivery stream
  • Repo = code boundary

Start there.

Do not over-split on day one, but do not let everything pile into one bucket either.

A clean Azure DevOps structure makes every other part of DevOps easier: pipelines, boards, permissions, onboarding, and ownership.

That is the real W.

References

  1. Microsoft Learn, About organizations, projects, and collections
    https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/about-accounts
  2. Microsoft Learn, Create a project in Azure DevOps
    https://learn.microsoft.com/en-us/azure/devops/organizations/projects/create-project
  3. Microsoft Learn, About Azure Repos
    https://learn.microsoft.com/en-us/azure/devops/repos/get-started/what-is-repos
  4. Microsoft Learn, What is Azure Pipelines?
    https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/what-is-azure-pipelines

A Simple React Hook for localStorage State, Expiry, and Sync

When we build React apps, we often need to save small values in the browser.

For example:

  • selected theme
  • user settings
  • form draft
  • search filters
  • temporary session value
  • cached API response

The browser already gives us localStorage, but using it directly in React can become repetitive.

We usually need to:

  • read the value from localStorage
  • parse the saved JSON
  • handle missing or broken data
  • update React state
  • write the new value back to localStorage
  • remove the value when needed
  • sync the value between tabs
  • expire old values after some time

This is why I built @gks101/localyx.

It is a small React hook that helps you use localStorage like normal React state.

Main Features

  • simple useState-like API
  • saves state in localStorage
  • Same Tab and Cross Tab Sync
  • TTL expiry support
  • absolute and sliding expiry strategies
  • namespace support
  • custom serializer support
  • custom encrypt and decrypt support
  • helper function to check remaining TTL

Working demo:

https://localyx.vercel.app/

GitHub repo:

https://github.com/gaurav101/localyx

NPM package:

npm install @gks101/localyx

What Problem Does It Solve?

React state is simple:

const [value, setValue] = useState("hello");

But normal React state is lost when the page is refreshed.

localStorage can keep data after refresh, but it does not work like React state by default.

So I wanted a hook that feels like useState, but also saves the value in localStorage.

That is the main idea of useLocalStorageState.

Basic Usage

First install the package:

npm install @gks101/localyx

Then import the hook:

import { useLocalStorageState } from "@gks101/localyx";

Now use it inside a component:

import { useLocalStorageState } from "@gks101/localyx";

function ThemeToggle() {
  const [theme, setTheme, clearTheme] = useLocalStorageState("theme", "light");

  return (
    <div>
      <p>Current theme: {theme}</p>

      <button onClick={() => setTheme(theme === "light" ? "dark" : "light")}>
        Toggle Theme
      </button>

      <button onClick={clearTheme}>
        Reset Theme
      </button>
    </div>
  );
}

This looks very close to useState.

The hook returns three values:

const [state, setState, removeValue] = useLocalStorageState(key, initialValue);

Step 1: The Key

The first value is the localStorage key.

useLocalStorageState("theme", "light");

Here, "theme" is the key.

The hook will save the value in localStorage using this key.

Step 2: The Initial Value

The second value is the default value.

useLocalStorageState("theme", "light");

Here, "light" is the initial value.

The hook will use this value when:

  • nothing is saved in localStorage
  • the saved data is invalid
  • the saved data has expired
  • you remove the value

Step 3: Reading the State

The first returned value is the current state.

const [theme] = useLocalStorageState("theme", "light");

You can show it in your UI:

<p>Current theme: {theme}</p>

When the page refreshes, the hook reads the saved value from localStorage.

Step 4: Updating the State

The second returned value is the setter function.

const [theme, setTheme] = useLocalStorageState("theme", "light");

You can use it like setState:

setTheme("dark");

You can also use the previous value:

setTheme((oldTheme) => oldTheme === "light" ? "dark" : "light");

When you call setTheme, the hook updates React state and also saves the new value in localStorage.

Step 5: Removing the Value

The third returned value removes the saved data.

const [theme, setTheme, clearTheme] = useLocalStorageState("theme", "light");

Use it like this:

clearTheme();

This removes the value from localStorage and sets the state back to the initial value.

TTL: Expire Data After Some Time

Sometimes we do not want to keep data forever.

For example, a session value or cache value should expire after some time.

TTL means “Time To Live”.

You can pass ttl in milliseconds.

import { useLocalStorageState } from "@gks101/localyx";

function SessionExample() {
  const [session, setSession, clearSession] = useLocalStorageState<string | null>(
    "session",
    null,
    {
      ttl: 30 * 60 * 1000,
    }
  );

  return (
    <div>
      <p>{session ?? "No active session"}</p>

      <button onClick={() => setSession("my-session-token")}>
        Login
      </button>

      <button onClick={clearSession}>
        Logout
      </button>
    </div>
  );
}

In this example, the session expires after 30 minutes.

After expiry, the hook removes the old value and returns the initial value.

Absolute TTL

By default, TTL uses the absolute strategy.

This means the timer starts when the value is saved.

Example:

const [token, setToken] = useLocalStorageState("token", null, {
  ttl: 10 * 60 * 1000,
  ttlStrategy: "absolute",
});

If the value is saved at 10:00, it will expire at 10:10.

Even if the user keeps using the app, the value still expires at that time.

This is useful for things like:

  • login session expiry
  • one-time temporary values
  • short-lived cache

Sliding TTL

The hook also supports sliding TTL.

const [searchCache, setSearchCache] = useLocalStorageState("search_cache", "", {
  ttl: 5 * 60 * 1000,
  ttlStrategy: "sliding",
});

Sliding TTL refreshes the timer when the value is read or synced.

This means active data can stay alive while the user is still using it.

This is useful for:

  • active session-like data
  • recently used filters
  • temporary UI cache

Run Code When a Value Expires

You can pass onExpire.

This function runs when the hook finds that the value has expired.

const [token, setToken] = useLocalStorageState("token", null, {
  ttl: 10 * 60 * 1000,
  onExpire: ({ key, value }) => {
    console.log("Expired key:", key);
    console.log("Expired value:", value);
  },
});

This is useful when you want to:

  • show a message
  • logout the user
  • clear related data
  • send an analytics event

Namespaces

Sometimes many features use localStorage.

To avoid key conflicts, you can use a namespace.

const [profile, setProfile] = useLocalStorageState("profile", null, {
  namespace: "my-app",
});

This stores the value using this key:

my-app:profile

Namespaces are useful when:

  • you have many stored values
  • you are building a large app
  • you want to separate app versions
  • you want cleaner localStorage keys

Same Tab and Cross Tab Sync

If you open the same app in two browser tabs, the hook can keep values in sync.

For example:

  1. Open the app in Tab 1.
  2. Open the app in Tab 2.
  3. Change the theme in Tab 1.
  4. Tab 2 gets the new value too.

The hook also handles same-tab sync.

This is useful when two components use the same storage key on the same page.

Get Remaining TTL

The package also exports getRemainingTtl.

This helper tells you how much time is left before a saved value expires.

import { getRemainingTtl } from "@gks101/localyx";

const remainingMs = getRemainingTtl("session", 30 * 60 * 1000);

console.log(remainingMs);

It returns:

  • a number if time is remaining
  • 0 if the value is already expired but not cleaned yet
  • null if the key does not exist or the value has no TTL data

You can also use it with a namespace:

const remainingMs = getRemainingTtl("session", 30 * 60 * 1000, {
  namespace: "my-app",
});

Custom Serialization

By default, the hook uses JSON.

For most React apps, that is enough.

But you can provide your own serializer if you need custom behavior.

const [value, setValue] = useLocalStorageState("custom", "hello", {
  serializer: {
    stringify: (value) => JSON.stringify(value),
    parse: (value) => JSON.parse(value),
  },
});

This is useful when you want more control over how data is saved and loaded.

Encoding and Encryption

By default, the hook stores data with Base64 encoding.

Base64 is only obfuscation. It is not real security.

If you need real security, pass your own encrypt and decrypt functions.

const encrypt = (raw: string) => {
  return btoa(raw);
};

const decrypt = (raw: string) => {
  return atob(raw);
};

const [secret, setSecret] = useLocalStorageState("secret", "hello", {
  encrypt,
  decrypt,
});

If you want to store plain JSON without Base64 encoding, pass null:

const [value, setValue] = useLocalStorageState("plain", "hello", {
  encrypt: null,
  decrypt: null,
});

Important note: do not store highly sensitive data in localStorage unless you fully understand the security risks.

SSR Safe

The hook has guards for browser-only APIs.

This means it can be used in apps that render on the server, like Next.js.

It will not directly access window or localStorage when they are not available.

Where Can You Use This Hook?

You can use this hook in many React projects.

Good use cases:

  • dark mode or theme setting
  • language preference
  • sidebar open or closed state
  • table filters
  • search filters
  • sort order
  • form draft
  • shopping cart for small projects
  • temporary session value
  • API cache with expiry
  • feature flag cache
  • onboarding step progress
  • recently viewed items

Avoid using it for:

  • passwords
  • private tokens without a security plan
  • very large data
  • data that must always be correct on the server

You Can Also Copy the Hook Into Your Project

You do not have to use the npm package if you do not want another dependency.

You can also copy the hook file from the GitHub repo and use it directly in your project.

Repo link:

https://github.com/gaurav101/localyx

This can be useful when:

  • you want full control over the code
  • you want to change the hook for your app
  • you are learning how the hook works
  • you want to avoid adding a package

Full API

Main hook:

useLocalStorageState<T>(key, initialValue, options?)

It returns:

[state, setState, removeValue]

Options:

{
  ttl?: number;
  ttlStrategy?: "absolute" | "sliding";
  namespace?: string;
  onExpire?: (ctx: { key: string; value: T }) => void;
  now?: () => number;
  encrypt?: ((raw: string) => string) | null;
  decrypt?: ((raw: string) => string) | null;
  serializer?: {
    stringify: (value: T) => string;
    parse: (value: string) => T;
  };
}

Helper:

getRemainingTtl(key, ttl, options?)

Final Thoughts

I built @gks101/localyx because I wanted a simple way to use localStorage in React without writing the same logic again and again.

It gives you a useState-like API, but also adds useful features like expiry, sync, namespaces, and custom serialization.

If you are a React beginner, you can start with the basic example first.

Later, when you need expiry or sync, the hook already supports it.

Package:

npm install @gks101/localyx

GitHub:

https://github.com/gaurav101/localyx

I sold you on /scratchpad. Then I migrated to /note.

Note-vs-scratchpad workflow

This is the third post in a series that started with vibe guiding and continued with an efficiency audit. Two months ago I laid out the four-skill loop: /start-issue writes a /scratchpad, /tackle-scratchpad-block executes one step at a time, status flags transition from pending to done, and /finish-issue reads what shipped and writes the PR title + description.

Why I changed my mind

The driver was token consumption. My employer gave each team dashboards of our AI tool usage and I was always the top consumer on mine — fairly high on the department chart too. That’s when I started to suspect it was not what I was building but how I was working with the tools.

/scratchpad‘s JSON step block and /tackle-scratchpad-block chain meant every task paid the same ceremony tax: status flags to transition, JSON to validate, a fenced block to parse. Useful scaffolding for multi-hour work. Overhead I was paying on every session. If my process was the reason I topped the dashboard, revisiting it was the cheapest experiment I could run.

Once I started looking, the redundancy wasn’t only scratchpad-vs-note. The templates the composite skills were generating — what /start-issue, /start-side-quest, /tackle-pr-comment, and /finish-issue wrote into their output documents — carried weight they didn’t need. A few examples from the audit that led to PR 130:

  • ## Files to Modify re-grouped information the Plan steps already named, just organized by file.
  • ## Documentation & Discoverability was a pre-populated checklist that /finish-issue already re-derived systematically at wrap-up time.
  • ## Acceptance Criteria had Claude copy criteria verbatim from the issue body it already had in context.
  • ## Why Split This Out was three hardcoded bullets that matched every side-quest ever created.

Each of these was small on its own. Cumulatively they were a per-session surtax, paid every time a skill fired. Defaulting to /note instead of /scratchpad was the biggest single cut, but the pattern of the change was wider: stop having the LLM write the same information twice, in different shapes, into different documents.

Claude was changing too

The other thing that shifted in those months was Claude itself. When I first wrote about this, the hard stops at every step made sense for the Claude that was around then. By the time PR 130 landed, Claude had gotten noticeably better at multi-step self-organization across an issue. I could lower my guard a bit. The control-freak posture I had started with was no longer needed. If I wanted an explicit checkpoint to manually review and commit, I could simply add a step in the generated /note.

Parallelism shifted too. Claude gained the ability to fan out workers and run agents in parallel within a single session, which meant on issues with independent pieces Claude could organize its own throughput faster than I could shepherd it through the /tackle-scratchpad-block chain. The gates that had once added safety started adding latency. By gating with /tackle-scratchpad-block, I was slowing Claude down by asking for more control.

What /note looks like

A /scratchpad‘s ## Implementation Plan section is a fenced JSON block with status fields, dependency arrays, and task lists:

{
  "finish_issue_on_complete": false,
  "steps": [
    {
      "id": "S001",
      "title": "Swap the token library",
      "status": "pending",
      "done_when": "Old lib removed from package.json, new one imported and passing existing tests",
      "depends_on": [],
      "files": ["package.json", "src/auth/token.ts"],
      "tasks": [
        "npm install new-token-lib, npm remove old-token-lib",
        "Update src/auth/token.ts imports",
        "Run the token test suite"
      ]
    },
    {
      "id": "S002",
      "title": "Update callers to match new token API",
      "status": "pending",
      "depends_on": ["S001"],
      "files": ["src/middleware/auth.ts", "src/routes/login.ts"],
      "tasks": [
        "Update verifyToken() call sites to new API shape",
        "Run full test suite"
      ]
    }
  ]
}

A /note‘s ## Plan section is the same information without the scaffolding:

1. Swap the token library — npm install new-token-lib, update imports, run the token test suite
2. Update callers to match the new token API — verifyToken() call sites, full test suite

No fenced JSON block. No status: pending. The plan says what to do and in what order; the LLM organizes its own execution in-session. The hard stops from the original workflow are still there: I read the plan before saying “go ahead”, I review the diff, I commit when I’m satisfied. The commit lands at the end of the note, not after every step. If I want an earlier checkpoint, I add an explicit gate in the plan.

Note-vs-scratchpad workflow

/scratchpad is still there. You opt in two ways: pass --scratchpad to the invoking skill, or use one of the natural-language triggers the skill watches for (“use a scratchpad”, “with step tracking”, “formal plan”, “track steps”). I still reach for it when I’m working through a larger GitHub issue and want a commit after each step instead of one at the end. This is especially helpful when I’m juggling two or three worktrees on the same project — iterative commits and step tracking keep me oriented when my attention is split across parallel branches.

What I am taking away

The /note flow still stops for plan review before anything happens, then gives the LLM more autonomy to fan out and move faster. The /scratchpad flow is still powerful — but you don’t always need a bazooka to kill a fly. Most days the lighter tool is enough, ending on a high note.

If you already use these skills

Pull the latest from the repo and run ./setup.sh to symlink the updated set. /start-issue will produce a /note on your next invocation. Nothing else changes — the rest of the loop carries on. If you want the old behavior, type --scratchpad.

Written using the same skills it describes, starting from issue #139. The plan was a note this time.