Mert

Most teams start debugging AI agents the same way they debug normal software: logs.

That works until the failure is not a single exception.

AI agents fail across decisions:

• the model picked the wrong tool
• the tool returned ambiguous data
• the agent ignored relevant context
• a retry changed the path
• the final answer looked correct, but came from the wrong chain of decisions

A log line can tell you what happened.
It rarely tells you why the agent chose that path.

That difference matters a lot once agents start doing real work.

The problem with agent logs

Traditional logs are linear.

A request comes in. Your system calls a service. A response comes back. Something succeeds or fails.

For normal backend systems, that is often enough. If a database query times out, a log line can point you to the query. If an API returns a 500, the log can tell you which call failed.

AI agents are different.

An agent run is not just a sequence of function calls. It is a decision process:

1. read the task
2. inspect context
3. choose a next action
4. call a tool
5. interpret the result
6. update the plan
7. decide what to do next

The failure may not be in the tool call itself. It may be in the decision that led to the tool call.

That is where logs start to break down.

Example: a tool calling failure

Imagine an agent that should look up a customer invoice.

The user says:

Can you check whether ACME has paid invoice INV-1042?

The agent calls the CRM search tool and gets no result.

The logs show something like this:

crm.search_customer({ "name": "ACME" })
→ []

final_answer: "I couldn't find an invoice for ACME."

From the logs, this looks straightforward. The CRM returned no result, so the agent answered that no invoice was found.

But the real problem might be somewhere else:

• the agent searched by customer name instead of invoice ID
• the CRM tool expected an account ID, not a name
• the invoice number was available in context but ignored
• a previous tool returned partial data and the agent misread it
• a retry changed the search parameter
• the agent failed to call the billing tool after the CRM returned empty

The failure is not simply “CRM returned empty.”

The failure is: why did the agent decide that CRM search by customer name was the right next action?

A log line usually cannot answer that.

What engineers need to know

When an agent fails, the useful debugging questions are different from normal software debugging.

You need to know:

• What did the agent know at this step?
• What options did it have?
• Which tool did it choose?
• Why did it choose that tool?
• What did the tool return?
• How did the agent interpret the result?
• Where did the first bad decision happen?

Logs usually capture pieces of this, but not the decision context around it.

That is why teams end up doing log archaeology: reading prompts, tool inputs, tool outputs, retries, traces, and final answers separately, trying to reconstruct the run after the fact.

More logs are not the same as better debugging

A common reaction is to add more logging.

Log the prompt. Log the tool input. Log the tool output. Log the final answer. Log token counts. Log latency. Log cost.

All of that is useful.

But it still leaves a gap: logs are observations, not explanations.

If an agent calls the wrong tool, the log can show the wrong tool call. It does not automatically show why the agent thought that was correct.

If an agent ignores context, the log can show the prompt contained the context. It does not show which part of the context the agent used or skipped.

If an agent succeeds after a retry, the log can show the retry. It does not always show how the retry changed the path.

For agents, the key unit of debugging is not just the event. It is the decision.

What better agent debugging looks like

For production agents, useful debugging needs more than flat logs.

It needs a replayable structure of the run:

• the goal the agent received
• the context available at each step
• every decision point
• every tool call and result
• retries and alternate paths
• the final answer
• the first point where behavior diverged from what was expected

This is closer to replaying a session than reading a log file.

A good agent debugger should let you inspect a failed run step by step and answer:

At this exact moment, why did the agent do this?

That is the question that matters.

Decision graphs instead of timelines

Many observability tools show agent activity as a timeline.

Timelines are helpful, but agents are not always best understood as timelines. They are better understood as decision graphs.

A timeline tells you:

Step 1 → Step 2 → Step 3 → Step 4

A decision graph tells you:

The agent had these options.
It chose this one.
That choice led to this tool call.
The result changed the next decision.
This is where the run went wrong.

That structure is much more useful when you are trying to debug behavior instead of infrastructure.

Logs are still necessary

None of this means logs are bad.

You still need logs for:

• errors
• latency
• cost
• request volume
• tool availability
• API failures
• security audits

Logs are part of the debugging picture.

They are just not the whole picture.

For AI agents, logs tell you what happened. Replay and decision context tell you why it happened.

A practical checklist

If you are building agents, ask whether you can answer these questions for a failed run:

• Can you replay the exact run?
• Can you see each tool input and output?
• Can you inspect what context was available before each decision?
• Can you identify the first bad decision, not just the final bad answer?
• Can you compare a failed run to a successful one?
• Can you explain why the agent chose a specific tool or path?

If the answer is no, more log lines probably will not solve the problem.

You need decision-level debugging.

Closing thought

AI agents introduce a new debugging problem.

The hard part is not always knowing whether a tool failed. The hard part is understanding why the agent chose that tool, how it interpreted the result, and where the reasoning path first went wrong.

That requires moving beyond flat logs toward replayable traces and decision graphs.

If you are working on production agents and have felt this pain, Opswald is building around exactly that problem: making agent runs easier to replay, inspect, and explain.

https://www.opswald.com/

Every morning, I see the same thing on my Twitter feed or the GitHub Trending page. A new Ruby gem is released that promises to “revolutionize” how we handle state, or a new “minimalist” alternative to a standard Rails tool that has 500 stars in its first week.

As developers, we love new things. We love “cleaner” syntax and smaller dependencies. But after building and maintaining Rails apps for years, I’ve stopped chasing the hype.

In 2026, my Gemfile is intentionally boring. I prefer gems that have been around for a decade over the latest shiny trends. Here is why boring tech is actually a solo developer’s greatest competitive advantage.

1. The “Abandoned Gem” Trap

When you use a brand-new gem that just trended on GitHub, you are taking a massive risk.

The maintainer might be excited today, but what happens in six months when they get a new job or lose interest in the project? If that gem breaks when Rails 8.1 comes out, you are the one who has to fix the source code or find a way to migrate away from it.

“Boring” gems like Devise, Sidekiq, or Kaminari have survived multiple Rails version upgrades. They have a team of maintainers and a massive community ensuring they don’t die. For a solo developer, “stability” is a feature that is more important than “cool syntax.”

2. AI and Documentation Edge

In the era of Vibe Coding and AI assistants like Cursor or ChatGPT, boring gems have a huge advantage: Training Data.

AI models have read millions of lines of code involving the “standard” gems.

• If I ask an AI to “Add a password reset flow using Devise,” it gets it right 100% of the time.
• If I use a shiny new auth library released three months ago, the AI will hallucinate and give me broken code because it hasn’t seen enough examples yet.

By using the most popular gems, you make your AI assistant 10x more effective. You can find the answer to any bug on StackOverflow in 30 seconds. With shiny new gems, you are the one writing the first StackOverflow question.

3. The “Standard” Rails Way (Omakase)

Rails is built on the idea of Convention over Configuration. When you stick to the “Boring” defaults, your app stays standard.

If I hire a freelancer or bring in a partner to help me with a project, they already know how Devise works. They already know how to check a Sidekiq queue. If I have replaced every standard tool with a “shiny” alternative, I have to spend three days teaching them my unique, custom stack.

4. Don’t Waste Your “Innovation Tokens”

There is a great concept in engineering called Innovation Tokens. You only get a few of them for each project.

If you spend your tokens on a “revolutionary new way to handle database IDs” and a “bleeding-edge frontend reactive library,” you have no tokens left to spend on the Actual Business Logic that makes your app unique.

I save my innovation for the features that my customers pay for. I don’t want to innovate on “how to upload a file” or “how to paginate a list.” I want those things to be as boring and invisible as possible.

Summary: My “Boring” Checklist

Before I add a new gem to my project, I check three things:

1. Age: Has it been maintained for at least 2 or 3 years?
2. Usage: Does it have millions of downloads on RubyGems?
3. Recent Activity: Was there a commit or a release in the last 6 months?

If the answer to any of these is “No,” I usually stick to the standard Rails default, even if the new gem looks “fancier.”

Boring gems allow you to focus on shipping products instead of fixing your tools. In the world of indie hacking, the developer who ships the most wins – and boring tech helps you ship.

Your Terraform pipeline is green. The deployment completes without errors. You grab a coffee.

Ten minutes later, you test your new Enterprise RAG application. It throws a 403 Forbidden. You open the Azure Portal, check the OpenAI Networking tab, and there it is: your Shared Private Link from AI Search is sitting in Pending.

Nobody told Terraform to approve it. Nobody told you it even needed approving.

This is the CI/CD killer of Azure AI infrastructure.

Why It Happens

AI Search must call OpenAI to vectorize data. The azurerm provider can successfully request this Shared Private Link — but it cannot approve its own request. The target resource (OpenAI) must explicitly accept the inbound connection. The standard provider has no method for this. Pipeline deadlocked. Someone has to click “Approve” in the Portal manually. ClickOps in 2026.

The AzAPI State Machine Fix

We bypass the azurerm provider entirely and talk directly to the Azure Resource Manager REST API using the azapi provider.

Because Azure generates a random GUID for the incoming connection, we can’t hardcode the ID — we read it at runtime:

data "azapi_resource_list" "pe_connections" {
  type                   = "Microsoft.CognitiveServices/accounts/privateEndpointConnections@2023-05-01"
  parent_id              = azurerm_cognitive_account.openai.id
  response_export_values = ["value"]

  depends_on = [azurerm_search_shared_private_link_service.openai_link]
}

The depends_on is critical — without it, Terraform queries the connection list before the link is even requested, returns empty, and the approval silently fails.

Then we filter for the Pending connection and approve it:

resource "azapi_update_resource" "approve_shared_link" {
  type = "Microsoft.CognitiveServices/accounts/privateEndpointConnections@2023-05-01"

  resource_id = try(
    [for conn in jsondecode(data.azapi_resource_list.pe_connections.output).value :
      conn.id
      if conn.properties.privateLinkServiceConnectionState.status == "Pending"
    ][0],
    ""
  )

  body = jsonencode({
    properties = {
      privateLinkServiceConnectionState = {
        status      = "Approved"
        description = "Approved via Terraform AzAPI Pipeline"
      }
    }
  })
}

The try() wrapper is not optional. On terraform destroy, the Shared Private Link is deleted before this resource is evaluated. Without try(), indexing [0] on an empty array crashes the destroy run and leaves orphaned resources in Azure.

After terraform apply, the link goes from Pending to Approved in under 30 seconds. No Portal access required.

Identity Chaining — Kill the API Keys

Auto-approving the link lets AI Search reach OpenAI. But static API keys will fail your compliance audit. Keys leak. Keys get committed to Git.

Disable local auth and use a System Managed Identity instead:

resource "azurerm_search_service" "search" {
  name                         = var.search_service_name
  sku                          = "standard"
  public_network_access_enabled = false
  local_authentication_enabled  = false

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_role_assignment" "search_to_openai" {
  scope                = azurerm_cognitive_account.openai.id
  role_definition_name = "Cognitive Services OpenAI User"
  principal_id         = azurerm_search_service.search.identity[0].principal_id
}

Zero credential management. When the AI Search instance is deleted, its identity and permissions are destroyed automatically.

Don’t Forget Private DNS

If your-instance.openai.azure.com still resolves to a public IP, the Azure Firewall drops the traffic and you get another opaque 403. Both services need their DNS zones linked to the VNet:

resource "azurerm_private_dns_zone" "openai_dns" {
  name                = "privatelink.openai.azure.com"
  resource_group_name = var.resource_group_name
}

resource "azurerm_private_dns_zone_virtual_network_link" "openai_vnet_link" {
  name                  = "link-openai-vnet"
  resource_group_name   = var.resource_group_name
  private_dns_zone_name = azurerm_private_dns_zone.openai_dns.name
  virtual_network_id    = azurerm_virtual_network.vnet.id
  registration_enabled  = false
}

registration_enabled = false — always. Automatic registration conflicts with centralized Hub & Spoke DNS management.

The free baseline (AzAPI auto-approval + basic networking) is on GitHub. The full article with complete VNet injection, Private DNS automation, and RBAC Identity Chaining is on my blog.

👉 Full article on woitzik.dev
👉 Free GitHub repo

An article circulating this week argues that faculty AI buy-in in higher education is a human factors engineering problem. The framing is correct. The path the piece describes skips the only two steps that matter, and the reason it skips them is structural, not pedagogical.

Start with the framework on its own terms. Human factors engineering, as a discipline, is most rigorous in the places where mistakes kill people — aviation, medicine, nuclear operations, military command. In none of those places does participatory design mean asking operators to author protocols for systems they have not yet operated. Crew Resource Management in commercial aviation was built by pilots who had logged thousands of hours on the platform. The accident-investigation literature, the cognitive task analyses, the checklists, the cross-checks — all of it sits downstream of operator-grade familiarity. Mature HFE practice in industrial settings treats prerequisite familiarity as a precondition for authorship, not as a parallel track. The order is fixed: operate the system, then design the safeguards.

The Step the Article Skips

The piece on faculty AI buy-in moves directly from “engage faculty as co-designers” to the outcomes — trust, transparency, governance, alignment with academic values. The prerequisite that holds every successful HFE program together never appears in the prose. The article asks faculty to co-design governance for tools the average faculty member has used for less than ten hours total, primarily in artificial training contexts.

What the article describes as co-design is closer to structured surveying. Faculty in a one-hour ChatGPT workshop can tell you what the demo felt like. They cannot tell you which boundaries a graduate seminar in clinical psychology needs around hallucination, or which retention defaults a research-methods course needs around student-generated prompts, or which provenance attribution rules an introductory writing course needs to keep its rubric honest. Those are the governance questions that matter. Surface familiarity produces surface governance.

What the article wants — discipline-specific, boundary-aware, defensible against edge cases — requires sustained use in the actual work. Faculty have to teach with the tool, grade against the tool, fail against the tool, and revise around the tool, for weeks or semesters, before they can author governance worth shipping. The discipline has a name for this kind of sustained operation in the actual work, and the name is praxis.

The Sequence That Makes the Outcomes Hold

The order matters because what comes out of a co-design session is exactly proportional to what its participants have actually done with the tool. A committee composed of operators who have spent a semester working through real student artifacts produces governance that survives the first stress case. A committee composed of policy interpreters who watched a demo produces governance that fails on contact with real coursework.

The fix is a sequencing change: praxis programs first, in disciplines, with real workflows and instructional artifacts, for at least one cycle of student work. Governance authorship after. The order is not optional, and the patience required to hold it is the part most institutions cannot afford politically. The faculty AI committee is sitting now; the spring catalog is locked; the student-affairs office wants a policy by July. So the committee is asked to ship governance from surface familiarity, and the result is governance theater.

The Second Step Hidden in Plain Sight

There is a second reason most institutions cannot deliver participatory design on AI even when they want to, and this one has nothing to do with pedagogy. By the time the faculty AI committee convenes, the enterprise contract has already been signed. Microsoft 365 Copilot for Education was procured eighteen months ago. The Google Workspace AI add-on, the OpenAI Edu tier, the Canvas-integrated AI tutor — all already on the books, with contract terms negotiated by procurement and counsel against the vendor’s standard data-protection and indemnity language.

The actual policy surface — data flows, retention windows, opt-out defaults, training-data carve-outs, accountability allocation, liability for hallucinated outputs reaching students — was decided in that contract. What the faculty AI committee ships from here is acceptable-use guidance inside a perimeter that was drawn elsewhere by people the committee never met. Co-design at the policy layer is downstream of choices that already foreclosed most of what could be co-designed.

This is the same structural pattern that shows up whenever software arrives through the procurement door instead of the operator door. The real co-design moment is the moment the contract is being negotiated. The operators are not in that room. By the time the operators are in the room, the room has been redecorated, and the decisions that needed operator input are the wallpaper.

The Reframe

The vocabulary the discussion runs on is part of the trouble. Buy-in is a marketing term. It implies persuading a population to consent to a decision that has been made. Higher-ed faculty are operators of AI workflows in disciplines where errors compound — into student records, into transcripts, into citations, into degree credentials. Authorship is the target the framework actually requires.

Authorship requires praxis. Praxis requires sustained operation in the actual work. Sustained operation requires that the procurement phase admit it is the policy phase, and seat operators where the contract gets negotiated. The article describes the destination correctly. Trust, transparency, governance, alignment — all of those are the right outcomes. The path it draws skips the only two steps that can produce them.

What This Looks Like In Practice

For an institution willing to do the work, the program structure is concrete. A nine-to-twelve-month operator residency for each discipline before its AI governance is drafted, structured around real student artifacts and graded course outputs. A standing seat for faculty operators in the procurement workstream, with veto power on terms that touch retention, training-data use, and provenance. An explicit acknowledgment in published policy that the contract terms are the upstream constraint, named and dated, so the limits of faculty authorship are honest and visible. A sunset clause on every contract that returns the policy surface to renegotiation when the operator cohort says the boundary is wrong.

None of this is the part faculty AI committees are currently asked to produce. All of it is the part the human factors engineering frame, taken seriously, would require. The framework is right. The implementations being shipped this year are the framework with the prerequisites filed off.

Higher education will get AI governance worth defending only when the operators arrive before the contract is signed and the praxis arrives before the committee meets. Until then, what most institutions are calling co-design is a way of borrowing the legitimacy of participation without paying its operating cost.

We’ve been trying to figure out a real answer to a question that keeps coming up: how do you measure whether someone is actually good at Claude Code, Codex, and the other AI coding tools? Not “do they use them,” but how good are they at using AI.

The first metric we looked at, like everyone else, was token usage. It’s the only number you can pull out of the box. Anthropic and OpenAI hand you spend data in the console. Spend correlates with cost. Cost is something finance asks about. So token usage becomes an easy first answer.

But obviously counting tokens sucks as a metric.

What we noticed when we looked at the actual sessions

When we started reading session logs from people who were clearly good with these tools, and people who were clearly struggling, both groups burned tokens. Sometimes the strugglers burned more.

A senior developer who has refined their workflow ships in 100,000 tokens what a junior chews through a million on. The high-skill move is fewer turns, sharper prompts, smaller context windows, more planning up front.

Rank by token spend and you end up rewarding the things that make people slower:

• Padded context with files that aren’t relevant
• Brute-forcing with longer and longer prompts
• Staying in chat mode forever instead of building reusable workflows

On top of all this, once a company starts measuring token usage, the incentive flips from accidental to deliberate. If your performance review or “AI adoption KPI” depends on token spend, the rational move is to burn tokens on purpose. We’ve already heard about people writing scripts that loop the model on busywork just to pump their number. The metric becomes the work, and the work stops mattering.

We’ve seen the same critique made about lines of code and commit counts. Volume isn’t skill. It just looks like it on a dashboard.

So we tried looking at something else

We started watching for things you can read from local session activity that show how someone configured the tool, not how much they spent on it.

Eight things kept clustering together. People with two of them usually had four. People with five usually had close to all of them.

• Customization — CLAUDE.md, AGENTS.md, custom slash commands, hooks. How much did they shape the tool to their workflow, vs run defaults?
• Parallel Agents — Are they using multiple agents working at once, or one chat at a time?
• Background Work — Tasks delegated to run unattended, or babysitting every turn?
• Tool Breadth — To what degree do they have MCP servers, skills, plugins wired into the environment?
• Planning — Plan mode, structured /spec / /plan workflows, or jumping straight to file edits?
• Repetition — Skill breadth and skill depth, measured separately. A lot of people install skills they never actually use.
• Custom Skills — Written their own reusable workflows for things they do more than once?
• Multi-Tasking — AI treated as a team running in parallel, or as a single chat window?

Plus a few tool-specific ones we added later for Codex and Cowork.

Each of these is observable from session activity. No self-report, no interview answer to game.

So we built a free metric

We turned the framework into AIQ Rank. AIQ Rank reads local session activity from whatever AI coding tools you’re using (Claude Code, Codex, Cursor, OpenCode, Cowork) and scores you 0-1000 across the eleven dimensions. Think of it as a credit score, but for AI fluency.

We made it free. It runs locally — transcripts never leave your machine. You get a number, a per-dimension breakdown, and a profile URL to share if you want to.

The score is the hook. The breakdown is the part that’s actually useful. When we ran it on ourselves the first time, the dimension that surprised us wasn’t a strength we expected. It was a weakness we hadn’t noticed.

What we’d suggest doing with it

If you ran a quick sanity check on your team — top 10% by token spend, top 10% by AIQ Rank — we suspect the overlap would be smaller than you’d expect. Some of the token-heavy people are still brute-forcing every problem in chat mode. Some of the lower-spend people quietly built skills, wired up MCPs, learned plan mode, and run parallel agents.

That gap is the interesting part. Not because token spend is bad data, but because it’s a different question than “who’s good at this.”

If you want to do this comparison across your own team, AIQ Rank has private team leaderboards — invite people, scores aggregate to a board only your group sees. Transcripts still never leave anyone’s machine.

What do you think?

Let us know what changes/improvements we should make and please try it out! Do you agree/disagree with these parameters?

Enter the one command from aiqrank.com. It takes about a minute to run. We’d love any constructive feedback!