TL;DR: Ollama is the fastest path to running local LLMs (one command to install, one to run). The Mac Mini M4 Pro 48GB (~$1,999) is the best-value hardware. Q4_K_M is the sweet spot quantization for most users. Open-weight models like GLM-5, MiniMax M2, and Hermes 4 are impressively capable for a wide range of tasks. This guide covers 10 inference tools, every quantization format, hardware at every budget, and the builders making all of this possible.

I’ve been setting up local inference on my own hardware recently — an M4 Pro Mac Mini running Ollama — and I wanted to compile everything I’ve learned into one place. This guide is as much for my own reference as it is for anyone else exploring this space.

The tooling in 2026 has matured to the point where a $600 Mac Mini can run 14B parameter models and a $1,600 setup handles 70B. Whether you want to reduce API costs for simple tasks, keep sensitive data private, build offline-capable apps, or just understand how these models actually work, there are real options now.

I still use Claude Code as my primary coding tool — local models aren’t a replacement for frontier cloud inference on complex tasks. But they’re genuinely useful for a lot of workflows, and the ecosystem is worth understanding. This guide covers the tools, formats, hardware, and people building the open-source ecosystem.

Get the full 14-page StarMorph Research PDF — detailed comparison tables, hardware buying guide, and thought leader profiles in a premium dark-mode report.

Tool Comparison Matrix

Ten tools, compared across what matters. Stars reflect community adoption as of March 2026.

Every tool above except LM Studio is open-source. Most build on top of llama.cpp — the foundational C/C++ inference engine that pioneered running LLMs on consumer hardware.

Ollama — The Developer Default

Ollama is the fastest path from zero to running local models. One command to install, one to run, and you get an OpenAI-compatible API on localhost:11434. It’s open-source (MIT), written in Go, and has 166k GitHub stars — the largest open-source AI project on GitHub by a wide margin.

# Install
curl -fsSL https://ollama.com/install.sh | sh

# Run a model
ollama run llama3

That’s it. No Python environments, no CUDA toolkit, no configuration files.

Why developers default to Ollama

• OpenAI + Anthropic API compatibility — Claude Code and OpenAI Codex CLI can use Ollama as a local backend. Your existing API client code works with minimal changes.
• Largest model registry — 100+ models available with ollama pull. One-command downloads.
• Performance — M3 Pro generates 40-60 tok/s on 7B models. Benefits from all llama.cpp optimizations (up to 35% faster from CES 2026 NVIDIA improvements).
• Image generation — Added to macOS in January 2026.
• Web search + structured outputs — Both added in 2026.

Where Ollama falls short

• GGUF-only for native format — safetensors/PyTorch models require a conversion step via Modelfile
• No GUI — third-party frontends like Open WebUI fill this gap
• Slightly higher overhead than raw llama.cpp (the abstraction layer costs a few percent)
• Custom model importing requires creating a Modelfile rather than just pointing at a file

For most developers, Ollama is the right first tool. Start here, then graduate to other tools as your needs become more specific.

Get the free macOS Bootstrap Script — idempotent setup for Homebrew, Zsh, Node.js, Ollama, and 30+ dev tools in one command.

LM Studio — The Visual Explorer

LM Studio is the most beginner-friendly option — a desktop application where you browse models, click to download, and start chatting. Zero terminal knowledge required. Closed-source but free for personal use.

What makes it stand out:

• Built-in model browser with one-click downloads from Hugging Face
• MLX backend on Apple Silicon for optimized Mac inference
• Split-view chat for side-by-side model comparison
• v0.4.0 (January 2026) added parallel inference with continuous batching
• New headless “llmster” daemon enables server-only deployment on Linux boxes without the GUI

Formats: GGUF (llama.cpp backend), MLX (Apple Silicon only), safetensors. No EXL2 or GPTQ support.

API: OpenAI-compatible on localhost:1234. Python and TypeScript SDKs hit v1.0.0.

LM Studio is ideal for model evaluation — browse, download, compare side-by-side — before deploying with Ollama or vLLM in production.

vLLM — Production GPU Serving

If you’re deploying models on GPU infrastructure at scale, vLLM is the industry standard. It’s the performance leader with PagedAttention for memory-efficient KV cache management, continuous batching, and speculative decoding.

Benchmarks with Marlin kernels: AWQ achieves 741 tok/s, GPTQ achieves 712 tok/s. vLLM v0.16.0 (February 2026) expanded multi-GPU and multi-platform support to NVIDIA, AMD ROCm, Intel XPU, and TPU.

Formats: The widest range — safetensors, GPTQ, AWQ, FP8, NVFP4, bitsandbytes. This matters because GPU-optimized quantization formats like AWQ achieve better throughput than GGUF on NVIDIA hardware.

The catch: Linux-only for production, requires a dedicated NVIDIA/AMD GPU, complex setup compared to Ollama. Overkill for single-user local inference.

Use vLLM when: You’re serving multiple users, need maximum throughput on GPU hardware, or are deploying in production. The common developer workflow is: evaluate models with LM Studio, develop with Ollama, deploy with vLLM.

llama.cpp — The Foundation

llama.cpp is the C/C++ inference engine that everything else builds on. Created by Georgi Gerganov, it pioneered running LLMs on consumer hardware via quantization. In February 2026, the ggml/llama.cpp team joined Hugging Face.

Ollama, LM Studio, GPT4All, and KoboldCpp all use llama.cpp under the hood. It’s the engine — they’re the interfaces.

Why use it directly?

• Maximum control over inference parameters and model loading
• Widest platform support: macOS, Windows, Linux, Android, iOS, WebAssembly
• Best CPU inference performance — designed from the ground up for consumer hardware
• Defines and maintains the GGUF format standard

Stats: 98.6k GitHub stars, 1,038 contributors, 28 upstream commits per week. CES 2026 NVIDIA optimizations yielded up to 35% faster token generation.

Use llama.cpp directly when you need fine-grained control that Ollama or LM Studio don’t expose. Otherwise, use the higher-level tools — they give you 95% of the performance with much less configuration.

ExoLabs — Distributed Inference

Exo takes a fundamentally different approach: instead of running a model on one device, it splits the model across multiple devices connected peer-to-peer. No master-worker architecture — any device can contribute compute.

What’s been demonstrated:

• DeepSeek V3 (671B parameters) across 8 M4 Pro 64GB Mac Minis (512GB total memory) at ~5 tok/s
• DeepSeek R1 (671B) across 7 Mac Minis + 1 M4 Max MacBook Pro (496GB total)
• 2 NVIDIA DGX Spark + M3 Ultra Mac Studio = 2.8x benchmark improvement through disaggregated inference

Why this works with Apple Silicon: Unified memory is ideal for Mixture-of-Expert (MoE) models. All 671B parameters load across the cluster, but only 37B are computed per inference step. Apple devices become surprisingly cost-effective for MoE architectures.

Current status: Alpha (v0.0.15-alpha public, 1.0 not yet released). macOS native app requires Tahoe 26.2+.

If you have multiple Macs, Exo lets you pool them into a single inference cluster. The constraint is total unified memory across devices — and the network connecting them.

For a deep dive on which Mac Mini to buy for local inference (with current Amazon pricing and used market analysis), see my complete Mac Mini buying guide for local LLMs.

Other Notable Tools

Jan.ai

Open-source (AGPLv3) privacy-first desktop app. 41.1k stars, 5.3M+ downloads. Runs 100% offline via the Cortex engine (wraps llama.cpp). The standout feature is hybrid local + cloud switching — you can connect OpenAI, Anthropic, and local models in one interface, switching between them as needed. MCP integration for agentic workflows. Supports Windows ARM (Snapdragon).

LocalAI

The most comprehensive API-compatible local server. Drop-in replacement for OpenAI’s API that supports text, images, audio, video, embeddings, and voice cloning — all locally. Multi-backend support (llama.cpp, vLLM, transformers, diffusers, MLX). Anthropic API support added January 2026. Best for: developers with existing OpenAI API code who want to run locally with minimal changes.

KoboldCpp

Single-executable fork of llama.cpp with an integrated web UI. “One file, zero install” — download, double-click, select a model. Triple API compatibility (KoboldAI + OpenAI + Ollama endpoints). The best tool for creative writing and roleplay with built-in memory, world info, author’s notes, and SillyTavern integration.

GPT4All

Desktop app by Nomic AI with built-in LocalDocs for private document chat (RAG). The 2026 GPT4All Reasoner adds on-device reasoning with tool calling and code sandboxing. Backed by a funded company (Nomic AI). Best for non-technical users who want to chat with their documents privately.

MLX

Apple’s open-source ML framework purpose-built for Apple Silicon. Not a user-facing app — a framework that other tools use as a backend. Leverages unified memory with zero CPU-GPU data copying. Built-in mixed-precision quantization (4/6/8-bit per layer). M5 Neural Accelerators provide up to 4x speedup for time-to-first-token. Swift API for native macOS/iOS apps.

Quantization Formats and Tradeoffs

Quantization compresses model weights from 16 bits per weight (FP16/BF16) down to fewer bits. This is what makes it possible to run a 70B parameter model on consumer hardware.

GGUF: The Universal Format

GGUF was created by llama.cpp and is used by Ollama, LM Studio, KoboldCpp, GPT4All, and Jan.ai. The “K-quant” variants use mixed precision per layer, allocating more bits to important layers.

The practical ladder: Q4_K_M → Q5_K_M → Q6_K → Q8_0 as you get more memory. For most users, Q4_K_M is the sweet spot — 92% quality retention with 75% size reduction from FP16.

GPU-Optimized Formats

These formats are designed for NVIDIA GPUs and used by vLLM, ExLlamaV2, and transformers:

AWQ vs GPTQ: AWQ consistently outperforms GPTQ in both quality (95% vs 90%) and speed. AWQ preserves activation-aware important weights. For most GPU users, AWQ is the better choice.

GGUF vs AWQ/GPTQ: GGUF is universal — runs on CPU, GPU, and Apple Silicon. AWQ/GPTQ are GPU-only but provide better throughput on NVIDIA hardware. Use GGUF for flexibility, AWQ for maximum GPU throughput.

Choosing the Right Tool

By Use Case

By Skill Level

The Common Multi-Tool Workflow

Many developers in 2026 use a three-tool pipeline:

1. LM Studio for model discovery and evaluation (browse, download, compare side-by-side)
2. Ollama for development and integration (OpenAI-compatible API for app development)
3. vLLM for production deployment (maximum throughput on GPU infrastructure)

Hardware Buying Guide

The Fundamental Rule

For LLM inference, memory bandwidth is the bottleneck, not compute. A chip with higher GB/s generates tokens faster, even if it has fewer FLOPS. This is why an M3 Max (400 GB/s) generates tokens faster than an M4 Pro (273 GB/s) despite the M4 Pro being newer.

Memory Requirements by Model Size

Apple Silicon

Macs are uniquely suited for local LLMs because of unified memory — the GPU can access all system RAM, unlike discrete GPUs with fixed VRAM. RAM is not upgradeable on Apple Silicon. Buy the most you can afford.

Best value: Mac Mini M4 Pro 48GB (~$1,999) — runs 70B parameter models and costs less than a good GPU.

For a complete pricing breakdown of every Mac Mini configuration (new and used), with model compatibility tables and OpenClaw setup instructions, see my Mac Mini buying guide for local LLMs.

NVIDIA GPUs

VRAM is the limiting factor — models must fit in GPU VRAM or spill to CPU RAM at a significant speed penalty.

Budget Tiers

For building dedicated GPU inference servers at any budget ($150 to $5,000+), Digital Spaceport has the most comprehensive build guides I’ve found.

Get the free Ubuntu Bootstrap Script — 340-line idempotent VM setup for GPU inference servers with Zsh, Node.js, Docker, and 35+ tools.

Thought Leaders and Builder Strategies

These are the builders, researchers, and educators I’ve been learning from as I explore local inference. Whether they’re building tools, training models, or documenting hardware builds, they’re all making this ecosystem more accessible.

This list was inspired by 0xSero’s thread on people to follow in the local inference space. 0xSero is one of the most active voices in the open-source AI community, and his recommendations pointed me to many of the builders profiled below.

0xSero (@0xSero)

One of the most active builders in the local inference community. Publishes quantized models on Hugging Face using Intel AutoRound, making large models runnable on consumer hardware. Built vllm Studio for managing local models with chat template proxies that make Hermes, MiniMax, and GLM models compatible with OpenAI and Anthropic API formats. Also created ai-data-extraction for extracting chat and code context data from AI coding assistants for ML training, and fine-tuned models like sero-nouscoder-14b-sft trained on real coding conversations.

Andrej Karpathy (@karpathy)

The best teacher in AI. nanochat is the definitive entry point for understanding LLM training — a full-stack pipeline in ~8,300 lines of clean PyTorch covering tokenization, pretraining, SFT, and reinforcement learning. Trains a 561M ChatGPT clone in ~4 hours for ~$100 (or ~$15 on spot instances).

What makes nanochat uniquely effective for learning: one dial — transformer depth. This single integer auto-determines all other hyperparameters, so you can understand the full pipeline without needing hyperparameter tuning expertise.

His latest project, autoresearch, uses AI agents to autonomously optimize nanochat training configurations — AI improving AI training.

Peter Steinberger (@steipete)

His GitHub is a treasure trove. Peekaboo (macOS screenshot automation for AI agents), Summarize (CLI that extracts/summarizes any URL, YouTube, PDF, or audio), and OpenClaw (the fastest-growing GitHub project at 180k+ stars — an autonomous AI assistant that lives on your computer and self-modifies its own code).

His design principle: “CLIs are the universal interface that both humans and AI agents can actually use effectively.” Build CLI-first — it becomes the universal adapter between human workflows and agent automation.

Mario Zechner (@badlogicgames)

Pi is possibly the best, simplest open-source agentic loop to learn from. The pi-mono agent toolkit achieves power through radical minimalism: exactly 4 tools, a system prompt under 1,000 tokens, and a philosophy that “what you leave out matters more than what you put in.” Pi became the engine behind OpenClaw.

His anti-MCP argument is worth considering: popular MCP servers like Playwright MCP (21 tools, 13.7k tokens) consume 7-9% of context window before work begins. Pi’s alternative: CLI tools with README files — agents read the README only when needed, paying token cost only when necessary.

Takeaway: Start with 4 tools, not 40. Context engineering matters more than tool count.

Ahmad Osman (@theahmadosman)

The GPU king. Moderator of r/LocalLLaMA, deep practical knowledge across NVIDIA, Mac, and Tenstorrent hardware. Hosts GPU giveaways with NVIDIA (RTX PRO 6000 Blackwell for GTC 2026) and regularly interviews open-weight labs. His key blog post — Stop Wasting Your Multi-GPU Setup With llama.cpp: Use vLLM or ExLlamaV2 for Tensor Parallelism — is essential reading for anyone with multiple GPUs.

@sudoingX

Pushing the limits of single-GPU inference. Ran Qwopus (Claude Opus 4.6 reasoning distilled into Qwen 3.5 27B) on a single RTX 3090 at 29-35 tok/s with thinking mode. Ran Qwen 3.5 9B on a single RTX 3060 — “5.3 GB of model on a card most people bought to play Warzone.” Also discovered and published the fix for the Qwen 3.5 jinja template crash that broke OpenCode and Claude Code.

Takeaway: A single RTX 3090 can run 27B coding models at usable speeds — impressive for tasks like code completion and simpler agentic workflows.

Alex Cheema (@alexocheema)

Founder of ExoLabs. Oxford physics graduate. Pioneering distributed inference across Apple hardware — demonstrated 671B parameter models running across Mac Mini clusters. The Exo framework (42.7k stars) uses peer-to-peer topology with automatic device discovery and dynamic model partitioning. If you’re interested in Mac Mini and Mac Studio clustering, this is the person to follow.

Digital Spaceport (@gospaceport)

The homelab hardware teacher. End-to-end AI server builds at every budget — from $150 entry-level to $5,000 quad-3090 builds. His Proxmox guides for Ollama + Open WebUI and vLLM are the best I’ve found.

Numman Ali (@nummanali)

Prolific CLI tool builder. cc-mirror creates isolated Claude Code variants with custom providers — your main installation stays untouched. Supports Z.ai, MiniMax, OpenRouter, Ollama, and local LLMs. Quick start: npx cc-mirror quick --provider mirror --name mclaude. Also building OpenSkills (cross-agent skill sharing) and an agent-native SDLC pipeline.

Takeaway: You don’t need an Anthropic subscription to use Claude Code’s interface. cc-mirror lets you point it at local or alternative models.

Get the Claude Code Config Pack — CLAUDE.md template, settings, hooks, and keybindings for the complete AI coding setup.

Dax Raad (@thdxr)

Creator of OpenCode — an open-source terminal-first AI coding agent with 120k+ stars, 75+ LLM providers, and zero data storage. Also built SST and models.dev. His grounded take: “The productivity feeling is real. The productivity isn’t.” OpenCode is vendor lock-in free — use any model provider.

Julia Turc (@juliarturc)

The compression scientist. Her paper Well-Read Students Learn Better (706+ citations) proved that pre-training compact models before distillation yields compound improvements — foundational research for how modern quantized models work. Now building Storia.ai (YC S24). Her YouTube channel explains deep AI concepts without the hype.

Teknium (@Teknium1)

Head of Post-Training at Nous Research ($1B valuation). Co-creator of the Hermes 4 model family (open-weight, hybrid reasoning, up to 405B parameters). Built DataForge for graph-based synthetic data generation. The OpenHermes 2.5 dataset (1M samples) is openly available. Also drove decentralized training via INTELLECT-2 — a 32B model trained across 100+ GPUs on 3 continents.

Open-Weight Model Labs

Several people are driving the open-weight model ecosystem forward:

• Victor Mustar (@victormustar) — Head of Product at Hugging Face, shaping the UX of the platform hosting the world’s largest open model collection.
• Z.ai Community (@louszbd) — GLM-5 is 744B parameters (40B active), MIT licensed, #1 among open models on Text Arena with day-0 vLLM/SGLang support.
• Skyler Miao (@SkylerMiao7) — Head of Engineering at MiniMax. M2 is 230B total / 10B active, MoE architecture that scores well on benchmarks while being very cost-efficient to run. API pricing: $0.30/M input tokens.

Also Worth Following

• @Ex0byt — Making local inference on massive models possible on consumer hardware
• @alexinexxx — GPU kernel programming learner with strong drive and educational content
• @crystalsssup — Building top open-weight models and releasing research openly

Key Themes

1. The barrier to entry keeps dropping. Karpathy trains a ChatGPT clone for $15-100. Consumer hardware runs models that were data-center-only a year ago. You can start experimenting for $0 with Ollama on your existing machine.

2. Consumer GPUs are more capable than you’d expect. @sudoingX runs 27B coding models on a single RTX 3090 at usable speeds. Digital Spaceport documents builds starting at $150.

3. Apple Silicon clustering is an interesting frontier. Exo Labs runs 671B parameter models across Mac Mini clusters. Unified memory + MoE is surprisingly effective for the price.

4. Agent architecture should be minimal. Pi proves 4 tools and a 1,000-token system prompt outperforms bloated frameworks. Context engineering matters more than tool count.

5. Open-weight models are genuinely useful. GLM-5 (MIT), MiniMax M2, Hermes 4, Qwen — strong performance across many tasks, openly available. They’re great for simple workflows, privacy-sensitive tasks, and offline use. For complex reasoning and agentic coding, frontier cloud models still have a clear edge.

6. Local and cloud are complementary. cc-mirror and OpenCode let you use familiar interfaces with local or alternative models. The best setup for most developers is probably both — cloud for hard tasks, local for everything else.

This field evolves fast. I’m still early in my own local inference journey — learning what works, what’s overhyped, and where the real value is. If you’re curious, the easiest way to start is ollama run llama3 on your existing machine and see what it can do. No commitment, no cost.

Get the full 14-page StarMorph Research PDF — detailed comparison tables, hardware buying guide, and thought leader profiles.

Some links in this article are affiliate links. If you purchase through them, I may earn a small commission at no extra cost to you. I only recommend products I actually use.

Originally published at StarBlog

Liquid syntax error: Variable ‘{{% raw %}’ was not properly terminated with regexp: /}}/

Mock Service Worker (MSW) intercepts network requests at the service worker level. No monkey-patching fetch. No test-specific HTTP clients. Your code doesn’t know it’s being mocked.

Setup: Define Handlers

import { http, HttpResponse } from "msw";

export const handlers = [
  // GET request
  http.get("/api/products", () => {
    return HttpResponse.json([
      { id: 1, title: "Widget", price: 29.99 },
      { id: 2, title: "Gadget", price: 49.99 },
    ]);
  }),

  // POST with request body
  http.post("/api/products", async ({ request }) => {
    const body = await request.json();
    return HttpResponse.json({ id: 3, ...body }, { status: 201 });
  }),

  // Dynamic params
  http.get("/api/products/:id", ({ params }) => {
    return HttpResponse.json({ id: params.id, title: "Product" });
  }),

  // Query parameters
  http.get("/api/search", ({ request }) => {
    const url = new URL(request.url);
    const query = url.searchParams.get("q");
    return HttpResponse.json({ results: [], query });
  }),

  // Error responses
  http.delete("/api/products/:id", () => {
    return new HttpResponse(null, { status: 204 });
  }),
];

Testing: Vitest/Jest Integration

import { setupServer } from "msw/node";
import { handlers } from "./handlers";

const server = setupServer(...handlers);

beforeAll(() => server.listen());
afterEach(() => server.resetHandlers());
afterAll(() => server.close());

test("fetches products", async () => {
  const res = await fetch("/api/products");
  const data = await res.json();
  expect(data).toHaveLength(2);
  expect(data[0].title).toBe("Widget");
});

test("handles errors", async () => {
  // Override for this test only
  server.use(
    http.get("/api/products", () => {
      return HttpResponse.json({ error: "Server Error" }, { status: 500 });
    })
  );

  const res = await fetch("/api/products");
  expect(res.status).toBe(500);
});

Browser: Development Mocking

// src/mocks/browser.ts
import { setupWorker } from "msw/browser";
import { handlers } from "./handlers";

export const worker = setupWorker(...handlers);

// main.tsx
if (process.env.NODE_ENV === "development") {
  const { worker } = await import("./mocks/browser");
  await worker.start();
}

Network Behaviors

import { http, HttpResponse, delay } from "msw";

// Simulate slow network
http.get("/api/slow", async () => {
  await delay(3000);
  return HttpResponse.json({ data: "finally" });
});

// Simulate network error
http.get("/api/broken", () => {
  return HttpResponse.error();
});

// Streaming response
http.get("/api/stream", () => {
  const stream = new ReadableStream({
    async start(controller) {
      controller.enqueue(new TextEncoder().encode("chunk 1n"));
      await delay(100);
      controller.enqueue(new TextEncoder().encode("chunk 2n"));
      controller.close();
    },
  });
  return new HttpResponse(stream);
});

Passthrough: Let Some Requests Through

import { http, passthrough } from "msw";

http.get("/api/analytics/*", () => {
  return passthrough(); // Don't mock analytics
});

Testing scraping integrations? My Apify tools + MSW = reliable test suites for data pipelines.

Custom testing solution? Email [email protected]

My Thinking and Approach

Introduction

In this task, I worked with roles and permissions in SQL using the dvdrental database. The goal was to control access to different tables by creating roles, assigning privileges, and managing user groups.

This helped me understand how database security works in real-world applications.

Problem Statement

• Create roles with limited access
• Grant and revoke permissions
• Restrict access to specific columns
• Manage users using groups

My Initial Thought

At first, I thought:

• Roles are just users
• Permissions are simple

But I realized:

• Roles control access at different levels
• Permissions can be very specific
• Security is very important in databases

Key Observation

• GRANT is used to give permissions
• REVOKE is used to remove permissions
• Roles can be grouped for easier management
• Column-level access is possible

Solutions

Task 1: Create report_user with access to film table

CREATE ROLE report_user WITH LOGIN PASSWORD 'password';

GRANT SELECT ON film TO report_user;

Explanation:

• Creates a login role
• Allows only read access to film table

Task 2: Fix access to customer table

GRANT SELECT ON customer TO report_user;

Explanation:

• Grants permission to read customer table

Task 3: Restrict columns in customer table

REVOKE SELECT ON customer FROM report_user;

GRANT SELECT (customer_id, first_name, last_name) ON customer TO report_user;

Explanation:

• Removes full access
• Grants access only to specific columns

Task 4: Create support_user with limited permissions

CREATE ROLE support_user WITH LOGIN PASSWORD 'password';

GRANT SELECT ON customer TO support_user;

GRANT UPDATE (email) ON customer TO support_user;

Explanation:

• Can view customer data
• Can update only email column
• No DELETE permission given

Task 5: Remove film access from report_user

REVOKE SELECT ON film FROM report_user;

Explanation:

• Removes access to film table

Task 6: Create readonly_group

CREATE ROLE readonly_group;

GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly_group;

Explanation:

• Group with read-only access
• Applies to all tables

Task 7: Create users and assign to group

CREATE ROLE analyst1 WITH LOGIN PASSWORD 'password';
CREATE ROLE analyst2 WITH LOGIN PASSWORD 'password';

GRANT readonly_group TO analyst1;
GRANT readonly_group TO analyst2;

Explanation:

• Creates two users
• Adds them to readonly group

Final Understanding

• Roles help in managing permissions efficiently
• Access can be controlled at table and column level
• Groups simplify permission management for multiple users

Conclusion

This task helped me understand how to manage users, roles, and permissions in a database. It showed how important access control is for maintaining data security and integrity.

Why Existing Kubernetes Alone Falls Short for AI Inference Workloads

In March 2026, at KubeCon Europe 2026 in Amsterdam, NVIDIA officially announced the open-source project Grove. Grove is a Kubernetes API for declaratively defining and orchestrating complex AI inference systems in Kubernetes. The era of simply spinning up a single Pod is over.

Modern LLM inference architectures are evolving toward Disaggregated Inference patterns. The Prefill and Decode stages are separated, KV-Cache is shared, and each component must scale independently. The traditional Kubernetes combination of Deployment + HPA cannot properly manage such composite workloads.

Core Problem: When you scale out Prefill workers, you need to proportionally increase Decode capacity, and when adding new service instances, you must maintain the correct ratio of each component. Pod-level autoscaling cannot express these dependencies.

Grove was designed as a core component of the NVIDIA Dynamo framework to solve this problem, developed with integration into the CNCF ecosystem as a prerequisite. At the same KubeCon, NVIDIA also announced the historic decision to donate their GPU DRA driver to the CNCF.

Grove Architecture: 3-Tier CRD Model

Grove abstracts AI inference workloads into three Custom Resource Definition (CRD) layers. Each layer separates different levels of concerns while ensuring the entire system is managed as a single declarative resource.

[Grove 3-Tier CRD Architecture]

┌─────────────────────────────────────────────────────────────────┐
│ PodCliqueSet │
│ (Full inference service definition: startup order, scaling policies, Gang scheduling) │
│ │
│ ┌──────────────────────────┐ ┌──────────────────────────┐ │
│ │ PodCliqueScalingGroup │ │ PodCliqueScalingGroup │ │
│ │ (Groups that scale │ │ (Independent scale │ │
│ │ together) │ │ group) │ │
│ │ │ │ │ │
│ │ ┌─────────┐ ┌────────┐ │ │ ┌──────────────────┐ │ │
│ │ │PodClique│ │PodCliq.│ │ │ │ PodClique │ │ │
│ │ │(Prefill │ │(Prefill│ │ │ │ (Decode │ │ │
│ │ │ Leader) │ │Worker) │ │ │ │ Workers) │ │ │
│ │ └─────────┘ └────────┘ │ │ └──────────────────┘ │ │
│ └──────────────────────────┘ └──────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

PodClique: Role-Based Pod Groups

PodClique is a group of Pods performing a specific role. Each role – Prefill Leader, Prefill Worker, Decode Leader, Decode Worker – has independent configuration and scaling logic appropriate for its role. Each PodClique can have different GPU requirements, images, and commands.

PodCliqueScalingGroup: Tightly Coupled Scaling

PodCliqueScalingGroup bundles PodCliques that must be scaled together. For example, if 1 Prefill Leader and 4 Prefill Workers constitute a single model instance, they are managed as one ScalingGroup. This prevents inconsistencies where only Workers are scaled without the Leader during scale-out operations.

PodCliqueSet: Full Service Orchestration

PodCliqueSet defines the entire multi-component workload as a single declarative resource. It includes startup order (Workers start after Leader is ready), scaling policies, and Gang scheduling constraints.

In-Depth Analysis of Core Features

1. Multi-Level Auto-Scaling

Grove handles 3-tier autoscaling simultaneously. Independent yet coordinated scaling occurs at the individual component level (adding Prefill Workers during traffic spikes), related component group level (scaling Prefill Leader and Workers together), and entire service instance level (expanding overall capacity).

2. Flexible Gang Scheduling

Traditional Gang scheduling follows an “all-or-nothing” approach. Grove improves this by ensuring minimum viable component combinations while allowing flexible scaling. Prefill and Decode components can scale at different ratios based on workload patterns.

The Grove operator automatically creates PodGang resources when processing PodCliqueSet resources. PodGang encapsulates minimum replica guarantees, network topology preferences, and distribution constraints to pass to the scheduler.

3. Topology-Aware Placement

GPU topology directly impacts performance in AI inference. In NVIDIA GB200 NVL72 systems, placing related Prefill and Decode Pods in the same NVLink domain can significantly reduce KV-Cache transfer latency. Grove automatically performs such topology-aware placement by integrating with KAI Scheduler.

Tip: KAI Scheduler is an open-source subcomponent of NVIDIA Run:ai, recently adopted as a CNCF Sandbox project. It continuously monitors PodGang resources and makes placement decisions considering GPU topology and cluster locality.

4. Role-Aware Orchestration

Prefill Leaders and Workers execute different startup logic. Workers cannot start before the Leader is ready. Grove manages these explicit startup orders and role-specific configurations at the CRD level. When failed components restart, they reconnect correctly at the system level, and during rolling updates, network topology is preserved to maintain low-latency states.

Practical Deployment: Deploying Qwen3 Model with KV-Routing

Let’s walk through setting up an actual Grove environment and deploying a Qwen3 0.6B model using the Disaggregated Inference pattern step by step.

Step 1: Install Prerequisites

# Create namespace
kubectl create namespace vllm-v1-disagg-router

# Install Dynamo CRDs
export NAMESPACE=vllm-v1-disagg-router
export RELEASE_VERSION=0.5.1

helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz
helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz --namespace default

# Install Dynamo Operator + Enable Grove
helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz
helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz 
  --namespace ${NAMESPACE} --create-namespace 
  --set "grove.enabled=true"

Step 2: Verify CRD Installation

# Check Grove CRDs
kubectl get crd | grep grove

# Expected output:
# podcliques.grove.io
# podcliquescalinggroups.grove.io
# podcliquesets.grove.io
# podgangs.scheduler.grove.io
# podgangsets.grove.io

Step 3: Define DynamoGraphDeployment

Here’s the complete manifest for deploying the Qwen3 0.6B model with Prefill/Decode separation architecture:

# dynamo-grove.yaml
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: dynamo-grove
spec:
  services:
    # Frontend: KV-Routing based router
    Frontend:
      dynamoNamespace: vllm-v1-disagg-router
      componentType: frontend
      replicas: 1
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.1
      envs:
        - name: DYN_ROUTER_MODE
          value: kv   # Enable KV-Cache based routing

    # Decode Worker: Responsible for token generation
    VllmDecodeWorker:
      dynamoNamespace: vllm-v1-disagg-router
      envFromSecret: hf-token-secret
      componentType: worker
      replicas: 2   # Decode needs more instances than Prefill
      resources:
        limits:
          gpu: "1"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.1
          workingDir: /workspace/components/backends/vllm
          command: [python3, -m, dynamo.vllm]
          args:
            - --model
            - Qwen/Qwen3-0.6B

    # Prefill Worker: Responsible for processing input prompts
    VllmPrefillWorker:
      dynamoNamespace: vllm-v1-disagg-router
      envFromSecret: hf-token-secret
      componentType: worker
      replicas: 1
      resources:
        limits:
          gpu: "1"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.1
          workingDir: /workspace/components/backends/vllm
          command: [python3, -m, dynamo.vllm]
          args:
            - --model
            - Qwen/Qwen3-0.6B
            - --is-prefill-worker   # Prefill-only mode flag

Step 4: Deploy and Verify

# Deploy
kubectl apply -f dynamo-grove.yaml -n ${NAMESPACE}

# Check Pod status
kubectl get pods -n ${NAMESPACE}
# NAME                                          READY   STATUS    RESTARTS   AGE
# dynamo-grove-frontend-0                       1/1     Running   0          2m
# dynamo-grove-0-vllmdecodeworker-0             1/1     Running   0          2m
# dynamo-grove-0-vllmdecodeworker-1             1/1     Running   0          2m
# dynamo-grove-0-vllmprefillworker-0            1/1     Running   0          2m

# Check PodClique resource status
kubectl get podclique -n ${NAMESPACE}
# NAME                                   READY   REPLICAS
# dynamo-grove-0-vllmdecodeworker        2/2     2
# dynamo-grove-0-vllmprefillworker       1/1     1

# Test API endpoint
kubectl port-forward svc/dynamo-grove-frontend 8000:8000 -n ${NAMESPACE}

# Check model list
curl http://localhost:8000/v1/models

# Test inference request
curl -X POST http://localhost:8000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "What is Kubernetes?"}],
    "max_tokens": 512
  }'

Tip: Using KV-Routing mode (DYN_ROUTER_MODE=kv) allows the Frontend to select the optimal Decode Worker based on KV-Cache state. This increases cache hit rates and significantly improves TTFT (Time to First Token).

The Significance of NVIDIA GPU DRA Driver Donation to CNCF

At the same KubeCon Europe 2026, NVIDIA donated their GPU DRA (Dynamic Resource Allocation) driver to the CNCF. This is a historic event where GPU resource management governance transferred from a single vendor to the community.

DRA was promoted to GA (General Availability) in Kubernetes 1.34, fundamentally solving the limitations of the existing Device Plugin model. Key differences include:

With DRA, you can request resources based on detailed GPU attributes as follows:

# DRA ResourceClaim - GPU attribute-based request
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: gpu-for-inference
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      selectors:
      - cel:
          expression: >
            device.attributes["gpu.nvidia.com"].productName == "NVIDIA H100" &&
            device.attributes["gpu.nvidia.com"].memory >= 80000000000
    # Priority list: fallback to A100 if H100 unavailable
    - name: gpu-fallback
      deviceClassName: gpu.nvidia.com
      selectors:
      - cel:
          expression: >
            device.attributes["gpu.nvidia.com"].productName == "NVIDIA A100"

Grove + DRA + KAI Scheduler: Integrated Architecture

When these three projects combine, they complete a stack that can fully automate AI inference workloads in Kubernetes:

[Integrated Architecture Flow]

User Request
│
▼
┌──────────────────────────────┐
│ PodCliqueSet (Grove) │ ← Declare entire workload
│ │
│ ┌─── Frontend ────────┐ │
│ │ KV-Router │ │
│ └──────────────────────┘ │
│ │
│ ┌─── Prefill Group ───┐ │
│ │ Leader + Workers │ │ ← PodCliqueScalingGroup
│ └──────────────────────┘ │
│ │
│ ┌─── Decode Group ────┐ │
│ │ Leader + Workers │ │ ← PodCliqueScalingGroup
│ └──────────────────────┘ │
└──────────────────────────────┘
│
▼ Create PodGang
┌──────────────────────────────┐
│ KAI Scheduler │ ← Gang scheduling + topology placement
│ – GPU topology awareness │
│ – NVLink domain optimization │
│ – Resource fragmentation prevention │
└──────────────────────────────┘
│
▼ Process ResourceClaim
┌──────────────────────────────┐
│ DRA (GPU Driver) │ ← Attribute-based GPU allocation
│ – Automatic H100/A100 selection │
│ – MIG profile support │
│ – MPS sharing configuration │
└──────────────────────────────┘

Practical Adoption Considerations

Cluster Requirements

• Kubernetes Version: 1.34+ (DRA GA support required). For 1.33, DRA is beta and requires --feature-gates=DynamicResourceAllocation=true flag.

• GPU Nodes: Nodes with NVIDIA GPUs + NVIDIA DRA driver installation

• Helm 3.12+: Required for Dynamo chart installation

• Scheduler: KAI Scheduler or custom scheduler that understands PodGang resources

Cost Optimization Strategy

Companies adopting the Grove + DRA combination report GPU job queue wait times reduced from 15-45 minutes to 3-10 minutes, monthly GPU costs reduced by 30-40%, and developer experiment iteration speed improved by over 80%.

Caution: Grove is currently in v0.5.x stage, and production stability has not been fully validated. We recommend thorough testing in staging environments before production deployment. Particularly validate PodCliqueSet rolling update strategies and failure recovery scenarios in advance.

Integration with Existing Environments

Using DRA’s Extended Resource Mapping (alpha feature), existing workloads can use DRA-managed resources without modification. Traditional nvidia.com/gpu: 1 resource requests are internally processed through DRA, enabling gradual migration.

Ecosystem Outlook: GPUs Become First-Class Citizens in Kubernetes

Summarizing the trends from KubeCon Europe 2026, GPUs are now establishing themselves as first-class citizens in Kubernetes, equal to containers. NVIDIA’s donation of the DRA driver declares that GPU orchestration is not the domain of a specific vendor but a shared asset of the open-source community.

This ecosystem collaboration involving AWS, Broadcom, Canonical, Google Cloud, Microsoft, Nutanix, Red Hat, and SUSE is expected to accelerate the standardization of cloud-native AI infrastructure. Azure AKS already supports DRA and vGPU integration, and Google GKE officially supports DRA.

This change delivers a clear message to DevOps and MLOps teams. As AI workload complexity increases, Deployment and HPA alone are no longer sufficient. The new orchestration stack consisting of Grove, DRA, and KAI Scheduler represents essential technology that must be understood and mastered as Kubernetes evolves into the operating system for AI infrastructure.

References

• NVIDIA Grove GitHub Repository

• NVIDIA Technical Blog – Streamline Complex AI Inference with Grove

• NVIDIA at KubeCon Europe 2026 Announcements

• Kubernetes v1.34: DRA GA Updates

• NVIDIA DRA Driver for GPUs (CNCF)

• KAI Scheduler – CNCF Sandbox Project

• Grove Deployment Guide – NVIDIA Dynamo Docs

Originally published on ManoIT Tech Blog. Translated from Korean.

Your AI agent says “Done! Order placed successfully.”

But it ordered the wrong product. Or it ignored a tool error and hallucinated the rest. Or someone changed the system prompt mid-session and the agent quietly shifted its behavior.

The agent didn’t crash. It didn’t raise an exception. It just… did the wrong thing and reported success.

I’ve been building agents in production and I keep seeing the same failure patterns. Here are the 6 most common ones, with concrete code examples showing how each one happens — and how to detect it.

1. Hallucinated Tool Output

What happens: A tool returns an error, but the agent ignores it and proceeds as if the tool succeeded.

# The tool returns an error
search_result = search_api("Galaxy S25 Ultra")
# -> {"error": "Product not found"}

# But the agent's next decision says:
# "Based on the search results, the Galaxy S25 Ultra costs $470..."
#
# What search results?! The tool returned an error!

Why it’s dangerous: The agent builds its entire decision chain on data that doesn’t exist. Every subsequent step is based on a hallucination.

How to catch it: After every tool call that returns an error, check if the agent’s next reasoning acknowledges the failure:

# Check: did the agent mention the error in its reasoning?
tool_result = "error: not found"
next_reasoning = agent.last_reasoning

if "error" in tool_result and "error" not in next_reasoning:
    print("[WARNING] Agent ignored the tool error!")

2. Missing Approval for Critical Actions

What happens: The agent takes a high-stakes action (purchase, delete, send) without any approval checkpoint.

# Agent decides to purchase $32,900 worth of products
agent.decision("purchase 100 units of Galaxy S24 FE")

# But wait -- nobody approved this purchase!
# No human-in-the-loop, no policy check, no guardrail.
# The agent just... did it.

Why it’s dangerous: Financial transactions, data deletion, external communications — these should never happen without explicit approval. An autonomous agent with no guardrails is a liability.

How to catch it: Maintain a list of critical action keywords and check for a preceding approval:

critical_keywords = ["purchase", "delete", "send", "transfer", "pay"]

if any(kw in action.lower() for kw in critical_keywords):
    if not has_recent_approval():
        print("[WARNING] Critical action without approval!")

3. Silent Substitution

What happens: The user asks for Product A. Product A is unavailable. The agent delivers Product B without telling the user.

User: "Buy 100 units of Galaxy S25 Ultra"
Agent: searches... not found
Agent: finds Galaxy S24 FE instead
Agent: "Order completed! 100 units purchased for $32,900"

# The user thinks they got Galaxy S25 Ultra.
# They actually got Galaxy S24 FE.
# The agent never asked.

Why it’s dangerous: The user receives something they didn’t request. In B2B procurement, this can mean wrong specs, compatibility issues, or contract violations.

How to catch it: Compare the original request with the final output:

original_request = "Galaxy S25 Ultra"
final_output = agent.last_output

if original_request.lower() not in final_output.lower():
    # Agent delivered something different
    print("[WARNING] Output doesn't match the original request!")

4. Prompt Drift

What happens: The system prompt changes between agent steps — maybe an admin pushed a config update, maybe a middleware injected new instructions. The agent’s behavior silently shifts.

# Step 1: System prompt says "Always confirm purchases with the user"
# Agent: "Let me confirm this purchase with you..."

# --- someone changes the system prompt ---

# Step 3: System prompt now says "Prioritize order completion rate above 95%"
# Agent: "I'll substitute with an available product to complete the order"

# The agent's priorities changed mid-session.
# No one noticed.

Why it’s dangerous: Prompt drift can completely change agent behavior. If you’re not tracking the system prompt at each step, you can’t explain why the agent acted differently.

How to catch it: Record the system prompt at each step and diff it:

if previous_prompt != current_prompt:
    added = set(current_prompt.splitlines()) - set(previous_prompt.splitlines())
    removed = set(previous_prompt.splitlines()) - set(current_prompt.splitlines())
    print(f"[WARNING] PROMPT DRIFT: +{len(added)} lines, -{len(removed)} lines")

5. Repeated Failure (Blind Retries)

What happens: A tool fails, and the agent retries the exact same call multiple times without changing its approach.

Tool: flaky_api("query") -> timeout
Tool: flaky_api("query") -> timeout
Tool: flaky_api("query") -> timeout
Agent: "I'm having trouble, let me try again"
Tool: flaky_api("query") -> timeout

Why it’s dangerous: Wastes time, burns API quota, and the agent never adapts. A smart retry would try a different tool, change parameters, or escalate.

How to catch it: Count consecutive failures per tool:

tool_failures = {}
for event in trace:
    if event.type == "tool_error":
        tool_failures[event.tool] = tool_failures.get(event.tool, 0) + 1
        if tool_failures[event.tool] >= 3:
            print(f"[WARNING] {event.tool} failed {tool_failures[event.tool]} times!")

6. Retrieval Mismatch (Bad RAG Context)

What happens: The RAG pipeline retrieves a document with low relevance, and the agent uses it anyway.

# User asks about "refund policy for electronics"
# RAG retrieves: "laptop_reviews_2024.md" (similarity: 0.45)
#
# The agent uses this irrelevant document to answer
# the refund question, confidently citing wrong information.

Why it’s dangerous: Low-similarity retrieval means the context probably doesn’t match the query intent. The agent doesn’t know the context is wrong — it trusts whatever the RAG pipeline gives it.

How to catch it: Set a similarity threshold and flag anything below it:

if retrieval_result.similarity_score < 0.7:
    print(f"[WARNING] Low similarity ({retrieval_result.similarity_score}) -- context may be irrelevant")

The Real Problem: These Fail Silently

None of these failures crash your agent. No exception is raised. The agent completes successfully and reports a result.

The only way to catch them is to record the full decision trace and analyze it after the fact — like a flight recorder for AI agents.

I built Agent Forensics to do exactly this. It records every decision, tool call, and LLM interaction, then auto-detects all 6 patterns above:

pip install agent-forensics

from agent_forensics import Forensics

f = Forensics(session="order-123")

# Works with any framework -- or add one-line auto-capture:
agent.invoke({"input": "..."}, config={"callbacks": [f.langchain()]})

# Auto-detect all 6 failure patterns
failures = f.classify()
for fail in failures:
    print(f"[{fail['severity']}] {fail['type']}: {fail['description']}")

You can try the live demo that demonstrates patterns #1-4 in a single run:

git clone https://github.com/ilflow4592/agent-forensics.git
cd agent-forensics
pip install -e .
python demo.py --no-llm

What silent failures have you seen in your agents? I’d love to hear about patterns I might have missed.

This is a submission for the Notion MCP Challenge

What I Built

I built an automated workflow for my daily Chinese learning habit called “中国語3行日記” (3-Line Chinese Diary). Every day, I write a 3-sentence Chinese diary entry based on a daily theme, get it corrected by Claude, and post it to Bluesky — all without leaving Claude’s chat UI.

The system is powered by Notion + Claude MCP + GitHub Actions:

• Notion stores the daily themes (Questions DB) and my answers with correction notes (Answers DB)
• Claude MCP fetches the day’s theme, corrects my Chinese, records the answer and notes back to Notion, and helps me analyze my writing patterns over time
• GitHub Actions runs a daily cron job at 21:00 JST to automatically post scheduled answers to Bluesky

As of today, I’ve kept this habit going for over 225 days.

Video Demo

Show us the code

github.com/enoki-85/3-line-diary-scheduler

How I Used Notion MCP

The Daily Workflow

Each session follows this flow entirely within Claude’s chat UI:

1. I tell Claude which day’s theme to fetch → Claude retrieves it from the Notion Questions DB via MCP
2. I write my 3-sentence Chinese diary entry
3. Claude corrects my Chinese and explains the mistakes
4. Claude registers the final answer, pinyin, and correction notes to the Notion Answers DB via MCP
5. GitHub Actions picks it up at 21:00 JST and posts it to Bluesky automatically

Looking Back at My Writing Patterns

Beyond the daily workflow, I can ask Claude to retrieve past answers from Notion and review my correction notes. Since all notes are stored in Notion, I can look back at specific entries and spot recurring mistakes. I’m also planning to build a GitHub Actions workflow that summarizes each month’s entries into a dedicated Notion page, which would make longer-term pattern analysis much easier.

Why This Stack Works

Before this system, I went through a few iterations. I started by handwriting entries, then manually formatting and posting them. I later built a web app to streamline the process — but it required server costs, and since the themes are sourced from copyrighted material, I couldn’t make the repo public. I was also using the Claude API separately, adding unnecessary cost on top of my Claude MAX plan.

Switching to Notion MCP + Claude solved all of this:

• No extra server costs — GitHub Actions handles the automation for free
• No API costs — everything runs within my Claude MAX plan
• No friction — theme retrieval, correction, and Notion registration all happen in one chat
• The repo is private-safe — theme data lives in Notion, not in the codebase

The most important shift was removing everything except the actual learning. Writing and getting corrected is the core habit. Everything else — fetching themes, formatting posts, storing notes, posting to Bluesky — is now invisible. On days when motivation is low, having a smaller, frictionless target makes all the difference. Over 225 days in — though the Notion MCP setup is only about a month old — I feel like I can keep this going for a long time to come.