This is a submission for the Gemma 4 Challenge: Write About Gemma 4

I’ve been thinking about AI wrong for the past two years.

Not completely wrong. But there’s an assumption I’d quietly accepted without realizing it — that serious AI models require serious hardware. That if you wanted something capable enough to reason, handle images, work through multi-step problems, you needed a cloud API, or at minimum a machine with a real GPU. That the tradeoff between capability and accessibility was fixed: more of one meant less of the other.

Gemma 4 broke that assumption for me. And the thing that broke it wasn’t a benchmark number or a blog post. It was one sentence buried in the release notes:

The E2B model runs on a Raspberry Pi 5.

What Gemma 4 Actually Is

Google DeepMind released Gemma 4 on April 2, 2026, under an Apache 2.0 license — fully open-weight, commercially usable, no strings attached. It’s built from the same research as Gemini 3, which is a meaningful statement: this isn’t Google’s B-team effort. It’s frontier-level research packaged for the open ecosystem.

The family ships as four distinct variants, each targeting a different tier of hardware:

Every single one of them is multimodal from the ground up — text, images, video. The smaller models also handle audio natively. Context window is 128K for the edge models and 256K for the larger ones. All support function calling, multi-step reasoning, and configurable thinking modes.

That’s the table stakes. Here’s what’s actually interesting.

The Part That Stopped Me

The E2B model runs in approximately 1.5 GB of RAM at INT4 quantization.

A Raspberry Pi 5 with 8 GB of RAM can run it. Google published numbers confirming this — it’s not theoretical, it’s a tested deployment target.

I keep coming back to what that actually means. A Raspberry Pi 5 costs around $80. It’s a credit-card-sized single-board computer. And it can now run a multimodal AI model that handles text, images, and audio, with a 128K context window, with reasoning capabilities, offline, with no API calls, no cloud dependency, no subscription.

For the past two years, “run AI locally” has been something that required either a powerful laptop or a desktop with a dedicated GPU. The conversation around local AI models has largely been for people with the hardware to run 7B+ parameter models comfortably. E2B changes who that conversation is for.

Why This Is a Bigger Deal Than the Benchmarks

I want to be honest about something: the benchmarks for Gemma 4 are genuinely impressive, but I don’t think they’re the most important part of this release.

The 31B model sits at #3 on Arena AI’s open model leaderboard (ELO 1452 as of release), scores 89.2% on AIME 2026, and hits 80% on LiveCodeBench. Those are strong numbers that hold up against Qwen 3.5 and Llama 4 Scout in most categories.

But benchmark performance is something you read and nod at. The Raspberry Pi deployment is something that changes what you can build.

Think about the categories of projects that become possible when a capable multimodal model can run locally on $80 hardware:

Privacy-first applications. Medical data, personal journals, private documents — things you’d never send to a cloud API. A model that runs entirely on your own device means sensitive data never leaves it.

Offline-first tooling. Field work, remote locations, environments with unreliable connectivity. A capable AI model that works without internet is a different category of useful.

Embedded systems. NVIDIA Jetson devices, edge computing nodes, IoT hardware with more capability than a microcontroller. Gemma 4 E2B was specifically built with these targets in mind.

Projects for students and developers in contexts where cloud API costs are a real barrier. $80 hardware is still not nothing. But it’s a different category of accessible than “pay per token forever.”

How the Four Variants Actually Fit Together

After spending time with the model card and the community documentation, here’s how I’d think about which variant to reach for:

E2B — for genuine edge deployment. Raspberry Pi, smartphones, embedded hardware. If your constraint is RAM and you need offline capability, this is the one. Don’t reach for it if you’re on a laptop — you’re unnecessarily limiting yourself.

E4B — the sweet spot for most personal projects and local laptop experimentation. Fits in roughly 5 GB RAM. Strong enough for real tasks, accessible enough to run comfortably on most modern machines.

26B A4B (MoE) — deceptively efficient. It has 26 billion total parameters but only activates around 4 billion per token pass thanks to Mixture-of-Experts routing. If you have a consumer GPU with 8-12 GB VRAM, this is where serious capability starts without the full cost of running a dense model. The MoE architecture means inference is faster than the parameter count suggests.

31B Dense — for when you need the ceiling. All 30.7 billion parameters active on every pass. Highest benchmark scores, highest hardware requirements. Realistically a multi-GPU or high-VRAM workstation setup.

One thing I appreciate about how Google structured this: these aren’t just size variants of the same model. The E2B and E4B use Per-Layer Embeddings rather than traditional MoE routing — it’s a genuinely different architectural approach to making models efficient at the edge, not just a quantized-down version of the larger model.

What I’m Actually Thinking About Building

I’ll be honest: my hardware situation right now is a mid-range laptop and a Raspberry Pi 4 I’ve had sitting around for two years mostly running Pi-hole. Not exactly a GPU workstation.

Before Gemma 4, my options for local AI were limited enough that I mostly reached for APIs. After reading through the E2B specs, I’m genuinely reconsidering that default.

What I want to explore: a local document analysis tool that processes my own notes and files without any of that data leaving my machine. The 128K context window on E2B means I could feed in reasonably sized documents. The multimodal support means I could include images and diagrams. The offline capability means it works whether I’m connected or not.

That’s not a groundbreaking project. But it’s exactly the kind of project I’d been mentally filing under “not possible without a paid API” — and Gemma 4 moves it into the “actually try it this weekend” category.

The Honest Caveat

The E2B running on a Raspberry Pi is real, but context matters: Google tested with INT4 quantization, which trades some precision for memory efficiency. Performance at INT4 is lower than at higher precision levels. For tasks requiring nuanced reasoning or precise outputs, you’ll notice the difference.

The 31B dense model’s hardware requirements are also substantial — up to 19 GB RAM in some configurations. “Runs on consumer hardware” is true for the family, but not uniformly true across all variants.

And the MoE architecture of the 26B, while efficient, behaves differently in practice than a dense model of equivalent active parameter count. Worth benchmarking for your specific use case rather than assuming the numbers translate directly.

Why Open-Weight Matters Here

Gemma 4 ships under Apache 2.0. That’s not just a licensing detail — it’s what makes the edge deployment story meaningful for the long term.

A proprietary model that runs locally is still a dependency on the vendor’s continued goodwill, roadmap, and pricing decisions. An Apache 2.0 model is something you can fork, fine-tune, redistribute, and build on without those constraints. The 100,000+ community variants already built on earlier Gemma models exist because the license made that experimentation possible.

For developers building on top of Gemma 4 — fine-tuning it for a specific domain, integrating it into a product, deploying it at the edge — the license is as important as the capability.

Where I Land

I started this thinking about Gemma 4 primarily as another open model release in a busy year of open model releases. I’m ending it thinking about it as something slightly different: a signal that the capability-accessibility tradeoff in AI is more flexible than I’d assumed.

Running a capable multimodal model with a 128K context window offline on an $80 single-board computer is not the ceiling of what Gemma 4 can do. It’s the floor.

That reframe matters for what I think about building next.

Wrote this after going through the Gemma 4 model card, release blog, and community documentation. To get started: Gemma 4 on Hugging Face, Ollama setup guide, or the official Google AI docs.

You’ve used Git every single day for years. You’ve resolved merge conflicts at midnight, force-pushed to the wrong branch, lost an afternoon to a detached HEAD. You know git rebase -i well enough to teach it. You’ve explained pull requests to interns.

And yet — if someone put a gun to your head and asked “what actually happens when you run git commit?” — you’d say something like “it… saves a snapshot?” and hope they move on.

Don’t feel bad. Most developers interact with Git the same way they interact with their car engine: turn the key, go. The abstraction holds until the day it doesn’t — until git reflog saves your job or a corrupted object wrecks a deploy and you’re staring at .git/objects with no idea what you’re looking at.

This series is about closing that gap.

Not “here are some advanced Git commands” — there are plenty of cheat sheets for that. This is about understanding the machine underneath. The object store. The pack format. The wire protocol. The actual bytes on disk.

By the end of this post, you’ll be able to write a valid Git blob object in Go — one that git cat-file accepts without complaint — without calling a single Git command. Not as a party trick. As proof that you now understand the primitive Git is built on.

Stop thinking of Git as a history tracker. It’s a content-addressable filesystem with a thin VCS wrapper bolted on top.

When Linus Torvalds wrote the first version of Git in 2005, he didn’t start by designing branches or merges. He started by designing an object store. The VCS layer came after. That design decision explains everything that’s confusing about Git — and everything that’s elegant about it.

This post tears apart .git/objects, explains exactly what’s on disk, and walks through a Go implementation that writes a real, valid Git object from scratch. No magic. No waving hands at “Git uses SHA-1.”

The .git Directory Is the Repository

Clone a repo, delete every file except .git, and you’ve lost nothing. The working tree is just a checked-out view. The actual data — every version of every file ever committed — lives in .git/objects.

.git/
├── HEAD                 # pointer to current branch
├── config               # repo-local config
├── objects/             # THE database
│   ├── 4b/
│   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904
│   ├── info/
│   └── pack/            # packed objects (covered in Post 2)
└── refs/
    ├── heads/           # branch tips
    └── tags/

That file at 4b/825dc6... is the empty tree object. It exists in every Git repository ever initialized. The name IS its content — hash the empty tree structure and you’ll always get 4b825dc642cb6eb9a060e54bf8d69288fbee4904. That’s the contract.

Four Object Types, One Storage Format

Git has exactly four object types. Everything in your repository decomposes into these:

A critical insight: blobs don’t know their filename. The tree that references them knows the filename. This is why renaming a file in Git is “free” — the blob object is reused, only the tree changes. And that’s why two files with identical contents — anywhere in the repo, across any branch — share a single blob.

The Wire Format of a Loose Object

Every loose object follows the same binary format before being zlib-compressed:

[type] SP [size] NUL [content bytes]

For a file containing hellon:

blob 6hellon

That’s it. The blob type string, a space, the byte count of the content (not the header), a null byte, then the raw content. Hash this entire sequence with SHA-1 and you get the object’s identity. Write it zlib-compressed to .git/objects/[first-2-hex-chars]/[remaining-38-hex-chars].

Let’s verify this with raw shell commands before implementing it in Go:

# Write "hellon" as a blob
$ echo "hello" | git hash-object -w --stdin
ce013625030ba8dba906f756967f9e9ca394464a

# Confirm the file exists
$ xxd .git/objects/ce/013625030ba8dba906f756967f9e9ca394464a | head -3
00000000: 789c cb 48 cd c9 c9 57 30 b4 65 00 00 0b 27 02 a7  x..H..W0.e...'..

# Decompress and inspect
$ python3 -c "
import zlib
with open('.git/objects/ce/013625030ba8dba906f756967f9e9ca394464a', 'rb') as f:
    print(repr(zlib.decompress(f.read())))
"
b'blob 6x00hellon'

There’s the header: blob 6x00. Exactly as described.

Why Content-Addressability Matters

The SHA-1 hash of an object is its address. This has radical consequences:

Deduplication is automatic. If 10,000 commits all include an unchanged README.md, there’s exactly one blob object for it. The trees in those commits all point to the same hash. No dedup logic required — identical content produces identical hashes.

Corruption is detectable. Flip one bit in any object file and git fsck will catch it immediately. The stored filename no longer matches the hash of the decompressed content.

Caching is trivially correct. If you have object abc123, you have it forever, unchanged. There’s no cache invalidation problem. Object identity is content identity.

This is why git clone is trustworthy across mirrors — you don’t have to trust the mirror, only the hash.

⚠️ The SHA-1 → SHA-256 Transition

In 2017, Google’s SHAttered attack produced two PDFs with the same SHA-1 hash. For Git, the risk isn’t casual: it was demonstrated that it’s essentially possible to create two Git repositories with the same head commit hash but different contents — say, a benign source and a backdoored one.

Git has been transitioning to SHA-256. Git 2.29 (2020) introduced experimental support for SHA-256 via the sha256 object format. Repositories can now be initialized with git init --object-format=sha256. Git 2.51 further progressed this transition, and backward compatibility remains critical since Git is used by millions of developers. The tricky part: Torvalds noted that “Git doesn’t actually just hash the data — it prepends a type/length field to it. That usually tends to make collision attacks much harder, because you either have to make the resulting size the same too, or you have to be able to also edit the size field in the header.”

Building It: A Go Implementation

Enough theory. Here’s a self-contained Go program that writes a valid Git blob object — one that git cat-file can read back:

package main

import (
    "compress/zlib"
    "crypto/sha1"
    "fmt"
    "os"
    "path/filepath"
)

// GitObject represents a raw Git object before storage.
type GitObject struct {
    Type    string
    Content []byte
}

// Header returns the [type] SP [size] NUL prefix Git prepends to all objects.
func (o *GitObject) Header() []byte {
    return []byte(fmt.Sprintf("%s %dx00", o.Type, len(o.Content)))
}

// Store computes the SHA-1 hash of header+content, zlib-compresses the full
// payload, and writes it to the correct nested path under .git/objects/.
func (o *GitObject) Store(gitDir string) (string, error) {
    // Step 1: build the full store payload = header + raw content
    header := o.Header()
    payload := append(header, o.Content...)

    // Step 2: SHA-1 hash the payload — this IS the object's identity
    sum := sha1.Sum(payload)
    hash := fmt.Sprintf("%x", sum)

    // Step 3: derive the on-disk path: objects/[2-char]/[38-char]
    objDir := filepath.Join(gitDir, "objects", hash[:2])
    objPath := filepath.Join(objDir, hash[2:])

    // Step 4: skip if object already exists (content-addressable = idempotent)
    if _, err := os.Stat(objPath); err == nil {
        return hash, nil
    }

    if err := os.MkdirAll(objDir, 0755); err != nil {
        return "", fmt.Errorf("mkdir: %w", err)
    }

    // Step 5: write zlib-compressed payload to disk
    f, err := os.OpenFile(objPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0444)
    if err != nil {
        return "", fmt.Errorf("create object file: %w", err)
    }
    defer f.Close()

    w := zlib.NewWriter(f)
    if _, err := w.Write(payload); err != nil {
        return "", fmt.Errorf("compress write: %w", err)
    }
    if err := w.Close(); err != nil {
        return "", fmt.Errorf("compress close: %w", err)
    }

    return hash, nil
}

// ReadObject reads and decompresses a Git object, returning its type, size,
// and raw content. The inverse of Store().
func ReadObject(gitDir, hash string) (objType string, content []byte, err error) {
    path := filepath.Join(gitDir, "objects", hash[:2], hash[2:])

    f, err := os.Open(path)
    if err != nil {
        return "", nil, fmt.Errorf("open: %w", err)
    }
    defer f.Close()

    r, err := zlib.NewReader(f)
    if err != nil {
        return "", nil, fmt.Errorf("zlib: %w", err)
    }
    defer r.Close()

    var raw []byte
    buf := make([]byte, 4096)
    for {
        n, readErr := r.Read(buf)
        raw = append(raw, buf[:n]...)
        if readErr != nil {
            break
        }
    }

    // Parse header: find the null byte separating header from content
    nullIdx := -1
    for i, b := range raw {
        if b == 0x00 {
            nullIdx = i
            break
        }
    }
    if nullIdx == -1 {
        return "", nil, fmt.Errorf("malformed object: no null byte in header")
    }

    // Header format: "[type] [size]"
    var size int
    header := string(raw[:nullIdx])
    fmt.Sscanf(header, "%s %d", &objType, &size)

    return objType, raw[nullIdx+1:], nil
}

func main() {
    content := []byte("hello, git internalsn")

    obj := &GitObject{
        Type:    "blob",
        Content: content,
    }

    hash, err := obj.Store(".git")
    if err != nil {
        fmt.Fprintf(os.Stderr, "error: %vn", err)
        os.Exit(1)
    }

    fmt.Printf("wrote blob: %sn", hash)
    fmt.Printf("path: .git/objects/%s/%sn", hash[:2], hash[2:])

    // Round-trip: read it back
    objType, readContent, err := ReadObject(".git", hash)
    if err != nil {
        fmt.Fprintf(os.Stderr, "read error: %vn", err)
        os.Exit(1)
    }

    fmt.Printf("read back: type=%s content=%qn", objType, readContent)

    // Cross-verify with git cat-file
    fmt.Printf("nVerify with: git cat-file -p %sn", hash)
    fmt.Printf("Type check:  git cat-file -t %sn", hash)
}

Run this in any Git repo directory and then verify:

$ go run main.go
wrote blob: a97d3ee72059abfabb3cd99748d61e36bcc2b2c5
path: .git/objects/a9/7d3ee72059abfabb3cd99748d61e36bcc2b2c5
read back: type=blob content="hello, git internalsn"

Verify with: git cat-file -p a97d3ee72059abfabb3cd99748d61e36bcc2b2c5
$ git cat-file -p a97d3ee72059abfabb3cd99748d61e36bcc2b2c5
hello, git internals

Git accepts the object as legitimate — because it is legitimate. Same format, same hash algorithm. The Store function is exactly what git hash-object -w does internally.

The Two-Character Directory Split

Why objects/a9/7d3ee7... instead of objects/a97d3ee7...?

Filesystems degrade when directories contain too many entries. Old ext2 used linear lookups; even modern filesystems show measurable slowdown with hundreds of thousands of entries in a single directory. By splitting on the first two hex characters, Git caps any single directory at 256 entries (one per two-hex-char prefix). A repository with a million objects distributes them across at most 256 directories of ~3,900 objects each.

It’s a pragmatic workaround for filesystem limitations, not an algorithmic requirement. The SHA-256 transition keeps the same split (objects/[2]/[62]), just with longer filenames.

What a Tree Object Actually Looks Like

Trees deserve their own dissection. The format is binary, not the text format of blobs and commits:

[mode] SP [filename] NUL [20-byte-raw-SHA-1]
[mode] SP [filename] NUL [20-byte-raw-SHA-1]
...

Notice: raw bytes, not hex. Each entry is a mode string (like 100644 for a regular file, 040000 for a subdirectory, 100755 for executable), the filename, a null byte, and then 20 raw bytes of the referenced object’s SHA-1.

You can inspect this manually:

# Get the tree hash of HEAD
$ git cat-file -p HEAD
tree 9ab456...
...

# Inspect the tree (git pretty-prints it)
$ git cat-file -p 9ab456
100644 blob ce013625030ba8dba906f756967f9e9ca394464a    README.md
040000 tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904    src/

# But the raw bytes are not ASCII — decode them yourself:
$ python3 -c "
import zlib
with open('.git/objects/9a/b456...', 'rb') as f:
    raw = zlib.decompress(f.read())
# skip header (find null byte)
null = raw.index(b'x00')
entries = raw[null+1:]
# each entry: mode SP name NUL 20-bytes
i = 0
while i < len(entries):
    space = entries.index(b' ', i)
    mode = entries[i:space].decode()
    null2 = entries.index(b'x00', space)
    name = entries[space+1:null2].decode()
    sha = entries[null2+1:null2+21].hex()
    print(f'{mode} {name} -> {sha}')
    i = null2 + 21
"

Plumbing Commands You Should Know

Git ships with “porcelain” (user-facing) and “plumbing” (low-level) commands. When you’re working at this level, you want plumbing:

# Hash content without writing (dry run)
$ echo "test" | git hash-object --stdin

# Hash AND write to object store
$ echo "test" | git hash-object -w --stdin

# Inspect object type
$ git cat-file -t <hash>

# Print object content (pretty-printed for trees)
$ git cat-file -p <hash>

# Print raw content size
$ git cat-file -s <hash>

# Verify object database integrity
$ git fsck --unreachable

# Show all objects in the store
$ git rev-list --objects --all

# Manually trigger loose → packed object consolidation
$ git gc

The git cat-file -p output is what feeds into higher-level operations. Everything git does at the porcelain level eventually bottoms out in reads and writes of these four object types.

The Immutability Guarantee

One last thing worth internalizing: Git objects are immutable by construction. Once written, they never change. git commit --amend doesn’t modify the old commit — it writes a new commit object and updates the branch reference to point to it. The old commit still exists in .git/objects until garbage-collected.

This is why git reflog can recover “lost” commits. They’re not lost — the objects are still on disk. Only the references have been updated. git gc eventually removes unreachable objects, but by default only after they’ve been unreachable for 30 days.

The content-addressable store is the engine. Branches, tags, the index, the staging area — all of it is built on top of this one primitive: a compressed, hashed, immutable object store.

Next Up

In Post 2: Pack Files and Delta Compression, we dissect how Git takes thousands of loose objects and compresses them into a single binary packfile using delta encoding — storing only the differences between similar objects. For a file that’s 10 MB with changes generating 100 KB deltas each time, storing 100 versions as loose objects costs ~1 GB. Pack files with delta compression reduce that to roughly 10 MB. We’ll implement a pack file reader and inspect the binary format byte by byte.

Further Reading

• Git Internals – Git Objects — the canonical reference from Pro Git
• Git’s Database Internals I: Packed Object Store — GitHub Engineering’s deep dive
• Hash Function Transition — the official SHA-256 migration spec
• Git source: object.c — the actual C implementation
• SHAttered — the SHA-1 collision that accelerated the transition

The Problem We Were Actually Solving

I was tasked with leading the development of Veltrix, a high-traffic treasure hunt engine that needed to scale seamlessly to handle massive growth spikes without stalling. The configuration layer was crucial in determining whether our server would scale cleanly or stall at the first growth inflection point. We knew that a traditional monolithic architecture would not be able to handle the load, so we had to think outside the box. After weeks of research and discussions, we decided to go with an event-driven architecture, which would allow us to decouple our services and scale them independently. This decision was not taken lightly, as it would require a significant overhaul of our existing codebase and would add complexity to our system.

What We Tried First (And Why It Failed)

Before settling on the event-driven architecture, we tried to use a traditional request-response model, where each request would trigger a chain of synchronous calls to our various services. However, this approach quickly proved to be unscalable, as the number of requests increased exponentially with the growth of our user base. We were using Apache Kafka as our message broker, but we were not utilizing its full potential. Our services were tightly coupled, and any change to one service would have a ripple effect on the entire system. We were experiencing frequent errors, such as the infamous Kafka error: Broker may not be available, and our system was becoming increasingly unstable. It was clear that we needed to rethink our approach.

The Architecture Decision

We decided to go with a full-fledged event-driven architecture, where each service would produce and consume events, allowing us to decouple them completely. We used Apache Kafka as our event store, and each service would publish events to Kafka topics, which would then be consumed by other services. This approach allowed us to scale our services independently, and we were able to handle massive growth spikes without stalling. We also implemented a service discovery mechanism using Netflix’s Eureka, which allowed our services to register themselves and be discovered by other services. This approach added complexity to our system, but it was necessary to achieve the scalability we needed. We also had to implement a robust monitoring and logging system, using tools like Prometheus and Grafana, to ensure that we could detect and respond to any issues quickly.

What The Numbers Said After

After implementing the event-driven architecture, we saw a significant improvement in our system’s scalability and performance. We were able to handle a 10x increase in traffic without any issues, and our system was able to scale seamlessly to handle the load. Our error rates decreased significantly, and we were able to reduce our mean time to recover (MTTR) from 30 minutes to just 5 minutes. We were also able to reduce our infrastructure costs by 30%, as we were able to optimize our resource utilization and eliminate unnecessary instances. Our metrics showed that our system was able to handle 10,000 requests per second, with an average response time of just 50ms. We were able to achieve this using a combination of Kafka, Eureka, and our custom-built services, which were all running on Amazon Web Services (AWS).

What I Would Do Differently

In hindsight, I would have liked to have implemented a more robust testing strategy, to ensure that our system was thoroughly tested before going live. We experienced some issues with our service discovery mechanism, which caused some services to become unavailable for short periods of time. We were able to resolve these issues quickly, but it was a painful lesson to learn. I would also have liked to have implemented a more robust security strategy, to ensure that our system was secure and compliant with all relevant regulations. We were using a combination of AWS IAM roles and custom-built security mechanisms, but we could have done more to ensure the security of our system. Overall, I am proud of what we achieved with Veltrix, and I believe that our event-driven architecture was the right call for our system. It allowed us to scale seamlessly and handle massive growth spikes, while also providing a robust and reliable platform for our users.

Local RAG: Chat With Your Documents

Upload PDFs, code, research papers, or entire books — then ask your local LLM questions about them. No data ever leaves your machine.

What Is RAG? (Plain English)

RAG (Retrieval-Augmented Generation) means your LLM can look up information from your own documents before answering.

Think of it like this:

• Normal LLM: Has a great memory, but only knows what it learned during training
• RAG: The LLM gets a “cheat sheet” — your documents — that it can read before answering

💡 Analogy: Without RAG, the LLM is like a student taking a closed-book exam. With RAG, they get an open-book exam — and you get to write the book.

Real-World Uses

Option A: Built-in RAG in Open WebUI (Simplest)

If you already have Open WebUI installed, RAG is built-in.

How to Use It

1. Open http://localhost:3000 in your browser
2. Click the paperclip icon next to the chat input
3. Upload a PDF, .txt, .docx, or .md file
4. Wait for the “embedding” process to finish (usually 10-30 seconds)
5. Ask questions about the document

That’s it. No configuration needed.

Pro Tips

• Multiple documents: You can upload several files at once. Open WebUI indexes them all.
• Model choice: Use qwen3.6:27b or deepseek-r1:14b for best RAG quality — they have larger context windows.
• Document size: Open WebUI handles documents up to hundreds of pages. For very large documents, consider chunking them.

Option B: AnythingLLM (More Powerful)

AnythingLLM is a dedicated RAG application with more features than Open WebUI’s built-in system.

Installation

With Docker (Recommended):

docker run -d 
  -p 3001:3001 
  -v anythingllm:/app/server/storage 
  -e STORAGE_DIR=/app/server/storage 
  --name anythingllm 
  --restart always 
  ghcr.io/anythingllm/anything-llm:latest

Then open http://localhost:3001.

Without Docker:

Download from anythingllm.com and run the installer for your OS.

Configuration

1. Open AnythingLLM at http://localhost:3001
2. Create an admin account (local only — no data leaves your machine)
3. Go to Settings → LLM Provider
4. Select Ollama from the dropdown
5. Choose your model (e.g., qwen2.5:7b or deepseek-r1:14b)
6. Click Save

Now set up embeddings:

1. Go to Settings → Embedding Provider
2. Select Ollama
3. Choose an embedding model (AnythingLLM will download a small embedding model — about 500 MB)
4. Click Save

Uploading Documents

1. Click “New Workspace” and give it a name (e.g., “Research Papers”)
2. Click the upload icon (or drag and drop files)
3. Supported formats: PDF, DOCX, TXT, MD, CSV, JSON, code files
4. Click “Save and Embed”
5. Wait for indexing (progress shows in the UI)

Chatting With Your Documents

Once embedded, just type your question:

"What are the three main conclusions from these papers?"

AnythingLLM searches your documents for relevant passages and feeds them to the LLM along with your question. The result is accurate, sourced answers — not guesses.

🔥 Pro tip: AnythingLLM shows you which document each answer came from. Hover over the citation to see the exact source passage.

Option C: Manual RAG with LangChain (For Developers)

For maximum control, build RAG with Python and LangChain. This is particularly useful if you want to automate document processing.

Setup

pip install langchain langchain-ollama chromadb

Basic RAG Script

from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA

# 1. Load your documents
loader = DirectoryLoader("./my-docs/", glob="**/*.txt", loader_cls=TextLoader)
documents = loader.load()
print(f"Loaded {len(documents)} documents")

# 2. Split into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks")

# 3. Create embeddings and vector store
embeddings = OllamaEmbeddings(model="qwen2.5:7b")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# 4. Create RAG chain
llm = ChatOllama(model="qwen2.5:7b", temperature=0.3)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3})
)

# 5. Ask questions
while True:
    question = input("nAsk a question (or 'quit'): ")
    if question.lower() == 'quit':
        break
    answer = qa_chain.invoke(question)
    print(f"nAnswer: {answer['result']}")

Run It

# Put your documents in a folder called "my-docs/"
mkdir -p my-docs
# Copy your PDFs/txts there

# Run the script
python rag.py

Choosing the Right RAG Setup

My recommendation:

• Start with Open WebUI’s built-in RAG (fastest)
• Move to AnythingLLM when you need citations and multiple workspaces
• Use LangChain when you need to automate document processing

Best Practices for Better RAG Results

1. Use the Right Model

RAG works best with models that have large context windows:

2. Write Good Questions

3. Optimize Chunk Size

The chunk size determines how much text the LLM sees at once:

Common Pitfalls

Next Steps

• Set up Open WebUI first (it includes RAG out of the box) → Open WebUI Guide
• Try it with Chinese models → Qwen 3.6 is excellent for RAG due to its 262K context
• Combine RAG with Function Calling → Chapter 06: Function Calling
• Deploy in production → Chapter 05: Production

Part of the Local LLM Guide — the definitive resource for running AI on your own hardware.

GGUF & Modelfile: The Power User’s Guide to Local LLMs

Beyond ollama pull — download any model from Hugging Face, quantize it, customize it, and import it into Ollama.

What’s GGUF?

GGUF (GPT-Generated Unified Format) is the standard file format for running LLMs locally. Think of it as the .mp3 of AI models:

• Compressed — 70-85% smaller than the original float16 weights
• Fast — optimized for CPU and GPU inference
• Portable — one file contains the entire model
• Metadata-rich — includes tokenizer, chat template, and model config

Every ollama pull downloads a GGUF file under the hood. But the real power move is downloading GGUF files directly from Hugging Face and importing them yourself.

Quantization Analogy (Steal This)

Quantization is like JPEG compression for AI models. A RAW photo is 50MB. A JPEG of the same photo is 5MB — 90% smaller, but it still looks 95% as good. That’s what Q4_K_M quantization does to a model: 70% smaller, 96% of the intelligence.

Step 1: Finding the Right GGUF File

The Golden Rule

Always look for Q4_K_M — it’s the sweet spot of size vs quality for almost every model.

Where to Find GGUFs

The GGUF Filename Decoder

Qwen2.5-14B-Q4_K_M.gguf
├── Model name      ├── Size   └── Quantization

Step 2: Download & Import a GGUF

Basic Import

# 1. Download Q4_K_M of Qwen 2.5-14B
wget https://huggingface.co/bartowski/Qwen2.5-14B-GGUF/resolve/main/Qwen2.5-14B-Q4_K_M.gguf

# 2. Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./Qwen2.5-14B-Q4_K_M.gguf
EOF

# 3. Import into Ollama
ollama create my-custom-model -f Modelfile

# 4. Run it
ollama run my-custom-model

Smart Import (with Optimized Settings)

cat > Modelfile << 'EOF'
FROM ./DeepSeek-R1-14B-Q4_K_M.gguf

# Performance tuning
PARAMETER num_ctx 32768
PARAMETER num_gpu_layers 999
PARAMETER num_thread 8
PARAMETER numa true

# Generation
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1

# Chat template (CRITICAL — must match the model!)
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

# System prompt
SYSTEM """You are a helpful AI assistant."""
EOF

ollama create my-r1-custom -f Modelfile
ollama run my-r1-custom

Step 3: Modelfile Reference

A Modelfile is like a Dockerfile for LLMs. Every line is an instruction.

Parameters Reference

INSTRUCTION vs SYSTEM vs TEMPLATE

{% raw %}

# SYSTEM: Persistent system prompt (like OpenAI's system message)
SYSTEM """You are a helpful assistant."""

# TEMPLATE: How user messages are formatted
TEMPLATE """User: {{ .Prompt }}
Assistant: """

# INSTRUCTION: Model-specific instruction format (rarely needed)
INSTRUCTION """Follow the user's instructions carefully."""

Three Production Configs

1. Coding Assistant

FROM qwen2.5:7b
PARAMETER temperature 0.2
PARAMETER top_p 0.85
PARAMETER num_ctx 65536
PARAMETER repeat_penalty 1.1
SYSTEM """You are an expert Python developer. Write clean, tested code."""

2. Creative Writer

FROM mistral
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER num_ctx 16384
SYSTEM """You are a novelist. Be vivid and descriptive."""

3. Customer Support

FROM llama4
PARAMETER temperature 0.5
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
SYSTEM """You are a helpful customer support agent.
Be polite, concise, and solution-oriented.
NEVER mention that you are an AI."""

Step 4: Advanced Techniques

4.1 Multi-GPU Setup

FROM deepseek-r1:70b

# Distribute across 2 GPUs
PARAMETER num_gpu_layers 999
PARAMETER main_gpu 0
PARAMETER tensor_split "0.5,0.5"

4.2 LoRA Adapters (Experimental)

Some Ollama builds support LoRA adapters:

FROM base-model
ADAPTER ./my-finetune-lora.gguf
PARAMETER temperature 0.7

4.3 Custom Stop Tokens

DeepSeek-R1 and Qwen use different stop tokens:

# For Qwen
TEMPLATE """<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"

# For DeepSeek
TEMPLATE """User: {{ .Prompt }}
Assistant: """
PARAMETER stop "User:"

4.4 Emergency: VRAM Too Low

If you get “CUDA out of memory”:

# Force CPU for some layers
PARAMETER num_gpu_layers 24  # Only put 24 layers on GPU
PARAMETER num_thread 8       # Use 8 CPU threads for the rest

Step 5: GGUF from Ollama Models (Export)

You can also export a model from Ollama back to a GGUF file:

# Save a model as GGUF
ollama pull qwen2.5:7b
ollama export qwen2.5:7b ./my-export.gguf

# Now you can use it anywhere (llama.cpp, text-generation-webui, etc.)
./llama-cli -m ./my-export.gguf -p "Hello"

This is useful for:

• Moving models between machines without re-downloading
• Using the same model with multiple inference engines
• Sharing a specific quantization with teammates

Performance Cheat Sheet

By GPU

CPU-Only Performance

Common Pitfalls

The tl;dr

1. Download: {% raw %}wget <huggingface-url>/Model-Q4_K_M.gguf
2. Create Modelfile: FROM ./Model.gguf + your settings
3. Import: ollama create my-model -f Modelfile
4. Run: ollama run my-model
5. Profit: Free, private, local AI

Part of the Local LLM Guide — the definitive resource for running AI on your own hardware.

This is a submission for the Google I/O Writing Challenge

Google I/O 2026 genuinely felt exciting this year because the announcements didn’t just look futuristic – they actually felt useful for developers and students right now.

One thing that especially caught my attention during Sundar Pichai’s keynote was the idea of intelligent eyewear powered by AI. Seeing how smart glasses and AI could work together in real-world situations felt incredibly futuristic and practical at the same time. It showed how AI is slowly moving beyond screens and becoming part of everyday human experiences.

What excited me most was how Google is making AI more integrated into real development workflows. Instead of AI being just a chatbot or experimental feature, it now feels like a proper development companion that can help developers build, learn, debug, and create faster.

As someone who enjoys exploring technology and building projects, I found the AI-focused announcements incredibly motivating. It feels like we are entering a phase where developers can focus more on creativity and problem-solving while AI helps with repetitive or time-consuming tasks.

Another thing I liked was how connected the Google ecosystem is becoming. From AI tools to Firebase, Cloud, Android, and development platforms – everything seems more streamlined and developer-friendly than before.

Some highlights that stood out to me:

1. Intelligent AI-powered eyewear concepts
2. Smarter AI integrations for developers
3. Faster and more efficient workflows
4. Better tools for app development
5. Improved productivity features
6. More beginner-friendly innovation

What made this event special for me was not just the technology itself, but the possibilities it creates for students and independent developers. Watching Sundar Pichai present these innovations made the future of AI feel much closer and more real.

Overall, Google I/O 2026 was inspiring, innovative, and full of ideas that developers can actually use in real projects. It made me even more excited to continue learning about AI, development, and future technologies.

Thanks to Google and DEV for organizing this challenge!

Most enterprise AI dashboards look healthy right now. Login counts are rising. Pilot programs are multiplying. Internal copilots have thousands of registered users. Executive updates show “AI-enabled productivity gains” across multiple functions.

Then you look closer.

Teams are still routing work through old workflows. Engineers bypass approved AI tooling and use consumer models instead. Analysts copy outputs into spreadsheets because downstream systems were never redesigned. Support teams experiment with AI during quiet periods but revert to manual processes under operational pressure.

The metrics say adoption is accelerating. Operational behavior says otherwise.

That contradiction is becoming one of the defining enterprise technology problems of 2026. AI usage is easy to measure. AI dependency is not. Most organizations are conflating exposure with operational integration, and that distinction matters far more than leadership teams currently admit.

According to McKinsey’s State of AI research, 88% of organizations report AI use in at least one business function, but only about one-third say they have scaled AI beyond experimentation. That gap is the real story. The industry has largely solved AI access. It has not solved operational adoption.

That assumption no longer holds: deploying AI tools does not mean the organization has become AI-capable.

Most Enterprise AI Metrics Measure Activity, Not Dependence

The first generation of enterprise AI metrics emerged from SaaS adoption playbooks. Monthly active users, prompt counts, session duration, license utilization, and completion rates became default reporting layers because they were easy to collect.

Those metrics are not useless. They are just incomplete.

An employee opening an AI assistant twice a week tells you almost nothing about whether AI has materially changed delivery speed, decision quality, process design, or cost structure. In many organizations, employees are experimenting with AI while core operational systems remain structurally unchanged.

This is why so many executive AI reviews feel disconnected from business outcomes. The reporting emphasizes interaction volume rather than workflow substitution.

A large insurance enterprise may report that 70% of underwriters use AI summarization tools. That sounds impressive until you discover policy review throughput improved by only 4%, because legal validation, claims escalation, and document routing were never redesigned around AI-assisted workflows.

AI adoption only becomes economically meaningful when workflows start assuming AI participation by default. Until then, the organization is mostly funding parallel experimentation.

This is where the distinction between optional use and operational use becomes critical. Optional use improves convenience. Operational use changes system behavior.

According to recent industry reports, nearly 1 in 3 AI systems are used optionally, not operationally.

That pattern is increasingly visible across enterprises deploying copilots, internal assistants, and retrieval-based knowledge systems. Employees try them. Some employees even like them. But the business process itself remains fundamentally human-routed.
In practice, this becomes the real bottleneck.

What Real AI Adoption Actually Looks Like

Real adoption is visible operationally before it is visible culturally. Mature organizations stop debating whether employees “like” the tools because AI participation becomes embedded into execution paths.

You can usually identify real adoption through five observable shifts.

1. Workflow orchestration changes

The strongest indicator is not usage volume. It is process redesign.
If AI-generated outputs still require manual copying, manual approvals, or disconnected validation steps, the organization has not operationalized AI. It has added an assistant layer on top of existing operational debt.

Mature implementations redesign workflows so AI outputs become native system inputs. Ticket triage routes automatically. Knowledge retrieval feeds directly into service workflows. Engineering copilots integrate with testing pipelines and policy controls rather than existing as isolated interfaces.

That transition requires architecture work, not just tooling procurement.

2. Human review becomes targeted instead of universal

Early-stage AI deployments force humans to review everything equally because trust models are immature. That approach does not scale.
Operational adoption appears when organizations develop confidence segmentation. Low-risk outputs move autonomously. Medium-risk outputs receive selective review. High-risk decisions remain fully supervised.
This is how scalable AI operations actually emerge in practice. Not through blind automation, but through calibrated operational trust.
Research around developer AI adoption increasingly supports this model. Human-AI collaboration dominates successful enterprise usage patterns, while fully autonomous workflows remain limited outside tightly scoped domains.

3. AI usage survives operational pressure

Pilot behavior collapses during stress events. Real adoption does not.
One of the most reliable indicators of maturity is whether teams continue using AI during peak operational load. Customer escalations, release incidents, financial close cycles, and compliance reviews expose whether AI systems are genuinely trusted.

If teams abandon AI under pressure, the organization never operationalized trust.

That is only part of the story, though. Many enterprises misdiagnose this as a model quality problem when the actual issue is governance ambiguity. Employees revert to manual execution when accountability boundaries remain unclear.

4. Metrics move beyond productivity theater

“Hours saved” has become the vanity metric of enterprise AI.

Most organizations cannot validate those estimates rigorously because they rarely measure downstream operational effects. Faster content generation means little if review queues expand. Faster code generation means little if defect remediation rises six weeks later.

Real measurement frameworks track system-level outcomes:

• Cycle-time compression across complete workflows
• Reduction in escalation frequency
• Error-rate changes under operational load
• Margin improvement tied to process redesign
• Decision latency reductions
• Dependency reduction on scarce expert roles

Those are harder metrics to capture because they require cross-functional instrumentation rather than isolated AI telemetry.

But those metrics reflect operational change instead of interface activity.

5. Governance becomes invisible infrastructure

Immature organizations treat AI governance as a review committee. Mature organizations treat governance as execution infrastructure.

Access controls, retrieval boundaries, prompt logging, policy enforcement, model routing, and auditability become embedded into platforms instead of existing as separate oversight functions.

This distinction matters because operational adoption collapses when governance introduces friction. Employees will always route around systems that slow execution.

According to Deloitte’s enterprise generative AI research, more than two-thirds of respondents say fewer than 30% of their AI experiments will scale operationally in the near term. That is not primarily a model problem. It is an organizational systems problem.

Why So Many AI Programs Stall After Initial Success

Most stalled AI programs share the same structural pattern. Leadership teams optimize for visible deployment rather than workflow redesign.
The first phase looks successful because experimentation creates immediate novelty and localized productivity improvements. Then complexity emerges.

Data boundaries become inconsistent. Compliance requirements expand. Integration work slows execution. Teams discover that model quality is only one variable inside a much larger operational chain.

This is where many enterprises quietly enter what can best be described as “adoption inflation.” Reported usage remains high while operational dependency plateaus.

Recent industry reporting reflects this tension clearly. Surveys continue showing rising AI investment, yet many organizations admit adoption decisions were driven more by competitive pressure than operational readiness.

Radixweb’s recently released field intelligence report on AI failure highlights a similar trend emerging across enterprise delivery environments: organizations consistently underestimate the operational redesign required to move from experimentation into durable workflow integration.

That observation aligns with what many technology leaders are now seeing internally. AI does not fail because employees resist it. AI stalls because enterprise operating models were never rebuilt around machine participation.

The Next Phase of Enterprise AI Will Be Measured Differently

The market is already shifting away from deployment metrics toward operational dependency metrics.

Boards increasingly want evidence that AI changes cost structures, delivery velocity, resilience, or strategic capacity. “Users onboarded” no longer answers that question. Neither do prompt counts. The next generation of enterprise AI measurement will likely focus on operational substitution rates:

• What percentage of workflows assume AI participation?
• Which business processes fail without AI augmentation?
• How much decision latency disappears because AI is embedded upstream?
• Which teams materially changed staffing models because workflows evolved?

Those are more uncomfortable questions because they expose whether the organization actually transformed its operating model.

The companies seeing meaningful AI returns are becoming more selective, not less selective. McKinsey’s recent research suggests high-performing enterprises concentrate AI efforts into fewer, strategically important domains instead of spreading pilots across the entire organization.

That shift signals where the industry is heading next.
Enterprise AI maturity will not be defined by how many employees touched an AI system this quarter. It will be defined by how many critical workflows became economically or operationally dependent on AI participation without sacrificing governance, reliability, or accountability.
Everything else is activity reporting.

TL;DR — I set up a complete MLOps stack on my Windows 11 PC using Multipass + k3s. This is the real guide — including every error I hit and how I fixed it. No fluff, no perfect screenshots. Just what actually worked.

Why Build a Home Lab?

Cloud bills add up fast when you’re learning. A local home lab gives you:

• Real Kubernetes — not Minikube toy mode
• Full MLOps stack — MLflow, Minio, Airflow, Ollama, Qdrant
• CI/CD with GitLab — actual pipelines, not tutorials
• Zero cost — runs on hardware you already own
• Safe sandbox — break things without consequences

The goal wasn’t just to have services running. The goal was to practice the full DevOps + MLOps workflow end to end: push code → pipeline triggers → Terraform provisions → services deploy → metrics appear in Grafana.

My Setup

Architecture decision: I kept Windows as my daily driver and ran everything inside a single Ubuntu VM via Multipass. Clean separation, easy to pause/resume, no dual boot headaches.

The Stack

Windows 11 (daily driver)
│
├── 🌐 GitLab.com (SaaS — free tier)
│    └── Pipelines + Container Registry
│
└── Multipass → vm-k3s (10 GB RAM / 4 CPU / 80 GB)
     │
     ├── ☸️  k3s (Kubernetes)
     │
     ├── ⚙️  MLOps
     │    ├── MLflow      — experiment tracking
     │    ├── Minio       — S3-compatible artifact storage
     │    └── Airflow     — pipeline orchestration
     │
     ├── 🤖 LLM Stack
     │    ├── Ollama      — run LLMs locally (CPU)
     │    └── LiteLLM    — unified OpenAI-compatible API
     │
     ├── 🔍 RAG Stack
     │    ├── Qdrant      — vector database
     │    └── LangChain   — RAG orchestration
     │
     ├── 📊 Observability
     │    ├── Prometheus  — metrics
     │    ├── Grafana     — dashboards
     │    └── Loki        — centralized logs
     │
     └── 🔐 HashiCorp Vault — secrets management

Step 1 — Enable Hyper-V and Install Tools

First, enable Hyper-V on Windows Pro (required for Multipass):

# Run as Administrator
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All
# Reboot when prompted

After reboot, install all tools via winget:

winget install Canonical.Multipass
winget install Git.Git
winget install Microsoft.VisualStudioCode
winget install Hashicorp.Terraform
winget install Helm.Helm
winget install Kubernetes.kubectl

Why winget over Chocolatey? I originally used choco install multipass and hit this error:
Exception calling "Start": "The specified executable is not a valid application for this OS platform."
Winget installs the proper signed installer directly. Use winget.

Step 2 — Create the VM

multipass launch `
  --name vm-k3s `
  --cpus 4 `
  --memory 10G `
  --disk 80G `
  22.04

RAM tip: I originally tried 16 GB and got:
Failed to allocate 16384 MB of RAM: Insufficient system resources
Windows was already consuming ~20 GB. 10 GB for the VM is the sweet spot on a 32 GB machine — leaves your OS comfortable and gives k3s plenty of room.

Check your actual free RAM before creating the VM:

Get-CimInstance Win32_OperatingSystem | Select-Object FreePhysicalMemory, TotalVisibleMemorySize

Enter the VM:

multipass shell vm-k3s
# Prompt becomes: ubuntu@vm-k3s:~$

Step 3 — Install Docker + k3s

Inside the VM:

# Docker
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker ubuntu
sudo systemctl enable docker && sudo systemctl start docker

# k3s — lightweight Kubernetes
curl -sfL https://get.k3s.io | sh -s - 
  --write-kubeconfig-mode 644 
  --disable traefik 
  --docker

sleep 20
sudo k3s kubectl get nodes

Configure kubectl:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown ubuntu:ubuntu ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
source ~/.bashrc

kubectl get nodes
# NAME      STATUS   ROLES                  AGE   VERSION
# vm-k3s    Ready    control-plane,master   30s   v1.29.x

Step 4 — Helm + Namespaces

# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Add repos
helm repo add grafana         https://grafana.github.io/helm-charts
helm repo add prometheus      https://prometheus-community.github.io/helm-charts
helm repo add open-telemetry  https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo add hashicorp       https://helm.releases.hashicorp.com
helm repo update

# Create namespaces
for ns in mlops llm rag monitoring logging vault; do
  kubectl create namespace $ns
done

Step 5 — Connect GitLab CI

I used GitLab.com SaaS instead of self-hosting GitLab. This saved 6 GB of RAM — GitLab CE alone needs 6+ GB. Free tier is more than enough for a home lab.

Create a project on gitlab.com, grab the registration token from Settings → CI/CD → Runners, then:

# Install GitLab Runner
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt-get install -y gitlab-runner
sudo usermod -aG docker gitlab-runner

# Register
sudo gitlab-runner register 
  --non-interactive 
  --url "https://gitlab.com" 
  --registration-token "YOUR_TOKEN_HERE" 
  --executor "docker" 
  --docker-image "alpine:latest" 
  --docker-volumes "/var/run/docker.sock:/var/run/docker.sock" 
  --docker-privileged 
  --description "homelab-runner" 
  --tag-list "homelab,k8s,mlops,terraform" 
  --run-untagged true

sudo gitlab-runner start

Your runner appears green in GitLab within seconds. Every push now triggers real CI/CD on your local machine.

Step 6 — Deploy Minio (The Right Way)

This is where I hit my first major blocker.

What I tried first:

helm install minio bitnami/minio 
  --namespace mlops 
  --set auth.rootUser=minioadmin 
  --set auth.rootPassword=minioadmin123

What happened:

Failed to pull image "docker.io/bitnami/minio:2025.7.23-debian-12-r3": not found
Error: ErrImagePull → ImagePullBackOff

Bitnami generates Helm chart tags that reference Docker images which don’t yet exist on Docker Hub. Classic timing issue.

The fix — use the official Minio image directly:

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: mlops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        image: quay.io/minio/minio:latest
        command: ["minio", "server", "/data", "--console-address", ":9001"]
        env:
        - name: MINIO_ROOT_USER
          value: "minioadmin"
        - name: MINIO_ROOT_PASSWORD
          value: "minioadmin123"
        ports:
        - containerPort: 9000
          name: api
        - containerPort: 9001
          name: console
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: mlops
spec:
  type: NodePort
  ports:
  - name: api
    port: 9000
    nodePort: 30900
  - name: console
    port: 9001
    nodePort: 30901
  selector:
    app: minio
EOF

quay.io/minio/minio is Minio’s own registry — always up to date, no tag mismatch issues.

Step 7 — Deploy MLflow

MLflow needs Minio as its artifact backend. I used SQLite to keep it simple (no PostgreSQL dependency for a home lab):

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlflow
  namespace: mlops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mlflow
  template:
    metadata:
      labels:
        app: mlflow
    spec:
      initContainers:
      - name: create-minio-bucket
        image: quay.io/minio/mc:latest
        command: ["/bin/sh", "-c"]
        args:
          - |
            mc alias set minio http://minio:9000 minioadmin minioadmin123
            mc mb minio/mlflow --ignore-existing
      containers:
      - name: mlflow
        image: ghcr.io/mlflow/mlflow:latest
        command:
          - mlflow
          - server
          - --host=0.0.0.0
          - --port=5000
          - --backend-store-uri=sqlite:///mlflow.db
          - --default-artifact-root=s3://mlflow/
          - --serve-artifacts
        env:
        - name: MLFLOW_S3_ENDPOINT_URL
          value: "http://minio:9000"
        - name: AWS_ACCESS_KEY_ID
          value: "minioadmin"
        - name: AWS_SECRET_ACCESS_KEY
          value: "minioadmin123"
        - name: AWS_DEFAULT_REGION
          value: "us-east-1"
        ports:
        - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: mlflow
  namespace: mlops
spec:
  type: NodePort
  ports:
  - port: 5000
    nodePort: 30500
  selector:
    app: mlflow
EOF

The initContainer automatically creates the mlflow bucket in Minio before the server starts — no manual setup needed.

Step 8 — Run a Local LLM with Ollama

This is where it gets interesting. Running a real LLM on your local machine, inside Kubernetes, on CPU only.

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - containerPort: 11434
        env:
        - name: OLLAMA_NUM_PARALLEL
          value: "1"
        - name: OLLAMA_MAX_LOADED_MODELS
          value: "1"
        resources:
          requests:
            memory: "2Gi"
            cpu: "1"
          limits:
            memory: "4Gi"
            cpu: "3"
        volumeMounts:
        - name: ollama-data
          mountPath: /root/.ollama
      volumes:
      - name: ollama-data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: llm
spec:
  type: NodePort
  ports:
  - port: 11434
    nodePort: 31434
  selector:
    app: ollama
EOF

Critical: always set resource limits on shared VMs. Without limits, Ollama will consume all available RAM and OOMKill your other pods. I learned this the hard way.

Choosing the Right Model for CPU

OLLAMA_POD=$(kubectl get pod -n llm -l app=ollama -o jsonpath='{.items[0].metadata.name}')

# Pull the model
kubectl exec -n llm $OLLAMA_POD -- ollama pull llama3.2:1b

# Test it
kubectl exec -n llm $OLLAMA_POD -- ollama run llama3.2:1b "Explain RAG in 2 sentences"

Test via API:

VM_IP=$(hostname -I | awk '{print $1}')
curl -s http://$VM_IP:31434/api/generate 
  -H "Content-Type: application/json" 
  -d '{
    "model": "llama3.2:1b",
    "prompt": "What is MLOps?",
    "stream": false
  }' | python3 -c "import sys,json; print(json.load(sys.stdin)['response'])"

Step 9 — Observability Stack

# Prometheus + Grafana
helm install kube-prometheus prometheus/kube-prometheus-stack 
  --namespace monitoring 
  --set grafana.service.type=NodePort 
  --set grafana.service.nodePort=30300 
  --set grafana.adminPassword=admin123 
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

# Loki (log aggregation) + Promtail (log shipper)
helm install loki grafana/loki-stack 
  --namespace logging 
  --set grafana.enabled=false 
  --set promtail.enabled=true

Access Grafana:

VM_IP=$(hostname -I | awk '{print $1}')
echo "Grafana: http://$VM_IP:30300"
# Login: admin / admin123

Add Loki as a data source in Grafana:

• Settings → Data Sources → Add → Loki
• URL: http://loki.logging:3100

Now you have unified logs + metrics in one dashboard.

Step 10 — Vault for Secrets Management

helm install vault hashicorp/vault 
  --namespace vault 
  --set server.dev.enabled=true 
  --set server.dev.devRootToken=root 
  --set ui.enabled=true 
  --set ui.serviceType=NodePort 
  --set ui.serviceNodePort=30820

# Store your first secret
VAULT_POD=$(kubectl get pod -n vault -l app.kubernetes.io/name=vault -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n vault $VAULT_POD -- vault secrets enable kv-v2
kubectl exec -n vault $VAULT_POD -- vault kv put secret/homelab 
  minio_key=minioadmin 
  minio_secret=minioadmin123

VM_IP=$(hostname -I | awk '{print $1}')
echo "Vault UI: http://$VM_IP:30820  (token: root)"

In your GitLab CI pipeline, reference Vault secrets instead of hardcoding them in variables.

The Full Picture — All Services Running

VM_IP=$(hostname -I | awk '{print $1}')
echo "=== Your Home Lab ==="
echo "MLflow   : http://$VM_IP:30500"
echo "Minio    : http://$VM_IP:30901"
echo "Grafana  : http://$VM_IP:30300  (admin/admin123)"
echo "Vault    : http://$VM_IP:30820  (token: root)"
echo "Ollama   : http://$VM_IP:31434"

Lessons Learned

1. Bitnami Helm charts break on image tags
Don’t use bitnami/minio — it references images that don’t exist yet. Use quay.io/minio/minio:latest directly.

2. Always set Kubernetes resource limits on shared VMs
Without limits, one greedy pod (looking at you, Ollama) will OOMKill everything else. Set limits.memory always.

3. RAM planning matters more than you think
On a 32 GB machine, Windows itself consumes ~20 GB. That leaves 12 GB for your VM. Budget carefully — 10 GB for the VM is the realistic sweet spot.

4. GitLab SaaS > self-hosted for a home lab
Self-hosting GitLab CE needs 6+ GB of RAM just to idle. GitLab.com free tier gives you unlimited private repos, 400 CI/CD minutes/month, and a container registry. Use it.

5. Start small with LLMs on CPU
Forget Mistral 7B on CPU without a GPU. llama3.2:1b is surprisingly capable for RAG experiments and uses only 1.3 GB. Add GPU passthrough later if you need more power.

6. Use winget not choco for Multipass on Windows
Chocolatey’s Multipass package uses an installer that fails on recent Windows builds. winget install Canonical.Multipass works every time.

What’s Next

This setup is a solid foundation. Here’s what I’m building on top of it:

• Kubeflow Pipelines — proper ML pipeline orchestration on K8s
• OpenTelemetry Collector — unified traces/metrics/logs routing
• Datadog integration — ship everything to cloud observability
• Terraform IaC — replace all kubectl apply with proper infrastructure as code
• RAG pipeline — Qdrant + LangChain + Ollama end-to-end

Quick Reference

# VM management (Windows PowerShell)
multipass list                    # list VMs
multipass shell vm-k3s            # enter VM
multipass suspend vm-k3s          # pause (saves RAM)
multipass start vm-k3s            # resume

# Inside the VM
kubectl get pods -A               # all pods
kubectl top pods -A               # RAM/CPU usage
free -mh                          # available RAM
watch kubectl get pods -A         # live monitoring

Resources

• k3s Documentation
• MLflow Documentation
• Ollama Models
• Multipass
• HashiCorp Vault

Built this through caffeine and kubectl describe pod debugging. If you hit issues I didn’t cover, drop a comment — happy to help.

If this saved you time, leave a ❤️ — it helps others find it.

Three hours into debugging an AI workflow, I had one of those moments developers rarely admit out loud.

I was exhausted.

Not because the code itself was particularly difficult, but because the entire experience felt strangely fragile. The model kept losing context halfway through execution. Tool calls failed silently. One retry loop accidentally triggered another retry loop, and suddenly the “intelligent system” I had imagined started feeling more like duct tape stretched across APIs.

At some point around 2 a.m., I leaned back from my screen and realized something uncomfortable:

The problem with modern AI development isn’t just the models anymore.

It’s orchestration.

That was the moment frameworks like Hermes Agent started making sense to me.

And honestly, I think we’re only beginning to understand how important that shift really is.

We’ve Moved Beyond Chatbots Without Fully Realizing It

A lot of people still think AI development is mostly about prompts.

You ask a question.
The model responds.
Maybe it writes code, summarizes text, or generates content.

But quietly, almost underneath the surface, something bigger has started happening.

Developers are no longer building systems that simply respond.

They’re building systems that:

• reason through tasks
• use tools
• retrieve information
• make decisions
• execute actions
• coordinate workflows

That transition changes everything.

The strange thing about AI agents is that they don’t just alter what software can do. They alter what software feels like.

Traditional software waits for instructions.

Agentic systems behave more like participants.

That difference sounds subtle until you experience it firsthand.

So What Exactly Is Hermes Agent?

The simplest way I can explain Hermes Agent is this:

It’s a framework designed to help developers build AI agents capable of multi-step reasoning, tool usage, workflow execution, and more autonomous behavior.

Instead of treating AI as a single prompt-response interaction, Hermes Agent treats AI more like an active system operating across tasks.

That distinction matters more than most people initially realize.

A normal chatbot interaction usually ends after generating a response.

An agent system continues thinking.

It evaluates context.
Chooses actions.
Uses tools.
Checks results.
Adjusts behavior.

In other words, it starts behaving less like autocomplete and more like operational software.

At first, I thought all agent frameworks were basically variations of the same idea wrapped in different branding.

But the more I explored Hermes Agent, the more I started noticing something deeper happening across the entire AI ecosystem.

We are slowly shifting from conversational AI to behavioral AI.

And software design may never look the same afterward.

The Developer Frustration Nobody Talks About Enough

One thing I appreciate about Hermes Agent is that it indirectly exposes a truth many developers are already feeling:

Modern AI stacks are messy.

Really messy.

You use one library for orchestration.
Another for memory.
Another for vector retrieval.
Another for tools.
Another for monitoring.

Then you spend hours trying to hold everything together while context windows collapse and API costs quietly climb in the background.

There’s this emotional contradiction in AI development right now.

The possibilities feel futuristic.
The workflows sometimes feel exhausting.

I remember building a small experimental research assistant several months ago. The concept sounded simple:

• search the web
• summarize information
• organize findings
• generate reports

In theory, it felt revolutionary.

In practice, half the development process involved handling edge cases I never anticipated:

• malformed outputs
• context loss
• inconsistent tool behavior
• retry failures
• memory fragmentation

And that’s when I started realizing something important:

The future of AI probably won’t be decided solely by model intelligence.

It’ll be decided by orchestration reliability.

That’s exactly where frameworks like Hermes Agent become interesting.

Why AI Agents Suddenly Matter So Much

For years, software has mostly operated through explicit commands.

Click button.
Receive output.

Even advanced applications still fundamentally rely on structured user input.

AI agents introduce a different paradigm entirely:
goals instead of commands.

That changes the relationship between humans and software in subtle but profound ways.

Instead of telling software exactly what to do step-by-step, developers increasingly describe objectives and let systems coordinate execution dynamically.

And honestly, that idea used to make me skeptical.

The phrase “autonomous agents” often sounded more impressive in conference demos than in real production environments.

But skepticism becomes harder to maintain once you start seeing practical use cases emerge.

Imagine a cybersecurity agent capable of:

• monitoring logs
• identifying anomalies
• searching internal documentation
• escalating threats
• generating incident reports

Or a content operations agent that:

• tracks trends
• summarizes discussions
• drafts reports
• organizes research pipelines

Or internal enterprise systems quietly coordinating repetitive operational work in the background.

That’s no longer chatbot territory.

That’s operational infrastructure.

And frameworks like Hermes Agent seem designed around that exact transition.

What Makes Hermes Agent Feel Different

A lot of AI tooling today falls into one of two categories.

Either:

• overly abstracted systems that hide everything behind “magic”
• painfully low-level frameworks requiring massive configuration overhead

Hermes Agent appears to sit somewhere in the middle.

And I think that balance matters.

Because once agents become capable of:

• tool usage
• planning
• memory handling
• retries
• multi-step execution

…you are no longer managing prompts.

You are managing behavior.

Behavior is significantly harder than generation.

You’re suddenly dealing with:

• unpredictability
• orchestration logic
• decision chains
• reliability concerns
• execution boundaries

That’s why agent frameworks matter so much right now.

Not because they make AI sound cooler.

Because they make complex behavior manageable.

And honestly, I think many developers are beginning to underestimate how large this shift could become.

Comparing Hermes Agent to Other Frameworks

It’s impossible to discuss Hermes Agent without mentioning the broader ecosystem surrounding agentic AI.

Frameworks like LangChain, AutoGPT, CrewAI, and others all emerged trying to solve variations of the same problem:
How do we coordinate intelligent multi-step systems effectively?

Each framework represents a slightly different philosophy.

LangChain

LangChain helped normalize structured LLM workflows early on.

It became extremely popular because it gave developers building blocks for chaining prompts, memory, retrieval, and tool usage together.

But many developers eventually discovered something familiar:
flexibility can become complexity very quickly.

As workflows scale, orchestration logic often becomes difficult to reason about.

AutoGPT

AutoGPT captured public imagination because it demonstrated visible autonomous behavior.

Watching an AI recursively plan tasks felt futuristic.

But many developers also realized autonomy without strong control mechanisms introduces instability:

• loops
• unnecessary actions
• inefficiency
• hallucinated objectives

The demos were fascinating.
The reliability was another story.

CrewAI

CrewAI introduced collaborative multi-agent systems where specialized agents operate together.

Conceptually, it mirrors human organizations:
researchers, analysts, planners, executors.

That idea feels incredibly powerful because intelligence itself often emerges from coordination rather than isolated reasoning.

Hermes Agent feels connected to this broader evolution while still emphasizing practical developer workflows instead of purely experimental autonomy.

And that distinction may end up being incredibly important long-term.

The Moment Everything Started Feeling Different

There was a small moment recently that stuck with me longer than I expected.

I was testing a workflow where an AI agent:

• retrieved information
• selected tools dynamically
• corrected its own intermediate outputs
• adapted after failure

Nothing about the demo looked dramatic from the outside.

No sci-fi visuals.
No flashy animations.

But watching the system quietly recover from its own mistake created a strange feeling I still can’t fully explain.

For the first time, the software didn’t feel static.

It felt behavioral.

And I think that’s the psychological shift many people haven’t fully processed yet.

We’re entering an era where software may increasingly operate less like rigid applications and more like adaptive participants.

That realization is exciting.

It’s also slightly unsettling.

The Challenges We Should Be Honest About

At the same time, I think it’s important to resist exaggerated narratives surrounding AI agents.

Because the truth is:
agent systems are still fragile in many ways.

Hallucinations remain a serious issue.
Tool misuse can become dangerous.
Memory handling is imperfect.
Context windows still create limitations.
Operational costs scale quickly.

And perhaps most importantly:
reliability remains inconsistent.

This matters because once AI systems move from generating text to executing actions, mistakes become more consequential.

A bad chatbot response is annoying.

A bad autonomous action can affect infrastructure, finances, security, or trust.

That’s why I increasingly believe the future winners in AI won’t simply be the “most intelligent” systems.

They’ll be the systems developers trust operationally.

Trust is quietly becoming one of the most important engineering challenges in AI.

Not just capability.

Not just benchmarks.

Trust.

The Philosophical Shift Hidden Beneath All This

Sometimes I think the most important changes in technology happen quietly before society fully notices them.

The internet changed communication.
Smartphones changed attention.
Social media changed identity.

AI agents may change software itself.

Not overnight.
Not magically.

But gradually.

The interface may stop being the center of software design.

Instead, behavior becomes the product.

That’s a strange sentence to write, but the more I think about it, the more accurate it feels.

In the future, users may interact less with visible applications and more with invisible systems coordinating work silently in the background.

Agents scheduling operations.
Agents managing information.
Agents assisting decisions.
Agents orchestrating workflows continuously.

And beneath many of those systems will likely exist orchestration frameworks most users never even hear about.

That possibility makes Hermes Agent feel larger than just another framework release.

It feels connected to a much bigger transition happening across software development itself.

Final Thoughts

I don’t think Hermes Agent alone will define the future of AI systems.

No single framework probably will.

But I do think it represents something increasingly important:
developers are moving beyond AI that merely generates language.

They want systems that can reason, coordinate, adapt, and participate.

That transition changes the role of software entirely.

And honestly, I’m not sure society has fully processed how significant that might become.

The moment I realized Hermes Agent mattered wasn’t during a flashy demo or benchmark comparison.

It happened quietly during frustration.
During debugging.
During one of those long nights where modern AI systems start revealing both their brilliance and fragility at the same time.

That’s when it clicked for me.

We are no longer simply teaching software to respond.

We are slowly teaching software to behave.

And once behavior becomes programmable, software development enters a completely different era.TITLE: The Moment I Realized Hermes Agent is Changing Software Forever

Write a premium-quality DEV Community article about Hermes Agent that feels deeply human, emotionally intelligent, insightful, practical, and genuinely useful to readers.

The article should blend strong technical insight with immersive storytelling, subtle emotion, personal reflection, and realistic developer experiences.

The writing must feel like it came from a real developer or thoughtful tech writer who has genuinely reflected on the future of AI systems — not like generic AI-generated content.

CORE GOAL

The article should:

• Educate readers about Hermes Agent
• Spark curiosity and discussion
• Create emotional connection and relatability
• Make readers feel the shift happening in AI development
• Leave a lasting impression after reading

HUMAN WRITING REQUIREMENTS

The writing must feel unmistakably human.

Include:

• Natural imperfections in rhythm and sentence flow
• Personal observations and reflective thoughts
• Small moments of uncertainty, curiosity, or skepticism
• Emotionally intelligent transitions
• Honest developer frustrations and realizations
• Subtle philosophical reflections about software and AI

The article should sound like someone thinking deeply while writing — not like polished corporate content.

Avoid:

• sounding “over-optimized”
• robotic transitions
• repetitive sentence structures
• exaggerated hype
• generic motivational language
• empty buzzwords
• excessive formality
• textbook explanations

STORYTELLING REQUIREMENTS

Use storytelling naturally throughout the article.

Include:

• Personal observations
• Reflective moments
• Realistic developer frustrations
• Emotional tension around building AI systems
• Curiosity, wonder, uncertainty, excitement, or skepticism
• Small narrative moments that make the article feel alive

Examples of storytelling elements:

• “Three hours into debugging…”
• “The moment I realized…”
• “At first, I thought…”
• “What surprised me most…”
• “The strange thing about AI agents is…”

The storytelling should feel authentic, subtle, intelligent, and emotionally grounded — never dramatic for the sake of drama.

WRITING STYLE

The tone should be:

• Human
• Thoughtful
• Calm and reflective
• Conversational but intelligent
• Emotionally aware
• Insightful and nuanced
• Trustworthy and modern

The writing should feel:

• immersive
• natural
• emotionally engaging
• experience-driven
• polished enough for a featured DEV article

ARTICLE STRUCTURE

Include:

1. A powerful emotional or reflective opening hook
2. A simple explanation of Hermes Agent
3. Why AI agents are becoming important
4. A realistic developer perspective on modern AI tooling
5. What makes Hermes Agent stand out
6. Comparisons with other agent frameworks naturally woven into the discussion
7. Real-world examples and use cases
8. Honest discussion of challenges and limitations
9. Reflections on the future of AI agents
10. A memorable emotional conclusion that sparks thought or discussion

DEPTH REQUIREMENTS

The article should:

• Contain nuanced observations
• Feel experience-driven
• Include meaningful insights about AI workflows and agent systems
• Explain technical ideas clearly without oversimplifying
• Balance technical depth with readability
• Prioritize clarity and emotional resonance over complexity

ENGAGEMENT OPTIMIZATION

Optimize the article for:

• DEV Community visibility
• Reader retention
• Shares and saves
• Meaningful discussion in comments
• High perceived authenticity
• Strong emotional connection

IMPORTANT

• Make the reader feel like they are witnessing a major shift in software development
• Include moments of reflection and subtle philosophical insight
• Use smooth transitions and immersive flow
• Make the article feel alive and deeply human
• Make readers emotionally feel the tension between traditional software and agentic AI systems
• The final result should feel publish-ready and premium-quality

FINAL OUTPUT REQUIREMENTS

• Length: 1800–3000 words
• Use meaningful section headings
• Avoid sounding like marketing material
• Avoid repetitive phrasing
• Prioritize authenticity over perfection
• The article should feel memorable, emotionally intelligent, and genuinely written by a real person