Mert

The digital landscape is witnessing a significant shift in how we perceive creativity and collaboration, propelled largely by advancements in artificial intelligence (AI). One of the most promising platforms to emerge in this arena is Collab-Public: Collaborator, a space designed to foster creativity through the use of AI agents. This platform not only provides tools for enhanced collaborative effort but also exemplifies the evolutionary trajectory of digital collaboration, marking a critical juncture in the intersection between technology and creativity.

As we stand at this juncture, it is crucial to understand the historical context that has paved the way for innovations like Collab-Public. From the early days of simple text editing to sophisticated cloud-based collaborative tools, the evolution of collaboration technology has been driven by the need for efficiency, accessibility, and innovation. In this article, we will explore the features of Collab-Public, examine its implications for creators and teams, and look towards the future of collaborative AI creation.

The Evolution of Collaborative Platforms

Early Tools: From Isolation to Collaboration

The concept of collaboration predates the digital era, with communities gathering for collective problem-solving. However, the advent of personal computing in the 1980s marked the beginning of a new era. Early tools like word processors often isolated users, focusing more on individual productivity than collaboration. It was not until the late 1990s and early 2000s that the first collaborative platforms emerged, enabling multiple users to work on documents simultaneously.

Tools such as Microsoft Word’s track changes feature allowed for basic collaboration, but they were limited in scope. With the rise of the internet, the need for real-time collaboration became apparent, leading to the development of cloud-based platforms like Google Docs. These platforms revolutionized how teams could work together, regardless of their physical location.

The Rise of AI and Its Impact on Collaboration

As technology advanced, AI began to infiltrate these collaborative platforms, enhancing their capabilities. AI agents could now assist users in real-time, offering suggestions and automating repetitive tasks. The idea of a collaborative environment powered by AI became increasingly appealing, as it promised not only efficiency but also a reduction in the cognitive load placed on human users.

Platforms began integrating AI features such as intelligent content generation, predictive text, and data-driven insights. By employing machine learning algorithms, these tools could learn user preferences and facilitate a more tailored collaborative experience.

Introducing Collab-Public: Collaborator

Launched against this backdrop of rapid technological advancement, Collab-Public represents a significant leap forward in collaborative creation. With its smart integration of AI agents, the platform allows users to harness the capabilities of AI in ways that were previously unimaginable. Users can collaborate not only among themselves but also with AI, creating an environment where human creativity is amplified by artificial intelligence.

Collab-Public stands out for its user-friendly interface and robust set of tools designed for various creative endeavors, from writing and graphic design to software development. The platform leverages AI agents that can analyze user input, provide suggestions, and even generate content autonomously.

Key Features of Collab-Public

Interactive AI Agents

At the heart of Collab-Public are its interactive AI agents, which can assist users in diverse tasks. These agents function as collaborative partners, helping to brainstorm ideas, generate content, and refine outputs. For example, a writer can collaborate with an AI agent that suggests plot points or provides stylistic feedback, thereby enriching the creative process.

“AI agents are not just assistants; they are co-creators that can elevate the quality of output while reducing the time and effort required.”

Versatile Collaboration Spaces

Collab-Public offers a variety of collaboration spaces tailored to specific needs, such as brainstorming rooms, design workshops, and coding labs. Each space comes equipped with tools suited to its purpose, ensuring that the collaborative process is as effective as possible.

A designer might use a specific space that integrates design software and AI capabilities to create visuals collaboratively, while software developers might utilize another space with programming tools that allow real-time coding and debugging.

Real-Time Feedback and Learning

One of the most compelling features of Collab-Public is its ability to provide real-time feedback through AI analysis. Users can receive instant critiques and suggestions on their work, making the iterative process of creation more dynamic and fluid.

For instance, a team developing a marketing strategy can instantly evaluate the effectiveness of their messaging through AI tools that measure potential engagement.

Integration with Other Tools

Collab-Public recognizes that no single platform can fulfill all creative needs. Therefore, it offers seamless integration with other popular tools and platforms, such as design software and project management applications. This interoperability enhances the user experience by allowing users to switch between tools without losing their workflow.

Accessibility and Inclusivity

With an emphasis on accessibility, Collab-Public is designed to accommodate users of varying skill levels and backgrounds. The platform includes tutorials and resources to help newcomers become proficient in utilizing AI effectively. Moreover, the AI agents can also adapt their responses based on the user’s expertise, ensuring that everyone can engage meaningfully with the technology.

Use Cases and Practical Examples

Creative Writing

Consider a novelist using Collab-Public to draft their next book. The author can collaborate with an AI agent that helps outline chapters, suggests character development arcs, and even aids in generating dialogue. The iterative feedback from the AI agent can guide the author in refining their prose while maintaining their unique voice.

Graphic Design

A graphic designer collaborating on a marketing project can work in real-time with an AI agent that proposes design variations based on current trends and aesthetics. The platform could automatically analyze successful campaigns in similar markets and suggest visuals that resonate with potential audiences.

Software Development

In a coding team, developers can leverage Collab-Public to collaborate on coding projects with AI agents that assist in debugging, suggesting code optimizations, and even generating boilerplate code. This allows developers to focus on more complex problem-solving and innovation.

Educational Environments

Educational institutions can implement Collab-Public as a collaborative learning platform where students work alongside AI agents on various projects. For instance, students could co-create presentations or research papers, receiving insights and suggestions from AI agents as they progress.

Remote Teams

With remote work becoming a norm, Collab-Public serves as an effective tool for geographically dispersed teams. The ability to collaborate in real-time with AI agents enhances communication and productivity, fostering creativity despite physical barriers.

Looking Ahead: The Future of Collaborative AI

As we delve into the future of platforms like Collab-Public, several trends and predictions emerge.

Enhanced Personalization

One potential direction for the collaboration space is the continued personalization of AI agents. As machine learning algorithms improve, these agents will become more adept at understanding individual users’ preferences and working styles, thereby creating highly tailored collaborative experiences.

Greater Interoperability

We may also see an increase in the interoperability of collaborative platforms. As organizations utilize multiple tools, seamless integration will become essential. Future iterations of Collab-Public could pave the way for unified ecosystems where information and workflows are fluid across different applications.

Ethical Considerations

As AI becomes more integrated into creative processes, ethical considerations will come to the forefront. Questions surrounding authorship, data usage, and the potential for biases in AI-generated content will need to be addressed. The development of guidelines and standards will be critical to mitigate these challenges.

Expansion of AI Capabilities

The capabilities of AI within collaborative environments will continue to expand. Future AI agents may excel in understanding context, generating content with greater nuance, and even collaborating on emotional intelligence levels, leading to deeper connections between human and machine creativity.

Conclusion

Collab-Public: Collaborator is not merely a technological tool; it represents an evolution in the way we conceptualize collaboration and creativity. By harnessing the power of AI, the platform is setting a new standard for collaborative efforts across various domains.

As we navigate this new frontier, understanding the historical context and the trajectory of collaborative tools will be crucial in harnessing the full potential of platforms like Collab-Public. The future of collaborative creation is one where AI and humans work side by side, each enhancing the other’s capabilities, and together, they will push the boundaries of innovation and creativity.

Your trading bot needs to lend idle USDC on Aave, earn stSOL rewards on Kamino, and maybe grab some Hyperliquid perp exposure — but integrating 14 different DeFi protocols means 14 different APIs, authentication schemes, and transaction formats. What if there was a better way?

Building multi-protocol DeFi strategies shouldn’t require a PhD in protocol-specific SDKs. Every protocol has its own API design, error handling, and transaction building process. Jupiter wants one format, Aave expects another, Kamino has its own thing entirely. Your code becomes a sprawling mess of protocol adapters, and adding a new venue means weeks of integration work.

One API for All DeFi Protocols

WAIaaS provides a unified REST API across 14 DeFi protocols on both EVM and Solana. Instead of learning each protocol’s quirks, you make standard HTTP calls with consistent request/response formats. Want to lend on Aave? Same API pattern as staking on Lido or swapping on Jupiter.

The architecture is built around “action providers” — each protocol gets wrapped in a standard interface that handles authentication, transaction building, and error formatting. Your application code stays clean and protocol-agnostic.

Here’s what lending USDC on Aave V3 looks like:

curl -X POST http://127.0.0.1:3100/v1/actions/aave-v3/supply 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "asset": "0xA0b86a33E6441b55434DD5031Ee5CC8a0b080eb6",
    "amount": "1000000000",
    "onBehalfOf": "0x742d35cc6bf8cf8d3d7b87b4e0b3c3c7a8b9e2e1"
  }'

The same pattern works for Kamino lending on Solana:

curl -X POST http://127.0.0.1:3100/v1/actions/kamino/lend 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "reserve": "USDC_RESERVE_ADDRESS",
    "amount": "1000000"
  }'

Behind the scenes, WAIaaS handles the protocol-specific transaction building, gas estimation, and execution. You get a consistent transaction ID back and can poll for status using the same endpoints regardless of which protocol you’re using.

Multi-Protocol Yield Strategy Example

Let’s build a simple yield optimization bot that:

1. Checks current positions across protocols
2. Moves funds to the highest-yield opportunity
3. Maintains target allocation percentages

First, check your current DeFi positions across all protocols:

curl http://127.0.0.1:3100/v1/wallet/defi-positions 
  -H "Authorization: Bearer wai_sess_<token>"

This returns a unified view of lending positions, liquid staking balances, and perpetual futures across Aave, Kamino, Lido, Jito, Hyperliquid, and more. No need to query each protocol separately.

Now implement the rebalancing logic. If Kamino is offering better USDC yields than Aave, withdraw from Aave and deposit to Kamino:

# Withdraw from Aave
curl -X POST http://127.0.0.1:3100/v1/actions/aave-v3/withdraw 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "asset": "0xA0b86a33E6441b55434DD5031Ee5CC8a0b080eb6",
    "amount": "500000000"
  }'

# Deposit to Kamino
curl -X POST http://127.0.0.1:3100/v1/actions/kamino/lend 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "reserve": "USDC_RESERVE_ADDRESS", 
    "amount": "500000"
  }'

Both transactions use the same authentication, error handling, and status polling. Your bot doesn’t need to understand Aave’s lending pool contracts or Kamino’s instruction formats.

Cross-Chain Arbitrage Made Simple

WAIaaS includes cross-chain bridging via LI.FI and Across protocols, so you can easily move funds between chains when opportunities arise. If Solana DeFi is offering better yields than Ethereum, bridge your assets automatically:

# Bridge USDC from Ethereum to Solana via LI.FI
curl -X POST http://127.0.0.1:3100/v1/actions/lifi/bridge 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "fromChain": "ethereum",
    "toChain": "solana", 
    "fromToken": "0xA0b86a33E6441b55434DD5031Ee5CC8a0b080eb6",
    "toToken": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
    "amount": "1000000000"
  }'

Once the bridge completes, deploy the funds on Solana DeFi:

# Lend on Kamino
curl -X POST http://127.0.0.1:3100/v1/actions/kamino/lend 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "reserve": "USDC_RESERVE_ADDRESS",
    "amount": "1000000"
  }'

The same API handles Ethereum DeFi, Solana DeFi, and cross-chain bridging. Your arbitrage bot becomes chain-agnostic.

Advanced Features for DeFi Developers

Policy-Based Risk Management

Set spending limits, allowed tokens, and maximum leverage automatically:

curl -X POST http://127.0.0.1:3100/v1/policies 
  -H "Content-Type: application/json" 
  -H "X-Master-Password: <password>" 
  -d '{
    "walletId": "<wallet-uuid>",
    "type": "LENDING_LTV_LIMIT",
    "rules": {"maxLtv": 0.7}
  }'

This prevents your lending positions from exceeding 70% LTV across all protocols, automatically.

Transaction Simulation

Test strategies without spending gas:

curl -X POST http://127.0.0.1:3100/v1/actions/aave-v3/supply 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "asset": "0xA0b86a33E6441b55434DD5031Ee5CC8a0b080eb6",
    "amount": "1000000000",
    "dryRun": true
  }'

Get gas estimates, success probability, and return calculations before executing real transactions.

Health Factor Monitoring

Track liquidation risk across all lending positions:

curl http://127.0.0.1:3100/v1/wallet/health-factor 
  -H "Authorization: Bearer wai_sess_<token>"

Returns a unified health factor considering positions across Aave, Kamino, and other lending protocols.

Get Started in 5 Minutes

1. Install and start WAIaaS:

npm install -g @waiaas/cli
waiaas init
waiaas start

1. Create wallets for both chains:

waiaas quickset --mode mainnet

1. Fund your wallets and create a session:

# Creates API session token
curl -X POST http://127.0.0.1:3100/v1/sessions 
  -H "Content-Type: application/json" 
  -H "X-Master-Password: <your-password>" 
  -d '{"walletId": "<wallet-uuid>"}'

1. Test a simple lending operation:

curl -X POST http://127.0.0.1:3100/v1/actions/aave-v3/supply 
  -H "Content-Type: application/json" 
  -H "Authorization: Bearer wai_sess_<token>" 
  -d '{
    "asset": "0xA0b86a33E6441b55434DD5031Ee5CC8a0b080eb6",
    "amount": "1000000"
  }'

1. Check your positions:

curl http://127.0.0.1:3100/v1/wallet/defi-positions 
  -H "Authorization: Bearer wai_sess_<token>"

You’re now lending on Aave through a unified API. Adding Kamino, Jupiter swaps, or Hyperliquid perps uses the same authentication and request patterns.

What’s Next

WAIaaS includes 39 REST API routes, 45 MCP tools for Claude integration, and comprehensive policy management for production DeFi applications. The complete protocol list includes Aave V3, Jupiter, Kamino, Lido, Hyperliquid, Drift, Polymarket, and more.

Ready to unify your DeFi integrations? Check out the full documentation and codebase at https://github.com/minhoyoo-iotrust/WAIaaS or visit https://waiaas.ai to get started.

Most AI chat apps make a quiet assumption that costs them a lot: one model is good enough for everything. It isn’t.

When I started building Chymera, I wanted to fix that. The idea was simple — instead of locking the user into a single LLM, the system should figure out what kind of question is being asked and send it to the model best suited to answer it.

This is the story of how I built that routing layer, what I got wrong the first time, and what the working version actually looks like.

The Problem With Single-Model Architectures

Every major AI chat product — ChatGPT, Claude, Gemini — lets you switch models manually. But users don’t think in terms of models. They just ask questions. The mental overhead of “hmm, should I use GPT-4o or o1 for this?” is friction that shouldn’t exist.

Beyond UX, there’s a real capability argument. Llama 3.3 70B via Groq is exceptional at code generation, while Qwen QwQ 32B has unusually strong multi-step reasoning. Gemini 2.5 Flash is fast and has native tool use that pairs well with live web search.

No single model wins every category. So why force users to pick?

Designing the Classifier

The first instinct was to use an LLM to classify intent. Send the query to a lightweight model, get back a category, then route accordingly. I tried this briefly and dropped it immediately. The latency was unacceptable — you’re adding a full round trip before the actual answer even starts.

The solution was a rule-based classifier written in plain JavaScript. No API call, no model inference, zero latency. It runs synchronously before any request goes out.

The classifier returns one of seven categories: chitchat, coding, reasoning, creative, search, factual, or general.

How It Works

function classifyQuery(query) {
  const q = query.toLowerCase().trim();
  const words = q.split(/s+/);
  const len = words.length;

  // Chitchat: short messages matching known conversational phrases
  const cleanQ = q.replace(/[.,!?]/g, '');
  const isChitchat = len <= 5 && chitchatPhrases.some(p =>
    new RegExp(`\b${p}\b`, 'i').test(cleanQ) || cleanQ === p
  );
  if (isChitchat) return 'chitchat';

  // Search: time-sensitive signals win before anything else
  if (any(q, realtimeSignals)) return 'search';

  // Recent year + question word = almost always needs live data
  const hasRecentYear = /b(2024|2025|2026)b/.test(q);
  const hasQuestionWord = any(q, ['what', 'who', 'when', 'latest', 'current', 'now']);
  if (hasRecentYear && hasQuestionWord && len >= 4) return 'search';

  // Reasoning, coding, creative, factual — keyword matching in priority order
  if (any(q, reasoningSignals)) return 'reasoning';
  if (any(q, codingSignals)) return 'coding';
  if (any(q, creativeSignals)) return 'creative';
  if (len >= 3 && any(q, factualSignals)) return 'factual';

  return 'general';
}

function any(text, keywords) {
  return keywords.some(k => text.includes(k));
}

The ordering matters more than the keyword lists. Search detection runs before factual detection — because “what is the current price of ETH” needs live data even though it matches factual patterns (what is). Getting this priority order wrong was my first significant bug.

Mapping Categories to Models

Once the classifier returns a category, model selection is a simple switch:

function getModel(queryType) {
  switch (queryType) {
    case 'coding':    return 'llama-3.3-70b-versatile';
    case 'reasoning': return 'qwen-qwq-32b-preview';
    case 'creative':  return 'gemini-2.5-flash';
    case 'search':    return 'llama-3.3-70b-versatile';
    case 'factual':   return 'llama-3.3-70b-versatile';
    case 'chitchat':  return 'llama-3.3-70b-versatile';
    default:          return 'gemini-2.5-flash';
  }
}

Groq handles Llama and Qwen. Gemini 2.5 Flash runs through Google’s Generative AI SDK. The two providers need different streaming implementations, so I extracted a shared aiCore.js module that normalises both into a single SSE pipeline. The chat handler doesn’t need to know which provider it’s talking to.

The Rate Limit Problem

Running on multiple free-tier API keys creates an obvious problem: 429 errors. The fix was a key rotation pool:

function getKeyPool(prefix) {
  const keys = [];
  for (let i = 1; i <= 10; i++) {
    const k = process.env[`${prefix}_${i}`];
    if (k) keys.push(k);
  }
  if (process.env[prefix]) keys.push(process.env[prefix]);
  return [...new Set(keys)].filter(Boolean);
}

let groqIdx = 0;

function nextGroqKey() {
  const key = groqKeys[groqIdx % groqKeys.length];
  groqIdx++;
  return key;
}

Each request picks the next key via round-robin. If a key returns 429, the request retries with the next one. With 10 keys in the pool, the effective rate limit headroom multiplies by pool size.

What Runs Where

The architecture has three independently deployed tiers:

• Frontend — React 18 SPA on Netlify. Handles auth state, renders the chat UI, consumes the SSE stream token by token.
• API — Express on Railway. Runs the classifier, selects the model, manages the key pool, and pipes the LLM stream back via Server-Sent Events.
• Data layer — Supabase for PostgreSQL and auth. Row-level security means each user can only read their own chat history — one policy handles it without any application-level filtering.

Mem0 sits alongside the API and handles semantic memory. Before each non-trivial request, the system searches for relevant memories from past sessions and injects them into the system prompt. The user never needs to re-explain who they are or what they’re working on.

What I’d Do Differently

1. Handle ambiguity with confidence scores
Queries like “is Next.js better than Remix?” match reasoning signals but are also partly factual. The current system picks one and commits. A confidence score with fallback logic would handle edge cases better.

2. Improve chitchat detection on longer messages
The len <= 5 check prevents false positives but a longer casual message doesn’t classify cleanly. A separate tone detector would help here.

3. Stateful key rotation
The round-robin index resets on every serverless cold start. A distributed counter in Redis would distribute load more accurately across concurrent requests.

Try It

Chymera is live here:

Ask it something technical, then ask it something that needs a live answer — you’ll see the model badge in the UI change between responses. The classifier runs in under a millisecond and the routing is completely invisible to the user.

If you’ve built something similar or approached the routing problem differently, I’d like to hear how.

Every project I start has the same 20 minutes of setup.

Write the HTTP client with retry logic. Set up the file watcher. Write the CSV parser. Build the rate limiter. Wire up the scheduler. I know exactly how to do all of it — I’ve done it fifty times. But I still write it from scratch every time, because I never had a clean canonical version I trusted enough to copy.

This year I stopped doing that. I went back through my last 2 years of projects and pulled out every script that (a) I’d rebuilt more than twice and (b) had survived production without modifications. I got 25 scripts. I cleaned them up, documented them properly, and packaged them.

Here’s what’s in there and why I made the choices I made.

What “production-ready” means in practice

Tutorial scripts have three failure modes:

1. No error handling — they work until they don’t, with no useful error output
2. Hardcoded config — you have to edit the file to change any parameter
3. No CLI — you can only run them from another Python script, not from a shell or scheduler

Every script in the cookbook fixes all three. They use argparse for CLI, handle errors with useful exit codes and messages, and are completely self-contained — no shared state, no internal imports required.

The goal: you should be able to drop any script into a new project, read it in 5 minutes, and run it. Then adapt it if you need to.

The pattern I found across all 25 scripts

When I went back through 2 years of projects to pull the scripts worth keeping, I expected to filter by “does this work.” That turned out to be the wrong filter. Plenty of scripts worked. What I actually filtered on was: has this been reused without modification?

The scripts that survived that filter shared four properties. None of them are surprising in isolation. What surprised me was how consistently tutorials and quick solutions violate all four.

1. They fail loudly with useful exit codes and error messages, not silently.

A script that swallows an exception and exits 0 is worse than one that crashes — at least the crash tells you something broke. The production-worthy version catches exceptions at the boundary, logs a message that includes the error type and the value that caused it, and exits with a non-zero code. A cron job that returns exit 1 triggers an alert. One that returns exit 0 silently drops your data with no indication anything went wrong.

2. Every configurable value is a CLI arg with a documented default, never hardcoded.

Hardcoded values look like shortcuts when you’re writing the script and look like landmines six months later when someone runs it in a slightly different context. File paths, timeouts, retry counts, output directories — all of these appear as argparse arguments with sensible defaults. If you need to run the same script against a different endpoint or with a longer timeout, you pass a flag; you don’t edit the file.

3. They’re idempotent — running them twice produces the same result.

This matters most for scripts run by schedulers or CI pipelines, where the same script might run twice due to a retry or a race condition. A script that creates a file should check if it already exists. A script that inserts records should handle duplicates. A script that syncs a directory should be a no-op if the directories are already in sync. Idempotence is not free, but the cost of writing it once is much lower than the cost of debugging a corruption caused by a double-run at 3 AM.

4. They do one thing and expose a clean interface.

The scripts that didn’t survive tended to grow. Start as a file downloader, acquire some transformation logic, pick up a notification step, accumulate a few flags that change the behavior in incompatible ways. A script doing four things is four things you can’t independently test, reuse, or replace. The ones worth keeping are narrow. The composition of narrow scripts produces complex behavior; the script itself stays simple.

Tutorial code violates all four of these constantly — not because the authors are wrong, but because tutorials optimize for clarity at the moment of reading, not for reliability at the moment of production failure. AI-generated scripts have the same problem: they optimize for plausible-looking output, not for what happens when the rate limiter fires at 2 AM and your CI has no one watching it. Every script in this cookbook was debugged in production before it ended up here. None of them are generated.

4 scripts worth showing in detail

1. HTTP client with retry logic and rate limiting

The problem: most requests implementations handle 200s and 500s. They don’t handle rate limits (429), transient failures (503), or connection timeouts gracefully. You find out when something breaks at 2 AM.

What the script does:

• Configurable backoff (exponential with jitter, adjustable base/max/multiplier)
• Handles 429 with Retry-After header parsing — waits the correct amount, not a fixed sleep
• Distinguishes retryable errors (5xx, timeout, connection error) from permanent ones (4xx except 429)
• Session reuse for performance
• Structured logging of every retry attempt with reason

CLI: python http_client.py --url https://api.example.com/endpoint --retries 5 --timeout 30

# Core retry logic — simplified excerpt
def request_with_retry(url, method="GET", max_retries=3, backoff_base=1.0, **kwargs):
    session = requests.Session()
    last_error = None

    for attempt in range(max_retries + 1):
        try:
            response = session.request(method, url, **kwargs)

            if response.status_code == 429:
                retry_after = int(response.headers.get("Retry-After", backoff_base * (2 ** attempt)))
                logger.warning(f"Rate limited. Waiting {retry_after}s (attempt {attempt+1}/{max_retries})")
                time.sleep(retry_after)
                continue

            if response.status_code >= 500:
                wait = backoff_base * (2 ** attempt) + random.uniform(0, 1)
                logger.warning(f"Server error {response.status_code}. Retrying in {wait:.1f}s")
                time.sleep(wait)
                continue

            response.raise_for_status()
            return response

        except (requests.Timeout, requests.ConnectionError) as e:
            wait = backoff_base * (2 ** attempt) + random.uniform(0, 1)
            logger.warning(f"Connection error: {e}. Retrying in {wait:.1f}s")
            last_error = e
            time.sleep(wait)

    raise last_error or requests.RequestException(f"Max retries exceeded for {url}")

2. File watcher with debouncing

The problem: watchdog is great but every implementation I’ve seen either (a) fires duplicate events on a single save or (b) doesn’t handle the “file replaced” pattern that editors like vim use.

What the script does:

• Debounces file events with configurable delay (default 500ms) — one callback per logical change, not per filesystem event
• Handles create/modify/delete/move events separately with distinct callbacks
• Ignores temp files and swap files by default (configurable patterns)
• Recursive or non-recursive watching
• Graceful shutdown on SIGINT/SIGTERM

# Debouncing logic — simplified excerpt
class DebouncedHandler(FileSystemEventHandler):
    def __init__(self, callback, debounce_ms=500, ignore_patterns=None):
        self.callback = callback
        self.debounce_delay = debounce_ms / 1000
        self.ignore_patterns = ignore_patterns or [r'.*.swp$', r'.*.tmp$', r'~$']
        self._timers = {}

    def _should_ignore(self, path):
        return any(re.match(p, path) for p in self.ignore_patterns)

    def on_modified(self, event):
        if event.is_directory or self._should_ignore(event.src_path):
            return
        self._schedule_callback(event.src_path, event)

    def _schedule_callback(self, path, event):
        if path in self._timers:
            self._timers[path].cancel()
        timer = threading.Timer(self.debounce_delay, self.callback, args=[event])
        self._timers[path] = timer
        timer.start()

3. Paginated API collector

The problem: every API paginates differently. Some use page/per_page, some use offset/limit, some use cursor tokens, some use next_url in the response. Writing a collector for each one is tedious and error-prone.

What the script does:

• Supports four pagination strategies: page-based, offset-based, cursor-based, and next-URL-based
• Configurable via JSON schema describing the API’s pagination parameters
• Yields records as they arrive (generator) — works on APIs returning millions of records
• Rate limiting between requests
• Configurable stop conditions (max records, max pages, or custom predicate)

CLI: python api_collector.py --config api_config.json --output records.jsonl

4. Structured CSV/JSON normalizer

The problem: real-world data is inconsistent. Column names vary across exports. Types need coercion. Nested JSON needs flattening. Nulls are represented as empty strings, “NULL”, “N/A”, “none”, or actual nulls depending on the source.

What the script does:

• Configurable field mapping (rename columns via config file)
• Type coercion with null handling (configurable null representations)
• JSON column flattening with configurable depth limit
• Duplicate detection and deduplication
• Validation with configurable error handling (skip / raise / log)
• Output to CSV, JSONL, or SQLite

CLI: python normalizer.py --input data.csv --config schema.json --output clean.jsonl --on-error log

Full list of 25 scripts

HTTP & APIs (6)

• http_client.py — retry logic, rate limiting, session reuse
• api_collector.py — paginated API collection (4 pagination strategies)
• webhook_sender.py — send webhooks with HMAC signing, retry, and delivery tracking
• webhook_receiver.py — receive and validate webhooks (Flask-based, signature verification)
• oauth_client.py — OAuth 2.0 client with token refresh
• graphql_client.py — GraphQL query client with variable substitution

File Operations (5)

• file_watcher.py — file system monitoring with debouncing
• batch_processor.py — process files in a directory with parallelism control
• file_sync.py — sync two directories (local or S3), with diff-only mode
• archive_manager.py — compress/decompress with progress, integrity checking
• log_rotator.py — rotate and compress log files with configurable retention

Data (5)

• normalizer.py — CSV/JSON normalization with type coercion and mapping
• deduplicator.py — deduplicate records by configurable key fields
• csv_merger.py — merge multiple CSV files with conflict resolution
• json_flattener.py — flatten nested JSON to tabular structure
• schema_validator.py — validate records against JSON Schema with detailed error output

Scheduling & Jobs (4)

• cron_wrapper.py — wrap any command as a cron job with logging, alerting, and lock files
• task_queue.py — simple file-based task queue (no Redis required)
• retry_runner.py — run a command with retry logic and configurable backoff
• job_scheduler.py — schedule jobs with APScheduler, persistent state

System & Cloud (5)

• db_backup.py — backup PostgreSQL/MySQL/SQLite with rotation and S3 upload
• s3_sync.py — sync local directory to S3 with change detection
• process_monitor.py — monitor a process, restart if dead, alert on anomalies
• ssl_checker.py — check SSL certificate expiry across a list of domains
• docker_cleanup.py — prune stopped containers, dangling images, unused volumes

Scripts I almost cut (and why they survived)

The hardest part of building the cookbook was the cutting phase. I had more than 40 scripts that fit the “rebuilt twice, survived production” criteria. Getting to 25 meant making the case for each one.

A few were hard to cut because their value only shows up in edge cases — the cases you only hit once, at the worst possible time.

cron_wrapper.py was almost cut because every team already has cron. The counterargument came from a near-miss. A data processing job had been running nightly for three months without issues. One night the job took longer than expected and the next scheduled run started before the first one finished. Both instances wrote to the same output file, the second one truncated it mid-write, and the first one finished writing to a file that was now shorter than it started. The corruption was silent — the file was valid CSV, just missing the last third of the records. We found it four days later when a downstream report came up short. cron_wrapper.py uses a lock file to prevent concurrent execution and alerts on lock contention. The lock behavior isn’t useful 99% of the time. It’s essential the one time it would have mattered.

process_monitor.py was almost cut because modern infrastructure uses systemd for process management. The counterargument: systemd manages services. process_monitor.py manages scripts that aren’t services — a nightly batch job, a file processor that runs on demand, a data sync that needs to run continuously but doesn’t warrant a full service definition. For these, you want restart-on-failure and anomaly alerting (CPU spike, memory leak, unusual runtime) without the overhead of writing and maintaining a systemd unit file. The script fills that gap and has been used on three different projects specifically for batch jobs that grew beyond what a bare cron job could handle.

retry_runner.py was almost cut because it seemed redundant with http_client.py. The distinction: http_client.py handles retry logic for HTTP requests specifically. retry_runner.py handles retry logic for arbitrary shell commands — a database migration that occasionally fails on the first attempt due to connection pool exhaustion, a file upload to a flaky external service, a test suite that reliably passes on the second run when given a few seconds after the first. It wraps any command, not just HTTP calls, and that generality turned out to matter.

How to adapt these to your codebase

These are starting points, not drop-in libraries. The distinction matters because “drop-in library” implies a black box — you configure it, call it, and don’t need to understand what’s inside. That’s the wrong model for scripts you’re going to run in production on your own infrastructure. You should understand what they do before you trust them.

Every script in the cookbook is under 200 lines, with clear section comments that separate the argument parsing, the core logic, and the error handling. The structure is intentional: it makes the scripts easy to read in one sitting, and easy to find the specific part you want to change.

The most common adaptation pattern: copy the script, read it once to understand the flow, find the 5-10 lines that implement the specific behavior you need, strip everything else, extend from there. You’re not supposed to use the full script as-is in every case. You’re supposed to extract the part that solves your problem and build on it.

A concrete example: the retry logic in http_client.py — the exponential backoff with jitter, the 429 handling with Retry-After header parsing, the distinction between retryable and permanent errors — has been extracted and adapted for at least three different internal tools this year. One was a Slack API client that needed to handle rate limits differently from the default. One was a webhook sender for an API that returned 503 during deployments and needed more aggressive backoff. One was an internal data pipeline that called a rate-limited external service with custom authentication. The core retry logic was the same in all three; the adaptation was in the error classification and the authentication layer. Starting from the script saved 30-60 minutes of writing and debugging the retry edge cases from scratch each time.

Why one-time price

Most tools that help you automate things are SaaS. Monthly subscription, usage limits, API keys you don’t control. For a set of scripts you run on your own infrastructure, that model doesn’t make sense.

The cookbook is $39 one-time at https://kazdispatch.gumroad.com/l/hydbq. You get the scripts, no subscription, no usage tracking.

30-day refund if it doesn’t deliver value. No questions asked.

What’s the automation you keep rewriting? Leave it in the comments — file operations, API calls, scheduling, data pipeline boilerplate — and I’ll either point you to the script that covers it or admit it’s not in there. If three people ask for the same thing, I’ll write it. (#discuss)

(Part 2 covers how to chain these scripts together with prompt_chainer.py — building multi-step automation pipelines that use AI to handle the decisions.)

If you want to know when Part 2 drops — and get the free conftest.py template I use on every Python project — sign up here. No spam, just the occasional Python automation post.

Ollama works great on bare metal. It gets even more interesting when you treat it like a service: a stable endpoint, pinned versions, persistent storage, and a GPU that is either available or it is not.

This post focuses on one goal: a reproducible local or single-node Ollama “server” using Docker Compose, with GPU acceleration and persistent model storage.

It intentionally skips generic Docker and Compose basics. When you need a compact list of the commands you reach for most often (images, containers, volumes, docker compose), the Docker Cheatsheet is a good companion.

When you want HTTPS in front of Ollama, correct streaming and WebSocket proxying, and edge controls (auth, timeouts, rate limits), see Ollama behind a reverse proxy with Caddy or Nginx for HTTPS streaming.

For how Ollama fits alongside vLLM, Docker Model Runner, LocalAI, and cloud hosting trade-offs, see
LLM Hosting in 2026: Local, Self-Hosted & Cloud Infrastructure Compared.

When Compose beats a bare metal install

A native install is frictionless for one developer on one machine. The moment you have any of the following, Compose starts to win on ergonomics:

A team setup benefits because the service definition is a file you can review, version, and share. A single-node server benefits because upgrades turn into an image tag bump and a restart, while your model storage stays put (as long as it is on a volume). Ollama also tends to live next to sidecars: a Web UI, a reverse proxy, an auth gateway, a vector DB, or an agent runtime. Compose is good at “one command to start the whole stack”, without turning your host into a snowflake.

This approach aligns well with how the official Ollama container is designed: the image runs ollama serve by default, exposes port 11434, and is meant to keep state under a mountable directory.

A Compose skeleton that is actually useful for Ollama

Start with two decisions:

First, how you will pin versions. The Docker Hub image is ollama/ollama, so you can pin a specific tag in .env instead of relying on latest.

Second, where the model data will live. The official docs mount a volume to /root/.ollama so models are not re-downloaded each time the container is replaced.

Here is a Compose file that bakes those decisions in, and keeps the “knobs” close to the service:

services:
  ollama:
    image: ollama/ollama:${OLLAMA_IMAGE_TAG:-latest}
    container_name: ollama
    restart: unless-stopped

    # Keep it local by default, expose it later if you need to.
    ports:
      - "${OLLAMA_BIND_IP:-127.0.0.1}:11434:11434"

    # Persistent models and server state.
    volumes:
      - ollama:/root/.ollama

    environment:
      # The official image already defaults to 0.0.0.0:11434 inside the container,
      # but keeping it explicit helps when you override things later.
      - OLLAMA_HOST=0.0.0.0:11434

      # Service tuning.
      - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE:-5m}
      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL:-1}
      - OLLAMA_MAX_LOADED_MODELS=${OLLAMA_MAX_LOADED_MODELS:-1}

      # Optional, but relevant when a browser-based UI talks to Ollama directly.
      # See the Networking section for why this exists.
      - OLLAMA_ORIGINS=${OLLAMA_ORIGINS:-}

    # GPU reservation is a separate section below.
    # Add it only on hosts that actually have NVIDIA GPUs.

volumes:
  ollama: {}

A matching .env keeps upgrades boring:

# Pin the image version you have tested.
OLLAMA_IMAGE_TAG=latest

# Local by default. Change to 0.0.0.0 when you intentionally expose it.
OLLAMA_BIND_IP=127.0.0.1

# Keep-alive tweaks cold-start latency vs memory footprint.
OLLAMA_KEEP_ALIVE=5m

# Concurrency knobs.
OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_LOADED_MODELS=1

# Leave empty unless you are serving browser clients that hit Ollama directly.
OLLAMA_ORIGINS=

A small but important nuance: Ollama itself has a default host bind of 127.0.0.1:11434 in the general configuration, but the official container image sets OLLAMA_HOST=0.0.0.0:11434 so the service is reachable through published ports.

If you want a quick sanity check without involving any client SDKs, the Ollama API includes a “list local models” endpoint at GET /api/tags.

Persistent model storage and the least painful way to move it

If you only remember one thing, make it this: the container must have persistent storage, otherwise every rebuild is a re-download.

Ollama lets you choose the models directory using OLLAMA_MODELS. In the reference implementation, the default is $HOME/.ollama/models, and setting OLLAMA_MODELS overrides that.

Inside the official Docker image, $HOME maps naturally to the /root layout used by the documented volume mount (/root/.ollama), which is exactly why the official docker run examples mount that directory.

There are two storage patterns that tend to work well in practice:

A named Docker volume is simplest and portable. It is also easy to accidentally orphan, so it is worth naming it intentionally (for example ollama) and keeping it stable across Compose refactors.

A bind mount to a dedicated disk is better when model sizes start to dominate your root filesystem. In that case, you either mount the whole /root/.ollama to that disk, or you mount a custom directory and point OLLAMA_MODELS at it.

If you are actively reorganising storage, this is where an explicit “move models” playbook helps. See: move-ollama-models .

NVIDIA GPU support with Compose and the NVIDIA Container Toolkit

Ollama can use NVIDIA GPUs in Docker, but the image cannot magic a GPU into existence. The host needs working NVIDIA drivers and the NVIDIA Container Toolkit, and Docker must be configured to use it. The Ollama Docker docs explicitly call out installing nvidia-container-toolkit, configuring the runtime via nvidia-ctk runtime configure --runtime=docker, and restarting Docker.

On the Compose side, the clean, modern way is device reservations. Docker documents GPU access in Compose using deploy.resources.reservations.devices, with capabilities: [gpu], driver: nvidia, and either count (including all) or device_ids.

Add this to the ollama service when you are on an NVIDIA host:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]

If you have multiple GPUs and want to keep Ollama on specific devices, switch from count to device_ids as documented by Docker (they are mutually exclusive).

You will sometimes see legacy Compose examples that use runtime: nvidia. That can fail on newer setups with errors like “unknown or invalid runtime name: nvidia”, which is a strong hint that you should move to the supported device reservation pattern and make sure the toolkit is configured on the host.

A useful detail hiding in plain sight: the official ollama/ollama image sets NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES=compute,utility. These are standard knobs recognised by the NVIDIA container runtime, and they are already present unless you overwrite them.

To confirm whether you are actually getting GPU inference (not just a container that starts), Ollama recommends using ollama ps and checking the “Processor” column, which shows whether the model is on GPU memory.

Platform reality check: Ollama notes that GPU acceleration in Docker is available on Linux (and Windows with WSL2), and not available on Docker Desktop for macOS due to the lack of GPU passthrough.

Networking choices: host vs bridge, ports, and CORS

Networking is where most “it runs but my app cannot connect” bugs come from.

Bridge networking with published ports

The default Compose network is a bridge network. In this setup, publishing 11434:11434 makes Ollama reachable from the host on port 11434, while other containers should talk to it using the service name ollama (not localhost). A lot of people trip on this because localhost inside a container means “this container”, not “the Ollama container”.

Ollama itself runs an HTTP server on port 11434 (the image exposes it), and the common convention is that clients use http://localhost:11434 on the host when ports are published.

Host networking

network_mode: host can be tempting on a single-node server because it removes port publishing and makes localhost semantics simpler. The trade-off is you lose the isolation and namespacing benefits of a bridge network, and you are more likely to hit port conflicts.

Exposing Ollama intentionally

Ollama on a normal install binds to 127.0.0.1 by default, and the documented way to change the bind address is OLLAMA_HOST.

In Docker, you have two layers:

Ollama bind address, controlled by OLLAMA_HOST (the container image defaults to binding on all interfaces inside the container).

Reachability from outside the container, controlled by Compose ports and the host firewall.

A pattern I like is “bind locally by default” via 127.0.0.1:11434:11434, then switch to 0.0.0.0:11434:11434 only when I have a reason to expose it.

Browser clients and OLLAMA_ORIGINS

If a browser-based UI or extension calls Ollama directly, you are in CORS territory. Ollama allows cross-origin requests from 127.0.0.1 and 0.0.0.0 by default, and you can configure additional origins using OLLAMA_ORIGINS.

This matters even on a single node, because “it works with curl” does not mean “it works from a browser app”.

Upgrade and rollback patterns that fit a single-node server

Ollama evolves quickly. Your Compose file can make that a calm process instead of a late-night surprise.

Upgrade by bumping a tag, not by hoping “latest” behaves

The most practical upgrade strategy is to pin the image to a known-good tag in .env, and bump it intentionally. The image is published as ollama/ollama on Docker Hub.

Because model data and server state are stored under a mounted directory (/root/.ollama in the official docs), replacing the container does not imply re-downloading models.

Rollback is just switching the tag back

Rollback is the same mechanism in reverse: set the previous tag, recreate the container, keep the same volume. This is where pinning pays for itself.

Data migration is mostly about storage paths

Most “migrations” in a single-node setup are not about database schemas. They are about disk layout. If you change the models directory (via OLLAMA_MODELS) or move the mounted volume to a new disk, you are doing a data migration whether you call it that or not.

If you want a practical guide for reorganising the model directory on real machines, see: move-ollama-models .

A final note that is easy to miss: Ollama’s API documentation explicitly says the API is expected to be stable and backwards compatible, with rare deprecations announced in release notes. That makes “upgrade the server, keep clients working” a reasonable default expectation for a single-node service endpoint.

Common failures: GPU permissions, driver mismatch, and OOM

This section is deliberately symptom-driven. The goal is not “every possible Docker error”, only the failures that show up specifically in Ollama + GPU + persistent storage setups.

GPU visible on the host, missing in the container

If the host has a working NVIDIA driver but the container does not see a GPU, the common causes are:

The NVIDIA Container Toolkit is not installed or the Docker runtime is not configured via nvidia-ctk. Ollama’s Docker docs call this out directly.

Compose is not reserving a GPU device. The supported way is deploy.resources.reservations.devices with the gpu capability as documented by Docker.

A legacy runtime: nvidia configuration is being used on a daemon that does not recognise it, producing “unknown or invalid runtime name: nvidia”.

For validation, ollama ps gives you a pragmatic check: it shows whether a model is loaded in GPU memory.

Permission denied on GPU devices

The “permission denied” flavour of GPU failures typically points to environment constraints rather than Ollama itself. Examples include running rootless Docker, security policies, or device nodes not being exposed as expected. The Docker Compose GPU support docs are explicit that the host must have GPU devices and that the Docker daemon must be set accordingly.

When in doubt, reduce the variables: confirm the toolkit configuration (host), then confirm GPU reservation (Compose), then confirm GPU usage (ollama ps).

Wrong driver, wrong expectation

Ollama in Docker relies on the host driver stack. If the host driver is missing, too old, or misconfigured, you will see failures that look like “Ollama is broken” but are really “CUDA stack is not usable”. The official docs place the container toolkit and Docker daemon configuration as prerequisites for NVIDIA GPU usage.

Out of memory: VRAM or RAM disappears fast

OOM is the most predictable failure mode for local inference, and it is usually self-inflicted by configuration.

Ollama supports concurrent processing through multiple loaded models and parallel request handling, but it is constrained by available memory (system RAM on CPU inference, VRAM on GPU inference). When GPU inference is used, new models must fit in VRAM to allow concurrent model loads.

Two configuration details are worth treating as first-class “server settings”:

OLLAMA_NUM_PARALLEL increases parallel request processing per model, but required memory scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.

OLLAMA_KEEP_ALIVE controls how long models remain loaded (default is 5 minutes). Keeping models loaded reduces cold-start latency, but it also pins memory.

If you are stabilising a single-node service under load, the non-dramatic fixes usually look like:

Lower parallelism and context defaults before you change anything else.

Limit how many models are allowed to remain loaded concurrently.

Consider memory-reduction features like Flash Attention (OLLAMA_FLASH_ATTENTION=1) and lower precision K/V cache types (OLLAMA_KV_CACHE_TYPE) when your bottleneck is memory, not raw compute.

When it is not Ollama: choosing Docker Model Runner instead

Sometimes the “failure” is really a tooling mismatch. If your organisation already standardises on Docker-native artifacts and workflows, Docker Model Runner (DMR) can be a better fit than running Ollama as a long-lived service container.

Docker positions DMR as a way to manage, run, and serve models directly via Docker, pulling from Docker Hub or other OCI registries, and serving OpenAI-compatible and Ollama-compatible APIs.

It also supports multiple inference engines (including llama.cpp, and vLLM on Linux with NVIDIA GPUs), which can matter if you care about throughput characteristics, not just “run one model locally”.

If you want a practical command reference and a deeper comparison angle, see: Docker Model Runner Cheatsheet: Commands & Examples.

Running Ollama behind a reverse proxy is the simplest way to get HTTPS, optional access control, and predictable streaming behaviour.

This post focuses on Caddy and Nginx ingress for the Ollama API, not on client code.

If you already have Python or Go clients talking to Ollama, this post is the missing piece: ingress and transport for the same API.

For how Ollama fits alongside vLLM, Docker Model Runner, LocalAI, and cloud hosting trade-offs, see
LLM Hosting in 2026: Local, Self-Hosted & Cloud Infrastructure Compared.

For request examples and client code, see
Ollama CLI Cheatsheet.

For UI and multi-user layers, see
Open WebUI overview, quickstart and alternatives.

For the bigger picture on self-hosting and data control, see
LLM self-hosting and AI sovereignty.

For a reproducible single-node Ollama service in Docker Compose (persistent volumes, OLLAMA_HOST, NVIDIA GPUs, upgrades), see
Ollama in Docker Compose with GPU and Persistent Model Storage.

Why you should proxy Ollama instead of exposing port 11434

Ollama is designed to run locally first. Out of the box it binds to localhost on port 11434, which is great for a developer workstation and a not-so-subtle hint that the raw port is not meant to be internet-facing.

I treat port 11434 as an internal, high-cost API. If it is reachable from the public internet, anyone who finds it can burn your CPU or GPU time, fill your disk by pulling models, or just keep connections open until something times out. A reverse proxy does not make Ollama safer by magic, but it gives you a place to put the controls that matter at the edge: TLS, authentication, timeouts, rate limits, and logs.

This matters because the local Ollama API does not ship with a built-in authentication layer. If you expose it, you typically add auth at the edge or keep it private and reachable only over a trusted network.

The second reason is UX. Ollama streams responses by default. If the proxy buffers or compresses in the wrong spot, streaming feels broken and UIs look like they are “thinking” with no output.

Minimal architecture and binding strategy

A clean minimum looks like this:

Client (curl, Python, Go, UI)
        |
        | HTTPS (optional Basic Auth or SSO)
        v
Reverse proxy (Caddy or Nginx)
        |
        | HTTP (private LAN, localhost, or Docker network)
        v
Ollama server (ollama serve on 127.0.0.1:11434)

Two practical rules keep this boring in the best way.

First, keep Ollama private and move exposure to the proxy. If Caddy or Nginx runs on the same host, proxy to 127.0.0.1:11434 and do not change Ollama’s bind address. If the proxy runs elsewhere (separate host, separate VM, or a container network), bind Ollama to a private interface, not 0.0.0.0 on the public NIC, and lean on a firewall.

Second, decide early whether browsers will call Ollama directly. If a browser-based tool hits Ollama from a different origin, you may need to deal with CORS. If everything is served from one domain via the proxy (recommended for sanity), you can often avoid CORS entirely and keep Ollama strict.

Reverse proxy configs for streaming and WebSockets

Ollama’s API is regular HTTP, and its streaming is newline-delimited JSON (NDJSON). That means you want a proxy that can do three things well:

• Do not buffer streaming responses.
• Do not kill long-running requests just because the model took a while to speak.
• If a UI uses WebSockets (some do), forward Upgrade cleanly.

You can keep this simple. In many cases, “correct WebSockets handling” is just having a config that is Upgrade-safe even if the upstream does not use WebSockets today.

Caddy Caddyfile example

Caddy is the “less config, more defaults” option. If you put a public domain name in the site address, Caddy will typically obtain and renew certificates automatically.

Minimal reverse proxy, HTTPS, and streaming-friendly settings:

# ollama.example.com A/AAAA -> your proxy host
ollama.example.com {

    # Optional Basic Auth at the edge.
    # Generate a password hash with:
    #   caddy hash-password --algorithm bcrypt
    #
    # basic_auth {
    #   alice $2a$12$REDACTED...
    # }

    reverse_proxy 127.0.0.1:11434 {

        # Some setups prefer pinning the upstream Host header.
        # Ollama's own docs show this pattern for Nginx.
        header_up Host localhost:11434

        # For streaming or chat-like workloads, prefer low latency.
        # NDJSON streaming usually flushes immediately anyway, but this makes it explicit.
        flush_interval -1

        transport http {
            # Avoid upstream gzip negotiation if it interferes with streaming.
            compression off

            # Give Ollama time to load a model and produce the first chunk.
            response_header_timeout 10m
            dial_timeout 10s
        }
    }
}

If you already have an SSO gateway (oauth2-proxy, Authelia, authentik outpost, etc), Caddy has an opinionated forward auth directive. The pattern is “auth first, then proxy”:

ollama.example.com {
    forward_auth 127.0.0.1:4180 {
        uri /oauth2/auth
        # Copy the identity headers your gateway returns, if you need them.
        copy_headers X-Auth-Request-User X-Auth-Request-Email Authorization
    }

    reverse_proxy 127.0.0.1:11434
}

Nginx server block example

Nginx gives you a bit more rope. The upside is that the knobs are explicit, and it has built-in primitives for rate limiting and connection limiting. The footgun is buffering: Nginx buffers proxied responses by default, which is the opposite of what you want for NDJSON streaming.

This example includes:

• HTTP to HTTPS redirect
• TLS certificate paths (Certbot style)
• WebSocket-safe Upgrade forwarding
• Streaming-friendly proxy_buffering off
• Longer timeouts than the default 60s

# /etc/nginx/conf.d/ollama.conf

# WebSocket-safe Connection header handling
map $http_upgrade $connection_upgrade {
    default upgrade;
    ""      close;
}

# Optional request rate limiting (IP-based)
# limit_req_zone $binary_remote_addr zone=ollama_rate:10m rate=10r/s;

server {
    listen 80;
    server_name ollama.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name ollama.example.com;

    ssl_certificate     /etc/letsencrypt/live/ollama.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ollama.example.com/privkey.pem;

    # Optional Basic Auth at the edge.
    # auth_basic "Ollama";
    # auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        # Optional rate limit
        # limit_req zone=ollama_rate burst=20 nodelay;

        proxy_pass http://127.0.0.1:11434;

        # Match Ollama docs pattern when proxying to localhost.
        proxy_set_header Host localhost:11434;

        # WebSocket Upgrade handling (harmless if unused).
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # Critical for NDJSON streaming.
        proxy_buffering off;

        # Prevent 60s idle timeouts while waiting for tokens.
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }
}

If you want an SSO-style gate in Nginx, the equivalent pattern is auth_request. Nginx sends a subrequest to your auth service, and only proxies to Ollama when auth returns 2xx.

TLS automation and renewal gotchas

For TLS, the operational split is simple.

With Caddy, TLS is usually “part of the reverse proxy”. Automatic HTTPS is one of its flagship features, so certificate issuance and renewal are coupled to keeping Caddy running, having working DNS, and exposing ports 80 and 443.

With Nginx, TLS is usually “a separate ACME client plus Nginx”. The common failure mode is not crypto, it is plumbing:

• Port 80 not reachable for HTTP-01 challenges.
• Certificates stored in a container but not persisted.
• Rate limits when doing repeated fresh installs or test deploys.

A subtle point that matters for long-lived services is that certificate lifetimes are short by design. Treat renewals as a background automation requirement, not an annual calendar event.

Authentication, abuse control, and verification

This is the part that makes an internet-facing LLM endpoint feel professional.

Authentication options, from blunt to elegant

Basic Auth at the proxy is blunt, but surprisingly effective for a private endpoint. It is also easy to apply to both HTTP requests and WebSocket upgrades.

If you want browser-friendly login flows, forward auth and auth_request are the common pattern. Your proxy stays stateless, and an auth gateway owns sessions and MFA. The trade-off is more moving parts.

If you are already running Open WebUI, you can also rely on its app-level authentication and keep Ollama itself private. The proxy then protects Open WebUI, not Ollama directly.

If you do not need public access at all, a network-only approach can be cleaner. For example, Tailscale Serve can expose a local service inside your tailnet without opening inbound ports on your router.

Abuse basics for an expensive API

Ollama is a powerful local API, and its surface goes beyond generation. It has endpoints for chat, embeddings, listing models, and version checks. Treat the whole API as sensitive.

Official API reference (endpoints and streaming): https://docs.ollama.com/api

At the proxy layer, there are three low-effort controls that reduce day-one pain:

• Rate limiting per IP on generation endpoints.
• Connection limits to stop a small number of clients holding everything open.
• Conservative timeouts that match your model and hardware reality, not generic web defaults.

At the Ollama layer, it can also reject overload with 503 and has server-side knobs for queueing. Proxy rate limiting keeps you from getting there as often.

Verification checklist

Use the same checks you would use for any streaming API.

1. Basic connectivity and TLS

   • curl -sS https://ollama.example.com/api/version
   • curl -sS https://ollama.example.com/api/tags | head

2. Streaming works end to end (no buffering)

   • curl -N https://ollama.example.com/api/generate -H "Content-Type: application/json" -d '{"model":"mistral","prompt":"Write 10 words only.","stream":true}'

If you are behind Basic Auth:

• curl -N -u alice:REDACTED https://ollama.example.com/api/generate -H "Content-Type: application/json" -d '{"model":"mistral","prompt":"Write 10 words only.","stream":true}'

1. Browser UI sanity
   • Load your chat UI and trigger a response.
   • If the UI uses WebSockets, confirm you do not see 400 or 426 errors and the connection stays open during generation.

If the curl output only appears at the end, it is almost always buffering at the proxy. Re-check proxy_buffering off in Nginx, and consider forcing low-latency flushing in Caddy for the Ollama site block.