VICIdial Dial Hopper: How It Works and Why Yours Is Empty

Last updated: March 2026 | Reading time: ~18 minutes

Your agents are sitting idle. The predictive dialer paused itself. You pull up the Real-Time Report and there it is — Hopper: 0. Campaign is active, lists are loaded, agents are logged in, and absolutely nothing is happening.

This is the single most common crisis in VICIdial operations, and it’s almost never about running out of leads. It’s about a chain of settings, filters, and cron jobs that have to align perfectly for leads to flow from your lists into the hopper and then into the dialer.

If you’ve been running VICIdial for any length of time, you’ve hit this wall. Let’s pull it apart.

What the Dial Hopper Actually Is

The dial hopper is a staging table — vicidial_hopper — that sits between your lead lists and the dialer engine. Think of it as a buffer. The dialer doesn’t reach into vicidial_list directly to find the next number to call. Instead, a background process (the VDHopper cron job) continuously scans your lists, applies every filter and rule you’ve configured, and loads qualifying leads into the hopper. The dialer then pulls from this pre-filtered, pre-sorted buffer.

Why the indirection? Performance. vicidial_list on a busy system can have tens of millions of rows. Doing a full table scan with all your filters applied every time the dialer needs a lead would hammer the database. The hopper keeps a small, ready-to-dial set in memory so the dialer can grab leads in microseconds.

Here’s the flow:

vicidial_list (millions of leads)
        │
        ▼
  VDHopper cron (runs every minute)
  Applies: list status, lead status, timezone, filters,
           DNC check, lead recycling rules, dial count limits
        │
        ▼
  vicidial_hopper (small buffer, typically 50-1000 leads)
        │
        ▼
  VDauto_dial (predictive engine)
  Grabs leads → dials → routes to agents

The hopper_level setting in your campaign controls how many leads the VDHopper process tries to keep in the hopper at any given time. Default is usually 50. For high-volume campaigns with 20+ agents, you need 200-500.

The VDHopper Cron Job: Your Hopper’s Heartbeat

Everything starts with the cron job. If this isn’t running, your hopper never fills. Period.

Check if it’s running:

# See if VDHopper is in the cron
crontab -l | grep VDHopper

# Expected output (every minute):
# * * * * * /usr/share/astguiclient/AST_VDhopper.pl

# Check if it's actually executing
ps aux | grep VDHopper

On a ViciBox install, this is set up automatically. On manual installs or after system migrations, it’s the first thing that breaks. The script lives at /usr/share/astguiclient/AST_VDhopper.pl and it needs to run every minute.

If the cron is running but the hopper is still empty, the problem is downstream — in your campaign settings, list configuration, or filtering rules.

VDHopper Execution Time

On systems with large lists (5M+ leads), the VDHopper script can take longer than 60 seconds to complete. When that happens, you get overlapping executions that fight over database locks, or worse, the hopper falls behind your dial velocity.

Check execution time:

# Find how long VDHopper takes to run
grep "VDHopper" /var/log/astguiclient/process_log.txt | tail -20

If execution consistently exceeds 45 seconds, you need to either reduce the number of active lists, add a MySQL index, or increase your database server’s resources. We’ve seen shops running 15M-lead lists on a single-core VM wondering why their hopper is empty — the cron literally can’t finish before it’s supposed to start again.

Why Your Hopper Is Empty: The 12 Most Common Causes

I’m going to go through these in order of how often we see them. In eight years of running VICIdial systems, cause #1 accounts for maybe 40% of empty-hopper tickets by itself.

1. No Dialable Leads Left (But You Think There Are)

You loaded 100,000 leads. You’ve been dialing for three days. You check vicidial_list and see 87,000 records. Plenty of leads, right?

Wrong. What matters isn’t total leads — it’s leads with a dialable status that haven’t exceeded your dial count limit.

In the VICIdial admin, go to Campaign Settings and look at Dial Status. This is the list of lead statuses that the hopper considers dialable. By default it’s usually NEW — meaning only leads that have never been called. After one dial attempt, a lead gets a disposition (NA, B, A, etc.) and it’s no longer NEW. Unless you’ve added those statuses to your dialable list, those leads are invisible to the hopper.

The fix: In Campaign Settings, add statuses like NA (No Answer), B (Busy), A (Answering Machine), CALLBK (Callback) to your Dial Status list. Also check your Dial Count Limit — if it’s set to 1, leads get exactly one attempt and then they’re done.

2. Lists Are Not Active

Every list in VICIdial has a status: Active or Inactive. The hopper only pulls from Active lists that are assigned to the campaign.

Go to Lists in the admin, find your lists, and verify:

  • List status is Active
  • The list is assigned to the correct Campaign
  • The list hasn’t expired (check the Expiration Date field)

We’ve seen operations run into this after a list import where the import completed successfully but the list defaulted to Inactive. You load leads, start the campaign, and wonder why nothing happens.

3. Timezone Restrictions Filtering Out Everything

VICIdial has built-in timezone-aware dialing. If your campaign has Local Call Time enforcement enabled (and it should be, for TCPA), the hopper only loads leads whose timezone + local time falls within your allowed calling window.

So if it’s 9 AM Eastern and your calling window is 9 AM – 9 PM local time, leads in Pacific timezone (where it’s 6 AM) won’t load into the hopper. This is working correctly. But if ALL your leads are in a timezone that’s currently outside the window, your hopper shows zero.

Check your campaign’s Local Call Time setting. Then check what timezones your leads are in. VICIdial assigns timezone by area code using the vicidial_phone_codes table. If your leads have malformed phone numbers or area codes that don’t map to a timezone, they can fall through the cracks.

4. Lead Filter Is Too Restrictive

Campaign lead filters are SQL WHERE clauses that get appended to the hopper query. They’re powerful, but they can also silently kill your hopper.

In Campaign Settings, check the Lead Filter field. If there’s a filter active, it might be filtering out more leads than you expect. A filter like state='CA' when your list has states stored as California will match zero leads.

Common filter mistakes:

  • Case sensitivity (state='ca' vs state='CA')
  • Field doesn’t exist or is misspelled
  • Date comparisons with wrong format
  • Filter references a field that’s empty for most leads

The fix: Test your filter by checking how many leads it actually matches. Look at the Dialable Leads Count in the campaign detail page — this number already accounts for filters, statuses, and dial count limits.

5. DNC List Is Blocking Everything

If you have an active DNC (Do Not Call) list loaded and your leads overlap heavily with that list, the hopper will skip those leads. This is correct behavior, but it can be surprising when you load a purchased list that turns out to be 80% DNC numbers.

Check your DNC settings in Campaign Settings. VICIdial supports both internal DNC lists and campaign-level DNC lists.

6. Hopper Level Is Too Low

The default hopper_level of 50 means the system tries to keep 50 leads ready in the hopper. With a predictive dialer running at an auto-dial level of 3.0 and 10 agents, you’re burning through 30 leads per minute (10 agents × 3 dials per agent). A hopper of 50 gives you less than two minutes of buffer.

If VDHopper takes 60 seconds to run (which it does on large lists), you can drain the hopper faster than it refills.

The fix: Set hopper_level proportional to your dial velocity. Rule of thumb: agents × auto_dial_level × 5. For 20 agents at dial level 3.0, that’s 300. For 50 agents, 750 or higher.

In the admin GUI, go to Campaign Settings and adjust Hopper Level.

7. Database Performance Issues

The VDHopper script runs a series of MySQL queries against vicidial_list. On large tables with poor indexes, these queries can timeout or take so long the cron can’t keep up.

Signs of database bottlenecks:

# Check for slow queries
mysqladmin processlist | grep -i hopper

# Check vicidial_list table size
mysql -e "SELECT COUNT(*) FROM vicidial_list;" asterisk

# Check if the table needs optimization
mysql -e "SHOW TABLE STATUS LIKE 'vicidial_list';" asterisk

If vicidial_list has more than 10 million rows and you’re running on hardware with less than 16 GB of RAM, the hopper query can become a bottleneck. The solution is usually proper indexing, query caching, or splitting old leads into archive tables.

8. Lead Recycling Not Configured

After your first pass through a list, most leads end up in non-dialable statuses. Without lead recycling, those leads are permanently dead to the hopper.

Lead recycling lets you automatically move leads back to a dialable status after a waiting period. For example: leads with status NA (No Answer) get recycled back to dialable after 4 hours.

Configure this in Campaign Settings → Lead Recycling. Set the statuses you want to recycle, the delay between attempts, and the maximum number of recycle attempts.

9. Duplicates Handling Blocking Leads

If you have Duplicate Check enabled at the campaign or system level, the hopper will skip leads where the phone number already exists in the hopper or has been recently dialed. This is usually what you want — nobody likes getting called twice in 10 minutes. But if your list has high duplication rates, this can significantly reduce your dialable pool.

10. Call Count Limits Hit

Two settings interact here: Dial Count Limit (how many times the system will attempt a lead total) and Call Count Limit (per-campaign cap on attempts). If either is reached, that lead is dead to the hopper.

Check both settings in Campaign Settings. A Dial Count Limit of 3 means after three attempts (regardless of disposition), that lead won’t load into the hopper again.

11. Agent Availability Affecting Hopper

In predictive mode, the dialer scales its dial velocity based on agent availability. If no agents are in READY status (all paused, all on calls, all in wrap-up), the dialer stops pulling from the hopper, which can make it appear that the hopper is the problem when it’s actually an agent staffing issue.

Check the Real-Time Report. If you see agents in PAUSED status or all on active calls, the issue isn’t the hopper — it’s that the dialer has nobody to route calls to.

12. Time-Based Campaign Restrictions

Campaigns can have start and stop times configured. Outside these windows, the hopper process skips the campaign entirely.

Check Campaign Settings → Active Time. Also check if the campaign itself is set to Active. A campaign in “Paused” state won’t get hopper loads.

Hopper Settings Deep Dive

Let’s go through every hopper-related setting in the Campaign admin page and what it actually does.

hopper_level

How many leads the system tries to maintain in the hopper. This is a target, not a hard limit. VDHopper will try to fill up to this number on each pass.

Agents Auto-Dial Level Recommended hopper_level
1-5 1.0-2.0 50-100
5-15 2.0-3.0 100-300
15-30 3.0-4.0 300-500
30-50 3.0-5.0 500-750
50+ 4.0-8.0 750-1500

lead_order

Controls the sort order of leads loaded into the hopper. Options include:

  • DOWN — Highest lead_id first (newest leads)
  • UP — Lowest lead_id first (oldest leads)
  • DOWN COUNT — Leads with fewer dial attempts first
  • UP COUNT — Leads with most dial attempts first (unusual, but useful for aggressive recycling)
  • DOWN PHONE — Descending phone number
  • UP PHONE — Ascending phone number
  • RANDOM — Random order each hopper load
  • UP LAST CALL TIME — Leads called longest ago first (good for recycling)

For most outbound campaigns, DOWN COUNT or UP LAST CALL TIME are the best choices. DOWN COUNT ensures fresh leads get dialed first while recycled leads are backfilled. UP LAST CALL TIME spreads out call attempts over time, which is better for contact rates and less likely to annoy people.

list_order_mix

When a campaign has multiple active lists, this controls how leads from different lists are mixed in the hopper. Options:

  • NONE — No mixing, lists are processed in order of list ID
  • EVEN — Equal distribution across lists
  • MULTI — Custom ratio per list (you set percentages)

For campaigns running multiple lead sources (e.g., web leads + purchased list + callbacks), use MULTI and weight toward your highest-value source.

adaptive_dropped_percentage

This affects how aggressively the dialer pulls from the hopper. If your drop rate exceeds this target percentage, the dialer slows down and pulls fewer leads. This indirectly affects hopper drain rate.

Default is 3% (the FTC safe harbor limit). Don’t change this unless you understand the TCPA implications.

Monitoring Hopper Health

Real-Time Report

The fastest way to check hopper status is the Real-Time Report in the VICIdial admin. The hopper count shows next to each campaign. Green is good. Red (or zero) means you’re about to have idle agents.

Watch for these patterns:

  • Hopper steadily at zero — One of the 12 causes above. Start debugging.
  • Hopper oscillates between 0 and hopper_level — VDHopper is running but the dialer drains it faster than it fills. Increase hopper_level or check cron frequency.
  • Hopper full but no calls going out — The problem isn’t the hopper. Check agent availability, trunk registration, and Asterisk channel availability.

Asterisk CLI

# Check active channels (are calls actually going out?)
asterisk -rx "core show channels"

# Check SIP trunk status (can we even make calls?)
asterisk -rx "sip show peers"

# Check current dial attempts
asterisk -rx "core show channels verbose" | head -30

Database Queries

Sometimes you need to look directly at the hopper table. These are read-only queries — we’re just looking, not changing anything:

# How many leads in hopper right now, by campaign
mysql -e "SELECT campaign_id, COUNT(*) FROM vicidial_hopper GROUP BY campaign_id;" asterisk

# Check hopper lead details
mysql -e "SELECT * FROM vicidial_hopper WHERE campaign_id='YOURCAMPAIGN' LIMIT 10;" asterisk

# Count dialable leads in your list (matching campaign dial statuses)
mysql -e "SELECT status, COUNT(*) FROM vicidial_list WHERE list_id='YOUR_LIST_ID' GROUP BY status;" asterisk

That last query is the one that tells you the truth. If 95% of your leads are in status SALE, DNC, or XFER, you don’t have a hopper problem — you have a “you’ve already dialed the whole list” problem.

Performance Tuning for High-Volume Operations

If you’re running 50+ agents and processing 500K+ dials per day, the standard hopper configuration won’t cut it. Here’s what production systems at that scale look like.

Database Optimization

The hopper query joins vicidial_list with vicidial_hopper, vicidial_dnc, and potentially your lead filter conditions. On large tables, this join can be brutal.

Key indexes that help:

# Check if critical indexes exist
mysql -e "SHOW INDEX FROM vicidial_list WHERE Key_name IN ('index_phone', 'status');" asterisk

If you’re running MariaDB 10.11+ (ViciBox 12.0.2 default), you get better query optimization out of the box. But if you migrated from an older system, your indexes might be suboptimal.

Splitting the Database Load

On large clusters, the standard approach is to run the hopper cron on a dedicated database server or read replica. The VDHopper script hammers vicidial_list with SELECT queries, and on the same box that’s handling agent phone queries and CDR writes, it competes for I/O.

VICIdial supports multi-server setups where the database runs on its own hardware. If your hopper is slow because of database contention, this is the real fix.

Hopper Pre-warming

Before a shift starts, you want the hopper full and ready. Some operations run the VDHopper script manually before agents log in:

# Manually trigger hopper fill
/usr/share/astguiclient/AST_VDhopper.pl --force-fill

This preloads the hopper so there’s no initial lag when agents start taking calls.

Common Mistakes and Misconceptions

“My hopper should always be at hopper_level” — Not necessarily. The hopper drains between cron runs. If your dial velocity exceeds refill rate, the hopper will oscillate. This is normal as long as it doesn’t stay at zero.

“I’ll just set hopper_level to 10,000” — Don’t. A huge hopper_level means the VDHopper script takes longer to run because it’s loading more leads. It also means your lead_order is less meaningful because you’re pre-loading leads that won’t be dialed for hours. Keep it proportional to your actual dial velocity.

“The hopper is empty so I need more leads” — Maybe. But nine times out of ten, you have plenty of leads. They’re just in non-dialable statuses, filtered out, or in the wrong timezone. Check your Dialable Leads Count first.

“I’ll just restart Asterisk” — Restarting Asterisk drops all active calls. It doesn’t fix the hopper. The hopper is a database table populated by a Perl script. Asterisk has nothing to do with it.

Hopper Troubleshooting Flowchart

When the hopper hits zero and agents are going idle, follow this path:

Step 1: Is VDHopper cron running?

  • Check crontab -l | grep VDHopper
  • If not running → Add the cron entry

Step 2: Is the campaign active with agents in READY?

  • Check Real-Time Report
  • If campaign paused or no agents ready → Fix staffing

Step 3: Are lists assigned, active, and non-expired?

  • Admin → Lists → Check status for each list in campaign
  • If inactive or expired → Activate or extend

Step 4: Check Dialable Leads Count in Campaign Settings

  • If zero → Your leads are exhausted or all filtered out
  • If > 0 → The filter/timezone/DNC is blocking the hopper load

Step 5: Check timezone restrictions

  • Is it currently within your Local Call Time for any timezone in your lead set?
  • If leads are all Pacific and it’s 6 AM PT → Wait for the window to open

Step 6: Check lead filter

  • Remove the filter temporarily
  • Does the hopper fill? → Your filter is the problem, fix the SQL

Step 7: Check database performance

  • Look at VDHopper execution time
  • If > 45 seconds → Database bottleneck, optimize or scale

The Hopper in Multi-Server Clusters

In a VICIdial cluster (separate web, telephony, and database servers), the VDHopper cron should run on the web/admin server — the one that has direct, low-latency access to the database. Running it on a telephony server that connects to the database over the network adds latency to every hopper query.

ViciBox clusters configure this automatically. If you built your cluster manually, verify which server is running the hopper cron:

# On each server in the cluster
crontab -l | grep VDHopper

Only one server should be running VDHopper for each campaign. Running it on multiple servers doesn’t load the hopper faster — it creates race conditions where both instances try to load the same leads.

Lead Recycling and Hopper Interaction

Lead recycling is the feature that keeps your hopper alive after the first pass through a list. Without it, once every lead has been dialed once (assuming Dial Count Limit of 1), you’re done.

Here’s how recycling works with the hopper:

  1. Lead gets dialed, agent dispositions it as NA (No Answer)
  2. Lead’s status in vicidial_list is now NA
  3. If NA is in your Dial Status list AND lead recycling is configured for NA, the hopper will pick it up again after the recycling delay
  4. The delay is configurable per status — e.g., NA waits 2 hours, B (Busy) waits 30 minutes, A (Answering Machine) waits 4 hours

Configure recycling in Campaign Settings → Lead Recycling.

The common mistake: setting Dial Status to include NA but NOT configuring a recycling delay. This means NA leads get immediately re-loaded into the hopper and re-dialed. Your leads get called every few minutes, which destroys your reputation and probably violates calling regulations.

Set realistic delays:

  • NA (No Answer): 2-4 hours
  • B (Busy): 30-60 minutes
  • A (Answering Machine): 4-8 hours
  • CALLBK (Callback): Handled separately by the callback system

When the Hopper Isn’t the Problem

Sometimes you’re staring at the hopper count and it looks fine — 200, 300, 400 leads — but calls still aren’t going out. The hopper is full and the dialer just isn’t pulling from it.

Possible causes:

  • No trunks available: Your SIP carrier is down or at capacity. Check asterisk -rx "sip show peers".
  • All agents paused: The predictive dialer scales to zero if nobody is in READY status.
  • Auto-dial level at 0: Someone set the dial level to zero. Check Campaign Settings.
  • Asterisk maxchannels hit: If your Asterisk instance is at its channel limit, no new calls can originate regardless of hopper state.
  • Max trunk lines exceeded: Check your carrier’s concurrent call limit.

Wrapping Up

The dial hopper is the most misunderstood component in VICIdial. It’s simple in concept (a buffer between your lists and the dialer) but complex in practice because it’s affected by every campaign setting, every filter, every timezone rule, and the database performance underneath it all.

When your hopper goes empty, resist the urge to blame the software. 95% of the time, it’s a configuration issue that’s fixable in the admin GUI within five minutes once you know where to look.

Need someone to look for you? ViciStack provides fully managed VICIdial infrastructure with proactive hopper monitoring, auto-scaling dial levels, and 24/7 coverage. We’ve fixed this exact problem on hundreds of production systems. Our standard engagement is $5K ($1K down, $4K on completion) and the hopper issue is usually resolved in the first 48 hours along with the other optimization work. Get in touch if staring at a zero hopper is costing you revenue.

Related: VICIdial Predictive Dialer Settings | VICIdial Lead Recycling | VICIdial List Management | VICIdial Auto Dial Level Tuning

Building Reliable Agents with the Transactional Outbox Pattern and Redis Streams

Diagram 1

AI agents are pretty good at deciding what should happen next, given a well-defined business workflow. In the case of a customer support agent, for example, they can read a conversation, apply a policy, and return a response like “approve the refund” or “escalate this case.” That part is exciting, and it is usually what gets demoed first. But the hard part starts right after the decision is made.

In a real system, a decision only matters if the rest of the platform can trust it. If an agent decides a customer should get a refund, that decision still has to turn into real work across the rest of the application. The support case needs to be updated, and billing needs to issue the refund. The customer may need an email, and the CRM probably needs the updated status, too.

If the app updates the case and then crashes before billing gets the event, you now have a case that says “refund approved” and a customer who never actually got refunded. That is the kind of bug that makes a system feel flaky even when the model made the right call. But the worst part is the damage to customer experience. I would be really mad at the company if this happened to me.

For scenarios like this, the Transactional Outbox pattern exists. Instead of treating “update the case” and “tell the rest of the system” as two separate operations, we commit them together and let the rest of the platform react asynchronously afterward. This pattern became fairly famous in the context of microservices, as they often need a reliable way to hand off tasks. I think the pattern is also useful for agents, because the fundamental problem is the same.

In this post, I will discuss the Transactional Outbox pattern in the context of agents and provide an opinionated view of why I believe it is a best practice for agentic applications. I will discuss the pattern around the following question: once an agent makes a business decision, how can you ensure the rest of the system can rely on it?

The problem is the handoff

Developers often stress about designing systems that can survive unimaginable production incidents. But the reality is, you don’t need a major outage or some exotic distributed systems incident to witness the worst. Sometimes it is as simple as the need for one service to try to do two related things in two separate steps. Like, first, it updates the business state, then it publishes an event for downstream systems.

That looks harmless until something fails in between. If the state update succeeds and the event publish does not, the source of truth has moved forward, but the rest of the workflow has not.

Here is that failure in one picture:

Diagram 1

What makes this annoying is that nothing looks obviously broken at first. During an incident investigation, if someone checks the case record, it looks correct. The problem only shows up later when billing never acts, the customer complains, or support has to manually reconcile what happened. That is why I think this is not an AI problem. It is a handoff problem.

Motivation for the Transactional Outbox pattern

The Transactional Outbox pattern exists because “save state, then publish the event” is fragile by design. The pattern gives you a cleaner contract: when business state changes, the application also writes an outbox event in the same atomic operation.

That one change removes the worst failure mode. You no longer end up in the state where the case changed, but the event silently disappeared. It also keeps the request path honest. The service does not have to directly coordinate billing, notifications, CRM sync, and everything else just to be correct.

Instead, the request path only needs to guarantee one thing: the decision and the outbox event are committed together. Once that happens, everything else becomes recoverable instead of fragile.

That is why this pattern fits agentic systems so well. Agents make decisions that trigger follow-up work, but those decisions need a durable “this happened” moment before the rest of the system can safely react.

Why is “Just Retry the Publish” not enough?

Every time I promote a discussion with developers around the Transactional Outbox pattern, I often hear them saying, “Why not just retry if the publish fails?” I think that happens because implementing the pattern correctly requires certain design decisions and technology stacks. The instinct is usually to look for a simpler alternative.

For this reason, I like to stress the following: the use of retries is reasonable until you look closely at where the failure occurs. This means that retries only help if the application still knows it has something to retry. If the process crashes after the state update but before the event is durably recorded anywhere, there is nothing left to retry.

That is the key difference between retries and an outbox. Retries help you deliver an event that already exists, while the outbox ensures the event exists in the first place. Once you look at it that way, the pattern feels less like a ceremony and more like basic design principles. If the business state changes, the system needs a durable record of the event that describes that change.

Redis Streams is great for this pattern

Redis Streams are a good fit for this kind of outbox because they already behave like the commit log we want. You can append events to them, consume them in order, track what is pending, and let different consumer groups process the same stream independently. That matters because the outbox is not really a queue in the narrow sense. It is a commit log for business events.

Admittedly, the Transactional Outbox pattern is often implemented using Apache Kafka and technologies such as Debezium. That is where the pattern became most notorious. I helped many developers implement this pattern with Kafka, and it works great for getting things done. However, because I have tons of experience with Kafka, I can say that the implementation effort can sometimes exceed the main problem they were trying to solve due to Kafka’s inherent complexity. You spend more time dealing with Kafka than the actual problem.

Redis Streams, on the other hand, makes that pretty natural. A single event can be appended once and then processed independently by several downstream concerns. The other reason Streams fit well is that they sit comfortably inside Redis. If your support case state also lives in Redis, the state change and the outbox append can share one commit boundary.

That part is important. The pattern is strongest when the business state and the outbox live in the same datastore, because that gives you a single atomic write instead of a dual-write problem wearing different clothes. With Kafka, you would need to handle two different distributed systems: the commit log itself and the data store where the update must occur.

Diving deep into the architecture

For this example, the support case state and the outbox both live in Redis. The current case state is stored in a hash, and the outbox is stored in a Redis Stream.

A case key might look like support:{tenant-acme}:case:case-123, while the outbox stream might be support:{tenant-acme}:outbox. The use of hash tags here is important because you must be intentional about where the data will be stored in Redis. During development, you may work with a single server which is the equivalent of a single shard. The data will naturally live in the same place. However, in production, you may have a clustered Redis environment with multiple shards.

The shared hash tag keeps both keys in the same slot in clustered Redis, which is what lets them participate in the same transaction. That gives us a clean split of responsibilities. The case record tells us what is true now, and the outbox stream tells us what happened and what the rest of the platform still needs to process. Yes, a relatively simple use of key prefixes could make this entire implementation useless if not carefully thought out.

From there, downstream concerns consume the stream through their own consumer groups. Billing can issue the refund, notifications can contact the customer, and CRM sync can update external systems, all without forcing the support service to orchestrate them directly in the request path.

That flow looks like this:

Diagram 2

The thing I like most about the Transactional Outbox pattern is that it keeps responsibilities clear. The support service is responsible for making the decision durable, and the rest of the platform is responsible for responding to it.

Trade-offs that are interesting to consider

The basic implementation of the pattern is simple. The design choices around it are where things get interesting. One of the first questions you must ask is where your source of truth lives. If the support case is also in Redis, the case update and the outbox append can share one transaction. If the case lives somewhere else and Redis only holds the stream, you are back in dual-write territory.

Another big choice is partitioning. It is tempting to imagine a single global outbox stream for the whole application, but that often becomes awkward in a clustered Redis setup. A per-tenant stream is often a better balance. It keeps related events together, provides useful ordering, and avoids making every transactional write depend on a single global key. It also makes querying and data retrieval a bit easier during investigation scenarios.

Consumer isolation is another trade-off that is worth saying out loud. One consumer group per downstream concern is a very nice model operationally, because billing, notifications, and CRM sync can all move at their own pace. The flip side is that you now own several background workflows. Each one has lag, retries, health, and recovery behavior to think about. This is where the world of microservices cross paths again with agentic systems. Each agent is not only a set of code and resources. They also bring operational complexities that must be owned by someone.

Retention matters too. An outbox is a log, and logs grow. If you trim too aggressively, you lose the replay window and the investigation history. If you never trim at all, the stream just keeps growing and eventually becomes an operational problem in its own right. Deciding how large the stream is allowed to grow must be a discussion that takes place before the app even goes to production. Not an afterthought.

Durability is another place where the architecture gets real fast. If the outbox carries important business decisions like refunds, escalations, or account changes, Redis is no longer “just a cache” in this design. It is part of the system’s correctness model. You must treat Redis as a single source of truth, and as such, think carefully about how to handle details like replication, failover, and geographic disasters.

Finally, there is idempotency. The outbox makes the handoff reliable, but it does not magically make downstream effects exactly-once behavior in the business sense. If a worker crashes after reading but before acknowledging, another worker may retry the same event later. That means the side effect needs to be safe to run more than once. The usual instinct for developers is to write the worker as a function that hooks into the stream, pulls the latest records, and processes them as if the data is simply mutable. Nope, you must treat them as immutable objects.

Okay, let’s see some code

This post is not meant to be a complete implementation reference, but I know that, as a developer, looking at code helps make understanding concrete. I will try to provide the example with fewer details so you can understand the design principles. I’m sure your coding agent can help with your actual final code. Also, I will use Java because it comes naturally to me — but feel free to ask your coding agent to translate it into another language.

Let’s start by looking for a runtime helper class that instantiates Jedis, a Redis client for Java:

import redis.clients.jedis.RedisClient;
import redis.clients.jedis.UnifiedJedis;

public final class RuntimeSupport {

    public UnifiedJedis createJedisFromEnv() {
        String redisHost = System.getenv().getOrDefault("REDIS_HOST", "localhost");
        int redisPort = Integer.parseInt(System.getenv().getOrDefault("REDIS_PORT", "6379"));

        return RedisClient.builder()
                .hostAndPort(redisHost, redisPort)
                .build();
    }
}

Next, let’s take a look at how keys and group naming are handled in a small constants class instead of scattering strings through the code.

public final class SupportConstants {
    public static final String STREAM_GROUP_START_ID = "0-0";
    public static final String BILLING_GROUP_NAME = "billing-cg";
    public static final String NOTIFICATIONS_GROUP_NAME = "notifications-cg";
    public static final String CRM_SYNC_GROUP_NAME = "crm-sync-cg";

    private SupportConstants() {}
}

For the Redis keys themselves, a small helper record keeps the slotting decision obvious:

public record SupportKeys(String caseKey, String outboxKey) {

    public static SupportKeys forCase(String tenantId, String caseId) {
        String hashTag = "{" + tenantId + "}";
        return new SupportKeys(
                "support:" + hashTag + ":case:" + caseId,
                "support:" + hashTag + ":outbox"
        );
    }
}

The core write path is where the architectural idea becomes real. When the support service accepts the agent’s decision, it updates the case state and appends a RefundApproved event in a single Redis transaction.

import redis.clients.jedis.AbstractTransaction;
import redis.clients.jedis.Response;
import redis.clients.jedis.StreamEntryID;
import redis.clients.jedis.UnifiedJedis;

import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;

public final class RefundApprovalService {
    private final UnifiedJedis jedis;

    public RefundApprovalService(UnifiedJedis jedis) {
        this.jedis = Objects.requireNonNull(jedis, "jedis must not be null");
    }

    public RefundCommitted approveRefund(RefundDecision decision) {
        SupportKeys keys = SupportKeys.forCase(decision.tenantId(), decision.caseId());

        Map<String, String> caseFields = new LinkedHashMap<>();
        caseFields.put("case_id", decision.caseId());
        caseFields.put("customer_id", decision.customerId());
        caseFields.put("refund_id", decision.refundId());
        caseFields.put("status", "refund_approved");
        caseFields.put("decision_source", "support-agent");
        caseFields.put("updated_at", decision.decidedAt().toString());

        Map<String, String> outboxFields = new LinkedHashMap<>();
        outboxFields.put("event_id", decision.eventId());
        outboxFields.put("event_type", "RefundApproved");
        outboxFields.put("case_id", decision.caseId());
        outboxFields.put("customer_id", decision.customerId());
        outboxFields.put("refund_id", decision.refundId());
        outboxFields.put("decision_source", "support-agent");
        outboxFields.put("occurred_at", decision.decidedAt().toString());

        try (AbstractTransaction redisTx = jedis.multi()) {
            redisTx.hset(keys.caseKey(), caseFields);
            Response<StreamEntryID> streamEntryId =
                    redisTx.xadd(keys.outboxKey(), StreamEntryID.NEW_ENTRY, outboxFields);

            List<Object> execResults = redisTx.exec();
            if (execResults == null) {
                throw new IllegalStateException("Refund approval transaction aborted");
            }

            return new RefundCommitted(
                    decision.caseId(),
                    decision.eventId(),
                    streamEntryId.get().toString()
            );
        }
    }

    public record RefundDecision(
            String tenantId,
            String caseId,
            String customerId,
            String refundId,
            String eventId,
            Instant decidedAt
    ) {
        public static RefundDecision create(
                String tenantId,
                String caseId,
                String customerId,
                String refundId
        ) {
            return new RefundDecision(
                    tenantId,
                    caseId,
                    customerId,
                    refundId,
                    UUID.randomUUID().toString(),
                    Instant.now()
            );
        }
    }

    public record RefundCommitted(
            String caseId,
            String eventId,
            String streamEntryId
    ) {}
}

This one method is the whole architectural point made concrete. If the transaction does not complete, neither the case update nor the outbox event exists. If it does complete, both exist. That is the durability boundary that the rest of the workflow can rely on.

Here is the same moment as a diagram:

Diagram 3

On the consumer side, the worker will act on the message written to the stream.

import redis.clients.jedis.StreamEntryID;
import redis.clients.jedis.UnifiedJedis;
import redis.clients.jedis.exceptions.JedisDataException;
import redis.clients.jedis.params.XReadGroupParams;
import redis.clients.jedis.resps.StreamEntry;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Objects;

import static redis.clients.jedis.StreamEntryID.XREADGROUP_UNDELIVERED_ENTRY;

public final class BillingConsumer {
    private static final StreamEntryID PENDING_ID =
            new StreamEntryID(SupportConstants.STREAM_GROUP_START_ID);
    private static final StreamEntryID NEW_ENTRY_ID = XREADGROUP_UNDELIVERED_ENTRY;

    private final UnifiedJedis jedis;
    private final BillingGateway billingGateway;
    private final String consumerName;

    public BillingConsumer(
            UnifiedJedis jedis,
            BillingGateway billingGateway,
            String consumerName
    ) {
        this.jedis = Objects.requireNonNull(jedis, "jedis must not be null");
        this.billingGateway = Objects.requireNonNull(billingGateway, "billingGateway must not be null");
        this.consumerName = Objects.requireNonNull(consumerName, "consumerName must not be null");
    }

    public void run(String tenantId) throws InterruptedException {
        String outboxKey = SupportKeys.forCase(tenantId, "unused").outboxKey();
        createConsumerGroup(outboxKey);

        while (!Thread.currentThread().isInterrupted()) {
            List<StreamMessage> pendingEntries = readGroup(outboxKey, PENDING_ID, 10);
            if (!pendingEntries.isEmpty()) {
                processEntries(outboxKey, pendingEntries);
                continue;
            }

            List<StreamMessage> newEntries = readGroup(outboxKey, NEW_ENTRY_ID, 10);
            if (!newEntries.isEmpty()) {
                processEntries(outboxKey, newEntries);
            } else {
                Thread.sleep(200L);
            }
        }
    }

    private void createConsumerGroup(String outboxKey) {
        try {
            jedis.xgroupCreate(
                    outboxKey,
                    SupportConstants.BILLING_GROUP_NAME,
                    new StreamEntryID(SupportConstants.STREAM_GROUP_START_ID),
                    true
            );
        } catch (JedisDataException e) {
            if (!e.getMessage().contains("BUSYGROUP")) {
                throw e;
            }
        }
    }

    private List<StreamMessage> readGroup(String outboxKey, StreamEntryID streamEntryID, int count) {
        XReadGroupParams params = XReadGroupParams.xReadGroupParams().count(count);

        List<Map.Entry<String, List<StreamEntry>>> rawEntries = jedis.xreadGroup(
                SupportConstants.BILLING_GROUP_NAME,
                consumerName,
                params,
                Map.of(outboxKey, streamEntryID)
        );

        return parseEntries(rawEntries);
    }

    private void processEntries(String outboxKey, List<StreamMessage> entries) {
        for (StreamMessage entry : entries) {
            if (!"RefundApproved".equals(entry.fields().get("event_type"))) {
                jedis.xack(
                        outboxKey,
                        SupportConstants.BILLING_GROUP_NAME,
                        new StreamEntryID(entry.id())
                );
                continue;
            }

            billingGateway.issueRefund(
                    entry.fields().get("refund_id"),
                    entry.fields().get("customer_id"),
                    entry.fields().get("event_id")
            );

            jedis.xack(
                    outboxKey,
                    SupportConstants.BILLING_GROUP_NAME,
                    new StreamEntryID(entry.id())
            );
        }
    }

    private static List<StreamMessage> parseEntries(List<Map.Entry<String, List<StreamEntry>>> rawEntries) {
        if (rawEntries == null || rawEntries.isEmpty()) {
            return Collections.emptyList();
        }

        List<StreamMessage> entries = new ArrayList<>();
        for (Map.Entry<String, List<StreamEntry>> streamData : rawEntries) {
            for (StreamEntry streamEntry : streamData.getValue()) {
                entries.add(new StreamMessage(
                        streamEntry.getID().toString(),
                        streamEntry.getFields()
                ));
            }
        }

        return entries;
    }

    private record StreamMessage(String id, Map<String, String> fields) {}

    public interface BillingGateway {
        void issueRefund(String refundId, String customerId, String idempotencyKey);
    }
}

Closing

What I like most about the Transactional Outbox pattern is that it respects the actual shape of agentic systems. Agents are good at deciding what should happen next given a flow, but the platform is still responsible for turning that decision into a durable state and letting the rest of the workflow react safely. The pattern gives you a clean handoff for that.

Redis Streams make it practical when your application state and the outbox both live in Redis. That doesn’t make the design free of trade-offs. You still need to think about partitioning, retention, durability, lag, and idempotency. It just prevents you from thinking about dual-write problems. It gives you a system where an agent’s decision becomes a durable fact before the rest of the platform starts depending on it.

By applying the Transactional Outbox pattern in your agents, you can be the difference between an agent that looks clever in a demo and a system you can actually trust.

Your AI agents are a security nightmare

Amazon CloudTrail logs showing a unique identifier for actions taken

I’ve noticed something about the tech industry over the past couple of months. Through the use of agentic code editors, developers are building small, custom solutions to problems they’re having. Recently, there’s been a surge in developers writing their own content platforms, abandoning bigger platforms like Hashnode and Medium. While I think this is genuinely amazing, we’re creating a gap that we have to address sooner rather than later.

That gap, of course, is maintenance. Even though AI coding is genuinely good now, new development is only a fraction of the software lifecycle. What happens when you get your app to prod and it suddenly breaks?

Do you know the codebase like you did back when everything was hand coded? Would you even know where to begin troubleshooting if your API started returning a bunch of 500s?

Let’s assume the answer is yes to both of those. Now let me ask the hard-hitting one: do you have time to troubleshoot and fix issues? This is a side project, not your day job. Many of us don’t have the luxury of dropping the “real work” tasks for a few hours while we figure out what’s going on.

I’m no exception. I’ve been building Good Roots Network, an app that connects local gardeners with people in their communities who need food. Growers list produce, gatherers like social workers and non-profits request it, and the system handles the coordination from availability through pickup. It’s API-driven from the ground up. State machines, role-based access, real-time inventory transitions. Real production infrastructure running in my personal AWS account.

There’s a lot of surface area to cover for a solo project. And when something goes wrong, I don’t have the bandwidth to dig in and figure out how to get it back on its feet – which means I’m potentially blocking people from getting access to fresh produce for themselves or others.

So I built an agent to be on call.

Before I continue, thank you to Teleport for sponsoring this post. All opinions are my own.

Building an agent that runs locally

The oncall-agent is a local, persistent process that automatically responds to production incidents. It subscribes to a Momento topic for real-time notifications and when something fires, it gets to work on its own. No need to wait for me to react.

The agent has an internal state machine: receive the incident signal, authenticate against AWS, investigate, then post a summary to Slack. During investigation, the agent loop runs on Amazon Bedrock, using the AWS CLI to query CloudWatch logs and metrics, inspect Lambda functions and DynamoDB tables. It also has access to my source code and deployments via a GitHub app to diagnose root cause and open a PR with the recommended code change.

I run it locally (rather than as a deployed service) for a couple of reasons. First, I specifically want it to make code changes if they’re needed, and pulling the source into a Lambda function in my account doesn’t seem like a great use of resources. Second, and more importantly, it touches production infrastructure directly. I want full control over the process, and I want the blast radius of any security issue limited to my machine, not a cloud-hosted process that’s running 24/7.

By the time I check Slack in the morning, the investigation and proposed solution is done and awaiting my approval.

Keeping the agent secure for a change

It’s tempting to just open up the doors on your machine and give the agent everything it needs to be successful (and then some). But as we’ve seen throughout 2026, doing so is a security nightmare. I don’t want to be another statistic who let an agent wreak havoc on my system because it was easy.

When it came time to give the agent AWS access, my first instinct was to put an access key in the .env file. Long-lived credentials. It’s the familiar pattern, it takes 30 seconds, and it works. But the more I sat with that approach, the more it felt like I was solving the wrong problem.

My issue with that approach was that the agent didn’t really exist as an identity in my system. It was just a privileged background process acting on my behalf with standing access.

So, I reconsidered the agent as a first-class principal. If it was going to investigate production issues autonomously, it needed to prove who it was at runtime, operate within a scoped session, and leave behind a traceable chain of actions.

Once I started thinking about the design this way, long-lived keys stopped making sense entirely. 👎

Identity is not the same thing as access

With this in mind, I needed something more than a vault. I needed a way for the agent to establish its identity at runtime.

That’s where Teleport fits into the design. Instead of access tokens being defined in a config file, the agent starts every investigation by authenticating into a new session. From there, it can request scoped AWS access and perform specific actions with clear attribution.

What I can now see in CloudTrail is a stable GUID that uniquely identifies the agent as its own principal. I can filter to that identity and see everything the agent has ever touched.

Amazon CloudTrail logs showing a unique identifier for actions taken

If something unexpected shows up, I know immediately it was the agent, and not a human or another process. That’s already a huge improvement over a shared access key where attribution is basically impossible.

Now, I have a legitimate session-bound identity operating on my system. Something that only has the permissions I gave it, but more importantly, is independently auditable.

When the agent runs tsh login, I set it up so I have to approve the MFA challenge myself. Fully autonomous credential rotation via Teleport Machine ID is available–and it’s the natural next step–but I’m not there yet. I want a few more practice reps with the agent before I remove myself from the auth flow entirely. Trust has to be earned, and I’d rather manually approve an MFA prompt than wake up to an agent that went sideways with no human checkpoint anywhere in the chain.

Agent security posture is something I’m really honing in on. I’m changing my mental approach from “does this agent have the right key?” to “was this agent authorized to do this specific thing, in this specific context, at this specific moment?” It might feel like a subtle difference, but it’s how we need to be thinking before we give agents the keys to our production systems. And a static credential sitting in an env file isn’t the way to answer it.

This isn’t a side project problem

The most common pattern I see these days is the same one I almost fell into: drop a long-lived token into the environment, grant it the permissions it needs, and ship it. Even if you do it as a “temporary stop gap” and swear you’ll come back to fix it, you won’t.

It always feels like it’s fine until something unexpected happens. The problem with unexpected behavior in autonomous systems is that you need more audit resolution, not less. Imagine an agent made unusual calls to your production services. Was it a bug? A prompt injection? A legitimate investigation path you just didn’t anticipate? With static credentials you can’t tell. The trail goes cold at “this IAM principal made these calls.”

Agentic AI is exposing some serious issues with the access model many of us are running. IAM was originally designed around humans doing episodic work. Even automated pipelines are episodic in that they run, finish, and stop. But that doesn’t map cleanly to processes that persist indefinitely, reason autonomously, and act faster than the blink of an eye.

When I think about what agents need to operate safely, I land on a few things. They need a cryptographic identity of their own, meaning something that can be uniquely identified (much like an individual developer). They need access that’s scoped and enforced at runtime–not granted once like an API key, and left open. They need some way to detect when they’re behaving outside expected parameters, or when the operating context has been tampered with. And if they’re coordinating with other agents or MCP servers, those connections need to be governed, too (you can’t have a secured agent talking to an unsecured tool).

These are all things Teleport is building with their Agentic Identity Framework. My agent only scratches the surface of all this with its short-lived credentials and isolated identity. But starting there forces you to think about the agent as a principal that needs its own security model, which is the right mental shift we need heading into mid-2026.

This probably sounds a little familiar

We’ve all seen this play out before in different forms. When containerization became mainstream, teams that had good practices with VM security often skipped the basics on containers because it felt like a “smaller” problem. When Lambda came out, teams that carefully managed server permissions dropped service roles for wildcard policies because it was a new execution model and they were moving fast (hopefully I’m not the only one who did this 😬).

Agentic AI is the same thing happening again. A new execution model, a lot of excitement, and a temptation to skip the identity and access fundamentals–because it feels like a side project, a POC, or just something you’re trying out.

The lesson is the same every time: the important components in production are still important even if the thing running is new, or small, or experimental. An agent with standing access and no attribution is a liability, whether it’s running inside a Fortune 500 or on your laptop handling incidents for a community gardening app.

The oncall-agent repo is open source. The Teleport integration is in the state machine if you want to see how it fits together in practice. And if you’re experimenting with agents at all, it’s worth triple-checking how they establish identity and access production systems. Move a small workflow to session-bound credentials to help you make better architectural decisions and make your automations easier to trust.

Tools like Teleport make that change practical without completely reworking your stack. It’s a small change in implementation, but a big change in how much autonomy you’re willing to give your systems.

Happy coding!

Qdrant Has a Free Vector Database Optimized for AI and Semantic Search

Qdrant is a vector similarity search engine built in Rust. It stores and searches high-dimensional vectors for AI applications like RAG, recommendations, and semantic search.

What You Get for Free

  • HNSW indexing — fast approximate nearest neighbor search
  • Filtering — combine vector search with payload filters
  • Quantization — reduce memory usage by 4x
  • Distributed mode — horizontal scaling
  • REST + gRPC APIs — multiple client libraries
  • Snapshots — backup and restore
  • Multi-tenancy — tenant isolation via payload

Quick Start

docker run -p 6333:6333 qdrant/qdrant

Store and Search Vectors (Python)

from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct

client = QdrantClient('localhost', port=6333)

client.create_collection('docs', vectors_config=VectorParams(size=384, distance=Distance.COSINE))

client.upsert('docs', points=[
    PointStruct(id=1, vector=[0.1]*384, payload={'text': 'Hello world'}),
])

results = client.search('docs', query_vector=[0.1]*384, limit=5)

Qdrant vs Pinecone

Feature Qdrant Pinecone
Price Free (OSS) Freemium
Hosting Self-hosted + cloud Cloud only
Filtering Rich payload filters Metadata filters
Performance Rust (fast) Managed

Need vector search setup? Check my work on GitHub or email [email protected] for consulting.