I lost hours debugging a Next.js 16 caching issue that had no error, no warning, and only showed up in production.

The Next.js 16 caching model is genuinely good. But it introduces a class of bugs that are harder to detect than anything in previous versions: bugs that look correct, compile without errors, deploy successfully, and then silently misbehave in production.

These are the most common ones I’ve seen across real projects. Every one comes from real production incidents.

(Assumes you have cacheComponents: true enabled in next.config.ts.)

This is a follow-up to my previous post where I built a dev-only debugger to surface these issues during development. That tool helps you detect them. This post breaks down the exact failure cases behind those warnings.

Bug 1: 'use cache' on the Wrapper Instead of Inside the Function

// This looks cached. It is not.
export const getProducts = someWrapper(async function() {
  'use cache'
  cacheLife('hours')
  return db.query('SELECT * FROM products')
})

The 'use cache' directive tells the Next.js compiler to treat that function as a cache boundary. When you wrap it, the compiler sees the wrapper as the entry point. The inner function may be cached, but the wrapper becomes the execution boundary, so you still end up running it on every request.

No error. No warning. Just a function running on every request when it should be cached.

Fix:

async function _getProducts() {
  'use cache'
  cacheLife('hours')
  cacheTag('products')
  return db.query('SELECT * FROM products')
}

// Wrapper receives the already-cached function
export const getProducts = someWrapper(_getProducts)

The rule is simple: 'use cache' lives inside the data function, never on anything that wraps it.

Bug 2: Deprecated revalidateTag That Compiles and Uses Legacy Behavior

// Next.js 15: correct
// Next.js 16: TypeScript error, silently uses legacy behavior in loose tsconfig
revalidateTag('products')

In Next.js 16, revalidateTag without a second argument is deprecated and produces a TypeScript error. But if your tsconfig is not in strict mode (common in older projects), it compiles cleanly and falls back to legacy invalidation behavior instead of the new SWR-based system.

Pages stop reflecting mutations. No error anywhere.

Fix:

revalidateTag('products', 'max')         // SWR, recommended for most content
revalidateTag('products', { expire: 0 }) // Immediate expiry for webhooks/payments

Run npx @next/codemod@canary upgrade latest during your migration, it handles this automatically. But check your tsconfig strictness regardless.

Bug 3: Tag String Mismatch Across Files

// lib/data.ts -- written by developer A
async function getProducts() {
  'use cache'
  cacheTag('product-list')  // Tag is 'product-list'
  return db.query('...')
}

// app/actions/products.ts -- written by developer B
export async function createProduct(data: ProductData) {
  await db.query('INSERT INTO products ...', [...])
  revalidateTag('products', 'max')  // Different string, invalidates nothing
}

Two different strings. Zero errors. The product list never refreshes after a new product is created. Users see stale data until cacheLife expires on its own.

Fix:

// lib/tags.ts -- one source of truth
export const tags = {
  products: 'product-list',
  product: (id: string) => `product-${id}`,
  user: (id: string) => `user-${id}`,
} as const

// Both files import from tags
cacheTag(tags.products)
revalidateTag(tags.products, 'max')

Tag typos become TypeScript errors. The string mismatch bug becomes structurally impossible.

Bug 4: Server Action Mutation Where the Acting User Sees Stale Data

'use server'
export async function updateProductPrice(id: string, newPrice: number) {
  await db.query('UPDATE products SET price = $1 WHERE id = $2', [newPrice, id])
  revalidateTag(`product-${id}`, 'max')
}

revalidateTag with any named profile uses stale-while-revalidate. It marks the cache as stale. The next request still gets the cached version while fresh data loads in the background.

For the admin who just clicked save, that means they navigate to the product page and see the old price. Looks like the save failed. Causes confusion and duplicate mutations.

Fix:

'use server'
import { revalidateTag, updateTag } from 'next/cache'

export async function updateProductPrice(id: string, newPrice: number) {
  await db.query('UPDATE products SET price = $1 WHERE id = $2', [newPrice, id])
  updateTag(`product-${id}`)            // Acting user sees change immediately
  revalidateTag(`product-${id}`, 'max') // Everyone else gets SWR
  revalidateTag('products', 'max')      // Product list also refreshes
}

updateTag expires the cache entry immediately. The next request waits for fresh data. The admin sees their change. Everyone else gets the fast SWR treatment.

Constraint: updateTag only works inside Server Actions. In Route Handlers, use revalidateTag(tag, { expire: 0 }) instead.

Bug 5: updateTag in a Route Handler

// app/api/webhooks/stripe/route.ts
import { updateTag } from 'next/cache'

export async function POST(req: Request) {
  const event = await parseStripeWebhook(req)
  if (event.type === 'price.updated') {
    updateTag('products')  // Throws at runtime
  }
  return new Response('ok', { status: 200 })
}

This compiles. It deploys. On the first real webhook call from Stripe, it throws at runtime. updateTag only works inside Server Actions. Calling it anywhere else throws.

Fix:

import { revalidateTag } from 'next/cache'

export async function POST(req: Request) {
  const event = await parseStripeWebhook(req)
  if (event.type === 'price.updated') {
    revalidateTag('products', { expire: 0 })  // Immediate expiry in Route Handlers
  }
  return new Response('ok', { status: 200 })
}

Bug 6: Short cacheLife That Silently Affects PPR

async function LiveStockStatus({ productId }: { productId: string }) {
  'use cache'
  cacheLife('seconds')  // Seems right for live stock data
  cacheTag(`stock-${productId}`)
  return fetchStockLevel(productId)
}

cacheLife('seconds'), revalidate: 0, and expire under 5 minutes are automatically excluded from the PPR static shell. They become dynamic holes that run at request time.

One component with cacheLife('seconds') can push parts of the page out of the static shell and turn them into request-time work. No warning. The page still works. It just becomes fully dynamic without any obvious signal.

Fix, if the data can tolerate a short delay:

cacheLife('minutes')  // Now included in the static shell

Fix, if it genuinely needs to be live:

// Parent page
<Suspense fallback={<StockSkeleton />}>
  <LiveStockStatus productId={id} />  {/* Streams in after static shell */}
</Suspense>

The second approach is the correct PPR pattern for truly dynamic data. The static shell renders instantly and the live data streams in after.

Bug 7: Runtime API Inside a Cached Scope

async function UserHeader() {
  'use cache'
  const cookieStore = await cookies()  // Throws at build time
  const user = await getUser(cookieStore.get('user-id')?.value)
  return <div>{user.name}</div>
}

cookies(), headers(), and draftMode() are runtime APIs. They read request-specific data. They cannot live inside a 'use cache' scope because cached output is stored and replayed across requests.

This one at least throws at build time with “Uncached data was accessed outside of Suspense”. But the error gives you no component name, no file path, and no useful stack trace. You get to play binary search across your codebase to find it.

Fix:

// Read runtime values OUTSIDE the cached scope
async function UserHeader() {
  const cookieStore = await cookies()
  const userId = cookieStore.get('user-id')?.value
  return <CachedUserProfile userId={userId} />
}

// Pass the VALUE as a serializable prop to the cached component
async function CachedUserProfile({ userId }: { userId?: string }) {
  'use cache'
  cacheLife('hours')
  cacheTag(`user-${userId}`)
  if (!userId) return <GuestGreeting />
  const user = await getUser(userId)
  return <div>{user.name}</div>
}

userId is a string so it becomes part of the cache key automatically. Different users produce different cache entries without any manual key construction.

The Common Thread

None of these have good error messages. Five of the seven compile and deploy without complaint. The other two throw, but either without enough information to find the cause quickly or only after the first real production request.

The pattern across all of them is the same: the new caching model requires explicit correctness. When you get something wrong, it does not always tell you.

If you are in the middle of a Next.js 16 migration and want to catch these during development rather than in production, I ended up building a free dev-only debugger that logs cache misses, dynamic holes, missing tags, and deprecated invalidation calls directly in your terminal. Zero production cost, one .tsx file: Next.js cache debugger.

And if you want these patterns enforced at the type level so the wrong call is a compile error rather than a runtime surprise, the production enforcement layer is Cache Pro Kit.

Have you run into any of these? Or something even stranger? I’m curious what the distribution looks like across different kinds of projects.

Ask any LLM “is NVDA bullish here?” and it will answer with total confidence — a cup-and-handle here, a win rate there, an “expect a 4% move.” The problem: most of it is invented. Language models are fluent in the vocabulary of technical analysis but have no grounded source for the numbers. They’ll give you a win rate they made up.

If you’re building an agent that touches markets, that’s a real liability. The fix isn’t a better prompt — it’s a grounded tool the agent can call instead of guessing.

The idea: a base-rate engine, exposed over MCP

Chart Library is an MCP server that answers one question with real history: “given a setup that looks like this, what did statistically similar setups do next?” It returns the cohort of ~40–50 historical analogs across 10 years and 19K+ symbols, plus the forward-return distribution (median, win rate, percentile bands) — every number a verified historical fact, not a generation.

Because it speaks the Model Context Protocol, any MCP-aware client — Claude Desktop, Cursor, Cline, LangChain, the OpenAI Agents SDK — can use it in a couple of lines.

30 seconds to grounded answers

pip install chartlibrary-mcp

LangChain / LangGraph:

from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
import os, asyncio

async def main():
    client = MultiServerMCPClient({
        "chartlibrary": {
            "command": "chartlibrary-mcp",
            "transport": "stdio",
            "env": {"CHART_LIBRARY_API_KEY": os.environ["CHART_LIBRARY_API_KEY"]},
        }
    })
    agent = create_react_agent(ChatAnthropic(model="claude-sonnet-4-6"),
                               await client.get_tools())
    out = await agent.ainvoke({"messages": [{"role": "user",
        "content": "What did setups like NVDA on 2024-08-05 do next? Cite the 5-day distribution."}]})
    print(out["messages"][-1].content)

asyncio.run(main())

The agent calls the cohort tool, gets the real distribution, and answers from it. No hallucinated stats.

The honest part (which is the whole point)

Here’s something most “AI alpha” tools won’t tell you, validated on 300K+ historical observations: chart shape predicts the size of the next move far better than the direction. Conditioning on a “bullish-looking” cohort earns a real magnitude edge (bigger average moves) but no win-rate edge — direction stays a coin flip. Our backtest shows it cleanly: a “cohort bullish” rule prints +0.77% avg vs +0.34% baseline, but 52% wins vs 54% baseline.

That’s not a bug — it’s the truth, and it’s why grounding matters. A tool that returns calibrated base rates (and tells you when it doesn’t know) is more useful to an agent than one that confidently predicts a direction it can’t.

What the agent gets

Eight read-only tools (all annotated readOnlyHint): search, cohort (the distribution primitive), discover, analyze, context, narrative, explain, portfolio. Free Sandbox tier (200 calls/day, no auth) covers search + context + explain; the full cohort depth is a paid tier.

Try it

• Install: pip install chartlibrary-mcp
• Docs + setup for every client: https://chartlibrary.io/guides/mcp-server-for-finance
• It’s in the official MCP Registry (io.github.grahammccain/chart-library)

If you’re building anything that reasons about markets, give your agent a source of truth instead of a confident guess.

The Problem We Were Actually Solving

As an engineer, my primary focus was ensuring our store could handle any data we might collect, considering our restrictions. Traditional e-commerce platforms were out, so I built our store directly on top of IPFS. The IPFS advantage of decentralized data storage allowed us to work around the usual payment issues. With a monorepo setup, I could handle all frontend and server-side logic in a single repository. The idea behind this decision was to simplify the development process and reduce the load on our servers, which are usually the bottleneck in complex applications.

What We Tried First (And Why It Failed)

Initially, I used a Next.js monorepo setup, with a clear separation of frontend and server logic. While this seemed like a good idea, it led to a significant increase in our build times. As we started to onboard new team members, it became apparent that managing the build process was almost as difficult as writing the application itself. The monorepo was meant to simplify things, but we quickly found ourselves wasting hours on merging code and debugging build-related issues.

The Architecture Decision

To get around these issues, I switched to a modular, multi-repo setup with a separate repository for our server logic. This allowed us to easily manage separate build processes for our frontend and backend code. We then used a third-party service for our IPFS data storage, which significantly reduced our development time and made it easier for the team to manage our codebase. In addition, we replaced the complex setup of our Next.js monorepo with a more straightforward React setup, which allowed our developers to focus on the logic of the application rather than dealing with an overly complex build process.

What The Numbers Said After

After implementing our new architecture, our build times decreased by 30%, and our code quality improved significantly. Our team spent less time on debugging and more time on actually writing code that mattered. The number of pull requests merged daily also went up, signaling that our team had made a crucial step forward in productivity.

What I Would Do Differently

If I had to do it all over again, I’d implement the modular setup from the start. It’s clear now that this approach would have saved us time and resources in the long run. I’d also consider using a more robust tooling setup to manage our monorepo, like Yarn Workspaces or Nx. In hindsight, making the switch to a multi-repo setup might have been overkill, but the benefits we achieved were worth the investment.

Moving Beyond the Black Box: How I Built a Real-Time Voice Fitness Coach using Next.js 15, Convex, & Vapi.ai

I’ve used a lot of fitness apps. I’ve followed their calorie numbers, done their workout splits, and trusted their recommendations – all without ever understanding where any of it came from. The number just appeared, authoritative and unexplained, and I was supposed to trust it.

Eventually, I stopped. Not because the numbers were wrong. Because I had no reason to believe they were right.

That frustration became FitExplain – a full-stack web application that generates personalized fitness and nutrition plans through a voice conversation, and then shows you exactly how every number was calculated. No black box. No mystery algorithm. Just metabolic science, made visible.

This post is a complete technical breakdown of how I built it.

🧠 The Core Problem I Was Solving

Most fitness apps share a quiet design flaw: they treat the user like they can’t handle the truth.

You open the app, enter your weight and height, tap through a few goal screens, and get back a daily calorie target and a workout plan. The numbers look precise. They come with decimal points. But if you ask why 1,940 calories – why not 1,800, why not 2,100 – the app has no answer. It never does.

This isn’t accidental. It’s a design assumption: that authority is enough, that users will follow a number simply because the app said so.

Behavioral research consistently shows this assumption is wrong. People follow health guidance longer and more faithfully when they understand the reasoning behind it. Unexplained AI output, however accurate, produces shallow trust – and shallow trust doesn’t survive the friction of real life.

FitExplain was built around the opposite assumption: show your work.

🏗️ Architecture Overview: The Triple-Cloud Handshake

Before diving into each piece, here’s how the system fits together. I call it the Triple-Cloud Handshake:

User's Browser
    │
    ├──── speaks to ──────► Vapi.ai (Voice AI)
    │                           │
    │                           └── webhook ──► Convex Backend
    │                                               │
    ├──── subscribes to ─────────────────────────► Convex (Reactive DB)
    │                                               │
    └──── authenticated by ──► Clerk ──webhook──►  Convex

Four independent cloud services, each best-in-class for its specific role:

The browser talks to Vapi for voice and to Convex for data. Clerk and Vapi each communicate with Convex via authenticated server-side webhooks — invisible to the user, but the backbone of the system.

🎙️ Layer 1: Vapi.ai — Voice as the Intake Form

The first question I had to answer was: why voice at all?

Most fitness apps use forms. Forms are fine — until you realize what they actually produce. A dropdown asking “How active are you?” with options like “Sedentary,” “Lightly Active,” and “Moderately Active” doesn’t capture useful data. Users don’t know what “moderately active” means in caloric terms, so they pick whatever sounds right and end up with a generic output.

Voice is different. A conversation can ask follow-up questions. It can confirm ambiguous answers. It can be designed to elicit specific, structured information — in a format that feels completely natural to the user.

The Vapi Workflow: 10 Nodes, 8 Parameters

I built a ten-node conversation workflow in Vapi that collects exactly eight parameters before allowing the session to close:

1. Age – integer
2. Current weight – numeric, with unit confirmation
3. Height — numeric, with unit confirmation
4. Existing injuries – free text, normalized
5. Primary fitness goal – categorized (fat loss / muscle gain / endurance / general fitness)
6. Preferred weekly training sessions – integer (1–7)
7. Current fitness level – categorized (beginner / intermediate / advanced)
8. Dietary restrictions or allergies – free text, normalized

The workflow uses guard conditions – Vapi will not advance to plan generation unless every required parameter has been captured and confirmed. If a user says something ambiguous like “I work out sometimes,” the workflow probes: “About how many days a week would you say?” The session runs between 30 and 77 seconds across all tested users.

The Race Condition I Spent an Hour Debugging

Here’s something that cost me a full debugging session: Vapi’s call.ended webhook fires before the call analysis is complete. I had built my Convex action to extract parameters from the analysis object – which wasn’t populated yet when the webhook arrived.

The fix was to listen for end-of-call-report instead of call.ended. The report event fires after analysis completes and includes the full structured output. A one-line change, but I only found it after reading the Vapi docs three times and adding console logs to every webhook handler.

// convex/http.ts - webhook handler
export const vapiWebhook = httpAction(async (ctx, request) => {
  const payload = await request.json();

  // Use end-of-call-report, NOT call.ended
  if (payload.message?.type === "end-of-call-report") {
    const analysis = payload.message.analysis;
    const structuredData = analysis?.structuredData;

    if (structuredData) {
      await ctx.runAction(internal.fitplan.generatePlan, {
        callId: payload.message.call.id,
        userData: structuredData,
      });
    }
  }

  return new Response("OK", { status: 200 });
});

⚡ Layer 2: Convex — The Reactive Backend

This was the most interesting architectural decision in the project, and the one I’d recommend most strongly to other developers.

Why Not Just Use a REST API?

The standard approach would be: Vapi webhook hits an API route → API route calls Gemini → API route writes to a database → user refreshes the page to see results.

That works. But it means the user has to do something to see their plan. They have to know to refresh. The experience has a gap in it.

Convex eliminates the gap entirely.

How Convex Reactivity Works

In Convex, queries are subscriptions. When the frontend runs a query through the Convex React SDK, it opens a persistent WebSocket connection. Convex tracks which database documents that query depends on. The moment any of those documents change — from any mutation, anywhere in the system – Convex re-executes the query and pushes the updated result to all subscribed clients.

The developer writes a query function. Convex handles subscription, invalidation, and delivery automatically.

// convex/fitplans.ts
export const getUserPlan = query({
  args: { userId: v.string() },
  handler: async (ctx, { userId }) => {
    return await ctx.db
      .query("fitPlans")
      .withIndex("by_user", (q) => q.eq("userId", userId))
      .order("desc")
      .first();
  },
});

// app/dashboard/page.tsx — React component
const plan = useQuery(api.fitplans.getUserPlan, { userId: user.id });

// 'plan' updates automatically when Convex pushes a new value.
// No polling. No useEffect with fetch. No page reload.

For FitExplain, this meant: the moment the Vapi webhook triggers a plan creation mutation in Convex, the user’s dashboard re-renders with the new plan. Latency between database write and screen update is WebSocket round-trip time plus Convex re-execution time — under 200ms combined.

The Schema: Keeping Gemini Honest

One subtle but important decision: I defined a strict Convex schema for fitness plans, then validated all Gemini output against it before writing to the database. This caught a bug that took me a while to find — Gemini was returning "2200" (string) instead of 2200 (integer) for calorie values. The Convex validator rejected it, which surfaced the issue immediately rather than letting corrupt data reach the frontend.

// convex/schema.ts
fitPlans: defineTable({
  userId: v.string(),
  callId: v.string(),
  createdAt: v.number(),
  userProfile: v.object({
    age: v.number(),
    weightKg: v.number(),
    heightCm: v.number(),
    goal: v.string(),
    fitnessLevel: v.string(),
    weeklySessionTarget: v.number(),
    injuries: v.string(),
    dietaryRestrictions: v.string(),
  }),
  metabolics: v.object({
    bmr: v.number(),         // Mifflin-St Jeor result
    tdee: v.number(),        // BMR × activity multiplier
    targetCalories: v.number(), // TDEE adjusted for goal
    proteinGrams: v.number(),
    carbGrams: v.number(),
    fatGrams: v.number(),
  }),
  workoutPlan: v.array(v.object({
    day: v.string(),
    focus: v.string(),
    exercises: v.array(v.object({
      name: v.string(),
      sets: v.number(),
      reps: v.string(),
      rest: v.string(),
    })),
  })),
  mealPlan: v.array(v.object({
    meal: v.string(),
    foods: v.array(v.string()),
    calories: v.number(),
  })),
}).index("by_user", ["userId"]),

🤖 Layer 3: Google Gemini 1.5 Flash – Structured AI Output

Most LLM integrations treat the model as a conversational partner. FitExplain treats Gemini as a structured data generator.

The distinction matters. A conversational response (“Here’s your plan! On Monday, you should do…”) cannot be stored cleanly in a database or rendered in a structured dashboard UI. Schema-compliant JSON can.

The Prompt Engineering

The system prompt does three things:

1. Instructs Gemini to calculate BMR using Mifflin-St Jeor explicitly
2. Instructs Gemini to derive TDEE using the standard activity multiplier table
3. Requires output as valid JSON matching a predefined schema – no prose, no markdown, no preamble

// convex/fitplan.ts - Gemini prompt construction
const systemPrompt = `You are a fitness and nutrition AI that generates structured plans.
You MUST respond with ONLY valid JSON. No markdown. No explanation. No preamble.

Calculate BMR using Mifflin-St Jeor:
- Men: (10 × weight_kg) + (6.25 × height_cm) - (5 × age) + 5
- Women: (10 × weight_kg) + (6.25 × height_cm) - (5 × age) - 161

Apply TDEE multiplier:
- Sedentary (1-2 days/week): BMR × 1.2
- Lightly active (3 days/week): BMR × 1.375
- Moderately active (4-5 days/week): BMR × 1.55
- Very active (6-7 days/week): BMR × 1.725

Adjust target calories for goal:
- Fat loss: TDEE - 500
- Muscle gain: TDEE + 300
- Endurance/General: TDEE

Output JSON matching this exact schema: ${JSON.stringify(planSchema)}`;

The apiVersion Bug

The Gemini Node.js SDK was returning HTTP 404 errors on every call. The fix was non-obvious: I needed to pass apiVersion: 'v1' explicitly when initializing the client. The default was pointing to a preview endpoint that wasn’t available in my region.

import { GoogleGenerativeAI } from "@google/generative-ai";

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!, {
  apiVersion: 'v1', // Required — default causes 404 in some regions
});

One line. Forty-five minutes to find it.

🔐 Layer 4: Clerk – Authentication Done Right

I chose Clerk over NextAuth for one reason: its webhook system integrates cleanly with Convex. When a user signs up, Clerk fires a user.created webhook to Convex, which creates a user document in the database. Session tokens from Clerk are verified in every Convex query and mutation, so user data is always scoped correctly.

// convex/http.ts — Clerk webhook
export const clerkWebhook = httpAction(async (ctx, request) => {
  const event = await validateClerkWebhook(request);

  if (event.type === "user.created") {
    await ctx.runMutation(internal.users.createUser, {
      clerkId: event.data.id,
      email: event.data.email_addresses[0].email_address,
      name: `${event.data.first_name} ${event.data.last_name}`,
    });
  }

  return new Response("OK", { status: 200 });
});

📊 The XAI Part: Making the Math Visible

The Explainable AI component of FitExplain isn’t a separate feature – it’s built into the dashboard display. Every number shown to the user has a derivation trace:

Your Daily Calorie Target: 1,940 kcal

How we calculated this:
├── BMR (Mifflin-St Jeor): 1,847 kcal
│   └── (10 × 75kg) + (6.25 × 178cm) - (5 × 24) + 5
├── TDEE (activity multiplier): 2,440 kcal
│   └── BMR × 1.325 (4 sessions/week = lightly-to-moderately active)
└── Fat loss adjustment: -500 kcal
    └── Standard 500kcal deficit targets ~0.5kg/week loss

This is the core design principle: explainability as engineering input, not post-hoc annotation. I didn’t generate a plan and then try to explain it afterward. I started from the explanation — the formula, the multiplier table, the adjustment logic — and used Gemini to instantiate that explanation for each individual user’s parameters.

The AI doesn’t replace the science. It applies it.

🧪 Testing Results

I ran 13 live voice sessions across users with different profiles:

The two schema validation failures were both the string/integer type mismatch on calorie values — Gemini occasionally returns numeric values as strings. I added a normalization step in the Convex action that coerces all numeric fields before validation, which resolved both failures.

🚀 The Full Tech Stack

Frontend:     Next.js 15 (App Router), TypeScript, Tailwind CSS
Deployment:   Vercel (automatic from GitHub)
Voice AI:     Vapi.ai (10-node workflow, guard-conditioned)
Database:     Convex (reactive, serverless, TypeScript-native)
AI Model:     Google Gemini 1.5 Flash (structured JSON output)
Auth:         Clerk (webhooks + session verification)

💡 What I’d Do Differently

1. Schema validation from day one. I added the Convex schema validator after encountering the string/integer bug. I should have defined it before writing any Gemini integration code. Strict schema-first development would have caught that issue in local testing rather than in a live session.

2. Vapi workflow versioning. The Vapi workflow is configured in their dashboard UI, which makes version control awkward. I’d build a script to export and commit the workflow JSON alongside the codebase from the start.

3. Gemini output temperature. I used the default temperature (1.0) throughout development. In retrospect, setting temperature to 0.3–0.5 for structured JSON generation would have produced more consistent outputs and fewer edge cases in the schema normalization layer.

🔭 What’s Next

The current system generates a plan once per voice session. The obvious next step is longitudinal tracking — storing multiple plans over time, detecting progress, and adjusting recommendations based on what’s changed. Convex’s reactive queries make this architecturally straightforward; it’s primarily a UI and prompt engineering problem.

Other directions I want to explore:

• Wearable integration – pulling actual activity data from Apple Health or Garmin rather than relying on self-reported activity levels
• Plan revision via voice – letting users update specific parameters (new injury, changed schedule) without redoing the full intake conversation
• Nutrition logging – tracking actual meals against the generated plan, with Convex reactively updating macro targets based on what’s been logged

Wrapping Up

The thing I kept thinking about while building FitExplain is that the transparency problem in fitness apps isn’t really a hard technical problem. It’s a design choice. Someone decided that showing users the formula would confuse them, or slow them down, or make the app feel less magical.

I think that’s wrong. Showing the formula is what builds real trust — the kind that survives a bad week and a missed workout, instead of evaporating the moment the user questions the number.

With cloud-native tools available today — Vapi for voice, Convex for reactive data, Gemini for structured AI generation — you can build a system that is both intelligent and transparent without trading one for the other.

That’s what FitExplain does. And it’s what I think a lot more fitness software should do.

The full project is documented in my final year report at Manipal University Jaipur. If you have questions about any specific part of the implementation — the Vapi workflow design, the Convex schema, the Gemini prompt structure — drop them in the comments.

Tags: #ai #gemini #frontend #database

The Problem Nobody Has a Tool For

Here is the scenario that wakes engineers up at 2am.

Your payment service has been running fine for months. The consumer reads events from a Kafka topic, processes them, sends confirmations. The status field is a plain String – "pending", "failed", "done".

A bug is discovered: the status field accepts invalid values like "ok" and "finished" from different producer teams, causing downstream analytics to break. The correct fix is converting status from String to a strict Enum with only four valid values: PENDING, PROCESSING, COMPLETED, FAILED.

The fix gets deployed at 9:45 PM.

Nobody checked the DLQ first.

There were 23,000 messages sitting in payments.dlq from an earlier consumer failure that afternoon. The new V2 consumer starts processing them. Each one crashes immediately:

InvalidFormatException: Cannot deserialize value of type `PaymentStatus` 
from String "pending": not one of the values accepted for Enum class: 
[PENDING, PROCESSING, COMPLETED, FAILED]

All 23,000 messages are now permanently stuck. The V2 consumer cannot read them. They cannot be redriven as-is. PagerDuty fires at 2:00 AM.

Why Schema Registry Doesn’t Help Here

The first suggestion you’ll get is “just use Schema Registry.”

Schema Registry is a seatbelt. It prevents new schema-incompatible events from being produced. It does nothing for messages already sitting in your DLQ from before the schema change.

Those messages are stored as bytes in Kafka. Schema Registry has no knowledge of them. It cannot transform them. Cannot validate them. Cannot redrive them.

The 23,000 stuck messages are entirely your problem.

What Engineers Actually Do (The Manual Reality)

I posted this exact scenario on r/apachekafka and asked how senior teams handle it. Four experienced engineers replied. Not one named a tool.

Their answers:

BroBroMate (Senior Engineer): “Something I’ve done in the past is just throw a quick Kafka Streams app up to do mass transformations… And easy to unit test before rolling out.”

KTCrisis: “You could also spin up a v2 topic for the new consumers and keep a specific v1 consumer around just to drain the DLQ. But it adds a new topic and a dedicated consumer for a one-shot issue.”

Every solution described is either:

• Building a temporary application from scratch, using it once, deleting it
• Keeping zombie infrastructure alive until the queue drains
• Writing a throwaway Python script with no tests, no audit trail, no validation

The time cost: 3-8 hours of a senior engineer’s time. At 2am. Every single incident.

The Gap No Existing Tool Fills

I looked at every existing solution:

DLQMan (irori-ab) – routes messages based on exception type headers. No transformation engine. If your messages have the wrong data format, DLQMan cannot help.

Confluent Control Center – has basic redrive. No schema transformation. Requires full Confluent Enterprise stack.

kafka-rewind-tools – handles the redrive part. You still need to write the mutation yourself.

Custom Kafka Streams apps – what everyone builds from scratch, uses once, and throws away.

The gap: no tool handles the transformation step between “message has wrong format” and “message is safe to redrive”.

Building DLQ Revive

I spent 5 weeks building the tool that should exist.

DLQ Revive is an open-source Kafka Dead Letter Queue mutation and redrive engine. Here is what it does and the architectural decisions behind each feature.

1. Browse DLQ Messages Safely

// WRONG - what most people do
consumer.subscribe(List.of("payments.dlq"));  // joins consumer group!

// CORRECT - what DLQ Revive does
consumer.assign(List.of(new TopicPartition("payments.dlq", 0)));
consumer.seek(new TopicPartition("payments.dlq", 0), fromOffset);

Using subscribe() joins your application to Kafka’s consumer group. Kafka can then trigger a rebalance and assign partitions to your read-only viewer – stealing them from your production consumer. Your 2am debugging tool accidentally takes down your production pipeline.

assign() + seek() bypasses group membership entirely. You read exactly what you want. The production consumer is untouched. DLQ Revive never calls commitSync() in view mode.

Reads are paginated at max 100 messages per API call. No full topic loads. No JVM OutOfMemoryError on topics with 500,000 stuck messages.

2. Transform Schema With JSONata

The transformation step is what makes DLQ recovery possible without writing custom code.

DLQ Revive uses JSONata – a declarative JSON-to-JSON mapping language. To fix the String to Enum problem from our scenario:

{
  "orderId": orderId,
  "amount": amount,
  "currency": currency,
  "status": $uppercase(status),
  "processedAt": $now()
}

One expression. Applies to all 23,000 messages. Preview 5 samples before committing. See the before and after side by side.

Why not Groovy? User-submitted Groovy on your backend can call Runtime.getRuntime().exec("rm -rf /"). It is an RCE vulnerability built into the product if you allow it. JSONata is purely declarative – it has no access to the file system, network, or system commands. Zero RCE surface.

3. Validate Before Redriving

BroBroMate’s advice from that Reddit thread was exactly right:

“Whatever way you do it, validate every ‘fixed’ record against an agreed good schema before producing it – there’s nothing worse than dumping another X thousand bad messages on the DLQ.”

DLQ Revive validates each transformed message against the target schema before producing. Failed validation shows per-message errors. You see exactly which messages will fail before redriving a single one.

4. Idempotency Guard – Pod-Restart Safe

-- Before producing any message:
SELECT 1 FROM redrive_log 
WHERE topic = ? AND partition = ? AND offset = ?

-- If not found, insert THEN produce:
INSERT INTO redrive_log (topic, partition, offset, redriven_at, redriven_by)
VALUES (?, ?, ?, NOW(), ?)

The redrive_log table has a UNIQUE(topic, partition, offset) constraint.

If your application crashes halfway through redriving 10,000 messages and restarts, it will not reprocess the messages it already produced. In a fintech payment pipeline, that is the difference between a successful recovery and double-charging 5,000 customers.

5. Full Audit Trail

Every browse and redrive action is logged to PostgreSQL with timestamp, user, session ID, message count, and source/target topics. For teams in regulated industries (PCI-DSS, SOC 2), this is the compliance evidence that a manual throwaway script can never provide.

Quick Start

git clone https://github.com/Saifulhuq01/dlq-revive.git
cd dlq-revive
docker compose -f docker/docker-compose.yml up -d
# Open http://localhost:4200

That starts Kafka, Zookeeper, PostgreSQL, the Spring Boot backend, and the Angular dashboard together.

Stack: Java 17, Spring Boot 3.x, Apache Kafka 3.x, Angular 13+, PostgreSQL 15, JSONata, Docker Compose, GitHub Actions CI.

The Kafka Consumer Safety Details

A few things worth understanding if you are building Kafka tooling:

Why assign() instead of subscribe():
subscribe() participates in consumer group coordination. Kafka’s group coordinator can reassign partitions at any time during a rebalance. For a read-only tool this is dangerous – you don’t want Kafka moving partitions between your tool and the production consumer mid-operation.

assign() with seek() gives you direct partition access. No group membership. No rebalance risk. The offset you seek to is exactly where you start reading. You consume exactly limit records and stop.

Why pagination at the Kafka offset level:
Standard REST pagination (page 1, page 2) doesn’t work well for Kafka topics because the “next page” needs to know the exact offset where the previous page ended. DLQ Revive uses cursor-based pagination: fromOffset=0&limit=50 returns messages at offsets 0-49. The next call uses fromOffset=50. This is memory-safe regardless of topic size.

Why no commitSync() in view mode:
Committing offsets in view mode would change the consumer group’s position – affecting what the production consumer reads next if they share a group ID. DLQ Revive uses a unique, isolated consumer group and never commits in view mode.

What’s Next

The open-source core handles:

• Paginated DLQ browsing
• JSONata schema transformation
• Preview before redriving
• Idempotency-safe redrive
• Full PostgreSQL audit trail
• One-command Docker setup

Free tier limit: 100 messages per redrive session. This is enough for testing and small incidents. Bulk redrive (>100 messages) and team features are in the cloud tier.

GitHub: github.com/Saifulhuq01/dlq-revive

MIT licensed. If you’ve dealt with DLQ schema recovery at work and want to try it against a real scenario, I’d genuinely value your feedback. Especially on the JSONata approach and whether the safety guarantees hold up under your specific Kafka setup.

Mohammed Saifulhuq – Apache Fineract contributor, building DLQ Revive