Custom API vs Custom Action vs Azure Function: Dataverse decision

A client needs to expose a “calculate the loyalty rebate for this customer” operation. It reads three Dataverse tables, applies some business rules, writes a result. Every consumer – the Dynamics web app, a Power Automate flow, an external integration – should call the same operation.

Three places we could put it. Three different cost, latency, and scale profiles. Here is the matrix we now run on every “new operation” request.

The three options

Custom Action (legacy): a process defined in Dataverse that can be invoked through the SDK. Steps are Workflow Activity Actions. Old-school but still widely deployed.

Custom API (modern): the successor to custom actions. Defined as Dataverse entity rows (customapi, customapiRequestParameter, customapiResponseProperty), backed by a plugin that implements the logic. Exposed through the Web API with a typed OpenAPI schema.

Azure Function: fully external .NET function, invoked from Power Automate or direct HTTP. Runs in its own compute, scales separately, has its own pricing.

The three solve overlapping problems. The choice is about latency, cost structure, and who maintains the code.

Decision matrix

DimensionCustom ActionCustom APIAzure FunctionLatencyLow (in-process)Low (in-process)Medium (cross-service)Concurrency limitDataverse’sDataverse’sFunction App’sTimeout2 minutes2 minutes10 minutes (Consumption), configurable higherCostIncluded in DataverseIncluded in DataversePer-execution + computeLong-running workNoNoYes (durable functions)External dependency callsLimited (sandbox)Limited (sandbox)Full flexibilityOpenAPI schemaNoYesManualInvocable from flowYesYes (first-class)Yes (HTTP connector)Maintenance.NET + Workflow.NET.NET (preferred)

When each one is correct

Custom Action is correct when you are maintaining an existing system that already uses them. For new work, skip them – custom APIs are the modern equivalent with better tooling.

Custom API is correct when:

  • The logic is primarily Dataverse queries and writes
  • Latency matters (under 500ms per call)
  • The caller is inside the Microsoft stack (flow, another plugin, Dynamics app)
  • The operation completes in under 2 minutes

Azure Function is correct when:

  • The logic calls external services with uncertain latency
  • The operation may take longer than 2 minutes
  • Durability, retries, or checkpointing matter (durable functions)
  • The code needs packages the Dataverse sandbox blocks (file IO, sockets, native dependencies)
  • You want the compute cost on a different meter from Dataverse

Cost reality

Custom APIs consume Dataverse capacity. A high-volume custom API (tens of thousands of calls per day) is paid for by your existing Dataverse licensing – no incremental per-call charge.

Azure Functions on Consumption plan: $0.000016 per GB-second and $0.20 per million executions. A typical custom function costing 128MB for 300ms runs at roughly $0.00000006 per call. A million calls per month is around $0.25 in compute plus $0.20 in executions – effectively free.

The cost math usually favors custom API for Dataverse-heavy work and Azure Function for external-dependency-heavy work. Mixing the two via a custom API that calls a function (rare pattern) pays both bills.

Gotchas per option

Custom API gotcha: the plugin that implements a custom API registers against a synthetic “Custom API message” step. Debugging is via plugin trace logs, same as any plugin. Deployment is through the solution. OpenAPI schema is auto-generated from the custom api row definitions – mis-typing a parameter name once in Dataverse and once in the plugin code produces a runtime failure that the solution checker does not catch.

Azure Function gotcha: cold start on Consumption plan can add 2-5 seconds for the first call after idle. For a user-facing interactive call, this is unacceptable. Options: Premium plan (always warm, higher baseline cost), Dedicated App Service plan, or accept the cold-start delay if the call is async.

Authentication gotcha for Azure Function: calling Dataverse from an Azure Function requires auth. The cleanest pattern is a managed identity on the Function App with an application user in Dataverse. Tutorials that use a user’s personal credentials in app settings are security incidents waiting to happen.

The decision we ran last month

Requirement: “Calculate and apply a loyalty rebate on every Order created.”

  • Reads three tables (Customer, OrderHistory, RebateProgram)
  • Writes a calculated value back to the Order
  • Must run on every order creation
  • Expected volume: 2,000 orders per day, bursty up to 50/minute
  • Latency target: under one second on order save

Walking the matrix:

  • Latency: sub-second is tight for Azure Function with cold start. Custom API stays in-process.
  • External dependencies: none. All Dataverse.
  • Long-running: no, under 500ms estimated.
  • Cost: Dataverse is already paid for; Azure Function would be cheap but not free.

Custom API won clearly. Implementation was a plugin registered against a custom API definition, called from both a post-operation plugin on Order Create (synchronous, blocks save) and directly from a Power Automate flow (for retroactive recalculation).

A year later, the client wanted to integrate with an external tax service that takes 3-4 seconds per call. We did not put that in the same custom API – the 2-minute timeout plus the unpredictability of the external service made it fragile. We built a separate Azure Function for the tax call, invoked async from a Power Automate flow. Two tools for two different latency profiles, as it should be.

The rule of thumb

If the operation stays inside Dataverse, start with a custom API. You get latency, integrated ALM, and zero incremental cost. You can always move it to an Azure Function later if the requirements shift.

If the operation has any external dependency with unpredictable behavior, start with an Azure Function. You get the durability patterns and the 10-minute timeout that the Dataverse sandbox cannot give.

If you find yourself using custom actions for new work, stop. Custom APIs have everything custom actions did plus a schema and better tooling.

Cutting agent latency from 30s to 8s without model swap

A founder pinged us with a UX problem disguised as an engineering question. His team had launched an AI chat product. Users were abandoning the conversation before the agent finished responding. The team had measured p95 response latency at 31 seconds. Their assumption was that they needed to switch to a faster model.

The actual model was responsible for about 35% of the total latency. The other 65% was sequential tool calls, unnecessary intermediate LLM steps, and a missing streaming layer. Switching to a smaller model would have cut maybe 5 seconds off the worst case while degrading response quality.

We made four changes that did not touch the model. P95 latency dropped from 31 seconds to 8 seconds. User abandonment rate dropped 70%. The model stayed the same.

This is the latency stack that most teams either do not measure or do not know how to optimize. Here is the pattern.

Where the time actually goes

Before any optimization, instrument the agent and trace where time is spent. The breakdown for a typical multi-step agent looks something like:

  • LLM calls (cumulative): 30-50% of total latency
  • Tool calls (cumulative): 30-40% of total latency
  • Network and serialization overhead: 5-10%
  • Application logic between steps: 5-10%

The LLM call portion is hard to optimize without changing the model. The other 50-70% is where most of the win is, and most teams do not look there because the LLM feels like the obvious bottleneck.

The founder’s agent had:

  • 8 LLM calls per response (planner + 4 tool dispatchers + critic + 2 retries)
  • 6 tool calls (4 distinct tools, with 2 retries)
  • All sequential
  • No streaming back to the user

Total p95 was 31 seconds. The model was responsible for 11 seconds of that. The other 20 seconds were structural.

Change 1: Parallelize independent tool calls

The biggest win in most agent systems is making tool calls concurrent.

The founder’s agent had a step where it needed to look up the user’s account information, the customer’s order history, and the related support tickets. The original code did this sequentially:

  1. Look up account (1.5s)
  2. Look up orders (2.0s)
  3. Look up tickets (1.8s)

Total: 5.3 seconds, all wait time.

The fix was async-await with parallel dispatch:

  1. Dispatch all three lookups concurrently
  2. Wait for all to complete
  3. Total: 2.0 seconds (the slowest of the three)

That single change saved 3.3 seconds per request. The actual code change was 8 lines.

The pattern: any time the agent calls multiple tools and the results are independent, those tool calls can run concurrently. Use asyncio.gather in Python, Promise.all in JavaScript. Modern agent frameworks (LangGraph, CrewAI Flows) support this natively.

Audit your agent for serial tool calls that could run in parallel. There is almost always one or two.

Change 2: Cut the unnecessary intermediate steps

The original agent had a planner LLM that generated a plan, then a separate critic LLM that reviewed the plan, then the executor.

The critic was rejecting roughly 8% of plans, which meant 92% of the time it was an extra LLM call (3 seconds) for nothing. And of the 8% it rejected, the planner produced a similar plan on the next iteration anyway, suggesting the critic was not adding much real value.

We removed the critic, kept the planner, and added validation rules instead (does the plan reference real tools, does it have any cycles, does it stay under the step limit). The validation runs in milliseconds, not seconds, and catches the same class of bad plans the critic was catching.

3 seconds saved per request, on average. Quality measured no different on the eval set.

The pattern: every LLM call in the chain should be earning its keep. If a step rejects only a small fraction of inputs and the alternative is a deterministic check, the deterministic check is faster and usually as good.

Change 3: Stream the response back to the user

The original agent waited for the full response to be generated before returning anything to the user. The user saw a loading spinner for 31 seconds, then the full response appeared.

Streaming changes the perceived latency dramatically. The user starts seeing tokens within 1-2 seconds, even if the full response takes 8. The agent is not actually faster, but the UX feels faster, and abandonment rate drops because users get visible feedback.

The implementation is a few lines of code. The OpenAI and Anthropic APIs both support streaming natively. The frontend needs to handle server-sent events (SSE) or websockets. The user experience improvement is large and immediate.

For multi-step agents, you can also stream intermediate progress: “Looking up your account… Checking order history… Generating response…” This is qualitatively different from a spinner. Users will wait 8 seconds for visible progress; they will not wait 8 seconds for a spinner.

Change 4: Cache aggressively

Some queries repeat. Some intermediate steps repeat across queries. Both are cacheable.

The founder’s agent had a “policy lookup” tool that retrieved from a small KB of company policies. The policies changed maybe once a month. The agent was hitting the KB and running a vector search every time. We added a simple in-memory cache with a 5-minute TTL. The cache hit rate was 35%, saving 0.8 seconds per cached request.

Semantic caching is the more advanced version: cache LLM responses keyed on the embedding of the query. If a new query is semantically similar to a recent one, return the cached response. We use this carefully (only for queries with high confidence and low staleness risk), but it can save 5+ seconds on cache hits.

Aggressive caching: every tool call and every LLM call should be evaluated for cacheability. Most are. Most teams do not bother. The latency improvement compounds.

What the founder shipped

After all four changes:

  • Parallel tool calls: -3 seconds
  • Removed critic LLM: -3 seconds
  • Streaming: perceived latency dropped from 31s to 1-2s for first token
  • Caching: -2 seconds average
  • Total p95: 31s → 8s

User abandonment rate dropped 70% in the first two weeks. Customer satisfaction scores went up. The model and the agent’s core logic did not change.

The team’s original instinct (switch to a smaller model) would have saved maybe 5 seconds while degrading quality. The structural changes saved 23 seconds while keeping quality the same.

The latency optimization checklist

Before approving “switch to a faster model” as the latency fix, walk through:

  • Are tool calls running sequentially when they could run in parallel? Almost always at least one set is serial that could be concurrent.
  • Is every LLM step earning its keep? Critics, validators, refiners that fire on every request might be removable in favor of deterministic checks.
  • Is the user seeing intermediate progress, or just a spinner? Streaming alone changes the perceived latency more than any model swap.
  • Are repeated queries cached? Tool calls, LLM calls, retrieved chunks: most are cacheable with TTL.
  • Are smaller models being used for non-critical steps? Use GPT-4o for the final answer if quality matters; use GPT-4o-mini for the planner, the router, the validator.

Most agent systems have 50-70% latency improvement available without touching the primary model. Audit before you swap.

The model swap is the last resort

Switching to a smaller, faster model can be the right call for some agents. But it is the last resort, not the first. The cost is response quality, and that cost is permanent. The structural improvements are free.

The pattern Sapota recommends: optimize the structure first, measure the new latency, then decide whether a model swap is still needed. Often it is not. When it is, you have a much smaller gap to close, and the trade-off is more justified.

If your agent’s latency is killing UX

If your team has launched an AI agent and users are abandoning conversations before the response arrives, the issue is rarely the model. It is usually the structure.

Sapota offers a one-week latency audit that traces your agent’s actual time spent, identifies the parallelizable tool calls, the removable LLM steps, and the missing caching opportunities, and ships the optimizations as working code. We have done this for chat products, customer support tools, and research assistants. The typical improvement is 60-75% latency reduction without changing the model.

Reach out via the AI engineering page with your current p95 latency and a sample trace if you have one. If you do not have traces, we will install observability first and audit second.

When recall plateaus: the late-interaction technique most teams skip

A founder we work with had been stuck on the same problem for two months. Their RAG retrieval recall was sitting at 58%. They had tried OpenAI’s embedding-3-small, then embedding-3-large, then BGE-M3, then Voyage. Each swap added a couple of points, then the curve flattened. The team was about to start fine-tuning their own embedding model.

We told them to stop and add a reranker first. The number went from 58% to 81% in a single afternoon. The fine-tuning project was cancelled.

This is the moment most teams discover that the bottleneck was never the embedding model. It was the architecture choice of using a single embedding per chunk to begin with. Late interaction is the family of techniques that fixes it, and it is the one most teams skip because the name sounds intimidating.

What a single embedding per chunk loses

A bi-encoder (which is what every standard embedding model is) takes a chunk of text, compresses it into a single fixed-length vector, and stores it. At query time, the user’s question is also compressed into a single vector, and similarity is computed between the two.

The compression is the problem. A 500-token chunk that mentions five different concepts gets averaged into one vector. The vector represents the chunk roughly, but it loses the distinction between “this chunk is mostly about X with a brief mention of Y” and “this chunk is mostly about Y with a brief mention of X.” When the user query is about Y, both chunks look equally relevant by cosine distance, even though one is the right answer and the other is noise.

This is why every benchmark of “best embedding model” shows diminishing returns past a certain point. The embedding model is doing the best it can with the information bottleneck of a single vector. The architecture is the limit.

How late interaction works

ColBERT (the original, 2020) keeps the per-token embeddings instead of pooling them into one vector. A 500-token chunk becomes 500 vectors. A 10-token query becomes 10 vectors. At scoring time, you compute the maximum similarity between each query token and any chunk token, then sum those max scores into the final relevance score.

The math is the same dot products you would do for any vector search. The difference is that “how well does this query match this chunk” is now a sum of “for each query token, what is the best matching chunk token,” which preserves the fine-grained signal that pooling threw away.

In practice this looks like:

  • Query “GPT-4o pricing breakdown” tokenizes to roughly 4 tokens.
  • Each token finds its best match in the candidate chunk.
  • “GPT-4o” matches the “GPT-4o” token in the chunk strongly.
  • “pricing” matches “cost” or “price” or “pricing” in the chunk.
  • “breakdown” matches “table” or “structure” or “breakdown.”
  • The summed max scores give a relevance number that reflects all four query terms, not just the average semantic similarity.

This is what catches the recall ceiling.

Two patterns we ship

Sapota uses ColBERT in two patterns depending on the corpus size and latency budget.

Pattern 1: ColBERT as a reranker over bi-encoder retrieval. The first stage is a standard bi-encoder vector search returning the top 50 candidates. The second stage is ColBERT reranking those 50 down to the top 5. This is the pattern we use for most production deployments. The first stage is fast (millisecond range, scales to billions of vectors). The second stage is slow but only runs on 50 candidates, not the full index.

Pattern 2: ColBERT as the only retriever. For corpora under a few million chunks, ColBERT can be the primary retriever using PLAID or similar index structures that make late-interaction search tractable at scale. Latency is higher than a bi-encoder (10x to 50x depending on index size), but recall is the highest of any retrieval method we have benchmarked.

We default to Pattern 1 unless the corpus is small enough that Pattern 2 is feasible and the recall lift justifies it.

ColPali: the same trick for documents

ColPali extends the late-interaction idea to entire document pages treated as images. Instead of token-level embeddings of text, it uses patch-level embeddings of an image of the page (each page split into a 32×32 grid of patches). Query tokens match against image patches using the same MaxSim mechanism.

The implications:

  • The OCR step disappears. The model sees the page as a vision-language model would.
  • Layout, charts, tables, and figures are all preserved as part of the same embedding space.
  • Cross-modal queries (text query against visual content) work natively.

The cost is storage (1024 vectors per page vs 1 vector per chunk) and indexing speed (vision encoder inference is GPU-bound). Binary quantization brings the storage cost down by 32x and the latency down by an order of magnitude, which is what makes ColPali production-viable.

For document-heavy corpora (research papers, financial filings, slide decks, regulatory submissions), ColPali outperforms both bi-encoder text RAG and CLIP-based multimodal RAG on published benchmarks. We use it when the corpus is genuinely visual and the budget supports the storage and GPU inference cost.

The cost conversation

Late interaction is not free. The honest trade-off:

  • Storage. ColBERT chunks store roughly 100x more vectors than bi-encoder chunks (one per token vs one per chunk). ColPali stores 1024 vectors per page. Plan for this in vector DB sizing.
  • Index time. Building the index takes longer because there are more vectors to compute. Not catastrophic, usually a few hours for a million-chunk corpus on a single GPU.
  • Query time. A reranker pattern adds 50ms to 200ms to the p50 latency, depending on how many candidates the reranker consumes. A pure-ColBERT setup adds more.
  • Operational complexity. Vector databases that support multi-vector with MaxSim comparators are a smaller set than the ones that support standard cosine search. Qdrant supports it natively. Most others do not.

The number to weigh against this is the recall lift. In every audit Sapota has run where the team’s recall plateaued in the 50% to 70% range, adding late-interaction reranking pushed it into the 80% to 90% range. That delta is the difference between “the AI is unreliable” and “the AI is the best search interface we have.”

When NOT to add late interaction

A reranker is not always the answer. Skip it when:

  • Recall is already above 90% (you are past the useful range; further gains come from prompt or generation work).
  • The corpus is small enough (under 50,000 chunks) that a cross-encoder reranker (BGE-reranker, Jina-reranker) gets you most of the same lift with less infrastructure.
  • The latency budget is hard-sub-100ms (a real-time conversational interface where the reranker overhead breaks the UX).
  • The team does not have a vector DB that supports multi-vector with MaxSim, and migrating is not on the roadmap.

For most production RAG systems sitting at recall in the 60% to 75% range, late interaction is the next move. Cross-encoder rerankers are the lighter alternative if the team is not ready for the full ColBERT setup.

What changed for the founder

The fix took half a day. We added a Jina-reranker stage between their existing Qdrant retrieval and the LLM call. Recall jumped from 58% to 81%. Faithfulness (because the LLM was now seeing better context) went from 0.79 to 0.93. The fine-tuning project was cancelled the same week.

The next conversation is whether to upgrade the cross-encoder to a full ColBERT setup, which would push recall another 4 to 6 points based on what we have seen on similar corpora. For their current scale and budget, the cross-encoder is the right floor. Full ColBERT is the v2.

If your retrieval has plateaued

If your team has been swapping embedding models and watching the recall curve flatten, the bottleneck is almost certainly the architecture, not the model. Sapota runs a one-week reranker integration engagement that adds the cross-encoder or ColBERT stage as a working PR plus a side-by-side eval against the current setup.

Reach out via the AI engineering page with the recall numbers you are seeing and the embedding models you have already tried. The diagnosis is usually the same conversation.

Mobile stack decision: FlutterFlow vs React Native vs Flutter

A founder asked us last month which mobile framework was “the best.” We get the question often enough that we have a rehearsed answer: none of them is best. Each fits a specific set of constraints, and picking the wrong one for your specific situation usually costs 2 to 4 months of rework eighteen months in.

The framework matters less than the fit. The fit depends on your timeline, your team, your design fidelity needs, and whether you are optimizing for the next twelve weeks or the next five years. Here is the decision framework Sapota walks every founder through.

The three options on the table

For most B2C and B2B mobile products in 2026, the realistic shortlist is FlutterFlow, React Native (with Expo), or Flutter native (writing Dart directly). Native iOS / Android (Swift, Kotlin) is still the right call for some specific cases, but it is rarely the default anymore. We will cover when native does win at the end.

The three frameworks differ on three dimensions that matter most: development speed, runtime control, and team future-proofing.

FlutterFlow

A visual development tool built on top of Flutter. You design screens by dragging components, configure data sources visually, and FlutterFlow generates Dart code under the hood. Custom logic drops into Dart custom actions and custom widgets when the visual editor cannot express what you need.

Strengths:

  • Fastest path to a shippable MVP. 12-week marketplace MVPs are realistic for a 3-engineer team.
  • Pixel-perfect design implementation. Figma to FlutterFlow is closer to one-to-one than any other framework we have used.
  • Theming and white-label setup are first-class concepts. Multi-tenant apps with brand variables ship in days, not weeks.
  • The founder can open the editor and request changes that you implement in minutes. Tightest design-to-development feedback loop in the industry.

Where it pushes back:

  • Anything custom-rendered (canvas drawing, complex animations, game-like UI) needs Dart custom widgets, which pulls you out of the visual flow.
  • State management at scale (3+ user roles, multi-tenant theming, real-time updates) outgrows app-state primitives. You end up moving state to the backend and treating the app as a thin client.
  • Performance profiling is shallow. No equivalent to React Native’s Flipper or Flutter’s DevTools. You debug performance through guesswork.
  • Vendor lock-in is real. Migrating off FlutterFlow means rebuilding in Flutter or another framework, not exporting and continuing.

Pick FlutterFlow when:

  • MVP timeline is under 16 weeks
  • 80%+ of screens are standard mobile UI patterns (auth, lists, details, forms)
  • Backend is decoupled (Supabase, Firebase, custom API)
  • Your team accepts that custom Dart will be needed for the bespoke 20%
  • Visual fidelity matters and you want predictable results across iOS and Android

Skip FlutterFlow when:

  • The product needs heavy real-time interactivity (multiplayer games, collaborative drawing, live markets)
  • Your engineering team will need to maintain the app for 5+ years and you want them comfortable with the underlying framework directly
  • You already have a strong React Native or Flutter team; the framework switch cost is higher than the visual development savings

React Native (with Expo)

Facebook’s framework for building native mobile apps using JavaScript and React. With Expo’s tooling (EAS Build, EAS Update, OTA patches), you get a managed workflow that handles a lot of native build complexity automatically.

Strengths:

  • Largest ecosystem of any cross-platform framework. Almost any native API has a JavaScript wrapper. Stuck on something? There is a npm package or a community fix.
  • JavaScript / TypeScript familiarity. Your web team can ramp on mobile faster than learning Dart or Swift.
  • OTA updates via Expo. Push bug fixes without going through App Store review. For early-stage products iterating weekly, this is significant.
  • Mature debugging tools (Flipper, React DevTools, native debuggers).
  • Best long-term hire-ability. JavaScript engineers are everywhere.

Where it pushes back:

  • Performance ceiling lower than Flutter or native for graphics-heavy or animation-heavy apps. The JavaScript bridge is faster than it used to be (with Hermes and the new architecture) but still not native.
  • Native module compatibility issues when you upgrade React Native. Major upgrades are rarely smooth. We budget 2-4 weeks for upgrades on production apps.
  • Visual fidelity work is more iterative than FlutterFlow. Designs render close but you spend more time on platform-specific edge cases (Android shadows, iOS safe areas, font rendering).
  • Easy to ship a slow app accidentally. The framework forgives sloppy patterns until production load reveals them.

Pick React Native when:

  • Your team has strong JavaScript / TypeScript background
  • The product needs OTA update capability (frequent iteration without store review)
  • You want hire-ability and ecosystem maturity over raw performance
  • The app is content-heavy or transactional (most B2B SaaS, content apps, e-commerce)
  • You expect 3+ years of active development with team turnover

Skip React Native when:

  • The app is graphics or animation-heavy
  • You need consistent 60+ fps under load
  • Your team has no JavaScript expertise and you would be teaching from scratch
  • The product is highly platform-specific (deep iOS-only or Android-only integrations)

Flutter native

Writing Dart directly against Flutter’s framework, no visual builder. Same underlying technology as FlutterFlow but with full control over every line of code.

Strengths:

  • Best performance of the three options. Compiles to native code, no JavaScript bridge, predictable 60+ fps even with complex UI.
  • Single codebase compiles to iOS, Android, web, desktop. The most truly cross-platform option of the three.
  • Beautiful animation framework. If your app’s differentiator is motion design or visual polish, Flutter native gives you the most expressive primitives.
  • Strong typing with Dart catches more bugs at compile time than JavaScript.
  • DevTools profiler is excellent. Performance debugging is straightforward.

Where it pushes back:

  • Slower to ship MVP than FlutterFlow. You write every screen by hand. For an MVP that FlutterFlow ships in 12 weeks, Flutter native usually takes 16-20.
  • Smaller ecosystem than React Native. Most native APIs have Flutter packages, but the long tail is smaller. You will write more native plugins yourself.
  • Hire-ability is improving but still behind JavaScript. Senior Flutter engineers in 2026 are easier to find than in 2022 but still command a premium.
  • Dart is a mid-popularity language. Comfortable for most engineers within a few weeks but not a transferable skill outside Flutter.

Pick Flutter native when:

  • Performance requirements are strict (60+ fps under all load conditions)
  • The app’s design includes complex animations or motion as a differentiator
  • You want one codebase across iOS, Android, web, and possibly desktop
  • The team is already Flutter-fluent or willing to invest in becoming so
  • Long-term ownership of the codebase matters more than fastest-possible MVP

Skip Flutter native when:

  • Your timeline is under 16 weeks for v1
  • You have no Flutter or Dart experience and the team’s bandwidth to ramp up is limited
  • The product is mostly standard CRUD screens (FlutterFlow ships these faster)
  • You need OTA updates as a core capability (Flutter has this but ecosystem is less mature than Expo)

When native (Swift, Kotlin) still wins

We bring up native iOS / Android less often than we used to, but it has not disappeared. The cases where we still recommend native:

  • Heavy platform-specific integrations: Apple Pay deep integration, ARKit / RealityKit, Watch app, CarPlay, App Clips, Live Activities. Same on Android: deep Auto, Wear OS, advanced camera features. Cross-platform frameworks reach these eventually but always with delay.
  • Performance-critical apps with platform-specific optimization: heavy image processing, on-device ML, AR / VR. Native gives you direct access to Metal, Core ML, ARKit without bridge overhead.
  • Apps where the team is already native and the cross-platform switch costs more than maintaining two codebases: rare but real. A team that has shipped 5 years of Swift will lose more by switching than by writing two apps.
  • Projects where the App Store review process is hostile to cross-platform (some categories of apps face stricter review when they look or behave non-native).

For most B2C and B2B SaaS products in 2026, native is overkill. The cross-platform frameworks have closed the gap on 95% of use cases.

The decision matrix

Eight criteria, four options each.

MVP speed (how fast you can ship v1):

  • FlutterFlow: Fastest
  • React Native: Fast
  • Flutter native: Medium
  • Native (Swift/Kotlin): Slowest

Performance ceiling (frame rate under load):

  • FlutterFlow: Medium
  • React Native: Medium
  • Flutter native: High
  • Native: Highest

Hire-ability in 2026 (how easy to find engineers):

  • FlutterFlow: Niche
  • React Native: High
  • Flutter native: Medium
  • Native: High per platform (Swift or Kotlin separately)

OTA updates (ship fixes without store review):

  • FlutterFlow: Yes via FlutterFlow Cloud
  • React Native: Yes via Expo
  • Flutter native: Limited
  • Native: Not supported

Long-term codebase health (5+ year ownership):

  • FlutterFlow: Risky, vendor lock-in
  • React Native: Stable
  • Flutter native: Stable
  • Native: Stable

White-label / multi-tenant theming:

  • FlutterFlow: Native first-class concept
  • React Native: Doable with effort
  • Flutter native: Doable with effort
  • Native: Doable with effort

Custom animations (motion-heavy products):

  • FlutterFlow: Limited
  • React Native: Medium
  • Flutter native: Strongest
  • Native: Strongest per platform

Visual fidelity to Figma:

  • FlutterFlow: Highest
  • React Native: Medium-High
  • Flutter native: High
  • Native: Highest per platform

How Sapota approaches the choice

The mistake we see most often is teams picking a framework based on what their current developers happen to know, then spending the next two years working around its limitations. The framework should fit the product, not the team’s existing comfort zone.

Our mobile engineers are trained across all four stacks (FlutterFlow, React Native, Flutter native, and the native platforms). When a new project lands, we run a fit assessment that scores the project against each framework on the criteria above. The recommendation comes from the assessment, not from “what we happen to specialize in.”

This sometimes means we recommend FlutterFlow when the founder expected React Native, or React Native when they expected Flutter. The conversation is occasionally awkward (founders have read articles, formed preferences) but it saves the months of rework that come from picking the wrong framework. We have done that rework on inherited projects often enough to know what it costs.

The cross-training is deliberate. A vendor who only knows FlutterFlow will recommend FlutterFlow. A vendor who only knows React Native will recommend React Native. The unbiased recommendation is the differentiator, and the only way to give it honestly is to be fluent across the options.

A simple decision tree

If you do not want to walk through the full matrix, the rough heuristic:

  • MVP in under 16 weeks, standard CRUD screens, single brand or simple white-label → FlutterFlow
  • 3+ year codebase, JavaScript team, frequent iteration with OTA → React Native
  • Performance-critical, animation-heavy, or codebase needs to span iOS / Android / web → Flutter native
  • Heavy platform-specific integration or your team is already native and shipping → Native

When in doubt, FlutterFlow for MVP, React Native for the production rebuild if you outgrow FlutterFlow’s limits.

If you are picking a stack right now

If your team is debating mobile frameworks and the conversation is not converging, the issue is usually that nobody has put the project’s specific constraints on the table next to each framework’s strengths. The decision is not abstract; it is a concrete fit for this product, this timeline, this team.

Sapota offers a one-week mobile stack assessment that takes your product spec, scores it against the four options, and returns a recommendation with the reasoning written down. We have done this for marketplaces, internal tools, content apps, and B2B SaaS products. The recommendation is unbiased because our engineers are trained across all four stacks.

Reach out via the mobile app development page with a description of what you are building, your timeline, and your team’s current language background. The first conversation usually surfaces the right fit within thirty minutes.

Plugin + Azure Function + Service Bus: async integration at scale

A Dataverse row changes. Five downstream systems need to know: an ERP that tracks financials, a search service that indexes the record, a Power BI dataset that feeds executive dashboards, a notification queue that messages field reps on mobile, and a data lake for analytics retention.

The junior version of this is a single async plugin that calls all five from the same post-operation step. It works in development. In production, it fails the first time any one of the five has a bad afternoon – the plugin step errors, retries ten times, fills the System Jobs queue, and operators get paged.

The pattern we ship at enterprise scale is different: the plugin’s only job is to publish a message to Azure Service Bus. Five independent consumers pull from that message topic and handle their respective destinations. Each consumer retries, dead-letters, and monitors independently. The plugin never calls out to anything but the queue.

Here is the architecture, the code, the failure semantics, and the six months of real-production experience that shaped it.

The architecture

A Topic (not a Queue) because the same event feeds multiple subscribers. Subscriptions filter so each consumer only sees events it cares about. Dead-letter per subscription, not global – the ERP and search services can fail independently without cross-contaminating.

Stage 1: the plugin

The plugin is small on purpose. Every line is a potential failure point; the less it does, the more reliable it is.

Three key decisions:

  1. Payload is metadata, not the full row. The message includes the entity name, ID, and which attributes changed. Consumers fetch the full row if they need it. Messages stay small (sub-1KB), which Service Bus charges less for and which avoids the edge case of payloads exceeding message size limits.
  2. CorrelationId from the plugin context. Every log entry and every downstream message carries the same correlation ID, making cross-system tracing possible with one query.
  3. Subject field for filtering. Subscribers filter on Subject LIKE ‘account.%’ or Subject = ‘order.Update’. Subject-based filtering is cheap server-side; body-based filtering is more expensive.

Stage 2: Service Bus Topic configuration

Topic-level settings:

  • Max message size: 1MB (default). Our payloads are ~500 bytes, nowhere near the limit.
  • Message time-to-live: 7 days. Long enough to survive a weekend outage, short enough that stale messages don’t haunt the system.
  • Duplicate detection: enabled with 10-minute window. If the same MessageId arrives twice within 10 minutes, the second is dropped. This guards against plugin retries causing duplicate downstream processing.

Subscription-level settings per subscriber:

  • Filter rule: SQL-like expression on message properties (Subject, custom headers).
  • Max delivery count: 5 (same reasoning as in the simpler Service Bus pattern).
  • Lock duration: tuned to each consumer’s processing time. The ERP consumer, which makes a remote call taking up to 30 seconds, has a 60-second lock. The search consumer, which is purely in-Azure and takes 1-2 seconds, has a 30-second lock.

Stage 3: the consumers

Each subscriber is an Azure Function with a Service Bus Topic subscription trigger. They share a common frame but implement different downstream logic.

Common frame:

The idempotency store is either Cosmos DB with TTL or Redis. Every successfully processed MessageId is recorded. The TTL matches the message time-to-live on the Topic plus a safety margin.

Core logic per consumer is where the differences live:

  • ERP consumer: fetches the full Dataverse row via Web API (using a managed identity), maps to the ERP’s schema, calls the ERP API with an idempotency key derived from MessageId.
  • Search consumer: fetches the row, calls the search service’s indexing API.
  • Power BI consumer: triggers a dataset partition refresh; uses the correlation ID to tag the refresh operation.
  • Notification consumer: looks up which users to notify based on the row’s owner and policy, sends via the notification service.
  • Data lake consumer: appends a compressed JSON record to partitioned ADLS Gen2 path by date.

Observability

The chain is not useful in production without observability. Our setup:

  • Application Insights per Function App, with shared workspace so queries span functions.
  • Every log entry includes CorrelationId and MessageId via the BeginScope pattern above.
  • Custom metrics: ProcessedMessages, FailedMessages, DeadLetteredMessages, ProcessingDuration.
  • Dashboards (Azure Monitor workbooks):End-to-end latency per event type (from Dataverse change timestamp to final destination write).Dead-letter queue depth per consumer.Error rate per consumer, per hour.
  • Alerts: DLQ depth > 10 for any consumer, processing duration p95 > SLA for any consumer.

The end-to-end trace for a single event: a KQL query across Application Insights joining by CorrelationId. One row change in Dataverse, traced through plugin → Service Bus publish → each consumer → each downstream call. When something fails, we know exactly where in the chain.

Six months in production: the numbers

A client with:

  • 50,000 Dataverse events per day
  • 5 consumers averaging 3 seconds processing each
  • Bursts to 500 events/minute during peak hours

Current state:

  • P50 end-to-end latency (event → all consumers complete): 4.2 seconds
  • P95 end-to-end latency: 11 seconds
  • Dead-letter rate: 0.003% (one in 33,000 messages)
  • Consumer error rate: 0.1-0.2% (mostly transient, auto-retried)
  • Monthly Azure cost (Service Bus + Function Apps + Application Insights): ~$320

What we tuned after launch

Ratio of Function App instances to queue depth. Consumption plan auto-scales; the defaults were conservative. We adjusted maxConcurrentCalls per function to keep consumer processing near its per-instance capacity before spinning up more instances.

Filter rules on subscriptions. Initially each consumer received every event and filtered in code. Moving the filter to the subscription level (server-side) cut consumer costs by ~60% because consumers no longer woke up for irrelevant events.

Deduplication window. Initial 1-minute window was too short; plugin retries occasionally pushed duplicates more than a minute apart. Extended to 10 minutes after measuring actual retry intervals.

When not to ship this

This pattern is overkill for:

  • Projects with one or two downstream consumers – direct Power Automate flow is simpler.
  • Low-volume scenarios (< 1000 events/day) – the operational complexity outweighs the throughput benefit.
  • Teams without Azure experience – maintaining Service Bus, Function Apps, and Application Insights is real ops burden. Pick a simpler pattern until the operational muscle exists.

For the projects where it fits (enterprise-scale, multi-consumer, event-critical), this is the architecture we reach for first. It survives load, it surfaces failures cleanly, and when something goes wrong, the fix is almost always in a known location with clear visibility.

SFMC Data Model and Cardinality: Wire DEs Together Without Regret

Teams new to SFMC create Data Extensions as requirements arrive. Today’s send needs a list, tomorrow’s loyalty feature needs a separate DE, next week’s CRM sync adds three more. Three months in: 20 DEs, nothing can be joined, no cross-data segmentation works, no audit trail.

The fix is 30 minutes on a whiteboard before the first DE exists. Two concepts carry it: Database of Record and Cardinality.

Database of Record (DBOR)

The DBOR is the single source of truth for subscriber identity. Every other data source points at it.

Discovery question for the client:

One answer. If the client says “all of them,” help them decide – usually by picking the system that currently drives business operations (order fulfillment, customer service queries).

The DBOR defines the Subscriber Key. All SFMC DEs that hold subscriber data use an ID that maps back to the DBOR.

Subscriber Key strategy

Once DBOR is decided, the Subscriber Key follows:

  • If DBOR is Salesforce CRM, use the CRM ContactID as Subscriber Key.
  • If DBOR is e-commerce, use the e-commerce customer ID.
  • If DBOR is a data warehouse, use whatever stable ID that warehouse assigns (often a UUID).

The Subscriber Key must be stable – never changes over a customer’s lifetime. Email addresses change; Shopify IDs can reset on a store migration; a purpose-built CustomerID is safest.

Document the Subscriber Key rule in the data model doc. When new data sources arrive later, they get plumbed through the same Subscriber Key.

The hub-and-spoke model we use most

Rather than one giant DE with every column, split by function:

Master holds the subscriber-facing attributes. Spokes hold the detail. Joins happen via AMPscript Lookup or Automation Studio SQL, not by duplicating columns.

Cardinality in Contact Builder

When you link DEs in Contact Builder for cross-DE segmentation, SFMC asks about cardinality – the relationship between two DEs:

CardinalityMeaningExample1:1One row in A relates to one row in BCustomer -> primary Address (each customer has one primary)1:ManyOne row in A relates to many rows in BCustomer -> Orders (one customer, many orders)Many:ManyMany rows in A relate to many in BCustomer -> Product (via Order_Items – each customer can buy many products, each product bought by many)

Getting cardinality wrong breaks segmentation:

  • Declaring 1:1 when it’s actually 1:Many means SFMC ignores all but the first matched row.
  • Declaring 1:Many when it’s actually 1:1 is harmless but lets you accidentally pick multiple rows.
  • Many:Many relationships require a junction DE (Order_Items) to resolve properly.

The data model doc we produce

Before touching SFMC for a new engagement, we write a one-page data model:

  1. DBOR – which system is the source of truth.
  2. Subscriber Key – exact field name, data type, where it originates.
  3. Master DE – list of fields, types, nullability, source system.
  4. Spoke DEs – name, purpose, cardinality to Master, source system.
  5. Lookup DEs – name, purpose, primary key.
  6. Contact Builder relationships – which DEs get linked, cardinality.

Review with the client. Once signed off, building in SFMC follows the doc rather than the other way around.

Why this matters on inherited engagements

When picking up a poorly-modeled SFMC account, the symptoms are predictable:

  • Segmentation that should be possible isn’t.
  • Same person appears multiple times in audiences.
  • Cross-DE reports have to be built with SQL outside SFMC.
  • Contact Builder has 15 linked DEs and nobody knows the cardinalities.

The remediation is the same whether you’re starting fresh or remodeling: define DBOR, fix Subscriber Key, introduce a Master DE, migrate downstream.

Takeaway

Data modeling doesn’t feel productive because you’re not building anything yet. On SFMC engagements it’s the most leveraged hour of the whole project. Write the data model doc, get client sign-off, then build. The alternative is the 20-DE sprawl that eats your Q2.

Modeling an SFMC data architecture? Our Salesforce team designs data models, Subscriber Key strategies, and Contact Builder relationships on production engagements. Get in touch ->

See our full platform services for the stack we cover.

Custom connector with OAuth2: three auth pitfalls we debugged

A client uses a third-party logistics API that is not in Power Automate’s built-in connector catalog. The API speaks OAuth2 authorization code flow. The platform has a “Create a custom connector” flow that claims to handle OAuth2 in a couple of clicks. The first two connectors we built this way worked. The third hit three separate issues that took a combined week to diagnose.

Here is what those three were, and the patterns we now check up-front on every OAuth2 custom connector.

What the custom connector OAuth2 flow is supposed to do

You fill in a form with:

  • Client ID
  • Client Secret
  • Authorization URL
  • Token URL
  • Refresh URL (usually same as Token URL)
  • Scope
  • Redirect URL (auto-generated by Power Platform)

The platform handles the rest: redirect the user through consent, capture the authorization code, exchange it for an access token, refresh when expired. When a flow uses the connector, the current access token is injected into the Authorization header automatically.

This works when the identity provider follows the OAuth2 spec faithfully. It breaks in ways the platform does not surface clearly when the provider takes any of the common non-spec shortcuts.

Pitfall 1: the redirect URL is tenant-specific and you hardcoded it in the IdP

You configured the logistics API’s OAuth app with a redirect URL copied from your Dev environment. The Dev custom connector works. You export the solution to UAT, import it, and the first OAuth consent in UAT fails with “redirect_uri_mismatch.”

Root cause: Power Platform generates a redirect URL that includes the environment-specific tenant subdomain. The URL from Dev is not the URL from UAT.

Fix: register multiple redirect URLs in the identity provider, one per Power Platform environment. The OAuth app on the IdP side needs an entry for every https://global.consent.azure-apim.net/redirect/* URL that corresponds to your environments.

Complication: some identity providers allow wildcards in redirect URL registration (https://global.consent.azure-apim.net/redirect/*), others do not. For the strict ones, you maintain a list and add/remove entries when environments come and go.

Pitfall 2: refresh token handling

The API you are connecting to issues access tokens valid for 1 hour and refresh tokens valid for 30 days. Power Platform should refresh the access token automatically when it expires; you should not need to re-authorize for 30 days.

What happened: every 2 hours, the flow started failing with 401 Unauthorized. Manual reauthorization of the connection fixed it; 2 hours later, same failure.

Diagnosis took a while. The flow’s run history showed the token being used, the API rejecting it, but no refresh attempt. We eventually traced it to how the connector was declaring its refresh behavior in the OpenAPI schema: the connector had no refreshUrl configured, so the platform treated every token as final and never attempted a refresh.

Fix: set the refreshUrl explicitly in the connector’s security section, even if it is the same as the token URL. Without it, the platform assumes no refresh is supported.

A secondary issue we found: some APIs rotate the refresh token on every refresh (return a new refresh token alongside the new access token). Power Platform handles this correctly if the response includes refresh_token in the JSON. APIs that only include refresh_token on the initial grant and not on refresh will eventually expire the stored refresh token, and re-authorization becomes necessary. This is a provider behavior you cannot fix in the connector – you can only schedule manual re-auth before the refresh token’s lifetime runs out.

Pitfall 3: the API expects a custom parameter name that OAuth2 spec does not mention

Standard OAuth2 token exchange sends grant_type=authorization_code, code=, redirect_uri=, etc. in the form body.

The API we were connecting to wanted an additional tenant_id parameter on every token exchange. It is not part of the spec; the provider added it for their multi-tenant SaaS model. Power Platform's custom connector OAuth2 configuration does not have a field for "extra parameters to send on token exchange." Default behavior sends only the spec-defined fields. The fix we used was hacky: move the OAuth flow behind an Azure Function. The flow calls the function's HTTPS endpoint, the function does the token exchange with the full set of parameters and returns the access token, the flow uses the access token via a separate connector that expects Bearer auth. This doubled the integration complexity (an extra hop, an extra resource to monitor) but was the only working path. We later migrated to direct connector when the API provider added spec-compliant OAuth as an option. The broader lesson: before building a custom connector with OAuth2, confirm the target API follows OAuth2 strictly enough to work in the platform's constrained configuration. Providers that require custom token exchange parameters, non-standard response fields, or DPoP/mTLS cannot use the built-in OAuth2 flow - and the connector wizard does not warn you. ## The pre-check we run before building an OAuth2 connector Ten minutes of reading provider docs before clicking "New custom connector" saves hours later. Our checklist: 1. Does the provider document the authorize URL, token URL, and refresh URL? (All three needed; "refresh works same as token" is the common case but confirm.) 2. Does the provider accept wildcard or multiple redirect URLs, or only one? (If only one, you need an extra OAuth app per environment.) 3. Does the provider rotate refresh tokens on refresh, and what are the lifetimes? (Shorter refresh lifetime means more re-auth pain.) 4. Does the token exchange require any parameters beyond the OAuth2 standard set? (If yes, you likely need an Azure Function proxy.) 5. Does the API require PKCE? (Power Platform supports it for some flows; confirm.) Any "no, not sure" on these is a reason to de-risk before committing the full integration. ## The connection reference hygiene A custom connector deployed to managed solution uses connection references for per-environment auth. One connector, one connection reference, N connections (one per environment, each bound to its own OAuth authorization). For the auth to actually carry across environments: - Create a long-lived connection per env at initial setup (authorize once, test works) - Ship the connection ID in the deployment-settings.json for each env - Rotate/reauthorize only when the refresh token lifetime runs out A connection reference that is re-bound mid-release will break every running flow. We now treat OAuth-backed connection references as nearly immutable - rebinding is an ops event, planned, communicated. ## The fourth pitfall we almost had On a subsequent project, we almost repeated a mistake: committing an OAuth client secret into the deployment-settings file. The file is in git. The secret would have been in git history forever. Pattern we now enforce: OAuth client secrets live in Azure Key Vault. A secret-type environment variable references the Key Vault secret URI. The deployment-settings file references only the env variable schema name, not the secret value. Git gets the schema name; Key Vault gets the secret; nothing sensitive lives in both. Four pitfalls, one prevention pattern each. Built into the project template, the custom-connector work that used to eat weeks now takes days.

Four forensics when a production AI agent fails

A founder messaged us at 11pm on a Friday: “Our agent is broken. Customers are complaining. My on-call engineer has no idea where to start. Can you help?”

The agent was a customer support tool that had launched the previous Monday. By Friday evening, the company’s support inbox had filled with users reporting that the AI was giving wrong answers, taking forever to respond, or just timing out. The engineering team was treating it as one big problem. It was actually four problems stacked on top of each other.

This is the failure pattern most production agent teams hit at some point. The symptoms compound, the team panics, and they start trying random fixes. Here is the forensics order Sapota walks through, and the four most common failure modes that account for the majority of post-launch incidents.

Forensics order: traces first

Before debugging anything else, look at the traces. If your agent is in production without traces, that is the first problem to solve, even mid-incident. Pull a request that is failing, look at the trace, and see where the time is being spent and what is failing.

The pattern we look for in the trace:

  • Where does the request actually fail? A specific tool call? A specific LLM step? A timeout?
  • What does the failure look like? HTTP error? LLM hallucination? Output schema mismatch? Output that passes validation but is wrong?
  • What changed recently? Compare a failing request to a working request from a week ago. What is different?

In the founder’s case, the traces showed three different failure patterns appearing in the same week. The team had been treating them as one problem because the customer-facing symptom was the same: “the AI is broken.”

Failure mode 1: External dependency degradation

The most common production agent failure is an external dependency getting slower or less reliable. The agent itself is fine; the world around it changed.

Common culprits:

  • LLM provider rate limits. OpenAI or Anthropic starts throttling because your traffic increased past your tier limit. Each request now retries 3 times before succeeding, tripling latency.
  • Retrieval system slowdown. Your vector database is under more load than at launch and the p95 query latency went from 50ms to 800ms.
  • External API drift. A tool you call (CRM API, billing system, search service) had a quiet update that changed response shape or timing.
  • Knowledge base growth. Your KB has tripled in size since launch, and retrieval recall has dropped because you never tuned for the larger corpus.

The diagnostic: check your tool latency and error metrics for the past week. If any tool’s p95 latency is 2x what it was at launch, or its error rate is up more than 1%, that is the candidate.

The fix depends on the specific dependency. Rate limits: upgrade your tier or implement exponential backoff. Slow retrieval: tune the index or scale the database. API drift: update the integration. KB growth: re-tune chunking and retrieval parameters.

In the founder’s case, the LLM provider had pushed a quiet model update on Wednesday. The new model interpreted the routing prompt slightly differently, causing the agent to loop more often before settling on an answer. Average iterations went from 2.3 to 4.1. Cost and latency both jumped. The fix was a tighter routing prompt with three more few-shot examples.

Failure mode 2: Validation gates that aren’t being triggered

The opposite failure: a validation gate is supposed to be catching bad outputs, but it is not firing because the gate logic has a bug or the threshold is wrong.

Common patterns:

  • Faithfulness threshold too low. Set at 0.5, the gate passes responses that are mostly hallucinated. Should be 0.85+ for production.
  • Schema validation that allows nulls. The output schema requires “answer” field but allows it to be empty string. Empty answers ship to users as “I don’t know” without the agent realizing it failed.
  • Toxicity filter not loaded. The filter library was supposed to be imported, but a refactor moved the import and now it silently no-ops.
  • PII redaction running on the wrong field. Redacts user input but ships PII through in the response.

The diagnostic: look at a sample of bad responses customers reported. Trace what should have caught them. If a validation gate exists for that failure mode, check whether it actually fired.

In the founder’s system, the faithfulness threshold was set at 0.7, which was permissive. We tightened it to 0.85, the rejection rate went from 2% to 9%, and the customer complaints about wrong answers dropped immediately. The “rejected” responses were replaced with honest “I cannot find that in our knowledge base” messages, which users preferred to wrong answers.

Failure mode 3: Cost runaway from edge cases

Production query distribution is different from test distribution. Specific query patterns can be much more expensive than the average, and a few of those can dominate the bill.

The pattern: a small fraction of users (often 1-5%) generate a large fraction of cost (often 30-60%). Either through legitimate complex queries, abuse, or because their input triggers a degenerate code path in the agent.

The diagnostic: pull cost-per-user statistics for the last week. Sort descending. Look at the top 10 users. Are they sending normal queries? Or is one user looping their integration with bad inputs? Or is a specific query class (long documents, malformed input, multi-turn deep into rare topics) eating budget?

The fixes vary:

  • Per-user rate limit and cost cap. Hard limit per user per day. Cuts off abuse without affecting legitimate use.
  • Input length cap. Most LLM cost scales with input tokens. Cap user input at a reasonable max (say, 10k characters) and politely ask them to summarize for longer queries.
  • Query type routing. If a specific query type is expensive, route it to a simpler/cheaper handler when possible. “Generate a comprehensive report” is expensive; route it to async batch processing instead of synchronous chat.
  • Iteration cap per request. Prevent the agent from looping indefinitely on a single request. We default to 5-10 iterations max.

In the founder’s system, two users were sending repeated multi-paragraph product comparison requests, generating about 40% of the daily cost between them. We added a per-user daily cost cap and a length limit on inputs. Cost dropped 35% within 48 hours. Neither user complained because both were testing internal features and the cap was generous enough for normal use.

Failure mode 4: Silent quality drift

The hardest failure to detect: nothing is broken, no errors, latency is fine, cost is normal. But the responses are getting worse. Customers complain, the team cannot reproduce, and the dashboards all look green.

Causes:

  • Prompt drift. Engineering edits to the prompt template that look small but change behavior. A removed example, a reworded instruction, a clarification that the LLM interprets differently.
  • Model provider updates. As above, the underlying model can change without you knowing. Quality on your specific use case can shift in either direction.
  • Corpus drift. Your KB has accumulated content that pollutes retrieval. Old documents that should have been deprecated still rank highly. New documents conflict with old documents.
  • Eval set staleness. Your eval set was written six months ago against an older product version. It does not reflect what users actually ask now.

The diagnostic: run your eval pipeline against the current production agent. Compare against the score from launch. If the score has dropped, you have quality drift. If the score is the same but customers are complaining, your eval set has gone stale.

The fix: refresh the eval set. Sample 50-100 actual production queries, write expected answers for each, run the eval, and tune from there. Most teams refresh eval sets quarterly. Teams in fast-moving domains do it monthly.

What we shipped for the founder’s incident

The four-hour Friday-night triage:

  1. External dependency: identified the LLM provider’s model update, tightened the routing prompt, agent stopped looping. Latency and cost recovered.
  2. Validation gate: tightened the faithfulness threshold from 0.7 to 0.85, rejected responses now return honest “I don’t know” instead of hallucinations.
  3. Cost runaway: added per-user daily cost cap of $5, input length limit of 10k characters. Cost dropped 35%.
  4. Quality drift: ran eval, score had dropped from 0.84 to 0.71 since launch. Refreshed eval set with 50 recent production queries, identified three categories of questions the agent was failing, added documentation coverage for those categories. Score returned to 0.86 after a week.

Customer complaints stopped within 72 hours. The team’s mood went from “we built a broken thing” to “we built a thing that needs operational rigor we did not anticipate.” That second framing is the one that produces a better product.

The recommendation: rehearse the forensics

The founder’s team had no playbook for “the agent is broken in production.” They were debugging in panic mode, which slowed every step. After the incident, we wrote a one-page runbook with the four failure modes, the diagnostic for each, and the most common fixes.

Six weeks later, when a similar issue happened (a tool API outage), the on-call engineer worked through the runbook, identified the cause in 20 minutes, applied the documented fix, and was done in an hour. No panic, no escalation, no Friday-night call to a consultant.

This is what production agent operations looks like at maturity. Not “nothing ever goes wrong” but “when things go wrong, the team has a known process to find the cause.”

If your agent had a launch that went sideways

If your team launched an AI agent and the first few weeks have been more painful than expected, the right intervention is usually a forensic audit, not more development. Most launch issues are not new bugs in the agent code. They are operational gaps that surface only at production scale.

Sapota offers a one-week post-launch audit that walks through traces, validation, dependencies, and quality drift, identifies which of the four failure modes is responsible for which symptoms, and ships fixes plus a runbook for future incidents. We have done this for half a dozen B2B SaaS clients in the first three months after their AI launches.

Reach out via the AI engineering page with a description of what your agent does and what kind of failures you are seeing. The first conversation is usually the diagnostic.

Send Log DE in SFMC: Tracking What the Defaults Miss

SFMC’s built-in tracking is useful but incomplete. The system _Open, _Click, _Sent, _Bounce data views cover email events with SubscriberKey, timestamp, and basic identifiers. For most analysis, they’re enough.

The gaps show up when you need business context: which campaign this was part of, which DE it came from, what subject line was used, which product ID was personalized into the email. The defaults don’t store that.

Send Log DE is how you capture it.

What Send Log does

Send Log DE is a custom Data Extension that SFMC populates on every send, in addition to the default tracking. You define what fields to capture, and AMPscript inside the email writes the values into Send Log for each recipient at send time.

It’s additive – you’re not replacing the default tracking, just adding a parallel log with richer metadata.

Setup: create the Send Log DE

Create a new DE from the TriggeredSendDataExtension template (for Triggered Sends) or from scratch for other use cases. Typical fields:

The built-in fields (SendID, SubscriberKey, SendDate) populate automatically when Send Log is wired up. The custom fields are up to you.

Wiring Send Log into the email

In the email body, add AMPscript that writes to Send Log:

Runs once per recipient at send time. Produces one row per subscriber per send.

Send Log extends default retention

The default _SendLog system data view retains records for 10 days. A custom Send Log DE retains as long as your retention policy allows – 90 days, 1 year, or indefinitely.

For an audit or long-term reporting requirement (“show me all sends to this customer over the last 12 months”), a custom Send Log DE is how you satisfy it.

Archiving pattern

Pair Send Log with a nightly archive automation:

This handles the system _SendLog retention problem. Separately, the custom Send Log DE captures your additional fields at send time.

Things that don’t go into Send Log

Send Log writes at send time, which means you have access to:

  • Subscriber attributes (AMPscript can read the DE)
  • Static per-send values (subject, campaign ID)
  • Personalization values used in the email

It does not have access to:

  • Subsequent engagement (opens, clicks happen after the send, covered by default _Open / _Click)
  • Bounce information (comes later, in _Bounce)
  • Unsubscribe events (covered by _Subscribers status changes)

For joined analysis (“did the customer open the email that had ProductID X?”), join Send Log with _Open on SendID + SubscriberKey using SQL Query.

One gotcha: Test Send doesn’t write to Send Log

Test Sends skip the Send Log write. If you’re testing and expecting the custom Send Log to populate, it won’t. Only production sends write.

Helps when testing – you don’t pollute the log with test data. Confuses when you’re trying to debug Send Log wiring, because Test Send success doesn’t confirm Send Log works.

Takeaway

Send Log DE is the bridge between SFMC’s send infrastructure and your analytics or audit needs. The default tracking views answer “who opened, who clicked.” Send Log answers “which campaign, which subject, which product was in the email.” Five minutes of AMPscript at the top of the email template, and the analytics team gets data they can actually work with.

Setting up custom SFMC tracking or audit logs? Our Salesforce team designs Send Log strategies and reporting pipelines on production engagements. Get in touch ->

See our full platform services for the stack we cover.

Model-driven ribbon: Command Designer vs Ribbon Workbench in 2026

Adding a custom button to a model-driven form’s command bar used to be a single-tool problem. Ribbon Workbench (a community XrmToolBox tool) was the answer for years. Microsoft then shipped Command Designer as the modern first-party replacement. For a while the two coexisted uneasily; the choice was political more than technical.

As of 2026, Command Designer has matured enough that it is the right first choice for new work, with specific exceptions. Here is the current comparison, and the migration pattern when you inherit a project with Ribbon Workbench customizations.

The current state

Command Designer (Power Apps maker portal):

  • First-party, UI-based, no additional tool installation
  • Uses Power Fx for command logic (visibility, action, label)
  • Ships in the solution as commandDefinition entries
  • Works with the “modern” command bar (the one on new model-driven apps)
  • Does not work with the classic command bar

Ribbon Workbench (XrmToolBox community tool):

  • Third-party, XML-based, requires installing XrmToolBox
  • Uses JavaScript for command logic
  • Ships as custom RibbonDiffXml in solution XML
  • Works with both classic and modern command bars (but modern support is limited for newer features)

What Command Designer is good at

  • Simple conditional visibility: “show this button if the record status is Draft” is a one-line Power Fx expression.
  • Direct calls to Power Fx patchable operations: Update a field, launch a Canvas dialog, navigate to a record. No JavaScript glue needed.
  • Source control friendliness: commandDefinition entries are structured XML in the solution; the diff is readable.
  • New-team-member onboarding: makers who know Power Fx can author commands without learning JavaScript and the classic ribbon XML schema.

What Ribbon Workbench still wins at

  • Complex conditional logic: visibility rules involving multiple fields, related record values, or security role checks are easier in JavaScript than in Power Fx.
  • Integration with existing JavaScript: if your project has a form script library with utility functions, commands can call into it directly.
  • Mature patterns for classic apps: if you maintain a classic app (Unified Interface but pre-modern command bar), Ribbon Workbench is the only practical tool.
  • Fine control over ribbon structure: hiding out-of-box buttons, repositioning, custom groups – Ribbon Workbench’s XML editor exposes the whole surface.

Our decision rule

For new model-driven apps on the modern command bar: start with Command Designer. Use Power Fx for command logic. Only fall back to Ribbon Workbench if you hit a feature gap.

For existing apps on the classic command bar: stay with Ribbon Workbench until the app is migrated to modern. Mixing the two tools on the same app creates confusion.

For apps that explicitly need JavaScript logic: Ribbon Workbench if the logic is complex; Command Designer with a minimal JavaScript bridge if the logic is small.

Migration pattern: Ribbon Workbench to Command Designer

Projects inherit Ribbon Workbench commands that were authored years ago. Moving them to Command Designer is usually worth it when:

  • You are upgrading the app to the modern command bar
  • You are simplifying the team’s skill requirements (one tool, one language)
  • The existing commands are maintenance burden (JavaScript with no tests, cryptic logic)

The migration is per-command, not all-or-nothing. Our process:

Step 1: inventory the existing commands. Export the solution, parse the RibbonDiffXml to list every custom command with its logic.

Step 2: rebuild each in Command Designer. Create the equivalent command in the maker portal. Power Fx the visibility and action logic. Test on a form.

Step 3: remove the Ribbon Workbench definition for that command. Edit the RibbonDiffXml to remove the old entry. Re-export the solution.

Step 4: deploy both changes together. The deploy adds the Command Designer command and removes the Ribbon Workbench command atomically.

Steps 2-4 per command. For a project with 15 custom commands, budget a day per complex command, a couple hours for simple ones. Four or five working days total.

What breaks in migration

JavaScript that called utility libraries. Commands that invoked shared JavaScript functions from the form library don’t translate directly to Power Fx. Two paths:

  1. Rewrite the utility in Power Fx (if it’s simple enough).
  2. Keep a small JavaScript bridge: the Command Designer command calls a Power Fx function, which calls into the JavaScript library via the Launch() function’s custom connector or a custom API endpoint.

Path 2 sounds hacky but works. For utility logic that is genuinely shared across forms (e.g., “format phone number by country”), keeping the JavaScript and bridging is simpler than rewriting.

Conditional visibility based on related records. “Show this button if the account’s parent has status X” requires a lookup from the current record to its parent, then a field check. In JavaScript this is a Xrm.WebApi call; in Power Fx it is a LookUp that may or may not delegate.

For simple parent checks, Power Fx works. For complex graph traversals, either denormalize the needed field onto the current table (via a rollup or plugin) or keep the command in Ribbon Workbench.

Button placement customization. Ribbon Workbench lets you specify exact positioning: “place this button to the left of Save, in the Actions group.” Command Designer’s placement controls are more limited – you pick a group, not an exact slot. If pixel-perfect placement matters, Ribbon Workbench retains an edge.

The one gotcha: solutions and ownership

Command Designer commands ship in the solution via new commandDefinition and commandComponent entities. Ribbon Workbench customizations ship via ribbon XML on the entity or application.

When both tools have touched the same entity’s ribbon, solution imports can get confused. The safe pattern:

  • One tool per entity’s ribbon at a time. If migrating, finish the entity before moving on.
  • Keep managed-solution layering in mind. A Ribbon Workbench customization from an older managed layer plus a Command Designer customization from the current solution can produce unpredictable behavior at runtime.

The one-year outlook

Microsoft’s direction is toward Command Designer as the canonical tool. New command-related features (command bar on Canvas components, AI-assisted command generation) are shipping to Command Designer first.

Ribbon Workbench will remain supported for the foreseeable future as long as classic command bar exists, but the feature gap is widening. For any project with a five-year horizon, investing in Command Designer skills over Ribbon Workbench is the right bet.

If you have a Ribbon Workbench-heavy codebase, start migrating the commands you touch anyway on feature work. Don’t do a big-bang migration unless you are already upgrading the app for other reasons. Incremental migration is the path that has worked across three projects we have moved.