Mert

Teams new to SFMC create Data Extensions as requirements arrive. Today’s send needs a list, tomorrow’s loyalty feature needs a separate DE, next week’s CRM sync adds three more. Three months in: 20 DEs, nothing can be joined, no cross-data segmentation works, no audit trail.

The fix is 30 minutes on a whiteboard before the first DE exists. Two concepts carry it: Database of Record and Cardinality.

Database of Record (DBOR)

The DBOR is the single source of truth for subscriber identity. Every other data source points at it.

Discovery question for the client:

One answer. If the client says “all of them,” help them decide – usually by picking the system that currently drives business operations (order fulfillment, customer service queries).

The DBOR defines the Subscriber Key. All SFMC DEs that hold subscriber data use an ID that maps back to the DBOR.

Subscriber Key strategy

Once DBOR is decided, the Subscriber Key follows:

• If DBOR is Salesforce CRM, use the CRM ContactID as Subscriber Key.
• If DBOR is e-commerce, use the e-commerce customer ID.
• If DBOR is a data warehouse, use whatever stable ID that warehouse assigns (often a UUID).

The Subscriber Key must be stable – never changes over a customer’s lifetime. Email addresses change; Shopify IDs can reset on a store migration; a purpose-built CustomerID is safest.

Document the Subscriber Key rule in the data model doc. When new data sources arrive later, they get plumbed through the same Subscriber Key.

The hub-and-spoke model we use most

Rather than one giant DE with every column, split by function:

Master holds the subscriber-facing attributes. Spokes hold the detail. Joins happen via AMPscript Lookup or Automation Studio SQL, not by duplicating columns.

Cardinality in Contact Builder

When you link DEs in Contact Builder for cross-DE segmentation, SFMC asks about cardinality – the relationship between two DEs:

CardinalityMeaningExample1:1One row in A relates to one row in BCustomer -> primary Address (each customer has one primary)1:ManyOne row in A relates to many rows in BCustomer -> Orders (one customer, many orders)Many:ManyMany rows in A relate to many in BCustomer -> Product (via Order_Items – each customer can buy many products, each product bought by many)

Getting cardinality wrong breaks segmentation:

• Declaring 1:1 when it’s actually 1:Many means SFMC ignores all but the first matched row.
• Declaring 1:Many when it’s actually 1:1 is harmless but lets you accidentally pick multiple rows.
• Many:Many relationships require a junction DE (Order_Items) to resolve properly.

The data model doc we produce

Before touching SFMC for a new engagement, we write a one-page data model:

1. DBOR – which system is the source of truth.
2. Subscriber Key – exact field name, data type, where it originates.
3. Master DE – list of fields, types, nullability, source system.
4. Spoke DEs – name, purpose, cardinality to Master, source system.
5. Lookup DEs – name, purpose, primary key.
6. Contact Builder relationships – which DEs get linked, cardinality.

Review with the client. Once signed off, building in SFMC follows the doc rather than the other way around.

Why this matters on inherited engagements

When picking up a poorly-modeled SFMC account, the symptoms are predictable:

• Segmentation that should be possible isn’t.
• Same person appears multiple times in audiences.
• Cross-DE reports have to be built with SQL outside SFMC.
• Contact Builder has 15 linked DEs and nobody knows the cardinalities.

The remediation is the same whether you’re starting fresh or remodeling: define DBOR, fix Subscriber Key, introduce a Master DE, migrate downstream.

Takeaway

Data modeling doesn’t feel productive because you’re not building anything yet. On SFMC engagements it’s the most leveraged hour of the whole project. Write the data model doc, get client sign-off, then build. The alternative is the 20-DE sprawl that eats your Q2.

Modeling an SFMC data architecture? Our Salesforce team designs data models, Subscriber Key strategies, and Contact Builder relationships on production engagements. Get in touch ->

See our full platform services for the stack we cover.

A client uses a third-party logistics API that is not in Power Automate’s built-in connector catalog. The API speaks OAuth2 authorization code flow. The platform has a “Create a custom connector” flow that claims to handle OAuth2 in a couple of clicks. The first two connectors we built this way worked. The third hit three separate issues that took a combined week to diagnose.

Here is what those three were, and the patterns we now check up-front on every OAuth2 custom connector.

What the custom connector OAuth2 flow is supposed to do

You fill in a form with:

• Client ID
• Client Secret
• Authorization URL
• Token URL
• Refresh URL (usually same as Token URL)
• Scope
• Redirect URL (auto-generated by Power Platform)

The platform handles the rest: redirect the user through consent, capture the authorization code, exchange it for an access token, refresh when expired. When a flow uses the connector, the current access token is injected into the Authorization header automatically.

This works when the identity provider follows the OAuth2 spec faithfully. It breaks in ways the platform does not surface clearly when the provider takes any of the common non-spec shortcuts.

Pitfall 1: the redirect URL is tenant-specific and you hardcoded it in the IdP

You configured the logistics API’s OAuth app with a redirect URL copied from your Dev environment. The Dev custom connector works. You export the solution to UAT, import it, and the first OAuth consent in UAT fails with “redirect_uri_mismatch.”

Root cause: Power Platform generates a redirect URL that includes the environment-specific tenant subdomain. The URL from Dev is not the URL from UAT.

Fix: register multiple redirect URLs in the identity provider, one per Power Platform environment. The OAuth app on the IdP side needs an entry for every https://global.consent.azure-apim.net/redirect/* URL that corresponds to your environments.

Complication: some identity providers allow wildcards in redirect URL registration (https://global.consent.azure-apim.net/redirect/*), others do not. For the strict ones, you maintain a list and add/remove entries when environments come and go.

Pitfall 2: refresh token handling

The API you are connecting to issues access tokens valid for 1 hour and refresh tokens valid for 30 days. Power Platform should refresh the access token automatically when it expires; you should not need to re-authorize for 30 days.

What happened: every 2 hours, the flow started failing with 401 Unauthorized. Manual reauthorization of the connection fixed it; 2 hours later, same failure.

Diagnosis took a while. The flow’s run history showed the token being used, the API rejecting it, but no refresh attempt. We eventually traced it to how the connector was declaring its refresh behavior in the OpenAPI schema: the connector had no refreshUrl configured, so the platform treated every token as final and never attempted a refresh.

Fix: set the refreshUrl explicitly in the connector’s security section, even if it is the same as the token URL. Without it, the platform assumes no refresh is supported.

A secondary issue we found: some APIs rotate the refresh token on every refresh (return a new refresh token alongside the new access token). Power Platform handles this correctly if the response includes refresh_token in the JSON. APIs that only include refresh_token on the initial grant and not on refresh will eventually expire the stored refresh token, and re-authorization becomes necessary. This is a provider behavior you cannot fix in the connector – you can only schedule manual re-auth before the refresh token’s lifetime runs out.

Pitfall 3: the API expects a custom parameter name that OAuth2 spec does not mention

Standard OAuth2 token exchange sends grant_type=authorization_code, code=, redirect_uri=, etc. in the form body.

The API we were connecting to wanted an additional tenant_id parameter on every token exchange. It is not part of the spec; the provider added it for their multi-tenant SaaS model. Power Platform's custom connector OAuth2 configuration does not have a field for "extra parameters to send on token exchange." Default behavior sends only the spec-defined fields. The fix we used was hacky: move the OAuth flow behind an Azure Function. The flow calls the function's HTTPS endpoint, the function does the token exchange with the full set of parameters and returns the access token, the flow uses the access token via a separate connector that expects Bearer auth. This doubled the integration complexity (an extra hop, an extra resource to monitor) but was the only working path. We later migrated to direct connector when the API provider added spec-compliant OAuth as an option. The broader lesson: before building a custom connector with OAuth2, confirm the target API follows OAuth2 strictly enough to work in the platform's constrained configuration. Providers that require custom token exchange parameters, non-standard response fields, or DPoP/mTLS cannot use the built-in OAuth2 flow - and the connector wizard does not warn you. ## The pre-check we run before building an OAuth2 connector Ten minutes of reading provider docs before clicking "New custom connector" saves hours later. Our checklist: 1. Does the provider document the authorize URL, token URL, and refresh URL? (All three needed; "refresh works same as token" is the common case but confirm.) 2. Does the provider accept wildcard or multiple redirect URLs, or only one? (If only one, you need an extra OAuth app per environment.) 3. Does the provider rotate refresh tokens on refresh, and what are the lifetimes? (Shorter refresh lifetime means more re-auth pain.) 4. Does the token exchange require any parameters beyond the OAuth2 standard set? (If yes, you likely need an Azure Function proxy.) 5. Does the API require PKCE? (Power Platform supports it for some flows; confirm.) Any "no, not sure" on these is a reason to de-risk before committing the full integration. ## The connection reference hygiene A custom connector deployed to managed solution uses connection references for per-environment auth. One connector, one connection reference, N connections (one per environment, each bound to its own OAuth authorization). For the auth to actually carry across environments: - Create a long-lived connection per env at initial setup (authorize once, test works) - Ship the connection ID in the deployment-settings.json for each env - Rotate/reauthorize only when the refresh token lifetime runs out A connection reference that is re-bound mid-release will break every running flow. We now treat OAuth-backed connection references as nearly immutable - rebinding is an ops event, planned, communicated. ## The fourth pitfall we almost had On a subsequent project, we almost repeated a mistake: committing an OAuth client secret into the deployment-settings file. The file is in git. The secret would have been in git history forever. Pattern we now enforce: OAuth client secrets live in Azure Key Vault. A secret-type environment variable references the Key Vault secret URI. The deployment-settings file references only the env variable schema name, not the secret value. Git gets the schema name; Key Vault gets the secret; nothing sensitive lives in both. Four pitfalls, one prevention pattern each. Built into the project template, the custom-connector work that used to eat weeks now takes days.

A founder messaged us at 11pm on a Friday: “Our agent is broken. Customers are complaining. My on-call engineer has no idea where to start. Can you help?”

The agent was a customer support tool that had launched the previous Monday. By Friday evening, the company’s support inbox had filled with users reporting that the AI was giving wrong answers, taking forever to respond, or just timing out. The engineering team was treating it as one big problem. It was actually four problems stacked on top of each other.

This is the failure pattern most production agent teams hit at some point. The symptoms compound, the team panics, and they start trying random fixes. Here is the forensics order Sapota walks through, and the four most common failure modes that account for the majority of post-launch incidents.

Forensics order: traces first

Before debugging anything else, look at the traces. If your agent is in production without traces, that is the first problem to solve, even mid-incident. Pull a request that is failing, look at the trace, and see where the time is being spent and what is failing.

The pattern we look for in the trace:

• Where does the request actually fail? A specific tool call? A specific LLM step? A timeout?
• What does the failure look like? HTTP error? LLM hallucination? Output schema mismatch? Output that passes validation but is wrong?
• What changed recently? Compare a failing request to a working request from a week ago. What is different?

In the founder’s case, the traces showed three different failure patterns appearing in the same week. The team had been treating them as one problem because the customer-facing symptom was the same: “the AI is broken.”

Failure mode 1: External dependency degradation

The most common production agent failure is an external dependency getting slower or less reliable. The agent itself is fine; the world around it changed.

Common culprits:

• LLM provider rate limits. OpenAI or Anthropic starts throttling because your traffic increased past your tier limit. Each request now retries 3 times before succeeding, tripling latency.
• Retrieval system slowdown. Your vector database is under more load than at launch and the p95 query latency went from 50ms to 800ms.
• External API drift. A tool you call (CRM API, billing system, search service) had a quiet update that changed response shape or timing.
• Knowledge base growth. Your KB has tripled in size since launch, and retrieval recall has dropped because you never tuned for the larger corpus.

The diagnostic: check your tool latency and error metrics for the past week. If any tool’s p95 latency is 2x what it was at launch, or its error rate is up more than 1%, that is the candidate.

The fix depends on the specific dependency. Rate limits: upgrade your tier or implement exponential backoff. Slow retrieval: tune the index or scale the database. API drift: update the integration. KB growth: re-tune chunking and retrieval parameters.

In the founder’s case, the LLM provider had pushed a quiet model update on Wednesday. The new model interpreted the routing prompt slightly differently, causing the agent to loop more often before settling on an answer. Average iterations went from 2.3 to 4.1. Cost and latency both jumped. The fix was a tighter routing prompt with three more few-shot examples.

Failure mode 2: Validation gates that aren’t being triggered

The opposite failure: a validation gate is supposed to be catching bad outputs, but it is not firing because the gate logic has a bug or the threshold is wrong.

Common patterns:

• Faithfulness threshold too low. Set at 0.5, the gate passes responses that are mostly hallucinated. Should be 0.85+ for production.
• Schema validation that allows nulls. The output schema requires “answer” field but allows it to be empty string. Empty answers ship to users as “I don’t know” without the agent realizing it failed.
• Toxicity filter not loaded. The filter library was supposed to be imported, but a refactor moved the import and now it silently no-ops.
• PII redaction running on the wrong field. Redacts user input but ships PII through in the response.

The diagnostic: look at a sample of bad responses customers reported. Trace what should have caught them. If a validation gate exists for that failure mode, check whether it actually fired.

In the founder’s system, the faithfulness threshold was set at 0.7, which was permissive. We tightened it to 0.85, the rejection rate went from 2% to 9%, and the customer complaints about wrong answers dropped immediately. The “rejected” responses were replaced with honest “I cannot find that in our knowledge base” messages, which users preferred to wrong answers.

Failure mode 3: Cost runaway from edge cases

Production query distribution is different from test distribution. Specific query patterns can be much more expensive than the average, and a few of those can dominate the bill.

The pattern: a small fraction of users (often 1-5%) generate a large fraction of cost (often 30-60%). Either through legitimate complex queries, abuse, or because their input triggers a degenerate code path in the agent.

The diagnostic: pull cost-per-user statistics for the last week. Sort descending. Look at the top 10 users. Are they sending normal queries? Or is one user looping their integration with bad inputs? Or is a specific query class (long documents, malformed input, multi-turn deep into rare topics) eating budget?

The fixes vary:

• Per-user rate limit and cost cap. Hard limit per user per day. Cuts off abuse without affecting legitimate use.
• Input length cap. Most LLM cost scales with input tokens. Cap user input at a reasonable max (say, 10k characters) and politely ask them to summarize for longer queries.
• Query type routing. If a specific query type is expensive, route it to a simpler/cheaper handler when possible. “Generate a comprehensive report” is expensive; route it to async batch processing instead of synchronous chat.
• Iteration cap per request. Prevent the agent from looping indefinitely on a single request. We default to 5-10 iterations max.

In the founder’s system, two users were sending repeated multi-paragraph product comparison requests, generating about 40% of the daily cost between them. We added a per-user daily cost cap and a length limit on inputs. Cost dropped 35% within 48 hours. Neither user complained because both were testing internal features and the cap was generous enough for normal use.

Failure mode 4: Silent quality drift

The hardest failure to detect: nothing is broken, no errors, latency is fine, cost is normal. But the responses are getting worse. Customers complain, the team cannot reproduce, and the dashboards all look green.

Causes:

• Prompt drift. Engineering edits to the prompt template that look small but change behavior. A removed example, a reworded instruction, a clarification that the LLM interprets differently.
• Model provider updates. As above, the underlying model can change without you knowing. Quality on your specific use case can shift in either direction.
• Corpus drift. Your KB has accumulated content that pollutes retrieval. Old documents that should have been deprecated still rank highly. New documents conflict with old documents.
• Eval set staleness. Your eval set was written six months ago against an older product version. It does not reflect what users actually ask now.

The diagnostic: run your eval pipeline against the current production agent. Compare against the score from launch. If the score has dropped, you have quality drift. If the score is the same but customers are complaining, your eval set has gone stale.

The fix: refresh the eval set. Sample 50-100 actual production queries, write expected answers for each, run the eval, and tune from there. Most teams refresh eval sets quarterly. Teams in fast-moving domains do it monthly.

What we shipped for the founder’s incident

The four-hour Friday-night triage:

1. External dependency: identified the LLM provider’s model update, tightened the routing prompt, agent stopped looping. Latency and cost recovered.
2. Validation gate: tightened the faithfulness threshold from 0.7 to 0.85, rejected responses now return honest “I don’t know” instead of hallucinations.
3. Cost runaway: added per-user daily cost cap of $5, input length limit of 10k characters. Cost dropped 35%.
4. Quality drift: ran eval, score had dropped from 0.84 to 0.71 since launch. Refreshed eval set with 50 recent production queries, identified three categories of questions the agent was failing, added documentation coverage for those categories. Score returned to 0.86 after a week.

Customer complaints stopped within 72 hours. The team’s mood went from “we built a broken thing” to “we built a thing that needs operational rigor we did not anticipate.” That second framing is the one that produces a better product.

The recommendation: rehearse the forensics

The founder’s team had no playbook for “the agent is broken in production.” They were debugging in panic mode, which slowed every step. After the incident, we wrote a one-page runbook with the four failure modes, the diagnostic for each, and the most common fixes.

Six weeks later, when a similar issue happened (a tool API outage), the on-call engineer worked through the runbook, identified the cause in 20 minutes, applied the documented fix, and was done in an hour. No panic, no escalation, no Friday-night call to a consultant.

This is what production agent operations looks like at maturity. Not “nothing ever goes wrong” but “when things go wrong, the team has a known process to find the cause.”

If your agent had a launch that went sideways

If your team launched an AI agent and the first few weeks have been more painful than expected, the right intervention is usually a forensic audit, not more development. Most launch issues are not new bugs in the agent code. They are operational gaps that surface only at production scale.

Sapota offers a one-week post-launch audit that walks through traces, validation, dependencies, and quality drift, identifies which of the four failure modes is responsible for which symptoms, and ships fixes plus a runbook for future incidents. We have done this for half a dozen B2B SaaS clients in the first three months after their AI launches.

Reach out via the AI engineering page with a description of what your agent does and what kind of failures you are seeing. The first conversation is usually the diagnostic.

SFMC’s built-in tracking is useful but incomplete. The system _Open, _Click, _Sent, _Bounce data views cover email events with SubscriberKey, timestamp, and basic identifiers. For most analysis, they’re enough.

The gaps show up when you need business context: which campaign this was part of, which DE it came from, what subject line was used, which product ID was personalized into the email. The defaults don’t store that.

Send Log DE is how you capture it.

What Send Log does

Send Log DE is a custom Data Extension that SFMC populates on every send, in addition to the default tracking. You define what fields to capture, and AMPscript inside the email writes the values into Send Log for each recipient at send time.

It’s additive – you’re not replacing the default tracking, just adding a parallel log with richer metadata.

Setup: create the Send Log DE

Create a new DE from the TriggeredSendDataExtension template (for Triggered Sends) or from scratch for other use cases. Typical fields:

The built-in fields (SendID, SubscriberKey, SendDate) populate automatically when Send Log is wired up. The custom fields are up to you.

Wiring Send Log into the email

In the email body, add AMPscript that writes to Send Log:

Runs once per recipient at send time. Produces one row per subscriber per send.

Send Log extends default retention

The default _SendLog system data view retains records for 10 days. A custom Send Log DE retains as long as your retention policy allows – 90 days, 1 year, or indefinitely.

For an audit or long-term reporting requirement (“show me all sends to this customer over the last 12 months”), a custom Send Log DE is how you satisfy it.

Archiving pattern

Pair Send Log with a nightly archive automation:

This handles the system _SendLog retention problem. Separately, the custom Send Log DE captures your additional fields at send time.

Things that don’t go into Send Log

Send Log writes at send time, which means you have access to:

• Subscriber attributes (AMPscript can read the DE)
• Static per-send values (subject, campaign ID)
• Personalization values used in the email

It does not have access to:

• Subsequent engagement (opens, clicks happen after the send, covered by default _Open / _Click)
• Bounce information (comes later, in _Bounce)
• Unsubscribe events (covered by _Subscribers status changes)

For joined analysis (“did the customer open the email that had ProductID X?”), join Send Log with _Open on SendID + SubscriberKey using SQL Query.

One gotcha: Test Send doesn’t write to Send Log

Test Sends skip the Send Log write. If you’re testing and expecting the custom Send Log to populate, it won’t. Only production sends write.

Helps when testing – you don’t pollute the log with test data. Confuses when you’re trying to debug Send Log wiring, because Test Send success doesn’t confirm Send Log works.

Takeaway

Send Log DE is the bridge between SFMC’s send infrastructure and your analytics or audit needs. The default tracking views answer “who opened, who clicked.” Send Log answers “which campaign, which subject, which product was in the email.” Five minutes of AMPscript at the top of the email template, and the analytics team gets data they can actually work with.

Setting up custom SFMC tracking or audit logs? Our Salesforce team designs Send Log strategies and reporting pipelines on production engagements. Get in touch ->

See our full platform services for the stack we cover.

Adding a custom button to a model-driven form’s command bar used to be a single-tool problem. Ribbon Workbench (a community XrmToolBox tool) was the answer for years. Microsoft then shipped Command Designer as the modern first-party replacement. For a while the two coexisted uneasily; the choice was political more than technical.

As of 2026, Command Designer has matured enough that it is the right first choice for new work, with specific exceptions. Here is the current comparison, and the migration pattern when you inherit a project with Ribbon Workbench customizations.

The current state

Command Designer (Power Apps maker portal):

• First-party, UI-based, no additional tool installation
• Uses Power Fx for command logic (visibility, action, label)
• Ships in the solution as commandDefinition entries
• Works with the “modern” command bar (the one on new model-driven apps)
• Does not work with the classic command bar

Ribbon Workbench (XrmToolBox community tool):

• Third-party, XML-based, requires installing XrmToolBox
• Uses JavaScript for command logic
• Ships as custom RibbonDiffXml in solution XML
• Works with both classic and modern command bars (but modern support is limited for newer features)

What Command Designer is good at

• Simple conditional visibility: “show this button if the record status is Draft” is a one-line Power Fx expression.
• Direct calls to Power Fx patchable operations: Update a field, launch a Canvas dialog, navigate to a record. No JavaScript glue needed.
• Source control friendliness: commandDefinition entries are structured XML in the solution; the diff is readable.
• New-team-member onboarding: makers who know Power Fx can author commands without learning JavaScript and the classic ribbon XML schema.

What Ribbon Workbench still wins at

• Complex conditional logic: visibility rules involving multiple fields, related record values, or security role checks are easier in JavaScript than in Power Fx.
• Integration with existing JavaScript: if your project has a form script library with utility functions, commands can call into it directly.
• Mature patterns for classic apps: if you maintain a classic app (Unified Interface but pre-modern command bar), Ribbon Workbench is the only practical tool.
• Fine control over ribbon structure: hiding out-of-box buttons, repositioning, custom groups – Ribbon Workbench’s XML editor exposes the whole surface.

Our decision rule

For new model-driven apps on the modern command bar: start with Command Designer. Use Power Fx for command logic. Only fall back to Ribbon Workbench if you hit a feature gap.

For existing apps on the classic command bar: stay with Ribbon Workbench until the app is migrated to modern. Mixing the two tools on the same app creates confusion.

For apps that explicitly need JavaScript logic: Ribbon Workbench if the logic is complex; Command Designer with a minimal JavaScript bridge if the logic is small.

Migration pattern: Ribbon Workbench to Command Designer

Projects inherit Ribbon Workbench commands that were authored years ago. Moving them to Command Designer is usually worth it when:

• You are upgrading the app to the modern command bar
• You are simplifying the team’s skill requirements (one tool, one language)
• The existing commands are maintenance burden (JavaScript with no tests, cryptic logic)

The migration is per-command, not all-or-nothing. Our process:

Step 1: inventory the existing commands. Export the solution, parse the RibbonDiffXml to list every custom command with its logic.

Step 2: rebuild each in Command Designer. Create the equivalent command in the maker portal. Power Fx the visibility and action logic. Test on a form.

Step 3: remove the Ribbon Workbench definition for that command. Edit the RibbonDiffXml to remove the old entry. Re-export the solution.

Step 4: deploy both changes together. The deploy adds the Command Designer command and removes the Ribbon Workbench command atomically.

Steps 2-4 per command. For a project with 15 custom commands, budget a day per complex command, a couple hours for simple ones. Four or five working days total.

What breaks in migration

JavaScript that called utility libraries. Commands that invoked shared JavaScript functions from the form library don’t translate directly to Power Fx. Two paths:

1. Rewrite the utility in Power Fx (if it’s simple enough).
2. Keep a small JavaScript bridge: the Command Designer command calls a Power Fx function, which calls into the JavaScript library via the Launch() function’s custom connector or a custom API endpoint.

Path 2 sounds hacky but works. For utility logic that is genuinely shared across forms (e.g., “format phone number by country”), keeping the JavaScript and bridging is simpler than rewriting.

Conditional visibility based on related records. “Show this button if the account’s parent has status X” requires a lookup from the current record to its parent, then a field check. In JavaScript this is a Xrm.WebApi call; in Power Fx it is a LookUp that may or may not delegate.

For simple parent checks, Power Fx works. For complex graph traversals, either denormalize the needed field onto the current table (via a rollup or plugin) or keep the command in Ribbon Workbench.

Button placement customization. Ribbon Workbench lets you specify exact positioning: “place this button to the left of Save, in the Actions group.” Command Designer’s placement controls are more limited – you pick a group, not an exact slot. If pixel-perfect placement matters, Ribbon Workbench retains an edge.

The one gotcha: solutions and ownership

Command Designer commands ship in the solution via new commandDefinition and commandComponent entities. Ribbon Workbench customizations ship via ribbon XML on the entity or application.

When both tools have touched the same entity’s ribbon, solution imports can get confused. The safe pattern:

• One tool per entity’s ribbon at a time. If migrating, finish the entity before moving on.
• Keep managed-solution layering in mind. A Ribbon Workbench customization from an older managed layer plus a Command Designer customization from the current solution can produce unpredictable behavior at runtime.

The one-year outlook

Microsoft’s direction is toward Command Designer as the canonical tool. New command-related features (command bar on Canvas components, AI-assisted command generation) are shipping to Command Designer first.

Ribbon Workbench will remain supported for the foreseeable future as long as classic command bar exists, but the feature gap is widening. For any project with a five-year horizon, investing in Command Designer skills over Ribbon Workbench is the right bet.

If you have a Ribbon Workbench-heavy codebase, start migrating the commands you touch anyway on feature work. Don’t do a big-bang migration unless you are already upgrading the app for other reasons. Incremental migration is the path that has worked across three projects we have moved.

Distributors processing hundreds of purchase orders daily across multiple warehouses need approval workflows that handle real variation: item categories with different approval thresholds, vendor risk tiers that escalate approval, amount-based routing, and exception handling for urgent procurements that can’t wait for the normal chain.

Three approaches get proposed. Two fail the “hundreds of POs daily” test the moment they’re tried at scale.

What doesn’t scale

Basic workflows with manual override allowances. Fine for small volumes where individual attention fits. Fails when buyers override thresholds casually to keep POs moving, eroding the internal controls the workflow exists to enforce. Audit findings follow.

Allow urgent orders to bypass workflows entirely, log approvals via email. Creates a parallel approval system that nobody can audit. Urgent becomes “whenever a buyer feels pressured.” Internal controls become advisory. Inevitable SOX or internal-audit finding.

Excel for procurement rules externally, upload approved POs to D365. Shifts the problem outside the system entirely. No enforcement in D365, no audit trail of why specific POs were approved, manual data entry at upload time. Scales poorly and introduces data integrity risk.

The pattern that handles the volume

Procurement and Sourcing workflows configured with rule-based conditions, exception escalations, and procurement category hierarchies.

Components of the working architecture:

Procurement categories are the taxonomy that drives category-specific approval. Office supplies, IT hardware, capital equipment, raw materials – each category can have its own approval rules. The category is set on each PO line; the rule engine evaluates against the line’s category.

Rule-based workflow conditions on the PO workflow:

• Amount thresholds (stepped: under $X, $X-Y, over Y)
• Category-specific thresholds (capital goes up faster than office supplies)
• Vendor performance rating (poor-rated vendor requires senior approval)
• First-time vendor flag (always requires senior approval)
• Buyer-specific overrides (certain buyers have higher blanket authority)

Exception escalation for urgent procurement:

• “Urgent” flag on the PO triggers a fast-path workflow with delegated approval
• Senior-buyer delegation list configured for after-hours approvals
• Exception is still workflow-approved; just routes differently than normal

Approval hierarchies tied to position – the approver is the person in a specific organizational position, not a named individual. When the person in the position changes, the workflow still works without reconfiguration.

Workflow scalability at hundreds-per-day volume

Hundreds of POs per day tests workflow performance. Specific configuration matters:

• Batch evaluation for workflow conditions – avoid complex X++ in conditions; precompute rule-relevant values in custom fields
• Parallel approval where multiple approvers can act simultaneously
• Conditional approval so low-risk POs skip unnecessary steps
• Auto-approval for well-defined routine categories within tight thresholds (e.g., office supplies under $500 to an approved vendor – no human approval needed)

Auto-approval isn’t laziness when it’s explicitly configured for scenarios that present no controls risk. The workflow still generates the audit record.

Vendor performance integration

Using vendor performance in approval rules requires:

• Vendor performance scoring in the vendor master, refreshed by a scheduled job pulling PO on-time, quality, and completeness metrics
• Score thresholds defining the risk tiers (green, yellow, red)
• Workflow condition reading the vendor’s current tier at PO submission
• Override with justification for buying from a red-tier vendor (sometimes unavoidable; requires extra approval + documented reason)

This makes vendor performance a live signal, not a quarterly review finding.

Exception handling for urgent procurement

The urgent path has to be fast and still auditable:

• Urgent flag requires buyer to record justification
• Fast-track approval routes to senior buyer or VP of procurement
• After-hours delegation – when primary approver is unavailable, pre-configured delegate can approve
• Post-hoc review – urgent POs flagged for procurement-committee review next week

Compliance accepts this because the audit trail is intact. Internal controls aren’t bypassed; they’re expedited.

Reporting and continuous improvement

The workflow produces data for ongoing tuning:

• Approval time per category – where’s the bottleneck?
• Urgent-flag frequency per buyer – is one buyer over-using the escape hatch?
• Vendor-performance-driven escalations – which vendors trigger frequent extra approvals? Time to negotiate or replace them.
• Threshold analysis – are the current thresholds too loose or too tight based on actual approval reject rates?

Quarterly workflow review with procurement leadership uses this data to refine rules.

Integration with budget controls

PO approval and budget control work together but aren’t the same. Budget control blocks POs that would exceed budget; approval workflow routes POs that are within budget for sign-off. Both need to fire:

• Budget check first – if it fails, PO can’t submit for approval
• Approval workflow second – if budget check passes, routes per rules
• Budget update on approval – reserves the amount so subsequent POs see reduced availability

Standard functionality, but the sequence and interaction need architect attention during configuration.

What ships with the pattern

A working high-volume PO approval setup has:

• Procurement category hierarchy defined and maintained
• Rule-based approval workflows with category, amount, vendor, and buyer dimensions
• Urgent-procurement fast path with documented controls
• Auto-approval configured for well-defined routine scenarios
• Vendor performance scoring integrated into approval conditions
• Budget control integrated with workflow
• Reporting on approval metrics for quarterly review

Standard D365 workflow engine. The architect’s work is in the rules, categories, and governance – not in custom workflow code.

One of the most confusing bugs in SFMC: the customer updated their email in the client’s system, the nightly file import brought the new address into the Data Extension, but emails still go to the old address.

We’ve seen this on multiple engagements. Takes someone who’s seen it before to diagnose in under an hour.

The two-list architecture

SFMC keeps email addresses in two independent places:

1. Data Extension – the email column of whichever DE the send uses.
2. All Subscribers List – the account-level master list, keyed by Subscriber Key.

When SFMC sends, and the subscriber’s Subscriber Key already exists in the All Subscribers List, it uses the email from the All Subscribers List – not the email in the Data Extension.

So even if the DE has the new email, the All Subscribers List still holds the old one. SFMC follows the old one, email goes to the wrong address.

The mental model: think of the DE as a working copy and All Subscribers as the master register. The Send Engine never trusts the working copy when the master has an entry for that Subscriber Key – the master’s value always wins.

Why it’s designed this way

The All Subscribers List is SFMC’s single source of truth for who a subscriber is at the account level. Subscribers can belong to multiple DEs and multiple sends, but their “identity” (Subscriber Key -> email) lives once in All Subscribers.

When you send from a DE, SFMC reconciles the DE rows against All Subscribers. If there’s a conflict (different email for the same Subscriber Key), the All Subscribers version wins.

This is actually useful in normal operation – it prevents inconsistent email addresses drifting between multiple DEs. It’s only surprising when you’re trying to update the email.

Audit: how many wrong sends are queued up?

Before fixing the pipeline, find out the size of the existing problem. Run this SQL Query Activity against your sending DE joined to the _Subscribers data view to count subscribers whose DE email no longer matches the All Subscribers email:

SELECT de.SubscriberKey, de.EmailAddress AS DE_Email, sub.EmailAddress AS AllSubscribers_Email FROM [Your_Sending_DE] de INNER JOIN _Subscribers sub ON de.SubscriberKey = sub.SubscriberKey WHERE de.EmailAddress <> sub.EmailAddress

Each row in the result is a subscriber currently set up to receive sends at the wrong address. If the count is high, that’s also the size of the silent goodwill loss already running through your account.

Save the result into a sync DE (call it Subscriber_Email_Sync) – it’s the input for the fix below.

The fix: update the All Subscribers List too

Updating the DE alone isn’t enough. You also need to push the email change into the All Subscribers List. Three methods, in order of how we usually deploy them:

Method 1: Scheduled sync via Import Activity

For ongoing pipelines, build an Automation that pushes the audit DE into All Subscribers:

Schedule (daily or hourly, after the upstream DE refresh) -> SQL Query Activity (the audit query above, output to Subscriber_Email_Sync DE) -> Import File Activity (target: All Subscribers List, NOT a DE) (update type: Add and Update) (key: Subscriber Key) (fields: SubscriberKey, EmailAddress)

The Import Activity targeting All Subscribers is the critical part. By default the destination picker drops you into a DE – you have to switch to All Subscribers explicitly at step 1 of the activity setup. This is the single click most teams miss the first time and then spend an hour wondering why the email column never updates.

Method 2: Direct UI import for one-off fixes

If you just need to fix a single batch of stale records (not set up an ongoing pipeline), use the Import Subscriber Wizard from Email Studio > Subscribers > All Subscribers > Import. Upload a file with Subscriber Key + EmailAddress. Update type: Add and Update.

The wizard does the same thing as the Import Activity – this method just skips the Automation Studio plumbing for ad-hoc cleanups.

Method 3: AMPscript UpsertContacts for real-time

For real-time updates from a Cloud Page, API call, or Journey Custom Activity, AMPscript has UpsertContacts() that pushes into the All Subscribers side directly:

%%[ var @rows set @rows = CreateArray() set @row = CreateObject(“Property”) SetObjectProperty(@row, “ContactKey”, @subscriberKey) SetObjectProperty(@row, “EmailAddress”, @newEmail) Add(@rows, @row) UpsertContacts(@rows) ]%%

Use this when DE updates are too slow for the use case (e.g. a customer changes email on a self-service portal and immediately triggers a confirmation email).

What about brand new subscribers?

If a Subscriber Key doesn’t yet exist in the All Subscribers List, SFMC auto-creates the record using the email from the DE on the first send. No override problem because there’s nothing to override.

The bug only appears when the Subscriber Key already exists with an old email value – which is also why it tends to surface late. Things look fine for weeks while the audience is mostly first-send subscribers, until enough customers update their email and a few of them complain.

Diagnosing it on a live ticket

When marketing ops files a “customer reports wrong address” ticket, the first place to check is All Subscribers – not the DE. Order of operations:

1. Query the DE for that Subscriber Key. Confirm DE has the new email.
2. Open Subscribers > All Subscribers in Email Studio, search for the Subscriber Key. Read the email value stored there.
3. If DE has new email but All Subscribers has old, you’ve found it.
4. Test: edit the All Subscribers record manually (it’s editable in the UI), do a test send. If it goes to the new address, diagnosis confirmed.

The screenshot shows the All Subscribers panel with the Subscriber Key column visible — this is the table you read against, not the DE, when diagnosing wrong-address tickets.

This is the single shift that saves teams about a day per case – skip debugging the DE, skip the import logs, go straight to All Subscribers.

Prevention on new engagements

Include an All Subscribers sync in the data pipeline design from day one. Whenever customer email updates flow into SFMC, they should update both the relevant DE(s) and the All Subscribers List. A separate Import Activity targeting All Subscribers runs alongside the DE import, on the same schedule.

Document this in the runbook so it doesn’t get dropped when someone new picks up the engagement. We add it to every SFMC handover doc precisely because it’s the kind of thing that takes a fresh engineer a full day to rediscover otherwise.

Takeaway

The DE is not the source of truth for email addresses – the All Subscribers List is. Update both when email addresses change. Build the sync into the data pipeline from day one, not as a patch after the first angry client call.

Shipping an SFMC subscriber management pipeline? Our Salesforce team sets up All Subscribers sync, Contact Builder attributes, and identity resolution on production engagements. Get in touch ->

See our full platform services for the stack we cover.

The Power Apps Component Framework (PCF) lets you drop a custom React component onto a Dataverse form or a Canvas app and treat it as if it were native. The official getting-started docs walk you through a working “hello world” in 20 minutes. The gap from there to something you would put into a managed solution and ship to production is wider than the tutorials admit.

We have built eleven PCF controls across client projects. Each taught us something we baked into a template repo. The current version of that template – React 18, TypeScript strict mode, Jest, ESLint, and a CI pipeline that produces a solution zip – is what we reach for on every new control.

Here is what is in it and why.

The shape of the template

Nothing exotic. The details matter more than the shape.

Why React and TypeScript (strict)

PCF originally shipped with plain TypeScript and DOM-native rendering. React is optional but heavily supported – Microsoft’s Fluent UI library assumes React, and the productivity gain for anything beyond a simple input is significant.

We use React 18 with the concurrent rendering features (createRoot rather than ReactDOM.render). This matters for controls that need transitions or suspense boundaries. If you ship with React 17, you opt out of features the newer Fluent UI components assume.

TypeScript strict mode is non-negotiable. PCF exposes ComponentFramework.Context, and the input/output types are generated from the manifest. Without strict mode, you can pass the wrong type to context.parameters.xxx and find out at runtime. With strict mode, the compiler rejects it.

The ControlManifest decisions

Three choices in the manifest make the template shareable across projects:

1. Virtual control (): the modern shape, React-aware, required for React 18 compatibility.
2. Dataset-bound or field-bound variants of the template: two separate starter branches because the Context API is subtly different between the two.
3. Resource declarations include a localization strings file (resx): every user-facing string gets a key, never hardcoded. Internationalization later is free.

Services: the wrapper layer over context.webAPI

PCF exposes context.webAPI for Dataverse calls. The raw API is callback-style, untyped, and verbose. Wrapping it in a typed service with Promise-returning methods is five minutes of work that pays off on every subsequent call.

Components import the service, never the raw webApi. Testing becomes possible because the service is a pure class that can be mocked.

The test harness

PCF’s runtime environment is not Node – it is a host page in Dynamics or a Canvas app. You cannot run a PCF control outside of that host directly. But you can test the React components and the service classes in isolation.

Jest plus @testing-library/react covers 80% of what matters:

The pattern: pass services in as props, not as new DataverseService(context.webApi) inside the component. Dependency injection at the top level makes every component test-mockable.

We require 70% line coverage on new PCFs. Not because the number is magic, but because a coverage floor forces the team to structure code so it can be tested.

ESLint rules that earn their slot

The base config we use:

no-floating-promises catches the most common PCF bug: calling webApi.retrieveRecord without awaiting it, which silently fails on error. no-explicit-any forces proper typing of the context API surface. exhaustive-deps catches stale-closure bugs in useEffect.

The CI pipeline

Three jobs run on every pull request:

1. Install + build: npm ci then npm run build (compiles TypeScript and runs the PCF build step). Fails if build errors.
2. Test: npm test (Jest) + coverage report. Fails if coverage drops below 70%.
3. Solution pack: uses pac pcf push or the CreatePowerPlatformSolution pipeline task to produce a solution zip artifact.

The solution zip is the artifact we deploy from. A pull request into main triggers a build that produces a versioned zip; the release pipeline then imports that zip into the target environment.

Things we stopped doing

• Bundling Fluent UI’s entire core library. The size hit on the PCF bundle is noticeable on slow connections. We import only the components we use (Button, TextField, specific icons).
• Running TypeScript in –strict false mode to “move faster.” It was always slower in the end.
• Skipping manifest versioning. The PCF manifest has a version field. Bumping it on every deploy was annoying until we had a PCF fail to load because the cached version matched the new version number – forcing users to hard-refresh. Now the version bumps automatically in CI.

When to not use PCF at all

Not every customization needs a PCF. Model-driven forms cover 80% of requirements with out-of-the-box components. Canvas apps offer drag-and-drop custom layouts with Power Fx. PCF is for:

• Interactive visualizations that Power Fx cannot produce (complex charts, map overlays)
• Third-party JavaScript library integrations (Chart.js, specific mapping SDKs)
• Custom form controls that replace a native field with richer UX (a date range picker, a search-autocomplete)

If the requirement maps to a Fluent UI or out-of-box component, use the native option. PCF adds a build pipeline, a deployment artifact, and a code surface to maintain. The threshold for choosing PCF should be “no other Power Platform option fits.”

The deliverable the client actually cares about

After all the engineering, the client handoff is: “here is a managed solution zip. Import it and your new field control appears on the form.” They do not know or care about React 18, strict mode, or Jest.

But every time they ask for a change six months later, the coverage, the linting, and the test suite are what make the change possible in hours instead of days. That is the real value of the template.

If you are starting your first PCF, skip past the hello-world tutorials and invest half a day in this shape. The hello-world code will become the first thing you have to throw away.

A SaaS founder asked us to second-opinion a roadmap his CTO had come back with after an agentic RAG vendor pitch. The plan was eighteen tools, a planner LLM, an executor LLM, a critic LLM, and a synthesis LLM, with autonomous multi-step reasoning over their support and CRM data. The vendor’s slides showed accuracy lifts from 67% to 89%. The implied infrastructure bill was around 5x their current model spend.

The product was a real-time chat assistant for their internal sales team. The latency budget was three seconds end-to-end. The team had not measured what fraction of production queries actually required multi-step reasoning.

We told the founder to ship a router first and the agent second, only on the queries that needed it. That conversation cut the projected agentic budget by roughly 80% while preserving most of the accuracy lift. Here is the framework.

What agentic RAG actually does

Naive RAG is a single retrieval followed by a single generation: embed the query, fetch the top-k, prompt the LLM, return the answer. Total cost is roughly two LLM calls (one to embed, one to generate). Total latency is the sum of the embedding, vector search, and generation.

Agentic RAG breaks the query into sub-tasks and runs the retrieve-and-reason loop multiple times. A typical pattern:

1. Planner breaks the query into sub-questions.
2. Retrieval runs once per sub-question.
3. Reasoner synthesizes intermediate results.
4. Executor decides whether to retrieve more, run a calculation, or call a tool.
5. Critic checks the answer against the question and triggers a revision if needed.
6. Synthesizer combines everything into the final response.

Each step is an LLM call. A typical agentic loop runs 3 to 8 LLM calls per query. Some patterns (ReAct with reflection, multi-agent collaboration) run 10 to 20.

The accuracy lift is real. On genuinely multi-step queries, agentic RAG outperforms naive RAG by 20 to 40 percentage points in published benchmarks and in the audits we have run. The cost lift is also real: 5x to 15x the LLM spend per query, plus 3x to 10x the latency.

The fraction-of-queries question

The conversation the vendor pitch never starts with: what fraction of production queries actually require multi-step reasoning?

For most B2B SaaS products we have audited, the breakdown looks roughly like this:

• 65 to 75% of queries are single-hop lookups. “What is the refund policy?” “How do I export reports?” “What does error code 502 mean?” These are textbook RAG questions. A naive pipeline answers them correctly at 85% recall and a fraction of the agentic cost.
• 15 to 25% of queries are multi-step. “Which enterprise customers complained about pricing in Q4?” “Compare the deployment guides for AWS and Azure and tell me which is faster to set up.” These benefit from agentic patterns. The accuracy lift is real and the latency hit is acceptable because the queries are inherently slower-thinking.
• 5 to 10% of queries are not RAG questions at all. They are SQL queries disguised as natural language (“total MRR by region last quarter”) or transactional requests (“create a ticket for me”). These should be routed to a dedicated agent or directly to the database, not to a RAG pipeline.

Running an agentic pipeline on the 65% to 75% of queries that are single-hop is paying 10x cost for no accuracy lift. The agent loop just does more steps to arrive at the same answer the naive pipeline produces.

What we ship instead

Sapota’s default for any production RAG product where agentic capabilities are on the roadmap:

Step 1: build the naive RAG pipeline first. Hybrid search, reranker, faithfulness gate, evaluation. This is the floor. Everything else is layered on top.

Step 2: add a query router. A small fast LLM (Llama 3.1 8B is plenty) classifies each incoming query as single-hop, multi-hop, or structured. This costs one extra LLM call per query, around 50ms of latency. The router accuracy needs to be measured and tracked, because misroutes are expensive in both directions.

Step 3: dispatch by class. Single-hop queries go to the naive pipeline. Multi-hop queries go to the agentic pipeline. Structured queries go to a SQL agent or whatever non-RAG tool fits.

Step 4: build the agentic pipeline only for the queries that need it. This is where the vendor pitch starts. The point is that the agent is not the entire product. It is one branch the router dispatches to.

The cost math: if 70% of queries hit the naive pipeline at $0.005 per query and 25% hit the agent at $0.05 per query, the blended average is $0.016 per query. If everything hits the agent, it is $0.05. The router pays for itself many times over.

When the agent is genuinely the product

Some products are built around multi-step reasoning by design. The agent is not an upgrade, it is the core value proposition. Cases we have seen and recommend full-agent for:

• Research assistants that take a complex question and produce a structured report (think: AI consultant for due diligence). The user expects 30-second responses. Multi-step reasoning is the value.
• Code agents that read a codebase, plan a refactor, and execute it across multiple files. The agent’s planning and tool-use capabilities are the entire product.
• Customer support agents that need to read account history, query backend systems, generate responses, and escalate when uncertain. The decision-making is the work.

For these products, the question is not whether to use agents, but how to keep the agent loop efficient (fewer steps, smarter tools, better caching).

For everything else (most B2B SaaS RAG products, most internal knowledge assistants, most customer-facing chatbots), the router-then-agent pattern is the right architecture.

The latency conversation

Agentic RAG is slow. A 5-step loop with 2-second LLM calls is a 10-second response. For batch use cases (overnight report generation, async customer email triage), this is fine. For real-time chat, it is the end of the user experience.

Mitigation patterns we use:

• Streaming intermediate steps to the user so the perceived latency is the time to first token, not the time to final answer. Users will wait if they see progress.
• Smaller models in the loop. The planner and the critic do not need GPT-4. Llama 3.1 8B or Haiku is enough. Reserve the expensive model for the final synthesis.
• Aggressive caching of intermediate results. Sub-questions that recur across queries (which they do, in narrow domains) hit a cache instead of re-running.
• Concurrent tool calls where the steps are independent. Most agent frameworks support parallel tool dispatch. Many implementations do not use it.
• Hard step limits. Cap the loop at 5 steps. If the agent has not converged by then, return a partial answer with a “I could not fully answer this; here is what I found” caveat. Better than a 30-second timeout.

These are the techniques that make agentic patterns viable for products with sub-10-second latency budgets.

What we sent the founder

The recommendation was three lines:

1. Ship the naive pipeline first. Spend two weeks getting it to 85% recall on the eval set.
2. Add the router as the second sprint. Measure misroute rate; aim for under 5%.
3. Build the agent only for the multi-hop branch. Start with a 3-step ReAct loop. Add reflection only if the eval shows it is worth the cost.

The vendor’s eighteen-tool plan was a v3 conversation, not a v1. By the time the team has the routing and the agent for the multi-hop branch working, they will know which tools they actually need. Most of the eighteen will not survive contact with the production query log.

The founder’s CTO sent the vendor a polite “let’s revisit this in six months” email.

If a vendor is pitching agents at you

If your team is being told that agentic RAG is the next quarter’s roadmap, the question to ask before signing is what fraction of your production queries actually need it. If the vendor cannot answer that with your data, they are pitching a product, not a solution.

Sapota offers a one-week query distribution audit that takes your production query log, classifies the queries by complexity, and produces a sized recommendation for which fraction needs agents and which is better served by naive RAG with a router. The output is a budget-and-architecture document the team can take to the vendor, the board, or both.

Reach out via the AI engineering page with your current RAG architecture and approximate query volume. The diagnostic conversation almost always reframes the conversation from “agents or not” to “how much of the product is agents and what is the rest.”