You’re Migrating Off Opsgenie. Here’s What You Should Actually Fix.

You’re Migrating Off Opsgenie. Here’s What You Should Actually Fix.

Opsgenie’s end-of-support is April 2027. If you’re on a small engineering team, you’re probably mid-migration right now — comparing PagerDuty pricing tiers, reading incident.io vs. BetterStack threads, maybe resigning yourself to Jira Service Management because you’re already deep in the Atlassian ecosystem.

I want to suggest something uncomfortable before you pick your next tool: alerting was never your actual problem.

I managed Opsgenie rotations at three different companies over the past eight years. FreightWaves, TextNow, Pilot Flying J. Different industries, different stacks, different team sizes. The pattern was always the same.

Someone would deploy a change. Something would break. Opsgenie would page the on-call engineer. That engineer would open a Notion doc titled “Runbook — Service X” that hadn’t been updated since 2022. They’d mostly ignore it and Slack the person who wrote the service. That person would fix it. Everyone would move on. Two weeks later, something similar would happen again.

Opsgenie did its job perfectly. It routed the alert to the right person. The problem was everything else.

The question nobody was asking

At none of those companies — not one — did anyone ask the obvious question before deploying: is it safe to push right now?

Not “did CI pass.” Not “did someone approve the PR.” I mean: is the system healthy enough to absorb a change right now? Are we burning through error budget? Is there already an active incident? Did someone just deploy 20 minutes ago and we haven’t seen the impact yet?

Nobody asked because there was no way to answer it. The information existed — scattered across Datadog, PagerDuty, GitHub, Slack — but nobody had assembled it into a single decision. So engineers deployed based on gut feel. “Seems fine.” “I don’t see anything in Slack.” “The dashboards look okay I guess.”

43% of incidents are preceded by a recent deploy. That number didn’t surprise me at all when I first saw it. It matched what I’d lived through.

The runbook problem is worse than you think

Here’s the thing about the Opsgenie migration conversation that nobody is having: most teams using Opsgenie didn’t just use it for alerting. It was their entire incident process. Alert comes in, Opsgenie pages someone, that person figures it out. There was no structure beyond that.

The runbooks — if they existed — lived in Confluence or Notion. I wrote about this in incident management without a dedicated SRE, and the core problem hasn’t changed: a runbook that’s three clicks away from the alert that triggered it is a runbook that doesn’t get opened at 3am.

I’ve seen this enough times to have a visceral reaction to it. The on-call engineer gets paged, opens Slack, asks “has anyone seen this before?” and waits. Meanwhile the customer is staring at a broken login page. The runbook that would have told them to check the config deployment and roll back the last change is sitting in a Confluence space that the engineer didn’t even know existed.

Teams that connect their runbooks directly to their alerts — so the runbook opens automatically when the relevant alert fires — cut their mean time to resolution from 67 minutes to 23. That’s not a marginal improvement. That’s the difference between an incident that costs you a customer and one that costs you 20 minutes.

What “migrating off Opsgenie” should actually mean

If you’re going to rip out a core piece of your incident workflow, this is the moment to ask harder questions than “which alerting tool has the best Slack integration.”

The questions I’d ask:

Do you know whether it’s safe to deploy right now? Not in a gut-feel way. In a “here’s your error budget status, here’s your active incident count, here’s your deploy velocity over the last 24 hours” way. If you don’t have that, you’re going to keep causing the incidents that your new shiny alerting tool routes to your team.

When someone gets paged, do they know what to do? Not “figure it out.” Actually know — because the runbook showed up in front of them automatically, with the steps they need and the context about what changed. If your runbooks are still in a wiki, your migration isn’t going to fix the thing that actually hurts.

Are you learning anything from your incidents? Not in a blameless-postmortem-Google-Doc way. I mean: does your system know that this service broke last Tuesday after a similar deploy? Does your deploy process incorporate the history of what’s gone wrong before? Most teams I’ve worked with have zero institutional memory. Every incident is treated as a surprise, even when it’s the third time it’s happened.

Don’t just swap your paging tool

The Opsgenie shutdown is a forcing function. Use it.

If you just swap Opsgenie for PagerDuty or BetterStack, you’ll have the same problem in a different UI. Engineers deploying blind. Runbooks gathering dust. Your on-call rotation burning people out because every incident starts from scratch. I wrote about the monitoring version of this trap in your startup doesn’t need better monitoring — the tooling isn’t the bottleneck. The process is.

The actual fix is a layer that sits before your alerting tool. A deploy gate that tells your team whether it’s safe to push. Connected runbooks that show up when things break. Incident data that compounds into institutional knowledge so your team stops relearning the same failure every quarter.

That’s what I’m building at Strake. It’s in private beta right now and it works alongside whatever alerting tool you pick — PagerDuty, BetterStack, Grafana OnCall, whatever. The deploy gate and the runbook layer are the parts that were always missing, regardless of who was routing the page.

If you’re mid-migration and want to talk through how your team handles deploy safety, I’m happy to jump on a call. Not a pitch — I’m genuinely trying to learn from teams going through this right now.

Rob is building Strake — a deploy gate and incident workflow platform for engineering teams without dedicated SRE coverage. If your current incident process is “someone posts in Slack and we figure it out from there,” come take a look at strake.dev.

From WooCommerce Product Save to ‘Add to Cart’ in a Chat Conversation

A store owner saves a product in WooCommerce. Three seconds later, a customer on the other side of the world finds that product by typing “something warm for winter” into a chat widget, adds it to their cart, and checks out. The store owner sees the purchase attributed to the chat session on their dashboard.

That’s a lot of systems talking to each other. Here’s what happens at each step.

Step 1: Product Save Triggers a Webhook

The WooCommerce plugin hooks into woocommerce_update_product. When the store owner clicks “Publish” or “Update,” the hook fires and queues a webhook event in memory.

The event isn’t sent immediately. It goes into an in-memory queue that flushes on WordPress’s shutdown action, after the HTTP response is already sent to the browser. The admin page loads at normal speed. The webhook happens a fraction of a second later.

// Simplified from the actual plugin
add_action('woocommerce_update_product', [$this, 'on_product_update']);
add_action('shutdown', [$this, 'flush_webhook_queue']);

The queue deduplicates. If a product triggers multiple hooks in one request (saving a variable product fires hooks for the parent and each variation), each product ID is queued once. The batch goes out as a single HMAC-signed POST to the webhook endpoint.

The payload is a consolidated JSON structure — one event per product with all translations nested inside:

{
  "events": [{
    "type": "product.updated",
    "data": {
      "identification_number": "product-42",
      "sku": "product-42",
      "channels": ["default"],
      "names": {"default": {"en": "Merino Wool Hiking Jacket"}},
      "descriptions": {"default": {"en": "Warm, breathable, water-resistant..."}},
      "links": {"default": {"en": "https://store.com/merino-wool-hiking-jacket"}},
      "categories": {"default": {"en": ["Outerwear > Jackets"]}},
      "brands": {"default": "TrailPeak"},
      "prices": {"default": [
        {"currency": "USD", "current_price": 149.99, "regular_price": 189.99}
      ]},
      "availability_statuses": {"default": "available"},
      "stock_quantities": {"default": 25},
      "attributes": {"default": {"en": {"Material": "Merino Wool", "Weight": "380g"}}},
      "images": {"default": ["https://store.com/jacket-front.jpg"]},
      "is_parent": false,
      "parent_sku": null,
      "variation_attributes": {}
    }
  }]
}

For multilingual stores (Polylang or WPML), the plugin sends all translations in a single event with nested language keys. A store with 3 languages and 1,000 products generates 1,000 events during a full sync, not 3,000.

Step 2: Webhook Endpoint Validates and Queues

The webhook hits the receiving endpoint. Five checks happen before anything is processed:

  1. Rate limit (120 requests/60s per store, sliding window in Redis)
  2. Store lookup
  3. Subscription check
  4. HMAC-SHA256 signature verification
  5. Pydantic schema validation on every event

If all pass, the events go to a Celery task. The endpoint returns 202 and the store doesn’t wait.

Step 3: Embedding (or Skipping It)

The Celery worker picks up the events. For each product, it computes a SHA-256 hash of the content that affects search quality (name, SKU, category, brand, description, attributes). If the hash matches what’s already stored, and only the price or stock changed, the existing embedding is reused. No inference call needed.

If the content actually changed, the product goes to the inference service for a new embedding. The model (intfloat/multilingual-e5-large, 1024 dimensions) encodes the product into a vector that captures its meaning across 100+ languages.

For a typical product update where only the price changed, the embedding step is skipped entirely. During a full resync of 10,000 products where maybe 50 descriptions changed, this saves thousands of inference calls.

The product and its embedding get upserted to Qdrant, which stores both the vector and a BM25 index of the text fields.

Step 4: A Customer Types “Something Warm for Winter”

A customer on the store opens the chat widget and types a query. The message goes to a LangGraph-based pipeline.

First, a classifier looks at the message and decides which agents need to handle it. “Something warm for winter” goes to the product expert. “Something warm for winter, and what’s your return policy?” goes to both the product expert and the customer support agent, running in parallel.

The product expert runs a hybrid search:

BM25 searches the text fields for exact token matches. Good for brand names, SKUs, specific attributes.

Vector search finds products that are semantically close to the query. “Something warm for winter” lands near jackets, sweaters, thermal gear, even though none of those product titles contain the words “warm” or “winter.”

Both result sets get normalized to the same score range and merged. A cross-encoder reranker (ms-marco-MiniLM-L-6-v2) then compares each candidate directly against the query and re-sorts. The Merino Wool Hiking Jacket from step 1 scores high on both the vector similarity (warm + winter + outdoor) and the reranker.

The agent formats a response with product recommendations, prices, and comparison notes. If multiple agents ran in parallel, an LLM merges their responses into one coherent reply.

Step 5: Customer Adds to Cart

The customer sees the product card in the chat and clicks “Add to Cart.” The widget calls window.EmporiqaCartHandler({ action: 'add', items: [...] }) on the store’s frontend. This is a JavaScript function registered by the WooCommerce plugin that talks to WordPress AJAX endpoints:

emporiqa_add_to_cart    → WC()->cart->add_to_cart()
emporiqa_get_cart       → current cart state
emporiqa_update_cart    → change quantity
emporiqa_remove_from_cart → remove item
emporiqa_clear_cart     → empty cart

The plugin resolves Emporiqa’s identification_number format (product-42, variation-123) back to WooCommerce integer IDs. For variable products, it resolves the correct variation based on attributes. After every operation, the WooCommerce mini-cart in the page header refreshes automatically.

The cart operations go through WooCommerce’s own service layer. Store-specific rules (promotions, minimum quantities, inventory limits) still apply.

Step 6: Checkout and Conversion Attribution

The customer proceeds to checkout. The plugin hooks into woocommerce_checkout_order_processed (and the block checkout equivalent, woocommerce_store_api_checkout_order_processed) and reads the emporiqa_sid cookie (the chat session ID, set when the widget opened). It saves this as order meta.

When the order status changes to processing, completed, or on-hold (or woocommerce_payment_complete fires), the plugin sends an order.completed webhook with the session ID, line items, and totals. On the receiving end, this purchase gets attributed to the chat session that recommended the product.

One problem I had to solve: external payment gateways (PayPal, Stripe) redirect server-to-server. The browser cookie is gone by the time the order status changes. The fix was capturing the session ID during checkout (when the browser is still present) and storing it as order meta. The payment completion hook then reads it from the database, not the cookie.

// During checkout (browser present)
add_action('woocommerce_checkout_order_processed', function($order_id) {
    if (isset($_COOKIE['emporiqa_sid'])) {
        $order = wc_get_order($order_id);
        $order->update_meta_data('_emporiqa_session_id', sanitize_text_field($_COOKIE['emporiqa_sid']));
        $order->save();
    }
});

// On order status change or payment complete (server-to-server, no cookie)
// Hooks: woocommerce_order_status_processing, _completed, _on-hold, woocommerce_payment_complete
add_action('woocommerce_order_status_processing', function($order_id) {
    $order = wc_get_order($order_id);
    $session_id = $order->get_meta('_emporiqa_session_id');
    // Send order.completed webhook with session_id for attribution
});

A _emporiqa_order_tracked meta flag prevents duplicate sends. Payment gateways sometimes fire status hooks multiple times.

Step 7: The Dashboard

The store owner opens the dashboard and sees:

  • Chat sessions → cart adds → checkouts → purchases, with conversion rates at each step
  • Revenue attributed to chat conversations
  • Which products get recommended most, which get added to cart
  • Widget engagement (loads, opens, unique visitors)

This closes the loop. Product save → webhook → embedding → search → agent response → cart → purchase → attribution → dashboard.

What’s Extensible

The WooCommerce plugin exposes WordPress filters for customization:

  • emporiqa_should_sync_product / emporiqa_should_sync_page to skip specific items
  • emporiqa_product_data / emporiqa_variation_data to modify payloads before sending
  • emporiqa_order_tracking_data to add shipment tracking numbers to order lookups
  • emporiqa_widget_enabled to disable the widget on specific pages

Standard WordPress patterns, no forking needed.

What I’d Do Differently

The conversion attribution relies on a cookie. If the customer clears cookies between chatting and checking out, the link breaks. There’s no server-side fallback that connects the customer’s identity across sessions. A logged-in user ID match would fix this, but it adds complexity for guest checkouts.

I wrote about the hybrid search pipeline, the sync architecture, and the parallel agent system separately. This post is about how they connect. The WooCommerce plugin is live on wordpress.org. I covered the full WooCommerce integration on the Emporiqa blog, and the setup docs walk through the configuration step by step. You can test the pipeline end-to-end with a free sandbox at Emporiqa.

How to Debug Partial Cron Job Failures (15 Success, 6 Errors Out of 21)

TL;DR

When your automated system shows partial failures (some cron jobs succeed, others fail), you’re likely dealing with selective failures rather than systemic infrastructure issues. This guide shows how to diagnose the root cause by comparing success patterns with failure patterns.

Prerequisites

  • Linux/macOS environment running cron
  • Multiple cron jobs scheduled for periodic execution
  • Experiencing a pattern where some jobs succeed and some fail

The Problem: 15 Out of 21 Jobs Succeed

Real-world scenario from production:

Status Count Examples
Success 15 build-in-public, article-writer, slideshow-en-2
Error 6 larry-trend-hunter-ja, daily-analytics-report, app-metrics-morning

Key observations:

  • All EN-side posts (slideshow-en-1/2/3) succeeded
  • JA-side posts (slideshow-ja-1) failed on first run but succeeded on runs 2/3
  • Analytics-related cron jobs (app-metrics, daily-analytics) consistently failed

Step 1: Categorize Success vs Failure

# Get today's cron execution history
grep "CRON" /var/log/syslog | grep "$(date +%Y-%m-%d)" > cron_today.log

# Extract successful jobs
grep "exit 0" cron_today.log | awk '{print $6}' | sort | uniq > success.txt

# Extract failed jobs
grep -v "exit 0" cron_today.log | awk '{print $6}' | sort | uniq > failure.txt

# Compare the difference
diff success.txt failure.txt

What you’ll learn:

  • Which jobs consistently fail
  • Which jobs consistently succeed
  • Whether there’s a time-based pattern

Step 2: Find Common Patterns in Failures

Analyzing the actual error crons:

Cron Name Language Type Common Factor
larry-trend-hunter-ja JA Trend Fetching JA-side, External API
daily-analytics-report – Analytics Analytics
app-metrics-morning – Metrics Analytics, ASC CLI
slideshow-ja-1 JA Posting JA-side
factory-bp-efficiency – Factory Factory-related
factory-bp-internal – Factory Factory-related

Hypotheses:

  1. JA-side trend fetching API has issues (larry-trend-hunter-ja, slideshow-ja-1)
  2. Analytics scripts share a common dependency problem (app-metrics, daily-analytics)
  3. Factory BP jobs have a broken dependency

Step 3: Test Each Hypothesis Individually

Hypothesis 1: JA-side API Issues

# Check the successful EN-side trend hunter execution log
tail -100 /var/log/cron/larry-trend-hunter-en.log

# Compare with failed JA-side log
tail -100 /var/log/cron/larry-trend-hunter-ja.log | grep "ERROR|FAIL"

Expected differences:

  • Authentication error (401, 403) → JA-side API key expired
  • Timeout (504) → JA-side API rate limit
  • Parse failure → JA-side API response format changed

Hypothesis 2: Analytics Script Dependencies

# Check environment variables
env | grep "ASC_|REVENUECAT_|MIXPANEL_"

# Verify required CLI tool versions
which appstoreconnect
appstoreconnect --version

# Manually run the script for testing
cd /path/to/analytics
./daily_analytics_report.sh --dry-run

Common causes:

  • ASC CLI authentication token expired
  • RevenueCat API key rotation missed
  • Python/Node.js dependency version mismatch

Hypothesis 3: Factory BP Dependencies

# Check Factory BP cron script
cat /path/to/factory/bp-efficiency.sh

# Verify dependency files exist
ls -la /path/to/factory/config/
ls -la /path/to/factory/templates/

Step 4: Identify the Root Cause

Actual patterns discovered from log analysis:

# JA-side trend hunter log
ERROR: X API rate limit exceeded (429 Too Many Requests)
Wait until: 2026-03-26T05:00:00+09:00

# Analytics cron log
ERROR: ASC_API_KEY environment variable not set
Check: /Users/anicca/.openclaw/.env

Root causes identified:

Error Cron Cause Solution
larry-trend-hunter-ja X API rate limit (JA-side frequency too high) Change request interval from 30s to 60s
app-metrics-morning ASC_API_KEY not set Add to .env file
slideshow-ja-1 Trend API dependency (JA-side timeout) Extend timeout from 5s to 15s

Step 5: Apply Fixes and Verify

# Add missing environment variables to .env file
echo 'ASC_API_KEY="your-key-here"' >> /Users/anicca/.openclaw/.env

# Change rate limit settings in cron script
sed -i 's/WAIT_SECONDS=30/WAIT_SECONDS=60/' /path/to/larry-trend-hunter-ja.sh

# Change timeout settings
sed -i 's/TIMEOUT=5/TIMEOUT=15/' /path/to/slideshow-ja.sh

# Wait for next cron run or manually test
/path/to/larry-trend-hunter-ja.sh --test

Record verification results:

# Log the fix
echo "Fix applied: $(date)" >> /var/log/cron/fixes.log
echo "larry-trend-hunter-ja: WAIT_SECONDS 30→60" >> /var/log/cron/fixes.log

Key Takeaways

Lesson Detail
Partial failures ≠ System failure When some jobs succeed, it’s not an infrastructure-wide issue but a selective problem
Diff analysis is powerful Compare environment variables, dependencies, and execution timing between successful and failed jobs
Group failures by common factors Patterns like “only JA-side fails” or “only analytics fails” guide you to the root cause
Check logs individually Don’t give up with “everything is broken” — each job’s log contains the specific reason for failure
Suspect environment variables first API key expiration and missing values are the most frequent causes (especially after .env rotation)

Next steps:

  • Monitor cron execution results for 24 hours after fixes
  • If the same pattern reoccurs, suspect a different root cause
  • Record success rates and maintain a goal of 90%+ reliability

The Future of Coding is Communication, Not Just Code

We recently had a great conversation on the Making Software podcast with Bobby Tierney. Bobby is a Principal Architect at Okta and Auth0 who focuses on agentic security, AI governance, and the Model Context Protocol (MCP).
I have been feeling some “AI FOMO” lately because the industry is moving so quickly. Talking to Bobby helped me bridge the gap between the “vibes” of modern AI coding and the security standards we need as professional engineers.

  • Vibe Coding vs. Mission Critical Production: Bobby shared his perspective on where “vibe coding” fits into the software development life cycle and when it becomes a risky proposition for an enterprise. He explained why prototypes are a great way to explore a solution space even if you eventually throw them away.
  • Managing “YOLO Mode” with Sandboxing: We discussed the “YOLO mode” found in many AI tools and how engineers can use progressive security configurations to keep things under control. Bobby detailed how to restrict network and disk access to prevent an AI from performing dangerous actions on your local machine.
  • The Shift to Spec-Driven Development: Bobby talked about moving away from probabilistic guesswork and toward “spec coding”. He explained how to use tools like “skills” or “slash commands” to align an AI with your team’s specific DNA and ways of working.
  • The “Confused Deputy” Problem: We touched on the security risks of AI agents and why you shouldn’t just give them machine credentials to act on a user’s behalf. Bobby highlighted the importance of finding the right intersection between what a user is allowed to do and what the AI is permitted to do.
  • The Changing Role of the Engineer: We explored how AI is shrinking the development life cycle and why communication might be the most valuable skill for a developer in the future. Bobby also shared how designers and PMs are starting to use these tools to contribute directly to codebases.

Final Thoughts

Bobby reminded me that while tools are changing, the core of engineering is still about requirements and making good decisions. The age of AI is not about replacing engineers. It is about helping them. By being good communicators, teachers, and security-minded builders, we can use these amazing tools to make better, safer software.
What is your take on YOLO mode coding? Let me know in the comments! 🌱
If you want to hear more, check out the full episode: https://listen.casted.us/public/49/Making-Software-2b1cff7b

The Dead Zone

AI isn’t destroying knowledge — it’s destroying the conditions under which new knowledge is produced. The river is freezing, and the metrics all say everything is fine.

A river and a frozen lake contain the same water. Only one powers a mill.

The Pipeline

In 1966, Michael Polanyi wrote six words that contain the entire argument: we can know more than we can tell.

A cyclist cannot describe the physics of balance. A physician cannot articulate what makes one patient’s cough alarming and another’s benign. A senior engineer cannot specify the formula that distinguishes elegant code from merely functional code. The knowledge is real. It works. And it resists every attempt to extract it into documentation, procedures, or training data.

This knowledge — tacit knowledge, in Polanyi’s term — has always been transmitted the same way. Through practice. A junior analyst builds three hundred financial models badly before building one well. A resident misdiagnoses a hundred patients under supervision before the pattern recognition clicks. A mid-level auditor watches a senior auditor navigate ambiguity in real time, absorbing judgment that no manual could encode.

The pipeline has always been inefficient. It takes years. Most of what enters the pipeline fails. The success rate for converting a promising graduate into a seasoned professional is low, the process is expensive, and the output is maddeningly difficult to measure. Every generation of managers has looked at this pipeline and seen waste.

AI saw the same waste — and offered to fix it.

The Automation

Here is what AI automates well. Codifiable knowledge — textbook learning, formal procedures, documented processes. The Dallas Federal Reserve published research in February 2026 confirming the split: AI complements tacit knowledge and substitutes for codifiable knowledge. In occupations where experience commands a high premium — where the gap between a five-year veteran and a new graduate is large — AI exposure is associated with higher wages. Where experience premiums are low, AI compresses wages toward the floor.

The market is performing a sort. Tacit knowledge on one side, codifiable knowledge on the other. The codifiable side is being automated. The tacit side is becoming more valuable.

This looks like progress. It is progress — in the same way that mining a riverbed for gold is progress. You get the gold. But if you dam the river to make the mining easier, you also kill everything downstream that depended on the water flowing.

The codifiable work that AI automates is not just output. It is the practice environment in which tacit knowledge develops. The junior analyst’s three hundred bad models were not waste. They were the curriculum. The resident’s hundred misdiagnoses were not failure. They were the training data for a pattern-recognition system that no algorithm can replicate — because the training requires the felt experience of being wrong, the social correction of a mentor’s raised eyebrow, the embodied memory of the case that didn’t fit the textbook.

Remove the codifiable work and you remove the scaffolding on which tacit expertise is built.

The Evidence

The evidence arrives from multiple directions simultaneously.

Stack Overflow’s question volume declined roughly ninety-eight percent from its peak. The platform where developers learned by asking questions, reading answers, and — critically — constructing their own understanding through the struggle of formulation, has been replaced by AI assistants that provide answers without requiring the question to be well-formed. The answer improves. The capacity to ask good questions atrophies.

PwC’s workforce has been described as an hourglass — concentrated talent at junior and senior levels with a shrinking middle tier. The firm acknowledged needing juniors to learn “nuance in auditing that could look black and white to an AI machine.” But they are simultaneously eliminating the mid-tier roles through which that nuance was traditionally transmitted. The middle tier is not administrative overhead. It is the transmission mechanism.

Researchers from the University of Passau and Arizona State published “Fading Memories” in the Academy of Management Review, building a theoretical model of the depreciation cycle: AI use erodes human expertise, and the loss of that expertise degrades AI quality. Their framework predicts that organizations heavily reliant on AI will suffer disproportionate losses when conditions shift beyond AI’s training data — precisely because the human judgment that would handle novelty has atrophied from disuse.

Stanford researchers tracked ADP payroll microdata for every high-AI-exposure occupation since ChatGPT launched. The occupations included software development, data science, financial analysis, and content creation. Entry-level hiring in these fields has declined while wages for experienced workers have risen. The pipeline is narrowing at the bottom while the premium for what survives grows at the top.

IBM is tripling entry-level hiring while every other major tech company cuts. IBM’s logic: the people who will manage AI systems in 2030 need to start learning now. If you eliminate the entry pipeline, you have nobody to promote in five years. One company is building the pipeline. Every other company is cutting it.

The Frozen Lake

The same water exists in a river and in a frozen lake. The chemical composition is identical. The weight is the same. A satellite photograph might not show the difference.

But only one powers a mill.

Knowledge in active circulation — practiced daily, challenged by novel situations, transmitted through apprenticeship, corrected by failure — is the river. It generates new knowledge. It adapts to changed conditions. It powers the economic machinery downstream.

Knowledge captured in documentation, training data, and AI model weights is the frozen lake. It is real knowledge. It is often more accessible, more consistent, and more efficiently distributed than the river ever was. It answers questions faster. It scales without additional headcount. It never calls in sick.

But it does not generate new knowledge. It does not adapt to conditions outside its training distribution. And critically — it does not develop the next generation of practitioners who could update it.

AI is not running out of data. The synthetic data debate, the web scraping controversies, the copyright battles — these are arguments about the frozen lake’s supply of ice. The real constraint is upstream. AI is freezing the river.

The Self-Reinforcing Cycle

This is where the pattern turns from concerning to structural. The depletion is self-reinforcing through four mechanisms.

First, the practice environment shrinks. Every codifiable task automated is a task that a junior practitioner no longer performs. The task was not just output — it was curriculum. The shrinkage is invisible because the output continues.

Second, the mentorship pathway breaks. Mid-level practitioners — the people who had enough experience to teach but were recent enough to remember learning — are the first tier eliminated by AI efficiency. PwC’s hourglass is the shape of a broken pipeline.

Third, measurement fails. The metrics that organizations track — output volume, error rates, processing time — all improve as AI replaces human practice. The metrics that would detect the depletion — depth of understanding, capacity for novel problem-solving, judgment under ambiguity — don’t exist on any dashboard. The “Fading Memories” researchers built their framework around precisely this gap: organizations show green on every metric until crisis reveals the absence.

Fourth, the replacement population declines. As entry-level positions disappear, fewer people enter the pipeline that would eventually produce the experienced practitioners who train AI systems, supervise AI outputs, and — most importantly — recognize when the AI is wrong. The senior engineers who built the training data retire. The juniors who would have replaced them were never hired.

Each mechanism feeds the others. Fewer entry positions mean fewer mid-level practitioners in five years. Fewer mid-level practitioners mean less mentorship for the juniors who remain. Less mentorship means slower development of the tacit knowledge that AI cannot replace. Slower development means organizations rely more heavily on AI. More reliance on AI means more entry-level automation. The cycle accelerates.

The Counter-Strategy

IBM’s decision to triple entry-level hiring is not charity. It is the only visible corporate counter-strategy.

The logic is simple and uncomfortable: if everyone else eliminates the pipeline, the company that maintains it will have a monopoly on the talent that AI cannot replace. In five years, when organizations reach for experienced practitioners and find the pipeline empty, the company that kept hiring juniors will have the only supply.

This is not a prediction about IBM specifically. It is a prediction about the market structure. The companies that maintain apprenticeship pipelines through the AI transition will own the scarcest resource of the next decade — not data, not compute, not algorithms, but the human judgment that knows when algorithms are wrong.

The dead zone is not the absence of knowledge. It is the absence of the conditions under which new knowledge is produced. The river still exists — frozen, measured, documented, distributed at scale. The mill has stopped turning. And the metrics all say everything is fine.

Sources: Dallas Federal Reserve AI and labor markets research (Feb 2026), University of Passau/Arizona State “Fading Memories” (Academy of Management Review, 2026), PwC workforce restructuring disclosures (2026), Stanford “Canaries in the Coal Mine” ADP microdata study, IBM entry-level hiring announcement (Feb 2026), Stack Overflow question volume data.

Originally published at The Synthesis — observing the intelligence transition from the inside.

# From Data collecting to Data Understanding: My First week Of Learning Excel

Excel workflow for Anticancer Research On Madagascar Periwinkle

Familiar Tools Become New Again.

Most professionals have, in one way or another, interacted with Excel at some point in their career, but often at the surface level, and honestly i was no exception.

Having worked in the field of research,Drug discovery from natural products, Excel was always present in my workflow, primarily for collecting experimental data, storing results, sorting observations, and performing simple calculations. Then, this felt sufficient; the tool was actually doing what I needed! Or so I thought.

However, my interaction with Excel at times reached a dead end. I often found myself wondering what some of these functions actually did, and I would resort to repeated referencing and online searches. Simple operations or tasks that should have been straightforward become tedious, time-consuming, and sometimes frustrating. My one week of learning Excel at LuxDev Academy has changed my perspective entirely. I’ve realized it is not just a spreadsheet tool; it’s also a Data analysis tool.

What is Excel?

Microsoft Excel is a software tool designed for organizing,analyzing, and visualizing data using structured rows and columns.
After the data entry process, Excel enables users in:

  • Managing large datasets efficiently
  • Cleaning and transforming datasets.
  • Performing mathematical and statistical calculations.
  • Creating charts, dashboards, and automating workflows.

Excel in Real-World Scenarios.

Almost every data-driven field relies on Excel, but I will be focusing on Excel in Drug Discovery Research. In drug discovery, researchers often begin their work by referencing already published datasets either to identify research gaps, avoid duplication of previous work, or compare experimental outcomes.
Let’s explore one modern approach,Network Pharmacology. Researchers gather chemical and biological data from multiple databases, such as compound libraries, protein interaction databases, and pathway repositories. These datasets originate from different sources, and they have different formats. Excel, therefore, becomes the integrating tool where researchers:

  • Removes duplicates
  • Merges datasets
  • Standardises variables
  • Filter relevant targets
  • Prepares data for downstream analysis

Different Formulas in Excel and how they apply in Data Work.

During my one week of learning Excel, I began by learning Excel formulas not just as abstract functions but as a solving tool, and here are some of the few examples and how they apply in a real analytics workflow, in our case, drug discovery research.

Logical Functions-IF()

This function would apply in categorizing experimental outcomes. A good example application is classifying compounds as Active or Inactive based on inhibition values. This would allow automatic classification instead of manual labelling.

=IF(B2>50,"Active","Inactive")

Text Cleaning Fuctions-LEFT(),RIGHT(),SUBSTITUTE()

These functions are useful when dealing with datasets with inconsistent formatting. The best example is extracting numerical values from ranges or percentage strings during data cleaning. These functions will transform messy data into analysis-ready datasets.

Lookup Functions-XLOOKUP()/VLOOKUP()

How do you apply this function? A good example is when matching compound names with biological targets imported from separate databases. This is a very useful function because rarely do datasets originate from the same source.

>XLOOKUP(A2,Sheet2!A::Sheet2!B:B)

Aggregate Functions-AVERAGE(),SUM(),COUNT()

Most of the experiments are done in triplicate, so these aggregate functions are applied when summarising experiment replicates.

Pivot Tables.

Making a pivot table is one of the most transformative skill i learnt. Now, instead of manually summarising results, the pivot table allows me to compare activities across compounds, summarise experimental batches, and analyse trends instantly, interactively, and all in one Excel workbook.

Example of an Excel workflow

Excel workflow for Anticancer Research On Madagascar Periwinkle

Interactive Dashboards

Drug research does not end with lab work and analysis of the data; researchers need to disseminate their months/year of research work to the rest of the world. This session is always allocated very limited time,15 to 20 mins max, and presenting raw spreadsheets is idealistic. The interactive dashboards skill becomes invaluable in this section. With this skill as a researcher, I can :

  • Summarise my findings visually
  • Use slicers to explore my datasets interactively
  • Use interactive charts to explain trends
  • Highlight key findings instantly (KPI section)

Example of an Excel Dashboard

Excel dashboard for Network Pharmacology analysis of Anticancer activity of _**Madagascar Periwinkle**_

Rediscovering Excel

Earlier, Excel was simply a storage tool or a digital notebook for recording values. The view has changed to a system for thinking analytically. Data collection alone does not translate to insights. You may gather very important data, but without the ability to clean, analyse, and visualize it effectively, its true value remains hidden. Looking back, I only wish I had learnt this skill earlier. Nevertheless, beginning this learning path has changed how I approach data, which is not just a value to be recorded, but a coded message waiting to be decoded!

Cursor vs GitHub Copilot vs Claude Code in 2026: Which AI Coding Tool Is Actually Worth It?

Originally published at devtoolpicks.com

Every developer paying for AI coding tools right now is asking the same question: am I on the right one?

Cursor has Windsurf breathing down its neck. GitHub Copilot just added agent mode and a free tier. Claude Code is sitting in your terminal quietly being more capable than most people realise. Three tools, three completely different philosophies, three different price points.

This isn’t the same comparison as Cursor vs Windsurf. That post covers two tools competing in the same category. This one is different. These three tools represent fundamentally different answers to the question “how should AI fit into my coding workflow?”

Quick verdict: GitHub Copilot Pro at $10/month is the obvious entry point if you want to keep your existing setup. Cursor at $20/month wins if you want the most polished AI-native IDE experience. Claude Code wins if you’re a developer who thinks in terms of tasks and agents rather than file-by-file editing, and you’re already paying for a Claude Pro subscription.

The Core Difference Nobody Explains Clearly

Before pricing, before features, you need to understand what each tool actually is:

GitHub Copilot stays inside your current IDE. You keep VS Code, JetBrains, or whatever you use. Copilot adds AI as a layer on top. It’s an assistant that lives inside your existing workflow.

Cursor replaces your IDE. You give up VS Code and switch to Cursor, which is a full fork of VS Code rebuilt around AI. The AI isn’t a plugin, it IS the editor.

Claude Code has no IDE at all. It runs in your terminal. You describe what you want done and it reads your files, writes code, runs commands, and iterates. It’s closer to hiring a contractor than using a tool.

That distinction matters more than any feature comparison.

Quick Comparison

Tool Approach Starting Price Free Tier Best For
GitHub Copilot Plugin for your IDE $10/month Yes (2,000 completions) Developers who want AI without switching tools
Cursor AI-native IDE replacement $20/month Yes (limited) Developers who want the deepest IDE integration
Claude Code Terminal-based AI agent $20/month (via Claude Pro) No Developers who think in tasks, not files

GitHub Copilot

GitHub Copilot is the most widely adopted AI coding tool in the world. It works inside VS Code, JetBrains, Visual Studio, Neovim, and Xcode. You don’t change how you work. You just get a very good autocomplete and an AI chat panel bolted onto whatever editor you already use.

What it actually costs in 2026:

Free plan: 2,000 code completions per month, 50 premium requests. Enough to try it properly, not enough for daily professional use. Pro: $10/month (or $100/year). Unlimited completions, 300 premium requests per month. This is the sweet spot for individual developers. Pro+: $39/month. 1,500 premium requests, access to every available model including Claude Opus 4.6 and OpenAI o3. Business: $19/user/month for teams needing central management.

Premium requests power the advanced features: Chat with complex questions, agent mode, code review, and manual model selection. Simple autocomplete doesn’t consume premium requests at all. The 300/month on Pro covers most developers comfortably. Only heavy agent mode users hit the ceiling.

The real pros:

  • Stays inside your existing editor. Zero workflow disruption
  • $10/month is genuinely affordable, half the cost of Cursor
  • 2,000-completion free tier is the most generous free option of the three
  • Works across more IDEs than any competitor
  • Deep GitHub integration: pull request summaries, code review, codebase chat on github.com

The real cons:

  • Multi-file editing and complex refactors are weaker than Cursor or Claude Code
  • Agent mode is newer and less capable than Cursor’s or Claude Code’s agentic abilities
  • Premium request limits can frustrate heavy users before they realise they need Pro+
  • The free tier’s 50 premium requests disappear in about two days of real use

Who should NOT use Copilot:
If you’re building full features end-to-end using AI (not just getting autocomplete help), Copilot’s agent mode lags behind Cursor and Claude Code. It’s also not the right tool if you want AI to read your entire codebase, understand architecture, and make decisions across multiple files. That’s where the other two pull ahead.

Cursor

Cursor is what happens when you build an AI model into the editor itself rather than bolting it on as a plugin. It’s a fork of VS Code, so your extensions, keybindings, and muscle memory transfer. But the AI isn’t layered on top. It’s woven into every part of the experience.

Composer mode lets you describe a multi-file change in plain English and Cursor executes it across your entire codebase. Tab completion predicts not just the next line but the next logical block. The codebase indexing means Cursor genuinely understands your project structure, not just the file you have open.

What it actually costs in 2026:

Free tier: limited Composer uses and a capped number of premium model requests. Enough to evaluate it properly. Pro: $20/month. Unlimited Composer uses, 500 fast premium requests per month, access to Claude Sonnet 4.6 and GPT-4.1. Business: $40/user/month adds privacy mode and centralised team management.

The 500 fast requests on Pro are the main constraint. Heavy Composer users hit it. When you do, you fall back to slower requests (still functional, just slower). For most solo developers doing focused sessions rather than running Cursor all day, Pro is sufficient.

The real pros:

  • The most polished AI IDE experience available right now
  • Multi-file Composer is genuinely impressive for feature work
  • Tab completion that predicts logical blocks, not just lines
  • Deep codebase indexing means context-aware suggestions across your whole project
  • VS Code compatibility means your existing setup transfers almost entirely

The real cons:

  • $20/month is double Copilot Pro
  • You are locked into Cursor as your IDE. If you prefer JetBrains or Vim, this doesn’t work
  • Heavy users hit the 500 fast request limit and feel it
  • Switching IDEs has a real cost even if extensions transfer

Who should NOT use Cursor:
If you’re a JetBrains user (IntelliJ, PhpStorm, WebStorm), Cursor doesn’t serve you. It’s VS Code only. If you’re already happy with Copilot and primarily want autocomplete rather than full feature generation, the extra $10/month isn’t justified. And if your workflow is more task-oriented than file-oriented, Claude Code may suit you better.

Claude Code

Claude Code is the odd one out. There’s no IDE. No editor. You run it in your terminal, give it a task in plain English, and it reads your files, writes code, runs tests, commits to git, and iterates until the job is done. You watch it work rather than working alongside it.

This sounds abstract but in practice it’s genuinely different from the other two. You don’t write code with Claude Code so much as review code it wrote. The workflow is: describe what needs to happen, review the diff, approve or iterate.

What it actually costs in 2026:

Claude Code is included in your Claude subscription. Claude Pro ($20/month): access to Claude Code, roughly 45 messages per 5-hour window. Claude Max 5x ($100/month): 5x the Pro usage, priority access. Claude Max 20x ($200/month): 20x Pro, essentially no practical limits for most full-day developers. API (pay-as-you-go): Claude Sonnet 4.6 at $3 per million input tokens / $15 per million output tokens. Average cost via API is $6 per developer per day for typical usage, or $100-200/month for heavy use.

The key insight: if you’re already paying $20/month for Claude Pro to use claude.ai, Claude Code costs you nothing extra. That changes the value calculation significantly.

The real pros:

  • If you already pay for Claude Pro, it’s free. No additional subscription
  • Best agentic capability of the three for complex, multi-step tasks
  • Can run terminal commands, execute tests, and manage git, not just edit files
  • Agent Teams (multiple parallel Claude Code instances) for large codebases
  • Works with any editor. Completely editor-agnostic
  • 1 million token context window means it can hold your entire codebase in memory

The real cons:

  • No IDE means no inline suggestions or autocomplete as you type
  • Pro plan’s 45 messages per 5-hour window runs out fast during intensive sessions
  • The terminal-only workflow takes adjustment. It’s not familiar to most developers
  • Heavy daily usage on Pro limits is frustrating. Max plans are expensive
  • Less suitable for quick line-by-line edits where Cursor or Copilot feel faster

Who should NOT use Claude Code:
If you want AI help while actively typing code (autocomplete, inline suggestions), Claude Code doesn’t do that. It’s for task-level work, not keystroke-level assistance. Also avoid it as your primary tool if you’re on Pro and doing intensive sessions, as you’ll hit limits quickly. And if you’re not already paying for Claude, you need to add a subscription just to use it.

Head-to-Head: What Actually Matters

Price for solo developers:
Copilot Pro wins at $10/month. Claude Code wins if you already have Claude Pro (net $0 extra). Cursor is $20/month. Copilot is the cheapest standalone option.

Autocomplete and inline suggestions:
Cursor is best, with contextual block-level prediction. Copilot is strong and works in more editors. Claude Code has none. It’s not that kind of tool.

Multi-file feature development:
Claude Code is the strongest here. Give it a specification and it executes across your entire codebase. Cursor Composer is close and more interactive. Copilot’s agent mode is improving but lags.

Laravel / PHP development specifically:
All three work with PHP and Laravel. Copilot has the broadest IDE support including PhpStorm. Cursor is VS Code only. Claude Code is editor-agnostic and handles large Laravel codebases well in the terminal.

Best for building a SaaS from scratch:
Cursor for the build phase where you’re typing a lot. Claude Code for larger refactors, adding features across many files, or automation tasks. Many developers use both.

Team use:
Copilot Business at $19/user is the most established team option. Cursor Business at $40/user is expensive. Claude Code Team at $100/seat/month is premium but powerful for engineering teams.

The Honest Recommendation by Developer Type

Use GitHub Copilot Pro if: You want AI coding help with zero workflow change, you’re on a budget ($10/month is hard to beat), or you use JetBrains, Vim, Xcode, or any non-VS-Code editor.

Use Cursor Pro if: You’re primarily a VS Code user, you want the most polished AI-native IDE, and you’re comfortable paying $20/month for a noticeably better autocomplete and multi-file editing experience than Copilot.

Use Claude Code if: You already pay for Claude Pro and want to get more from that subscription, you prefer describing tasks over typing code, or you’re working on large refactors and multi-file features where its agentic approach outperforms the IDE-based tools.

The combination most solo developers end up with:
Cursor or Copilot for daily coding, Claude Code for the bigger tasks. They’re not mutually exclusive, and several of the tools in our automation tools comparison are built by developers using exactly this stack.

FAQ

Is GitHub Copilot free in 2026?

Yes. GitHub Copilot has a free tier with 2,000 code completions and 50 premium requests per month. It’s the most generous free tier of the three tools here. The Pro plan at $10/month unlocks unlimited completions and 300 premium requests.

Do I need a separate subscription for Claude Code?

No. Claude Code is included in your Claude subscription. Claude Pro at $20/month gives you access. If you’re already paying for Claude Pro to use claude.ai, adding Claude Code costs nothing extra.

Can I use Cursor with JetBrains?

No. Cursor is a fork of VS Code and only works as a VS Code replacement. If you use IntelliJ, PhpStorm, WebStorm, or any other JetBrains IDE, GitHub Copilot is the better option since it supports all major IDEs natively.

Which AI coding tool is best for Laravel development?

All three support PHP and Laravel well. GitHub Copilot works inside PhpStorm which is the preferred IDE for many Laravel developers. Cursor is VS Code only. Claude Code is terminal-based and works regardless of your editor, making it useful for Laravel developers who want to offload large refactors or generate boilerplate.

Is Claude Code better than Cursor?

They’re different tools for different jobs. Cursor excels at inline assistance, autocomplete, and interactive file editing. Claude Code excels at autonomous multi-step tasks. Most developers find them complementary rather than competing. If forced to pick one: Cursor for daily coding flow, Claude Code for bigger feature work.

Conclusion

The AI coding tool market in 2026 has settled into three clear lanes. GitHub Copilot is the affordable, low-disruption option that fits into any existing workflow. Cursor is the premium IDE-replacement for VS Code users who want the deepest AI integration. Claude Code is the agentic option for developers who want to describe tasks and review results rather than type alongside an AI.

None of them is obviously the best for everyone. The right answer depends on how you code, what editor you use, and whether you’re already paying for Claude.

Start with Copilot’s free tier. If you want more, upgrade to Pro at $10/month. Only switch to Cursor or add Claude Code when you have a clear reason to. Not because of FOMO.

Canonical tag mistakes in WordPress sites before launch

Part of the series: WordPress Pre-Launch Technical Checks

When reviewing a WordPress site before launch, there are a few technical signals that are easy to overlook. One of them is the canonical tag. It’s not something you see on the page, so it often gets less attention than it should.

In practice, this tag plays a key role in how search engines interpret the structure of the site. If it’s not configured properly, different URLs can end up competing with each other without anyone noticing.

Before delivering a project, it’s usually worth checking that canonical behavior is clean and consistent across the site.

What the canonical tag actually does

The canonical tag lives in the HTML head and tells search engines which version of a page should be treated as the main one.

A typical example looks like this:

<link rel="canonical" href="https://example.com/page/" />

It might seem like a small detail, but it helps avoid situations where multiple URLs point to the same content without a clear reference.

This happens more often than expected, especially with parameters, alternate paths or small variations of the same page.

Why these issues appear in WordPress

WordPress generates a wide range of URLs by default. Archives, paginated pages, previews or parameter-based URLs are all part of its normal behavior.

Because of that, canonical signals are usually handled automatically by themes or SEO plugins. And that’s where things can get a bit messy if no one reviews the final setup.

It’s quite easy to assume everything is fine, when in reality the signals don’t fully match the intended structure.

What tends to go wrong

Missing canonical tags

Some pages, especially custom ones, end up without a canonical tag. It’s not always obvious, but it leaves search engines without a clear reference.

Canonical pointing to the wrong place

After migrations, it’s not unusual to find canonicals still pointing to staging domains or older URLs. It’s one of those things that slips through if no one checks it directly.

Multiple canonical tags

Sometimes both the theme and an SEO plugin generate their own canonical tag. When that happens, you can end up with conflicting signals in the page head.

Inconsistent behavior across templates

Different templates can produce slightly different canonical logic. On their own it might not seem critical, but across the site it creates mixed signals.

A quick review before launch

At this stage, a simple check is usually enough. Just confirm that important pages have a canonical tag, that it points to the correct production URL, and that there are no duplicates or leftover references.

It doesn’t take much time, but it helps avoid confusion later on.

Why this belongs in a pre-launch checklist

Canonical issues rarely break anything visually. The site loads, navigation works, everything seems fine.

But under the surface, they affect how search engines interpret the site structure. Fixing them after launch is possible, but it’s cleaner to get it right from the start.

Where PreFlight fits in

PreFlight focuses on reviewing these kinds of technical details before a WordPress site is delivered or published. It’s about catching small inconsistencies before they turn into bigger issues.

If you want to run a quick technical check before launch, you can start here:
https://preflightstandard.com/

Final thought

The canonical tag is a small piece of code, but it carries a lot of weight when it comes to how a site is understood.

Taking a moment to review it before launch helps keep everything aligned from day one.

SAP Commerce Catalog System Deep Dive: Staged Content, Synchronization, and Multi-Catalog Architecture

The catalog system is one of SAP Commerce’s most powerful — and most misunderstood — features. It provides a staging mechanism that lets business users prepare product data, pricing, and content changes in isolation, then publish everything to the live storefront in a single atomic operation. When used correctly, it eliminates the “edit live data and hope nothing breaks” anti-pattern. When used incorrectly, it becomes a source of synchronization headaches, data inconsistency, and performance bottlenecks.

This article breaks down how the catalog system actually works, how synchronization moves data from staged to online, and how to architect multi-catalog setups for complex B2B and multi-brand scenarios.

The Two-Version Model

Every catalog in SAP Commerce has at least two catalog versions:

┌─────────────────────────────────────────┐
│  Product Catalog: "electronicsProductCatalog"  │
│                                         │
│  ┌───────────────┐    ┌──────────────┐ │
│  │   Staged       │───>│   Online     │ │
│  │   Version      │sync│   Version    │ │
│  │               │───>│              │ │
│  │ • Edit here   │    │ • Read-only  │ │
│  │ • Preview     │    │ • Live data  │ │
│  │ • No customer │    │ • Storefront │ │
│  │   impact      │    │   serves this│ │
│  └───────────────┘    └──────────────┘ │
└─────────────────────────────────────────┘

Staged: The working version. Business users, product managers, and content editors make all their changes here. Products are created, prices are updated, categories are reorganized — all without affecting what customers see.

Online: The published version. The storefront (OCC API, Accelerator, Spartacus) reads data from this version. It’s effectively read-only from a business perspective — changes should only come through synchronization from Staged.

Why This Matters

Without the staged/online model, every product edit would immediately be visible to customers. Imagine changing a product price from $99 to $79 for a Black Friday promotion — but the promotion doesn’t start until Friday at midnight. With direct editing, you’d have to set the new price at exactly midnight. With staged/online, you set the price in Staged days in advance and synchronize at midnight.

Setting Up a Catalog

Catalogs are created via ImpEx:

INSERT_UPDATE Catalog;id[unique=true];name[lang=en];defaultCatalog
;myProductCatalog;My Product Catalog;true

# Create catalog versions
INSERT_UPDATE CatalogVersion;catalog(id)[unique=true];version[unique=true];active;languages(isocode)
;myProductCatalog;Staged;false;en,de,fr
;myProductCatalog;Online;true;en,de,fr

The active=true flag marks the Online version as the one served to customers.

How Synchronization Works

Synchronization is the process of copying data from one catalog version to another — typically from Staged to Online.

The Sync Job

A CatalogVersionSyncJob defines what to synchronize:

INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];sourceVersion(catalog(id),version)[unique=true];targetVersion(catalog(id),version)[unique=true];createNewItems;removeMissingItems
;myProductCatalogSyncJob;myProductCatalog:Staged;myProductCatalog:Online;true;true
Parameter Meaning
createNewItems If true, new items in Staged are created in Online
removeMissingItems If true, items deleted from Staged are deleted from Online

What Gets Synchronized

The sync job synchronizes all catalog-aware item types. An item type is catalog-aware if it has a catalogVersion attribute. The standard catalog-aware types include:

  • Product (and subtypes like VariantProduct)
  • Category
  • Media
  • Keyword
  • CatalogVersionSyncScheduleMedia

Custom types become catalog-aware when you add a catalogVersion attribute:

<itemtype code="CustomContentBlock" extends="GenericItem" autocreate="true" generate="true">
    <deployment table="CustomContentBlocks" typecode="25050"/>
    <attributes>
        <attribute qualifier="code" type="java.lang.String">
            <persistence type="property"/>
            <modifiers unique="true"/>
        </attribute>
        <attribute qualifier="catalogVersion" type="CatalogVersion">
            <persistence type="property"/>
            <modifiers read="true" write="true"/>
        </attribute>
        <!-- other attributes -->
    </attributes>
</itemtype>

Sync Internals: What Actually Happens

When a sync job runs, the platform:

  1. Identifies changed items — compares timestamps between source and target catalog versions
  2. Copies item data — creates or updates items in the target version with data from the source
  3. Resolves references — translates references from source catalog version to target (e.g., a product’s category reference in Staged maps to the corresponding category in Online)
  4. Handles media — copies media items and their binary data
  5. Maintains link consistency — ensures all relations (product-to-category, product-to-media) are consistent in the target version
Sync Process Flow:
┌─────────────────────────────────────────────────┐
│ 1. Collect modified items from source (Staged)   │
│ 2. For each item:                                │
│    a. Find/create counterpart in target (Online) │
│    b. Copy all synchronized attributes           │
│    c. Resolve catalog-version-specific references│
│    d. Update timestamps                          │
│ 3. Handle deletions (if removeMissingItems=true) │
│ 4. Update sync status and counters               │
└─────────────────────────────────────────────────┘

Counterpart Resolution

During synchronization, the platform needs to find the “counterpart” of a Staged item in Online. It uses unique keys for this:

  • For Product: code + catalogVersion
  • For Category: code + catalogVersion
  • For Media: code + catalogVersion

A product with code PROD-001 in myProductCatalog:Staged maps to PROD-001 in myProductCatalog:Online. If the Online counterpart doesn’t exist and createNewItems=true, it’s created.

Synchronization Performance

Catalog sync is one of the most resource-intensive operations in SAP Commerce. Understanding and optimizing it is critical for production environments.

Full vs. Incremental Sync

Full sync: Compares every item in source and target. Safe but slow. Use only during initial setup or when data consistency is questionable.

Incremental sync: Only processes items modified since the last sync. Much faster. Use for regular production synchronization.

// Programmatic incremental sync
SyncConfig config = new SyncConfig();
config.setSynchronizationType(SyncConfig.INCREMENTAL);
config.setCreateSavedValues(false);
config.setLogToDatabase(false);
config.setLogToFile(false);

catalogSynchronizationService.synchronize(syncJob, config);

Performance Tuning

# Number of parallel sync workers
catalog.sync.workers=4

# Batch size — number of items processed per transaction
catalog.sync.batch.size=100

# Disable interceptors during sync (significant speedup)
catalog.sync.enable.interceptors=false

# Skip saved values (audit trail) for speed
catalog.sync.create.saved.values=false

Timing and Scheduling

Schedule syncs during low-traffic periods:

INSERT_UPDATE Trigger;cronJob(code)[unique=true];cronExpression
;myProductCatalogSyncJob;0 0 3 * * ?

For time-critical syncs (flash sales, price changes), trigger programmatically:

@Resource
private CatalogSynchronizationService catalogSynchronizationService;

public void publishCatalogChanges() {
    CatalogVersionSyncJobModel syncJob = // lookup sync job
    SyncConfig config = new SyncConfig();
    config.setSynchronizationType(SyncConfig.INCREMENTAL);

    CatalogVersionSyncCronJobModel cronJob = catalogSynchronizationService.synchronize(syncJob, config);
    LOG.info("Sync started: {}", cronJob.getCode());
}

Content Catalogs

In addition to product catalogs, SAP Commerce uses content catalogs for CMS content (pages, components, slots). The same staged/online model applies:

INSERT_UPDATE ContentCatalog;id[unique=true];name[lang=en]
;myContentCatalog;My Content Catalog

INSERT_UPDATE CatalogVersion;catalog(id)[unique=true];version[unique=true];active
;myContentCatalog;Staged;false
;myContentCatalog;Online;true

CMS Content Structure

Content Catalog (Staged)
├── PageTemplate: ProductDetailPageTemplate
├── ContentPage: homepage
│   ├── ContentSlot: Section1
│   │   ├── BannerComponent: heroBanner
│   │   └── ProductCarouselComponent: newArrivals
│   ├── ContentSlot: Section2
│   │   └── CategoryFeatureComponent: topCategories
│   └── ContentSlot: Footer
│       └── FooterNavigationComponent: mainFooter
├── ContentPage: faqPage
└── ContentPage: contactPage

Content catalog sync works identically to product catalog sync — changes made to CMS pages in Staged are published to Online through synchronization.

SmartEdit Integration

SmartEdit (SAP Commerce’s visual CMS editor) works exclusively with the Staged content catalog. Editors drag and drop components, rearrange content slots, and preview pages — all in Staged. When they click “Sync,” the content catalog synchronization job runs and publishes changes to Online.

Multi-Catalog Architecture

Real-world projects often need multiple catalogs. Here are the common patterns.

Pattern 1: Shared Master Catalog

Multiple storefronts share a master product catalog but have separate content catalogs:

┌─────────────────────────────────┐
│     Master Product Catalog       │
│  (Staged → Online)              │
├────────────┬────────────────────┤
│            │                    │
│  ┌─────────▼──────┐ ┌─────────▼──────┐
│  │ US Content     │ │ EU Content     │
│  │ Catalog        │ │ Catalog        │
│  │ (Staged→Online)│ │ (Staged→Online)│
│  └────────────────┘ └────────────────┘
# Shared product catalog
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;globalProductCatalog;Global Product Catalog

# Region-specific content catalogs
INSERT_UPDATE ContentCatalog;id[unique=true];name[lang=en]
;usContentCatalog;US Content Catalog
;euContentCatalog;EU Content Catalog

Pattern 2: Multi-Country Product Catalogs

Each country has its own product catalog with country-specific pricing, availability, and product assortment:

┌──────────────────┐
│ Global Product    │
│ Catalog (Master)  │
│ Staged → Online   │
└────────┬─────────┘
         │ sync
    ┌────┴────┐
    │         │
┌───▼────┐ ┌─▼──────┐
│ US     │ │ DE     │
│ Product│ │ Product│
│ Catalog│ │ Catalog│
│ S → O  │ │ S → O  │
└────────┘ └────────┘
# Global master catalog
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;globalProductCatalog;Global Product Catalog

# Country catalogs
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;usProductCatalog;US Product Catalog
;deProductCatalog;DE Product Catalog

# Cross-catalog sync: Global Staged → US Staged
INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];sourceVersion(catalog(id),version);targetVersion(catalog(id),version)
;globalToUsSyncJob;globalProductCatalog:Staged;usProductCatalog:Staged
;globalToDeSyncJob;globalProductCatalog:Staged;deProductCatalog:Staged

# Then US Staged → US Online
INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];sourceVersion(catalog(id),version);targetVersion(catalog(id),version)
;usProductCatalogSyncJob;usProductCatalog:Staged;usProductCatalog:Online
;deProductCatalogSyncJob;deProductCatalog:Staged;deProductCatalog:Online

Pattern 3: B2B Multi-Catalog with Visibility Rules

B2B scenarios often require different product visibility per customer group:

# Base catalog visible to all customers
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;baseProductCatalog;Base Product Catalog

# Premium catalog with additional products
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;premiumProductCatalog;Premium Product Catalog

# Map catalogs to customer groups via CMSSite
INSERT_UPDATE CMSSite;uid[unique=true];contentCatalogs(id);productCatalogs(id)
;standardSite;standardContentCatalog;baseProductCatalog
;premiumSite;premiumContentCatalog;baseProductCatalog,premiumProductCatalog

Catalog-Aware Queries

When querying data, you must always specify the catalog version. This is one of the most common sources of bugs — forgetting the catalog version filter returns data from all versions (Staged + Online), causing duplicates.

FlexibleSearch with Catalog Version

// CORRECT: Always include catalogVersion in queries
String query = "SELECT {pk} FROM {Product} WHERE {code} = ?code AND {catalogVersion} = ?cv";
FlexibleSearchQuery fsq = new FlexibleSearchQuery(query);
fsq.addQueryParameter("code", "PROD-001");
fsq.addQueryParameter("cv", catalogVersionService.getCatalogVersion("myProductCatalog", "Online"));
// WRONG: Missing catalogVersion — returns products from ALL versions
String query = "SELECT {pk} FROM {Product} WHERE {code} = ?code";
// This returns PROD-001 from both Staged AND Online — likely a bug

Session Catalog Versions

SAP Commerce maintains a set of “session catalog versions” — the catalog versions the current user should see. The storefront sets Online versions; Backoffice sets Staged versions.

// Get the session catalog version
Collection<CatalogVersionModel> sessionVersions = catalogVersionService.getSessionCatalogVersions();

// Set session catalog versions (typically done by the storefront filter chain)
catalogVersionService.setSessionCatalogVersions(
    Collections.singletonList(
        catalogVersionService.getCatalogVersion("myProductCatalog", "Online")
    )
);

Services like ProductService.getProductForCode() automatically filter by session catalog versions, so you don’t need to pass the catalog version explicitly when using the service layer.

Category Hierarchies

Categories are catalog-version-aware and form a tree structure within each catalog version.

Defining Categories

$catalogVersion = catalogVersion(catalog(id[default='myProductCatalog']),version[default='Staged'])

INSERT_UPDATE Category;code[unique=true];$catalogVersion;name[lang=en];supercategories(code,$catalogVersion)
;electronics;;Electronics;
;cameras;;Cameras;electronics
;dslr;;DSLR Cameras;cameras
;mirrorless;;Mirrorless Cameras;cameras
;smartphones;;Smartphones;electronics
;accessories;;Accessories;electronics
;cases;;Phone Cases;accessories
;chargers;;Chargers;accessories

This creates a hierarchy:

Electronics
├── Cameras
│   ├── DSLR Cameras
│   └── Mirrorless Cameras
├── Smartphones
└── Accessories
    ├── Phone Cases
    └── Chargers

Assigning Products to Categories

$catalogVersion = catalogVersion(catalog(id[default='myProductCatalog']),version[default='Staged'])

INSERT_UPDATE CategoryProductRelation;source(code,$catalogVersion)[unique=true];target(code,$catalogVersion)[unique=true]
;dslr;CANON-EOS-R5
;dslr;NIKON-Z8
;mirrorless;SONY-A7IV
;smartphones;IPHONE-15
;cases;CASE-IPHONE-15

A product can belong to multiple categories. During synchronization, these relationships are copied from Staged to Online.

Handling Sync Conflicts and Edge Cases

Partial Sync

Sometimes you need to sync only specific items rather than the entire catalog:

// Sync specific products only
List<ItemModel> itemsToSync = Arrays.asList(product1, product2, product3);
SyncConfig config = new SyncConfig();
config.setSynchronizationType(SyncConfig.PARTIAL);

for (ItemModel item : itemsToSync) {
    catalogSynchronizationService.synchronizeItem(item, syncJob, config);
}

Sync Excludes

Exclude specific attributes from synchronization:

# Don't sync the 'internalNotes' attribute — keep it Staged-only
INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];&syncJobRef
;myProductCatalogSyncJob;syncJobRef

INSERT_UPDATE SyncAttributeDescriptorConfig;syncJob(&syncJobRef)[unique=true];attributeDescriptor(qualifier,enclosingType(code))[unique=true];includedInSync
;syncJobRef;internalNotes:Product;false

Avoiding Common Sync Problems

1. Missing references: If a product references a category that hasn’t been synced yet, the sync fails. Solution: sync categories before products, or sync both in the same job.

2. Orphaned items: If removeMissingItems=true and someone accidentally deletes items from Staged, they’ll be deleted from Online too. Consider setting removeMissingItems=false for safety and handling deletions manually.

3. Large media files: Syncing catalogs with thousands of high-resolution images is slow. Consider using external media (CDN URLs) instead of storing media in the catalog.

4. Concurrent modifications: If two users modify the same product in Staged while a sync is running, the sync might pick up an inconsistent state. Best practice: lock the catalog (or communicate a sync window) during synchronization.

Monitoring Sync Status

Via HAC

Navigate to Platform → System → Catalog Version Syncing to see:

  • Last sync date
  • Number of items synced
  • Sync status (SUCCESS, ERROR, RUNNING)
  • Error details

Programmatically

CatalogVersionSyncCronJobModel syncCronJob = // lookup
LOG.info("Sync status: {}", syncCronJob.getStatus());
LOG.info("Sync result: {}", syncCronJob.getResult());
LOG.info("Items processed: {}", syncCronJob.getProcessedItemsCount());
LOG.info("Items synced: {}", syncCronJob.getSyncedItemsCount());
LOG.info("Errors: {}", syncCronJob.getFailedItemsCount());

Sync Audit

Enable saved values to track what changed during each sync:

catalog.sync.create.saved.values=true

This creates SavedValues items that record the before/after state of each synced attribute. Useful for debugging but adds significant overhead — disable in production unless needed.

Best Practices

  1. Always work in Staged — never modify Online directly. All changes should flow through sync.
  2. Use incremental sync — full sync is only for initial setup or data repair.
  3. Schedule syncs during off-peak hours — sync is CPU and memory intensive.
  4. Separate product and content syncs — they have different schedules and different business owners.
  5. Test sync with production-like data volumes — sync that works with 100 products may fail with 500,000.
  6. Monitor sync duration trends — increasing sync times indicate growing data complexity or missing indexes.
  7. Always include catalogVersion in FlexibleSearch queries — the most common source of data bugs.
  8. Plan your multi-catalog architecture early — changing catalog structure after go-live requires data migration.

Summary

The SAP Commerce catalog system is a content staging and publishing framework. Its two-version model provides safety for business operations, while synchronization gives you atomic publishing of changes. The key principles:

  1. Staged is for editing, Online is for serving — never break this boundary
  2. Synchronization copies data from Staged to Online — it resolves references, handles media, and maintains consistency
  3. Incremental sync is your production workhorse — full sync is for exceptional circumstances
  4. Multi-catalog architectures serve multi-country and B2B scenarios — plan the catalog hierarchy carefully
  5. Always filter by catalog version in queries — missing this filter is the most common catalog-related bug
  6. Monitor and tune sync performance — it’s one of the most resource-intensive platform operations

Understanding the catalog system deeply is what allows you to build commerce solutions that scale — both technically and operationally.