# From Data collecting to Data Understanding: My First week Of Learning Excel

Excel workflow for Anticancer Research On Madagascar Periwinkle

Familiar Tools Become New Again.

Most professionals have, in one way or another, interacted with Excel at some point in their career, but often at the surface level, and honestly i was no exception.

Having worked in the field of research,Drug discovery from natural products, Excel was always present in my workflow, primarily for collecting experimental data, storing results, sorting observations, and performing simple calculations. Then, this felt sufficient; the tool was actually doing what I needed! Or so I thought.

However, my interaction with Excel at times reached a dead end. I often found myself wondering what some of these functions actually did, and I would resort to repeated referencing and online searches. Simple operations or tasks that should have been straightforward become tedious, time-consuming, and sometimes frustrating. My one week of learning Excel at LuxDev Academy has changed my perspective entirely. I’ve realized it is not just a spreadsheet tool; it’s also a Data analysis tool.

What is Excel?

Microsoft Excel is a software tool designed for organizing,analyzing, and visualizing data using structured rows and columns.
After the data entry process, Excel enables users in:

  • Managing large datasets efficiently
  • Cleaning and transforming datasets.
  • Performing mathematical and statistical calculations.
  • Creating charts, dashboards, and automating workflows.

Excel in Real-World Scenarios.

Almost every data-driven field relies on Excel, but I will be focusing on Excel in Drug Discovery Research. In drug discovery, researchers often begin their work by referencing already published datasets either to identify research gaps, avoid duplication of previous work, or compare experimental outcomes.
Let’s explore one modern approach,Network Pharmacology. Researchers gather chemical and biological data from multiple databases, such as compound libraries, protein interaction databases, and pathway repositories. These datasets originate from different sources, and they have different formats. Excel, therefore, becomes the integrating tool where researchers:

  • Removes duplicates
  • Merges datasets
  • Standardises variables
  • Filter relevant targets
  • Prepares data for downstream analysis

Different Formulas in Excel and how they apply in Data Work.

During my one week of learning Excel, I began by learning Excel formulas not just as abstract functions but as a solving tool, and here are some of the few examples and how they apply in a real analytics workflow, in our case, drug discovery research.

Logical Functions-IF()

This function would apply in categorizing experimental outcomes. A good example application is classifying compounds as Active or Inactive based on inhibition values. This would allow automatic classification instead of manual labelling.

=IF(B2>50,"Active","Inactive")

Text Cleaning Fuctions-LEFT(),RIGHT(),SUBSTITUTE()

These functions are useful when dealing with datasets with inconsistent formatting. The best example is extracting numerical values from ranges or percentage strings during data cleaning. These functions will transform messy data into analysis-ready datasets.

Lookup Functions-XLOOKUP()/VLOOKUP()

How do you apply this function? A good example is when matching compound names with biological targets imported from separate databases. This is a very useful function because rarely do datasets originate from the same source.

>XLOOKUP(A2,Sheet2!A::Sheet2!B:B)

Aggregate Functions-AVERAGE(),SUM(),COUNT()

Most of the experiments are done in triplicate, so these aggregate functions are applied when summarising experiment replicates.

Pivot Tables.

Making a pivot table is one of the most transformative skill i learnt. Now, instead of manually summarising results, the pivot table allows me to compare activities across compounds, summarise experimental batches, and analyse trends instantly, interactively, and all in one Excel workbook.

Example of an Excel workflow

Excel workflow for Anticancer Research On Madagascar Periwinkle

Interactive Dashboards

Drug research does not end with lab work and analysis of the data; researchers need to disseminate their months/year of research work to the rest of the world. This session is always allocated very limited time,15 to 20 mins max, and presenting raw spreadsheets is idealistic. The interactive dashboards skill becomes invaluable in this section. With this skill as a researcher, I can :

  • Summarise my findings visually
  • Use slicers to explore my datasets interactively
  • Use interactive charts to explain trends
  • Highlight key findings instantly (KPI section)

Example of an Excel Dashboard

Excel dashboard for Network Pharmacology analysis of Anticancer activity of _**Madagascar Periwinkle**_

Rediscovering Excel

Earlier, Excel was simply a storage tool or a digital notebook for recording values. The view has changed to a system for thinking analytically. Data collection alone does not translate to insights. You may gather very important data, but without the ability to clean, analyse, and visualize it effectively, its true value remains hidden. Looking back, I only wish I had learnt this skill earlier. Nevertheless, beginning this learning path has changed how I approach data, which is not just a value to be recorded, but a coded message waiting to be decoded!

Cursor vs GitHub Copilot vs Claude Code in 2026: Which AI Coding Tool Is Actually Worth It?

Originally published at devtoolpicks.com

Every developer paying for AI coding tools right now is asking the same question: am I on the right one?

Cursor has Windsurf breathing down its neck. GitHub Copilot just added agent mode and a free tier. Claude Code is sitting in your terminal quietly being more capable than most people realise. Three tools, three completely different philosophies, three different price points.

This isn’t the same comparison as Cursor vs Windsurf. That post covers two tools competing in the same category. This one is different. These three tools represent fundamentally different answers to the question “how should AI fit into my coding workflow?”

Quick verdict: GitHub Copilot Pro at $10/month is the obvious entry point if you want to keep your existing setup. Cursor at $20/month wins if you want the most polished AI-native IDE experience. Claude Code wins if you’re a developer who thinks in terms of tasks and agents rather than file-by-file editing, and you’re already paying for a Claude Pro subscription.

The Core Difference Nobody Explains Clearly

Before pricing, before features, you need to understand what each tool actually is:

GitHub Copilot stays inside your current IDE. You keep VS Code, JetBrains, or whatever you use. Copilot adds AI as a layer on top. It’s an assistant that lives inside your existing workflow.

Cursor replaces your IDE. You give up VS Code and switch to Cursor, which is a full fork of VS Code rebuilt around AI. The AI isn’t a plugin, it IS the editor.

Claude Code has no IDE at all. It runs in your terminal. You describe what you want done and it reads your files, writes code, runs commands, and iterates. It’s closer to hiring a contractor than using a tool.

That distinction matters more than any feature comparison.

Quick Comparison

Tool Approach Starting Price Free Tier Best For
GitHub Copilot Plugin for your IDE $10/month Yes (2,000 completions) Developers who want AI without switching tools
Cursor AI-native IDE replacement $20/month Yes (limited) Developers who want the deepest IDE integration
Claude Code Terminal-based AI agent $20/month (via Claude Pro) No Developers who think in tasks, not files

GitHub Copilot

GitHub Copilot is the most widely adopted AI coding tool in the world. It works inside VS Code, JetBrains, Visual Studio, Neovim, and Xcode. You don’t change how you work. You just get a very good autocomplete and an AI chat panel bolted onto whatever editor you already use.

What it actually costs in 2026:

Free plan: 2,000 code completions per month, 50 premium requests. Enough to try it properly, not enough for daily professional use. Pro: $10/month (or $100/year). Unlimited completions, 300 premium requests per month. This is the sweet spot for individual developers. Pro+: $39/month. 1,500 premium requests, access to every available model including Claude Opus 4.6 and OpenAI o3. Business: $19/user/month for teams needing central management.

Premium requests power the advanced features: Chat with complex questions, agent mode, code review, and manual model selection. Simple autocomplete doesn’t consume premium requests at all. The 300/month on Pro covers most developers comfortably. Only heavy agent mode users hit the ceiling.

The real pros:

  • Stays inside your existing editor. Zero workflow disruption
  • $10/month is genuinely affordable, half the cost of Cursor
  • 2,000-completion free tier is the most generous free option of the three
  • Works across more IDEs than any competitor
  • Deep GitHub integration: pull request summaries, code review, codebase chat on github.com

The real cons:

  • Multi-file editing and complex refactors are weaker than Cursor or Claude Code
  • Agent mode is newer and less capable than Cursor’s or Claude Code’s agentic abilities
  • Premium request limits can frustrate heavy users before they realise they need Pro+
  • The free tier’s 50 premium requests disappear in about two days of real use

Who should NOT use Copilot:
If you’re building full features end-to-end using AI (not just getting autocomplete help), Copilot’s agent mode lags behind Cursor and Claude Code. It’s also not the right tool if you want AI to read your entire codebase, understand architecture, and make decisions across multiple files. That’s where the other two pull ahead.

Cursor

Cursor is what happens when you build an AI model into the editor itself rather than bolting it on as a plugin. It’s a fork of VS Code, so your extensions, keybindings, and muscle memory transfer. But the AI isn’t layered on top. It’s woven into every part of the experience.

Composer mode lets you describe a multi-file change in plain English and Cursor executes it across your entire codebase. Tab completion predicts not just the next line but the next logical block. The codebase indexing means Cursor genuinely understands your project structure, not just the file you have open.

What it actually costs in 2026:

Free tier: limited Composer uses and a capped number of premium model requests. Enough to evaluate it properly. Pro: $20/month. Unlimited Composer uses, 500 fast premium requests per month, access to Claude Sonnet 4.6 and GPT-4.1. Business: $40/user/month adds privacy mode and centralised team management.

The 500 fast requests on Pro are the main constraint. Heavy Composer users hit it. When you do, you fall back to slower requests (still functional, just slower). For most solo developers doing focused sessions rather than running Cursor all day, Pro is sufficient.

The real pros:

  • The most polished AI IDE experience available right now
  • Multi-file Composer is genuinely impressive for feature work
  • Tab completion that predicts logical blocks, not just lines
  • Deep codebase indexing means context-aware suggestions across your whole project
  • VS Code compatibility means your existing setup transfers almost entirely

The real cons:

  • $20/month is double Copilot Pro
  • You are locked into Cursor as your IDE. If you prefer JetBrains or Vim, this doesn’t work
  • Heavy users hit the 500 fast request limit and feel it
  • Switching IDEs has a real cost even if extensions transfer

Who should NOT use Cursor:
If you’re a JetBrains user (IntelliJ, PhpStorm, WebStorm), Cursor doesn’t serve you. It’s VS Code only. If you’re already happy with Copilot and primarily want autocomplete rather than full feature generation, the extra $10/month isn’t justified. And if your workflow is more task-oriented than file-oriented, Claude Code may suit you better.

Claude Code

Claude Code is the odd one out. There’s no IDE. No editor. You run it in your terminal, give it a task in plain English, and it reads your files, writes code, runs tests, commits to git, and iterates until the job is done. You watch it work rather than working alongside it.

This sounds abstract but in practice it’s genuinely different from the other two. You don’t write code with Claude Code so much as review code it wrote. The workflow is: describe what needs to happen, review the diff, approve or iterate.

What it actually costs in 2026:

Claude Code is included in your Claude subscription. Claude Pro ($20/month): access to Claude Code, roughly 45 messages per 5-hour window. Claude Max 5x ($100/month): 5x the Pro usage, priority access. Claude Max 20x ($200/month): 20x Pro, essentially no practical limits for most full-day developers. API (pay-as-you-go): Claude Sonnet 4.6 at $3 per million input tokens / $15 per million output tokens. Average cost via API is $6 per developer per day for typical usage, or $100-200/month for heavy use.

The key insight: if you’re already paying $20/month for Claude Pro to use claude.ai, Claude Code costs you nothing extra. That changes the value calculation significantly.

The real pros:

  • If you already pay for Claude Pro, it’s free. No additional subscription
  • Best agentic capability of the three for complex, multi-step tasks
  • Can run terminal commands, execute tests, and manage git, not just edit files
  • Agent Teams (multiple parallel Claude Code instances) for large codebases
  • Works with any editor. Completely editor-agnostic
  • 1 million token context window means it can hold your entire codebase in memory

The real cons:

  • No IDE means no inline suggestions or autocomplete as you type
  • Pro plan’s 45 messages per 5-hour window runs out fast during intensive sessions
  • The terminal-only workflow takes adjustment. It’s not familiar to most developers
  • Heavy daily usage on Pro limits is frustrating. Max plans are expensive
  • Less suitable for quick line-by-line edits where Cursor or Copilot feel faster

Who should NOT use Claude Code:
If you want AI help while actively typing code (autocomplete, inline suggestions), Claude Code doesn’t do that. It’s for task-level work, not keystroke-level assistance. Also avoid it as your primary tool if you’re on Pro and doing intensive sessions, as you’ll hit limits quickly. And if you’re not already paying for Claude, you need to add a subscription just to use it.

Head-to-Head: What Actually Matters

Price for solo developers:
Copilot Pro wins at $10/month. Claude Code wins if you already have Claude Pro (net $0 extra). Cursor is $20/month. Copilot is the cheapest standalone option.

Autocomplete and inline suggestions:
Cursor is best, with contextual block-level prediction. Copilot is strong and works in more editors. Claude Code has none. It’s not that kind of tool.

Multi-file feature development:
Claude Code is the strongest here. Give it a specification and it executes across your entire codebase. Cursor Composer is close and more interactive. Copilot’s agent mode is improving but lags.

Laravel / PHP development specifically:
All three work with PHP and Laravel. Copilot has the broadest IDE support including PhpStorm. Cursor is VS Code only. Claude Code is editor-agnostic and handles large Laravel codebases well in the terminal.

Best for building a SaaS from scratch:
Cursor for the build phase where you’re typing a lot. Claude Code for larger refactors, adding features across many files, or automation tasks. Many developers use both.

Team use:
Copilot Business at $19/user is the most established team option. Cursor Business at $40/user is expensive. Claude Code Team at $100/seat/month is premium but powerful for engineering teams.

The Honest Recommendation by Developer Type

Use GitHub Copilot Pro if: You want AI coding help with zero workflow change, you’re on a budget ($10/month is hard to beat), or you use JetBrains, Vim, Xcode, or any non-VS-Code editor.

Use Cursor Pro if: You’re primarily a VS Code user, you want the most polished AI-native IDE, and you’re comfortable paying $20/month for a noticeably better autocomplete and multi-file editing experience than Copilot.

Use Claude Code if: You already pay for Claude Pro and want to get more from that subscription, you prefer describing tasks over typing code, or you’re working on large refactors and multi-file features where its agentic approach outperforms the IDE-based tools.

The combination most solo developers end up with:
Cursor or Copilot for daily coding, Claude Code for the bigger tasks. They’re not mutually exclusive, and several of the tools in our automation tools comparison are built by developers using exactly this stack.

FAQ

Is GitHub Copilot free in 2026?

Yes. GitHub Copilot has a free tier with 2,000 code completions and 50 premium requests per month. It’s the most generous free tier of the three tools here. The Pro plan at $10/month unlocks unlimited completions and 300 premium requests.

Do I need a separate subscription for Claude Code?

No. Claude Code is included in your Claude subscription. Claude Pro at $20/month gives you access. If you’re already paying for Claude Pro to use claude.ai, adding Claude Code costs nothing extra.

Can I use Cursor with JetBrains?

No. Cursor is a fork of VS Code and only works as a VS Code replacement. If you use IntelliJ, PhpStorm, WebStorm, or any other JetBrains IDE, GitHub Copilot is the better option since it supports all major IDEs natively.

Which AI coding tool is best for Laravel development?

All three support PHP and Laravel well. GitHub Copilot works inside PhpStorm which is the preferred IDE for many Laravel developers. Cursor is VS Code only. Claude Code is terminal-based and works regardless of your editor, making it useful for Laravel developers who want to offload large refactors or generate boilerplate.

Is Claude Code better than Cursor?

They’re different tools for different jobs. Cursor excels at inline assistance, autocomplete, and interactive file editing. Claude Code excels at autonomous multi-step tasks. Most developers find them complementary rather than competing. If forced to pick one: Cursor for daily coding flow, Claude Code for bigger feature work.

Conclusion

The AI coding tool market in 2026 has settled into three clear lanes. GitHub Copilot is the affordable, low-disruption option that fits into any existing workflow. Cursor is the premium IDE-replacement for VS Code users who want the deepest AI integration. Claude Code is the agentic option for developers who want to describe tasks and review results rather than type alongside an AI.

None of them is obviously the best for everyone. The right answer depends on how you code, what editor you use, and whether you’re already paying for Claude.

Start with Copilot’s free tier. If you want more, upgrade to Pro at $10/month. Only switch to Cursor or add Claude Code when you have a clear reason to. Not because of FOMO.

Canonical tag mistakes in WordPress sites before launch

Part of the series: WordPress Pre-Launch Technical Checks

When reviewing a WordPress site before launch, there are a few technical signals that are easy to overlook. One of them is the canonical tag. It’s not something you see on the page, so it often gets less attention than it should.

In practice, this tag plays a key role in how search engines interpret the structure of the site. If it’s not configured properly, different URLs can end up competing with each other without anyone noticing.

Before delivering a project, it’s usually worth checking that canonical behavior is clean and consistent across the site.

What the canonical tag actually does

The canonical tag lives in the HTML head and tells search engines which version of a page should be treated as the main one.

A typical example looks like this:

<link rel="canonical" href="https://example.com/page/" />

It might seem like a small detail, but it helps avoid situations where multiple URLs point to the same content without a clear reference.

This happens more often than expected, especially with parameters, alternate paths or small variations of the same page.

Why these issues appear in WordPress

WordPress generates a wide range of URLs by default. Archives, paginated pages, previews or parameter-based URLs are all part of its normal behavior.

Because of that, canonical signals are usually handled automatically by themes or SEO plugins. And that’s where things can get a bit messy if no one reviews the final setup.

It’s quite easy to assume everything is fine, when in reality the signals don’t fully match the intended structure.

What tends to go wrong

Missing canonical tags

Some pages, especially custom ones, end up without a canonical tag. It’s not always obvious, but it leaves search engines without a clear reference.

Canonical pointing to the wrong place

After migrations, it’s not unusual to find canonicals still pointing to staging domains or older URLs. It’s one of those things that slips through if no one checks it directly.

Multiple canonical tags

Sometimes both the theme and an SEO plugin generate their own canonical tag. When that happens, you can end up with conflicting signals in the page head.

Inconsistent behavior across templates

Different templates can produce slightly different canonical logic. On their own it might not seem critical, but across the site it creates mixed signals.

A quick review before launch

At this stage, a simple check is usually enough. Just confirm that important pages have a canonical tag, that it points to the correct production URL, and that there are no duplicates or leftover references.

It doesn’t take much time, but it helps avoid confusion later on.

Why this belongs in a pre-launch checklist

Canonical issues rarely break anything visually. The site loads, navigation works, everything seems fine.

But under the surface, they affect how search engines interpret the site structure. Fixing them after launch is possible, but it’s cleaner to get it right from the start.

Where PreFlight fits in

PreFlight focuses on reviewing these kinds of technical details before a WordPress site is delivered or published. It’s about catching small inconsistencies before they turn into bigger issues.

If you want to run a quick technical check before launch, you can start here:
https://preflightstandard.com/

Final thought

The canonical tag is a small piece of code, but it carries a lot of weight when it comes to how a site is understood.

Taking a moment to review it before launch helps keep everything aligned from day one.

SAP Commerce Catalog System Deep Dive: Staged Content, Synchronization, and Multi-Catalog Architecture

The catalog system is one of SAP Commerce’s most powerful — and most misunderstood — features. It provides a staging mechanism that lets business users prepare product data, pricing, and content changes in isolation, then publish everything to the live storefront in a single atomic operation. When used correctly, it eliminates the “edit live data and hope nothing breaks” anti-pattern. When used incorrectly, it becomes a source of synchronization headaches, data inconsistency, and performance bottlenecks.

This article breaks down how the catalog system actually works, how synchronization moves data from staged to online, and how to architect multi-catalog setups for complex B2B and multi-brand scenarios.

The Two-Version Model

Every catalog in SAP Commerce has at least two catalog versions:

┌─────────────────────────────────────────┐
│  Product Catalog: "electronicsProductCatalog"  │
│                                         │
│  ┌───────────────┐    ┌──────────────┐ │
│  │   Staged       │───>│   Online     │ │
│  │   Version      │sync│   Version    │ │
│  │               │───>│              │ │
│  │ • Edit here   │    │ • Read-only  │ │
│  │ • Preview     │    │ • Live data  │ │
│  │ • No customer │    │ • Storefront │ │
│  │   impact      │    │   serves this│ │
│  └───────────────┘    └──────────────┘ │
└─────────────────────────────────────────┘

Staged: The working version. Business users, product managers, and content editors make all their changes here. Products are created, prices are updated, categories are reorganized — all without affecting what customers see.

Online: The published version. The storefront (OCC API, Accelerator, Spartacus) reads data from this version. It’s effectively read-only from a business perspective — changes should only come through synchronization from Staged.

Why This Matters

Without the staged/online model, every product edit would immediately be visible to customers. Imagine changing a product price from $99 to $79 for a Black Friday promotion — but the promotion doesn’t start until Friday at midnight. With direct editing, you’d have to set the new price at exactly midnight. With staged/online, you set the price in Staged days in advance and synchronize at midnight.

Setting Up a Catalog

Catalogs are created via ImpEx:

INSERT_UPDATE Catalog;id[unique=true];name[lang=en];defaultCatalog
;myProductCatalog;My Product Catalog;true

# Create catalog versions
INSERT_UPDATE CatalogVersion;catalog(id)[unique=true];version[unique=true];active;languages(isocode)
;myProductCatalog;Staged;false;en,de,fr
;myProductCatalog;Online;true;en,de,fr

The active=true flag marks the Online version as the one served to customers.

How Synchronization Works

Synchronization is the process of copying data from one catalog version to another — typically from Staged to Online.

The Sync Job

A CatalogVersionSyncJob defines what to synchronize:

INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];sourceVersion(catalog(id),version)[unique=true];targetVersion(catalog(id),version)[unique=true];createNewItems;removeMissingItems
;myProductCatalogSyncJob;myProductCatalog:Staged;myProductCatalog:Online;true;true
Parameter Meaning
createNewItems If true, new items in Staged are created in Online
removeMissingItems If true, items deleted from Staged are deleted from Online

What Gets Synchronized

The sync job synchronizes all catalog-aware item types. An item type is catalog-aware if it has a catalogVersion attribute. The standard catalog-aware types include:

  • Product (and subtypes like VariantProduct)
  • Category
  • Media
  • Keyword
  • CatalogVersionSyncScheduleMedia

Custom types become catalog-aware when you add a catalogVersion attribute:

<itemtype code="CustomContentBlock" extends="GenericItem" autocreate="true" generate="true">
    <deployment table="CustomContentBlocks" typecode="25050"/>
    <attributes>
        <attribute qualifier="code" type="java.lang.String">
            <persistence type="property"/>
            <modifiers unique="true"/>
        </attribute>
        <attribute qualifier="catalogVersion" type="CatalogVersion">
            <persistence type="property"/>
            <modifiers read="true" write="true"/>
        </attribute>
        <!-- other attributes -->
    </attributes>
</itemtype>

Sync Internals: What Actually Happens

When a sync job runs, the platform:

  1. Identifies changed items — compares timestamps between source and target catalog versions
  2. Copies item data — creates or updates items in the target version with data from the source
  3. Resolves references — translates references from source catalog version to target (e.g., a product’s category reference in Staged maps to the corresponding category in Online)
  4. Handles media — copies media items and their binary data
  5. Maintains link consistency — ensures all relations (product-to-category, product-to-media) are consistent in the target version
Sync Process Flow:
┌─────────────────────────────────────────────────┐
│ 1. Collect modified items from source (Staged)   │
│ 2. For each item:                                │
│    a. Find/create counterpart in target (Online) │
│    b. Copy all synchronized attributes           │
│    c. Resolve catalog-version-specific references│
│    d. Update timestamps                          │
│ 3. Handle deletions (if removeMissingItems=true) │
│ 4. Update sync status and counters               │
└─────────────────────────────────────────────────┘

Counterpart Resolution

During synchronization, the platform needs to find the “counterpart” of a Staged item in Online. It uses unique keys for this:

  • For Product: code + catalogVersion
  • For Category: code + catalogVersion
  • For Media: code + catalogVersion

A product with code PROD-001 in myProductCatalog:Staged maps to PROD-001 in myProductCatalog:Online. If the Online counterpart doesn’t exist and createNewItems=true, it’s created.

Synchronization Performance

Catalog sync is one of the most resource-intensive operations in SAP Commerce. Understanding and optimizing it is critical for production environments.

Full vs. Incremental Sync

Full sync: Compares every item in source and target. Safe but slow. Use only during initial setup or when data consistency is questionable.

Incremental sync: Only processes items modified since the last sync. Much faster. Use for regular production synchronization.

// Programmatic incremental sync
SyncConfig config = new SyncConfig();
config.setSynchronizationType(SyncConfig.INCREMENTAL);
config.setCreateSavedValues(false);
config.setLogToDatabase(false);
config.setLogToFile(false);

catalogSynchronizationService.synchronize(syncJob, config);

Performance Tuning

# Number of parallel sync workers
catalog.sync.workers=4

# Batch size — number of items processed per transaction
catalog.sync.batch.size=100

# Disable interceptors during sync (significant speedup)
catalog.sync.enable.interceptors=false

# Skip saved values (audit trail) for speed
catalog.sync.create.saved.values=false

Timing and Scheduling

Schedule syncs during low-traffic periods:

INSERT_UPDATE Trigger;cronJob(code)[unique=true];cronExpression
;myProductCatalogSyncJob;0 0 3 * * ?

For time-critical syncs (flash sales, price changes), trigger programmatically:

@Resource
private CatalogSynchronizationService catalogSynchronizationService;

public void publishCatalogChanges() {
    CatalogVersionSyncJobModel syncJob = // lookup sync job
    SyncConfig config = new SyncConfig();
    config.setSynchronizationType(SyncConfig.INCREMENTAL);

    CatalogVersionSyncCronJobModel cronJob = catalogSynchronizationService.synchronize(syncJob, config);
    LOG.info("Sync started: {}", cronJob.getCode());
}

Content Catalogs

In addition to product catalogs, SAP Commerce uses content catalogs for CMS content (pages, components, slots). The same staged/online model applies:

INSERT_UPDATE ContentCatalog;id[unique=true];name[lang=en]
;myContentCatalog;My Content Catalog

INSERT_UPDATE CatalogVersion;catalog(id)[unique=true];version[unique=true];active
;myContentCatalog;Staged;false
;myContentCatalog;Online;true

CMS Content Structure

Content Catalog (Staged)
├── PageTemplate: ProductDetailPageTemplate
├── ContentPage: homepage
│   ├── ContentSlot: Section1
│   │   ├── BannerComponent: heroBanner
│   │   └── ProductCarouselComponent: newArrivals
│   ├── ContentSlot: Section2
│   │   └── CategoryFeatureComponent: topCategories
│   └── ContentSlot: Footer
│       └── FooterNavigationComponent: mainFooter
├── ContentPage: faqPage
└── ContentPage: contactPage

Content catalog sync works identically to product catalog sync — changes made to CMS pages in Staged are published to Online through synchronization.

SmartEdit Integration

SmartEdit (SAP Commerce’s visual CMS editor) works exclusively with the Staged content catalog. Editors drag and drop components, rearrange content slots, and preview pages — all in Staged. When they click “Sync,” the content catalog synchronization job runs and publishes changes to Online.

Multi-Catalog Architecture

Real-world projects often need multiple catalogs. Here are the common patterns.

Pattern 1: Shared Master Catalog

Multiple storefronts share a master product catalog but have separate content catalogs:

┌─────────────────────────────────┐
│     Master Product Catalog       │
│  (Staged → Online)              │
├────────────┬────────────────────┤
│            │                    │
│  ┌─────────▼──────┐ ┌─────────▼──────┐
│  │ US Content     │ │ EU Content     │
│  │ Catalog        │ │ Catalog        │
│  │ (Staged→Online)│ │ (Staged→Online)│
│  └────────────────┘ └────────────────┘
# Shared product catalog
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;globalProductCatalog;Global Product Catalog

# Region-specific content catalogs
INSERT_UPDATE ContentCatalog;id[unique=true];name[lang=en]
;usContentCatalog;US Content Catalog
;euContentCatalog;EU Content Catalog

Pattern 2: Multi-Country Product Catalogs

Each country has its own product catalog with country-specific pricing, availability, and product assortment:

┌──────────────────┐
│ Global Product    │
│ Catalog (Master)  │
│ Staged → Online   │
└────────┬─────────┘
         │ sync
    ┌────┴────┐
    │         │
┌───▼────┐ ┌─▼──────┐
│ US     │ │ DE     │
│ Product│ │ Product│
│ Catalog│ │ Catalog│
│ S → O  │ │ S → O  │
└────────┘ └────────┘
# Global master catalog
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;globalProductCatalog;Global Product Catalog

# Country catalogs
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;usProductCatalog;US Product Catalog
;deProductCatalog;DE Product Catalog

# Cross-catalog sync: Global Staged → US Staged
INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];sourceVersion(catalog(id),version);targetVersion(catalog(id),version)
;globalToUsSyncJob;globalProductCatalog:Staged;usProductCatalog:Staged
;globalToDeSyncJob;globalProductCatalog:Staged;deProductCatalog:Staged

# Then US Staged → US Online
INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];sourceVersion(catalog(id),version);targetVersion(catalog(id),version)
;usProductCatalogSyncJob;usProductCatalog:Staged;usProductCatalog:Online
;deProductCatalogSyncJob;deProductCatalog:Staged;deProductCatalog:Online

Pattern 3: B2B Multi-Catalog with Visibility Rules

B2B scenarios often require different product visibility per customer group:

# Base catalog visible to all customers
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;baseProductCatalog;Base Product Catalog

# Premium catalog with additional products
INSERT_UPDATE Catalog;id[unique=true];name[lang=en]
;premiumProductCatalog;Premium Product Catalog

# Map catalogs to customer groups via CMSSite
INSERT_UPDATE CMSSite;uid[unique=true];contentCatalogs(id);productCatalogs(id)
;standardSite;standardContentCatalog;baseProductCatalog
;premiumSite;premiumContentCatalog;baseProductCatalog,premiumProductCatalog

Catalog-Aware Queries

When querying data, you must always specify the catalog version. This is one of the most common sources of bugs — forgetting the catalog version filter returns data from all versions (Staged + Online), causing duplicates.

FlexibleSearch with Catalog Version

// CORRECT: Always include catalogVersion in queries
String query = "SELECT {pk} FROM {Product} WHERE {code} = ?code AND {catalogVersion} = ?cv";
FlexibleSearchQuery fsq = new FlexibleSearchQuery(query);
fsq.addQueryParameter("code", "PROD-001");
fsq.addQueryParameter("cv", catalogVersionService.getCatalogVersion("myProductCatalog", "Online"));
// WRONG: Missing catalogVersion — returns products from ALL versions
String query = "SELECT {pk} FROM {Product} WHERE {code} = ?code";
// This returns PROD-001 from both Staged AND Online — likely a bug

Session Catalog Versions

SAP Commerce maintains a set of “session catalog versions” — the catalog versions the current user should see. The storefront sets Online versions; Backoffice sets Staged versions.

// Get the session catalog version
Collection<CatalogVersionModel> sessionVersions = catalogVersionService.getSessionCatalogVersions();

// Set session catalog versions (typically done by the storefront filter chain)
catalogVersionService.setSessionCatalogVersions(
    Collections.singletonList(
        catalogVersionService.getCatalogVersion("myProductCatalog", "Online")
    )
);

Services like ProductService.getProductForCode() automatically filter by session catalog versions, so you don’t need to pass the catalog version explicitly when using the service layer.

Category Hierarchies

Categories are catalog-version-aware and form a tree structure within each catalog version.

Defining Categories

$catalogVersion = catalogVersion(catalog(id[default='myProductCatalog']),version[default='Staged'])

INSERT_UPDATE Category;code[unique=true];$catalogVersion;name[lang=en];supercategories(code,$catalogVersion)
;electronics;;Electronics;
;cameras;;Cameras;electronics
;dslr;;DSLR Cameras;cameras
;mirrorless;;Mirrorless Cameras;cameras
;smartphones;;Smartphones;electronics
;accessories;;Accessories;electronics
;cases;;Phone Cases;accessories
;chargers;;Chargers;accessories

This creates a hierarchy:

Electronics
├── Cameras
│   ├── DSLR Cameras
│   └── Mirrorless Cameras
├── Smartphones
└── Accessories
    ├── Phone Cases
    └── Chargers

Assigning Products to Categories

$catalogVersion = catalogVersion(catalog(id[default='myProductCatalog']),version[default='Staged'])

INSERT_UPDATE CategoryProductRelation;source(code,$catalogVersion)[unique=true];target(code,$catalogVersion)[unique=true]
;dslr;CANON-EOS-R5
;dslr;NIKON-Z8
;mirrorless;SONY-A7IV
;smartphones;IPHONE-15
;cases;CASE-IPHONE-15

A product can belong to multiple categories. During synchronization, these relationships are copied from Staged to Online.

Handling Sync Conflicts and Edge Cases

Partial Sync

Sometimes you need to sync only specific items rather than the entire catalog:

// Sync specific products only
List<ItemModel> itemsToSync = Arrays.asList(product1, product2, product3);
SyncConfig config = new SyncConfig();
config.setSynchronizationType(SyncConfig.PARTIAL);

for (ItemModel item : itemsToSync) {
    catalogSynchronizationService.synchronizeItem(item, syncJob, config);
}

Sync Excludes

Exclude specific attributes from synchronization:

# Don't sync the 'internalNotes' attribute — keep it Staged-only
INSERT_UPDATE CatalogVersionSyncJob;code[unique=true];&syncJobRef
;myProductCatalogSyncJob;syncJobRef

INSERT_UPDATE SyncAttributeDescriptorConfig;syncJob(&syncJobRef)[unique=true];attributeDescriptor(qualifier,enclosingType(code))[unique=true];includedInSync
;syncJobRef;internalNotes:Product;false

Avoiding Common Sync Problems

1. Missing references: If a product references a category that hasn’t been synced yet, the sync fails. Solution: sync categories before products, or sync both in the same job.

2. Orphaned items: If removeMissingItems=true and someone accidentally deletes items from Staged, they’ll be deleted from Online too. Consider setting removeMissingItems=false for safety and handling deletions manually.

3. Large media files: Syncing catalogs with thousands of high-resolution images is slow. Consider using external media (CDN URLs) instead of storing media in the catalog.

4. Concurrent modifications: If two users modify the same product in Staged while a sync is running, the sync might pick up an inconsistent state. Best practice: lock the catalog (or communicate a sync window) during synchronization.

Monitoring Sync Status

Via HAC

Navigate to Platform → System → Catalog Version Syncing to see:

  • Last sync date
  • Number of items synced
  • Sync status (SUCCESS, ERROR, RUNNING)
  • Error details

Programmatically

CatalogVersionSyncCronJobModel syncCronJob = // lookup
LOG.info("Sync status: {}", syncCronJob.getStatus());
LOG.info("Sync result: {}", syncCronJob.getResult());
LOG.info("Items processed: {}", syncCronJob.getProcessedItemsCount());
LOG.info("Items synced: {}", syncCronJob.getSyncedItemsCount());
LOG.info("Errors: {}", syncCronJob.getFailedItemsCount());

Sync Audit

Enable saved values to track what changed during each sync:

catalog.sync.create.saved.values=true

This creates SavedValues items that record the before/after state of each synced attribute. Useful for debugging but adds significant overhead — disable in production unless needed.

Best Practices

  1. Always work in Staged — never modify Online directly. All changes should flow through sync.
  2. Use incremental sync — full sync is only for initial setup or data repair.
  3. Schedule syncs during off-peak hours — sync is CPU and memory intensive.
  4. Separate product and content syncs — they have different schedules and different business owners.
  5. Test sync with production-like data volumes — sync that works with 100 products may fail with 500,000.
  6. Monitor sync duration trends — increasing sync times indicate growing data complexity or missing indexes.
  7. Always include catalogVersion in FlexibleSearch queries — the most common source of data bugs.
  8. Plan your multi-catalog architecture early — changing catalog structure after go-live requires data migration.

Summary

The SAP Commerce catalog system is a content staging and publishing framework. Its two-version model provides safety for business operations, while synchronization gives you atomic publishing of changes. The key principles:

  1. Staged is for editing, Online is for serving — never break this boundary
  2. Synchronization copies data from Staged to Online — it resolves references, handles media, and maintains consistency
  3. Incremental sync is your production workhorse — full sync is for exceptional circumstances
  4. Multi-catalog architectures serve multi-country and B2B scenarios — plan the catalog hierarchy carefully
  5. Always filter by catalog version in queries — missing this filter is the most common catalog-related bug
  6. Monitor and tune sync performance — it’s one of the most resource-intensive platform operations

Understanding the catalog system deeply is what allows you to build commerce solutions that scale — both technically and operationally.

The Architectural Problem With Compliance-as-a-Service

A major compliance automation platform was just exposed for allegedly fabricating audit evidence at scale. 493 of 494 SOC 2 reports contained the same typo. 259 Type II audits all claimed zero incidents. Pre-written conclusions were generated before any control was tested.

If you’re an engineer at a company that relies on a compliance vendor, this should bother you — not just ethically, but architecturally.

The separation-of-concerns problem

In software engineering, we understand that the component producing data should not also be the component validating it. You don’t let a service write its own health checks and also be the only thing reading them.

Compliance frameworks have the same requirement. AICPA AT-C Section 205 mandates that the entity helping you implement controls cannot also produce the audit conclusions about those controls. This isn’t bureaucratic overhead. It’s the same separation of concerns we apply to every distributed system.

The platform in question broke this fundamental principle. It generated evidence, wrote the audit conclusions, and routed everything through audit firms that were essentially shell entities. The auditor and the implementer were in the same system.

What real compliance evidence looks like

If your compliance tool is doing its job, the evidence it produces should be traceable. Not template text that a human clicked “Accept” on, but actual artifacts pulled from your infrastructure:

  • Cloud configuration snapshots from AWS, Azure, or GCP via API integration
  • Access control logs pulled from your identity provider
  • Code review and deployment evidence from your CI/CD pipeline
  • Encryption-at-rest and in-transit configurations verified against live infrastructure
  • Incident response records tied to your actual ticketing system

This is why we built Cyberbase the way we did. Our platform connects directly to your infrastructure and pulls evidence from your live environment. When it says a control is in place, it’s because an integration verified it — not because someone populated a template.

The trust center as a source of truth

Your trust page is increasingly the first thing a prospective customer’s security team checks before they engage with sales. If the claims on that page don’t correspond to implemented, tested, verified controls, you have a liability problem.

We created Cyberbase’s Trust Center to reflect your real security posture. It surfaces the controls you’ve actually implemented and keeps them synchronized with your environment. YSecurity’s consultants and virtual CISOs provide the expert human layer that ensures those controls are designed correctly in the first place.

Five questions for your next vendor review

  1. Can you trace every piece of evidence back to a real event, configuration, or action in your environment?
  2. Does the platform that generates evidence also produce or influence the audit conclusions?
  3. Are the integrations reading from your actual infrastructure, or just from a configuration form?
  4. Is the audit firm performing your examination genuinely independent?
  5. Does your trust page reflect verified controls, or aspirational checkboxes?

The only compliance that matters is the kind that holds up when someone actually checks.

Originally published at https://www.cyberbase.ai/blog/the-compliance-industry-has-a-trust-problem-here-s-how-we-fix-it
Cyberbase is an AI-powered compliance platform. YSecurity is a cybersecurity services company. Together: compliance built on substance.

Redis INCR: O comando atômico que você deveria conhecer

Em sistemas distribuídos, contar algo corretamente é mais difícil do que parece. Dois processos lendo e atualizando o mesmo valor podem facilmente gerar race conditions e inconsistências.

É exatamente aqui que o comando INCR do Redis brilha.

O INCR resolve 3 problemas clássicos:

  • contadores distribuídos
  • rate limiting
  • geração de IDs

Vamos entender por que esse comando aparentemente simples é tão poderoso.

O que é o comando INCR?

O INCR incrementa o valor numérico armazenado em uma chave por 1. Se a chave não existir, ela é criada com valor 0 antes do incremento, resultando em 1.

> SET contador 10
OK
> INCR contador
(integer) 11
> INCR contador
(integer) 12

Se a chave não existe:

> INCR novo_contador
(integer) 1

A família completa de comandos de incremento inclui:

Comando Descrição
INCR Incrementa por 1
INCRBY Incrementa por um valor inteiro específico
INCRBYFLOAT Incrementa por um valor decimal
DECR Decrementa por 1
DECRBY Decrementa por um valor inteiro específico

Por que o INCR é especial?

A característica mais importante do INCR é sua atomicidade. Em sistemas distribuídos, onde múltiplas instâncias da aplicação podem tentar modificar o mesmo valor simultaneamente, a atomicidade garante que cada operação seja executada de forma isolada.

A atomicidade do INCR é possível porque o Redis executa comandos de forma sequencial em um único thread. Isso significa que não existe concorrência interna entre comandos: cada operação é processada por completo antes da próxima iniciar. Essa característica de single-thread para execução de comandos é o que torna operações como INCR naturalmente seguras em ambientes concorrentes.

Considere o cenário clássico de race condition:

Tempo    Processo A          Processo B          Valor Real
─────────────────────────────────────────────────────────────
T1       Lê valor: 10                            10
T2                           Lê valor: 10        10
T3       Calcula: 10 + 1                         10
T4                           Calcula: 10 + 1     10
T5       Escreve: 11                             11
T6                           Escreve: 11         11  ← Perdemos um incremento!

Com INCR, o Redis garante que isso não acontece. A operação de leitura-incremento-escrita é executada como uma única unidade atômica.

Casos de Uso Práticos

1. Contadores de visualização

Um dos casos mais comuns é contar visualizações de páginas ou recursos:

public class ViewCounterService
{
    private readonly IDatabase _redis;
    private readonly TimeSpan _cacheExpiry = TimeSpan.FromHours(24);

    public ViewCounterService(IConnectionMultiplexer connection)
    {
        _redis = connection.GetDatabase();
    }

    public async Task<long> IncrementViewCountAsync(string resourceId)
    {
        var key = $"views:{resourceId}";
        var count = await _redis.StringIncrementAsync(key);

        // Define TTL apenas na primeira visualização
        if (count == 1)
        {
            await _redis.KeyExpireAsync(key, _cacheExpiry);
        }

        return count;
    }

    public async Task<long> GetViewCountAsync(string resourceId)
    {
        var key = $"views:{resourceId}";
        var value = await _redis.StringGetAsync(key);
        return value.HasValue ? (long)value : 0;
    }
}

2. Rate Limiting com Fixed Window

O rate limiting é essencial para proteger APIs contra abusos. O INCR combinado com TTL cria um rate limiter baseado em fixed window, simples e eficiente:

public class RateLimiter
{
    private readonly IDatabase _redis;

    public RateLimiter(IConnectionMultiplexer connection)
    {
        _redis = connection.GetDatabase();
    }

    public async Task<RateLimitResult> CheckRateLimitAsync(
        string clientId, 
        int maxRequests, 
        TimeSpan window)
    {
        var key = $"ratelimit:{clientId}:{DateTime.UtcNow.Ticks / window.Ticks}";

        var count = await _redis.StringIncrementAsync(key);

        if (count == 1)
        {
            await _redis.KeyExpireAsync(key, window);
        }

        return new RateLimitResult
        {
            IsAllowed = count <= maxRequests,
            CurrentCount = count,
            Limit = maxRequests,
            RetryAfter = count > maxRequests 
                ? await _redis.KeyTimeToLiveAsync(key) 
                : null
        };
    }
}

public record RateLimitResult
{
    public bool IsAllowed { get; init; }
    public long CurrentCount { get; init; }
    public int Limit { get; init; }
    public TimeSpan? RetryAfter { get; init; }
}

Nota: Essa abordagem usa fixed window counter, onde cada janela de tempo tem seu próprio contador. Para cenários que exigem sliding window real, considere usar Sorted Sets com ZADD/ZRANGEBYSCORE ou um algoritmo de token bucket. Além disso, vale observar que o padrão INCR seguido de EXPIRE pode sofrer uma race condition sutil: outro processo pode incrementar a chave entre o INCR e o EXPIRE. Para maior robustez, você pode encapsular ambos em um script Lua, garantindo a execução atômica de INCR + EXPIRE.

Uso em um middleware ASP.NET Core:

public class RateLimitingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly RateLimiter _rateLimiter;

    public RateLimitingMiddleware(RequestDelegate next, RateLimiter rateLimiter)
    {
        _next = next;
        _rateLimiter = rateLimiter;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        var clientId = context.Connection.RemoteIpAddress?.ToString() ?? "unknown";
        var result = await _rateLimiter.CheckRateLimitAsync(
            clientId, 
            maxRequests: 100, 
            window: TimeSpan.FromMinutes(1));

        context.Response.Headers["X-RateLimit-Limit"] = result.Limit.ToString();
        context.Response.Headers["X-RateLimit-Remaining"] = 
            Math.Max(0, result.Limit - result.CurrentCount).ToString();

        if (!result.IsAllowed)
        {
            context.Response.StatusCode = StatusCodes.Status429TooManyRequests;
            if (result.RetryAfter.HasValue)
            {
                context.Response.Headers["Retry-After"] = 
                    ((int)result.RetryAfter.Value.TotalSeconds).ToString();
            }
            return;
        }

        await _next(context);
    }
}

3. Geração de IDs sequenciais

Quando você precisa de IDs sequenciais em um sistema distribuído:

public class SequentialIdGenerator
{
    private readonly IDatabase _redis;
    private readonly string _prefix;

    public SequentialIdGenerator(IConnectionMultiplexer connection, string prefix = "id")
    {
        _redis = connection.GetDatabase();
        _prefix = prefix;
    }

    public async Task<long> GenerateIdAsync(string entity)
    {
        var key = $"{_prefix}:{entity}:sequence";
        return await _redis.StringIncrementAsync(key);
    }

    public async Task<string> GenerateFormattedIdAsync(string entity, string format = "{0}-{1:D8}")
    {
        var id = await GenerateIdAsync(entity);
        return string.Format(format, entity.ToUpper(), id);
    }
}

// Uso
var generator = new SequentialIdGenerator(connection);
var orderId = await generator.GenerateFormattedIdAsync("order"); 
// Resultado: "ORDER-00000001"

4. Contadores de sessões ativas

Monitorar quantos usuários estão ativos em tempo real:

public class ActiveSessionTracker
{
    private readonly IDatabase _redis;
    private const string ActiveSessionsKey = "sessions:active:count";
    private const string SessionPrefix = "session:";

    public ActiveSessionTracker(IConnectionMultiplexer connection)
    {
        _redis = connection.GetDatabase();
    }

    public async Task<string> StartSessionAsync(string userId)
    {
        var sessionId = Guid.NewGuid().ToString("N");
        var sessionKey = $"{SessionPrefix}{sessionId}";

        var transaction = _redis.CreateTransaction();

        _ = transaction.StringSetAsync(sessionKey, userId, TimeSpan.FromMinutes(30));
        _ = transaction.StringIncrementAsync(ActiveSessionsKey);

        await transaction.ExecuteAsync();

        return sessionId;
    }

    public async Task EndSessionAsync(string sessionId)
    {
        var sessionKey = $"{SessionPrefix}{sessionId}";

        var existed = await _redis.KeyDeleteAsync(sessionKey);

        if (existed)
        {
            await _redis.StringDecrementAsync(ActiveSessionsKey);
        }
    }

    public async Task<long> GetActiveSessionCountAsync()
    {
        var value = await _redis.StringGetAsync(ActiveSessionsKey);
        return value.HasValue ? (long)value : 0;
    }
}

Nota: Nessa abordagem, se uma sessão expirar automaticamente pelo TTL, o contador ActiveSessionsKey não será decrementado, o que pode gerar drift ao longo do tempo. Para maior robustez em produção, considere usar um Sorted Set (ZSET) com timestamps para rastrear sessões ativas, permitindo que você calcule o total real a qualquer momento com ZRANGEBYSCORE, ou implemente um processo periódico de reconciliação do contador.

5. Métricas por intervalo de tempo

Coletar métricas agregadas por minuto/hora/dia:

public class MetricsCollector
{
    private readonly IDatabase _redis;

    public MetricsCollector(IConnectionMultiplexer connection)
    {
        _redis = connection.GetDatabase();
    }

    public async Task RecordMetricAsync(string metricName, long value = 1)
    {
        var now = DateTime.UtcNow;

        var keys = new[]
        {
            $"metrics:{metricName}:minute:{now:yyyyMMddHHmm}",
            $"metrics:{metricName}:hour:{now:yyyyMMddHH}",
            $"metrics:{metricName}:day:{now:yyyyMMdd}"
        };

        var expiries = new[]
        {
            TimeSpan.FromHours(2),
            TimeSpan.FromDays(2),
            TimeSpan.FromDays(60)
        };

        var batch = _redis.CreateBatch();

        for (int i = 0; i < keys.Length; i++)
        {
            var key = keys[i];
            var expiry = expiries[i];

            _ = batch.StringIncrementAsync(key, value);
            _ = batch.KeyExpireAsync(key, expiry, ExpireWhen.HasNoExpiry);
        }

        batch.Execute();
    }

    public async Task<Dictionary<string, long>> GetHourlyMetricsAsync(
        string metricName, 
        DateTime date)
    {
        var results = new Dictionary<string, long>();

        for (int hour = 0; hour < 24; hour++)
        {
            var key = $"metrics:{metricName}:hour:{date:yyyyMMdd}{hour:D2}";
            var value = await _redis.StringGetAsync(key);
            results[$"{hour:D2}:00"] = value.HasValue ? (long)value : 0;
        }

        return results;
    }
}

Sistemas Distribuídos com múltiplos pods

Em ambientes Kubernetes ou qualquer arquitetura com múltiplas instâncias, o Redis INCR se torna ainda mais valioso. Cada pod opera de forma independente, sem conhecimento dos outros, mas todos precisam coordenar acesso a recursos compartilhados.

O Problema do estado compartilhado

Imagine 5 pods processando requisições simultaneamente. Sem coordenação central, cada pod manteria seu próprio contador local:

Pod 1: contador local = 47
Pod 2: contador local = 52
Pod 3: contador local = 38
Pod 4: contador local = 61
Pod 5: contador local = 44
                         ───────
Total real de requisições: 242
Mas cada pod "acha" que processou apenas seu número local

Com Redis como ponto central de coordenação, todos os pods compartilham a mesma visão do estado.

E em Redis Cluster?

Em arquiteturas maiores que utilizam Redis Cluster, o INCR funciona normalmente desde que as chaves relacionadas estejam no mesmo hash slot. O Redis Cluster distribui chaves entre nós usando CRC16, então se você precisa que múltiplas chaves sejam processadas no mesmo nó, utilize hash tags. Por exemplo, {rate}:user:123 e {rate}:user:456 serão direcionadas para o mesmo slot por compartilharem a tag {rate}. Isso é especialmente relevante quando você usa transactions ou scripts Lua que operam sobre múltiplas chaves.

Considerações de Performance

O INCR é extremamente rápido, com complexidade O(1). No entanto, existem algumas práticas para otimizar ainda mais:

Batch de operações: Quando você precisa incrementar múltiplos contadores, use batching:

public async Task IncrementMultipleAsync(Dictionary<string, long> counters)
{
    var batch = _redis.CreateBatch();
    var tasks = new List<Task<long>>();

    foreach (var (key, increment) in counters)
    {
        tasks.Add(batch.StringIncrementAsync(key, increment));
    }

    batch.Execute();
    await Task.WhenAll(tasks);
}

Pipeline: Para operações que não dependem do resultado imediato:

public void RecordEventsFireAndForget(IEnumerable<string> eventKeys)
{
    foreach (var key in eventKeys)
    {
        _redis.StringIncrement(key, flags: CommandFlags.FireAndForget);
    }
}

Armadilhas Comuns

1. Não definir TTL: Contadores sem expiração podem acumular indefinidamente. Sempre considere se o contador precisa de um TTL.

2. Usar GET seguido de SET: Nunca faça isso para incrementar valores. Use sempre INCR:

// ERRADO: Race condition!
var value = await _redis.StringGetAsync(key);
await _redis.StringSetAsync(key, (long)value + 1);

// CORRETO: Atômico
await _redis.StringIncrementAsync(key);

3. Ignorar overflow: O Redis armazena inteiros como strings de até 64 bits signed. Em cenários extremos, considere verificar limites.

4. Não tratar valores não numéricos: Se uma chave contiver um valor não numérico, INCR retornará erro. Valide ou use chaves dedicadas.

Conclusão

O INCR do Redis é uma ferramenta fundamental para construir sistemas distribuídos robustos. Sua atomicidade elimina race conditions, sua velocidade permite uso em cenários de alta performance, e sua simplicidade torna a implementação direta.

Os casos de uso vão desde contadores simples até rate limiters sofisticados e sistemas de métricas em tempo real. O segredo está em entender que operações atômicas simples, quando combinadas corretamente, resolvem problemas complexos de concorrência.

O poder do Redis muitas vezes não está em comandos complexos, mas em operações simples que são atômicas, rápidas e distribuídas.

Na próxima vez que você precisar de um contador distribuído, pense no INCR antes de partir para soluções mais complexas. Frequentemente, a resposta mais simples é também a mais elegante.

I Built a Monitoring Tool Because Uptime Checks Kept Lying to Me

If you deploy sites to Netlify, Vercel, Cloudflare Pages or pretty much any modern platform — you’ve probably set up some kind of uptime monitoring. Ping every minute, get an alert if the server doesn’t respond. Pretty standard stuff.

I had the same setup across all my sites. And for a long time, I thought I was covered.

I wasn’t.

The Problem

A few months back I pushed an update to one of my Nuxt apps on Netlify. Build passed, deploy preview looked fine, dashboard showed green. I moved on.

Turns out the site was broken for hours. Not down — the server was responding just fine. 200 OK on every request. But the JavaScript bundle the page needed wasn’t loading, so the entire app was a blank page.

Here’s what happened: the new deploy generated fresh hashed bundles. main.c8d13.js replaced main.a4f2c.js. But some CDN edge nodes were still serving the old HTML, which referenced the old filename. Browser tries to fetch main.a4f2c.js, gets a 404, and nothing renders.

My uptime monitor? Green the entire time. Because the HTML document loaded. 200 OK. Job done, as far as it was concerned.

It Keeps Happening

Once I started paying attention, I found this happens a lot more than you’d expect:

MIME type mismatches — Your JS file gets served as text/html by the CDN. Browser silently refuses to execute it. No console error you’d notice unless you’re specifically looking. I had this on one of my sites behind Cloudflare for days without knowing.

Redirect chains that loop — CDN redirect + origin redirect + app redirect that conflict with each other. Browser hits the hop limit and gives up. Uptime tool sees the first 301 and moves on.

DNS still pointing at old infrastructure — After a migration, a subdomain resolves to the old origin. It responds with stale content from months ago. Technically “up.” Practically useless.

All return 200. All invisible to standard monitoring.

Why I Built Sitewatch

I looked around for tools that caught this stuff. Most uptime monitors check if the server responds — and that’s it. Some offer headless browser checks with Puppeteer, but they’re slow, expensive, and fragile at any real scale. Not great when you’re watching a bunch of sites.

What I needed was something in between. Goes deeper than a status code check, doesn’t require a full browser session. Catches the broken deploy, the MIME type issue, the redirect loop — automatically.

So I built Sitewatch.

What It Does

Instead of just checking if your server responds, Sitewatch verifies whether your site actually works:

Asset verification — It fetches your page, parses the HTML, finds your JS and CSS bundle references, and checks each one. Correct status code? Correct MIME type? Actually loadable? If your HTML references a script that 404s or gets served as text/html, you get alerted.

Full redirect chain resolution — Follows redirects all the way to the final destination. Catches loops, unexpected landings, and chains that time out.

Content fingerprinting — Generates hashes of your page content and tracks them over time. If your homepage suddenly returns completely different content (stale cache, CDN error page that returns 200), the fingerprint shift triggers an alert.

Multi-region checks — CDN failures are often edge-specific. Your site works from Europe but serves broken assets in Asia. Sitewatch checks from multiple locations.

Confirmation before alerting — Uses a retry model before firing alerts. No 3am wake-ups because of a transient CDN blip.

It also classifies the root cause — infrastructure, deploy artifact, DNS, CDN config — so you know where to look instead of just knowing “something broke.”

Some Technical Decisions

A few choices that might be relevant if you’ve thought about building something similar:

No headless browser. It’s tempting because it’s “real” rendering, but it doesn’t scale well. Sitewatch parses HTML and makes targeted follow-up requests for critical assets. Gets you the meaningful signal without the overhead.

Stack detection. Different frameworks break differently. A Nuxt app has different failure patterns than a WordPress site or a Rails app behind Nginx. Sitewatch detects the tech stack from response headers and adjusts what it checks for.

Alert deduplication. If the same issue persists across checks, you don’t want repeated alerts. Content fingerprints with a cooldown window — you get notified when something breaks and again when it’s resolved, not every 5 minutes in between.

Try It

If you’re running sites on modern deployment platforms — especially if you’re managing multiple — this is a blind spot worth covering.

There’s a free tier for one site. Takes about 30 seconds to set up: getsitewatch.com

That about wraps it up! If you found this useful, I’d love to know — leave a comment or connect with me on Twitter | LinkedIn or visit my website.

Integrate Custom Tools with OpenAI Function Calling

Large Language Models (LLMs) are incredibly powerful for generating text, summarizing information, and answering questions. However, their knowledge is typically limited to their training data. To perform real-world actions like checking the weather, sending emails, or querying a database, LLMs need to interact with external tools. This is where “function calling” or “tool use” becomes essential. It’s the mechanism that allows an LLM to not just generate text, but to intelligently decide when and how to invoke external functions, transforming it from a text generator into a capable agent.

In this guide, we’ll dive into OpenAI’s Function Calling feature, a robust way to connect your LLM applications with custom tools and APIs. We’ll explore the core concepts, walk through practical Python code examples, and discuss common pitfalls to help you build more dynamic and powerful AI-driven applications.

Understanding OpenAI Function Calling

OpenAI Function Calling empowers models like gpt-4o to detect when a user’s prompt might require an external function call, generate the necessary arguments for that call, and present this structured information back to your application. Your application then executes the function and feeds the result back to the LLM, allowing it to complete the user’s request with up-to-date, real-world data or perform specific actions.

This isn’t the LLM executing code directly. Instead, it acts as a smart orchestrator:

  1. You describe your tools: You provide the LLM with a JSON Schema definition of functions it can “call” (their names, descriptions, and expected parameters).
  2. User prompt: A user asks a question or makes a request.
  3. LLM decides: The LLM analyzes the prompt, compares it against the available tool descriptions, and decides if a tool is needed.
  4. LLM generates tool call: If a tool is needed, the LLM generates a structured JSON object containing the tool’s name and the arguments it inferred from the user’s prompt.
  5. Your application executes: Your code receives this JSON, calls your actual Python function (or external API) with the provided arguments.
  6. Tool output is returned: The result of your function call is then sent back to the LLM.
  7. LLM synthesizes response: The LLM uses this real-world data to formulate a natural language response to the user.

Let’s illustrate this with a common example: fetching current weather information.

Step 1: Describing Your Tools to the LLM

The first crucial step is to define the capabilities of your external tools in a way the LLM can understand. OpenAI uses a JSON Schema-like structure for this. Each tool needs a descriptive name, a clear description (this is key for the LLM’s understanding), and a parameters object detailing the inputs it expects.

Consider a simple get_current_weather function that takes a location and an optional unit.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

This Python list defines a single tool. The type is specified as “function”, indicating it’s a callable operation. Within the function object, we provide a name (get_current_weather) that your actual code will use, and a description that the LLM reads to understand the tool’s purpose. The parameters field, written in JSON Schema format, details the expected inputs: location as a required string and unit as an optional string that must be either “celsius” or “fahrenheit”.

Step 2: Triggering a Function Call from the LLM

Now that our tool is defined, we can pass it to the OpenAI API when making a chat completion request. The tool_choice parameter gives you control over whether the model should use a tool, always use a specific tool, or never use one. Setting it to "auto" (the default) lets the LLM decide.

from openai import OpenAI
import json # Used for parsing arguments later

client = OpenAI()

messages = [{"role": "user", "content": "What's the weather like in Boston?"}]

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools, # Our defined tools
    tool_choice="auto", # Let the model decide if a tool is needed
)
response_message = response.choices[0].message

# Check if the model decided to call a tool
if response_message.tool_calls:
    tool_call = response_message.tool_calls[0] # Assuming one tool call for simplicity
    print(f"LLM wants to call a tool: {tool_call.function.name}")
    print(f"Arguments: {tool_call.function.arguments}")
else:
    print("LLM responded directly:", response_message.content)

In this snippet, we initialize the OpenAI client and prepare a list of messages containing the user’s query. The core of this step is the call to client.chat.completions.create. We pass our messages, specify a model (gpt-4o is good for function calling), and crucially, include our tools definition. tool_choice="auto" instructs the model to automatically determine if a function call is appropriate. The response_message then holds the model’s decision; if response_message.tool_calls is populated, it means the LLM has generated a call to one of our defined tools, providing the function name and its parsed arguments as a JSON string.

Step 3: Executing the Tool and Sending Results Back

Receiving a tool_calls object means your application now needs to execute the identified function. This involves parsing the arguments provided by the LLM, calling your actual local function, and then sending the result back to the LLM to complete the conversation.

# Mock tool function (in a real application, this would query a weather API, database, etc.)
def get_current_weather(location, unit="fahrenheit"):
    # Simplified logic for demonstration
    if "boston" in location.lower():
        return json.dumps({"location": location, "temperature": "65", "unit": unit, "forecast": "partly cloudy"})
    elif "new york" in location.lower():
        return json.dumps({"location": location, "temperature": "72", "unit": unit, "forecast": "sunny"})
    return json.dumps({"location": location, "temperature": "unknown", "unit": unit, "forecast": "unavailable"})

# --- Continuation from Step 2, assuming response_message.tool_calls exists ---
if response_message.tool_calls:
    tool_call = response_message.tool_calls[0]
    function_name = tool_call.function.name
    function_args = json.loads(tool_call.function.arguments) # Parse the JSON string into a Python dict

    # Execute the function based on its name
    if function_name == "get_current_weather":
        function_response = get_current_weather(**function_args) # Unpack arguments with **

        # Append both the LLM's tool call and the tool's output to the messages history
        messages.append(response_message) # The model's request to call the tool
        messages.append(
            {
                "tool_call_id": tool_call.id,
                "role": "tool",
                "name": function_name,
                "content": function_response, # The actual result from our function
            }
        )

        # Send the updated messages back to the LLM to get a final, human-readable response
        second_response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
        )
        print("Final LLM response:", second_response.choices[0].message.content)
    else:
        print(f"Error: Unknown function requested: {function_name}")
else:
    # This case would be handled by the direct response logic in Step 2
    pass

This expanded block handles the actual execution. We define a placeholder get_current_weather function that simulates an API call. After receiving the LLM’s tool_call, we parse its arguments (which are a JSON string) into a Python dictionary using json.loads(). We then dynamically call our get_current_weather function, unpacking the arguments using **function_args. Crucially, we append two new messages to our messages history: first, the response_message from the LLM (which contained the tool_calls), and second, a new message with role="tool". This tool message includes the tool_call_id (linking it back to the original call), the function name, and the content which is the actual output from our get_current_weather function. Finally, we make a second API call to client.chat.completions.create with this updated message history, allowing the LLM to synthesize a natural language response based on the tool’s output.

Common Mistakes and Gotchas

Working with Function Calling, while powerful, can introduce specific challenges:

  • Incorrect Schema Definition: The description of your tool and its parameters are vital. If they are vague or misleading, the LLM might struggle to correctly identify when to call your tool or how to map user input to its arguments. Ensure required parameters are correctly marked.
  • Forgetting to Send Tool Output: A common oversight is to call the tool but not feed its results back into the LLM’s messages history. Without the role="tool" message, the LLM won’t know the outcome of its requested action and cannot complete the user’s request contextually.
  • Infinite Loops: In more complex scenarios, an LLM might continuously request the same tool if the tool’s output doesn’t provide enough information or resolve the initial query. Ensure your tool outputs are comprehensive, and consider adding retry limits or human intervention points.
  • Security Implications: Since your application executes the functions, be extremely cautious about what functions you expose and how you handle their arguments. Malicious prompts could potentially lead to unexpected function calls or data exposure if not properly validated. Always sanitize and validate inputs before executing actions.
  • Over-reliance on tool_choice="auto": While convenient, auto might not always pick the desired tool or might pick one when a direct response is better. For critical flows, consider explicit tool_choice values, e.g., {"type": "function", "function": {"name": "my_specific_tool"}}, or none to force a direct textual response.

Key Takeaways

OpenAI Function Calling fundamentally shifts how we interact with LLMs, moving beyond simple text generation to building truly intelligent, agentic systems. By providing a clear contract for your external tools through JSON Schema, you enable LLMs to:

  • Perform real-world actions: Interact with APIs, databases, or local services.
  • Access up-to-date information: Overcome knowledge cutoffs by fetching current data.
  • Improve user experience: Provide more accurate and actionable responses.

This pattern is a cornerstone for building sophisticated AI assistants, task automation agents, and advanced chatbots that can understand user intent and execute complex workflows.

Conclusion: Build Smarter, Not Harder

Integrating custom tools with LLMs via function calling unlocks a new dimension of possibilities for developers. It’s a critical pattern for extending the capabilities of models beyond their inherent knowledge, turning them into versatile problem-solvers. The ability to dynamically connect LLMs to your existing backend services means you can create applications that are not just conversational, but genuinely functional and integrated.

Start experimenting with OpenAI Function Calling in your projects today. Define a simple tool, feed it to the model, and observe how your LLM begins to intelligently orchestrate actions. The future of AI applications lies in these powerful integrations.