Stop Wasting API Calls: Detect What’s Blocking Your Agent Before You Solve

Most AI agents handle web blocks the same way: hit a wall, throw a CAPTCHA solver at it, hope for the best.

The problem? Half the time it’s not a CAPTCHA.

I’ve been building GateSolve, a CAPTCHA solving API for AI agents. And the most common failure pattern I see is agents submitting solve requests for pages that aren’t actually blocked by a CAPTCHA.

They’re blocked by:

  • JS challenges (Cloudflare interstitials that auto-resolve in 5-10 seconds)
  • Auth walls (login pages that need credentials, not CAPTCHA tokens)
  • Silent blocks (403 responses with no CAPTCHA at all)
  • Rate limits (429 responses that just need a backoff)

Solving a CAPTCHA on an auth-walled page does nothing. You burn credits and get nowhere.

The Fix: Classify Before You Solve

I built a free endpoint that classifies any URL’s access block:

curl "https://gatesolve.dev/api/detect?url=https://example.com"

Response:

{
  "url": "https://example.com",
  "classification": "public-ok",
  "httpStatus": 200,
  "solvable": false,
  "confidence": 0.9,
  "details": "Page accessible, no blocks detected",
  "recommendation": "No action needed. URL is publicly accessible."
}

The classifications:

Classification What it means Solvable?
public-ok Page is accessible No action needed
captcha CAPTCHA widget detected Yes – use GateSolve
js-challenge Cloudflare interstitial Wait or use stealth browser
auth-wall Login/credentials required No – authenticate instead
blocked-silent 403 with no pattern No – change IP/UA
rate-limited 429 Too Many Requests No – back off

Real Example: CAPTCHA Detected

When a real CAPTCHA is found:

{
  "url": "https://some-protected-site.com",
  "classification": "captcha",
  "captchaType": "cloudflare-turnstile",
  "httpStatus": 403,
  "solvable": true,
  "confidence": 0.9,
  "recommendation": "Submit to GateSolve /api/solve with type="cloudflare-turnstile". Extract siteKey from the page source."
}

Now you know what you’re dealing with before spending a solve credit.

For Browser Automation: Client-Side Detection

If you’re using Puppeteer or Playwright, our plugins now include detectBlock() — a DOM-level check that classifies the page after goto():

import { detectBlock, solveOnPage } from '@gatesolve/puppeteer-plugin';

const page = await browser.newPage();
await page.goto('https://target-site.com');

const block = await detectBlock(page);

if (block.classification === 'captcha' && block.solvable) {
  // Actually a CAPTCHA - solve it
  await solveOnPage(page, { apiKey: 'gs_...' });
} else if (block.classification === 'js-challenge') {
  // Just wait for Cloudflare to resolve
  await page.waitForTimeout(10000);
} else if (block.classification === 'auth-wall') {
  // Need credentials, not a CAPTCHA solve
  console.log('Login required');
}

This prevents the most common automation failure: your agent assumes the page loaded, tries to find elements on a Cloudflare interstitial, fails, retries, and spirals into a loop.

Install

# Puppeteer
npm install @gatesolve/puppeteer-plugin

# Playwright
npm install @gatesolve/playwright-plugin

# Python SDK
pip install gatesolve

The /api/detect endpoint requires no API key and no signup. Just hit it with a URL.

Links:

Building agent infrastructure is weird. The biggest wins come from helping agents NOT do things — like not wasting a CAPTCHA solve on a login page.

18,883 MCP servers. Five Chinese tech giants joined this week. Zero security audits.

MCP Supply Chain Security

MCP Supply Chain Security

MCP Supply Chain Risk Diagram

On March 24, someone put malware in litellm, a popular Python library for calling LLM APIs. Versions 1.82.7 and 1.82.8 on PyPI stole API keys for OpenAI, Anthropic, and Gemini. Two Hacker News posts about it got 1,159 points total and 364 comments.

The same week, five Chinese tech companies appeared on the MCP.so trending page. Tencent, Zhipu AI, Amap (owned by Alibaba), Baidu, and MiniMax. They all published MCP servers between March 23 and 25. The total number of MCP servers has now reached 18,883.

Everyone noticed litellm. Nobody noticed the Chinese MCP servers. No security review. No public discussion. Nothing.

npm vs MCP structural comparison

MCP has the same supply chain problem as npm, but worse

npm’s supply chain attacks are well known. event-stream in 2018. ua-parser-js in 2021. MCP is heading down the same path, but with two differences that make it worse.

The first is data direction. When you install an npm package, code runs on your machine. It stays local. When you connect an MCP server, your data leaves your machine. A map server gets your location queries. A search server gets your search queries. A deploy server gets your HTML source code. npm packages do not send data out by default. MCP servers do. That is what they are built for.

The second is geopolitics. npm was mostly neutral. No government had a special interest in controlling JavaScript module registries. MCP is different. Five Chinese tech companies now provide servers that handle location data, search queries, and deployment workflows. Data that flows through these servers may fall under China’s Cybersecurity Law (2017), which requires data to stay in China, and China’s National Intelligence Law (2017), which requires organizations to cooperate with intelligence work. npm never had this kind of issue.

npm vs MCP timeline comparison

Three events happened in one week. Nobody connected them.

March 19. Security firm Qualys published a blog post: “MCP Servers: The Shadow IT of the AI Agent Era.” They warned that developers install MCP servers without approval from IT departments. The same pattern as Shadow IT in the 2010s, but with AI tools.

March 24. litellm versions 1.82.7 and 1.82.8 on PyPI were found to carry malware. A file called litellm_init.pth stole LLM API credentials. litellm is widely used in production because it lets you call OpenAI, Anthropic, and Gemini through one API. Hacker News gave the story 1,159 points.

March 23-25. Five Chinese tech companies showed up on MCP.so‘s trending page. Tencent EdgeOne deploys HTML and returns a public URL. Zhipu AI offers web search with intent recognition. Amap and Baidu provide map and location services. MiniMax provides text-to-speech, image generation, and video generation through MCP.

Nobody has connected these three events. litellm was reported. Qualys published their warning. But no one has asked: who is auditing the five Chinese MCP servers that appeared the same week?

This is not about China being dangerous

This article does not say Chinese MCP servers are dangerous. There is no evidence that any of them contain malware. Some publish their source code on GitHub.

The problem is structural. 18,883 MCP servers exist, and there is no standard audit process. After the event-stream attack in 2018, it took npm years to add package signing and scoped registries. MCP will need server verification and data flow monitoring. The difference is that MCP sends live data, not just code. The attack surface is wider.

Our previous article, “MCP tool spoofing: 100% success rate,” covered protocol-level risks. This is a different kind of problem. Not how MCP is designed, but who publishes servers, where data goes, and who checks.

Ten years of npm lessons. One year to learn them.

MCP is becoming the npm of AI. Same supply chain problem, but higher stakes. Live data instead of static code. National interests instead of neutral registries. npm needed ten years to learn its lessons. MCP needs to learn them in one. The litellm hack was the first warning sign. The next one might come from an MCP server itself.

回测的真相与陷阱:为什么「测了十年」不等于「靠谱」

目录

  1. 回测是什么,能解决什么问题
  2. 外行最容易误读的一句话:「我回测了十年」
  3. 时间跨度 vs 数据粒度:同一根 K 线里藏着多少故事
  4. 从日 K 到 Tick:精度越高,故事越接近真实
  5. 撮合引擎与交易所现实:你的回测里成交得太「乖」了
  6. 插针、流动性与极端行情:图表上看不见的伤口
  7. 为什么回测曲线往往比实盘好看
  8. 其它常见坑:从幸存者偏差到过拟合
  9. 可以怎么做:让回测更「诚实」
  10. 小结

回测是什么,能解决什么问题

回测(Backtesting) 指在历史数据上,用固定规则「假装」在过去交易,统计收益、回撤、胜率等指标。它的价值是:在不动真金白银的前提下,快速验证策略逻辑是否自洽、参数是否过于极端、对哪些市场环境敏感。

它不能保证未来表现,只能回答:「若在过去这段数据、在某种成交与费用假设下,这套规则会怎样表现。」

外行最容易误读的一句话:「我回测了十年」

十年听起来很长,心理上容易等价于「样本够大、结论可靠」。但在量化里,「十年」只描述了时间轴的长度,没有描述采样密度与成交假设。

同一段十年:

  • 按日 K 回测,大约只有 3,650 根左右 K 线(视交易日略少);
  • 按1 分钟 K 回测,同一段历史的 K 线数量大约是日 K 的数百倍;
  • 若按 Tick(逐笔成交) 或订单簿事件回放,信息量又比分钟 K 再高几个数量级。

信息量不同,对同一策略的结论可以完全不同。 外行听「十年」很踏实;内行会先问:十年是哪种颗粒度上的十年?

时间跨度 vs 数据粒度:同一根 K 线里藏着多少故事

OHLC 丢掉了什么

常见的 K 线只记录 开高低收(OHLC) 与成交量。你不知道这一段时间里价格走过的路径:是先跌后涨,还是先涨后跌?中间有没有快速扫过你的止损再回来?

对很多策略(尤其是带止损、带网格、带条件单的策略),路径依赖极强:同样的 OHLC,不同路径会导致完全不同的成交与盈亏。

粗粒度回测在做什么隐含假设

用日 K 回测时,往往隐含了类似假设:「我能在开盘价/收盘价/某种理想价位成交」「止损一定按我想的价格触发」等。这些假设在实盘中常常不成立——尤其在波动大、流动性分层明显的品种上。

从日 K 到 Tick:精度越高,故事越接近真实

下面用直觉对比(非绝对,但方向普遍成立):

粒度 典型用途 主要风险
日 K 粗筛趋势、长周期因子 路径未知,止损/网格等容易「过于乐观」
小时 / 15 分钟 中频策略初筛 仍可能低估盘中插针与滑点
1 分钟 较多零售级回测的上限 分钟 K 内仍可能有剧烈波动未体现
Tick / 订单簿事件 高频、窄价差、精细风控 数据量大、实现复杂,但更接近真实撮合顺序

对波动大、杠杆高、点差与滑点敏感的品种(典型如加密货币永续合约),若策略持仓时间短、挂单密、依赖精确成交顺序,只有日 K 或小时 K 的回测往往严重失真。此时业界会更认真看待 Tick 级或至少 L2 订单簿级 的回测与仿真——否则更像「在地图缩略图上讨论路况」。

注意:Tick 回测也不是「真理」本身:它还依赖历史 Tick 是否完整、是否含撤单与自成交噪声、以及你的撮合模型是否与真实交易所一致。

撮合引擎与交易所现实:你的回测里成交得太「乖」了

回测程序里常见的简化包括:

  • 立即成交:挂单一碰价就全成,不考虑排队与部分成交;
  • 固定滑点:全市场统一加几个基点,而真实滑点随波动与深度变化;
  • 无延迟:信号产生与下单在同一时刻,而实盘有网络、限频与撮合延迟;
  • 理想对手盘:永远假设有足够流动性吃掉你的单。

真实交易所的撮合是 价格—时间优先、队列、最小下单单位、Post-Only、Reduce-Only、强平与资金费率等规则共同作用的结果。撮合引擎不一样,同一策略的成交序列就会不一样——这不是小误差,而是系统性偏差。

插针、流动性与极端行情:图表上看不见的伤口

插针(Wick) 指价格短时间刺穿某价位又迅速收回。日 K 上可能只是一根长影线;在分钟或 Tick 上,可能是多次快速扫单。

若回测用粗 K 线:

  • 可能低估被针扫掉止损的次数;
  • 也可能在不当假设下高估某些「触碰即成交」的网格收益。

此外,还有 流动性枯竭、点差拉宽、无法成交 等情况——历史 K 线未必体现当时的盘口状态。你在回测里「成交了」,实盘可能只是挂在簿上排队。

为什么回测曲线往往比实盘好看

常见原因包括但不限于:

  1. 费用与资金费率:手续费、Maker/Taker 差异、永续合约资金费率未计入或低估。
  2. 滑点与冲击成本:尤其大单或薄深度品种。
  3. 成交假设过于理想:见上一节。
  4. 前视偏差(Look-ahead Bias):用「未来才知道的信息」做过去决策(例如用全日收盘价信号在开盘就成交)。
  5. 过拟合:参数在历史段上被调得太贴合噪音,换一段行情就失效。
  6. 幸存者偏差:只测「现在还活着」的币种或交易对,忽略已退市或归零样本。

因此:回测好是必要不充分条件;实盘是另一套系统(执行、风控、心理与运维)。

其它常见坑:从幸存者偏差到过拟合

  • 样本外测试:保留一段从未参与调参的历史做验证。
  • 走样本测试(Walk-forward):滚动窗口,看策略是否只在某一两段「碰巧」有效。
  • 换市场与换品种:在相关性不同的品种上复现,降低「只适配某一标的」的风险。
  • 极端日 stress:单独拉出重大事件日前后,看爆仓、断网、交易所宕机等操作风险(这些往往不在回测里)。

可以怎么做:让回测更「诚实」

  1. 明确声明粒度:日 / 小时 / 分钟 / Tick,以及是否含订单簿。
  2. 费用与费率模型:手续费档位、资金费率、是否吃单等,尽量贴近真实账户。
  3. 成交模型升级:从「收盘价成交」→「下一根开盘价」→「带滑点与延迟」→「事件驱动 + 简撮合」→「近 Tick 仿真」逐级加严。
  4. 纸交易 / 仿真盘:用实时行情但虚拟资金,专门检验执行层。
  5. 小资金实盘:在风险可控下验证最后一公里的差异。

目标不是追求「回测曲线难看」,而是追求可解释、可复现、与实盘偏差可预期。

小结

  • 「回测了十年」若只有日 K,与「十年 Tick 级仿真」不是同一句话;粒度决定信息量。
  • 波动大、杠杆高、挂单密的策略,要特别警惕粗 K 线 + 理想成交带来的乐观偏差。
  • 撮合与交易所规则不同,回测与实盘就会分叉;插针与流动性在 K 线摘要里常被低估。
  • 回测是推演工具,不是盈利承诺;严谨的做法是逐级加严假设,并用样本外与实盘小资金交叉验证。

相关文档

Day 2/100: The 4 Android Components — What Senior Engineer Get Wrong

This is Day 2 of my 100 Days to Senior Android Engineer series. Each post follows a consistent format: what I thought I knew → what I actually learned → interview implications.

🔍 The concept

Android has exactly four fundamental application components:

  • Activity — a single screen with a UI
  • Service — background work with no UI
  • BroadcastReceiver — responds to system-wide or app-wide events
  • ContentProvider — structured data sharing between apps

Every Android developer learns these in week one. What separates junior from senior isn’t knowing the list — it’s understanding the boundaries and the cost of each one.

💡 What I thought I knew

After years of building apps, my mental model was roughly:

  • Need a screen? → Activity (or Fragment)
  • Need background work? → Service
  • Need to react to a system event? → BroadcastReceiver
  • Need to share data with other apps? → ContentProvider

Simple enough. And for most day-to-day work, this is fine.

The problem is that this mental model is when to use them, not what they actually are — and the distinction matters enormously when things break.

😳 What I actually learned

Each component is a system entry point — not just a class

This is the insight that reframes everything.

When you declare a component in AndroidManifest.xml, you’re not just registering a class. You’re telling the Android system: “here is a door into my app.” The system can open that door independently of your app’s current state — even if no other part of your app is running.

This has real consequences:

User taps notification
    → System creates new Activity (entry point #1)

Device boots, BOOT_COMPLETED fires
    → System wakes your BroadcastReceiver (entry point #2)

Another app queries your ContentProvider
    → System starts your process just for the ContentProvider (entry point #3)

Your Application.onCreate() runs before all of these. Which means every entry point carries Application initialization cost — and if that initialization is slow or throws, it affects all of them.

The BroadcastReceiver trap most seniors fall into

// Registered in Manifest — looks innocent
class BootReceiver : BroadcastReceiver() {
    override fun onReceive(context: Context, intent: Intent) {
        // DON'T do this
        val db = Room.databaseBuilder(context, AppDatabase::class.java, "db").build()
        db.someDao().doHeavyWork() // runs on main thread, no coroutine scope
    }
}

onReceive() runs on the main thread and has a hard time limit of ~10 seconds before the system ANRs. There is no coroutine scope. There is no lifecycle. The receiver is considered dead the moment onReceive() returns.

The correct pattern for anything non-trivial:

class BootReceiver : BroadcastReceiver() {
    override fun onReceive(context: Context, intent: Intent) {
        // Hand off immediately — don't do work here
        SyncWorker.enqueue(context)
    }
}

Delegate to WorkManager and return. That’s it.

Service is not what you think in 2024

Service runs on the main thread by default. Not a background thread. Not a coroutine. The main thread.

class MyService : Service() {
    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        // This is still the main thread
        // Heavy work here = ANR
        return START_STICKY
    }
}

The distinction that matters now:

Type Visibility Use case
Service (background) None Legacy. Avoid on API 26+. System kills it aggressively.
ForegroundService Persistent notification Music playback, navigation, active uploads
WorkManager None Deferrable, guaranteed background work
CoroutineWorker None WorkManager + coroutines, the modern default

Since API 26 (Android 8.0), background services are killed almost immediately when your app goes to the background. If you’re still using plain Service for background processing, you’re relying on behavior that the system actively works against.

ContentProvider is the most misunderstood of the four

Most developers think: “I don’t share data with other apps, so I don’t need ContentProvider.”

But here’s what they’re missing — ContentProvider initializes before Application.onCreate().

Process starts
    → ContentProviders init (in manifest order)
    → Application.onCreate()
    → Activity/Service/Receiver entry point

Libraries like WorkManager, Firebase, and Jetpack Startup exploit this deliberately. They ship a ContentProvider with nothing inside — just an onCreate() — to auto-initialize without requiring you to add code to your Application class.

// From Jetpack App Startup — the entire trick
class InitializationProvider : ContentProvider() {
    override fun onCreate(): Boolean {
        // Runs before Application.onCreate()
        // Perfect for library auto-init
        AppInitializer.getInstance(context!!).discoverAndInitialize()
        return true
    }
    // query, insert, etc. — all throw UnsupportedOperationException
}

If you’ve ever wondered how a library “just works” without any setup code — this is usually how.

🧪 The mental model that stuck

Think of each component as having a lifecycle owner and a thread contract:

Component          | Lifecycle owner      | Default thread  | Survives process death?
-------------------|----------------------|-----------------|------------------------
Activity           | User (navigates)     | Main            | No (state via Bundle)
Fragment           | Activity / BackStack | Main            | No (state via Bundle)
Service            | You (stopSelf)       | Main (!)        | Maybe (START_STICKY)
ForegroundService  | You + notification   | Main (!)        | Yes, while notif shows
BroadcastReceiver  | Single onReceive()   | Main            | No — dies immediately
ContentProvider    | Process lifetime     | Binder thread   | Yes, while process lives
WorkManager        | System scheduler     | Background      | Yes, survives reboot

When something breaks — a crash, an ANR, unexpected behavior after process death — I now ask: which entry point triggered this, and what are its thread and lifecycle contracts?

That single question resolves about 60% of the confusion.

❓ The interview question

Here’s the version of this question that trips people up at the senior level:

“Your app needs to sync data when the device finishes charging. Walk me through the component choices and the tradeoffs of each approach.”

The naive answer: BroadcastReceiver for ACTION_BATTERY_CHANGED.

The senior answer considers:

  • Is this deferrable? (Yes → WorkManager with requiresCharging(true))
  • What if the sync takes > 10 seconds? (Rules out BroadcastReceiver body entirely)
  • What if the user force-stops the app? (WorkManager handles re-scheduling; a plain Receiver doesn’t)
  • What’s the battery and data impact? (Constraints API)
val syncRequest = PeriodicWorkRequestBuilder<SyncWorker>(1, TimeUnit.HOURS)
    .setConstraints(
        Constraints.Builder()
            .setRequiresCharging(true)
            .setRequiredNetworkType(NetworkType.UNMETERED)
            .build()
    )
    .build()

WorkManager.getInstance(context).enqueueUniquePeriodicWork(
    "periodic_sync",
    ExistingPeriodicWorkPolicy.KEEP,
    syncRequest
)

The question isn’t testing if you know WorkManager exists. It’s testing whether you reason about constraints, reliability, and system behavior — not just “which class do I instantiate.”

What surprised me revisiting this

I thought I knew these components cold. What I actually found:

  1. I’d internalized “Service for background work” without regularly questioning whether I needed a Service at all in 2024
  2. I’d never thought carefully about ContentProvider initialization order — and now I see it everywhere in library source code
  3. The “entry point” framing changed how I read crash reports. When I see a crash in a component I didn’t expect to be running, I now know to look for which entry point woke up the process

Tomorrow

Day 3 → Activity Lifecycle — the diagram you’ve seen a hundred times, and the three scenarios it doesn’t show you.

Found a gap in your own understanding? Drop it in the comments — I read every one.

← Day 1: Going Back to Basics as a Senior Android Dev

Onde e como usar o Schepta

Este artigo é para quem vai adotar o Schepta: planos futuros da arquitetura (outras factories além de forms), cenários de uso — multi-tenant, testes A/B, wizards, white-label — e por que a suite de testes garante que sua aplicação pode focar na parte de negócio. No final, um resumo prático de onde o projeto vive e da extensibilidade.

Planos futuros e arquitetura: outras factories, prioridade em forms

O core do Schepta (orquestrador, registry, middlewares, pipeline schema → render) é agnóstico do tipo de UI. O mesmo mecanismo que hoje alimenta o FormFactory pode alimentar outras factories — por exemplo MenuFactory (navegação dinâmica a partir de schema), dashboards, wizards. A arquitetura já prevê isso (tipos de componente como menu-item, menu-container na documentação de conceitos).

No momento foi priorizado forms porque grande parte da complexidade em aplicações multi-tenant está nos formulários: campos por tenant, validações por contexto, fluxos diferentes por cliente. Resolver bem “schema → form” desbloqueia o cenário mais doloroso; outras factories podem vir em seguida.

A arquitetura permite a criação de outras factories além de forms. A escolha de lançar com foco em forms é priorização (multi-tenant + complexidade), não limitação do desenho.

Outros usos além de multi-tenant

Multi-tenant já está no centro: um schema por tenant, UI consistente.

Testes A/B: o mesmo fluxo ou tela pode ser servido com schemas diferentes — por exemplo variação A com 3 campos, variação B com 5 campos ou ordem diferente. O backend ou a camada de feature flags entrega o schema da variação; o Schepta renderiza. Sem trocar código do app, só o JSON.

Wizards e fluxos dinâmicos: próximos passos ou telas dependentes de contexto (respostas do usuário, perfil, permissões) podem ser modelados como schemas diferentes por etapa; o Schepta renderiza cada etapa.

White-label / configuração por cliente: além de multi-tenant “duro”, cenários em que cada cliente tem sua própria definição de formulários (CRM, ferramentas internas); o schema vem de configuração ou API e o Schepta mantém a UI consistente.

Prototipagem e ferramentas low-code: ferramentas que geram ou editam schemas (por humano ou por IA) e usam o Schepta como runtime de preview ou produção.

Uso com IA e produtos no chat: um produto que roda diretamente dentro do chat (por exemplo via MCP — Model Context Protocol) mantém a consistência entre os componentes: o cliente cria o próprio MCP e disponibiliza nas tools o schema de acordo com o usuário e com o que ele está pedindo; o Schepta renderiza o formulário ou a UI no próprio chat. Assim, assistentes como o Claude podem invocar as tools (obter schema, validar, abrir preview) e a interface exibida é sempre a mesma engine Schepta, previsível e testável.

Schema-driven UI serve para vários problemas, não só “um form por tenant”.

Cenário Como o Schepta ajuda
Multi-tenant Um schema por tenant; mesma engine, UI consistente.
Testes A/B Schemas diferentes por variação; sem alterar código.
Wizards Um schema por etapa; renderização por contexto.
White-label Schema por cliente via config ou API.
Low-code / IA Runtime estável para schemas gerados ou editados.
IA / produto no chat Cliente expõe schema nas tools do MCP; Schepta renderiza no chat com a mesma engine.

Testes como garantia: o app foca no negócio

O Schepta é coberto por 103 testes unitários e 26 testes E2E (Playwright) que validam o fluxo completo: schema → resolução → render → interação em React, Vue e Vanilla, incluindo integrações com react-hook-form e Formik. Isso garante que a engine se comporta corretamente — resolução de componentes, middlewares, validação, binding de campos.

Ao adotar o Schepta, o time do app não precisa se preocupar com “será que o schema vira componente certo?”, “será que o middleware roda na ordem?”, “será que o form adapter conecta?”. Esses detalhes estão cobertos pela suite; regressões da engine são detectadas antes de chegar ao app.

Nos aplicativos que usam o Schepta, o time pode concentrar esforço na parte voltada para negócio: validações específicas do domínio, regras de negócio em middlewares, definições de forms por contexto ou tenant, integração com APIs e fluxos. A “infraestrutura” de schema→UI está testada e estável.

Os testes são a garantia de que as funcionalidades da engine funcionam, abstraindo esses detalhes para que, nos apps onde o Schepta será usado, as preocupações sejam apenas a parte voltada para negócio (validações específicas, definições específicas de forms, etc.).

Em resumo

  • Onde vive: monorepo com @schepta/core, adapters (react, vue, vanilla), factories por framework; documentação e exemplos em schepta.org.
  • Testes: 103 testes unitários e 26 testes E2E (React, Vue, Vanilla), incluindo react-hook-form e Formik — garantia da engine; o app foca em lógica de negócio.
  • Extensibilidade: middlewares para transformar props; registry para trocar componentes; renderers por tipo (field, container, button, content); expressões em props com $formValues e $externalContext.

Showcases por framework (React, Vue, Vanilla, MUI, Chakra, Vuetify, etc.) estão disponíveis na documentação.

Próximo passo

No último artigo da série entramos em como o Schepta funciona por dentro: factories, orquestradores, component registry e o pipeline de resolução até o render. Para quem quer profundidade técnica.

Enquanto isso: schepta.org para docs e exemplos.

We Built an Open-Source CRM for AI Agents — Here’s Why

We Built an Open-Source CRM for AI Agents — Here’s Why

Traditional CRMs were designed for humans clicking buttons. With 6,400+ MCP servers and AI agents entering every enterprise workflow, it’s time for infrastructure that’s built for them.

If you’re building AI agents that interact with business data, you’ve probably hit the same wall we did.

Your agent can call APIs, generate text, and reason over documents — but where does it store what it knows? Where does it persist contacts, deals, interactions, and the evidence behind every claim? How does it handle conflicting information without silently overwriting the truth?

Most teams wire up a vector database and call it memory. That works for semantic search. It doesn’t work when you need to know the exact deal value, who the decision maker is, and which email confirmed it — with a full audit trail.

We needed something better. So we built it and open-sourced it.

Open Tooling CRM: The Technical Case

Open Tooling CRM is a headless, local-first CRM designed for AI agents as the primary users. No UI. No cloud dependencies. Just a SQLite-backed data layer with two clean interfaces:

  • REST API — 29 endpoints. Standard HTTP/JSON with consistent pagination, filtering, idempotency keys, and Zod validation.

  • MCP Server — 27 tools purpose-built for Claude Desktop, Claude Code, and any MCP-compatible client. Entities, relationships, artifacts, observations, briefs, conflicts, graph traversal, FTS, import/export.

The Evidence Chain

The core differentiator is the memory layer. Instead of flat fields that get overwritten, Open Tooling CRM implements a four-level evidence chain:

Artifacts (raw evidence: emails, transcripts, documents)

↓ extraction

Observations (typed claims with lifecycle: current → superseded / retracted)

↓ derivation

Briefs (summaries that cite observations — always regeneratable)

↓ detection

Conflicts (when observations disagree — never silently resolved)

Every observation links to its source artifact. Every brief cites its observations. Every conflict references the disagreeing claims. Progressive retrieval means agents start with the brief and drill down to raw evidence when needed.

This isn’t academic — it’s practical. When your agent says “budget is $250K,” you can trace that through the observation to the artifact (the email from the prospect’s CFO on March 3rd). When the budget changes, the old observation is superseded, not deleted. The audit trail is preserved.

Architecture

Agent Client → MCP Tools → OpenCRM Core → SQLite

Agent Client → REST API → OpenCRM Core → SQLite

Both interfaces share the same core. All writes produce immutable events in an append-only ledger. Data model: typed entities with JSON properties, directed relationships, field-level provenance, FTS index, and the full memory layer.

MCP Tools at a Glance

  • Entities: create_entity, update_entity, get_entity, search_entities, archive_entity

  • Relationships: link_entities, unlink_entities, list_relationships, traverse_graph

  • Memory: ingest_artifact, add_observation, list_observations, supersede_observation, retract_observation

  • Briefs: create_brief, get_brief, list_briefs

  • Conflicts: create_conflict, get_conflict, list_conflicts, resolve_conflict

  • Data: export_data, import_data

Why Headless?

No bundled UI is a deliberate architectural choice. CRMs that couple their data layer to a rigid frontend create lock-in by design. Open Tooling inverts this: the API is the product.

Three ways to use it:

  • Agent-only. Connect the MCP server to Claude Desktop. Your CRM interface is natural language.

  • Vibe-code a frontend. Tell Claude or Cursor: “Build me a React dashboard for my deals pipeline using the Open Tooling CRM API at localhost:8787.” The API is conventional enough that any code-gen tool can scaffold a working UI in minutes.

  • Integrate into your stack. It’s a REST API — connect it to anything.

Part of a Bigger Vision

Open Tooling CRM is the first module in Open Tooling — an open-source family of AI-native SaaS tools. The same architecture (headless, local-first, evidence-based, MCP-native) applies across verticals like HR, Finance, Project Management, Procurement, and beyond.

We also ship a Claude plugin marketplace (Attri-Inc/open-tooling-plugins). Install it and Claude gets the knowledge layer to use Open Tooling CRM optimally — the right ingestion patterns, the evidence chain, progressive retrieval, conflict resolution.

Quick Start

git clone https://github.com/Attri-Inc/open-tooling.git

cd open-tooling/crm

npm install && cp .env.example .env

npm run dev # REST API at localhost:8787

npm run seed # Sample data

npm run mcp # MCP server (stdio)

Use It with Claude (Zero Config)

MCP Server:

Add the MCP config to Claude Desktop or Claude Code. 27 tools, full evidence chain, natural language CRM.

Claude Cowork Plugin:

One-click install via our plugin marketplace. Gives Claude the knowledge layer for optimal usage — ingestion patterns, evidence chain workflow, progressive retrieval, conflict resolution.

Cowork setup: Customize → Add plugin → Add marketplace → Attri-Inc/open-tooling-plugins → Sync.

[GIF: Claude Cowork plugin install — showing the marketplace setup flow]

⭐ GITHUB: github.com/Attri-Inc/open-tooling

🧩 PLUGIN MARKETPLACE: github.com/Attri-Inc/open-tooling-plugins

Apache 2.0 licensed. Contributions welcome.

Originally published on Attri AI Substack [link].

Built by Attri AI. We build AI-native tools for enterprises.

Lucid Software advances MCP Server offering and Lucid AI capabilities, debuts Process Agent to boost speed and clarity across teams

SOUTH JORDAN, Utah  – Lucid Software, the leader in AI-driven work acceleration, today announced new AI-enhanced collaboration features that speed up how teams create complex diagrams and documentation. With advancements to its Model Context Protocol (MCP) server, enhancements to Lucid AI, and the introduction of its Process Agent, Lucid is helping organizations scale their AI … continue reading

The post Lucid Software advances MCP Server offering and Lucid AI capabilities, debuts Process Agent to boost speed and clarity across teams appeared first on SD Times.

Zero-Downtime Deployments in Node.js: Blue-Green, Rolling, and Canary Explained

Zero-Downtime Deployments in Node.js: Blue-Green, Rolling, and Canary Explained

Every production deployment is a bet. You’re betting that your changes are correct, your infrastructure will cooperate, and your users won’t notice anything happened. Most of the time, the bet pays off. But when it doesn’t — when a bad deploy takes down your Node.js service at 2 PM on a Tuesday — the cost is real: lost revenue, damaged trust, and a team scrambling to roll back.

Zero-downtime deployment removes the bet. Instead of replacing your running service with the new version and hoping for the best, you introduce the new version alongside the old one, validate it, and shift traffic gradually. If something breaks, you catch it early and reverse — before your users do.

This guide covers three production-ready strategies: blue-green, rolling, and canary deployments. We’ll look at when to use each, how to implement them for Node.js specifically, and what you need to have in place for any of them to work.

Why Node.js Deployments Fail (And When)

Before the strategies, let’s be clear about the actual failure modes:

Process crashes on startup. Your new version has a syntax error, a missing environment variable, or a database migration conflict. The old process is gone, the new one won’t start, your load balancer is routing to nothing.

Graceful shutdown not implemented. Your deployment kills the old process mid-request. Active HTTP connections get dropped. In-flight jobs are lost.

Health checks that lie. Your new version responds 200 OK on /health but has a broken connection to its database. Traffic is routed to it. Errors spike.

Memory leaks in new code. The new version looks fine for 10 minutes, then memory climbs. By the time you notice, every instance is affected.

Zero-downtime strategies solve the first two directly. The other two require proper health checks and monitoring — which you need regardless of deployment strategy.

Prerequisites: What Every Strategy Requires

You cannot do zero-downtime deployments without these:

1. A working health check endpoint

// app.js
app.get('/health', async (req, res) => {
  const checks = {
    status: 'ok',
    uptime: process.uptime(),
    timestamp: new Date().toISOString(),
    db: await checkDatabase(),
    memory: process.memoryUsage().heapUsed / 1024 / 1024
  };

  const isHealthy = checks.db === 'connected' && checks.memory < 400; // MB
  res.status(isHealthy ? 200 : 503).json(checks);
});

async function checkDatabase() {
  try {
    await pool.query('SELECT 1');
    return 'connected';
  } catch {
    return 'disconnected';
  }
}

2. Graceful shutdown

// Handle SIGTERM (sent by process managers and orchestrators)
process.on('SIGTERM', async () => {
  console.log('SIGTERM received — starting graceful shutdown');

  // Stop accepting new connections
  server.close(async () => {
    // Wait for in-flight requests to complete
    await drainConnections();

    // Close database connections
    await pool.end();

    console.log('Graceful shutdown complete');
    process.exit(0);
  });

  // Force kill after 30 seconds if something hangs
  setTimeout(() => {
    console.error('Graceful shutdown timed out — forcing exit');
    process.exit(1);
  }, 30000);
});

3. A load balancer or reverse proxy

NGINX, HAProxy, AWS ALB, Cloudflare, or your PaaS provider’s built-in routing — you need something sitting in front of your Node.js instances that can be told “route traffic here, not there.”

With these in place, let’s look at the strategies.

Strategy 1: Blue-Green Deployment

The concept: Maintain two identical production environments — “blue” (current live) and “green” (new version). Deploy to green, validate it fully, then switch all traffic from blue to green in a single step. Blue becomes your instant rollback.

Before deploy:    [Users] → [Load Balancer] → [Blue: v1.2.3]    [Green: idle]
Deploy to green:  [Users] → [Load Balancer] → [Blue: v1.2.3]    [Green: v1.3.0 ← deploying]
After validation: [Users] → [Load Balancer] → [Blue: v1.2.3]    [Green: v1.3.0 ✓]
Switch traffic:   [Users] → [Load Balancer] → [Blue: v1.2.3]    [Green: v1.3.0 → LIVE]

Implementing blue-green with PM2 + NGINX:

# ecosystem.config.js defines both environments
// ecosystem.config.js
module.exports = {
  apps: [
    {
      name: 'app-blue',
      script: './dist/server.js',
      instances: 2,
      env: { PORT: 3000, NODE_ENV: 'production' }
    },
    {
      name: 'app-green',
      script: './dist/server.js',
      instances: 2,
      env: { PORT: 3001, NODE_ENV: 'production' }
    }
  ]
};
# nginx.conf
upstream active_app {
  server localhost:3000;  # Blue: change to 3001 to switch to Green
}

server {
  listen 80;
  location / {
    proxy_pass http://active_app;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
  }

  location /health {
    proxy_pass http://active_app/health;
    access_log off;
  }
}

Deployment script:

#!/bin/bash
# deploy-blue-green.sh

NEW_VERSION=$1
CURRENT=$(cat /var/app/active-slot)  # "blue" or "green"
NEW_SLOT=$([ "$CURRENT" = "blue" ] && echo "green" || echo "blue")
NEW_PORT=$([ "$NEW_SLOT" = "blue" ] && echo "3000" || echo "3001")

echo "Current: $CURRENT | Deploying to: $NEW_SLOT (port $NEW_PORT)"

# 1. Pull new code and build
git pull origin main
npm ci --production
npm run build

# 2. Start the new slot
pm2 start ecosystem.config.js --only "app-$NEW_SLOT"

# 3. Wait for it to be healthy
echo "Waiting for $NEW_SLOT to become healthy..."
for i in {1..30}; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:$NEW_PORT/health)
  if [ "$STATUS" = "200" ]; then
    echo "Health check passed"
    break
  fi
  echo "Attempt $i: status $STATUS — waiting..."
  sleep 2
  if [ $i -eq 30 ]; then
    echo "Health check failed after 60s — aborting"
    pm2 stop "app-$NEW_SLOT"
    exit 1
  fi
done

# 4. Switch NGINX traffic to new slot
sed -i "s/server localhost:[0-9]*/server localhost:$NEW_PORT/" /etc/nginx/conf.d/app.conf
nginx -t && nginx -s reload

# 5. Record the switch and stop old slot
echo "$NEW_SLOT" > /var/app/active-slot
sleep 5
pm2 stop "app-$CURRENT"

echo "Deploy complete. Active: $NEW_SLOT ($NEW_VERSION)"

When to use blue-green:

  • You need instant, complete rollback capability
  • Your migrations are additive (blue and green can share a database)
  • You can afford to run double infrastructure (even briefly)
  • You’re on a PaaS that supports slot-based deployment (Railway, Render, Azure App Service)

When not to: If your database schema changes are destructive (column drops, renames), blue-green is tricky — old blue can’t run against green’s schema. Use expand-contract migrations.

Strategy 2: Rolling Deployment

The concept: Replace instances of the old version one by one (or in small batches), while keeping the remaining old instances live. At any point during the rollout, both versions are serving traffic simultaneously.

Start:   [v1][v1][v1][v1]
Step 1:  [v2][v1][v1][v1]  ← replace 1 instance
Step 2:  [v2][v2][v1][v1]  ← replace 1 more
Step 3:  [v2][v2][v2][v1]  ← replace 1 more
Done:    [v2][v2][v2][v2]  ← all updated

Implementing rolling updates with Kubernetes:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-app
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1         # Add 1 new pod before removing old
      maxUnavailable: 0   # Never remove old until new is healthy
  template:
    spec:
      containers:
      - name: app
        image: my-app:1.3.0
        ports:
        - containerPort: 3000
        readinessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 5"]  # Drain in-flight requests
# Deploy with rollout monitoring
kubectl set image deployment/node-app app=my-app:1.3.0
kubectl rollout status deployment/node-app --timeout=5m

# Rollback if something goes wrong
kubectl rollout undo deployment/node-app

Implementing rolling updates with PM2 (no Kubernetes):

PM2’s built-in rolling restart handles this elegantly:

# Zero-downtime reload — restarts workers one at a time
pm2 reload ecosystem.config.js --update-env

# Or just the app name
pm2 reload app-name

What PM2 does: sends SIGINT to the first worker, waits for it to drain and exit, starts a new worker, waits for it to be ready, then moves to the next — while all other workers keep serving traffic.

When to use rolling:

  • Multiple instances of the same service (standard for any production service)
  • You’re already using Kubernetes, Docker Swarm, or PM2
  • Database changes are backward-compatible with the old version
  • You want fine-grained control over rollout speed

Key consideration: During a rolling deploy, v1 and v2 are simultaneously live. Your API changes must be backward-compatible. Adding a new optional field is fine. Removing a field that v1 uses is not.

Strategy 3: Canary Deployment

The concept: Route a small percentage of real traffic (5–10%) to the new version. Monitor error rates, latency, and business metrics. If healthy, gradually increase the percentage. If not, route all traffic back to the old version.

Canary phase:      [Users] → 95% → [v1 fleet]
                           → 5%  → [v2 canary]

Monitor for 30min: errors? latency? business metrics? ✓

Expand to 25%:     [Users] → 75% → [v1 fleet]
                           → 25% → [v2 canary]

Expand to 100%:    [Users] → 100% → [v2 fleet]

Implementing canary with NGINX weighted upstream:

upstream app_v1 {
  server app-v1:3000 weight=95;
}

upstream app_v2 {
  server app-v2:3000 weight=5;
}

# Custom upstream selection
split_clients "$request_id" $app_upstream {
  5%    app_v2;
  *     app_v1;
}

server {
  location / {
    proxy_pass http://$app_upstream;
  }
}

Implementing canary with feature flags (code-level canary):

Sometimes you don’t need infrastructure-level traffic splitting — you can canary at the application level:

// middleware/canary.js
const CANARY_FEATURES = {
  newCheckoutFlow: { percentage: 5, sticky: true },
  improvedSearch: { percentage: 10, sticky: false }
};

function isInCanary(userId, feature) {
  const config = CANARY_FEATURES[feature];
  if (!config) return false;

  // Sticky: same user always gets same experience
  if (config.sticky) {
    const hash = require('crypto')
      .createHash('md5')
      .update(`${userId}-${feature}`)
      .digest('hex');
    return parseInt(hash.slice(0, 8), 16) % 100 < config.percentage;
  }

  // Random: different on each request
  return Math.random() * 100 < config.percentage;
}

// Usage in route handler
app.post('/checkout', (req, res) => {
  if (isInCanary(req.user.id, 'newCheckoutFlow')) {
    return newCheckoutHandler(req, res);
  }
  return legacyCheckoutHandler(req, res);
});

Monitoring during canary (the only thing that makes it useful):

// Track canary vs. control metrics
const { Histogram, Counter } = require('prom-client');

const requestDuration = new Histogram({
  name: 'http_request_duration_ms',
  help: 'HTTP request duration in milliseconds',
  labelNames: ['method', 'route', 'status', 'version'],
  buckets: [5, 10, 25, 50, 100, 250, 500, 1000]
});

const errorCounter = new Counter({
  name: 'http_errors_total',
  help: 'Total HTTP errors',
  labelNames: ['route', 'error_type', 'version']
});

// Middleware
app.use((req, res, next) => {
  const version = process.env.APP_VERSION || 'unknown';
  const start = Date.now();

  res.on('finish', () => {
    const duration = Date.now() - start;
    requestDuration.labels(req.method, req.route?.path || 'unknown', res.statusCode, version)
      .observe(duration);

    if (res.statusCode >= 500) {
      errorCounter.labels(req.route?.path || 'unknown', 'server_error', version).inc();
    }
  });

  next();
});

Compare version="v2" metrics against version="v1" in your dashboards. If error rate or p99 latency climbs, roll back.

When to use canary:

  • High-risk changes (new payment flows, auth changes, database queries at scale)
  • You want real production validation without full exposure
  • You have observability in place to actually compare metrics
  • Your changes affect a subset of users (A/B test territory)

Choosing the Right Strategy

Blue-Green Rolling Canary
Rollback speed Instant Minutes Instant
Infrastructure cost 2x (brief) Normal + 1 instance Normal + small canary
Both versions live simultaneously No Yes Yes
Best for Major releases, DB migrations Standard weekly deploys High-risk or A/B changes
Requires Slot routing, 2x infra Orchestration (K8s, PM2) Traffic splitting + monitoring

For most Node.js services:

  • Daily/weekly deploys: Rolling with PM2 or Kubernetes
  • Major releases: Blue-green with database expand-contract
  • Risky feature launches: Canary with observability

The One Tool to Add Right Now

Before any of these strategies work, your application needs to correctly report its own health. Run npx node-deploy-check in your project root to audit your production readiness:

npx node-deploy-check

It checks for missing health endpoints, graceful shutdown handlers, environment variable validation, and 12+ other production essentials — in under 5 seconds.

Final Thought

Zero-downtime deployment isn’t a luxury — it’s the baseline for any service people depend on. The implementation complexity is real, but the alternatives (maintenance windows, crossed fingers, midnight rollbacks) are worse.

Pick one strategy that matches your infrastructure. Implement it fully, including health checks and graceful shutdown. Run it on a staging environment until it’s boring. Then your production deploys will be, too.

Follow the AXIOM experiment at axiom-experiment.hashnode.dev — an autonomous AI agent building a business from scratch, documented in real time.

This article is part of the **Node.js in Production* series. Check out the companion tool node-deploy-check on npm.*

Beyond the Terminal: My Evolution as a Fedora Contributor

From Newbie to Contributor: My Path in Fedora

Back in 2024, I started my journey into open source with curiosity, passion, and honestly… a lot of confusion.

I was especially interested in Linux systems i have always loved Linux and enjoyed exploring its tools, the terminal, and scripting ,that passion made me want to contribute to the Fedora Project. But at that time, I was a complete beginner. I didn’t know where to start, what to do, or even how to begin contributing.

That “Oh no” moment with nmstate

One of the first projects I came across was nmstate. I remember opening it, Everything looked complex, and when i opened the repo, I saw these massive YAML files defining MACVTAP interfaces and complex Ansible roles,I remember looking at a block of code like this:

- name: veth0
  type: macvlan
  state: up
  parent: eth0
  macvlan:
    mode: bridge

At first, I froze. I ended up feeling completely overwhelmed. I was used to simple scripts, but seeing low-level networking logic abstracted into declarative YAML was a different beast entirely. It felt like I was staring at a wall of text that I didn’t have the key to unlock. But instead of stepping back or closing the tab, I decided to learn, explore, and ask questions. I realized that if I wanted to move from being a user to a builder, I had to get comfortable with the fact that I didn’t know everything yet.

The slow climb

I joined the community for the first time, started reading discussions on Matrix and IRC, exploring documentation, and trying to understand how things work. It wasn’t easy. Sometimes it felt like I was not making progress at all,There were days when I’d spend hours reading a single documentation page only to feel more confused than when I started.
But step by step, things started to make more sense.

Small Wins

Even small achievements felt significant. Completing my Fedora Account System (FAS) profile was my first real step into the Fedora contributor world. Gradually, I learned more about NMState, explored its modules, and began thinking about ways I could contribute. Each small win reminded me that beginners can still make meaningful contributions.

Now here I am in 2026.

I’m still learning, still growing, and maybe still not an expert—but I’m no longer the same beginner I was before. I understand more, I feel more confident, and most importantly, I truly want to be part of the open source community.I’ve moved from just “loving Linux” to wanting to help shape its future.

Outreachy feels like a big opportunity for me. Not just to contribute, but to grow, to learn from others, and to finally become an active contributor to projects like Fedora.

This blog is part of that journey.

I want to document what I learn, share my progress, and show that even someone who started with no idea where to begin can move forward.I want to be honest about the struggles, because I know there is another beginner out there right now looking at a complex configuration and feeling just as lost as I did.

And this is just the beginning.

Bandcamp Artist Data Scraping: Music Research and Analytics

Bandcamp Artist Data Scraping: Music Research and Analytics

Bandcamp is the largest independent music platform, home to millions of artists selling directly to fans. For music researchers, label scouts, and analytics teams, Bandcamp data is incredibly valuable — but there is no public API.

This guide shows you how to extract artist data, pricing patterns, and genre trends from Bandcamp using Python.

What Data Can You Extract?

Bandcamp artist pages contain rich structured data:

  • Artist info: name, location, bio, genre tags
  • Discography: albums, EPs, singles with release dates
  • Pricing: album prices, track prices, “name your price” flags
  • Fan data: supporter counts, top supporters
  • Label info: label name, catalog number, other releases
  • Merch: physical products, bundles, pricing

Setting Up Your Scraper

First, install the required packages:

pip install requests beautifulsoup4 lxml

Bandcamp uses server-rendered HTML with embedded JSON-LD data, making it relatively straightforward to parse.

import requests
import json
import re
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import time

class BandcampScraper:
    def __init__(self, proxy_url=None):
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
        })
        if proxy_url:
            self.session.proxies = {"http": proxy_url, "https": proxy_url}

    def get_artist_info(self, artist_url):
        """Extract artist data from a Bandcamp artist page."""
        resp = self.session.get(artist_url)
        soup = BeautifulSoup(resp.text, "lxml")

        # Extract embedded JSON-LD data
        ld_json = soup.find("script", type="application/ld+json")
        data = json.loads(ld_json.string) if ld_json else {}

        # Extract band name and location
        band_name = soup.select_one("#band-name-location .title")  
        location = soup.select_one("#band-name-location .location")

        # Extract bio
        bio = soup.select_one(".signed-out-artists-bio-text")

        return {
            "name": band_name.text.strip() if band_name else data.get("name"),
            "location": location.text.strip() if location else "",
            "bio": bio.text.strip() if bio else "",
            "url": artist_url,
            "type": data.get("@type"),
            "image": data.get("image"),
        }

    def get_discography(self, artist_url):
        """Extract all releases from an artist page."""
        music_url = artist_url.rstrip("/") + "/music"
        resp = self.session.get(music_url)
        soup = BeautifulSoup(resp.text, "lxml")

        releases = []
        for item in soup.select(".music-grid-item"):
            link = item.select_one("a")
            title = item.select_one(".title")
            img = item.select_one("img")

            releases.append({
                "title": title.text.strip() if title else "",
                "url": urljoin(artist_url, link["href"]) if link else "",
                "art": img["src"] if img else ""
            })
        return releases

Extracting Album Pricing Data

Pricing analysis is where Bandcamp data gets really interesting for market research:

def get_album_details(self, album_url):
    """Extract pricing and track details from an album page."""
    resp = self.session.get(album_url)
    soup = BeautifulSoup(resp.text, "lxml")

    price_el = soup.select_one(".buyItem .base-text-color")
    nyp = soup.select_one(".buyItem .name-your-price")

    tracks = []
    for row in soup.select(".track_list .track_row_view"):
        track_title = row.select_one(".track-title")
        duration = row.select_one(".time")
        tracks.append({
            "title": track_title.text.strip() if track_title else "",
            "duration": duration.text.strip() if duration else ""
        })

    return {
        "url": album_url,
        "price": price_el.text.strip() if price_el else "Free",
        "name_your_price": nyp is not None,
        "tracks": tracks,
        "track_count": len(tracks)
    }

Genre and Tag Discovery

Bandcamp tags are excellent for music discovery and genre research:

def explore_tag(self, tag, sort="pop", page=1):
    """Explore releases by genre tag."""
    url = f"https://bandcamp.com/tag/{tag}?sort_field={sort}&page={page}"
    resp = self.session.get(url)
    soup = BeautifulSoup(resp.text, "lxml")

    results = []
    for item in soup.select(".item_list .item"):
        artist = item.select_one(".itemsubtext")
        title = item.select_one(".itemtext")
        results.append({
            "title": title.text.strip() if title else "",
            "artist": artist.text.strip() if artist else "",
            "tag": tag
        })
    return results

# Discover trending electronic music
scraper = BandcampScraper()
trending = scraper.explore_tag("electronic", sort="date")
for item in trending[:10]:
    print(f"{item['artist']} - {item['title']}")

Using Proxies for Scale

For large-scale Bandcamp research across thousands of artists, you need proxy rotation to avoid rate limiting. ThorData residential proxies provide reliable access with automatic IP rotation:

scraper = BandcampScraper(
    proxy_url="http://user:[email protected]:9000"
)

artist_urls = [
    "https://artist1.bandcamp.com",
    "https://artist2.bandcamp.com",
]

all_data = []
for url in artist_urls:
    info = scraper.get_artist_info(url)
    info["discography"] = scraper.get_discography(url)
    all_data.append(info)
    time.sleep(2)  # Respectful delay

Production-Ready Solution

For production use cases — label research, market analysis, or building music discovery tools — the Bandcamp Scraper on Apify handles all the edge cases: JavaScript rendering, pagination, rate limiting, and structured data output.

It exports clean JSON with artist profiles, discographies, pricing, and tags — ready for analysis or database import.

Use Cases for Bandcamp Data

Use Case Data Needed
Label scouting Artist info, genre tags, fan counts
Pricing research Album prices, NYP stats, currency
Genre analysis Tag exploration, trending releases
Music discovery Discographies, related artists
Market sizing Release counts, pricing distributions

Ethical Considerations

  • Respect rate limits: Add delays between requests (2+ seconds)
  • Cache responses: Do not re-scrape the same pages repeatedly
  • Public data only: Never attempt to access private fan data or sales figures
  • Attribution: Credit Bandcamp as your data source

Conclusion

Bandcamp is a goldmine for music industry research. While there is no official API, the structured HTML and embedded JSON-LD make it one of the cleaner sites to scrape. Combine Python extraction with ThorData proxies for reliable large-scale collection, or use a pre-built Bandcamp scraper for instant results.

Follow me for more web scraping tutorials covering music, social media, and data extraction guides.