I have been exploring backend development for a while now, and if there is one thing I have learned, it is that setting everything up from scratch is honestly the most frustrating part. You want to build something cool, but before you write a single line of actual logic, you are already dealing with logging setup, database connections, tracing, configs, and a hundred other things that have nothing to do with what you actually want to build.
That is exactly why I got excited when I came across GoFr.
So What Actually Is GoFr?
GoFr is an open source framework written in Go, and it is designed specifically for building microservices and backend APIs. The whole philosophy behind it is simple: give developers a solid, production ready foundation so they can focus on writing business logic instead of boilerplate.
It is not trying to be everything. It is trying to be the thing that gets you from zero to a working, production quality service as fast as possible.
The Problem GoFr Solves
Think about the last time you started a new backend project. You probably spent the first few hours just deciding which logging library to use, how to structure your folders, how to connect to your database, how to handle errors properly. And that is before you have even thought about observability, tracing, or health checks.
Most frameworks give you the routing and not much else. GoFr gives you the full picture. Logging, tracing, metrics, database support, Redis support, and a clean project structure are all included and already wired together properly. You do not have to make those decisions. You just start building.
What I Found Really Impressive
The first thing that stood out to me was how little code you need to get a working API running. A basic REST endpoint in GoFr is just a few lines. There is no magic happening behind the scenes that you cannot understand, but there is also no unnecessary repetition.
The built in observability is another big deal. In most projects, adding proper logging and tracing is a whole separate task that gets pushed to later. GoFr builds it in from the start, which means your service is production ready from day one and not just in theory.
The documentation is also genuinely good. It is clear, it has real examples, and it does not assume you already know everything. For someone who is still learning, that matters a lot.
The Community Side of Things
GoFr is fully open source and lives on GitHub. The maintainers are active, pull requests get reviewed, and the community is welcoming to newcomers. Whether you want to report a bug, suggest a feature, or just ask a question, there are real people on the other side who actually respond.
This is the kind of project where you can tell the people building it actually use it themselves. The decisions make sense. Nothing feels like it was added just to add it.
Who Should Try GoFr?
Honestly, if you are someone who works with Go or wants to start, GoFr is worth your time. It is also a great way to understand what a well structured microservice actually looks like in practice, because the framework basically teaches you good patterns just by using it.
Even if you are not planning to use it in production right away, spending a weekend with GoFr will change how you think about backend architecture.
How to Get Started
Head over to gofr.dev and take a look at the documentation. You can have a basic API running locally in under ten minutes. The getting started guide is beginner friendly and walks you through everything without assuming prior experience with Go frameworks.
After that, check out their GitHub repository at github.com/gofr-dev/gofr. Star it, explore the issues, and if you feel like contributing, the community is very open to new contributors whether you are fixing a bug, improving docs, or writing about your experience.
Final Thoughts
GoFr reminded me why I enjoy building things. Not because it is flashy or overhyped, but because it just works. It removes the friction, gives you a clean foundation, and lets you actually focus on the problem you are trying to solve.
If you have been looking for a reason to try Go, or if you have been frustrated with how much setup your current backend workflow requires, give GoFr a shot. I think you will be pleasantly surprised.

The Knowledge Gap Problem

If you’ve used AI coding agents, you’ve hit this wall: you ask for code using a new SDK, and the agent generates deprecated API calls. Google quantified this as the “Knowledge Gap” — the mismatch between static LLM training data and rapidly evolving software ecosystems.

Their benchmark puts it starkly: Gemini 3.0 Pro scored just 6.8% on Gemini SDK code generation without skill assistance. That means 93 out of 100 code generation attempts failed.

What Are Agent Skills?

An Agent Skill is a standalone instruction package that bridges this gap:

my-skill/
  SKILL.md           # Core instructions for the agent
  references/        # Official docs excerpts
  assets/            # Diagrams, images
  scripts/           # Helper scripts

The SKILL.md file loads into the agent’s context window, providing up-to-date instructions on how to use specific APIs and frameworks correctly.

The Benchmark Results

Google tested gemini-api-dev skill with 117 prompts across Python and TypeScript:

One file. 14x improvement. That’s not a marginal gain — it’s a paradigm shift in how we should be using AI coding agents.

ADK Integration

Google’s Agent Development Kit makes skill integration straightforward:

from google.adk.skills import SkillToolset
from google.adk.agents import Agent

skill = SkillToolset.load_skill_from_dir("./skills/gemini-api-dev")

agent = Agent(
    name="coding_agent",
    model="gemini-3.0-pro",
    tools=[skill],
    instruction="You are a Gemini API coding agent."
)

Cross-Platform Standard

Here’s what makes this particularly interesting: SKILL.md is becoming a de facto industry standard. The same skill file works across:

• Claude Code (Anthropic)
• Cursor (AI-native IDE)
• Windsurf (AI pair programming)
• Gemini CLI (Google)
• Codex CLI (OpenAI)
• GitHub Copilot

The awesome-agent-skills repo has 1,234+ skills registered, with management tools like skillport and openskills emerging.

Building Your Own Skill

Creating a skill is surprisingly simple:

1. Write SKILL.md — Instructions for the agent (what API to use, what’s deprecated)
2. Add references/ — Excerpted official docs, migration guides, code examples
3. Deploy — Place in .claude/skills/, .cursor/skills/, or load via ADK’s SkillToolset

The key is being specific about what’s correct (latest API patterns) and what’s forbidden (deprecated patterns).

Key Takeaway

The knowledge gap is a structural limitation of LLMs — no amount of model scaling fixes it. Agent Skills provide a practical engineering solution. Google’s 6.8% to 96% benchmark proves the approach works at scale.

Check out the gemini-skills repo and the Google Developers Blog post for details.

What frameworks are you building Agent Skills for? I’d love to hear about community experiences with SKILL.md across different AI coding tools.

Building a Clean Master-Detail App with Blazor Server, MudBlazor, and EF Core

When building business applications, one of the most common requirements is a master-detail workflow. Orders and order items are a perfect example.

You need a clean list view, a reliable detail page, inline editing, validation, filtering, and a structure that can grow without becoming messy.

That is exactly what BlazorOrderApp is designed to demonstrate.

This application is a production-ready Blazor Server solution built with MudBlazor for the UI, EF Core with SQL Server for persistence, and a clean architecture that separates concerns across Domain, Application, Infrastructure, and Web layers.

Instead of placing everything in pages or code-behind files, the app keeps business rules, data access, validation, and UI responsibilities clearly divided.

At the core of the app is a classic master-detail pattern. The Orders.razor page acts as the master list, giving users filtering, search, pagination, and a structured overview of all orders. The OrderDetail.razor page handles the detail side, where users can inspect customer information, manage order items, and update order status in a focused way.

A big strength of this project is its use of the Repository and Specification patterns. The repository layer keeps EF Core logic centralized and testable, while specifications allow queries to be defined as reusable objects instead of scattered LINQ expressions. This makes filtering, paging, and retrieving complex data much easier to maintain over time.

Validation is handled with FluentValidation, which keeps rules explicit and easy to evolve as business needs grow. Combined with MudBlazor dialogs for create, edit, and delete operations, the result is a workflow that feels polished and user-friendly without sacrificing structure behind the scenes.

Overall, BlazorOrderApp is a strong example of how to build a modern internal business application in Blazor. It shows that you can create a clean, scalable CRUD system with a professional UI, solid architecture, and maintainable backend patterns, all within the .NET ecosystem.

Source code : https://github.com/stevsharp/BlazorOrderApp

War diary. Real failures. Real fix. No inspirational BS.

I’ll be honest with you: I spent three weekends building a Vinted scraper that worked exactly once.

The second time I ran it, I got a 403. The third time, my entire datacenter IP range was silently blacklisted. By the fourth weekend, I wasn’t writing Python anymore – I was googling “residential proxy Vinted” at 2am and reading forum posts from people who had clearly given up.

This is the story of how I tried to build an arbitrage pipeline for Vinted, why everything broke, what I learned from every failure, and why I eventually stopped reinventing the wheel.

If you’re a dev or data engineer trying to extract data from Vinted for price monitoring, cross-border arbitrage, or inventory hunting – read this before you repeat my mistakes.

🔥 Why Traditional Vinted Scraping Fails (The Bot Protection Problem)

Vinted is not a public API. It never was. Their internal API is undocumented, versioned inconsistently, and protected by a stack of bot-detection layers that get more aggressive every quarter.

The first mistake most devs make – including me – is treating Vinted like a simple HTML scrape target.

It’s not. The listing data is rendered client-side via API calls. So step one is: forget BeautifulSoup. You’re not scraping HTML. You’re reverse-engineering JSON endpoints that change without warning.

That discovery costs you your first weekend.

Then come the real problems.

🛑 Datacenter IPs vs. Residential Proxies

Vinted’s bot protection stack fingerprints requests at multiple layers:

• IP reputation – Datacenter IPs (AWS, GCP, Hetzner, OVH) are flagged by default. Even if your request headers look perfect, your IP subnet is already on a deny list.
• TLS fingerprinting – Python’s requests library and standard httpx produce a TLS fingerprint that differs from a real browser. Vinted (like most modern platforms) uses JA3/JA4 fingerprinting to detect non-browser clients.
• Behavioral analysis – Consistent intervals, identical User-Agents, no mouse events, no referrer chains. Bots don’t browse like humans. Vinted knows.

I tested this empirically. Same request, two IP sources:

The pattern is clear: residential proxies extend your session, but they’re not a silver bullet if you’re hitting the wrong geo.

Which brings us to the next problem.

🌍 The Geo-Blocking Nightmare (19 European Countries)

Vinted operates across 19 European markets. Each market has its own subdomain:

• www.vinted.fr (France)
• www.vinted.de (Germany)
• www.vinted.pl (Poland)
• www.vinted.es (Spain)
• www.vinted.it (Italy)
• …and 14 more

Here’s what they don’t tell you: these aren’t just URL namespaces. The platform actively routes and restricts access based on your IP’s geolocation. Making a request to vinted.de from a French IP? You’ll get content, but category filtering, pricing, and shipping data will be inconsistent. Making the same request from a US IP? Silent redirect or empty results.

For vinted arbitrage, this is catastrophic. The entire point is to monitor price gaps between markets – buy cheap in Poland, sell at markup in France. If you can’t reliably query both markets from a consistent perspective, your arbitrage pipeline is garbage data.

To do real cross-border arbitrage monitoring, you need:

1. A residential proxy pool with per-country targeting
2. Session management per country (cookies, tokens, headers)
3. Rate limiting per country (they’re independent backends)

I built this. It took two more weekends. It worked for about a week, then Vinted updated their token rotation and the whole thing broke again.

⏳ The Pagination Bottleneck and Rate Limits

Even when you solve the IP problem, Vinted’s pagination is its own circle of hell.

Vinted’s search API uses cursor-based pagination. That sounds fine until you realize:

• The cursor becomes invalid after a few minutes
• Deep pagination (page 10+) triggers additional verification
• Rate limits are per-IP, not per-session, so rotating proxies too fast creates a new problem: you lose cursor continuity

Here’s what a naive pagination loop looks like:

cursor = None
while True:
    response = session.get(
        "https://www.vinted.fr/api/v2/catalog/items",
        params={"search_text": "Nike Dunk Low", "cursor": cursor},
        headers=headers,
        proxies=proxy
    )
    data = response.json()
    items = data.get("items", [])
    process(items)
    cursor = data.get("pagination", {}).get("next_cursor")
    if not cursor:
        break
    time.sleep(random.uniform(1.5, 3.5))

This works for the first two or three pages. Then one of these happens:

• cursor expires mid-loop – restart from page 1, duplicates everywhere
• Rate limit kicks in – 429, your proxy is flagged
• The session token rotates – 401, re-auth required
• The item count returns 0 unexpectedly – silent failure, you stop early without knowing

Handling all of this correctly requires retry logic, session refresh, cursor state management, and dead-letter queuing for failed pages. You’re not writing a scraper anymore. You’re writing a distributed job queue.

⚡ Why Asynchronous Requests Get You Banned Faster

The logical next step: “Let me make this faster with async.”

Wrong move.

I tried asyncio + aiohttp to parallelize requests across multiple proxies. Result: banned in under 10 minutes across my entire proxy pool.

Here’s why: async request patterns are a strong behavioral signal. Even with random delays, the statistical distribution of request timing differs from human browsing. Vinted’s rate limits aren’t just about volume – they’re about velocity and pattern regularity.

The sweet spot for staying undetected is low concurrency (1-2 simultaneous sessions per country), with jittered delays drawn from distributions that approximate real browsing behavior. Not uniform random – actual distributions skewed toward longer pauses.

By the time I had tuned this correctly, my “fast scraper” was running at 40 items/minute per country. At that rate, monitoring 5 categories across 5 countries would take hours per full cycle.

This is not a pipeline. This is a liability.

🏗️ Building the Ultimate Vinted Data Pipeline (Or Why I Stopped Trying)

Let me describe the architecture I eventually sketched out before I abandoned it.

What you’d need to build a production-grade Vinted scraper from scratch:

1. Residential proxy pool – country-targeted, with session stickiness per market. Cost: €80-200/month minimum for a reliable pool.
2. Browser fingerprint spoofing – TLS client hello matching a real browser, User-Agent rotation, Accept-Language matching the target country.
3. Session management – Vinted requires a guest token for API access. This token has to be obtained via a browser-like flow, then refreshed on a schedule. Store per-country, rotate on 401.
4. Rate limit tracker – per-country, per-proxy, with exponential backoff and cool-down periods.
5. Cursor state manager – checkpointing pagination cursors so a failure mid-run doesn’t restart from scratch.
6. Schema parser – Vinted’s API response schema changes. You need a versioned parser with fallback fields.
7. Output pipeline – JSON normalization, deduplication, webhook delivery (Discord, n8n, Make, custom endpoints).
8. Monitoring & alerting – because this thing will break silently at 3am.

Estimated build time: 4-6 weeks for a competent backend dev. Estimated maintenance: ongoing, every time Vinted updates their anti-bot stack.

And here’s the part nobody tells you: even if you build all of this, you’re in a constant arms race. Vinted has an entire security team whose job is to break your scraper. You’re one person.

I’m a developer. I understand the appeal of building this from scratch. I felt it too. But at some point, I did the math.

My time x its value vs. the cost of a working solution = I should stop building this.

🚀 Enter the Vinted Smart Scraper on Apify

After my third rebuild, a colleague pointed me to the Vinted Smart Scraper on Apify.

I was skeptical. I’d seen plenty of half-baked scrapers on Apify that stopped working after a month. But I read the documentation and ran a test, and here’s what I found:

The actor is built specifically for the problems I spent three weekends failing to solve. It’s not a generic scraper with Vinted bolted on. It’s purpose-built for the Vinted API surface, with the infrastructure already handled.

You pass in a search query, target category, and country. The actor handles:

• Proxy rotation and country-targeting automatically
• Session token management
• Pagination across the full result set
• Clean JSON output per item (title, price, condition, photos, seller info, URL, shipping)

The result set is consistent. The pagination completes. The data is structured.

Here’s what a basic run config looks like:

{
  "searchQuery": "Nike Dunk Low",
  "country": "FR",
  "maxItems": 500,
  "condition": ["Neuf avec étiquette", "Très bon état"]
}

And you get back clean, normalized items ready for your pipeline.

No proxy configuration. No session management. No cursor headaches.

🌐 Built-in Geo-Routing for European Arbitrage

The key feature for cross-border arbitrage: the Vinted Smart Scraper supports multi-country runs. You can run parallel actors per country and get consistent, comparable data across markets.

This is the part I couldn’t reliably build myself. Geo-routing residential proxies to match Vinted’s market segmentation is infrastructure-level work. It’s solved here.

For a sneaker arbitrage use case, a typical setup looks like:

1. Run actor for country: PL (Poland) – find underpriced listings
2. Run actor for country: FR (France) – check equivalent items
3. Compare price delta in n8n or a simple Python script
4. Trigger Discord webhook alert when spread exceeds threshold

The actor handles steps 1 and 2. You only have to write the delta logic.

📦 Clean JSON Output for n8n and Discord Webhooks

The output schema is consistent and webhook-ready. A sample item output:

{
  "id": "4829301",
  "title": "Nike Dunk Low Retro White Black",
  "price": 68.00,
  "currency": "EUR",
  "condition": "Très bon état",
  "size": "42",
  "brand": "Nike",
  "photos": ["https://images.vinted.net/..."],
  "url": "https://www.vinted.fr/items/4829301",
  "seller": {
    "username": "sneaker_seller_42",
    "rating": 4.9,
    "total_reviews": 230
  },
  "shipping_options": [...],
  "created_at": "2025-05-15T10:23:00Z"
}

This pipes directly into n8n HTTP nodes, Discord embed webhooks, or Google Sheets via any standard automation platform. No parsing gymnastics. No field guessing.

For those of us running lightweight arbitrage monitors on n8n or Make: this is the missing data source that makes the rest of the automation trivial.

📈 Stop Reinventing the Wheel, Start Scaling

Here’s what I actually run now:

• Vinted Smart Scraper scheduled on Apify (every 4 hours, 5 countries, 3 search queries each)
• n8n to receive the output webhook, compute price deltas, filter by condition
• Discord webhook to push alerts when a deal meets criteria (min 30% below market average)
• Google Sheets for deal history and price trend visualization

Total setup time: ~4 hours. Total recurring cost: a few euros/month of Apify compute credits.

Compare that to: 6+ weekends of my time, ongoing proxy costs (€80+/month), and a system that breaks every time Vinted updates their anti-bot.

The math is not close.

🧠 What I Would Have Told Myself at Weekend One

If you’re starting this journey now:

1. Don’t start with raw HTTP requests. Vinted’s bot protection will end you inside an hour on datacenter IPs.
2. Residential proxies help, but geo-targeting matters. A UK residential proxy cannot reliably query vinted.de without soft-blocks.
3. Async is a trap. High velocity is suspicious. Vinted’s defenses aren’t just rate limits – they’re behavioral fingerprinting.
4. Pagination is stateful. Cursor expiry plus proxy rotation equals data loss at scale.
5. Your time has a cost. Building this yourself is an option. Maintaining it forever is the hidden cost.

What I wish had existed at weekend one is exactly this: a purpose-built actor that solves the infrastructure problem so you never have to fight the proxy/token/cursor war yourself.

✅ Final Verdict

If you’re building a Vinted arbitrage system, a price monitor, or any kind of web scraping Vinted pipeline: the bottleneck isn’t your logic. It’s the IP ban, the geo-block, the cursor expiry, and the rotating tokens.

Solve the data access layer first. The Vinted Smart Scraper does exactly that.

Use it as your foundation. Build the arbitrage logic on top. Ship it in a weekend.

→ Get the Vinted Smart Scraper on Apify

Tags: #webdev #python #dataengineering #webscraping #vinted #sideproject

📋 Article Metadata (for dev.to publication)

Title: The Vinted Arbitrage War: Building a Scraper That Doesn’t Get IP-Banned

Cover image suggestion: Dark terminal screenshot with a 403 error and a Python traceback – mood-setting, developer-resonant

Tags: webdev, python, dataengineering, tutorial, beginners

Canonical URL: (set to your own domain if cross-posting)

Series suggestion: “Scraping the Hard Way”

Connecting LINE Bot to Home Lab Without cloudflared — WebSocket Relay Architecture

When running a LINE Bot in a local environment, the first wall you hit is receiving webhooks from the outside world. LINE sends HTTPS POST requests from their servers. Your local machine has no global IP.

The usual answers are ngrok or Cloudflare Tunnel (cloudflared). But both have friction. ngrok changes URLs on every restart with the free tier. cloudflared needs a persistent daemon and non-trivial setup.

At TechsFree, we took a different approach: a WebSocket relay server in the middle.

Architecture

LINE → HTTPS POST → Relay Server (linebot.techsfree.com)
                         ↓ WebSocket
                    Local Node (line_bridge.py)
                         ↓ HTTP POST
                    OpenClaw chatCompletions API (localhost:18789)
                         ↓
                    LINE Reply API → User

The relay server sits on the cloud side, always accepting HTTPS. The local line_bridge.py keeps a persistent outbound WebSocket connection to it. Since the connection is outbound (inside → outside), it passes through firewalls without any port forwarding. When a LINE webhook arrives, the relay forwards it over the WebSocket.

Why WebSocket?

HTTP polling adds latency. Server-Sent Events are unidirectional. WebSocket gives you bidirectional, low-latency, persistent connection — ideal for a relay pattern.

Implementation Highlights

1. Auto-Reconnect with Exponential Backoff

Networks always fail. Home routers reboot. Connections drop on sleep/wake. The bridge must recover on its own.

reconnect_delay = RECONNECT_DELAY  # 5 seconds
MAX_RECONNECT_DELAY = 60

while True:
    try:
        async with websockets.connect(relay_url, ping_interval=20, ping_timeout=10) as ws:
            reconnect_delay = RECONNECT_DELAY  # reset on success
            async for raw_message in ws:
                ...
    except ConnectionClosed as e:
        logger.warning(f"Connection closed: {e}")
    except Exception as e:
        logger.error(f"Connection error: {e}")

    await asyncio.sleep(reconnect_delay)
    reconnect_delay = min(reconnect_delay * 2, MAX_RECONNECT_DELAY)

The key detail: reset the delay on success. Only grow the backoff on errors. Cap at 60 seconds — no infinite waiting.

2. LINE Signature Verification

Every LINE webhook carries an x-line-signature header. Skip HMAC-SHA256 verification and you’ll process forged requests.

def verify_signature(body: str, signature: str, channel_secret: str) -> bool:
    hash_val = hmac.new(
        channel_secret.encode("utf-8"),
        body.encode("utf-8"),
        hashlib.sha256
    ).digest()
    expected = base64.b64encode(hash_val).decode("utf-8")
    return hmac.compare_digest(signature, expected)

Using hmac.compare_digest instead of == prevents timing attacks — string comparison time can leak whether characters match.

3. Config Reload on Every Webhook

The bridge reloads its config file on every incoming webhook:

elif msg_type == "webhook":
    current_config = load_config()  # reload every time
    asyncio.create_task(handle_webhook(msg, current_config))

This means LINE tokens and channel_secret can be swapped out without restarting the daemon. In our miniPC distribution scenario, we ship a unit to a customer, they fill in their credentials, and the bridge picks them up immediately — no restart needed.

4. OpenClaw chatCompletions API Integration

When a LINE message arrives, the bridge calls the local OpenClaw API:

async def call_openclaw(message: str, openclaw_url: str, gateway_token: str = "") -> str:
    payload = {
        "model": "gpt-4",
        "messages": [{"role": "user", "content": message}]
    }
    headers = {"Content-Type": "application/json"}
    if gateway_token:
        headers["Authorization"] = f"Bearer {gateway_token}"
    async with httpx.AsyncClient(timeout=60) as client:
        resp = await client.post(openclaw_url, json=payload, headers=headers)
        resp.raise_for_status()
        return resp.json()["choices"][0]["message"]["content"]

It uses the OpenAI-compatible chatCompletions format, so you can point it at Ollama or LM Studio just by changing the URL.

Designed for Distribution

line_bridge.py is built to ship inside the miniPCs TechsFree sells. Each unit gets a unique relay_token; the relay server uses it to route webhooks to the right local bridge.

{
  "relay_token": "gmk-xxxxxxxx",
  "channel_secret": "...",
  "channel_access_token": "...",
  "relay_url": "wss://linebot.techsfree.com/gmk/{token}/ws",
  "openclaw_url": "http://localhost:18789/v1/chat/completions"
}

One JSON file per unit. A GUI fills in the fields — that’s the entire setup for end users.

Comparison

cloudflared is powerful, but “dependency on an external service” was a concern. Running your own relay means authentication, logging, and routing are all under your control. For scenarios like managing many local environments — home labs or distributed miniPC products — the self-hosted relay approach is a clean fit.

There are over 4.8 million active Shopify stores worldwide. If you’re doing competitor research, building a price comparison tool, tracking product trends, or sourcing for dropshipping, Shopify stores are a goldmine of structured product data.

The best part? Shopify has a little-known public endpoint that makes scraping significantly easier than most ecommerce platforms.

The Shopify products.json Endpoint

Every Shopify store exposes a public JSON API at /products.json. No authentication needed:

https://store-name.myshopify.com/products.json
https://custom-domain.com/products.json

This returns up to 250 products per page with full details: titles, descriptions, prices, variants, images, inventory availability, and more.

Basic Shopify Scraper

Here’s a straightforward Python scraper that pulls every product from a Shopify store:

import requests
import time
import csv

def scrape_shopify_store(domain: str) -> list[dict]:
    """Scrape all products from a Shopify store."""
    products = []
    page = 1

    while True:
        url = f"https://{domain}/products.json?limit=250&page={page}"

        response = requests.get(url, timeout=30, headers={
            "User-Agent": "Mozilla/5.0 (compatible; ProductResearch/1.0)"
        })

        if response.status_code == 430:
            print(f"Rate limited on page {page}, waiting 30s...")
            time.sleep(30)
            continue

        if response.status_code != 200:
            print(f"Got status {response.status_code} on page {page}")
            break

        data = response.json()
        batch = data.get("products", [])

        if not batch:
            break

        for product in batch:
            for variant in product.get("variants", []):
                products.append({
                    "title": product["title"],
                    "product_type": product.get("product_type", ""),
                    "vendor": product.get("vendor", ""),
                    "variant": variant.get("title", "Default"),
                    "price": variant.get("price"),
                    "compare_at_price": variant.get("compare_at_price"),
                    "sku": variant.get("sku"),
                    "available": variant.get("available", False),
                    "created_at": product.get("created_at"),
                    "updated_at": product.get("updated_at"),
                    "tags": ", ".join(product.get("tags", [])),
                    "image": product.get("images", [{}])[0].get("src", ""),
                    "url": f"https://{domain}/products/{product['handle']}",
                })

        print(f"Page {page}: {len(batch)} products")
        page += 1
        time.sleep(1)

    return products

# Scrape a store
products = scrape_shopify_store("allbirds.com")
print(f"Total variants: {len(products)}")

Exporting to CSV

def export_to_csv(products: list[dict], filename: str = "products.csv"):
    """Export scraped products to CSV."""
    if not products:
        return

    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=products[0].keys())
        writer.writeheader()
        writer.writerows(products)

    print(f"Exported {len(products)} products to {filename}")

export_to_csv(products)

Scraping Multiple Stores for Competitive Analysis

For comparing competitors, batch scrape and save each store separately:

import json
from pathlib import Path

COMPETITORS = [
    "allbirds.com",
    "gymshark.com",
    "fashionnova.com",
    "colourpop.com",
    "kyliecosmetics.com",
]

def scrape_competitors(domains: list[str], output_dir: str = "data"):
    """Scrape multiple Shopify stores and save results."""
    Path(output_dir).mkdir(exist_ok=True)

    for domain in domains:
        print(f"nScraping {domain}...")
        try:
            products = scrape_shopify_store(domain)

            output_file = Path(output_dir) / f"{domain.replace('.', '_')}.json"
            with open(output_file, "w") as f:
                json.dump(products, f, indent=2)

            print(f"  Saved {len(products)} variants to {output_file}")
        except Exception as e:
            print(f"  Failed: {e}")

        time.sleep(5)  # Be nice between stores

scrape_competitors(COMPETITORS)

Detecting if a Site Runs Shopify

Not sure if a website is on Shopify? Check programmatically:

def is_shopify(domain: str) -> bool:
    """Detect if a website runs on Shopify."""
    try:
        resp = requests.get(
            f"https://{domain}/products.json?limit=1",
            timeout=10,
            allow_redirects=True,
        )
        if resp.status_code == 200:
            data = resp.json()
            return "products" in data
    except Exception:
        pass

    # Fallback: check for Shopify headers
    try:
        resp = requests.head(f"https://{domain}", timeout=10)
        return "shopify" in resp.headers.get("x-shopify-stage", "").lower() 
            or "shopify" in resp.headers.get("server", "").lower()
    except Exception:
        return False

# Test some domains
for domain in ["allbirds.com", "nike.com", "gymshark.com"]:
    print(f"{domain}: {'Shopify' if is_shopify(domain) else 'Not Shopify'}")

Handling Common Challenges

Rate Limiting (HTTP 430)

Shopify returns status 430 when you hit their rate limit. Solutions:

1. Add delays — 1-2 seconds between pages, 5+ seconds between stores
2. Use ScraperAPI — rotates IPs automatically so you spread requests across many addresses
3. Use ThorData residential proxies — a pool of real residential IPs makes your requests look like normal browser traffic

def scrape_with_proxy(domain: str, proxy_url: str) -> list[dict]:
    """Scrape with rotating proxy to avoid rate limits."""
    products = []
    page = 1

    while True:
        url = f"https://{domain}/products.json?limit=250&page={page}"

        response = requests.get(url, proxies={
            "http": proxy_url,
            "https": proxy_url,
        }, timeout=30)

        if response.status_code != 200:
            break

        data = response.json()
        batch = data.get("products", [])
        if not batch:
            break

        products.extend(batch)
        page += 1
        time.sleep(1)

    return products

Stores That Block products.json

Some merchants disable the public endpoint. In that case:

• Scrape the collection pages HTML instead (/collections/all)
• Use ScraperAPI with render=true to execute JavaScript
• Check out pre-built Shopify scrapers on Apify that handle edge cases like custom themes and pagination

Getting Collection and Category Data

def scrape_collections(domain: str) -> list[dict]:
    """Get all product collections/categories."""
    url = f"https://{domain}/collections.json"
    response = requests.get(url, timeout=30)

    if response.status_code != 200:
        return []

    collections = response.json().get("collections", [])
    return [{
        "title": c["title"],
        "handle": c["handle"],
        "products_count": c.get("products_count"),
        "url": f"https://{domain}/collections/{c['handle']}",
    } for c in collections]

Building a Price Monitor

The real power is in running this daily to track price changes:

import sqlite3
from datetime import datetime

def init_db(db_path: str = "prices.db"):
    conn = sqlite3.connect(db_path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS price_history (
            domain TEXT,
            sku TEXT,
            title TEXT,
            price REAL,
            compare_at_price REAL,
            available BOOLEAN,
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    return conn

def record_prices(conn, domain: str, products: list[dict]):
    for p in products:
        conn.execute(
            "INSERT INTO price_history (domain, sku, title, price, compare_at_price, available) VALUES (?, ?, ?, ?, ?, ?)",
            (domain, p["sku"], p["title"], float(p["price"]),
             float(p["compare_at_price"]) if p["compare_at_price"] else None,
             p["available"])
        )
    conn.commit()

# Run daily via cron
conn = init_db()
products = scrape_shopify_store("competitor.com")
record_prices(conn, "competitor.com", products)

Set this up as a cron job and you have a full competitor price tracker running for free.

Legal Notes

The products.json endpoint serves publicly available data that Shopify intentionally exposes. That said:

• Don’t hammer stores with high-frequency requests
• Don’t scrape and republish product descriptions (copyright)
• Don’t use scraped data for fake reviews or misleading comparisons
• Respect rate limits — the 430 status exists for a reason

Conclusion

Shopify’s public JSON API makes it one of the easiest ecommerce platforms to scrape. For stores that block the endpoint, tools like ScraperAPI and ThorData proxies handle the heavy lifting. And if you’d rather skip the code, Apify has cloud-based Shopify scrapers ready to go.

Happy scraping — let me know in the comments what you’re building!

For decades, the term “computer worm” meant malicious code exploiting binary vulnerabilities. From the 1988 Morris Worm to modern ransomware, we’ve been in a constant arms race.

But as we move from simple chatbots to complex Multi-Agent Systems (MAS), a new, more insidious threat has emerged: the AI Worm.

Unlike traditional malware, these “digital parasites” don’t target your source code. They target the very fabric of AI communication: language.

What exactly is an AI Worm?

An AI worm is a piece of self-replicating prompt malware. It’s a malicious instruction embedded within an innocuous-looking email or document.

When an AI agent processes this data, the prompt does two things:

1. Tricks the agent into performing an unwanted action (like exfiltrating data).
2. Compels the agent to replicate and spread that same instruction to other agents or systems.

This isn’t science fiction. Researchers have already demonstrated this with Morris II, a zero-click worm that targets generative AI ecosystems.

The Anatomy of a Self-Replicating Prompt

How does a string of text become a virus? It happens in three stages:

1. Replication

The attacker crafts a prompt that forces the LLM to include the malicious instruction in its own output. Think of it like a “jailbreak” that survives a summary. If an agent summarizes an infected document, the summary itself now contains the malware.

2. Propagation

This is where the “worm” part comes in. AI agents are often connected to tools such as email clients, Slack, or databases. The replicated prompt instructs the compromised agent to use these tools to send the malware to new targets.

Example: An AI email assistant summarizes an infected message and then forwards that summary to everyone in your contact list.

3. Payload

The final goal. This could be anything from stealing sensitive PII to launching automated spam campaigns. This often uses Indirect Prompt Injection (IPI), where the malware is hidden in data the AI processes naturally, making it incredibly hard to detect.

Why Multi-Agent Systems (MAS) are Vulnerable

In a MAS, agents collaborate and share information autonomously. This interconnectedness is a double-edged sword.

• Trust Assumptions: Developers often assume internal agent-to-agent communication is safe. If one agent is compromised, the infection can cascade through the entire system.
• Agentic RAG: Retrieval-Augmented Generation allows agents to pull data from external sources (web, emails, docs). This creates a massive attack surface for malicious prompts to enter the system.
• Tool Access: Modern agents have “hands”, they can send emails, update databases, or even trigger financial transactions. An AI worm uses these hands to spread itself.

The Enterprise Risk: Zero-Click Infections

The scariest part? Zero-click infections.

Unlike traditional phishing, where a human has to click a link, an AI worm can spread without any human interaction. If your agent is set to automatically process incoming support tickets or emails, it can become infected and start propagating the malware the moment it reads the text.

This leads to:

• Data Exfiltration: Sensitive customer or company data sent to unauthorized recipients.
• Poisoned Knowledge Bases: Malicious prompts subtly altering stored info, leading to flawed business decisions.
• Automated Spam/Misinformation: Your own agents being used to damage your brand reputation.

How to Secure Your Agentic Workflows

Building a secure MAS requires moving beyond traditional code-centric defenses. Here are some practical best practices:

1. Treat All LLM Outputs as Untrusted

Never assume an agent’s output is safe just because it’s “internal.” Implement rigorous input/output sanitization. Scan for known malicious patterns or unexpected commands before any agent-generated text is acted upon.

2. The Principle of Least Privilege

Give your agents only the tools they absolutely need. An email summarizer doesn’t need the ability to send emails or modify your database.

3. Human-in-the-Loop (HITL)

For high-stakes actions, like financial transactions or communicating with external clients, always require a human “circuit breaker” to approve the action.

4. Sandbox Your Agents

Isolate agents and their LLMs in sandboxed environments. If one agent gets infected, the sandbox prevents the malware from spreading laterally to the rest of your infrastructure.

Securing the Future

The future of AI security is the security of language. As we entrust more of our business logic to autonomous agents, we need specialized layers that can monitor and protect these linguistic interactions.

Solutions like NeuralTrust are designed for this exact purpose—providing the visibility and control needed to detect indirect prompt injections and stop self-replicating prompts before they can do damage.

Are you building with multi-agent systems? How are you handling prompt security? Let’s discuss in the comments!

Most developers answer “tell me about yourself” like a resume. That kills the interview in 60 seconds. Here are 5 patterns that turn it into a technical discussion about your strongest work.

1. Replace Titles With Impact Statements

Most answers start with a job title. That adds zero signal.

Before

I am a Senior Frontend Developer with 7 years of experience in React and TypeScript.

After

I build customer-facing dashboards for a fintech product processing $2M in daily transactions.

The second version compresses domain, scale, and responsibility into one sentence. You remove 6 words and add actual signal.

2. Anchor Your Story in One Measurable Result

Listing technologies is noise. One quantified result is sticky.

Before

I work with React, Node.js, PostgreSQL, Redis, Docker, and AWS.

After

Last year I rebuilt our checkout flow using React Server Components and reduced time-to-interactive from 3.2s to 900ms, increasing conversion by 12%.

Now the interviewer has something to ask about. Numbers trigger follow-up questions. No one asks about your tech list.

3. Use Architecture as a Hook, Not a Buzzword

Mentioning architecture without context sounds generic. Tie it to a constraint.

Before

I worked on microservices and scalable systems.

After

I redesigned our order pipeline to handle Black Friday traffic by moving to a queue-based architecture with Redis, cutting API latency from 1200ms to 180ms under load.

This creates a natural follow-up path into system design. If you want to go deeper into backend bottlenecks, this pairs well with the patterns in the Node.js memory leak debugging strategies that prevent production crashes.

4. Match Your Achievement to the Role’s Core Problem

Most candidates reuse the same story everywhere. That is a mistake.

Before

I built a social media app with React and Firebase.

After (for a performance-heavy role)

I optimized a real-time analytics dashboard rendering 5,000 data points at 60fps using virtualization and WebSocket streaming.

Same developer. Different positioning. The second answer aligns with what the company actually needs.

5. Close With a Technical Alignment, Not a Career Goal

“Looking for growth” is a dead sentence. Show direct overlap.

Before

I am looking for new challenges and opportunities to grow.

After

I saw your team is moving to Next.js with server components, and that is exactly the architecture I shipped in production last year.

This tells the interviewer you read the job description and already solved their problem somewhere else.

6. Control the Next Questions With Intentional Hooks

Your intro determines the next 10 minutes. Use that.

Before

I have experience across frontend and backend systems.

After

I led a migration from Angular to React that reduced bundle size by 40% and improved load time by 2 seconds across 200K users.

Now the interviewer will ask:

• how you handled the migration
• how you measured performance
• what broke during rollout

You just turned random questions into a deep dive on your strongest project.

7. Compress Everything Into 45 to 60 Seconds

Long answers signal lack of prioritization.

Before

I started programming in university, then worked at three companies, learned multiple stacks, contributed to different projects, and...

After

I build high-performance React applications. Recently I rebuilt a checkout flow that improved conversion by 12%. I saw your team is solving similar performance problems, and that is exactly where I add value.

Three sentences. Clear structure. No filler.

You do not need a better resume to pass interviews. You need a better first 60 seconds. Pick one project, attach numbers, align it to the role, and let the interviewer follow your lead.

How Ubuntu 24.04’s PEP 668 Broke Our Month-End Automation — SSH Multi-Node Workaround

Background

We run automation across multiple nodes in our home lab. At month-end, a script called monthly-endofmonth.sh handles time-tracking, invoice submission, and form filling automatically.

This month, it exploded on first run.

What Happened

The error surfaced like this:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, either use a
    distribution package (apt install python3-xxx) or create
    a virtual environment (python3 -m venv PATH)...

Running pip install playwright on Ubuntu 24.04 got blocked by PEP 668.

PEP 668, adopted in 2022, changes how Python package management works. On Ubuntu 24.04 (Noble), direct pip install to the system Python is forbidden. The goal is to prevent apt-managed Python and pip from conflicting and breaking the system.

Previously, pip install --break-system-packages could force through. Ubuntu 24.04 blocks even that in many cases.

The Real Root Cause: Phase 0 and Production Ran on Different Nodes

PEP 668 itself is well-known. The real problem was something else.

Phase 0 (pre-flight check) ran on our infra server — a node with playwright, openpyxl, and google-auth already set up. Everything passed.

But the actual month-end job was designed to run on a different Ubuntu 24.04 node that had none of those libraries.

Result: Phase 0 all green → production blew up immediately.

This environment mismatch is a classic multi-node trap. The infra equivalent of “works on my machine.”

The Fix: Route Python Jobs via SSH to the Prepared Node

We changed the approach: delegate Python execution to the infra server via SSH.

The key change in monthly-endofmonth.sh:

# Before: local execution on Ubuntu 24.04 node (blocked by PEP 668)
python3 /mnt/shared/scripts/make_timesheet.py

# After: run on infra server via SSH
ssh user@infra-server "python3 /mnt/shared/scripts/make_timesheet.py"

The infra server has a fixed LAN IP, and all scripts are on an NFS mount (/mnt/shared/), so both nodes access the same files regardless of which machine runs the command.

Post-fix results:

Lessons Learned

1. Run Phase 0 Checks on the Production Node

“It worked somewhere” doesn’t mean “it works everywhere.” Dependencies, Python environments, and file paths vary per node. Phase 0 should run on the actual node that will execute production jobs.

2. Move Work to the Prepared Environment, Not the Other Way Around

Fighting PEP 668 is the wrong battle. Moving the job to an already-configured environment is faster and safer.

Using venv is also an option, but in a multi-node NFS setup, managing venv paths across machines gets messy. SSH-delegating to a single trusted node is simpler.

3. Test Month-End Scripts Mid-Month

Testing one day before month-end is too late. Next month, we plan to add a --dry-run pass around the 25th to catch environment issues early.

Bonus: The SameFileError Bug

We found a bonus bug during debugging. Running make_timesheet.py twice on the same day produces:

shutil.SameFileError: '/mnt/shared/.../timesheet.xlsx' and '/mnt/shared/.../timesheet.xlsx' are the same file

shutil.copy() was passed the same path as both source and destination. A simple existence check before copying will fix it — planned for next month’s run.

Summary

• Ubuntu 24.04 enforces PEP 668, blocking system-wide pip install
• Multi-node setups need explicit awareness of which node runs which job
• SSH-delegating to a prepared environment is a clean, pragmatic solution
• Month-end automation needs mid-month dry-run testing, not last-minute manual checks

Fiverr is a goldmine of freelance market data — gig prices, seller ratings, delivery times, service categories, and buyer reviews. Whether you’re researching freelance pricing trends, building a competitor analysis tool, or studying the gig economy, scraping Fiverr gives you structured data that their platform doesn’t expose through any public API.

In this guide, I’ll show you how to scrape Fiverr gig listings, seller profiles, and reviews with Python. I’ll cover what works, what breaks, and how to handle Fiverr’s anti-bot measures.

What Data Can You Extract from Fiverr?

Here’s what’s available on public Fiverr pages:

• Gig listings — title, description, pricing tiers (Basic/Standard/Premium), delivery time
• Seller profiles — username, level (New/Level 1/2/Top Rated), response time, country, member since
• Reviews — star rating, review text, buyer country, date
• Category data — subcategories, number of services available
• Search results — gigs ranked by relevance/best selling/newest for any keyword

Step 1: Understanding Fiverr’s URL Structure

Fiverr search URLs are straightforward:

https://www.fiverr.com/search/gigs?query=web+scraping&source=top-bar&ref_ctx_id=...&page=1

Key parameters:

• query — search keywords
• page — pagination (starts at 1)
• category_id — filter by category
• delivery_time — filter by delivery speed

Individual gig pages follow the pattern:

https://www.fiverr.com/{seller_username}/{gig_slug}

Step 2: Scraping Search Results

Fiverr’s search results render with JavaScript, but the initial HTML contains enough structured data to get started. Here’s a scraper for gig listings:

import requests
from bs4 import BeautifulSoup
import json
import time
import random
import csv

def scrape_fiverr_search(query, pages=3):
    """Scrape Fiverr search results for gig listings."""

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Referer": "https://www.fiverr.com/",
    }

    all_gigs = []

    for page in range(1, pages + 1):
        url = f"https://www.fiverr.com/search/gigs?query={query}&page={page}"

        response = requests.get(url, headers=headers, timeout=30)

        if response.status_code != 200:
            print(f"Page {page}: Status {response.status_code}")
            continue

        soup = BeautifulSoup(response.text, "html.parser")

        # Fiverr embeds gig data in JSON within the page
        # Look for the Next.js data script
        for script in soup.select('script[id="__NEXT_DATA__"]'):
            try:
                data = json.loads(script.string)
                gigs_data = (
                    data.get("props", {})
                    .get("pageProps", {})
                    .get("searchResults", {})
                    .get("gigs", [])
                )

                for gig in gigs_data:
                    parsed = {
                        "title": gig.get("title"),
                        "seller": gig.get("seller", {}).get("username"),
                        "seller_level": gig.get("seller", {}).get("level"),
                        "seller_country": gig.get("seller", {}).get("country"),
                        "price": gig.get("price"),
                        "currency": gig.get("currency"),
                        "rating": gig.get("rating"),
                        "reviews_count": gig.get("reviews_count"),
                        "delivery_time": gig.get("delivery_time"),
                        "gig_url": f"https://www.fiverr.com{gig.get('url', '')}",
                    }
                    all_gigs.append(parsed)

            except (json.JSONDecodeError, TypeError, KeyError):
                continue

        # Fallback: parse HTML directly if JSON isn't available
        if not all_gigs:
            gig_cards = soup.select('.gig-card-layout')
            for card in gig_cards:
                gig = {}

                title_el = card.select_one('.text-display-7')
                gig["title"] = title_el.get_text(strip=True) if title_el else None

                seller_el = card.select_one('.seller-name')
                gig["seller"] = seller_el.get_text(strip=True) if seller_el else None

                price_el = card.select_one('.price')
                gig["price"] = price_el.get_text(strip=True) if price_el else None

                rating_el = card.select_one('.rating-score')
                gig["rating"] = rating_el.get_text(strip=True) if rating_el else None

                link_el = card.select_one('a[href*="/"]')
                gig["gig_url"] = f"https://www.fiverr.com{link_el['href']}" if link_el else None

                if gig["title"]:
                    all_gigs.append(gig)

        print(f"Page {page}: Found {len(all_gigs)} gigs total")
        time.sleep(random.uniform(3, 6))

    return all_gigs

# Usage
gigs = scrape_fiverr_search("web+scraping", pages=3)
print(f"nFound {len(gigs)} gigs")
for g in gigs[:5]:
    print(f"  {g['title'][:60]} — ${g['price']} — ★{g['rating']}")

Step 3: Scraping Individual Gig Pages

To get detailed gig information including all pricing tiers:

def scrape_gig_details(gig_url):
    """Extract detailed data from a single Fiverr gig page."""

    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }

    response = requests.get(gig_url, headers=headers, timeout=30)
    soup = BeautifulSoup(response.text, "html.parser")

    details = {}

    # Try extracting from embedded JSON first
    for script in soup.select('script[id="__NEXT_DATA__"]'):
        try:
            data = json.loads(script.string)
            gig_data = data.get("props", {}).get("pageProps", {}).get("gig", {})

            details["title"] = gig_data.get("title")
            details["description"] = gig_data.get("description")
            details["category"] = gig_data.get("category", {}).get("name")
            details["subcategory"] = gig_data.get("sub_category", {}).get("name")

            # Pricing tiers
            packages = gig_data.get("packages", [])
            for pkg in packages:
                tier = pkg.get("title", "unknown").lower()
                details[f"price_{tier}"] = pkg.get("price")
                details[f"delivery_{tier}"] = pkg.get("delivery_time")
                details[f"revisions_{tier}"] = pkg.get("revisions")
                details[f"description_{tier}"] = pkg.get("description")

            # Seller info
            seller = gig_data.get("seller", {})
            details["seller_username"] = seller.get("username")
            details["seller_level"] = seller.get("level")
            details["seller_country"] = seller.get("country")
            details["seller_response_time"] = seller.get("response_time")
            details["seller_member_since"] = seller.get("member_since")

            # Stats
            details["rating"] = gig_data.get("rating")
            details["reviews_count"] = gig_data.get("reviews_count")
            details["orders_in_queue"] = gig_data.get("orders_in_queue")

        except (json.JSONDecodeError, TypeError):
            continue

    return details

# Usage
gig = scrape_gig_details("https://www.fiverr.com/someuser/some-gig")
print(json.dumps(gig, indent=2))

Step 4: Scraping Seller Reviews

Reviews are critical for understanding service quality:

def scrape_gig_reviews(gig_url, pages=3):
    """Scrape reviews for a specific Fiverr gig."""

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }

    all_reviews = []

    for page in range(1, pages + 1):
        url = f"{gig_url}?page={page}#reviews"

        response = requests.get(url, headers=headers, timeout=30)
        soup = BeautifulSoup(response.text, "html.parser")

        review_items = soup.select('.review-item')

        for item in review_items:
            review = {}

            # Reviewer name and country
            name_el = item.select_one('.reviewer-name')
            review["reviewer"] = name_el.get_text(strip=True) if name_el else None

            country_el = item.select_one('.reviewer-country')
            review["country"] = country_el.get_text(strip=True) if country_el else None

            # Star rating
            stars = item.select('.star-rating .filled')
            review["stars"] = len(stars) if stars else None

            # Review text
            text_el = item.select_one('.review-description')
            review["text"] = text_el.get_text(strip=True) if text_el else None

            # Date
            date_el = item.select_one('.review-date')
            review["date"] = date_el.get_text(strip=True) if date_el else None

            # Price paid (sometimes visible)
            price_el = item.select_one('.review-price')
            review["price_paid"] = price_el.get_text(strip=True) if price_el else None

            if review["text"]:
                all_reviews.append(review)

        time.sleep(random.uniform(2, 4))

    return all_reviews

Step 5: Category Browsing

To map the entire Fiverr marketplace structure:

def scrape_category(category_slug, subcategory_slug=None, pages=2):
    """Browse Fiverr by category to find top gigs."""

    if subcategory_slug:
        url = f"https://www.fiverr.com/categories/{category_slug}/{subcategory_slug}"
    else:
        url = f"https://www.fiverr.com/categories/{category_slug}"

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36",
    }

    response = requests.get(url, headers=headers, timeout=30)
    soup = BeautifulSoup(response.text, "html.parser")

    # Extract subcategories if on main category page
    subcategories = []
    for link in soup.select('a[href*="/categories/"]'):
        href = link.get("href", "")
        text = link.get_text(strip=True)
        if text and category_slug in href:
            subcategories.append({"name": text, "url": href})

    return {
        "subcategories": subcategories,
        "url": url,
    }

# Example: browse programming category
cats = scrape_category("programming-tech", "web-scraping")

Handling Anti-Bot Protection

Fiverr uses Cloudflare and custom anti-bot measures. Here’s what you’ll face:

1. Cloudflare challenges — JavaScript challenges that block simple HTTP requests
2. Rate limiting — aggressive throttling after repeated requests
3. Browser fingerprinting — detection of automated browsers

For scraping at scale, you’ll need a scraping API that handles these challenges. ScraperAPI manages proxy rotation, CAPTCHA solving, and browser rendering automatically:

SCRAPER_API_KEY = "your_api_key"

def scrape_with_api(url):
    """Use ScraperAPI to handle anti-bot protection."""
    api_url = f"http://api.scraperapi.com?api_key={SCRAPER_API_KEY}&url={url}&render=true"

    response = requests.get(api_url, timeout=60)

    if response.status_code == 200:
        return BeautifulSoup(response.text, "html.parser")
    else:
        print(f"ScraperAPI error: {response.status_code}")
        return None

# Use it in place of direct requests
soup = scrape_with_api("https://www.fiverr.com/search/gigs?query=data+scraping")

This is significantly easier than managing your own proxy infrastructure, especially for Cloudflare-protected sites like Fiverr.

Building a Market Research Dataset

Here’s how to combine everything into a market research pipeline:

def build_market_dataset(queries, output_file="fiverr_market_data.csv"):
    """Build a dataset of gig listings across multiple search terms."""

    all_gigs = []

    for query in queries:
        print(f"nSearching for: {query}")
        gigs = scrape_fiverr_search(query, pages=2)

        for gig in gigs:
            gig["search_query"] = query
            all_gigs.append(gig)

        time.sleep(random.uniform(5, 10))

    # Save to CSV
    if all_gigs:
        keys = all_gigs[0].keys()
        with open(output_file, "w", newline="", encoding="utf-8") as f:
            writer = csv.DictWriter(f, fieldnames=keys)
            writer.writeheader()
            writer.writerows(all_gigs)

        print(f"nSaved {len(all_gigs)} gigs to {output_file}")

    return all_gigs

# Research the web scraping niche
queries = [
    "web+scraping",
    "data+scraping",
    "web+crawler",
    "scraping+bot",
    "api+scraping",
]

dataset = build_market_dataset(queries)

Practical Use Cases

What can you actually do with Fiverr data?

• Pricing research — understand market rates for any service category
• Competitor analysis — see how top sellers position their gigs
• Trend monitoring — track which services are growing in demand
• Quality assessment — analyze review patterns to find reliable sellers
• Niche discovery — find underserved categories with high demand

Limitations and Honest Assessment

1. Cloudflare is tough. Plain requests will get blocked quickly. You need either a scraping API or Playwright with stealth plugins.
2. Fiverr’s frontend changes often. The __NEXT_DATA__ JSON structure can change between deployments. Build your parser to handle missing fields gracefully.
3. No official API for this data. Fiverr’s API is only for sellers managing their own gigs. There’s no public API for browsing the marketplace.
4. Rate limit strictly. Fiverr will ban your IP fast if you’re aggressive. Keep delays between requests at 3+ seconds minimum.
5. Respect the platform. Don’t scrape personal data, don’t spam sellers, and don’t use the data for anything that violates Fiverr’s Terms of Service.

Pre-Built Solutions

If you’d rather skip the scraping infrastructure entirely, there are pre-built Fiverr scrapers available on platforms like Apify. These handle anti-bot protection, proxy rotation, and data formatting out of the box — just configure your search parameters and get structured JSON output.

Wrapping Up

Fiverr scraping is a great way to understand the freelance marketplace at scale. The combination of __NEXT_DATA__ JSON extraction and a scraping API like ScraperAPI for anti-bot handling gives you a reliable pipeline.

Start small with a single search query, verify your selectors work, then scale up gradually. The freelance economy generates fascinating data — pricing trends, demand shifts, and service quality patterns that aren’t visible from casual browsing.

Got questions about scraping other freelance platforms? Drop a comment below.