Check out this Pen I made!

Web Scraping for Beginners: Sell Data as a Service

As a developer, you’re likely aware of the power of web scraping in extracting valuable data from websites. But have you ever considered selling this data as a service? In this article, we’ll walk you through the steps of web scraping for beginners and explore the monetization opportunities that come with it.

Step 1: Choose Your Target Website

The first step in web scraping is to identify the website you want to scrape. This could be a website that provides valuable data such as stock prices, weather forecasts, or social media metrics. For this example, let’s use the website https://www.example.com.

Step 2: Inspect the Website’s HTML Structure

To scrape a website, you need to understand its HTML structure. Use your browser’s developer tools to inspect the website’s HTML elements. Identify the elements that contain the data you want to scrape. For example, if you want to scrape the website’s headings, you would look for <h1>, <h2>, etc. elements.

Step 3: Write Your Web Scraping Code

Now it’s time to write your web scraping code. You can use a programming language like Python or JavaScript to write your scraper. For this example, we’ll use Python with the requests and BeautifulSoup libraries.

import requests
from bs4 import BeautifulSoup

# Send a GET request to the website
url = "https://www.example.com"
response = requests.get(url)

# If the GET request is successful, the status code will be 200
if response.status_code == 200:
    # Get the content of the response
    page_content = response.content

    # Create a BeautifulSoup object and specify the parser
    soup = BeautifulSoup(page_content, 'html.parser')

    # Find all the headings on the webpage
    headings = soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5', 'h6'])

    # Print out the headings
    for heading in headings:
        print(heading.text)

Step 4: Handle Anti-Scraping Measures

Some websites have anti-scraping measures in place to prevent bots from scraping their data. These measures can include CAPTCHAs, rate limiting, and IP blocking. To handle these measures, you can use techniques such as:

• Rotating user agents to make your scraper look like a real browser
• Adding delays between requests to avoid rate limiting
• Using a proxy service to rotate IP addresses

python
import requests
from bs4 import BeautifulSoup
import random
import time

# List of user agents
user_agents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3',
    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0'
]

# Choose a random user agent
user_agent = random.choice(user_agents)

# Send a GET request to the website
url = "https://www.example.com"
headers = {'User-Agent': user_agent}
response = requests.get(url, headers=headers)

# If the GET request is successful, the status code will be 200
if response.status_code == 200:
    # Get the content of the response
    page_content = response.content

    # Create a BeautifulSoup object and specify the

Qdrant is a vector database built for AI applications — semantic search, recommendation systems, and RAG pipelines. REST and gRPC APIs with filtering and payload storage.

Setup

docker run -p 6333:6333 qdrant/qdrant

Create Collection

const response = await fetch('http://localhost:6333/collections/products', {
  method: 'PUT',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    vectors: { size: 384, distance: 'Cosine' }
  })
});

Index Documents with Embeddings

import { QdrantClient } from '@qdrant/js-client-rest';

const client = new QdrantClient({ url: 'http://localhost:6333' });

// Upsert points with vectors and payload
await client.upsert('products', {
  points: [
    {
      id: 1,
      vector: await getEmbedding('Wireless bluetooth headphones with noise cancellation'),
      payload: { name: 'Headphones', price: 89.99, category: 'audio' }
    },
    {
      id: 2,
      vector: await getEmbedding('Mechanical keyboard with Cherry MX switches'),
      payload: { name: 'Keyboard', price: 129.99, category: 'peripherals' }
    }
  ]
});

Semantic Search

const queryVector = await getEmbedding('good headphones for music');

const results = await client.search('products', {
  vector: queryVector,
  limit: 5,
  filter: {
    must: [{ key: 'price', range: { lte: 100 } }]
  },
  with_payload: true
});
// Returns headphones (semantically similar) under $100

Recommendation API

const recommendations = await client.recommend('products', {
  positive: [1, 5],   // IDs user liked
  negative: [3],       // IDs user disliked
  limit: 10,
  filter: { must: [{ key: 'category', match: { value: 'audio' } }] }
});

REST API

# Search
curl -X POST http://localhost:6333/collections/products/points/search 
  -d '{"vector":[0.1,0.2,...],"limit":5,"filter":{"must":[{"key":"category","match":{"value":"audio"}}]}}'

# Get collection info
curl http://localhost:6333/collections/products

# Scroll through points
curl -X POST http://localhost:6333/collections/products/points/scroll 
  -d '{"limit":10,"with_payload":true}'

Why This Matters

• Built for AI: First-class support for embeddings and RAG
• Filtering: Combine vector search with exact filters
• Payload storage: Store metadata alongside vectors
• Fast: Written in Rust, optimized for billion-scale

Need custom AI search or vector database tools? I build developer tools. Check out my web scraping actors on Apify or reach out at [email protected] for custom solutions.

Today I worked on a wallet system like GPay or PhonePe, where users can send money to each other. The main focus was Atomicity from ACID properties.

Atomicity means “all or nothing”.
If a transaction fails in between, everything should go back to the original state.

Transaction
I wrote a transaction like this

BEGIN;
UPDATE accounts
SET balance = balance - 200
WHERE name = 'Alice';
UPDATE accounts
SET balance = balance + 200
WHERE name = 'Bob';
COMMIT;

Now Testing Failure

I intentionally broke the second query (credit to Bob).

BEGIN;

UPDATE accounts
SET balance = balance - 200
WHERE name = 'Alice';
UPDATE accounts
SET balancee = balance + 200
WHERE name = 'Bob';
COMMIT;

BEGIN – start transaction
Do all operations
If success – COMMIT
If error – ROLLBACK

Atomicity ensures no partial transactions, which is very important in payment systems.

This is a submission for the Notion MCP Challenge

What I Built

I built Eco-Loom, a multi-facility sustainability operations workspace that helps teams track carbon emissions without drowning in spreadsheets, scattered documents, and disconnected reporting tools.

The idea came from a simple frustration:

carbon accounting is often treated like a reporting exercise at the end of the month—when it should feel like an operational system teams can use every day.

Eco-Loom turns that process into a live workflow.

Teams can:

• Capture emissions data
• Manage a carbon ledger
• Compare facilities and sites
• Monitor monthly trends
• Track sustainability goals
• Review gap analysis
• Generate report-ready outputs

All from a single workspace.

The product is designed around visibility and action.

Instead of asking:

“Where did this number come from?”

“Which site is off track?”

Eco-Loom makes those answers visible immediately.

🎥 Video Demo

Try it yourself:

👉 Live Demo Link

Demo Login Credentials:

• Email: [email protected]
• Password: eco-loom-local-admin

Video walkthrough:

In the demo, I show how a team can:

• Log into Eco-Loom and land in a facility-aware sustainability dashboard
• Switch between sites to see how emissions, trends, and goal health change
• Review emissions by category, gap analysis, and recent activity
• Manage carbon-ledger records and see how data feeds reporting workflows
• Generate reports with facility context included
• Use the Notion-connected workflow as a collaboration and review layer

💻 Show Us the Code

A key part of the implementation is the company-scoped Notion configuration layer.

Since Eco-Loom is multi-workspace and multi-facility, Notion cannot be treated as a single global integration.

export async function getCompanyNotionConfig(
  companyId: string,
  options: CompanyNotionConfigOptions = {}
) {
  const { createIfMissing = false, adoptLegacy = true } = options;

  const companyConfig = await db.notionConfig.findUnique({
    where: { companyId },
  });

  if (companyConfig) {
    return ensureEncryptedNotionApiKey(companyConfig);
  }

  if (createIfMissing) {
    return db.notionConfig.upsert({
      where: { companyId },
      update: {},
      create: {
        companyId,
        syncEnabled: true,
      },
    });
  }

  return null;
}

That small layer matters because it:

• Keeps each workspace isolated
• Makes Notion sync safer
• Supports a more production-ready sustainability workflow

🔗 How I Used Notion MCP

Notion MCP is one of the most important parts of Eco-Loom.

I used it to connect Eco-Loom with structured Notion databases for:

• Carbon Ledger
• Sustainability Goals
• Reports
• Facilities / Sites
• Operations follow-up

This means Notion is not just a note-taking add-on—it becomes the collaboration layer around the product.

With MCP, Eco-Loom can:

• Sync records into Notion
• Preserve workspace context
• Power a custom sustainability operations agent that:
  • Reviews data quality
  • Flags missing fields
  • Detects duplicate risks
  • Prepares monthly follow-ups

⚙️ What This Unlocks

This creates a much stronger workflow:

• Eco-Loom → structured sustainability system of record
• Notion → human collaboration and review environment
• MCP → bridge connecting data, workflows, and reporting

All together, they form a closed operational loop.

🚀 Final Thoughts

In short, Notion MCP helped turn Eco-Loom from a dashboard into a true sustainability operating system.

If you’ve ever opened LeetCode, stared at a problem, and immediately closed your laptop — you’re not alone. Grinding algorithm problems in isolation is brutal. No stakes, no community, no reward loop.

That’s why I built Algoora (https://algoora.vercel.app/) — a competitive coding platform designed to be the place you actually want to show up every day.

What Is Algoora?

Algoora is a free web-based competitive coding platform where developers solve algorithmic problems, compete in live tournaments, collaborate with teams, and earn real rewards for their progress. Built for everyone from curious beginners to engineers prepping for FAANG interviews.

Think LeetCode meets a competitive game — with XP, battle passes, cosmetic unlocks, real-time team sessions, and live competitions that make every session feel like it matters.

Core Features

🧩 Problem Library — Solo & Team Modes

Four difficulty tiers: Easy, Medium, Hard, and Extreme. What makes it different is Team Mode — certain problems are designed for two people. One takes the Coder role (writes the solution), the other takes the Explainer role (guides the logic out loud). Real pair programming, built into the platform.

Every problem includes:

• Hints and full editorials
• Time and space complexity breakdowns
• Company tags (know which firms ask what)
• Similar problems for topic-based drilling
• Video solutions on selected problems

⚔️ Live Competitions

Real-time coding tournaments with a live arena, global leaderboard, and time pressure. Race other developers, earn ranking rewards, and get that adrenaline hit of solving a Hard problem with 90 seconds left on the clock.

👥 Teams

Create or join a team, build your roster, and compete together. Teams have dashboards, shared stats, and WebSocket-powered live coding sessions where both teammates see every change in real time.

📈 XP, Levels & Streaks

Every solved problem gives XP. Daily challenges extend your streak. Level up and let your profile show the world how hard you grind.

🏆 Achievements & Badges

A full achievement system — “Solve 100 problems,” “Win your first competition,” “Maintain a 30-day streak.” Badges display publicly on your profile as proof of skill.

🎮 Coding Pass (Battle Pass)

A seasonal battle pass inspired by games like Valorant. Complete challenges, earn XP, unlock cosmetic rewards tier by tier. Premium pass holders get exclusive drops.

🛍️ Shop & Cosmetics

Earn coins by solving problems. Spend them on:

• Profile themes
• Custom borders and avatar frames
• Username effects
• Card effects

Your profile is your identity — make it yours.

🤖 AI-Powered Error Explanations

When your code fails, the AI doesn’t just show you “Wrong Answer.” It explains why in plain English — what went wrong conceptually and how to think about fixing it. Massive for learners still building their debugging intuition.

🔐 Two-Factor Authentication

TOTP-based 2FA built in. Your account is properly protected.

💎 Free + Premium Tiers

The entire core platform is free. Premium and Elite tiers unlock premium problems and exclusive cosmetics — purchasable with a subscription or with coins you earn in-app.

Who Is Algoora For?

Interview preppers — company tags, editorials, and time/space complexity on every problem.

Competitive programmers — real-time tournaments and global leaderboards.

Developer teams — team coding sessions and collaborative problem solving.

Beginners — hints, AI error explanations, and a gamified progression that makes the learning curve less terrifying.

Anyone who gets bored grinding alone — competitions, streaks, cosmetics, and a battle pass give you reasons to come back every day.

Algoora vs. LeetCode vs. HackerRank

The Stack

React + TypeScript (Vite) · Tailwind CSS · Node.js · MongoDB · WebSockets · Vercel + Render

Try It Free

No credit card. No friction. Full access to problems, competitions, and team features the moment you sign up.

👉 https://algoora.vercel.app/

Why are 80% of engineering organisations abandoning traditional DevOps for Platform Engineering?

The answer starts at Spotify — and it might change how you think about software infrastructure.

The Spotify Origin Story

Around 2018, Spotify faced a critical problem. As they grew from a startup to thousands of engineers, their infrastructure became fragmented. Developers weren’t building features — they were drowning.

Instead of shipping code, engineers spent hours asking:

• 🔍 “Where’s the API for that service?”
• 📦 “What framework version is everyone using?”
• 👤 “Who owns this broken service?”
• 📚 “Where’s the documentation?”

Context switching and cognitive overload were killing productivity.

The Solution: Backstage

Spotify’s answer was revolutionary: create an abstraction layer on top of all infrastructure tooling.

They called it Backstage — a developer portal powered by a centralised software catalog. One place for everything, accessible to everyone.

Here’s the impact:

• ✅ Open-sourced in 2020
• ✅ Now a CNCF incubating project
• ✅ 100+ adopters (Netflix, American Airlines, IKEA)

But Backstage was just the beginning of something much bigger.

What Exactly Is Platform Engineering?

Think of it as building an Internal Developer Platform (IDP) — a self-service layer that sits between your developers and your infrastructure complexity.

The Three-Layer Architecture

Layer 1: Developer Portal
Your single pane of glass. This is where developers go to discover services, read documentation, and trigger deployments. Tools: Backstage, Port, Cortex.

Layer 2: Service Catalog
The registry of everything. Every API, service, database, and deployment — catalogued and searchable. No more Slack archaeology.

Layer 3: Infrastructure Automation
The magic layer. Using tools like Terraform, Crossplane, or Kratix, you define golden paths — opinionated, pre-approved workflows that let developers deploy without tickets.

DevOps vs Platform Engineering

DevOps said “you build it, you run it.” Platform Engineering says “you build it, we make running it easy.”

The Tools Landscape

Developer Portals

• Backstage — Plugin ecosystem, CNCF-backed, most community support
• Port — Self-service catalogs, policy enforcement, GitOps native
• Cortex — Service reliability focus, scorecards, ownership tracking

Infrastructure Automation

• Crossplane — Kubernetes-native infrastructure as code
• Kratix — Platform-as-a-Product framework
• Humanitec — Commercial IDP with dynamic config management

The Key Insight

These aren’t just pretty UIs — they’re opinionated workflows. The platform team decides the “right way” to deploy, and developers follow golden paths instead of reinventing wheels.

When NOT to Use Platform Engineering

Platform Engineering isn’t a silver bullet.

❌ Skip it if:

• You have a small team (<20 engineers)
• Your infrastructure is already simple
• You don’t have dedicated platform resources
• You’re trying to “boil the ocean” (start small!)

✅ Consider it if:

• Developers spend >30% of time on non-feature work
• Onboarding takes weeks, not days
• You have multiple teams reinventing the same wheels
• Compliance/security requirements are complex

The Numbers

According to Gartner:

• 80% of large engineering orgs will adopt Platform Engineering by 2026
• 60% of Kubernetes enterprises are already forming platform teams
• 40% average reduction in deployment time after IDP adoption

Getting Started

1. Audit cognitive load — Where do developers waste time?
2. Pick ONE golden path — Don’t boil the ocean
3. Start with a portal — Backstage is free and extensible
4. Measure developer experience — Track DORA metrics + satisfaction

Watch the Full Video

I cover all of this in more depth — including architecture diagrams, real-world examples, and the trade-offs nobody talks about:

💬 Have you tried Platform Engineering? What tools are you using? Drop your experience in the comments!

Follow me for more deep dives on software architecture, DevOps evolution, and the tools shaping modern engineering.

AI-Driven Code Review: How I Reduced Bug Rate by 65%

Code review is broken.

We spend hours staring at PRs, catching the same bugs over and over. We miss critical issues because we’re tired or distracted. We delay releases because reviews take too long.

I know this because I lived it.

For 90 days, I tracked every code review I did. The data was sobering:

• Average review time: 45 minutes
• Bugs caught in review: 15%
• Bugs that made it to production: 85%
• Team satisfaction with reviews: 3/10

Then I implemented AI-driven code review. The results:

• Average review time: 22 minutes (50% reduction)
• Bugs caught in review: 65% (333% improvement)
• Bugs that made it to production: 35% (59% reduction)
• Team satisfaction: 8/10 (167% improvement)

Here’s exactly how I did it.

The Problem: Why Code Review Fails

1. Human Limitations

We’re not machines. We get tired. We get distracted. We miss things.

The data:

• Reviews after 4 PM: 40% less effective
• Reviews on Fridays: 35% less effective
• Reviews after 10 PRs in a day: 50% less effective

2. Inconsistency

Different reviewers catch different things. Style guides help, but they’re not enough.

The data:

• Reviewer A catches 20% of bugs
• Reviewer B catches 25% of bugs
• Reviewer C catches 18% of bugs
• Combined: Only 35% of bugs caught (overlap is real)

3. Scale Issues

As teams grow, reviews get slower. More code, more reviewers, more delays.

The data:

• 2-person team: 30-minute average review
• 5-person team: 45-minute average review
• 10-person team: 60-minute average review

The Solution: AI-Driven Code Review

I implemented a 3-layer AI review system:

Layer 1: Pre-Review Analysis

Tool: Custom AI Scripts + Static Analysis

Before a human ever sees the PR, AI analyzes it:

• Code quality checks: Style, complexity, duplication
• Security scans: Vulnerabilities, secrets, dependencies
• Performance analysis: Potential bottlenecks, memory leaks
• Test coverage: Gaps in test coverage

Results:

• 40% of issues caught before human review
• Review time reduced by 30%
• Consistent quality across all PRs

Layer 2: AI-Assisted Human Review

Tool: GitHub Copilot + Custom Prompts

When humans review, AI assists:

• Suggested improvements: “Consider using X instead of Y”
• Contextual explanations: “This pattern might cause Z”
• Risk assessment: “High risk change, needs extra attention”
• Automated comments: “Missing error handling here”

Results:

• Review quality improved by 50%
• Reviewer confidence increased by 40%
• Knowledge transfer improved by 60%

Layer 3: Post-Review Learning

Tool: AI Analytics + Feedback Loop

After reviews, AI learns:

• Pattern recognition: “This team often misses X”
• Trend analysis: “Bug rate increased after Y change”
• Reviewer insights: “Reviewer A is good at catching Z”
• Process optimization: “Reviews take longer on Fridays”

Results:

• Continuous improvement
• Personalized reviewer training
• Process optimization over time

The 14-Day Implementation Plan

Days 1-3: Setup and Baseline

Day 1: Measure Your Current State

• Track review time for 10 PRs
• Count bugs caught vs. bugs missed
• Survey team satisfaction

Day 2: Choose Your Tools

• Static analysis: SonarQube (free)
• AI assistance: GitHub Copilot ($10/mo)
• Custom scripts: Python + OpenAI API

Day 3: Set Up Pre-Review Analysis

• Configure static analysis
• Write custom AI scripts
• Test on sample PRs

Days 4-7: Implement AI-Assisted Review

Day 4: Create Review Templates

• Standardize review criteria
• Write AI prompts for common issues
• Create risk assessment framework

Day 5: Train Your Team

• Show how AI assistance works
• Explain when to trust AI vs. when to question it
• Set expectations for review quality

Day 6: Pilot with One Team

• Start with a small team
• Monitor results closely
• Gather feedback

Day 7: Review and Adjust

• Compare with baseline
• Adjust tools and processes
• Plan for rollout

Days 8-14: Rollout and Optimize

Day 8-10: Gradual Rollout

• Add one team at a time
• Provide support and training
• Monitor metrics

Day 11-13: Optimize and Scale

• Fine-tune AI prompts
• Automate more checks
• Expand to all teams

Day 14: Measure and Celebrate

• Compare with Day 1 baseline
• Share results with the team
• Plan next improvements

The Tools I Use

Total setup time: 13 hours
Total monthly cost: $40
ROI: 50x (based on time saved)

The Metrics That Matter

Before AI-Driven Review

• Average review time: 45 minutes
• Bugs caught in review: 15%
• Bugs in production: 85%
• Team satisfaction: 3/10

After AI-Driven Review

• Average review time: 22 minutes (50% reduction)
• Bugs caught in review: 65% (333% improvement)
• Bugs in production: 35% (59% reduction)
• Team satisfaction: 8/10 (167% improvement)

Financial Impact

• Time saved: 23 minutes/review × 100 reviews/month = 38 hours/month
• Value of time saved: 38 hours × $50/hr = $1,900/month
• Cost of tools: $40/month
• Net savings: $1,860/month
• Annual savings: $22,320

Common Pitfalls to Avoid

1. Over-Relying on AI

AI makes mistakes. Always review AI suggestions.

Rule of thumb: Trust AI for 80% of issues, verify the other 20%.

2. Ignoring Team Feedback

Your team knows your codebase better than any AI.

Best practice: Let reviewers override AI suggestions.

3. Not Customizing for Your Context

Generic AI prompts won’t work for every team.

Best practice: Customize prompts for your coding style and patterns.

4. Forgetting About Security

AI might miss security issues.

Best practice: Always run security scans separately.

The Results

After 90 days of using AI-driven code review:

Quality Improvements

• Bug rate: Down 65%
• Code coverage: Up 40%
• Technical debt: Down 30%

Efficiency Improvements

• Review time: Down 50%
• Time to merge: Down 40%
• Release frequency: Up 60%

Team Improvements

• Satisfaction: Up 167%
• Knowledge sharing: Up 80%
• Onboarding time: Down 50%

Business Impact

• Production bugs: Down 59%
• Customer complaints: Down 45%
• Development velocity: Up 70%

Your Turn

You don’t need to be an AI expert. You don’t need a huge budget. You just need to start.

Pick one layer of AI-driven review. Implement it this week.

Then come back and tell me how much time you saved.

Resources

Affiliate Disclosure: Some links below are affiliate links. If you purchase through these links, I may earn a commission at no extra cost to you.

• GitHub Copilot – AI-powered code review assistance
• SonarQube – Static code analysis (free)
• OpenAI API – Build custom AI tools
• GitHub Actions – Automate your review pipeline

Want to learn more? Check out my other articles on AI-powered development workflows.

Have questions? Drop them in the comments below. I reply to every comment.