I spent a week trying to scrape YellowPages.com with Python. Most tutorials online are from 2021-2023 and silently fail now. Here’s what I found.

The tutorials are all broken

If you Google “scrape yellowpages python”, you’ll find guides using requests + BeautifulSoup. They look clean, they make sense, and they don’t work.

import requests
from bs4 import BeautifulSoup

url = "https://www.yellowpages.com/search?search_terms=plumbers&geo_location_terms=Austin%2C+TX"
resp = requests.get(url)
# resp.status_code == 403. Every time.

YellowPages.com moved behind Cloudflare sometime in 2023. Every request now passes through a JavaScript challenge. requests can’t execute JavaScript, so it gets a 403 or an empty challenge page. Same with httpx, urllib3, or any pure HTTP library.

What about Selenium/Playwright?

Headless browsers can execute the JS challenge:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://www.yellowpages.com/search?search_terms=plumbers&geo_location_terms=Austin%2C+TX")
    page.wait_for_selector(".search-results")
    html = page.content()

This works… sometimes. Three problems kill it at scale:

1. IP blocks. Default Chromium uses your IP. After 10-20 requests, Cloudflare blocks you. You need residential proxies, and specifically US ones — non-US IPs get rejected.

2. Browser fingerprinting. Headless Chrome has detectable fingerprint differences. Cloudflare catches navigator.webdriver=true, missing plugins, and other signals. You need stealth patches.

3. Session reuse kills extraction. If you navigate to 50 business detail pages in one browser session, Cloudflare starts returning challenge pages instead of data after the 8th or 9th page. You need fresh browser contexts per page.

The proxy problem

Even with Playwright + stealth, you need proxies. Not just any proxies — residential US proxies specifically.

Free proxy lists? Dead within hours. Datacenter proxies? Blocked instantly. Shared residential proxies? Rate-limited because 100 other scrapers are on the same pool.

You’ll spend $50-100/month on a residential proxy service just to keep the scraper running. Then you need rotation logic, error handling for burned IPs, and retry logic.

What I actually use now

After fighting this for a week, I switched to using a pre-built scraper that handles all of this: Yellow Pages Scraper on Apify.

It handles:

• Cloudflare bypass with residential proxies (built in)
• Stealth browser automation (no fingerprint detection)
• Fresh browser sessions per detail page
• Automatic retries on blocked requests
• Structured output (JSON, CSV, Excel)

You put in search terms + locations and get back clean data:

{
  "name": "Joe's Plumbing LLC",
  "phone": "(512) 555-0142",
  "email": "[email protected]",
  "website": "https://joesplumbing.com",
  "address": "1234 Main St, Austin, TX 78701",
  "rating": 4.5,
  "reviewCount": 47,
  "categories": ["Plumbers", "Water Heaters"],
  "yearsInBusiness": 12
}

Cost: about $0.005 per result. A run of 100 businesses costs $0.60. Cheaper than maintaining your own proxy subscription.

When to roll your own

Building your own scraper makes sense if:

• You need custom post-processing that can’t be done after export
• You’re scraping at massive scale (10K+ results/day) and want to optimize costs
• You need real-time streaming rather than batch results
• You want to learn how browser automation and anti-bot bypass work

If you just need business leads from YellowPages, the pre-built tool saves you a week of proxy debugging.

Key technical lessons if you do build it yourself

1. Use Go or Playwright, not Selenium. Selenium is slower and has more detectable fingerprints.
2. Residential US proxies are mandatory. Budget $50-100/month minimum.
3. Fresh browser context per detail page. Session reuse drops email extraction from ~22% to ~3%.
4. Parse JSON-LD, not HTML. YellowPages embeds structured data as application/ld+json. It’s cleaner and more stable than CSS selectors.
5. Respect rate limits. 1-3 seconds between requests. Faster than that triggers immediate blocks.

I’m a data engineer building automation tools for lead generation. If you have questions about scraping public business directories, drop a comment.

When working with big data analytics in GBase 8a, you’ve probably noticed something strange: the same SQL query can be incredibly fast sometimes, yet painfully slow at others. The key difference often comes down to whether you’re leveraging Rough Index — a lightweight indexing mechanism built into the GBase 8a columnar storage engine.

In this article, I’ll walk you through what Rough Index is, how it works, and how to use it effectively to dramatically speed up your queries.

1. What Is Rough Index?
   Rough Index (RI) is a block‑level index that doesn’t store exact row positions like a B‑Tree. Instead, for each data block (typically 64KB or 128KB), it records the minimum and maximum values of a column within that block.

Example: Suppose the amount column in an orders table is split into 1,000 blocks. Rough Index might store:

Block 1: min=10, max=500
Block 2: min=501, max=1200
Block 3: min=1201, max=2500
…

Now when you run a query like WHERE amount > 2000, the GBase 8a optimizer scans the Rough Index first. Any block whose maximum value is ≤ 2000 is skipped entirely — no actual data is read. This is called block pruning.

When data is well‑ordered, Rough Index can prune 90% or more of I/O operations, making your queries 5 to 10 times faster with no extra hardware.

1. How to Check If Rough Index Is Working
   Use EXPLAIN to View the Execution Plan

EXPLAIN SELECT SUM(amount) FROM orders WHERE create_time >= '2024-01-01';

If you see “Rough Index Scan” in the output, or the number of blocks_scanned is much smaller than the total block count, Rough Index is active.

Query the System View

SELECT * FROM information_schema.GBASE_ROUGH_INDEX 
WHERE table_name = 'orders';

This view shows which columns have Rough Index enabled and their current status.

1. How to Maximize Rough Index Effectiveness
   Tip 1: Keep Data Ordered During Import
   Rough Index pruning works best when data is sorted by the column you frequently filter on. If values are randomly distributed across blocks, each block’s min/max range becomes wide, and pruning becomes less effective.
   Best practice: Sort your input file by the filter column before loading.

LOAD DATA INFILE '/data/orders_sorted.csv' INTO TABLE orders
FIELDS TERMINATED BY ',' (order_id, user_id, amount, create_time);
-- Make sure the file is sorted by create_time before loading

Tip 2: Rebuild Rough Index After Updates
If your table undergoes many updates or out‑of‑order inserts, you can manually rebuild the index:

ALTER TABLE orders REBUILD ROUGH INDEX;

Tip 3: Use Range Filters for Best Results
Rough Index performs best with range predicates (>, <, BETWEEN). It works less well with IN lists or !=.
✅ Good: Range query

SELECT * FROM orders 
WHERE create_time BETWEEN '2024-01-01' AND '2024-03-31';

❌ Not ideal: Discrete value list

SELECT * FROM orders 
WHERE status IN (1, 3, 7, 15);

1. Summary
   Rough Index is a powerful, built‑in performance feature in the gbase database. By understanding its block‑pruning mechanism and designing your data loading and queries accordingly, you can achieve massive speedups without adding hardware.

If you’re using GBase 8a for analytical workloads, take a moment to check your Rough Index usage — it might be the secret to making your queries run 10x faster!

Self-Supervised Temporal Pattern Mining for satellite anomaly response operations for extreme data sparsity scenarios

Introduction: The Silent Satellite Problem

I remember the first time I encountered what we called “the silent satellite problem” during my research at the European Space Agency’s data lab. We were analyzing telemetry from a decade-old Earth observation satellite that had started exhibiting mysterious power fluctuations. The anomaly logs were sparse—sometimes weeks between meaningful events—and the labeled data was practically non-existent. Traditional supervised approaches failed spectacularly, with our best models achieving barely 40% precision on anomaly detection.

This experience fundamentally changed my approach to AI for space systems. While exploring alternative methodologies, I discovered that the most valuable patterns weren’t in the anomalies themselves, but in the temporal relationships between seemingly normal operations. Through studying recent advances in self-supervised learning, I realized we could treat the satellite’s entire operational history as a single, continuous learning signal, even with extreme data sparsity.

Technical Background: The Challenge of Sparse Space Data

Satellite anomaly response presents unique challenges that make conventional machine learning approaches inadequate:

1. Extreme Data Sparsity: Anomalies might occur only once every few months
2. High-Dimensional Temporal Data: Hundreds of telemetry channels with complex interdependencies
3. Missing Labels: Most anomalies are discovered post-facto, if at all
4. Non-Stationary Systems: Satellites age, components degrade, and operational patterns evolve

During my investigation of self-supervised approaches for time series, I found that contrastive learning methods could learn robust representations even with sparse positive examples. The key insight was that temporal consistency—how systems behave over time—provides a powerful supervisory signal without explicit labels.

Core Methodology: Temporal Contrastive Learning

The Temporal Consistency Principle

One interesting finding from my experimentation with satellite telemetry was that normal operations exhibit predictable temporal patterns, while anomalies break these patterns in subtle ways. We can leverage this through temporal contrastive learning:

import torch
import torch.nn as nn
import torch.nn.functional as F

class TemporalContrastiveEncoder(nn.Module):
    def __init__(self, input_dim=128, hidden_dim=256, output_dim=128):
        super().__init__()
        self.temporal_encoder = nn.LSTM(
            input_dim, hidden_dim,
            num_layers=3,
            bidirectional=True,
            dropout=0.2,
            batch_first=True
        )
        self.projection_head = nn.Sequential(
            nn.Linear(hidden_dim * 2, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, output_dim)
        )

    def forward(self, x, lengths):
        packed = nn.utils.rnn.pack_padded_sequence(
            x, lengths.cpu(),
            batch_first=True,
            enforce_sorted=False
        )
        outputs, (hidden, cell) = self.temporal_encoder(packed)
        hidden = torch.cat([hidden[-2], hidden[-1]], dim=1)
        return F.normalize(self.projection_head(hidden), dim=1)

class TemporalContrastiveLoss(nn.Module):
    def __init__(self, temperature=0.1):
        super().__init__()
        self.temperature = temperature

    def forward(self, z_i, z_j):
        """Normalized temperature-scaled cross entropy loss"""
        batch_size = z_i.size(0)
        labels = torch.arange(batch_size).to(z_i.device)

        # Compute similarity matrix
        features = torch.cat([z_i, z_j], dim=0)
        similarity_matrix = F.cosine_similarity(
            features.unsqueeze(1),
            features.unsqueeze(0),
            dim=2
        )

        # Mask for positive pairs
        mask = torch.eye(batch_size, dtype=torch.bool).to(z_i.device)
        mask = mask.repeat(2, 2)
        mask.fill_diagonal_(0)

        positives = similarity_matrix[mask].view(2 * batch_size, -1)
        negatives = similarity_matrix[~mask].view(2 * batch_size, -1)

        logits = torch.cat([positives, negatives], dim=1)
        logits /= self.temperature

        loss = F.cross_entropy(logits, labels.repeat(2))
        return loss

Through studying this approach, I learned that creating positive pairs through temporal augmentation (shifting, scaling, and masking time windows) allows the model to learn invariant representations of normal operational patterns.

Multi-Scale Temporal Pattern Mining

My exploration of satellite telemetry revealed that anomalies manifest at different temporal scales. Some appear as sudden spikes, while others develop gradually over weeks. This led me to develop a multi-scale architecture:

class MultiScaleTemporalMiner(nn.Module):
    def __init__(self, input_channels, scales=[1, 7, 30]):
        super().__init__()
        self.scales = scales
        self.encoders = nn.ModuleList([
            TemporalEncoder(input_channels, scale_length=s)
            for s in scales
        ])
        self.attention = nn.MultiheadAttention(
            embed_dim=256,
            num_heads=8,
            dropout=0.1,
            batch_first=True
        )
        self.anomaly_scorer = nn.Sequential(
            nn.Linear(256 * len(scales), 512),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, 1)
        )

    def forward(self, x):
        """Process multi-scale temporal patterns"""
        scale_features = []

        for scale, encoder in zip(self.scales, self.encoders):
            # Extract features at each temporal scale
            scaled_x = self._resample_temporal(x, scale)
            features = encoder(scaled_x)
            scale_features.append(features)

        # Cross-scale attention
        combined = torch.cat(scale_features, dim=1)
        attended, _ = self.attention(combined, combined, combined)

        # Anomaly scoring
        scores = self.anomaly_scorer(attended)
        return scores, attended

    def _resample_temporal(self, x, scale_factor):
        """Adaptive temporal resampling"""
        if scale_factor == 1:
            return x

        batch_size, seq_len, features = x.shape
        new_len = seq_len // scale_factor

        # Use adaptive pooling for temporal resampling
        x_reshaped = x.transpose(1, 2).contiguous()
        resampled = F.adaptive_avg_pool1d(x_reshaped, new_len)
        return resampled.transpose(1, 2).contiguous()

During my experimentation with this architecture, I observed that the attention mechanism learns to weight different temporal scales based on the anomaly type, providing interpretable insights into failure modes.

Implementation Details: Handling Extreme Sparsity

Synthetic Minority Oversampling for Time Series

One of the biggest challenges I encountered was the extreme class imbalance. While researching solutions, I developed a temporal-aware oversampling technique:

import numpy as np
from scipy.interpolate import CubicSpline

class TemporalSMOTE:
    def __init__(self, k_neighbors=5, synthetic_ratio=0.5):
        self.k = k_neighbors
        self.ratio = synthetic_ratio

    def generate_synthetic(self, minority_samples, timestamps):
        """Generate synthetic temporal sequences"""
        n_samples, seq_len, n_features = minority_samples.shape
        n_synthetic = int(n_samples * self.ratio)

        synthetic_samples = []
        synthetic_timestamps = []

        for _ in range(n_synthetic):
            # Randomly select two samples
            idx1, idx2 = np.random.choice(n_samples, 2, replace=False)

            # Temporal alignment using DTW distance
            aligned_sample = self._temporal_interpolation(
                minority_samples[idx1],
                minority_samples[idx2],
                timestamps[idx1],
                timestamps[idx2]
            )

            synthetic_samples.append(aligned_sample)

            # Generate synthetic timestamp
            synthetic_ts = self._interpolate_timestamp(
                timestamps[idx1],
                timestamps[idx2]
            )
            synthetic_timestamps.append(synthetic_ts)

        return np.array(synthetic_samples), np.array(synthetic_timestamps)

    def _temporal_interpolation(self, seq1, seq2, ts1, ts2):
        """Dynamic time warping interpolation"""
        # Simplified DTW-based interpolation
        n_features = seq1.shape[1]
        interpolated = np.zeros_like(seq1)

        for f in range(n_features):
            # Create cubic splines for both sequences
            cs1 = CubicSpline(ts1, seq1[:, f])
            cs2 = CubicSpline(ts2, seq2[:, f])

            # Interpolate at combined time points
            combined_ts = np.sort(np.unique(np.concatenate([ts1, ts2])))
            interp1 = cs1(combined_ts)
            interp2 = cs2(combined_ts)

            # Weighted combination
            alpha = np.random.random()
            interpolated[:, f] = alpha * interp1 + (1 - alpha) * interp2

        return interpolated

Through studying various oversampling techniques, I found that temporal-aware methods significantly outperformed traditional SMOTE for time series anomaly detection.

Uncertainty-Aware Anomaly Scoring

My research revealed that quantifying uncertainty is crucial for operational decision-making. I implemented a Bayesian approach:

import pyro
import pyro.distributions as dist
from pyro.infer import SVI, Trace_ELBO
from pyro.optim import Adam

class BayesianAnomalyDetector(nn.Module):
    def __init__(self, input_dim, hidden_dim=128):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )

    def model(self, x, y=None):
        # Priors
        w_prior = dist.Normal(torch.zeros_like(self.encoder[0].weight), 1.)
        b_prior = dist.Normal(torch.zeros_like(self.encoder[0].bias), 1.)

        priors = {'encoder.0.weight': w_prior, 'encoder.0.bias': b_prior}

        # Lift parameters
        lifted_module = pyro.random_module("module", self.encoder, priors)
        lifted_reg_model = lifted_module()

        with pyro.plate("data", x.shape[0]):
            # Forward pass
            hidden = lifted_reg_model(x)
            mean = hidden.mean(1, keepdim=True)

            # Observation model
            obs = dist.Normal(mean, 0.1)
            pyro.sample("obs", obs, obs=y)

    def guide(self, x, y=None):
        # Variational parameters
        w_mu = pyro.param("w_mu", torch.randn_like(self.encoder[0].weight))
        w_sigma = pyro.param("w_sigma", torch.ones_like(self.encoder[0].weight),
                           constraint=dist.constraints.positive)

        b_mu = pyro.param("b_mu", torch.randn_like(self.encoder[0].bias))
        b_sigma = pyro.param("b_sigma", torch.ones_like(self.encoder[0].bias),
                           constraint=dist.constraints.positive)

        # Variational distributions
        w_dist = dist.Normal(w_mu, w_sigma)
        b_dist = dist.Normal(b_mu, b_sigma)

        variational = {'encoder.0.weight': w_dist, 'encoder.0.bias': b_dist}

        lifted_module = pyro.random_module("module", self.encoder, variational)
        return lifted_module()

    def predict_with_uncertainty(self, x, n_samples=100):
        """Monte Carlo dropout for uncertainty estimation"""
        predictions = []

        for _ in range(n_samples):
            with torch.no_grad():
                pred = self.encoder(x)
                predictions.append(pred)

        predictions = torch.stack(predictions)
        mean_pred = predictions.mean(0)
        std_pred = predictions.std(0)

        return mean_pred, std_pred

While experimenting with Bayesian methods, I discovered that uncertainty estimates were particularly valuable for prioritizing anomaly investigations in operational scenarios.

Real-World Applications: Satellite Operations Center Integration

Automated Anomaly Triage System

Based on my hands-on experience with satellite operations, I developed an integrated anomaly triage system:

class SatelliteAnomalyTriage:
    def __init__(self, model, confidence_threshold=0.85):
        self.model = model
        self.confidence_threshold = confidence_threshold
        self.anomaly_db = self._initialize_database()

    def process_telemetry_stream(self, telemetry_data, metadata):
        """Real-time anomaly detection and triage"""
        anomalies = []

        for satellite_id, data in telemetry_data.items():
            # Extract temporal features
            features = self._extract_temporal_features(data)

            # Get anomaly scores with uncertainty
            scores, uncertainty = self.model.predict_with_uncertainty(features)

            # Identify high-confidence anomalies
            high_conf_mask = (scores > 0.7) & (uncertainty < 0.2)
            high_conf_anomalies = scores[high_conf_mask]

            if len(high_conf_anomalies) > 0:
                # Classify anomaly type
                anomaly_type = self._classify_anomaly(
                    data, high_conf_anomalies
                )

                # Assess severity
                severity = self._assess_severity(
                    anomaly_type,
                    high_conf_anomalies.mean()
                )

                # Generate response plan
                response_plan = self._generate_response_plan(
                    satellite_id,
                    anomaly_type,
                    severity,
                    metadata
                )

                anomalies.append({
                    'satellite_id': satellite_id,
                    'anomaly_type': anomaly_type,
                    'severity': severity,
                    'confidence': 1 - uncertainty.mean().item(),
                    'response_plan': response_plan,
                    'timestamp': metadata['timestamp']
                })

        return anomalies

    def _classify_anomaly(self, data, anomaly_scores):
        """Classify anomaly using learned temporal patterns"""
        # Extract pattern signatures
        pattern_features = self._extract_pattern_signatures(data)

        # Compare with known anomaly patterns
        similarities = self._compute_pattern_similarity(pattern_features)

        # Return most similar anomaly type
        return self.anomaly_db.iloc[similarities.argmax()]['anomaly_type']

    def _generate_response_plan(self, satellite_id, anomaly_type, severity, metadata):
        """Generate automated response recommendations"""
        # Query historical responses for similar anomalies
        historical_responses = self._query_historical_responses(
            anomaly_type, severity
        )

        # Consider current operational context
        context = self._assess_operational_context(metadata)

        # Generate prioritized action list
        actions = self._prioritize_actions(
            historical_responses,
            context,
            satellite_id
        )

        return {
            'immediate_actions': actions['immediate'],
            'short_term_actions': actions['short_term'],
            'long_term_actions': actions['long_term'],
            'estimated_risk': self._estimate_risk(severity, context)
        }

During my investigation of operational integration, I found that providing confidence scores and recommended actions significantly improved operator response times.

Challenges and Solutions

Challenge 1: Non-Stationary Satellite Behavior

One significant problem I encountered was that satellites don’t behave consistently over their lifetime. Components degrade, software updates change behavior, and orbital adjustments alter thermal profiles.

Solution: I implemented an adaptive normalization scheme:

python
class AdaptiveNormalization:
    def __init__(self, window_size=1000, adaptation_rate=0.01):
        self.window_size = window_size
        self.rate = adaptation_rate
        self.running_stats = {}

    def update_and_normalize(self, satellite_id, new_data):
        """Adaptive normalization with concept drift detection"""
        if satellite_id not in self.running_stats:
            self.running_stats[satellite_id] = {
                'mean': new_data.mean(0),
                'std': new_data.std(0),
                'buffer': []
            }

        stats = self.running_stats[satellite_id]

        # Detect concept drift
        drift_score = self._detect_drift(new_data, stats)

        if drift_score > 0.1:  # Significant drift detected
            # Update statistics with adaptation
            new_mean = (1 - self.rate) * stats['mean'] + self.rate * new_data.mean(0)
            new_std = (1 - self.rate) * stats['std'] + self.rate * new_data.std(0)

            stats['mean'] = new_mean
            stats['std'] = new_std

            # Clear buffer for new regime
            stats['buffer'] = []

        # Update buffer
        stats['buffer'].append(new_data)
        if len(stats['buffer']) > self.window_size:
            stats['buffer'].pop(0)

        # Normalize data
        normalized = (new_data - stats['mean']) / (stats['std'] + 1e-8)

        return normalized, drift_score

    def _detect_drift(self, new_data, stats):
        """KL divergence-based drift detection"""
        # Compute distribution statistics
        current_mean = stats['mean']
        current_std = stats['std']

        # Compute KL divergence between distributions
        kl_div = self._compute_kl_diverg

If you work in KYC, compliance, sales, or due diligence, you’ve probably spent time on state Secretary of State websites manually searching for business entities. The experience is universally terrible — each state has a different system, most are built on 2000s-era ASP.NET, and none of them talk to each other.

There’s no national business entity database. Each of the 50 states maintains its own registry independently. If you need to verify that “Acme LLC” is registered and active, you have to know which state it’s registered in and search that state’s specific portal.

Here’s what I’ve learned building tools to automate this across multiple states.

The state-by-state landscape

California

• Portal: bizfileonline.sos.ca.gov
• Searchable by: entity name, entity number
• Data available: entity name, number, type, status, formation date, jurisdiction, agent info
• API: CA has an unofficial API that returns JSON. It’s the cleanest of any state.

Texas

• Portal: SOSDirect (sos.state.tx.us)
• Searchable by: entity name, entity number, registered agent, officer/director name
• Data available: entity name, type, status, formation date, registered agent, officers
• API: No public API. The portal is a classic ASP.NET form with ViewState.

New York

• Portal: appext20.dos.ny.gov/corp_public
• Searchable by: entity name, DOS ID
• Data available: entity name, type, status, jurisdiction, filing date, county, process address
• API: No public API. Returns HTML tables.

Florida

• Portal: search.sunbiz.org
• Searchable by: entity name, officer/agent name, document number
• Data available: entity name, type, status, filing date, officers, registered agent, annual reports
• API: No API, but FL publishes bulk data via SFTP (weekly). This is the exception.

The common problems

1. No standard schema. California calls it “Entity Number”, Texas calls it “Filing Number”, New York calls it “DOS ID”, Florida calls it “Document Number”. Same concept, different field name, different format.

2. Anti-bot protection is increasing. As of 2026, Florida and several other states have added Cloudflare or Akamai protection. Texas and New York still work with direct HTTP requests. California’s unofficial API is the easiest to work with.

3. Search is fuzzy in unhelpful ways. Some states return exact matches only. Others return wildcard matches that bury what you’re looking for in hundreds of results. Name formatting (LLC vs L.L.C. vs Limited Liability Company) causes missed matches.

4. Rate limits are invisible. No state documents their rate limits. Hit them too fast and you get soft-blocked (empty results or 503 errors) without any error message explaining why.

Building it yourself vs. using existing tools

If you need to search one state occasionally, the web portal is fine. If you need to:

• Search across multiple states in a single workflow
• Do bulk entity verification (KYC/compliance)
• Monitor entity status changes over time
• Feed results into a CRM or database automatically

…then you need programmatic access.

I built individual actors for each state that handle the portal differences, anti-bot measures, and output normalization:

• California Business Entity Search — uses CA’s API endpoint
• Texas Business Entity Search — handles SOSDirect ASP.NET forms
• New York Business Entity Search — parses DOS HTML tables
• Florida Business Filings Search — searches Sunbiz portal

Each one takes a search query and returns normalized JSON:

{
  "entityName": "ACME HOLDINGS LLC",
  "entityNumber": "202312345678",
  "entityType": "Limited Liability Company",
  "status": "Active",
  "formationDate": "2023-05-15",
  "jurisdiction": "California",
  "registeredAgent": "CT Corporation System",
  "principalAddress": "123 Market St, San Francisco, CA 94105"
}

Why fragmentation is the real problem (and the opportunity)

The reason no one has built a clean multi-state business entity search is that it’s 50 separate engineering problems. Each state has different:

• Web technology (ASP.NET, PHP, Java servlets, React)
• Search parameters and result formats
• Anti-bot measures
• Data fields and naming conventions
• Update frequency and data freshness

Services like OpenCorporates, Cobalt Intelligence, and LexisNexis aggregate this data but charge enterprise pricing ($500-5000/month). If you need occasional lookups or want to build your own pipeline, the per-state tools above cost pennies per search.

What’s coming

I’m expanding to cover all 50 states. The top 5 by entity count (CA, TX, NY, FL, IL) are done. Next: IL, PA, OH, GA, NJ. The goal is programmatic access to every state’s business entity registry with normalized output.

If you’re building KYC/compliance tooling and need specific states, drop a comment — I’ll prioritize based on demand.

I build automation tools for public records data. State SOS portals are my specialty — and my recurring nightmare.

Complete Guide to Building SaaS with Next.js and Supabase

Building a Software-as-a-Service (SaaS) application requires mastering multiple technologies and architectural patterns. This comprehensive guide walks you through building production-ready SaaS applications using Next.js 15 and Supabase—a powerful combination that enables indie developers to ship fast without sacrificing quality.

Why Next.js + Supabase for SaaS?

Next.js 15 provides:

• Server-side rendering and static generation
• API routes for backend logic
• Optimized performance out of the box
• Edge runtime support
• Built-in TypeScript support

Supabase offers:

• PostgreSQL database with real-time capabilities
• Built-in authentication
• Row-level security (RLS)
• Storage for files and media
• Auto-generated APIs

Together, they form a complete stack that lets you focus on building features instead of infrastructure.

1. Project Setup and Configuration

Initial Setup

## Create Next.js app
npx create-next-app@latest my-saas-app --typescript --tailwind --app

## Install Supabase
npm install @supabase/supabase-js @supabase/ssr

Environment Configuration

## .env.local
NEXT_PUBLIC_SUPABASE_URL=your-project-url
NEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key

Supabase Client Setup

// lib/supabase/client.ts
import { createBrowserClient } from '@supabase/ssr'

export function createClient() {
  return createBrowserClient(
    process.env.NEXT_PUBLIC_SUPABASE_URL!,
    process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!
  )
}

2. Database Design for SaaS

Multi-Tenant Architecture

Design your database schema to support multiple customers (tenants):

-- Organizations table
CREATE TABLE organizations (
  id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  name TEXT NOT NULL,
  slug TEXT UNIQUE NOT NULL,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Users table
CREATE TABLE users (
  id UUID PRIMARY KEY REFERENCES auth.users,
  email TEXT NOT NULL,
  full_name TEXT,
  avatar_url TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Organization members
CREATE TABLE organization_members (
  id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  organization_id UUID REFERENCES organizations(id) ON DELETE CASCADE,
  user_id UUID REFERENCES users(id) ON DELETE CASCADE,
  role TEXT NOT NULL CHECK (role IN ('owner', 'admin', 'member')),
  created_at TIMESTAMPTZ DEFAULT NOW(),
  UNIQUE(organization_id, user_id)
);

Row-Level Security (RLS)

Protect your data with RLS policies:

-- Enable RLS
ALTER TABLE organizations ENABLE ROW LEVEL SECURITY;
ALTER TABLE organization_members ENABLE ROW LEVEL SECURITY;

-- Users can only see organizations they're members of
CREATE POLICY "Users can view their organizations"
  ON organizations FOR SELECT
  USING (
    id IN (
      SELECT organization_id 
      FROM organization_members 
      WHERE user_id = auth.uid()
    )
  );

Related: Supabase Database Schema Design for SaaS Applications, Implement Soft Deletes in Supabase PostgreSQL

3. Authentication and Authorization

Email/Password Authentication

// app/auth/login/page.tsx
'use client'

import { createClient } from '@/lib/supabase/client'
import { useState } from 'react'

export default function LoginPage() {
  const [email, setEmail] = useState('')
  const [password, setPassword] = useState('')
  const supabase = createClient()

  async function handleLogin(e: React.FormEvent) {
    e.preventDefault()

    const { data, error } = await supabase.auth.signInWithPassword({
      email,
      password,
    })

    if (error) {
      console.error('Login error:', error.message)
      return
    }

    // Redirect to dashboard
    window.location.href = '/dashboard'
  }

  return (
    <form onSubmit={handleLogin}>
      <input
        type="email"
        value={email}
        onChange={(e) => setEmail(e.target.value)}
        placeholder="Email"
        required
      />
      <input
        type="password"
        value={password}
        onChange={(e) => setPassword(e.target.value)}
        placeholder="Password"
        required
      />
      <button type="submit">Sign In</button>
    </form>
  )
}

Social Authentication

// Sign in with Google
const { data, error } = await supabase.auth.signInWithOAuth({
  provider: 'google',
  options: {
    redirectTo: `${window.location.origin}/auth/callback`,
  },
})

Protected Routes

// middleware.ts
import { createServerClient } from '@supabase/ssr'
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'

export async function middleware(request: NextRequest) {
  const response = NextResponse.next()

  const supabase = createServerClient(
    process.env.NEXT_PUBLIC_SUPABASE_URL!,
    process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!,
    {
      cookies: {
        get(name: string) {
          return request.cookies.get(name)?.value
        },
        set(name: string, value: string, options: any) {
          response.cookies.set({ name, value, ...options })
        },
        remove(name: string, options: any) {
          response.cookies.set({ name, value: '', ...options })
        },
      },
    }
  )

  const { data: { session } } = await supabase.auth.getSession()

  if (!session && request.nextUrl.pathname.startsWith('/dashboard')) {
    return NextResponse.redirect(new URL('/auth/login', request.url))
  }

  return response
}

export const config = {
  matcher: ['/dashboard/:path*'],
}

Related: Implement Social Auth with Supabase Next.js Complete Guide, Build Protected Routes in Next.js 15 with Supabase

4. API Development

REST API with Route Handlers

// app/api/organizations/route.ts
import { createClient } from '@/lib/supabase/server'
import { NextResponse } from 'next/server'

export async function GET() {
  const supabase = createClient()

  const { data: { user } } = await supabase.auth.getUser()

  if (!user) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
  }

  const { data, error } = await supabase
    .from('organizations')
    .select(`
      *,
      organization_members!inner(role)
    `)
    .eq('organization_members.user_id', user.id)

  if (error) {
    return NextResponse.json({ error: error.message }, { status: 500 })
  }

  return NextResponse.json({ data })
}

export async function POST(request: Request) {
  const supabase = createClient()
  const { name, slug } = await request.json()

  const { data: { user } } = await supabase.auth.getUser()

  if (!user) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
  }

  // Create organization
  const { data: org, error: orgError } = await supabase
    .from('organizations')
    .insert({ name, slug })
    .select()
    .single()

  if (orgError) {
    return NextResponse.json({ error: orgError.message }, { status: 500 })
  }

  // Add user as owner
  const { error: memberError } = await supabase
    .from('organization_members')
    .insert({
      organization_id: org.id,
      user_id: user.id,
      role: 'owner',
    })

  if (memberError) {
    return NextResponse.json({ error: memberError.message }, { status: 500 })
  }

  return NextResponse.json({ data: org }, { status: 201 })
}

Server Actions

// app/actions/organizations.ts
'use server'

import { createClient } from '@/lib/supabase/server'
import { revalidatePath } from 'next/cache'

export async function createOrganization(formData: FormData) {
  const supabase = createClient()

  const name = formData.get('name') as string
  const slug = formData.get('slug') as string

  const { data: { user } } = await supabase.auth.getUser()

  if (!user) {
    return { error: 'Unauthorized' }
  }

  const { data, error } = await supabase
    .from('organizations')
    .insert({ name, slug })
    .select()
    .single()

  if (error) {
    return { error: error.message }
  }

  revalidatePath('/dashboard/organizations')
  return { data }
}

Related: Build REST API with Next.js 15 Route Handlers, Implement API Rate Limiting in Next.js Applications

5. Real-time Features

Real-time Subscriptions

'use client'

import { createClient } from '@/lib/supabase/client'
import { useEffect, useState } from 'react'

export function RealtimeMessages({ channelId }: { channelId: string }) {
  const [messages, setMessages] = useState<any[]>([])
  const supabase = createClient()

  useEffect(() => {
    // Fetch initial messages
    supabase
      .from('messages')
      .select('*')
      .eq('channel_id', channelId)
      .order('created_at', { ascending: true })
      .then(({ data }) => setMessages(data || []))

    // Subscribe to new messages
    const channel = supabase
      .channel(`messages:${channelId}`)
      .on(
        'postgres_changes',
        {
          event: 'INSERT',
          schema: 'public',
          table: 'messages',
          filter: `channel_id=eq.${channelId}`,
        },
        (payload) => {
          setMessages((current) => [...current, payload.new])
        }
      )
      .subscribe()

    return () => {
      supabase.removeChannel(channel)
    }
  }, [channelId])

  return (
    <div>
      {messages.map((message) => (
        <div key={message.id}>{message.content}</div>
      ))}
    </div>
  )
}

Related: Implement Supabase Realtime Subscriptions in Next.js 15, Build Real-Time Collaboration Features with Supabase

6. File Storage

Upload Files to Supabase Storage

'use client'

import { createClient } from '@/lib/supabase/client'
import { useState } from 'react'

export function FileUpload() {
  const [uploading, setUploading] = useState(false)
  const supabase = createClient()

  async function uploadFile(event: React.ChangeEvent<HTMLInputElement>) {
    try {
      setUploading(true)

      if (!event.target.files || event.target.files.length === 0) {
        throw new Error('You must select a file to upload.')
      }

      const file = event.target.files[0]
      const fileExt = file.name.split('.').pop()
      const fileName = `${Math.random()}.${fileExt}`
      const filePath = `${fileName}`

      const { error: uploadError } = await supabase.storage
        .from('uploads')
        .upload(filePath, file)

      if (uploadError) {
        throw uploadError
      }

      // Get public URL
      const { data } = supabase.storage
        .from('uploads')
        .getPublicUrl(filePath)

      console.log('File uploaded:', data.publicUrl)
    } catch (error) {
      alert('Error uploading file!')
      console.log(error)
    } finally {
      setUploading(false)
    }
  }

  return (
    <div>
      <input
        type="file"
        onChange={uploadFile}
        disabled={uploading}
      />
    </div>
  )
}

Related: Build File Upload with Progress Bar Supabase Next.js, Supabase Storage CDN Configuration for Next.js

7. Deployment and Scaling

Deploy to Vercel

## Install Vercel CLI
npm i -g vercel

## Deploy
vercel --prod

Environment Variables

Set these in Vercel dashboard:

• NEXT_PUBLIC_SUPABASE_URL
• NEXT_PUBLIC_SUPABASE_ANON_KEY
• SUPABASE_SERVICE_ROLE_KEY

Database Migrations

## Install Supabase CLI
npm install supabase --save-dev

## Create migration
npx supabase migration new add_organizations

## Apply migrations
npx supabase db push

Related: Deploy Next.js Supabase App to Vercel Production, Handle Database Migrations in Production Supabase

8. Monitoring and Optimization

Error Tracking with Sentry

// sentry.client.config.ts
import * as Sentry from "@sentry/nextjs"

Sentry.init({
  dsn: process.env.NEXT_PUBLIC_SENTRY_DSN,
  tracesSampleRate: 1.0,
})

Performance Monitoring

// lib/analytics.ts
export function trackEvent(name: string, properties?: Record<string, any>) {
  if (typeof window !== 'undefined') {
    // Track with your analytics provider
    console.log('Event:', name, properties)
  }
}

Related: Monitor Next.js Application Performance in Production, Set Up Error Tracking with Sentry in Next.js

9. Common Pitfalls and Solutions

Session Management Issues

Problem: Auth session not persisting after refresh

Solution: Use server-side session management with cookies

import { createServerClient } from '@supabase/ssr'
import { cookies } from 'next/headers'

export function createClient() {
  const cookieStore = cookies()

  return createServerClient(
    process.env.NEXT_PUBLIC_SUPABASE_URL!,
    process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!,
    {
      cookies: {
        get(name: string) {
          return cookieStore.get(name)?.value
        },
      },
    }
  )
}

RLS Policy Errors

Problem: Users can’t access their own data

Solution: Check RLS policies and ensure auth.uid() is used correctly

-- Debug RLS policies
SELECT * FROM pg_policies WHERE tablename = 'organizations';

-- Test as specific user
SET request.jwt.claim.sub = 'user-uuid-here';
SELECT * FROM organizations;

Performance Issues

Problem: Slow database queries

Solution: Add indexes and optimize queries

-- Add index on foreign keys
CREATE INDEX idx_organization_members_user_id 
  ON organization_members(user_id);

CREATE INDEX idx_organization_members_org_id 
  ON organization_members(organization_id);

10. Next Steps

Now that you have a solid foundation, explore these advanced topics:

• Implement Optimistic UI Updates in Next.js with Supabase
• Add Full-Text Search to Next.js App with Supabase
• Build Advanced Filtering System with Supabase
• Implement Vector Search with Supabase pgvector
• Add AI-Powered Features to Your SaaS

Frequently Asked Questions (FAQ)

What is the best tech stack for building a SaaS application?

Next.js 15 with Supabase is an excellent choice for SaaS applications. Next.js provides server-side rendering, API routes, and optimized performance, while Supabase offers PostgreSQL database, authentication, real-time capabilities, and storage—all without managing infrastructure.

How do I implement multi-tenancy in Supabase?

Create an organizations table and organization_members junction table. Use Row Level Security (RLS) policies to ensure users only access data from organizations they belong to. Filter queries using organization_id IN (SELECT organization_id FROM organization_members WHERE user_id = auth.uid()).

Should I use Server Components or Client Components for my SaaS?

Use Server Components by default for better performance and smaller bundle sizes. Only use Client Components when you need interactivity (forms, buttons), React hooks (useState, useEffect), or browser APIs. This can reduce your JavaScript bundle by 50%+.

How do I handle user authentication in a SaaS application?

Use Supabase Auth with email/password or social OAuth providers. Implement middleware to protect routes, create server and client Supabase clients for different contexts, and use RLS policies to secure data access at the database level.

What’s the best way to structure my database for SaaS?

Design for multi-tenancy from the start with organizations, users, and organization_members tables. Enable RLS on all tables, create proper indexes on foreign keys, and use database functions for complex operations that need transactions.

How do I implement role-based access control (RBAC)?

Create a roles table with permissions stored as JSONB, a user_roles junction table, and database functions to check permissions. Use these functions in RLS policies: CREATE POLICY ... USING (has_permission('resource.action')).

Should I use API routes or Server Actions?

Use Server Actions for form submissions and mutations—they’re simpler and more performant. Use API routes when you need RESTful endpoints for external integrations, webhooks, or when you need fine-grained control over HTTP responses.

How do I handle file uploads in my SaaS?

Use Supabase Storage with the Next.js file upload API. Create storage buckets with RLS policies, implement upload progress tracking, and use Supabase’s image transformation API for optimized delivery. Store file metadata in your database.

What’s the best deployment platform for Next.js + Supabase?

Vercel is the recommended platform—it’s built by the Next.js team, offers zero-config deployment, automatic HTTPS, preview deployments, and edge network distribution. The free tier is generous for indie developers and startups.

How do I implement real-time features?

Use Supabase Realtime to subscribe to database changes, presence tracking for online users, and broadcast channels for ephemeral data like typing indicators. Always clean up subscriptions when components unmount to prevent memory leaks.

Should I use TypeScript for my SaaS application?

Yes, TypeScript catches errors at compile time, provides better IDE support, and makes refactoring safer. Generate types from your Supabase database with npx supabase gen types typescript to keep your types in sync with your schema.

How do I handle database migrations in production?

Use Supabase CLI to create and manage migrations. Test migrations in staging first, use transactions for data migrations, and always create rollback migrations. Never modify the database schema directly in production.

Related Articles

• Next.js Performance Optimization for Indie Developers
• Supabase Authentication & Authorization Patterns
• Deploying Next.js + Supabase to Production
• AI Integration for Next.js + Supabase Applications

Conclusion

Building a SaaS application with Next.js and Supabase gives you a powerful, scalable foundation. Focus on shipping features quickly, iterate based on user feedback, and scale as you grow.

The combination of Next.js’s performance and Supabase’s backend capabilities means you can build production-ready applications without managing infrastructure—perfect for indie developers and small teams.

Start building today, and remember: ship fast, iterate faster.

Originally published at https://iloveblogs.blog

If you’ve ever needed to look up a doctor’s NPI number, verify a provider’s credentials, or build a list of specialists in a given area, you’ve probably used the NPPES website. And you’ve probably noticed it’s awful.

The search is slow, the results are paginated one painful page at a time, there’s no bulk export, and the interface looks like it hasn’t been updated since 2008. Which it might not have been.

The good news: there’s a free public API behind it that’s actually pretty decent. The bad news: the docs are scattered and the response format is… a lot.

What’s in the NPI Registry?

Every healthcare provider in the US that bills insurance has a National Provider Identifier — a 10-digit number assigned by CMS. The NPPES registry has over 7 million active records.

For individual providers (doctors, nurses, therapists, dentists), you get:

• Full name and credentials (MD, DO, NP, LCSW, etc.)
• Primary specialty and all taxonomy codes
• Practice address and phone number
• Mailing address
• License numbers by state
• Enumeration date (when they got their NPI)

For organizations (hospitals, clinics, group practices), you get the organization name, authorized official, address, specialties, and the same taxonomy data.

Who actually needs this data?

More people than you’d think. I’ve talked to folks in healthcare sales, credentialing, insurance verification, and compliance who all need NPI lookups at scale. A few common use cases:

Healthcare sales teams building prospect lists of cardiologists in Texas or dentists in the Bay Area. The NPI registry is basically a free healthcare provider directory with verified contact info.

Credentialing departments verifying that a provider is active, has the right specialty, and practices where they say they do. Manual lookups on the NPPES site take forever when you’re processing hundreds of applications.

Insurance companies and billing teams validating NPI numbers on claims. A wrong NPI means a rejected claim, and nobody wants to chase that down manually.

Researchers studying provider distribution, specialty density by geography, or healthcare access patterns. The data is rich enough for serious analysis if you can get it in bulk.

The NPPES API

CMS runs a free API at https://npiregistry.cms.hhs.gov/api/. No API key needed, no authentication, no signup. Just hit the endpoint.

Here’s a basic search for internal medicine doctors named Smith in California:

GET https://npiregistry.cms.hhs.gov/api/?version=2.1&first_name=Smith&enumeration_type=NPI-1&state=CA&taxonomy_description=Internal%20Medicine&limit=50

The response is JSON with a results array. Each result has nested objects for basic info, addresses, taxonomies, and other names.

The catch: the API maxes out at 200 results per request, and there’s no cursor or offset pagination. If your query matches more than 200 providers, you need to narrow your search (add city, postal code, etc.) or break it into multiple requests.

Also, the response structure is deeply nested. Addresses come in an array with different address_purpose values (“LOCATION” vs “MAILING”). Taxonomies are another array where you need to find the one with primary: true. It’s not hard, but it’s tedious to flatten into something usable.

The easy way

I built a tool that wraps the NPPES API, handles the pagination workaround, flattens the nested response into clean records, and exports to CSV/JSON/Excel:

NPPES NPI Lookup – Healthcare Provider Search

You search by name, NPI number, specialty, city, state, or ZIP code. It returns flat records with the fields you actually care about — name, credential, specialty, practice address, phone, NPI, status.

What it does that the raw API doesn’t

• Handles the 200-result limit by breaking large queries into smaller geographic chunks
• Flattens the nested response so you get one row per provider with clean columns
• Returns up to 10,000 results per run
• Exports directly to CSV, JSON, or Excel — no post-processing needed

Example: find all cardiologists in Houston

Set search type to “Individual Provider,” leave name blank, set specialty to “Cardiology,” city to “Houston,” state to “TX.” Run it. You get a clean spreadsheet of every cardiologist in Houston with their NPI, practice address, phone number, and credentials.

Pricing

$0.005 per result + $0.10 per run. 1,000 provider records costs about $5.10. No subscription, no monthly minimum.

Tips if you’re building your own

If you want to hit the API directly, a few things that tripped me up:

The taxonomy_description field is fuzzy-matched. Searching for “cardio” returns Cardiology, Cardiovascular Disease, Cardiovascular Surgery, etc. This is actually useful for discovery but annoying when you want exact matches.

Use enumeration_type to filter. NPI-1 is individual providers, NPI-2 is organizations. If you don’t specify, you get both mixed together.

The limit parameter caps at 200. And there’s no skip or offset. If you need more than 200 results, you have to subdivide your search. State + city combinations work well for this.

Rate limiting is generous but undocumented. I’ve hit it at around 2-3 requests per second. Adding a small delay between requests keeps things stable. The API doesn’t return rate limit headers, so you won’t know you’re hitting the limit until requests start failing.

Watch for deactivated NPIs. Some records have status: "D" (deactivated). If you’re building a prospect list or doing credentialing verification, filter these out.

Bulk download alternative

CMS also publishes a monthly full export of the NPI registry as a CSV file. It’s about 8GB uncompressed. If you need the entire dataset and don’t mind working with a massive file, that’s at https://download.cms.gov/nppes/NPI_Files.html.

For most use cases though, searching by criteria is more practical than downloading 7 million records and filtering locally.

If you’re doing healthcare provider lookups at scale, try the NPI Lookup tool on Apify. Questions or feature requests — hit me up on the Apify Discord or open an issue on the actor page.