Mert

If you have ever tried to wire up webhooks from more than three SaaS apps, you already know the punchline: every vendor invented their own conventions, and none of them are wrong, but none of them agree either.

I was building agent tooling that had to understand many of them. Halfway through, I stopped trying to keep it in my head and started writing it down. Then I kept writing it down. The result is now an open dataset.

1,119 webhook events. 30 platforms. One schema. Free, CC-BY-4.0.

• HuggingFace: automatelab/saas-webhooks
• Browsable index: automatelab.tech/products/datasets/saas-webhooks/

What it is

A normalized catalog covering Stripe, GitHub, Slack, Notion, Linear, Jira, HubSpot, Salesforce, Zendesk, Intercom, Discord, Twilio, Calendly, Mailchimp, Zoom, Microsoft Teams, PagerDuty, Pipedrive, Asana, ClickUp, Front, Help Scout, Loom, Greenhouse, Ashby, BambooHR, Gusto, Attio, Close, and Freshdesk.

For every event, you get:

• event_name and trigger_description
• payload_schema as JSON Schema (draft 2020-12)
• auth_method, signature_header, and the exact signing algorithm
• delivery_guarantees and retry_policy
• idempotency_key_header (if the vendor provides one)
• docs_url back to the canonical vendor docs
• Format: JSONL per vendor, plus Parquet for the full set

A sample row

Here is what a single Stripe event looks like, trimmed:

{
  "vendor": "stripe",
  "category": "payments",
  "event_name": "account.application.authorized",
  "trigger_description": "Fires when a user authorizes a Stripe application.",
  "auth_method": "hmac-sha256",
  "signature_header": "Stripe-Signature",
  "signature_algorithm_detail": "HMAC-SHA256 with versioned scheme; header contains timestamp and v1 hash. Verify timestamp to prevent replay attacks.",
  "delivery_guarantees": "at-least-once",
  "retry_policy": {
    "max_attempts": null,
    "backoff": "Exponential backoff over multiple hours.",
    "total_retry_window": "PT72H"
  },
  "payload_schema": { "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { "...": "..." } },
  "docs_url": "https://docs.stripe.com/api/events/types"
}

Same shape across all 30 vendors. That is the entire point.

The surprising stuff

A few things jumped out once everything was in one schema:

Signing is not standardized in any way. A small sampler:

Same algorithm, five different envelopes. Anyone writing one verifier and trying to reuse it has a bad afternoon ahead.

Retry policies are wildly different. Stripe retries for 72 hours with exponential backoff. GitHub does not retry by default at all (it depends on app type). Slack retries 3 times. Some vendors do not publish a retry policy in their docs, which means you should not rely on one.

Idempotency support is hit-or-miss. GitHub gives you X-GitHub-Delivery. Stripe gives you the event id. Several vendors give you nothing, which means you either dedupe by payload hash or accept duplicates.

Max payload sizes are mostly undocumented. GitHub publishes 25 MB. Most vendors do not say.

These are not opinions. They are facts I would rather not have had to learn the hard way.

Why this exists

I am building agents that integrate with many SaaS products. Two things have to be true for an agent to call any webhook tool correctly:

1. The agent needs the payload schema to know what fields it can rely on.
2. The agent needs the auth contract to know how to validate inbound deliveries.

If that information is scattered across 30 different docs sites in 30 different shapes, the agent cannot use it. Once it is in a single schema, the agent can.

Same logic applies to anything that has to interop with many vendors at once: an integration platform, a security scanner that audits webhook configurations, a docs site that needs to render comparison tables, a research notebook.

Use it however

from datasets import load_dataset

ds = load_dataset("automatelab/saas-webhooks")

stripe_events = ds["train"].filter(lambda r: r["vendor"] == "stripe")
print(stripe_events[0]["payload_schema"])

The dataset is CC-BY-4.0. No email gate, no sign-up. Attribution is appreciated when you ship something interesting on top of it.

It updates monthly from source. If you find a missing event or a vendor that should be there, the GitHub issue tracker is the right place to drop it.

Links

• Dataset on HuggingFace: automatelab/saas-webhooks
• Browsable catalog (one page per vendor): automatelab.tech/products/datasets/saas-webhooks/
• All open datasets: automatelab.tech/products/datasets/

If you build something on top of it I would genuinely like to know. The whole point of putting it under CC-BY is that the work compounds.

If you’ve ever tried to debug a DNSSEC misconfiguration using dig, you know the pain. You’re staring at a wall of raw text, manually cross-referencing DS records against DNSKEY records, tracing through TLD delegations one query at a time. It works — but it’s exhausting.

I wanted something better. So I built shohei — a Rust-powered DNS diagnostic CLI that makes the invisible visible.

github.com/kent-tokyo/shohei

What Is shohei?

shohei is a next-generation DNS diagnostic tool that goes well beyond what dig or drill offer. It renders DNS resolution as color-coded terminal trees, walks you through DNSSEC trust chains step by step, and supports modern transports like DoH and DoT — all from a single command.

Think of it as dig if dig was designed for humans first.

The Three Hard Problems shohei Solves

Building a DNS diagnostic tool sounds straightforward until you try to make it actually useful. Here are the problems that motivated most of the design.

1. DNSSEC Trust Chains Are Invisible in Standard Tools

DNSSEC forms a cryptographic chain from the root zone down to individual records. The chain looks like this:

Root KSK (trust anchor, hardcoded in resolvers)
  └── Root ZSK (signs root zone records)
        └── DS record for .com  ← hash of .com's KSK
              └── .com KSK
                    └── .com ZSK (signs .com zone)
                          └── DS record for example.com
                                └── example.com KSK
                                      └── example.com ZSK
                                            └── RRSIG on A record ← what you care about

When something breaks in this chain — a DS mismatch, an expired RRSIG, a missing DNSKEY — dig gives you raw records and leaves you to piece together the failure yourself. shohei walks the whole chain and tells you exactly where validation fails and why.

. (Root)
└── com.  [DS 30909 → DNSKEY 30909 ✓]
    └── example.com.  [DS 12345 → DNSKEY 12345 ✓]
        └── www.example.com.  A 93.184.216.34
            └── RRSIG (ZSK tag 12345) expires 2026-06-01 ✓

A broken chain looks like:

. (Root)
└── com.  [DS 30909 → DNSKEY 30909 ✓]
    └── example.com.  [DS 99999 → DNSKEY ✗ NO MATCHING KEY FOUND]
        └── ⚠ Validation failed: DS/DNSKEY mismatch at example.com.

2. Iterative Resolution Is a Black Box

Standard resolvers are recursive — they do the work for you and return an answer. That’s convenient, but useless when you’re debugging a delegation problem, a glue record mismatch, or propagation that hasn’t reached a specific nameserver yet.

shohei’s --trace mode sends queries the way a resolver actually would — manually, hop by hop:

[1] Query root servers for example.com. A
    → a.root-servers.net. (198.41.0.4)  4ms
    ← REFERRAL: com. NS [a.gtld-servers.net., ...]

[2] Query TLD servers for example.com. A
    → a.gtld-servers.net. (192.5.6.30)  11ms
    ← REFERRAL: example.com. NS [ns1.example.com., ns2.example.com.]
       GLUE: ns1.example.com. A 205.251.196.1

[3] Query authoritative servers for example.com. A
    → ns1.example.com. (205.251.196.1)  8ms
    ← ANSWER: www.example.com. A 93.184.216.34  TTL 3600

Each hop shows which server was queried, its IP, round-trip time, and exactly what was returned. Delegation loops, missing glue records, and lame delegations all become immediately visible.

3. Resolver Disagreements Are Hard to Diagnose

Split-horizon DNS, stale caches, and regional differences mean two resolvers can return completely different answers for the same query. Figuring out why usually involves running dig @8.8.8.8 and dig @1.1.1.1 separately, then comparing the output by hand.

shohei’s --compare mode queries both resolvers concurrently and shows a unified diff:

Comparing 8.8.8.8 vs 1.1.1.1 for example.com A

  ANSWER
  8.8.8.8:   93.184.216.34  TTL 3521
  1.1.1.1:   93.184.216.34  TTL 120

  AUTHORITY
+ 8.8.8.8:   example.com. NS ns1.example.com.  TTL 172800
- 1.1.1.1:   (none)

  FLAGS
  8.8.8.8:   QR AA TC RD RA AD  ← AD set (DNSSEC validated)
  1.1.1.1:   QR AA TC RD RA     ← AD not set

The TTL difference here tells you 1.1.1.1 has a nearly-fresh cache entry while 8.8.8.8 is serving a stale one. The missing AD flag on 1.1.1.1 is immediately visible.

Feature Overview

DNSSEC Chain Visualization

Full trust chain from root to record, color-coded by validation status. Green for valid, red for failures, yellow for warnings (e.g., signature expiring within 7 days).

Iterative Resolution Tracing (--trace)

Manual hop-by-hop resolution with timing, referral details, and glue record display. Shows exactly what a recursive resolver would do.

DoH and DoT Support

Modern DNS transports with no external dependencies:

shohei example.com --doh https://cloudflare-dns.com/dns-query
shohei example.com --dot 1.1.1.1

Useful for testing whether a domain resolves correctly over encrypted transports, or for bypassing a local resolver that might be interfering.

Resolver Comparison (--compare)

Side-by-side diff of two resolvers. Flags TTL differences, missing records, DNSSEC validation status mismatches, and flag differences:

shohei example.com --compare 8.8.8.8 1.1.1.1

Multi-Type Queries

Query multiple record types in a single command instead of running dig four times:

shohei example.com A AAAA MX TXT SOA

Output is grouped by type with consistent formatting.

Watch Mode (--watch)

Auto-refresh at a set interval. Useful for monitoring TTL countdown, propagation in real time, or catching flapping records:

shohei example.com --watch 5  # refresh every 5 seconds

The display updates in-place, so changes are immediately visible without scrolling.

Interactive TUI (--tui)

Built with ratatui, the TUI gives you three navigable panels in a single terminal window:

• Records panel — all RRsets for the queried domain, sorted by type
• DNSSEC panel — the full trust chain with validation status
• Trace panel — iterative resolution hops with timing

Switch between panels with Tab, scroll with arrow keys, and exit with q. Useful when you need to explore a domain’s DNS health interactively rather than in a single pass.

Reverse DNS (--ptr)

PTR lookups for IPv4 and IPv6, with automatic in-addr.arpa / ip6.arpa formatting:

shohei --ptr 8.8.8.8
shohei --ptr 2001:4860:4860::8888

Batch Mode & Script-Friendly Output

Pipe a list of domains for bulk querying, with JSON output for downstream processing:

cat domains.txt | shohei --batch --json | jq '.[] | select(.dnssec_valid == false)'

The --minimal flag strips color and tree formatting for plain-text pipelines.

Why Rust?

Three reasons:

Performance. DNS diagnostics involve parallel resolution paths — querying multiple nameservers concurrently, running resolver comparisons simultaneously, tracing iterative hops while fetching DNSSEC records in parallel. Rust’s async ecosystem via tokio handles this efficiently without GC pauses that would skew timing measurements.

Correctness. DNSSEC validation is stateful and involves cryptographic operations across multiple record types. Getting it wrong silently is worse than getting it wrong loudly. Rust’s type system lets you encode validation state as types — a validated chain is a different type than an unvalidated one, and the compiler enforces that distinction.

hickory-dns. The hickory-dns crate (formerly trust-dns) is a mature pure-Rust DNS implementation with first-class DNSSEC, DoH, and DoT support. It handles wire-format parsing, cryptographic validation, and transport negotiation — the parts that are easy to get subtly wrong — so I could focus on the UX layer.

Core Dependencies

Installation

cargo install shohei

Or build from source:

git clone https://github.com/kent-tokyo/shohei
cd shohei
cargo build --release
./target/release/shohei example.com

The TUI is included by default. If you want a smaller binary without ratatui, build with --no-default-features.

Quick Reference

# Basic query — records + DNSSEC chain
shohei example.com

# Multiple record types in one shot
shohei example.com A AAAA MX TXT SOA

# Iterative resolution trace (hop by hop)
shohei example.com --trace

# DNSSEC chain only
shohei example.com --dnssec

# Compare two resolvers
shohei example.com --compare 8.8.8.8 1.1.1.1

# Query over DoH
shohei example.com --doh https://cloudflare-dns.com/dns-query

# Query over DoT
shohei example.com --dot 1.1.1.1

# Watch mode (refresh every 5s)
shohei example.com --watch 5

# Reverse DNS
shohei --ptr 8.8.8.8

# Interactive TUI
shohei example.com --tui

# Batch mode with JSON output
cat domains.txt | shohei --batch --json

# Minimal output for scripting
shohei example.com --minimal

What’s Next

• DANE/TLSA validation — cross-reference TLS certificate fingerprints against DNS to verify DANE configurations
• NSEC/NSEC3 zone walking — for security research and CTF scenarios
• HTML report export — a shareable snapshot of a domain’s full DNS health, including DNSSEC chain and trace output
• CAA record checking — verify Certification Authority Authorization records alongside DNSSEC

Feedback Welcome

shohei is MIT licensed and open to contributions. If you work with DNS professionally — or just find yourself reaching for dig more than you’d like — I’d love to hear what features would be most useful.

github.com/kent-tokyo/shohei

Built with Rust, a healthy obsession with DNSSEC, and too many late nights staring at zone files.

If you have ever integrated Spring Security into an enterprise application, you know it feels like magic—until a random 401 Unauthorized or 403 Forbidden breaks your production build.

To build secure, predictable APIs, you have to look past the boilerplate annotations and understand how Spring Security coordinates filters, manages authentication contexts, and delegates authorization. Let’s break down how the internal engine works, look at optimal JWT strategies for microservices, and tackle common interview and production questions.

1. The Core Architecture Blueprint

Every request entering a secure Spring Boot application journeys through a layered ecosystem before it ever hits your @RestController.

The Request Lifecycle Pipeline

Request ➔ DelegatingFilterProxy ➔ SecurityFilterChain (Authentication Filter) 
        ➔ AuthenticationManager ➔ AuthenticationProvider ➔ UserDetailsService 
        ➔ SecurityContextHolder ➔ AuthorizationFilter ➔ Your Controller

1. The Gateway (DelegatingFilterProxy): Servlet containers don’t natively know about Spring beans. This proxy bridges the gap, handing the request over to Spring’s managed SecurityFilterChain.
2. The Interceptor (AuthenticationFilter): Filters extract raw credentials. For instance, a BasicAuthenticationFilter looks for a header starting with Basic, decodes the Base64 payload, and extracts the username and password.
3. The Tokenization: Once extracted, the filter wraps these raw credentials into an unauthenticated token object, such as a UsernamePasswordAuthenticationToken.
4. The Coordinator (AuthenticationManager): The manager (typically implemented as ProviderManager) holds a registry of available AuthenticationProvider instances. It loops through them, matching a provider via its .supports(Class<?> authentication) method.

// Simplified look under the hood of ProviderManager
for (AuthenticationProvider provider : getProviders()) {
   if (!provider.supports(toTest)) {
       continue;
   }
   try {
       result = provider.authenticate(authentication);
       if (result != null) {
           copyDetails(authentication, result);
           break; // Authentication succeeded!
       }
   } catch (AuthenticationException e) {
       // Handled appropriately or thrown down the line
   }
}

1. The Resolver (AuthenticationProvider & UserDetailsService): The provider calls your database or identity provider (via UserDetailsService), matches the passwords, and returns a fully populated, authenticated Authentication object.

2. Authentication Outcomes: Success vs. Failure

What happens right after the AuthenticationManager completes its evaluation?

On Success: Populating ThreadLocal

The engine constructs a Principal object containing user information and roles, then injects it directly into the SecurityContextHolder. By default, this uses a ThreadLocal strategy.

Architectural Win: Because it is stored in a ThreadLocal wrapper, the active user’s context stays accessible across any controller, service, or repository executed within that specific request thread—without forcing you to pass user objects as method arguments.

On Failure: The Exception Translation Layer

If credentials mismatch, an exception is thrown. The AuthenticationManager bubbles this back to the filters. This is where the ExceptionTranslationFilter shines: it intercepts the security exceptions and translates them into appropriate HTTP responses (like a 401 Unauthorized challenge or custom JSON payload) via the configured AuthenticationEntryPoint.

3. Stateful vs. Stateless Context Management

How your application remembers authenticated users depends heavily on your architecture layout:

4. The JWT Pattern for Distributed Microservices

When implementing stateless JSON Web Tokens, you will usually extend OncePerRequestFilter. This ensures your validation logic triggers exactly once per request lifecycle, bypassing edge cases where internal servlet forwards or redirects double-trigger a regular filter.

Incoming Request ➔ JWT Filter ➔ .parseClaimsJws(token) [Validates Signature & Expiry] ➔ Set SecurityContext ➔ Chain Continues

Propagating JWT Tokens Downstream

In a microservices ecosystem, you have two primary options when Service A needs to call Service B:

1. Token Propagation (Impersonation): Forward the original client token received by your controller to downstream services.
2. Service-to-Service Identity: Service A obtains its own dedicated machine-to-machine JWT from the Authorization Server (cached until expiry) and attaches it to calls bound for Service B.

If you are using the Token Propagation approach, you don’t need to rebuild the string payload manually from user fields. Spring Security caches the raw token value inside the security context:

// Extracting the original token inside a RestTemplate/WebClient Interceptor
String token = ((Jwt) SecurityContextHolder.getContext()
       .getAuthentication().getPrincipal())
       .getTokenValue();

// Attach 'token' to the Authorization Bearer header for outbound calls

5. Method-Level Security & Role Hierarchies

By dropping @EnableMethodSecurity on a configuration class, Spring creates runtime proxies around your beans, using Aspect-Oriented Programming (AOP) to evaluate security rules before or after a method fires.

Setting up a Clean Role Hierarchy

Instead of cluttering your code with complex expressions like @PreAuthorize("hasAnyRole('SALESREP', 'ADMIN')"), clean it up by declaring a logical hierarchy bean. This ensures an ADMIN implicitly inherits all capabilities belonging to a SALESREP.

@Bean
public RoleHierarchy roleHierarchy() {
    RoleHierarchyImpl hierarchy = new RoleHierarchyImpl();
    hierarchy.setHierarchy("ROLE_ADMIN > ROLE_SALESREP");
    return hierarchy;
}

Overriding Hierarchy Defaults & Custom Beans

If you run into an isolated edgecase where an endpoint must be accessed by a SALESREP but explicitly blocked for an ADMIN, combine logical operators or delegate to a custom Spring component:

// Inline evaluation override
@PreAuthorize("hasRole('SALESREP') and !hasRole('ADMIN')")

// Delegation to a specialized security service
@PreAuthorize("@securityService.canAccessLead(authentication)")

When to leverage @PostAuthorize

While @PreAuthorize prevents a method from starting if criteria aren’t met, @PostAuthorize allows a method to completely execute, evaluates the returned entity, and throws an Access Denied exception if conditions fail. This is ideal for domain-driven permissions:

@PostAuthorize("returnObject.isPublic() or returnObject.getOwner() == authentication.name")
public Document findDocumentById(Long id) {
    return documentRepository.findById(id); 
}

6. Real-World Production Deep Dive (Q&A)

Q: If a security chain features multiple filters, can we skip remaining filters once one successfully authenticates the request?

No, but also Yes. The structure of a FilterChain is static; once built, the request must physical pass through every node in that sequence to reach the servlet. However, you can skip the heavy internal execution logic inside your custom filters by wrapping your functional logic in a conditional check (e.g., executing only if SecurityContextHolder.getContext().getAuthentication() == null). If it is already authenticated, the filter can immediately invoke filterChain.doFilter(request, response) and return early.

Q: How do we handle Blacklisting/Logouts for stateless JWTs?

Because JWTs are self-contained and stateless, they remain valid until their expiration date, even if a user logs out.

• The Scalable Approach: Keep token life-cycles highly restricted (e.g., 15 minutes) to minimize windows of vulnerability.
• The Strict Approach: Upon logout, record the token signature into a fast, centralized key-value data cache like Redis with a Time-To-Live (TTL) tracking its remaining validity duration. On subsequent filter evaluations, query Redis. If the token matches the blacklist, halt the request execution.

Q: What if a JWT token expires right in the middle of downstream processing?

Spring Security introduces a default clock skew tolerance configuration (typically 60 seconds) to account for slight server time variances across instances. To survive unexpected mid-flight expiration scenarios across deeper, time-consuming service calls, ensure downstream services implement sensible internal network timeout/tolerance settings or use API gateways that renew downstream-bound tokens automatically.

Q: Why are users experiencing random 401 Unauthorized or 403 Forbidden errors?

• Clock Drift: If your distributed cluster nodes drift out of sync temporally, a token minted on Server A might be treated as prematurely expired or not-yet-valid by Server B. Ensure all cluster nodes use Network Time Protocol (NTP) daemons to keep system clocks tightly aligned.
• Configuration Conflicts: If a URL path pattern is protected via your SecurityFilterChain settings (e.g., requiring .hasRole("ADMIN")) but the underlying method controller is annotated with an explicit @PreAuthorize("hasRole('USER')"), both criteria must pass. A configuration mismatch here results in a frustrating 403 Forbidden response.
• Token Refresh Failures: Front-end clients missing precise execution loops for silent token-refresh operations often send expired signatures, causing unexpected 401 Unauthorized errors during normal user flows.

Introduction

In the realm of digital marketing and data analytics, Search Engine Results Page (SERP) data has always been crucial for understanding user behavior, optimizing content strategies, and monitoring competitors. However, with the rapid evolution of the search engine landscape, particularly the deep integration of Artificial Intelligence (AI), traditional methods of acquiring SERP data are facing unprecedented challenges. For developers, understanding and effectively utilizing SERP APIs is no longer optional but a prerequisite for building future intelligent applications and data-driven solutions. This article will delve into the latest trends, mainstream products, and their impact on developers in the 2026 SERP API industry, offering practical advice for choosing the right API.

Four Core Forces Shaping the 2026 SERP API Landscape

The SERP API market is undergoing a profound transformation driven by both technological advancements and market demands. Here are the four key factors influencing this domain:

1. The Sunset of Google Custom Search JSON API

The Google Custom Search JSON API was once a convenient way for many developers to obtain SERP data. However, according to Google’s developer documentation
, the API stopped accepting new customers in 2025 and is scheduled to be fully discontinued on January 1, 2027. This change forces thousands of teams relying on this API to seek alternatives. Due to its inherent limitations (e.g., a maximum of 10 results per request, a daily cap of 10,000 queries, and no inclusion of ads or AI Overviews), it has long been insufficient for modern data needs. Therefore, migrating to a more powerful and contemporary SERP API has become an urgent priority.

1. AI Overview Parsing Becomes Standard

The presence of AI Overviews (AIO) in SERPs has significantly increased. BrightEdge tracking indicates that AIOs are triggered in approximately 48% of tracked queries
. seoClarity data shows a 475% year-over-year growth in mobile AIOs between September 2024 and September 2025
. This implies that if a SERP API cannot parse and structurally return the generative content, cited sources, inline videos, and injected ads within an AIO, the data it provides will be partial and incomplete. Further research by Ahrefs points out that the overlap between pages cited in AIOs and those ranking in the organic top 10 has dropped from 76% to 38%
, indicating that ranking and AIO citations are now distinct signals. Developers need to track both to gain comprehensive search insights.

1. Multi-Engine Coverage Becomes the New Norm

With the rise of AI search, users are no longer solely dependent on Google. AI search surfaces like ChatGPT, Perplexity, Gemini, and Copilot are rapidly gaining market share. For instance, Conductor’s 2026 enterprise benchmark report shows that ChatGPT drives 87.4% of AI referral traffic
. Different AI search surfaces exhibit unique citation behaviors and ranking signals. Therefore, a modern SERP API must be able to abstract Google, Bing, and these emerging AI search surfaces, providing parsed data through a unified JSON interface. APIs lacking multi-engine coverage will lead to data fragmentation, increasing integration and management costs for developers.

1. True Cost Per Call Far Exceeds Stated Prices

After the deprecation of the n=100 parameter, the true cost of SERP API calls has become more complex. Stated prices are often misleading, and developers need to consider the following factors:

•
AI Overview Surcharge: Some APIs charge extra for enabling AIO parsing, while others include it
.

•
Pagination Multiplier: Each additional 10-result page may cost 75-100% of the base fee
. This means retrieving 100 results (n=100) could cost 5-9 times more than retrieving 10 results.

•
Monthly Minimums vs. Pay-Per-Call: Some APIs offer monthly plans with overage rates, while others adopt a purely pay-per-call model
.

•
Latency Tier Surcharge: Certain APIs provide different service tiers with varying speeds, where faster speeds incur higher costs
.

When choosing an API, developers must carefully calculate the true cost in a production environment, rather than just focusing on the advertised
“per thousand calls” price.

Overview of Mainstream SERP API Products

In the 2026 SERP API market, numerous excellent products have emerged, each with distinct focuses on features, pricing, and target users. Here’s an overview of some mainstream providers:

•
cloro: With its AI-native, cross-surface SERP intelligence as a core strength, cloro offers deep parsing of Google, ChatGPT, Perplexity, Gemini, Copilot, and other AI search surfaces, including AI Overview. Its pricing model is pay-per-call, primarily targeting SEO, AI search, and brand monitoring teams.

•
SerpApi: As a mature and stable product in the market, SerpApi is known for its broad engine support, covering over 80 search engines including Google, Bing, Baidu, Yandex, eBay, and YouTube, and provides reliable AI Overview parsing. It employs a monthly plan plus overage pricing model, suitable for developers and enterprises requiring broad data sources.

•
DataForSEO: Specializing in large-scale and comprehensive SEO data services, DataForSEO supports engines like Google, Bing, Yahoo, Baidu, and Yandex, and offers AI Overview parsing. Its pay-per-call (credit-based) model makes it an ideal choice for enterprises building SEO tools and those needing highly customized data.

•
Apify: As a web automation platform, Apify provides flexible scraping capabilities through its pre-built Google Search Actor, supporting AI Overview parsing via headless Chrome. Its pay-per-compute unit model attracts developers needing flexible scraping and automation capabilities.

•
SearchApi: This product strikes a good balance between multi-engine coverage and AI parsing, supporting Google, Bing, YouTube, ChatGPT, and Perplexity, and offering AI Overview parsing. Its monthly plan pricing model is suitable for developers seeking versatile and easy-to-use APIs.

•
Serper: Known for being lightweight and fast, Serper primarily supports Google and Bing, and offers partial AI Overview parsing. Its monthly plan pricing model is suitable for developers requiring speed and lightweight solutions.

In-depth Look at Selected Products:

•
cloro: As an AI-native solution, cloro focuses on providing parsed data for Google SERP, AI Overview, AI Mode, and multiple AI search surfaces like ChatGPT, Perplexity, Gemini, and Copilot
. Its unique selling point is the ability to return all this data through a single endpoint family, along with actual cited source URLs, which is crucial for teams needing to comprehensively track AI search citations and rankings.

•
SerpApi: A veteran player in the market, SerpApi is known for its extensive search engine coverage (supporting over 80 engines) and stable service
. It offers reliable AI Overview parsing, making it an ideal choice for developers who need structured SERP data from diverse sources.

•
DataForSEO: DataForSEO targets enterprise users, offering a comprehensive suite of SEO data APIs, including SERP data, keywords, and backlinks
. Its pay-per-call credit system and flexible query speed options make it a top choice for enterprises requiring large-scale, highly customized data. However, its complex documentation and multi-step integration process might be challenging for novice developers.

•
Apify: As a web automation platform, Apify provides SERP scraping services through its pre-built Google Search Actor
. It leverages headless browsers to handle CAPTCHAs and anti-bot mechanisms, ensuring data reliability. While potentially slightly slower than pure API solutions, its flexibility and powerful automation capabilities make it a strong tool for developers needing custom scraping logic and integration into broader automation workflows.

How to Choose the Right SERP API?

For developers, selecting an appropriate SERP API requires considering multiple factors:

1.
Data Requirements: What types of SERP data do you need? Just organic results, or also ads, local packs, knowledge panels, and most importantly, AI Overviews and multi-engine AI search results?

2.
Coverage: Which countries, languages, or devices (desktop/mobile) do you need to track search results for? Is support for search engines other than Google or AI search surfaces required?

3.
Real-time vs. Scale: How high are your real-time data requirements? Do you need instant results, or can queued processing be accepted? What is your expected query volume?

4.
Cost-Effectiveness: Carefully calculate the true cost in a production environment, including base fees, surcharges, pagination costs, and monthly minimums. Choose the pricing model that best matches your budget and usage patterns.

5.
Integration Difficulty & Developer Experience: Does the API provide clear documentation, SDKs, or example code? Is there active community support? Is the integration process complex?

6.
Reliability & Support: What is the API provider’s stability, SLA (Service Level Agreement), and customer support responsiveness?

Conclusion

The 2026 SERP API market is in an exciting period of transformation. The retirement of the Google Custom Search JSON API, the widespread adoption of AI Overviews, and the rise of multi-engine AI search collectively shape a more complex yet potentially richer ecosystem. For developers, this presents both challenges and opportunities. By deeply understanding these trends and carefully selecting the right SERP API based on project needs, developers can unlock unprecedented search data insights, laying a solid foundation for building the next generation of intelligent applications and data-driven products.

I’m not a full-time developer. I’m an SEO and WordPress specialist. So when I decided to build and launch a web tool from scratch, I knew I had to be smart about every decision — from the tech stack to how I’d get it in front of people.

This is the story of how I built randomletter.io — a free, fast, accessible random letter generator — and what I learned along the way.

Why I Built It

The idea was simple: I wanted to create a suite of small, useful generator tools under one brand. Random letter generator was Tool #1.

I looked at what was already out there. Most existing tools were cluttered, slow, or just plain ugly. I knew I could do better — and more importantly, I knew a clean, fast, accessible tool could rank well on Google with the right SEO foundation.

So I got to work.

The Stack

I kept it deliberately simple:

• WordPress as the CMS (I know it well, it’s fast to deploy)
• Kadence Theme — lightweight, performance-friendly
• Custom Vanilla JS plugin for the actual tool logic
• LiteSpeed Cache + Cloudflare for speed
• Rank Math SEO for on-page optimization
• Hostinger for hosting

No React. No npm. No build pipeline. Just clean, dependency-free Vanilla JS wrapped in a custom WordPress plugin.

Building the Tool Plugin

The core tool lives in a custom plugin at wp-content/plugins/. I built it entirely in Vanilla JS — no frameworks, no libraries.

Why Vanilla JS? A few reasons:

1. Performance — no library overhead. Every millisecond matters for PageSpeed scores.
2. Simplicity — fewer moving parts means fewer things to break.
3. Control — I own every line of code.

The plugin handles letter generation, display modes (single letter, multiple letters), and user settings like excluding vowels or consonants. It enqueues its own CSS and JS cleanly, without touching the theme.

One early mistake: I accidentally enqueued Google Fonts twice — once in the theme and once in the plugin. Fixed it, and PageSpeed jumped immediately. Small things like that compound fast.

Accessibility from Day One

This was non-negotiable for me. A tool that teachers and students use needs to work for everyone.

I made sure:

• All interactive elements have proper accessibility labels
• Colour contrast meets accessibility standards (I had to fix --rlg-text-muted — the default was too light)
• The tool is fully keyboard navigable
• Screen reader announcements work correctly when a new letter generates

Result: 100/100 Accessibility score on PageSpeed Insights. Not a nice-to-have — a baseline.

The PageSpeed Obsession

I’ll be honest — I became a little obsessed with PageSpeed scores.

After a lot of tweaking — removing duplicate font loads, deferring non-critical CSS, optimising the LiteSpeed Cache configuration, and setting up Cloudflare properly — here’s where I landed:

A perfect or near-perfect score doesn’t guarantee rankings — but it removes a barrier. Google won’t penalise you for being fast.

SEO Strategy: Content Over Tricks

I didn’t do anything clever with SEO. I just did the basics well:

• Targeted low-competition keywords
• Wrote genuinely useful supporting content
• Built clean internal linking
• Made sure the site was fully crawlable

No black hat. No shortcuts. Just content that actually helps people.

The Results (8 Weeks In)

As of May 2026, with zero ad spend:

• 7,870 impressions on Google Search Console
• 85 clicks
• Average position: 18.2 (and improving)
• 7 pages indexed
• Impressions in 145 countries — India, USA, UK, Japan, Canada, Philippines, and counting

Not viral numbers. But for an 8-week-old site with no backlinks budget, I’ll take it.

What’s Next

Continuing to grow organically — more content, more visibility, more tools under the same brand. Slow and steady.

No shortcuts. No rushing.

What I Learned

1. Vanilla JS is underrated. You don’t need a framework for a focused, single-purpose tool.
2. Accessibility and performance aren’t opposites. You can have both if you’re intentional from the start.
3. WordPress gets a bad reputation it doesn’t always deserve. With the right stack, it’s genuinely fast.
4. SEO is slow. That’s fine. Compound interest is slow too. Keep going.

If you’re thinking about building a small web tool — just start. Keep the scope small, the code clean, and the user experience tight.

You can check it out at randomletter.io — feedback always welcome.

So this is the Brains room on TryHackMe. Its a two part challenge, you hack into a box (red team) and then you investigate what happened (blue team). Pretty cool combo actually.

Here’s my video walkthrough if you wanna follow along:

Part 1: Hacking the Box

First thing, start the machine and grab the IP address. Then open a terminal and run nmap to see whats available:

nmap <target-ip>

You’ll see three things open:

• Port 22 (SSH)
• Port 80 (HTTP)
• Port 50000 (something weird, looks like a database but its not)

If you go to port 80 in the browser its just a blank page, nothing there. You can run gobuster to discover hidden folders and files on it but trust me, waste of time, there’s nothing.

The interesting one is port 50000. Try connecting with telnet:

telnet <target-ip> 50000

Hit enter a few times and you’ll see it returns HTML. So its actually a web server not a database. Open it in Firefox:

<target-ip>:50000

You’ll see TeamCity running. Note the version number, in this case 2023.11.3.

Finding the Exploit

Go to exploit-db.com and search for “TeamCity”. You’ll find a bunch of vulnerabilities, the one we want is the authentication bypass / remote code execution one. Note the CVE number and search for it on GitHub.

I used the exploit from wolf hacker. Copy the code, save it as a .py file. When you run it you might get a missing module error:

pip install faker

Then run the exploit pointing at your target:

python exploit.py -t <target-ip>

If it works youll get a shell. Test it:

whoami

You should see something like “ubuntu”. Then check the home folder:

ls /home/ubuntu

Youll find a flag.txt, cat it out:

cat /home/ubuntu/flag.txt

Thats your first flag, machine hacked.

Privilege Escalation

While your in there check what you can run as sudo:

sudo -l

In this case everything is allowed which means you can just do:

sudo whoami

And youll see root. Thats it, full privilege escalation. Pretty easy one but good to practice the concept.

Part 2: Investigation with Splunk

Now close that machine and start the second one (the blue team box). Put the new IP in your browser and youll see the Splunk interface.

Go to Search and Reporting. Start with a broad search to see everything:

index=*

Set the time to “All Time” and search. Youll see around 4000 events.

Question 1: What plugin was installed after exploitation?

Just search:

index=* plugin

Youll see an event about a plugin being uploaded. The full name is right there in the log, copy it.

Question 2: What malicious package was installed?

Search for:

dpkg

Then filter to look through installed packages. Youll see loads of normal Linux stuff, lib this, lib that. Keep going through the pages and youll spot one called “data-collector”. Thats the dodgy one, nothing legit should be called that.

Question 3: What is the backdoor user?

Search for:

useradd

Youll find an entry where useradd was run with the username “evil-user” and a home folder to match. Yeah not subtle lol. Thats your answer.

Done

Thats the full Brains room. Red team side you exploited a TeamCity auth bypass to get a shell and escalated to root. Blue team side you used Splunk to find the plugin, the malicious package and the backdoor user that the attacker left behind.

If you want to practice the Linux commands used in this walkthrough check out https://practicelinux.com