How to Build a Trading Platform: Architecture, Features, and the Hard Engineering Problems

The Core Components Every Trading Platform Needs

Why Trading Platforms Are Among the Hardest Software to Build

Most software has a generous margin for error. A bug in your e-commerce checkout means a failed transaction — annoying, recoverable. A bug in your trading platform’s order matching engine means incorrect executions, real financial losses, and potentially regulatory consequences. The gap between “it works” and “it works correctly under all market conditions” is wider in trading software than almost anywhere else.

I’ve spent time building and reviewing trading platforms across retail brokerage, institutional execution, and DeFi. This post is a practical engineering guide: the architecture decisions that matter, the features you can’t cut corners on, and the failure modes that will bite you if you’re not prepared.

This is not financial advice, and building a regulated trading platform requires legal and compliance expertise beyond the scope of any engineering post. What this covers is the engineering substance of the problem.

The Core Components Every Trading Platform Needs

The Core Components Every Trading Platform Needs

1. Order Management System (OMS)

The OMS is the heart of the platform. It receives orders from users, validates them, routes them for execution, tracks their lifecycle, and reconciles the results. Every other component interacts with it.

Key requirements:

  • Idempotency: Order submission must be idempotent. Network timeouts are common; if a user retries a submission, you must not create duplicate orders.
  • State machine correctness: An order has a defined lifecycle (pending → submitted → partially filled → filled, or pending → cancelled, etc.). Transitions must be atomic and auditable.
  • Audit trail: Every state change, every modification, every cancellation must be logged with timestamp, actor, and reason. This is not optional in any regulated context.
from enum import Enum
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
import uuid

class OrderStatus(str, Enum):
    PENDING = "pending"
    SUBMITTED = "submitted"
    PARTIALLY_FILLED = "partially_filled"
    FILLED = "filled"
    CANCELLED = "cancelled"
    REJECTED = "rejected"
    EXPIRED = "expired"

class OrderSide(str, Enum):
    BUY = "buy"
    SELL = "sell"

class OrderType(str, Enum):
    MARKET = "market"
    LIMIT = "limit"
    STOP = "stop"
    STOP_LIMIT = "stop_limit"

@dataclass
class Order:
    user_id: str
    symbol: str
    side: OrderSide
    order_type: OrderType
    quantity: float
    limit_price: Optional[float] = None
    stop_price: Optional[float] = None

    # System-managed fields
    order_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    client_order_id: Optional[str] = None  # Idempotency key from client
    status: OrderStatus = OrderStatus.PENDING
    filled_quantity: float = 0.0
    average_fill_price: Optional[float] = None
    created_at: datetime = field(default_factory=datetime.utcnow)
    updated_at: datetime = field(default_factory=datetime.utcnow)

    def validate(self) -> list[str]:
        """Validate order before submission. Returns list of error messages."""
        errors = []

        if self.quantity <= 0:
            errors.append("Quantity must be positive")

        if self.order_type in (OrderType.LIMIT, OrderType.STOP_LIMIT):
            if self.limit_price is None or self.limit_price <= 0:
                errors.append("Limit price required and must be positive")

        if self.order_type in (OrderType.STOP, OrderType.STOP_LIMIT):
            if self.stop_price is None or self.stop_price <= 0:
                errors.append("Stop price required and must be positive")

        return errors

    def can_transition_to(self, new_status: OrderStatus) -> bool:
        """Enforce valid state machine transitions."""
        valid_transitions = {
            OrderStatus.PENDING: {OrderStatus.SUBMITTED, OrderStatus.REJECTED},
            OrderStatus.SUBMITTED: {
                OrderStatus.PARTIALLY_FILLED, OrderStatus.FILLED,
                OrderStatus.CANCELLED, OrderStatus.EXPIRED
            },
            OrderStatus.PARTIALLY_FILLED: {
                OrderStatus.FILLED, OrderStatus.CANCELLED
            },
        }
        return new_status in valid_transitions.get(self.status, set())


class OrderManagementSystem:

    def __init__(self, db, risk_engine, execution_router, audit_log):
        self.db = db
        self.risk = risk_engine
        self.router = execution_router
        self.audit = audit_log

    def submit_order(self, order: Order) -> dict:
        # Idempotency check
        if order.client_order_id:
            existing = self.db.find_by_client_order_id(order.client_order_id)
            if existing:
                return {"status": "duplicate", "order_id": existing.order_id}

        # Validation
        errors = order.validate()
        if errors:
            return {"status": "rejected", "errors": errors}

        # Pre-trade risk checks
        risk_result = self.risk.check(order)
        if not risk_result.approved:
            order.status = OrderStatus.REJECTED
            self.db.save(order)
            self.audit.log("order_rejected", order, reason=risk_result.reason)
            return {"status": "rejected", "reason": risk_result.reason}

        # Submit
        order.status = OrderStatus.SUBMITTED
        self.db.save(order)
        self.audit.log("order_submitted", order)

        # Route to execution (async in production)
        self.router.route(order)

        return {"status": "submitted", "order_id": order.order_id}

2. Market Data Infrastructure

Your platform needs real-time market data: current prices, order book depth, trade history, and historical data for charts. This is harder than it looks because:

  • Volume is high: A single liquid equity can generate thousands of price updates per second
  • Latency matters: Stale prices cause bad user decisions and, in some architectures, bad executions
  • Data quality matters: Bad ticks (erroneous price prints) need to be filtered

The architecture decision is whether to build your own market data pipeline or use a managed provider. For most platforms, managed providers (Polygon.io, Alpaca, Interactive Brokers data feeds) are the right answer — the engineering investment in a production-grade market data system is substantial and the differentiation is minimal.

When you do need to build your own data handling layer, a time-series database is essential. TimescaleDB (Postgres extension) handles most use cases well without introducing a new operational dependency:

-- TimescaleDB hypertable for OHLCV data
CREATE TABLE ohlcv (
    time        TIMESTAMPTZ NOT NULL,
    symbol      TEXT NOT NULL,
    open        NUMERIC(18, 8) NOT NULL,
    high        NUMERIC(18, 8) NOT NULL,
    low         NUMERIC(18, 8) NOT NULL,
    close       NUMERIC(18, 8) NOT NULL,
    volume      NUMERIC(24, 8) NOT NULL
);

SELECT create_hypertable('ohlcv', 'time');
CREATE INDEX ON ohlcv (symbol, time DESC);

-- Continuous aggregate for 1-hour candles from tick data
CREATE MATERIALIZED VIEW ohlcv_1h
WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 hour', time) AS bucket,
    symbol,
    first(open, time) AS open,
    max(high) AS high,
    min(low) AS low,
    last(close, time) AS close,
    sum(volume) AS volume
FROM ohlcv
GROUP BY bucket, symbol;

3. Risk Engine

The risk engine sits between order submission and execution. It enforces position limits, buying power constraints, and market risk parameters. It is not optional.

Pre-trade risk checks for a retail platform typically include:

  • Buying power: Does the user have sufficient funds/margin to cover this order?
  • Position limits: Would this order exceed maximum allowed position size per symbol?
  • Order size limits: Is this order unreasonably large (potential fat-finger error)?
  • Market hours: Is this market currently open for the order type being submitted?
  • Symbol restrictions: Is this symbol available for trading on this platform?
from dataclasses import dataclass

@dataclass
class RiskCheckResult:
    approved: bool
    reason: Optional[str] = None
    warnings: list = field(default_factory=list)

class PreTradeRiskEngine:

    def __init__(self, account_service, position_service, config):
        self.accounts = account_service
        self.positions = position_service
        self.config = config

    def check(self, order: Order) -> RiskCheckResult:
        account = self.accounts.get(order.user_id)

        # Buying power check
        estimated_cost = self._estimate_order_cost(order)
        if account.available_cash < estimated_cost:
            return RiskCheckResult(
                approved=False,
                reason=f"Insufficient buying power. Required: {estimated_cost:.2f}, Available: {account.available_cash:.2f}"
            )

        # Position limit check
        current_position = self.positions.get(order.user_id, order.symbol)
        new_position = current_position.quantity + (
            order.quantity if order.side == OrderSide.BUY else -order.quantity
        )

        max_position = self.config.get_max_position(order.symbol, account.tier)
        if abs(new_position) > max_position:
            return RiskCheckResult(
                approved=False,
                reason=f"Order would exceed maximum position limit of {max_position} for {order.symbol}"
            )

        # Fat finger check
        if order.quantity > self.config.fat_finger_threshold:
            return RiskCheckResult(
                approved=False,
                reason=f"Order size {order.quantity} exceeds maximum single order size {self.config.fat_finger_threshold}"
            )

        return RiskCheckResult(approved=True)

    def _estimate_order_cost(self, order: Order) -> float:
        if order.order_type == OrderType.LIMIT and order.limit_price:
            return order.quantity * order.limit_price
        # For market orders, use last price with a buffer
        last_price = self.positions.get_last_price(order.symbol)
        return order.quantity * last_price * 1.02  # 2% buffer for market impact

4. Real-Time Portfolio and P&L

Users need to see their current positions, unrealised P&L, and account value in real time. This is a read-heavy workload that benefits from a separate read model updated by the execution feed.

WebSocket connections are the standard for pushing portfolio updates to frontend clients. The architecture: execution fills update a portfolio state store (Redis works well here for latency), and a WebSocket gateway pushes diffs to connected clients.

The Features You Cannot Cut Corners On

The Features You Cannot Cut Corners On

Order History and Statements

Every trade must be recorded and retrievable. Users need complete trade history for tax purposes. Regulators need it for compliance purposes. Your operations team needs it for reconciliation.

This means: immutable trade records, complete audit trails, export capabilities (CSV at minimum), and retention policies that meet your regulatory requirements. The retention requirement for financial records in most jurisdictions is 5-7 years.

Account Security

Trading accounts are high-value targets. The security requirements go beyond standard web application security:

  • MFA mandatory, not optional: SMS, TOTP, or hardware key
  • Session management: Short session timeouts, concurrent session detection, geographic anomaly alerts
  • Withdrawal address whitelisting: For crypto platforms, withdrawals only to pre-approved addresses
  • Transaction monitoring: Flag unusual patterns — unusually large trades, trading at unusual hours, rapid position changes

Reconciliation

End-of-day reconciliation between your internal records and your execution venue records is not optional. Discrepancies exist — execution venues make mistakes, network issues cause message loss, edge cases in your OMS create inconsistencies. Daily automated reconciliation with exception alerting catches these before they compound.

The Infrastructure Reality

A trading platform is not a typical web application. The requirements that differentiate it:

Latency: Order submission to acknowledgement needs to be fast — users notice delays above 200ms, and anything above 1 second creates trust issues. This means database query optimisation, connection pooling, and careful attention to your critical path.

Reliability: Trading platforms need 99.9%+ uptime during market hours. Planned maintenance windows need to be outside market hours. Unplanned outages during high-volatility market sessions are severe reputational events.

Consistency over availability: When you have to choose between availability and consistency (a partition tolerance scenario), trading platforms choose consistency. It is better to reject an order than to create an inconsistent state.

Disaster recovery: You need point-in-time recovery for your trade database, tested regularly. RTO (recovery time objective) and RPO (recovery point objective) need to be defined and designed for before you go live.

For teams building fintech and trading infrastructure, our team at Lycore has hands-on experience with the full stack — from order management systems to real-time market data pipelines to regulatory reporting. The complexity is significant but manageable with the right architecture from the start.

What Most Teams Get Wrong

Starting with the UI: The beautiful trading interface is the last thing to build, not the first. The OMS, risk engine, and execution connectivity need to be solid before the front end matters.

Underestimating reconciliation: Teams consistently underinvest in reconciliation infrastructure and spend months retrofitting it after launch. Build it in from day one.

Ignoring the operational side: A trading platform needs a full operational runbook, clear escalation paths for execution issues, and relationships with your execution venues’ technical support teams. You will have incidents. Being prepared for them is the difference between a recoverable situation and a crisis.

Not testing failure modes: Test what happens when your execution venue connection drops mid-order. Test what happens when the market data feed goes stale. Test what happens when your database primary fails over. These scenarios will occur in production.

Building something in the fintech or trading space? I’m happy to discuss architecture in the comments — the specifics vary a lot by asset class, regulatory jurisdiction, and execution model.

AI 週報 — 2026-05-15 to 2026-05-22 | 工具鏈下沉:Anthropic 買 Stainless 與企業 AI 的 actual 進展

本週一句話:當多數分析還在追逐模型 benchmark,Anthropic 收購 Stainless 已經把「收購整合」本身的風險放到了檯面上——技術團隊能否留住、產品路線圖是否偏移,這些問題比任何 benchmark 分數更直接決定這筆交易的成敗。

Anthropic 收購 Stainless:整合風險才是真正的故事

本週最重要的事件不是任何新模型發布,而是 Anthropic 收購 Stainless(API 開發工具公司,專注於 SDK 與 API 穩定性的自動化)。

這個收購的策略邏輯不新——Anthropic 的商業模式向下沈澱的路徑,過去 18 個月已有跡可循:與 PwC 合作滲透企業諮詢市場[6]、與 KPMG 覆蓋 27.6 萬人的內部部署[7]。這些動作的終點現在看清楚了:Anthropic 不只想當模型供應商,而是想把「用 Anthropic 模型開發」這件事的整條路徑都握住。

但真正值得追蹤的不是策略方向,而是執行風險。 Stainless 的價值在工程師社群——SDK 的穩定性與開發體驗取決於團隊。收購整合中技術團隊出走、產品路線圖因母公司優先順序而偏移,是這類交易的常見死法。這筆收購是否會像 Microsoft/GitHub 或 Google/Android 的整合那樣順利,目前沒有判斷依據——整合成敗通常在交易宣布後 12-18 個月才見分曉。

與替代方案的對比: 現有企業接入 LLM API 的標準工具鏈是 OpenAI 的 SDK、LangChain、以及各種開源中介層。Anthropic 若能透過 Stainless 提供更穩定、更整合的開發體驗,能在工具層形成 Pull——工程師因為 Stainless 生態而選擇 Claude,而不是因為 Claude 能力更強所以忍受工具鏈的不便。

企業 AI 落地:用「發布/可用/可商用」框架過濾噪音

本週有兩則大型企業合作公告:PwC 用 Claude 建構技術、執行交易、重塑企業功能[6];KPMG 將 Claude 整合進 27.6 萬員工的工作流程[7]。

員工總數是誤導性指標。 27.6 萬這個數字包含行政、后勤、職能部門,不等於實際在核心分析流程使用 Claude 的人數。真正有意義的指標是:每日活躍用戶數、錯誤率、延遲數據。這些數據比員工總數更能反映落地效果,但目前都沒有披露。

把這些公告視為「企業 AI 預算分配方向確認」,而非落地效果證明。

OpenAI 的政府夥伴關係:Malta 與新加坡

本週 OpenAI 宣佈與 Malta 達成合作,目標是讓所有公民使用 ChatGPT Plus[[2]],以及與新加坡建立國家級 AI 合作夥伴關係[[5]]。

切入角度: 這是分發渠道的制度性佔位。當一個國家政府出面將特定 AI 工具推向國民,會形成制度性依賴(stickiness)。這比任何廣告投放都有效。

但注意:

  • Malta 人口約 53 萬,規模相當於一個中型城市。這是 pilot,不是 scale。
  • 新加坡合作更具戰略價值——作為亞洲金融與技術樞紐,新加坡的 AI 政策有輻射效應。
  • 目前沒有公開的技術規格或合約細節,難以評估實質內容。

與 Anthropic 的對比: Anthropic 選擇大型顧問公司作為渠道(PwC、KPMG);OpenAI 選擇政府作為直接渠道。兩種模式代表不同的市場切入策略。

Google DeepMind 科研 AI:批量出現的信號意義

本週 DeepMind 密集發布了一系列科學應用公告,涵蓋:

  • Co-Scientist[10]:多 Agent 協作研究系統,輔助科學假說生成
  • ALS 研究合作[14]:與學術機構合作,用 AI 加速漸凍人症研究
  • 傳染病分子開關識別[15]:找出新品種傳染病的分子調控機制
  • 肝臟疾病機制發現[16]:加速肝病理解
  • 藥物重新定位(肝纖維化)[17]:現有藥物新用途探索
  • 老化研究[18]:生物老化機制的 AI 輔助發現
  • WeatherNext 預測颶風 Melissa[13]:提前預警極端天氣事件

這批公告的共同特徵: 全部是「AI 輔助科學研究」而非「AI 取代科學家」。Co-Scientist 的設計是「AI 夥伴」而非「AI 研究員」[10]——這個語義差異值得注意。

更有用的觀察: 數量與領域覆蓋範圍本身是信號。單點實驗不會同時出現在ALS、肝臟、傳染病、藥物重新定位、老化等多個領域。這顯示 AI 在科學發現上的應用正在從「單點實驗」進入「批量探索」階段。這是少數對人類有直接長期價值的 AI 應用方向之一。

WeatherNext 的實際價值要看預測準確率與現有氣象模型的對比——目前沒有公開 benchmark 數據,這點值得持續追蹤。

NVIDIA Vera CPU:硬體層的 Agent 訊號

NVIDIA 發表了 Vera[20]——他們首款為 AI Agent 場景設計的 CPU。同時 NVIDIA 與 Google Cloud 擴大合作[[11]],以及發布電信 AI 工廠的技術文件[[19]]。

真正的訊號: 硬體廠商(NVIDIA)明確在對 Agent 場景做硬體優化,而不是繼續只談 GPU 訓練能力。這是 Agent 應用正在脫離「demo 階段」的間接證明。

實質評估:

  • Vera 目前處於「發布」狀態,硬體出貨與效能 benchmark 尚未公開驗證。
  • Agent 場景的 CPU 需求與傳統伺服器不同:大量短任務切換、高頻 API 調用。這是需要專用硬體優化的真實需求。
  • 「電信 AI 工廠」概念(透過 NVIDIA 的 Telco 合作)[[19]]代表邊緣 AI 推理的商業化路徑,這是相對新的應用方向。

OpenAI Content Provenance:對工程師的實際意義

OpenAI 發布了內容出處(provenance)標準推進的更新[[4]],推動 C2PA 開源標準作為 AI 生成內容的元數據標記框架。

這是什麼: 簡單說,就是給 AI 生成的圖片、影片、文字加上不可竄改的「出生證明」。背後技術是 C2PA(Coalition for Content Provenance and Authenticity)開放標準。

工程師的實際相關性:

  • 如果你的產品輸出需要符合內容來源標記規範,現在有了一個可依賴的標準。
  • 但:標準落地需要整個生態採用。目前僅有部分工具支援,距離成為預設行為還有距離。
  • 這是「可用」狀態,不是「可商用」狀態——實務整合需要額外工程投入。

基礎設施對比:本週硬體與合作矩陣

公司/組織 動作 層次 實質階段
Anthropic 收購 Stainless 工具鏈 宣布(整合風險未知)
Anthropic PwC/KPMG 合作 分發/應用 宣布(無落地數據)
OpenAI Malta/Singapore 合作 分發/政策 宣布(無技術細節)
OpenAI Dell Codex on-prem 分發/部署 宣布(無延遲/定價)
NVIDIA Vera CPU for Agents 硬體 宣布(無 benchmark)
Google DeepMind Co-Scientist + 科研應用 應用 發布(批量覆蓋多領域)

結語

本週沒有 transformer 架構的新突破,沒有 GPT-5 發布,沒有 Claude 5。

但有兩件事值得記住:

第一,Anthropic 收購 Stainless 把「模型公司往下走」這件事從猜測變成事實。工具鏈控制權是下一階段的真正護城河,比 5% 的 benchmark 差距更有長期價值——但這筆交易的成敗,取決於整合執行,不是策略方向。

第二,DeepMind 這週密集發布的科研應用,數量與領域覆蓋範圍顯示 AI 在科學發現上的應用正在從「單點實驗」進入「批量探索」階段。這是少數對人類有直接長期價值的 AI 應用方向之一。

多做一步的驗證方向:關注 Stainless 收購完成後的產品整合公告,以及 KPMG/PwC 合作的實際內部採用率數據(通常會在 6-12 個月後出現)。

How AI Is Changing the QA Engineer Role in 2026: A Data Analysis

Breakdown of QA Engineer postings by AI requirement: 93.9% no AI mention, 3.1% generative AI only, 1.2% generative AI plus traditional ML, 1.7% traditional ML only

How Has the QA Engineer Job Description Changed in 2026?

Open a QA Engineer job posting from 2022 and you will find a familiar checklist: Selenium or Cypress, a primary scripting language (usually Python, Java, or JavaScript), test framework experience (JUnit, TestNG, pytest), a CI/CD pipeline, an API testing tool like Postman, and behavior-driven development (BDD) for the more progressive shops. Open one from May 2026 and that checklist is still there, but a new layer has begun to settle on top: test the AI features the product team just shipped, build evaluation harnesses for LLM outputs, and use AI assistants to author and maintain the test suite itself.

To put numbers on it, we looked at every active QA Engineer posting on the InterviewStack.io job board over the trailing 90 days as of May 2026, 16,376 listings, with AI skills extracted from descriptions and synonyms collapsed (so “ChatGPT”, “OpenAI API”, and “Anthropic Claude” each get counted under the right canonical concept).

The headline: AI is not yet pervasive in QA hiring (4.3% of postings explicitly mention new-wave generative AI), but the salary premium for AI-fluent QA Engineers is the largest we have measured across our AI-shift analyses to date (including Software Engineering), roughly 45% over the non-AI baseline. That gap signals scarcity, and scarcity is what early movers convert into offers.

Key Findings

  • 16,376 active QA Engineer postings analyzed across the live job board in May 2026.
  • 4.3% of postings explicitly mention new-wave generative AI skills (712 of 16,376). When traditional machine learning is included, the share rises to 6.1% (998 postings).
  • Median US base salary is $119,000 for postings that ask for new-wave AI skills (n=75), versus $82,041 for postings without AI (n=3,264). That is a $36,959 premium, or roughly 45% above the non-AI baseline.
  • Machine Learning is still the top AI-adjacent skill in QA postings at 2.8% (458 listings); LLMs (1.4%) and AI Agents (1.4%) are the leading new-wave entries.
  • Staff QA Engineers see the highest AI demand at 7.3%; senior follows at 4.8%, mid-level at 3.6%, junior at just 1.5%. AI work is concentrated at the experienced end of the ladder.
  • Professional services leads industries at 31.3% AI adoption among its QA Engineer postings, more than double any other sector.
  • Poland (18.1%), Mexico (10.0%), and India (9.9%) all post AI-related QA roles at a higher rate than the US (2.1%), reflecting consulting-led offshore work on AI testing engagements.

What Did the QA Engineer Role Look Like in 2022?

Three or four years ago, the QA Engineer description was one of the more stable job specs in software. The Stack Overflow Developer Survey 2022 captured the developer-side baseline well: Selenium and Cypress as the dominant browser-test frameworks, Jest as the rising JavaScript test runner, Postman for API testing, and JUnit / TestNG / pytest for unit and integration layers. The GitHub Octoverse 2022 report tracked the same migration toward open-source test stacks (Playwright grew sharply that year) and toward GitHub Actions as the default CI integration point.

A typical 2022 QA Engineer posting asked for some combination of: a primary scripting language (Python, Java, or JavaScript), one browser automation framework (Selenium or Cypress), a unit-test framework, API testing tools, CI/CD integration (Jenkins or GitHub Actions), JIRA, and either Agile/Scrum or a structured test-management tool. Manual testing was already being phased out as a stand-alone discipline; “QA Engineer” had largely consolidated around test automation. Generative AI was not in the working vocabulary: ChatGPT had only just launched in November 2022, and no hiring manager was writing “Prompt Engineering” into a QA job description.

The result was that “QA Engineer” was narrow and stable across the industry. A posting from a fintech and a posting from a healthcare SaaS company would look almost interchangeable at the top of the spec. That stability is what is now starting to crack.

What Share of QA Engineer Postings Now Ask for AI Skills?

The first thing the 2026 data shows is that AI has reached QA, but only at the margins. The growth is visible, the salary signal is loud, but the absolute share of QA postings asking for AI is much smaller than in Software Engineering.

Breakdown of QA Engineer postings by AI requirement: 93.9% no AI mention, 3.1% generative AI only, 1.2% generative AI plus traditional ML, 1.7% traditional ML only

Of 16,376 active QA Engineer postings, 6.1% mention some form of AI. New-wave generative AI (the LLM-era stack of Agents, Prompt Engineering, RAG, GitHub Copilot) appears in 4.3%, with 1.2% of postings asking for both new-wave AI and traditional ML.

Three things to pull out of that chart. First, the new-wave generative AI cohort (3.1% on its own, 4.3% including overlap) is already larger than the pure traditional-ML cohort (1.7%). In just over three years since ChatGPT launched, the LLM-era toolkit has overtaken classical ML as a QA hiring requirement, even though both are small. Second, only 1.2% of postings ask for both new-wave AI and traditional ML, almost always senior or staff roles at companies running production ML platforms that now also have generative AI features layered on top. Third, 93.9% of postings still ask for neither. Regression testing, browser automation, API testing, and manual test-case work still drive the vast majority of QA hiring. AI is the leading edge, not the median.

For a QA Engineer deciding whether to invest in AI skills now or wait, the “any AI” share (6.1%) understates how concentrated that demand is. As we will see in the salary and seniority cuts below, the AI-asking postings are clustered at the senior tier and pay a steep premium, which means they screen a much smaller pool of candidates.

Which AI Skills Are Reshaping the QA Engineer Role?

Drill into the AI skills that show up across QA postings and a clear two-track story emerges. The top of the list is dominated by what QA Engineers are being asked to test against (LLMs, AI Agents, Generative AI). The next tier is dominated by what QA Engineers are being asked to use (GitHub Copilot, ChatGPT, AI-assisted development).

Top AI skills demanded in QA Engineer postings: Machine Learning 2.8%, LLMs 1.4%, AI Agents 1.4%, Generative AI 1.2%, AI-Assisted Development 0.6%, GitHub Copilot 0.5%, Prompt Engineering 0.5%, ChatGPT 0.5%, OpenAI 0.4%, LangChain 0.3%, RAG 0.3%, Deep Learning 0.3%

Share of QA Engineer postings that mention each AI skill. Traditional ML (gray) has been in QA postings for years; the new-wave generative AI stack (highlighted) is what changed since 2023.

The ranking tells a clear story:

  1. Machine Learning (2.8%, 458 postings) is the largest single AI-related skill. Most of those are not new-wave generative AI: they are postings at companies running production ML pipelines (recommender systems, fraud models, demand forecasting) where the QA team validates training data, monitors model drift, and writes acceptance tests against ML outputs. This is not a 2026 phenomenon; it has been at roughly this level for several years.

  2. LLMs (1.4%, 234) and AI Agents (1.4%, 222) are the leading new-wave entries. Postings in this band ask QA Engineers to test against LLM-powered features: chat interfaces, summarization endpoints, agentic workflows that call tools and make decisions. The work pulls in evaluation-harness design, prompt-based regression suites, and accuracy/hallucination scoring. (Browse QA Engineer roles asking for LLMs.)

  3. Generative AI (1.2%) and Prompt Engineering (0.5%) form the second new-wave tier. Prompt Engineering specifically is interesting in a QA context: it appears as a testing technique (varying prompts to probe failure modes) more often than as a development skill. RAG (retrieval-augmented generation, where an LLM is grounded against a vector store) and OpenAI (0.4%) round out the application stack.

  4. AI-Assisted Development (0.6%, 91), GitHub Copilot (0.5%, 87), and ChatGPT (0.5%, 84) form a distinct cluster. These postings are not asking the QA Engineer to test AI products; they are asking the QA Engineer to use AI assistants to write tests, draft test plans, and generate test data faster. It is now an explicit hiring signal at a small but visible fraction of employers. (QA Engineer + GitHub Copilot openings.)

  5. LangChain (54), RAG (49), Vector Databases (16), and Anthropic / Claude (10) make up the long tail of framework-specific mentions. Specific framework brands stay niche; the volume sits on the underlying concepts (LLMs, Agents, RAG) rather than on tool names, which is what you would expect in a stack that is still settling.

The clearest signal across the ranking: AI is reaching QA on two distinct vectors at once, and the postings that pull both vectors together (test AI features and use AI to test) are the highest-paying and most senior. The next section quantifies the pay side directly.

How Much Does AI Knowledge Raise a QA Engineer’s Salary?

Among US postings (where wage-transparency laws produce consistent disclosure), the median QA Engineer base salary in postings that do not require AI skills is $82,041 (n=3,264). In postings that require new-wave generative AI skills, the median jumps to $119,000 (n=75), a $36,959 difference, or roughly 45% above the non-AI baseline. Equity, bonus, and sign-on are not disclosed in postings, so total compensation at AI-heavy employers (companies building agentic platforms, applied-AI startups, frontier labs) runs materially higher than these base figures suggest.

US base salary comparison for QA Engineer postings: $82,041 median without AI skills (n=3,264), $119,000 median with new-wave AI skills (n=75), a $36,959 premium (about 45% above baseline)

Median US base salary for QA Engineer postings with and without new-wave AI skill requirements. Equity and bonus are not in this dataset.

That premium is unusually steep. For context, our analysis of Software Engineer roles found a $21,000 (16.2%) premium for AI skills. QA Engineers see a premium more than 1.7 times larger in absolute dollars and nearly three times larger in percentage terms. The cleanest read of that gap is that AI-fluent QA Engineers are scarce relative to demand. Most QA candidates today are still optimizing for browser automation and API regression suites; the small group that has also built evaluation harnesses for LLM outputs or trained on production agentic workflows can name a number.

Two caveats are worth flagging. First, the AI-skill sample (n=75 US postings with disclosed salary) is much smaller than the no-AI sample (n=3,264). The point estimate of $119,000 has a wider band than the baseline. Second, the premium compounds with seniority: as the next section shows, AI demand clusters in the senior and staff tiers, which already pay more than the role average. Part of the $36,959 gap reflects seniority skew, not AI alone. Even so, the signal is unambiguous: AI knowledge is currently the single highest-paying skill class to add to a QA Engineer resume.

For a mid-level QA Engineer deciding whether to spend a quarter learning LLM evaluation, prompt-based testing, and one AI coding assistant, the payback is short. Targeting QA Engineer roles that ask for AI skills on your next job search compresses the salary jump even further.

Who Is Leading the Shift: Which QA Engineers, Industries, and Companies?

The shift is not happening evenly. Three cuts of the data make that obvious: by seniority, by industry, and by employer.

AI adoption rate by QA Engineer seniority level: Staff 7.3%, Senior 4.8%, Entry 3.6%, Mid-level 3.6%, Junior 1.5%

Share of postings at each seniority level that ask for AI skills. Staff and senior are the clear leaders.

The seniority pattern is monotonic at the top and tells a coherent story:

  • Staff QA Engineers (7.3%, 39 of 535) are the most-asked group. Companies need experienced QA leaders to design the evaluation harnesses, the AI-feature test strategies, and the data-quality gates that production LLM systems require. This is where the senior skills gap shows up most sharply.
  • Senior (4.8%, 487 of 10,123) is where the bulk of the absolute AI volume sits. The senior tier alone accounts for 61.8% of all QA Engineer postings, so even at a 4.8% adoption rate, it produces the largest cohort of AI-asking listings.
  • Mid-level (3.6%) and Entry-level (3.6%) are tied, but with very different stakes: entry-level postings are tiny in absolute count (506) and only 18 of them mention AI.
  • Junior (1.5%, 15 of 1,003) is the lowest cohort. The pattern is clear: companies are not pushing AI requirements onto early-career QA hires. They are pulling experienced testers up the stack to lead AI work.

The industry view amplifies the same point.

AI adoption rate by industry for QA Engineer postings: Professional Services 31.3%, Technology 12.3%, Pharmaceutical 8.6%, Software 7.7%, Finance 7.3%, Healthtech 7.1%, Fintech 4.5%, Other 3.3%, Insurance 3.0%, Consulting 2.6%, Education 2.5%, Manufacturing 2.3%

Share of QA Engineer postings within each industry that require AI skills. Professional services is the clear outlier.

  • Professional services (31.3%, 45 of 144) is a striking outlier, more than double the next group. These are the consulting and software-services firms staffing AI testing engagements at client companies that have not yet built the capability in-house. If you want exposure to a wide range of AI testing problems quickly, professional services is the fastest on-ramp; the trade-off is the project-rotation cadence rather than long-term ownership.
  • Technology (12.3%, 149) and Pharmaceutical (8.6%, 9) follow. Pharma’s appearance here is worth noting: it sits above software (7.7%) on AI adoption rate even with a much smaller posting base, reflecting how aggressively drug-discovery and lab-automation teams are layering AI on top of their existing validation work.
  • Software (7.7%) and Finance (7.3%) round out the leaders. Finance is using LLMs for document review, compliance triage, and customer-support automation, all of which need QA against hallucination and accuracy.
  • Manufacturing (2.3%), education (2.5%), and consulting (2.6%) trail at the bottom. If your current role is in those sectors, the AI signal in the job market is genuinely weaker; switching industries may matter more than switching companies.

Geography tells a parallel story that surprises most readers. The United States is the largest single QA market by volume (6,659 postings, 41% of the dataset), but its AI adoption rate is only 2.1%, well below the global average. Poland leads at 18.1% (37 of 204 postings), followed by Mexico at 10.0% (35 of 350) and India at 9.9% (139 of 1,399). Most of that international volume flows through global capability centers and software-services firms (the same firms driving the professional-services number above) serving US and UK clients on AI testing engagements, which is why the consulting-led offshore markets are running ahead of the domestic US one.

On the employer side, the highest-density hirers are revealing. Nebius Academy (34 AI postings out of 44 QA Engineer roles, 77% AI density) and AgileEngine (33 of 33, 100%) are the most aggressive volume hirers. Several firms post at 100% AI density across smaller samples, including Marvell Technology (18 of 18), VML (8 of 8), Centific (8 of 8), Hyland (8 of 8), and Faro Health (6 of 6). Among large-cap technology employers, NVIDIA Corporation (7 of 40 QA postings, 17.5%), Waymo (6 of 20, 30%), and Cisco Systems (6 of 28, 21%) are the most AI-heavy. PricewaterhouseCoopers (8 of 27, 30%) is the highest-density Big Four name in QA, and Royal Bank of Canada (7 of 21, 33%) is the leading financial-services employer. If you want to optimize for AI exposure in your next QA role, those second-cluster firms (and the AI-native consultancies above) are the obvious targets.

How Should QA Engineers Use This Data in Their Job Search?

Three things follow from the data that you can act on this quarter.

First, treat AI as an extension of your QA stack, not a separate career path. The data shows AI requirements appearing inside generalist QA Engineer postings, not just inside AI-specialist QA roles. That means a tester who adds LLM evaluation, prompt-based regression design, and one AI coding assistant is competing for higher-paying generalist roles, not switching tracks. Practice the systems side of AI testing (eval harnesses, hallucination scoring, agent trace validation) the same way you practiced API regression suites a few years ago. Our AI mock interviews include scenarios for designing tests against LLM features, agentic workflows, and RAG pipelines, with feedback calibrated to what hiring managers at AI-heavy companies are actually asking.

Second, drill the specific concepts that recur in the postings. LLM evaluation, AI Agent test design, prompt engineering as a testing technique, vector-retrieval verification, and AI-assisted test authoring come up across the AI-asking QA postings. The Question Bank groups questions by topic so you can drill these one at a time rather than wandering through generic interview prep. Pair the conceptual drilling with one or two real projects on GitHub: an open-source eval harness for an LLM feature, or a test suite that uses an AI assistant to maintain itself. Recruiters at AI-heavy QA employers value “built with” over “read about”, and a portfolio link compresses the screen.

Third, filter your search and follow the geography. Generic “QA Engineer” feeds will be 95.7% non-AI roles. If your goal is to find the AI premium, search the job board with AI-skill filters explicitly: QA Engineer + LLMs and QA Engineer + GitHub Copilot are good starting points. If you are open to remote-friendly consulting work, the full QA Engineer feed on InterviewStack.io is the right place to scan global capability-center listings, which currently lead the AI adoption rate by a wide margin. For company-specific interview structures, the preparation guides index covers the consulting and tech-firm processes that show up most often in AI-leaning QA hiring.

FAQ

Q. How is AI changing the QA Engineer role in 2026?

AI now appears in 6.1% of all QA Engineer postings (998 of 16,376 analyzed) and 4.3% specifically mention new-wave generative AI tools like LLMs, AI Agents, and GitHub Copilot. The shift is happening on two fronts: postings that ask QA Engineers to test AI-powered products, and postings that expect QA Engineers to use AI assistants to write and maintain tests. Adoption is still well below Software Engineering, but the US salary premium (about 45% over the non-AI baseline) is the largest we have measured across our AI-shift analyses to date, including Software Engineering.

Q. What is the salary premium for QA Engineers with AI skills?

Among US postings with disclosed base salary, QA Engineers with new-wave AI skills earn a median $119,000 (n=75), versus $82,041 (n=3,264) for postings without AI. That is a $36,959 premium, or roughly 45% above the non-AI baseline. Equity and bonus are not disclosed in postings, so total compensation at top employers runs higher than these numbers.

Q. Which AI skills appear most often in QA Engineer postings?

Machine Learning leads at 2.8% (458 postings), reflecting older test-automation work against ML systems. The new-wave generative AI stack follows: LLMs (1.4%, 234), AI Agents (1.4%, 222), Generative AI (1.2%, 189), AI-Assisted Development (0.6%, 91), GitHub Copilot (0.5%, 87), Prompt Engineering (0.5%, 86), and ChatGPT (0.5%, 84). Specific frameworks like LangChain (54) and RAG (49) sit further down the long tail.

Q. Are QA Engineer roles becoming entry-level AI jobs?

No. AI adoption is highest in Staff QA Engineer postings (7.3%, 39 of 535) and Senior postings (4.8%, 487 of 10,123). Junior postings sit at the bottom at 1.5%, and Entry-level matches Mid-level at 3.6%. The pattern is clear: companies want experienced QA Engineers to lead the AI work, not entry-level hires to learn it on the job.

Q. Which industries hire the most AI-aware QA Engineers?

Professional services leads dramatically at 31.3% AI adoption (45 of 144 postings), more than double the next group. Technology follows at 12.3% (149 of 1,212), then pharmaceutical (8.6%), software (7.7%), finance (7.3%), and healthtech (7.1%). The high professional-services number reflects consulting firms staffing AI testing engagements for client companies that have not yet built the capability in-house.

Q. Is the AI shift in QA bigger in some countries than others?

Yes, and the US is not in the lead. Poland posts AI-related QA roles at 18.1% (37 of 204), Mexico at 10.0% (35 of 350), and India at 9.9% (139 of 1,399), all well above the US rate of 2.1% (143 of 6,659). Most of that international volume flows through global capability centers and software-services firms serving US and UK clients, where AI testing engagements are concentrated.

Q. Should QA Engineers learn AI skills in 2026?

The salary data says yes if you are aiming for senior or staff roles. A $36,959 US base premium is large, and AI demand clusters in the senior tier (4.8%) and staff tier (7.3%) where the salary curve already steepens. Practical priorities: get fluent with one AI coding assistant for test authoring, understand LLMs and AI Agents well enough to design tests against them, and learn Prompt Engineering as a testing technique (it shows up in 86 postings already).

Final Thoughts

The QA Engineer title in 2026 is not yet being rewritten the way Software Engineer has been, but the leading edge is clearly visible. Roughly one in twenty QA postings already asks for some form of AI fluency, and the small group of QA Engineers who can credibly answer that ask is being paid a premium large enough that the rest of the field cannot ignore it for long. The testers who treat LLM evaluation and AI-assisted authoring as an extension of their existing automation discipline, rather than as a separate career, will track with where the role is heading next.

I Investigated 986,317 retro cards. Teams aren’t as negative as you think 🤔

columns

Every retro I’ve ever run starts the same way. Screen shared, columns loaded, and a long quiet second where everyone waits for someone else to drop the first card.

So I got curious about what actually lands on those boards. We host a lot of retrospectives on Kollabe, so I pulled the aggregate: 79,868 retros, 986,317 cards. Anonymised, counted, nothing fancy.

The thing that surprised me most: retros have a reputation as a place to air what’s broken, and the data says the opposite.

Teams write almost twice as much about what’s going right. The “What Went Well” column collects 5.4 cards on average. Its “What we want to improve” neighbour barely clears 3. “Glad” beats “Mad” by more than two to one. Whatever the format, the positive side fills up first.

And the loneliest column in agile? “Stop.” At 1.5 cards on average, it’s the one teams reach for least. Starting new things is easy. Admitting you should stop doing something, apparently, is not.

columns

The textbook format isn’t the popular one. Start, Stop, Continue is the one everyone learns. But the single most-used format is the plainer What Went Well / What Went Wrong / What We Want to Improve, ahead of Start/Stop/Continue and Glad/Sad/Mad. We don’t pre-select a format for anyone, so that’s a real preference rather than a default nudge.

Friday is retro day. Measured by when a board actually gets filled in (not when it was created the day before), Friday is the most popular day to run a retro, just ahead of Tuesday. Teams reflect at the end of the sprint and the end of the week.

Action items are where good intentions go to rest. A typical retro produces about three of them. Only around one in eight ever gets marked done. I’d love to say my own team’s board looks better than that. It does not.

There’s more in the full breakdown, with charts for formats, columns, timing, kudos and reactions: Retrospective Statistics 2026.

Popular formats

A rule of thumb I’ve started using: if your retros only ever fill the “what’s broken” column, you’re probably running a complaints session, not a retrospective. The good stuff is a signal too. It tells you what to protect when the next reorg or deadline comes for it.

The honest caveat: this is data from teams who chose Kollabe, so it skews toward teams who like running structured retros in the first place. Your mileage will vary. But a million cards is a lot of cards, and the positivity gap holds across every format we looked at.

If you want to run one this week, Kollabe retros are free to start. Or grab whatever board you already use, add a real “Stop” column, and watch how empty it stays.

Go Fiber vs Gin vs Echo: what I chose after building a 1,600-article platform

I have opinions about Go web frameworks now. I did not want to have opinions. I wanted to pick something, build the thing, and move on. Instead I spent three weeks evaluating options before starting, switched frameworks once during development, and now have a production system with strong views about what worked and what did not.

This is not a benchmark post. I do not care that Fiber handles 400k requests/second in a synthetic test. I care that I can render a server-side HTML template with a dynamic nav, handle file uploads, and get useful error messages when I forget a closing brace in a template. Real usage criteria.

What I built

A cybersecurity consulting site with 1,600+ articles, full-text search via Meilisearch, a custom SEO scoring engine, PDF generation hooks, an admin interface with article CRUD, and a handful of API endpoints for search and lead capture. Server-side rendered HTML using Go’s html/template. No React, no SPA. Just fast, boring HTML.

Traffic is moderate (a few thousand visits per day), but the codebase complexity is real: about 8,000 lines of Go across handlers, middleware, search integration, and a background task runner.

Gin: the safe choice I did not take

Gin is the most popular Go web framework by GitHub stars and the most likely answer when you ask “what Go framework should I use” on Reddit. It is mature, the ecosystem is large, and you will find a StackOverflow answer for almost any problem.

My issue with Gin was ergonomics for HTML rendering. Gin’s template support exists, but it felt bolted on. The context type (*gin.Context) has different method names than you might expect coming from other frameworks, and the error handling model — where you call Abort() to stop handler chain execution — tripped me up repeatedly. It works, but the mental model did not click for me.

Gin also uses net/http underneath, which is the standard library. That is not a criticism — it means Gin integrates cleanly with any middleware written for net/http. But Fasthttp, which Fiber uses, is measurably faster for I/O-heavy workloads and the difference shows up in time-to-first-byte on slow connections.

I would reach for Gin if I were building a JSON API for a service that will be maintained by many developers over many years. The ecosystem and community are unmatched.

Echo: the cleanest API

Echo has the best-designed API of the three. Routes are clean, the middleware interface is simple, context methods are named sensibly, and the error handling model (returning errors from handlers, which Echo routes to a centralized error handler) is genuinely good.

// Echo error handling — clean and centralized
e.HTTPErrorHandler = func(err error, c echo.Context) {
    code := http.StatusInternalServerError
    message := "Internal Server Error"
    if he, ok := err.(*echo.HTTPError); ok {
        code = he.Code
        message = fmt.Sprintf("%v", he.Message)
    }
    c.JSON(code, map[string]string{"error": message})
}

Echo’s template rendering also requires a small adapter (Echo uses its own Renderer interface rather than html/template directly), but the adapter is 10 lines of code and well-documented.

Where I found Echo lacking: the community is smaller than Gin, and when I hit an unusual problem with template caching behavior during development, I could not find anyone who had solved exactly the same thing. I ended up reading Echo source code, which is fine but not ideal when you are trying to move fast.

Fiber: what I chose and why

I chose Fiber. The primary reason was familiarity: Fiber’s API is intentionally modeled after Express.js, and I had spent years building things with Express. The route handler signature, the middleware pattern, the way you access request parameters — all of it mapped onto existing muscle memory.

// Fiber route handler — the middleware chain feels natural
func SetupRoutes(app *fiber.App, db *sql.DB, meili *meilisearch.Client) {
    // Middleware stack
    app.Use(logger.New())
    app.Use(compress.New())
    app.Use(limiter.New(limiter.Config{
        Max:        100,
        Expiration: 1 * time.Minute,
    }))

    // Public routes
    app.Get("/", handlers.HomePage)
    app.Get("/articles/:slug", handlers.ArticlePage)
    app.Get("/api/search", handlers.SearchArticles)

    // Admin group with auth middleware
    admin := app.Group("/admin", middleware.AuthRequired)
    admin.Get("/", handlers.AdminDashboard)
    admin.Post("/articles", handlers.CreateArticle)
    admin.Put("/articles/:id", handlers.UpdateArticle)
    admin.Delete("/articles/:id", handlers.DeleteArticle)
}

Fiber’s html/template integration is first-class. You pass a template engine to fiber.Config and it handles caching, reloading in development, and template function maps without needing adapters.

engine := html.New("./templates", ".html")
engine.AddFunc("formatDate", func(t time.Time) string {
    return t.Format("2 January 2006")
})

app := fiber.New(fiber.Config{
    Views:       engine,
    ViewsLayout: "layouts/main",
})

Fasthttp underneath means lower memory allocation per request, which matters for a server with 4GB of RAM serving both the web application and Meilisearch simultaneously.

The honest criticism of Fiber

Fiber has a real gotcha that has bitten me twice: you cannot pass a *fiber.Ctx to a goroutine.

Fiber reuses context objects for performance (Fasthttp’s zero-allocation design). If you start a goroutine inside a handler and pass the context to it, by the time the goroutine runs, the context has been recycled for another request. You get corrupted data silently.

// This will corrupt data — ctx is reused after handler returns
app.Post("/articles", func(ctx *fiber.Ctx) error {
    go func() {
        processInBackground(ctx) // WRONG — ctx is invalid here
    }()
    return ctx.SendStatus(202)
})

// Correct — copy what you need before spawning the goroutine
app.Post("/articles", func(ctx *fiber.Ctx) error {
    bodyBytes := make([]byte, len(ctx.Body()))
    copy(bodyBytes, ctx.Body())
    userID := ctx.Locals("user_id").(int)

    go func() {
        processInBackground(bodyBytes, userID) // safe
    }()
    return ctx.SendStatus(202)
})

This is documented, but it is the kind of thing that only becomes obvious after you spend an afternoon debugging a race condition that looks completely unrelated. If you are building something with heavy goroutine usage from request handlers, this is a significant ergonomic cost.

The actual decision factors

Factor Gin Echo Fiber
HTML template integration Adequate Good (with adapter) Excellent
Express-like API No Somewhat Yes
Community size Large Medium Medium
Error handling model Confusing Best Good
Goroutine safety Safe Safe Careful required
Performance (Fasthttp) No No Yes

For a server-side rendered content platform with moderate traffic and one developer maintaining it: Fiber. For a team JSON API: Gin. For clean code you are proud of: Echo.

The article library running on this stack currently serves 1,600+ pieces of content. Fiber has not been the bottleneck once. The database has.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish security hardening checklists for FortiGate, Palo Alto, Active Directory, and more — free PDF and Excel.

I built a Telegram Twitter auto-poster — here’s what OAuth 1.0a actually requires

I run a cybersecurity consulting firm. We publish a lot of content — threat intel, CVE summaries, analysis — first on a Telegram channel, then cross-posted to Twitter/X for reach. After the tenth time manually copying a post at 8am, I decided to automate it.

The result: a Python bot that reads a Telegram channel via the Bot API, filters for posts worth sharing, and pushes them to Twitter. It runs every 2 hours via cron. It’s capped at 400 tweets per month to stay on the free API tier. At 3 tweets per run, that’s exactly 360 per month — enough margin for reruns and edge cases.

Building it took maybe 4 hours. Debugging OAuth took 3 of those.

The deceptively simple problem

Twitter API v2 is the current API. It uses OAuth 2.0 Bearer tokens for read operations, and OAuth 1.0a for write operations (posting tweets). That’s annoying but fine — I’d dealt with OAuth 1.0a before.

The wrinkle: media uploads still go through the v1.1 endpoint (api.twitter.com/1.1/media/upload.json), not v2. And v1.1 uses OAuth 1.0a. So even if you’d rather forget OAuth 1.0a exists, you can’t. You need it for image posts.

The Twitter docs don’t exactly shout this at you. They mention it, but it’s buried under the v2 migration guides and easy to miss if you’re building fresh instead of migrating.

What OAuth 1.0a actually signs

Here’s where I lost two hours.

OAuth 1.0a works by building a signature from a concatenation of: the HTTP method, the base URL, and all the request parameters — both OAuth parameters and request body parameters — sorted alphabetically, percent-encoded, and joined with &. Then you HMAC-SHA1 that with your consumer secret and token secret.

The part the docs gloss over: for application/x-www-form-urlencoded requests, body parameters must be included in the signature base string. If you’re sending media_data=<base64> in the body, that parameter has to be in the signature. Skip it, and you get a 401 with {"code": 32, "message": "Could not authenticate you."} — the most useless error message in any API.

Here’s the signing function I ended up with:

import hmac
import hashlib
import base64
import time
import uuid
from urllib.parse import quote, urlencode

def build_oauth_header(method, url, consumer_key, consumer_secret,
                       token, token_secret, body_params=None):
    oauth_params = {
        "oauth_consumer_key": consumer_key,
        "oauth_nonce": uuid.uuid4().hex,
        "oauth_signature_method": "HMAC-SHA1",
        "oauth_timestamp": str(int(time.time())),
        "oauth_token": token,
        "oauth_version": "1.0",
    }

    # THIS is the part the docs bury: body_params go into the signature
    all_params = {**oauth_params}
    if body_params:
        all_params.update(body_params)

    # Sort all params, percent-encode keys and values
    sorted_params = sorted(
        (quote(str(k), safe=""), quote(str(v), safe=""))
        for k, v in all_params.items()
    )
    param_string = "&".join(f"{k}={v}" for k, v in sorted_params)

    signature_base = "&".join([
        method.upper(),
        quote(url, safe=""),
        quote(param_string, safe=""),
    ])

    signing_key = f"{quote(consumer_secret, safe='')}&{quote(token_secret, safe='')}"
    signature = base64.b64encode(
        hmac.new(signing_key.encode(), signature_base.encode(), hashlib.sha1).digest()
    ).decode()

    oauth_params["oauth_signature"] = signature

    header_parts = ", ".join(
        f'{quote(k, safe="")}="{quote(v, safe="")}"'
        for k, v in sorted(oauth_params.items())
    )
    return f"Authorization: OAuth {header_parts}"

The call that bit me: I was passing body_params=None for the media upload because I thought the body was handled separately by the requests library. It is — but you still have to include those same params in the signature calculation. The body and the signature are separate concerns.

The rate limit math

Twitter’s free API tier gives you 500 write operations per month. I set a hard cap of 400 in the script to leave buffer. The cron runs every 2 hours (12 times per day, 360 times per month), posting a maximum of 3 tweets per run.

400 tweets / 30 days = 13.3/day
13.3/day / 12 runs = 1.1 tweet/run average

In practice, some days have nothing worth posting, some days have 5 things. Setting the per-run max to 3 and tracking monthly usage in a local SQLite counter keeps it safe. If the monthly counter is above 380, the bot skips posting and logs it.

This is more reliable than trusting the Twitter API to return sensible errors when you hit limits — their rate limit error messages are also not great.

What the Telegram side looks like

Telegram’s Bot API is, in contrast, a pleasure to use. You forward messages from the channel to a dedicated bot, and poll getUpdates to retrieve them. No OAuth. Bearer token in the URL. JSON responses that actually make sense.

The filter logic: skip messages shorter than 100 characters (usually reactions or single-link drops), skip messages that are pure retweets or already have the Twitter URL pattern, prefer messages with media for higher engagement.

One gotcha: Telegram’s getUpdates gives you an update_id field. You have to track the last processed update_id and pass it as offset on the next call, or you’ll re-process every message from the beginning on every run. I lost this to a file I forgot to persist across container restarts — spent 20 minutes wondering why the bot was reposting 3-week-old content.

Honest take on OAuth 1.0a

OAuth 1.0a is not good. The signature mechanism is fiddly, the error messages when you get it wrong are deliberately unhelpful (presumably as a security measure, though it mostly just wastes developer time), and the interaction between different content types and what has to be in the signature base is genuinely underspecified in the RFC.

OAuth 2.0 Bearer tokens are significantly better for developer experience. The fact that Twitter still requires OAuth 1.0a for writes — and v1.1-era endpoints for media — suggests some infrastructure decisions were made a long time ago and are very expensive to change.

If you’re building against the Twitter API, use a library like tweepy for the OAuth layer. I chose not to for this project because I wanted to understand what was happening, and because the bot’s requirements were narrow enough that the added dependency felt like overkill. That’s a valid choice, but go in knowing you will spend time on the signature generation that you would not spend otherwise.

The bot has been running for three months without issue. The cron, the SQLite counter, and the Telegram offset tracking all work. Every 2 hours, relevant security news goes from Telegram to Twitter without me touching anything. That part is genuinely satisfying.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists for FortiGate, Palo Alto, pfSense, Sophos, Active Directory and more — PDF and Excel.

pfSense vs commercial firewalls: what I learned auditing 200+ networks

After auditing over 200 networks across SMBs, mid-market companies, and a handful of larger organizations, I have a fairly strong opinion about the pfSense debate: it’s the wrong question. The right question is “who is maintaining this firewall?”

That said, let me give you the honest comparison I wish existed when I was starting out.

What pfSense CE actually is

pfSense Community Edition is a FreeBSD-based firewall/router. It’s free, it’s open source, and it’s genuinely capable. The feature set covers everything a small-to-medium business needs: stateful packet filtering, NAT, VPN (OpenVPN, WireGuard, IPsec), traffic shaping, VLAN support, high availability with CARP, and a plugin ecosystem that adds IDS/IPS (Suricata), DNS filtering (pfBlockerNG), and more.

It’s not a toy. I’ve seen pfSense deployments protecting 500-user organizations with complex network topologies, and they were fine — when they were properly configured and maintained.

I’ve also seen pfSense deployments that were running firmware from 2019 with the admin interface exposed to the internet on default credentials. Both of these were labeled “we use pfSense” in the asset inventory.

The audit view: what actually fails

When I run a firewall audit — checking ruleset logic, egress filtering, administrative access controls, firmware currency, logging configuration — the failure modes split cleanly by category, not by vendor.

pfSense failures I see most often:

  • Outdated firmware (often 1-2 major versions behind)
  • No egress filtering — the default is “allow all out”
  • Admin interface accessible from the WAN or from the general LAN without MFA
  • pfBlockerNG installed but never updated
  • No log forwarding — alerts go nowhere

Commercial firewall failures I see most often:

  • Default admin credentials unchanged (yes, even on FortiGate)
  • Licenses expired, so threat intelligence feeds are stale
  • Rules accumulated over years with no cleanup — “allow any any” buried 50 rules deep
  • Configuration done by the VAR at deployment, never touched since

The truth is that commercial firewalls don’t protect you from operational neglect. They just have a vendor you can call when something goes wrong — which is valuable, but it’s not a security control.

Where pfSense wins

Cost. pfSense CE is free. Hardware can be a used Protectli box for $200 or a Netgate appliance if you want something supported. For a 50-person company, the difference between pfSense on a $300 box and a FortiGate 100F starts at around $2,500 for hardware plus $800/year for threat feed subscriptions. Over three years, that’s $4,900 vs. roughly $800 total for pfSense. That delta matters for SMBs.

WireGuard integration. pfSense added native WireGuard support and it works well. Setup is straightforward, performance is excellent, and it’s significantly simpler to configure than IPsec. For remote access VPN, this is now my recommended choice for pfSense deployments.

pfBlockerNG. This plugin does IP and domain-based blocking using community threat feeds — essentially what commercial firewalls charge $800/year for in their “advanced threat protection” subscriptions. The feed quality is different, but for a budget-constrained SMB, pfBlockerNG configured properly is genuinely useful.

Transparency. You can look at what pfSense is actually doing. The ruleset is visible, the logs are accessible, and if something breaks you can debug it. Commercial firewalls often abstract this away behind a GUI that hides complexity but also hides what’s actually happening.

Here’s a quick diagnostic I run on every pfSense engagement:

# Check state table and active rules from the pfSense shell
pfctl -sa | head -50

# Check for rules with "any" as source or destination
pfctl -sr | grep -E "from any|to any"

# Check firmware version and available updates
pkg version -v | grep pfSense

# Check if admin interface is accessible from WAN
# (run from outside the network)
curl -sk https://<wan-ip>/ -o /dev/null -w "%{http_code}"
# 200 here is a problem

Where commercial firewalls win

Support SLA. When the firewall goes down at 11pm on a Friday before a major client presentation, “you can post on the Netgate forum” is not the same as “call this number and a TAC engineer picks up.” For organizations where network downtime has real revenue impact, that support contract has value.

Threat intelligence. FortiGate’s FortiGuard, Palo Alto’s threat feeds, Sophos’s Synchronized Security — these are genuinely better than community feeds. Not by an order of magnitude, but better. If you’re in a regulated industry or handling sensitive data, that gap matters.

Compliance documentation. Auditors like documented vendor relationships. “We use FortiGate with an active support contract” checks a box that “we run pfSense on a Protectli box” does not, even if the pfSense deployment is technically better configured.

SSL inspection at scale. Commercial firewalls handle SSL/TLS decryption more gracefully at high throughput. pfSense can do it but it’s not where it shines.

The honest decision matrix

Factor pfSense CE Commercial (FortiGate/Sophos/etc.)
Budget Under $500 all-in $2k–$15k depending on size
Internal expertise Needs someone who knows BSD/networking GUI-driven, easier for generalists
Support requirements Community only (or paid Netgate) Vendor TAC available
Compliance pressure Harder to document Easier for auditors
Threat feed quality Good with pfBlockerNG Better with subscriptions
Update cadence Requires manual attention Auto-update options available
Remote access VPN Excellent (WireGuard) Good (vendor-specific clients)

My general recommendation after 200+ audits: pfSense is the right choice for budget-constrained SMBs with at least one person who can and will maintain it. If that person doesn’t exist, buy a commercial firewall with a support contract. A neglected pfSense is worse than a commercial firewall with default configuration, because at least the commercial vendor will reach out about expired licenses.

For organizations with 250+ users or in regulated sectors (finance, healthcare), go commercial. The support SLA and compliance documentation justify the cost.

If you’re running pfSense and want to audit your own configuration, I’ve published a detailed pfSense firewall hardening checklist that covers ruleset review, egress filtering, admin interface hardening, logging configuration, and pfBlockerNG setup. There’s also a Sophos SFOS hardening checklist if you’re on the commercial side. Both are free — PDF and Excel.

The full security checklists library has 17 checklists total covering firewalls, Active Directory, Windows, Linux, DNS, and more.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists for FortiGate, Palo Alto, pfSense, Sophos, Active Directory and more — PDF and Excel.

I indexed 500+ security terms into a searchable glossary — what I learned about knowledge bases

About eight months ago I decided that the cybersecurity site I run needed a proper glossary. Not a static page with 50 definitions, but a real searchable knowledge base — terms organized by category, each with a proper definition, related terms, difficulty level, and a clean URL.

We’re now at 500+ terms across 8 categories: Active Directory (102 terms), General Security (101), AI Security (working toward 100), Hacking (working toward 100), Compliance, Cloud Security, DevSecOps, Forensics, and OT/ICS. The backend is MySQL, the frontend uses Meilisearch for full-text search, and the whole thing is part of a Go Fiber application.

Here’s what I actually learned building it — not the “knowledge bases are great for SEO” takes, but the technical and editorial problems I didn’t anticipate.

Slug uniqueness is harder than you think

Every glossary term gets a URL slug. Simple enough. The problem is that cybersecurity is full of acronyms, and acronyms collide.

“APT” could be Advanced Persistent Threat or Advanced Package Tool (Linux). “CA” is Certificate Authority or Certification Authority (same thing, two common names). “IR” is Incident Response or Infrared. “SOC” is Security Operations Center or System-on-Chip.

My initial slug generation was naive:

import re
import unicodedata

def slugify(text):
    text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode()
    text = re.sub(r'[^ws-]', '', text.lower())
    return re.sub(r'[-s]+', '-', text).strip('-')

This works fine until you try to insert “Advanced Persistent Threat” and “Advanced Package Tool” and get two slugs that both want to be apt. Then you add the definition for “Active Directory” and it wants ad, which you’ve already given to “Access Denied.”

My solution was a suffix-based disambiguation system: if a slug already exists, append the category abbreviation (apt-securite vs apt-linux). If that also conflicts, append a counter. The database enforces uniqueness with a UNIQUE constraint:

CREATE TABLE glossary_terms (
    id          INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    term        VARCHAR(255) NOT NULL,
    slug        VARCHAR(255) NOT NULL UNIQUE,
    category    ENUM('general','active-directory','ia','hacking',
                     'conformite','cloud','devsecops','forensics','ot') NOT NULL,
    definition  TEXT NOT NULL,
    difficulty  TINYINT UNSIGNED DEFAULT 1 COMMENT '1=beginner, 2=intermediate, 3=expert',
    related     JSON COMMENT 'array of related term slugs',
    sources     JSON COMMENT 'array of {title, url} objects',
    created_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    INDEX idx_category (category),
    INDEX idx_difficulty (difficulty),
    FULLTEXT INDEX ft_term_def (term, definition)
);

The UNIQUE constraint on slug means slug conflicts fail hard at insert time, which is the right behavior — better a failed insert I can handle than a silent overwrite of an existing term.

Category taxonomy matters more than definitions

Early on I focused on writing good definitions. What I should have focused on was getting the category structure right first.

My original categories were too broad. “Security” as a category is meaningless when your entire glossary is about security. I split it five times before landing on the current structure. The problem with restructuring categories after you have 200 terms in the database is that you have to re-review every term’s placement, update all the category-filtered search indexes, and regenerate the category landing pages.

The Active Directory category ended up being my most visited by a significant margin — 102 terms, well-defined scope (everything related to AD, LDAP, Kerberos, GPO, LDAP injection, pass-the-hash, BloodHound, etc.), and a clear audience (sysadmins and pentesters). Category coherence matters for readers and for how search clusters results.

A lesson I’d apply to any knowledge base: design your taxonomy with at least 10 example terms per category before committing to it. If you can’t generate 10 clear examples without overlap, your categories are wrong.

LLM-generated definitions need human review — especially for CVEs

I used an LLM to draft a significant portion of the definitions. At the scale of 500+ terms, writing every definition from scratch is not feasible for a small team. The drafting speed is genuinely useful.

The problem is accuracy variance. For well-established terms with lots of training data — RSA, TLS handshake, SQL injection, VLAN — the drafts were good with minor edits needed. For specific CVEs, niche attack techniques, or recent additions (post mid-2024 anything), the quality dropped sharply. The model would describe the vulnerability pattern correctly but attribute wrong CVSSv3 scores, wrong affected versions, or wrong discovery timeline.

My review process became: auto-accept definitions for general/conceptual terms after a quick read, mandatory manual fact-checking for anything CVE-related or with version-specific technical claims. The CVE definitions specifically need a cross-reference against the NVD entry. This sounds obvious in retrospect.

For the AI security category, the term landscape itself is still evolving fast enough that definitions needed to be written fresh rather than drafted — the concepts are too recent and too inconsistently defined across sources for an LLM to have reliable training signal on them.

Search over a glossary needs fuzzy matching

A static MySQL FULLTEXT index works fine until users search for “kerberosting” (missing an ‘a’), “phising” (one ‘s’), or “ransomwar” (trailing character dropped). These are real queries from the logs. Exact-match fulltext search returns nothing.

Meilisearch handles this much better out of the box. Its typo-tolerance is configurable, and for a glossary where almost every query is a technical term, the right settings are:

  • minWordSizeForTypos.oneTypo: 4 — allow one typo for words of 4+ characters
  • minWordSizeForTypos.twoTypos: 8 — allow two typos for longer words
  • Set searchable attributes to ["term", "definition"] but rank term matches higher

The difference in user experience between exact-match and typo-tolerant search on a technical glossary is substantial. Someone who searches for “kerberosting” and gets no results doesn’t search again — they leave.

The parts that worked better than expected

Linking related terms was worth the effort. When a term has a related array pointing to 3-4 other slugs, readers navigate more than twice as many pages per session. For a knowledge base where depth of understanding matters (you can’t understand pass-the-hash without understanding NTLM hashes, which requires understanding authentication protocols), the links create a learning path rather than a dead end.

Difficulty levels were also useful in ways I didn’t fully anticipate. Filtering to “beginner” terms gives a coherent introduction path. The distribution ended up roughly 20% beginner, 60% intermediate, 20% expert — which felt right for an audience that’s mostly practitioners rather than newcomers.

The glossary is part of the broader content library at AYI NEDJIMI Consultants, where we also cover practical security topics — network hardening, compliance frameworks, threat analysis, and more. If you’re in the security space, the free hardening checklists are probably the most practically useful thing on the site — 17 checklists for FortiGate, pfSense, Sophos, Active Directory, Windows Server, Linux, DNS security, and others.

Building this taught me more about taxonomy design and search relevance tuning than about cybersecurity itself. The technical problems in a knowledge base project are mostly not the ones you expect.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists for FortiGate, Palo Alto, pfSense, Sophos, Active Directory and more — PDF and Excel.

LLM output validation: 5 patterns that actually work in production

LLMs are probabilistic text generators. In a notebook demo, that’s fine. In production, it means your pipeline will occasionally receive a Python dict where you expected JSON, a 900-word paragraph where you asked for three bullet points, or a hallucinated field name that breaks your downstream schema. This post is not about theory — it’s about five concrete patterns, each with working code, that handle these failures reliably.

The core problem

You’re calling an LLM API expecting structured output. The model has been prompted carefully. But over thousands of calls, you’ll see:

  • Malformed JSON (trailing commas, unquoted keys, markdown code fences wrapping the payload)
  • Responses that exceed or fall short of length constraints
  • Fields that exist in the schema but contain garbage (“confidence”: “I am quite certain”)
  • Duplicate entries in batch completions
  • The model evaluating its own output charitably when asked to self-check

Each pattern below addresses one failure mode.

import json
import re
import time
import hashlib
from openai import OpenAI

llm_client = OpenAI(
    api_key="your_api_key",
    base_url="https://api.your-llm-provider.com/v1",
)

def call_llm(messages: list[dict], model: str = "gpt-4o-mini",
             temperature: float = 0.3, max_tokens: int = 1000) -> str:
    response = llm_client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens,
    )
    return response.choices[0].message.content.strip()

Pattern 1: JSON schema validation with retry

Problem: The model returns valid JSON 98% of the time and something subtly broken the other 2%. Your parser crashes and you lose the request.

Bad solution: json.loads() with a bare except that returns None. You swallow errors silently and downstream code explodes later.

Good solution: Parse, validate against a schema, and retry with an error hint that tells the model exactly what went wrong.

import jsonschema

ARTICLE_SCHEMA = {
    "type": "object",
    "required": ["title", "summary", "tags", "difficulty"],
    "properties": {
        "title": {"type": "string", "minLength": 10, "maxLength": 120},
        "summary": {"type": "string", "minLength": 50},
        "tags": {"type": "array", "items": {"type": "string"}, "minItems": 1},
        "difficulty": {"type": "string", "enum": ["beginner", "intermediate", "advanced"]},
    },
    "additionalProperties": False,
}

def extract_json_from_response(text: str) -> str:
    """Strip markdown code fences if present."""
    match = re.search(r"```

(?:json)?s*([sS]*?)

```", text)
    if match:
        return match.group(1).strip()
    # Try to find raw JSON object
    match = re.search(r"{[sS]*}", text)
    if match:
        return match.group(0)
    return text

def call_with_json_schema(prompt: str, schema: dict,
                           max_retries: int = 3) -> dict:
    messages = [
        {"role": "system", "content": (
            "You are a data extraction assistant. "
            "Always respond with valid JSON matching the requested schema. "
            "No prose, no markdown fences, just the JSON object."
        )},
        {"role": "user", "content": prompt},
    ]

    last_error = None
    for attempt in range(max_retries):
        raw = call_llm(messages)
        json_str = extract_json_from_response(raw)

        try:
            data = json.loads(json_str)
            jsonschema.validate(instance=data, schema=schema)
            return data
        except json.JSONDecodeError as e:
            last_error = f"JSON parse error: {e}. Raw output was: {json_str[:200]}"
        except jsonschema.ValidationError as e:
            last_error = f"Schema validation failed: {e.message}"

        # Append error feedback and retry
        messages.append({"role": "assistant", "content": raw})
        messages.append({"role": "user", "content": (
            f"That response had an error: {last_error}n"
            "Please fix it and return only the corrected JSON."
        )})
        time.sleep(0.5 * (attempt + 1))  # back off slightly

    raise ValueError(f"Failed after {max_retries} attempts. Last error: {last_error}")

Pattern 2: Length constraint enforcement

Problem: You ask for a 2-sentence summary and get a paragraph. Or you ask for 500 words and get 80. Downstream rendering breaks.

Bad solution: Truncate with response[:500]. You cut mid-sentence and produce garbage.

Good solution: Measure, then retry with a correction hint that quantifies the delta.

def count_words(text: str) -> int:
    return len(text.split())

def call_with_length_constraint(prompt: str, min_words: int, max_words: int,
                                 max_retries: int = 3) -> str:
    messages = [
        {"role": "system", "content": (
            f"Write responses between {min_words} and {max_words} words. "
            "Count carefully before submitting."
        )},
        {"role": "user", "content": prompt},
    ]

    for attempt in range(max_retries):
        response = call_llm(messages, max_tokens=max_words * 2)
        word_count = count_words(response)

        if min_words <= word_count <= max_words:
            return response

        delta = word_count - max_words if word_count > max_words else min_words - word_count
        direction = "shorter" if word_count > max_words else "longer"
        hint = (
            f"Your response was {word_count} words. "
            f"It needs to be {abs(delta)} words {direction}. "
            f"Target: {min_words}–{max_words} words. Rewrite it."
        )
        messages.append({"role": "assistant", "content": response})
        messages.append({"role": "user", "content": hint})

    # Last resort: hard truncate/expand with note
    final = call_llm(messages, max_tokens=max_words * 2)
    words = final.split()
    if len(words) > max_words:
        return " ".join(words[:max_words])
    return final

Pattern 3: Regex-based field extraction as fallback

Problem: The model consistently wraps values in prose (“The severity is: HIGH”) instead of returning a clean value. JSON parsing fails; you can’t proceed.

Good solution: Regex extraction as a structured fallback — not a replacement for JSON, but a recovery layer when JSON fails.

FIELD_PATTERNS = {
    "severity": r"b(LOW|MEDIUM|HIGH|CRITICAL)b",
    "score": r"b(d+(?:.d+)?)s*(?:/s*10)?",
    "category": r"b(spam|phishing|legitimate|malware|unknown)b",
    "confidence": r"confidence[:s]+(d+(?:.d+)?)%?",
}

def extract_fields_with_regex(text: str,
                               fields: list[str]) -> dict:
    """
    Attempt to extract structured fields from prose output using regex.
    Returns None for fields that cannot be extracted.
    """
    result = {}
    text_upper = text.upper()

    for field in fields:
        pattern = FIELD_PATTERNS.get(field)
        if not pattern:
            result[field] = None
            continue

        match = re.search(pattern, text_upper if field == "severity" else text,
                          re.IGNORECASE)
        result[field] = match.group(1) if match else None

    return result

def classify_with_fallback(text_to_classify: str) -> dict:
    prompt = (
        f'Classify this text:nn"{text_to_classify}"nn'
        'Return JSON: {"category": "spam|phishing|legitimate", '
        '"severity": "LOW|MEDIUM|HIGH|CRITICAL", "confidence": 0-100}'
    )
    messages = [{"role": "user", "content": prompt}]
    raw = call_llm(messages, temperature=0.1)

    try:
        json_str = extract_json_from_response(raw)
        return json.loads(json_str)
    except (json.JSONDecodeError, ValueError):
        # Fallback: extract fields with regex
        extracted = extract_fields_with_regex(raw, ["category", "severity", "confidence"])
        extracted["_extraction_method"] = "regex_fallback"
        return extracted

Pattern 4: Confidence scoring via self-evaluation

Problem: The model answers confidently even when it’s guessing. You need a signal to route low-confidence answers to human review.

Key insight: Ask the model to evaluate its own answer in a separate call. Self-evaluation in the same call is biased upward.

def get_answer_with_confidence(question: str, context: str) -> dict:
    # Step 1: Generate answer
    answer_messages = [
        {"role": "system", "content": "Answer based strictly on the provided context."},
        {"role": "user", "content": f"Context:n{context}nnQuestion: {question}"},
    ]
    answer = call_llm(answer_messages, temperature=0.2)

    # Step 2: Evaluate in a separate call
    eval_messages = [
        {"role": "system", "content": (
            "You are an impartial evaluator. Assess answer quality strictly. "
            "Return JSON: {"confidence": 0-100, "issues": [list of concerns], "
            ""grounded": true/false}"
        )},
        {"role": "user", "content": (
            f"Question: {question}nn"
            f"Context provided:n{context}nn"
            f"Answer given:n{answer}nn"
            "Evaluate: Is this answer fully supported by the context? "
            "Are there unsupported claims? Score 0-100."
        )},
    ]
    eval_raw = call_llm(eval_messages, temperature=0.0)

    try:
        eval_data = json.loads(extract_json_from_response(eval_raw))
    except (json.JSONDecodeError, ValueError):
        eval_data = {"confidence": 50, "issues": ["evaluation_parse_failed"], "grounded": None}

    return {
        "answer": answer,
        "confidence": eval_data.get("confidence", 50),
        "issues": eval_data.get("issues", []),
        "grounded": eval_data.get("grounded"),
        "needs_review": eval_data.get("confidence", 50) < 70,
    }

Pattern 5: Deduplication across batch outputs

Problem: You process 50 documents in batch and ask the model to extract key entities from each. You get overlapping, near-duplicate entries that pollute your downstream data.

Good solution: Hash-based exact dedup combined with a lightweight similarity check for near-duplicates.

from difflib import SequenceMatcher

def deduplicate_outputs(items: list[str],
                         similarity_threshold: float = 0.85) -> list[str]:
    """
    Remove exact duplicates (hash) and near-duplicates (sequence similarity).
    Keeps the first occurrence of each unique item.
    """
    seen_hashes: set[str] = set()
    unique_items: list[str] = []

    for item in items:
        normalized = item.strip().lower()
        item_hash = hashlib.md5(normalized.encode()).hexdigest()

        if item_hash in seen_hashes:
            continue  # exact duplicate

        # Check near-duplicate against existing unique items
        is_near_dup = any(
            SequenceMatcher(None, normalized, existing.strip().lower()).ratio()
            >= similarity_threshold
            for existing in unique_items
        )

        if not is_near_dup:
            unique_items.append(item)
            seen_hashes.add(item_hash)

    return unique_items

def batch_extract_entities(documents: list[str], entity_type: str) -> list[str]:
    all_entities = []

    for doc in documents:
        messages = [
            {"role": "system", "content": (
                f"Extract all {entity_type} from the text. "
                "Return a JSON array of strings. Nothing else."
            )},
            {"role": "user", "content": doc},
        ]
        raw = call_llm(messages, temperature=0.1)
        try:
            entities = json.loads(extract_json_from_response(raw))
            if isinstance(entities, list):
                all_entities.extend(entities)
        except (json.JSONDecodeError, ValueError):
            pass  # log and continue — one bad doc shouldn't stop the batch

    return deduplicate_outputs(all_entities)

Putting it all together

These patterns compose. A production pipeline for classifying user-submitted content might chain them:

def robust_classify(text: str) -> dict:
    try:
        result = call_with_json_schema(
            prompt=f'Classify this text: "{text}"',
            schema={
                "type": "object",
                "required": ["category", "severity", "confidence"],
                "properties": {
                    "category": {"type": "string", "enum": ["spam", "phishing", "legitimate", "toxic"]},
                    "severity": {"type": "string", "enum": ["LOW", "MEDIUM", "HIGH", "CRITICAL"]},
                    "confidence": {"type": "number", "minimum": 0, "maximum": 100},
                },
            },
            max_retries=3,
        )
    except ValueError:
        # Pattern 3 fallback
        result = classify_with_fallback(text)

    # Pattern 4: flag for human review if uncertain
    result["needs_review"] = result.get("confidence", 100) < 65
    return result

These five patterns cover the vast majority of production failures. Start with Pattern 1 (JSON schema + retry) and Pattern 3 (regex fallback) — they handle 80% of output issues. Add Pattern 4 (self-evaluation) when you have a human review queue and need to route intelligently. For content pipelines like the moderation system described in practical security guides, Patterns 1 and 5 together eliminate most of the noise from batch LLM processing.

A practical guide to prompt engineering for structured data extraction

Extracting structured data from unstructured text is one of the most practical uses of language models in production. Advisory feeds, incident reports, job postings, legal documents — they all contain structured information buried in natural language. Getting that information out reliably requires more than a naive “respond in JSON” instruction.

This tutorial walks through the full stack: system prompt design, few-shot examples, chain-of-thought for ambiguous fields, JSON mode, and Pydantic validation with retry logic. The running example is CVE advisory extraction, which is genuinely hard because advisories vary wildly in format and verbosity.

What we are extracting

Given raw advisory text like this:

CERT-FR CERTFR-2025-AVI-0312
A critical vulnerability has been identified in Fortinet FortiGate versions
7.0.0 through 7.0.14. An unauthenticated remote attacker can exploit a
stack-based buffer overflow in the SSL-VPN web management interface to
execute arbitrary code. The CVSS v3.1 base score is 9.8 (Critical).
Affected versions: FortiOS 7.0.x < 7.0.15, 7.2.x < 7.2.9.
Apply the vendor patch immediately or disable SSL-VPN if patching is not
immediately possible.

We want this:

{
  "cve_id": null,
  "cert_id": "CERTFR-2025-AVI-0312",
  "cvss_score": 9.8,
  "severity": "Critical",
  "affected_product": "Fortinet FortiGate / FortiOS",
  "affected_versions": ["7.0.0–7.0.14", "7.2.x < 7.2.9"],
  "vulnerability_type": "Stack-based buffer overflow",
  "attack_vector": "Network (unauthenticated remote)",
  "remediation": "Apply vendor patch; disable SSL-VPN if patch cannot be applied immediately"
}

Setup

pip install openai pydantic tenacity python-dotenv
import os
import json
import re
from typing import Optional
from openai import OpenAI
from pydantic import BaseModel, Field, field_validator
from tenacity import retry, stop_after_attempt, wait_exponential

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

Step 1: Define the schema with Pydantic

Defining the output schema first forces you to be precise about what you actually need before you write a single prompt word.

class CVEExtraction(BaseModel):
    cve_id: Optional[str] = Field(
        None,
        description="CVE identifier if present, e.g. CVE-2025-12345"
    )
    cert_id: Optional[str] = Field(
        None,
        description="CERT advisory ID if present"
    )
    cvss_score: Optional[float] = Field(
        None,
        ge=0.0,
        le=10.0,
        description="CVSS v3.x base score as a float"
    )
    severity: Optional[str] = Field(
        None,
        description="Severity label: Critical, High, Medium, Low, or Informational"
    )
    affected_product: str = Field(
        description="Vendor and product name"
    )
    affected_versions: list[str] = Field(
        default_factory=list,
        description="List of affected version ranges"
    )
    vulnerability_type: str = Field(
        description="Technical class of the vulnerability"
    )
    attack_vector: str = Field(
        description="How the vulnerability is exploited (network/local/physical, auth required)"
    )
    remediation: str = Field(
        description="Recommended fix or workaround"
    )

    @field_validator("cve_id")
    @classmethod
    def validate_cve_format(cls, v):
        if v is not None and not re.match(r"CVE-d{4}-d{4,}", v):
            raise ValueError(f"Invalid CVE format: {v}")
        return v

    @field_validator("severity")
    @classmethod
    def validate_severity(cls, v):
        valid = {"Critical", "High", "Medium", "Low", "Informational", None}
        if v not in valid:
            raise ValueError(f"Invalid severity: {v}")
        return v

Step 2: System prompt design

The system prompt does the heavy lifting. Three principles:

  1. Give the model a role and a goal, not just instructions
  2. Specify output format explicitly — schema + example
  3. Handle ambiguity with explicit rules — what to do when a field is missing
SYSTEM_PROMPT = """You are a cybersecurity data extraction engine. Your task is to parse
security advisories and extract structured vulnerability information.

OUTPUT FORMAT: Return a single JSON object conforming exactly to this schema:
{
  "cve_id": string or null,
  "cert_id": string or null,
  "cvss_score": number (0.0–10.0) or null,
  "severity": "Critical" | "High" | "Medium" | "Low" | "Informational" | null,
  "affected_product": string (required),
  "affected_versions": array of strings,
  "vulnerability_type": string (required),
  "attack_vector": string (required),
  "remediation": string (required)
}

RULES:
- Return null for any field not present in the source text. Do NOT infer or guess.
- cvss_score must be numeric, not a string like "9.8 (Critical)".
- If the advisory lists multiple affected version ranges, list each as a separate array element.
- remediation should be a concise actionable summary, not a copy-paste of the full advisory.
- Return ONLY the JSON object. No explanation, no markdown fences, no commentary.
"""

Step 3: Few-shot examples

Few-shot examples dramatically improve consistency on edge cases. Include at least two: one clean advisory and one messy one where fields are implicit.

FEW_SHOT_EXAMPLES = [
    {
        "role": "user",
        "content": """Extract from this advisory:
CVE-2024-21762 affects FortiOS and FortiProxy. An out-of-bounds write
vulnerability (CWE-787) in SSL-VPN allows a remote unauthenticated attacker
to execute arbitrary code or commands via specially crafted HTTP requests.
CVSS: 9.6. Affected: FortiOS 6.0 all versions, 6.2 all versions, 6.4.0–6.4.14,
7.0.0–7.0.14, 7.2.0–7.2.7, 7.4.0–7.4.2. Workaround: disable SSL-VPN."""
    },
    {
        "role": "assistant",
        "content": json.dumps({
            "cve_id": "CVE-2024-21762",
            "cert_id": None,
            "cvss_score": 9.6,
            "severity": "Critical",
            "affected_product": "Fortinet FortiOS and FortiProxy",
            "affected_versions": [
                "FortiOS 6.0 all versions",
                "FortiOS 6.2 all versions",
                "FortiOS 6.4.0–6.4.14",
                "FortiOS 7.0.0–7.0.14",
                "FortiOS 7.2.0–7.2.7",
                "FortiOS 7.4.0–7.4.2"
            ],
            "vulnerability_type": "Out-of-bounds write (CWE-787)",
            "attack_vector": "Network (remote unauthenticated)",
            "remediation": "Disable SSL-VPN as workaround; apply vendor patch when available"
        }, indent=2)
    },
    {
        "role": "user",
        "content": """Extract from this advisory:
A researcher has disclosed a high-severity issue in Apache Struts. The flaw
allows attackers on the local network with valid credentials to escalate
privileges through a race condition in the file upload handler. No CVE assigned
yet. Vendor fix expected next quarter. Score: 7.5."""
    },
    {
        "role": "assistant",
        "content": json.dumps({
            "cve_id": None,
            "cert_id": None,
            "cvss_score": 7.5,
            "severity": "High",
            "affected_product": "Apache Struts",
            "affected_versions": [],
            "vulnerability_type": "Race condition / privilege escalation",
            "attack_vector": "Local network (authenticated)",
            "remediation": "No patch available; monitor vendor advisory for fix"
        }, indent=2)
    }
]

Step 4: Chain-of-thought for complex cases

Some advisories require reasoning to extract correctly — for example, when the CVSS score is described only in a table or when the affected product must be inferred from context. Add a CoT step before the final extraction on inputs flagged as complex.

def is_complex(text: str) -> bool:
    """Heuristic: long text or multiple products suggests complexity."""
    return len(text) > 800 or text.count("CVE-") > 1

def add_cot_prefix(text: str) -> str:
    return (
        "First, identify the key fields step by step:n"
        "1. Is there a CVE ID? If not, is there another advisory ID?n"
        "2. What is the CVSS score? What severity does it correspond to?n"
        "3. Which product and versions are affected?n"
        "4. What is the technical class of the vulnerability?n"
        "5. What does the attacker need (network access? credentials?)?n"
        "6. What is the fix or workaround?nn"
        "Now extract the JSON:nn"
        + text
    )

Step 5: Extraction with retry and validation

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10)
)
def extract_cve(advisory_text: str) -> CVEExtraction:
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        *FEW_SHOT_EXAMPLES,
    ]

    user_content = advisory_text
    if is_complex(advisory_text):
        user_content = add_cot_prefix(advisory_text)

    messages.append({"role": "user", "content": f"Extract from this advisory:n{user_content}"})

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        response_format={"type": "json_object"},  # JSON mode — forces valid JSON output
        temperature=0,  # deterministic extraction
    )

    raw = response.choices[0].message.content
    data = json.loads(raw)
    return CVEExtraction(**data)  # Pydantic validates types and constraints

The response_format={"type": "json_object"} parameter guarantees syntactically valid JSON. Pydantic then catches semantic errors (wrong types, out-of-range scores, malformed CVE IDs) and the tenacity decorator retries on any exception — network errors, validation failures, or parse errors alike.

Step 6: Batch processing

from pathlib import Path

def process_advisory_file(path: Path) -> dict:
    text = path.read_text(encoding="utf-8")
    try:
        result = extract_cve(text)
        return {"status": "ok", "file": path.name, "data": result.model_dump()}
    except Exception as e:
        return {"status": "error", "file": path.name, "error": str(e)}

def batch_extract(directory: Path) -> list[dict]:
    results = []
    for f in directory.glob("*.txt"):
        print(f"Processing {f.name}...")
        results.append(process_advisory_file(f))
    return results

Measuring extraction quality

Define a small labelled test set and track precision per field:

def evaluate(predictions: list[CVEExtraction], ground_truth: list[dict]) -> dict:
    fields = ["cve_id", "cvss_score", "severity", "affected_product", "vulnerability_type"]
    scores = {f: [] for f in fields}

    for pred, gt in zip(predictions, ground_truth):
        for field in fields:
            pred_val = getattr(pred, field)
            gt_val = gt.get(field)
            scores[field].append(1 if str(pred_val) == str(gt_val) else 0)

    return {f: sum(v) / len(v) for f, v in scores.items() if v}

What actually matters

A few lessons from running this in production on real advisory feeds:

  • Temperature = 0 is mandatory for extraction. Any randomness introduces inconsistency across identical inputs.
  • JSON mode alone is not enough — it prevents syntax errors but not semantic garbage. Always validate with Pydantic.
  • Few-shot examples outperform longer instructions for format compliance. Two well-chosen examples beat two paragraphs of “you must always…” text.
  • CoT is a latency trade-off — it adds tokens but reduces errors on complex multi-product advisories. Gate it with a heuristic.

Security teams processing vulnerability feeds at scale — like those using the advisory tracking systems described at AYI NEDJIMI Consultants — typically combine this extraction layer with a downstream enrichment step that cross-references extracted CVEs against asset inventories.

What to build next

  • Add a confidence score by asking the model to rate its own extraction (0–1 per field)
  • Stream results with stream=True for real-time processing of long advisories
  • Feed extracted data directly to a vector database for similarity search across historical advisories
  • Set up automatic reprocessing when Pydantic validation fails after all retries