Stop Letting AI Agents Go Rogue on Large Codebases: The large-scale-refactor Skill

A comprehensive guide to OpenSite AI’s open-source skill that gives AI coding agents enterprise-grade guardrails for migrations, multi-session refactors, and parallelized tasks.

We’ve all been there.

You ask an AI agent to “migrate my components to TypeScript.” An hour later you check back and it has renamed your props, refactored three utility functions, installed two new packages, reorganized your folder structure, and changed your ESLint config “while it was in there.”

The task was 40 files. It touched 140.

This is the core problem with AI agents on large, long-running tasks: they don’t have a scope reflex. Humans instinctively know when they’ve drifted. Agents don’t — and the bigger the task, the worse the drift gets.

The large-scale-refactor skill from OpenSite AI is a free, open-source agent skill that solves this by giving your AI agent a complete operating protocol for any task touching 50+ files — migrations, framework upgrades, codebase-wide renames, or anything running across multiple sessions or parallel agents.

This guide walks through everything: what the skill is, how every guardrail works, and a complete end-to-end walkthrough of a real JS→TS migration.

What Is an “Agent Skill”?

Before diving in, a quick note on terminology. An agent skill is a structured instruction set — loaded into the AI agent’s context — that defines how it should behave for a specific class of task. It’s not a library you install in your codebase. It’s a behavioral protocol for your agent.

The large-scale-refactor skill works with Claude Code, Codex, Cursor, GitHub Copilot, Qoder Quest, Factory Droid, and Devin Playbooks. On platforms that support automatic activation, the skill kicks in automatically when it detects patterns like "migrate * to" or "refactor * across" or when a task is estimated to touch 50+ files.

Install it once, then never think about it:

git clone https://github.com/opensite-ai/opensite-skills.git

For Claude Code, add it to your .claude/skills/ directory. For Cursor, drop it in your .cursor/skills/ directory. The SKILL.md frontmatter handles the rest.

The Core Insight: Agents Need a Scope Reflex

The skill’s design is built around one uncomfortable truth:

AI agents are pattern completers, not task completers. When they see something that could be improved, they improve it — even if that wasn’t the job.

This isn’t a bug in any particular model. It’s the fundamental nature of how these systems work. A language model trained to produce high-quality code will naturally drift toward “better” code whenever it has the opportunity.

The large-scale-refactor skill installs a scope reflex by making the agent ask a single question before every change:

“If I remove this change from the diff, does the task still fail?”

This is the Substitution Test (§ 2.2), and it is the single most important concept in the skill. If the task succeeds without a change, the change doesn’t belong in this PR.

The Eight Guardrails — A Deep Dive

The skill is organized into eight sections. Let’s go through each one in depth.

§ 1 — The Spec Gate: No Execution Without Approval

This is the foundational rule. Before the agent writes a single line of code, it must produce a written spec and wait for a human to explicitly approve it.

The spec format is precise by design:

TASK SPEC
=========
**Task Name**: js-to-ts-migration
**Date**: 2026-03-27
**Initiator**: Your Name

### What This Task Does
Convert all .js files in src/ to .ts/.tsx, adding TypeScript types.
No logic changes, no style changes, no dependency additions beyond @types/*.

### Explicit Scope Boundary
**IN SCOPE** (agent may touch):
- [x] File types: *.js files in src/components/, src/hooks/, src/utils/
- [x] Operations: Type annotations, file extension changes
- [x] Directories: src/components/, src/hooks/, src/utils/

**OUT OF SCOPE — DO NOT TOUCH**:
- [ ] CSS/styling, color tokens, theme configuration
- [ ] Business logic or algorithm changes
- [ ] package.json (except @types/*)
- [ ] Build configs, CI/CD, deployment
- [ ] node_modules/, config/, scripts/, public/

### Task Decomposition
1. Subtask A — src/components/common/ — 15 files
2. Subtask B — src/components/features/ — 22 files
3. Subtask C — src/hooks/ — 8 files
4. Subtask D — src/utils/ — 12 files
5. Subtask E — test files — 35 files

### Acceptance Criteria
- [ ] All in-scope .js files converted to .ts/.tsx
- [ ] All tests pass (or pre-existing failures documented)
- [ ] No files outside scope boundary modified
- [ ] Each subtask in its own reviewable commit

### Rollback Plan
Feature branch: refactor/js-to-ts-migration
Full rollback: git checkout main && git branch -D refactor/js-to-ts-migration

After generating this, the agent on Claude Code will output:

⏸ SPEC GATE: Please review and reply 'approved' to begin execution,
or provide corrections.

No code gets written until you say the word. The spec is your contract with the agent.

Why this matters: Most agent disasters happen in the first 10 minutes, before the human realizes the task is going sideways. The spec gate forces that realization to happen before any code is touched.

§ 2 — Scope Enforcement: Five Rules the Agent Cannot Break

Once execution begins, five rules govern every file touch and every line written.

2.1 The One Task Rule

The agent has exactly one job. Not one-and-a-half. Not “one plus reasonable improvements.” One.

When it notices a bug, a performance issue, or an architectural smell that isn’t in the spec, it does not fix it. It logs it to OBSERVATIONS.md and moves on:

## Observations (NOT acted upon — logged for human review)

| File | Observation | Severity |
|------|-------------|----------|
| src/api/user.ts | Possible N+1 query in fetchUsers | Medium |
| src/components/Button.tsx | Inline styles could use CSS vars | Low |

This file becomes a goldmine at the end of the task. You get a curated list of real technical debt, surfaced by an agent that touched every file, with zero action taken on any of it.

2.2 The Substitution Test

Before every change: “If I remove this from the diff, does the task still fail?” If the answer is no — don’t make the change.

2.3 No Emergent Systems

The agent is explicitly prohibited from creating anything that didn’t exist before the task began unless it’s a spec-defined output. No new utility libraries. No new abstractions. No folder reorganizations. No new config files.

If a new shared utility is genuinely necessary to avoid massive repetition, the agent proposes it in OBSERVATIONS.md and halts for approval before creating it.

2.4 Dependency Lockdown

No adding, removing, or upgrading dependencies unless explicitly listed in the spec (or @types/* for TypeScript migrations). Any dependency change is an automatic human checkpoint.

2.5 The 50-Line Net-New Code Threshold

If completing a single refactoring step requires writing more than 50 lines of net-new logic, the agent stops. This is a hard signal that something has gone wrong — either the task is being misunderstood, or the spec needs an architectural discussion.

A refactor should transform existing patterns, not invent new ones. 50 lines of net-new logic in a refactor is a red flag.

§ 3 — Execution Protocol: Atomic, Budgeted, and Drift-Checked

3.1 Pilot Batch First

Regardless of risk level, the first batch of any new refactor is limited to 10–20 files. This surfaces edge cases in the spec before you’ve committed to the pattern across 200 files.

3.2 File Diff Budgets by Risk Level

Risk Level Max Files per Session Review Cadence
Low (type renames, import fixes) 200 files End of session
Medium (logic-adjacent refactors) 50 files Every 25 files
High (framework migrations, API changes) 20 files Every 10 files

When a session hits its budget, it commits, pushes, and stops. Human reviews before the next session.

3.3 Parallel Agent Isolation

When running parallel agents (Qoder Quest’s Worktree mode, Factory Droid batch, multiple Devin instances), the rules are strict:

  • Every instance gets the approved spec as its first system message
  • Instances have non-overlapping, explicitly assigned file lists
  • No instance creates new shared abstractions
  • Instances do not observe each other’s output

3.4 Drift Detection Checkpoint (every N files)

At the configured cadence (or every 25 files by default), the agent performs a mandatory self-audit:

DRIFT CHECK
===========
Files touched so far: 47
Task: js-to-ts-migration

1. Does every changed file appear in the IN SCOPE list? YES
   Evidence: All files are in src/components/, src/hooks/, or src/utils/

2. Did I add any new files not defined in the spec? NO

3. Did I add, remove, or modify any dependency? NO

4. Did I make any change that fails the Substitution Test? NO

5. Did I create any new abstraction, utility, or system? NO

All answers NO — continuing.

If any answer is “yes,” it halts immediately and surfaces the issue before proceeding.

§ 4 — Human Checkpoints: Hard Stops That Cannot Be Talked Past

Every checkpoint in the skill is a hard stop. The agent does not reason its way around them or make “reasonable assumptions.” It halts and waits.

The checkpoint message format is structured to force clear thinking:

⏸ CHECKPOINT — dependency_required

**Trigger**: TypeScript compilation requires @types/react
**Context**: src/components/common/Button.tsx:3 - Cannot find module 'react'

**Options**:
  A. Add @types/react as devDependency (in spec allowance for TS migrations)
  B. Skip type checking for this batch (not recommended)
  C. Abort task and preserve current state for human review

**Recommendation**: Option A — this is explicitly allowed by the spec.

Awaiting instruction. No changes will be made until a response is received.

Checkpoint triggers include: spec gate, drift check failure, any out-of-scope file, any new dependency, any new abstraction, unexpected test failures, file budget reached, and spec ambiguity.

§ 5 — Output & Verification Requirements

5.1 The Change Manifest

After each subtask, the agent produces a CHANGE_MANIFEST.md:

CHANGE MANIFEST — subtask-a
==============================
Task: js-to-ts-migration
Completed: 2026-03-27T10:30:00Z
Files modified: 15
Files created: 0
Files deleted: 0

### Modified Files
| File | Change Type | Lines +/- | Notes |
|------|-------------|-----------|-------|
| src/components/common/Button.tsx | Type annotation | +12/-3 | Added Props interface |
| src/components/common/Modal.tsx  | Type annotation | +18/-2 | Added component types |

### Scope Compliance
- [x] All modified files were in the IN SCOPE list
- [x] No files created outside spec-defined outputs
- [x] No unauthorized dependencies added/removed
- [x] No new abstractions created
- [x] All drift checks passed

### Test Results
- Before: 450 passed, 12 failed
- After: 450 passed, 12 failed
- New failures: none

5.2 The Verification Sequence

Before marking any subtask complete, the agent runs this exact sequence and records the output:

# 1. Dependency check
git diff HEAD package.json package-lock.json yarn.lock Cargo.toml Gemfile*

# 2. New file audit
git diff HEAD --name-status | grep "^A"

# 3. Scope boundary check (the star of the show)
python scripts/verify_scope.py --strict

# 4. Test suite
npm test   # or your platform equivalent

# 5. Lint
npm run lint

The verify_scope.py script is a key part of the toolchain. It reads your .refactor-scope-allowlist file (generated automatically from the spec) and checks every changed file against it:

=== Scope Verification ===
Allowlist: .refactor-scope-allowlist
Allowed patterns (6):
  - src/components/
  - src/hooks/
  - src/utils/
  - *.js
  - *.ts
  - *.tsx

Changed files (15):
  - src/components/common/Button.tsx
  - src/components/common/Modal.tsx
  [...]

✅ All changed files are within approved scope
✅ No new files created
✅ No dependency manifests modified
✅ All checks passed. Scope is clean.

If it detects a violation, it exits non-zero in --strict mode — which makes it CI-friendly.

§ 6 — Context Persistence: The Session Handoff Protocol

Long-running tasks span days and multiple sessions. Context that fills up and gets flushed is the #1 cause of late-task drift. The skill solves this with two mechanisms.

6.1 The Session Handoff File

At the end of every session, the agent writes .refactor-session.md. At the start of the next session — whether it’s the same model, a different model, or a different platform entirely — it reads this file as its first action.

The template in templates/session-handoff.md is comprehensive: it covers completed subtasks with commit SHAs, the in-progress subtask with exact file-level progress, decisions made (with reasoning), edge cases discovered, files requiring special handling, the most recent drift check log, and active blockers.

A fresh agent context with a spec + session handoff file can resume a 5-day migration without re-reading the entire git log.

6.2 Context Flushing Protocol (Anti-Degradation)

After each batch:

  1. Commit and push all changes
  2. Write the session handoff file
  3. Discard from active context: all file diffs, modified file contents, and intermediate reasoning from the completed batch
  4. Reload into fresh context: approved spec + latest .refactor-session.md + next batch file list only

The flush signal list is explicit about when to do this mid-batch: “making changes that feel right based on pattern matching rather than spec compliance” is the most important one. Trust your gut on this — when an agent starts referencing decisions from 3 batches ago without consulting the handoff file, it’s degrading.

§ 7 — Platform-Specific Notes

The skill includes platform-specific guidance for Qoder Quest, Claude Code/Codex, Factory Droid/Devin Playbooks, and GitHub Copilot. The key points:

  • Qoder Quest: Always use “Code with Spec” scenario. Use Remote execution for tasks touching 100+ files. Use Worktree mode for parallel subtask isolation.
  • Claude Code: Core guardrails (§§ 1–4) are explicitly not subject to “deepening” sessions. They are non-negotiable.
  • Factory Droid/Devin: The spec must be injected as system prompt before delegation. Playbooks that operate on “all files matching pattern X” without per-instance file lists are prohibited.
  • Copilot: Scope the workspace to IN SCOPE directories only before starting.

§ 8 — The Quick Reference Table

The skill ends with a decision table that any agent can consult in-context:

Situation Action
“This file might be in scope, I’m not sure” Ask. Don’t touch.
“This would be cleaner if I also refactored X” Log in OBSERVATIONS.md. Don’t touch X.
“Tests are failing and I know how to fix it” Check if the fix is in spec. If not: halt, report.
“I found a bug while doing this refactor” Log in OBSERVATIONS.md. Leave the bug alone.
“This approach requires a new shared utility” Halt. Propose. Wait for approval.
“I could make this faster/better/cleaner” That is not the task. Log. Move on.
“The spec is ambiguous about this file” Surface the ambiguity. Await clarification.
“I hit the file budget for this session” Stop. Commit. Push. Report progress.

Complete End-to-End Walkthrough: 120-File JS→TS Migration

Let’s put it all together with a real scenario. You have a React app with 120 JavaScript files to migrate to TypeScript.

Step 1: Install and Invoke

# Clone the skills repo
git clone https://github.com/opensite-ai/opensite-skills.git

# Copy the skill to your platform's skill directory
# (or configure it per your platform's documentation)

# Create your feature branch
git checkout -b refactor/js-to-ts-migration

# Invoke the skill
@large-scale-refactor js-to-ts-migration   # Claude Code / Codex
/large-scale-refactor js-to-ts-migration   # Cursor / Copilot

Step 2: Review and Approve the Spec

The agent generates a full spec (see the spec template above). You review it, edit any scope boundaries that need adjusting, and reply approved.

Read the OUT OF SCOPE list carefully. This is where most migrations go wrong. Make sure config/, scripts/, build/, and anything CI-related is explicitly excluded.

Step 3: Generate the Scope Allowlist

python scripts/generate_allowlist.py TASK_SPEC.md --dry-run
# Review the output, then:
python scripts/generate_allowlist.py TASK_SPEC.md

This creates .refactor-scope-allowlist:

# Refactoring Scope Allowlist
# Generated by generate_allowlist.py from refactoring spec

src/components/
src/hooks/
src/utils/
src/api/
*.js
*.ts
*.tsx
tsconfig.json

Commit this file. It’s the source of truth for verify_scope.py throughout the migration.

Step 4: The Pilot Batch (10–20 files)

The agent starts with a pilot batch of 10–20 files — even though your file budget for a low-risk task like this is 200. This is non-negotiable by design. The pilot surfaces:

  • Files with non-standard patterns (e.g., forwardRef wrappers, custom hooks with unusual signatures)
  • Whether your TypeScript config is set up correctly
  • Whether your test suite handles .tsx imports
  • Edge cases the spec didn’t anticipate

After the pilot batch, you’ll have a much clearer picture of whether your transformation approach works.

Step 5: Full Execution with Drift Checks

The agent processes the remaining files in batches. Every 25 files (the default cadence for medium-risk tasks), it runs a drift check:

DRIFT CHECK
===========
Files touched so far: 25
Task: js-to-ts-migration

1. Does every changed file appear in the IN SCOPE list? YES
2. Did I add any new files not defined in the spec? NO  
3. Did I add, remove, or modify any dependency? NO
4. Did I make any change that fails the Substitution Test? NO
5. Did I create any new abstraction, utility, or system? NO

All answers NO — continuing.

After each subtask, verify_scope.py --strict runs automatically.

Step 6: Handle Checkpoints

You’ll likely hit at least one checkpoint. Common ones for TS migrations:

  • @types/react or @types/node needed (allowed by spec if you said @types/* is OK)
  • A file in src/ that turned out to be generated/build output (out of scope)
  • A component that requires a new shared type definition (propose in OBSERVATIONS.md first)

Each checkpoint presents options and a recommendation. You reply, the agent continues.

Step 7: Session Boundaries

After each session, .refactor-session.md is committed:

REFACTOR SESSION HANDOFF
========================
Task: js-to-ts-migration
Last session: 2026-03-27T15:45:00Z
Agent: Claude Sonnet 4.5 / Claude Code
Session number: 2

### Progress
- Completed: subtask-a (15 files), subtask-b (22 files)
- In-progress: subtask-c — 40% complete
- Remaining: subtask-c (cont.), subtask-d, subtask-e

### Files Remaining in subtask-c
src/components/layouts/Header.js
src/components/layouts/Footer.js
[...]

### Decisions made this session
- Decision: Use .tsx for all React components (not just those with JSX)
  Reason: Simpler rule, avoids mid-migration ambiguity
  Impact: All remaining components should get .tsx extension

### Edge cases discovered
- src/components/common/withAuth.js — HOC pattern requires special generic syntax

### Active Blockers
None.

The next session — same model, different model, or entirely different platform — reads this file first and resumes exactly where things left off.

Step 8: Final Verification

After all subtasks are complete:

# Full scope check
python scripts/verify_scope.py --strict

# TypeScript compilation
tsc --noEmit

# Full test suite
npm test

# Review observations log
cat OBSERVATIONS.md

The OBSERVATIONS.md file at this point is genuinely valuable. It contains every improvement the agent noticed but didn’t act on — organized by file, severity, and type. This becomes your technical debt backlog for the next sprint.

The Verification Scripts in Detail

The skill ships with two Python scripts that require no external dependencies (stdlib only).

verify_scope.py

The scope enforcement engine. It supports three match strategies:

  1. Directory prefix — src/components/ matches any file starting with that path (trailing slash prevents false matches like src/components-extra/)
  2. Glob patterns — *.js matches against both the full path and the basename, so *.js correctly catches src/utils/format.js without needing **/*.js
  3. Exact path — tsconfig.json matches only tsconfig.json
python scripts/verify_scope.py                       # report only
python scripts/verify_scope.py --strict              # exit 1 on any violation (CI-friendly)
python scripts/verify_scope.py --allowlist custom.txt
python scripts/verify_scope.py --base HEAD~5        # compare against specific ref

generate_allowlist.py

Parses your TASK_SPEC.md and extracts scope patterns from the IN SCOPE section automatically. Understands both - [x] checked items and plain bullets. Handles globs, directory paths, exact file paths, and bare extensions (.js → *.js).

python scripts/generate_allowlist.py TASK_SPEC.md --dry-run   # inspect first
python scripts/generate_allowlist.py TASK_SPEC.md              # write allowlist

Both scripts have a full test suite in scripts/test_verify_scope.py:

python -m pytest scripts/test_verify_scope.py -v

The Templates: What Gets Created

The skill includes four templates in templates/:

change-manifest.md — The post-subtask audit trail. Files modified, lines added/removed, scope compliance checklist, test results before/after.

observations.md — The “notice but don’t act” log. Includes a severity guide (Critical → High → Medium → Low) and a type guide (Bug, Performance, Security, Architecture, Dependency, Style, Debt). Critical/High entries get filed as separate issues. Medium/Low are batched for a future cleanup pass.

session-handoff.md — The context bridge between sessions. Structured to give a fresh agent everything it needs without re-reading the git log. Includes a drift check log section so the next session can verify the working tree was clean at handoff.

refactor-scope-allowlist.example — A heavily annotated example allowlist covering JS→TS migrations, CSS-in-JS→CSS Modules migrations, and Rails controller refactors. The “What Should NOT Appear In This File” section is worth reading — it explicitly calls out package.json, *.env, CI configs, and infrastructure files.

Why This Approach Works

The large-scale-refactor skill works because it addresses the four actual failure modes of agentic refactoring:

  1. Scope creep → Substitution Test + Drift Detection + Scope Allowlist
  2. Context degradation → Context Flushing Protocol + Session Handoff
  3. Silent compounding errors → Atomic subtask commits + Verification Sequence
  4. Parallel agent contamination → Non-overlapping file lists + Parallel Isolation rules

None of these are solved by “a smarter model.” They’re process failures, not capability failures. The skill is a process.

Getting Started

  1. Install: git clone https://github.com/opensite-ai/opensite-skills.git
  2. Configure for your platform: Drop large-scale-refactor/ into your platform’s skill directory
  3. Invoke: @large-scale-refactor [task-name] (Claude Code / Codex) or /large-scale-refactor [task-name] (Cursor / Copilot)
  4. Review the spec: The most important 5 minutes of the entire task
  5. Generate the allowlist: python scripts/generate_allowlist.py TASK_SPEC.md
  6. Let it run: Trust the pilot batch, trust the drift checks, trust the checkpoints

The skill is MIT licensed, works across all major AI coding platforms, and requires no dependencies beyond Python’s standard library. Contributions welcome via the GitHub repo.

The large-scale-refactor skill is part of the OpenSite Skills Library — open-source behavioral protocols for AI coding agents.

I Turned Substack Into a Research Surface Inside an MCP Client

Part 6 of 9

The project got a lot more interesting when it stopped being just a publishing tool.

That’s where the research layer came in:

  • research_substack
  • research_substack_post
  • research_substack_publication

Now the MCP can help discover writers, inspect posts, or break down a whole publication from an AI client.

That means it can help you publish, but also help you study a niche and understand what already works before you write a word.

Repo:
https://github.com/abanoub-ashraf/substack-mcp-plus

I built a React Native OTT video player with debugging tools (open core + pro)

I built a React Native OTT video player with debugging tools (open core + pro)

I’ve been working with React Native video for a while, and I kept running into the same problems:

  • hard to debug streaming issues
  • no built-in support for subtitles or quality switching
  • building custom controls takes too much time

Most libraries solve playback, but not the full streaming experience.

So I built MamoPlayer.

What is MamoPlayer?

MamoPlayer is a React Native video player designed for OTT-style apps.

It comes in two parts:

Core (free)

  • custom timeline
  • playback controls
  • gestures (double tap seek)
  • auto-hide controls

Pro

  • quality selection (HLS)
  • subtitles & multi-language audio
  • ads (pre/mid/post roll)
  • thumbnail previews while scrubbing
  • PiP
  • debug overlay for real-time playback insights

Why I built it

The biggest issue for me wasn’t playback — it was debugging and UX.

I wanted:

  • better visibility into playback state
  • easier handling of tracks (quality, audio, subtitles)
  • a modern OTT-style UI without rebuilding everything

Links

Docs: https://mamoplayer.com
npm (Core): https://www.npmjs.com/package/@mamoplayer/core

Feedback

If you’re working on video in React Native, I’d really appreciate your feedback.

What’s missing? What would you need to adopt something like this?

Day 3/100: Activity Lifecycle — The Diagram You’ve Seen 100 Times

This is Day 3 of my 100 Days to Senior Android Engineer series. Each post: what I thought I knew → what I actually learned → interview implications.

🔍 The concept

Every Android developer has seen this diagram:

          onCreate()
              ↓
          onStart()
              ↓
          onResume()  ←──────────────────────┐
              ↓                              │
         [Activity running]                  │
              ↓                              │
          onPause()                          │
              ↓                         onRestart()
          onStop()   ──────────────────────→ │
              ↓
          onDestroy()

You could draw this in your sleep. I certainly could.

The problem is that this diagram shows the happy path. It doesn’t show what happens in the three scenarios that actually matter in production — and that every senior interview will eventually probe.

💡 What I thought I knew

My mental model was: callbacks fire in a predictable sequence, each one is the right place to do certain things (init in onCreate, release in onDestroy), and as long as I follow the pattern, everything works.

That model is mostly right. Mostly.

😳 What I actually learned

The diagram has a ghost: the back stack

The standard diagram shows a single Activity in isolation. But your app never has just one Activity in isolation — it has a back stack, and the lifecycle behaves differently depending on why a callback fires.

Consider onStop(). It gets called in two very different situations:

Situation A: User presses Home
    → Activity goes to background
    → onPause() → onStop()
    → Activity is still in memory (probably)
    → User comes back → onRestart() → onStart() → onResume()

Situation B: User navigates to another Activity
    → Current Activity partially covered
    → onPause() → onStop()
    → User presses Back → onRestart() → onStart() → onResume()

Situation C: System kills the process (low memory)
    → onPause() → onStop() → [process killed, no onDestroy()]
    → User comes back → onCreate() with savedInstanceState Bundle

Situation C is the one that breaks apps. onDestroy() is not guaranteed to be called. If you’re releasing resources, closing connections, or saving state in onDestroy(), you have a bug waiting to happen under memory pressure.

The configuration change trap

Rotate the screen. What happens?

onPause()
onStop()
onDestroy()   ← called this time, because it's a controlled teardown
onCreate()    ← brand new instance
onStart()
onResume()

The Activity is destroyed and recreated. Which means:

  • Any data stored in instance variables is gone
  • Any network request in flight is cancelled (if tied to the Activity’s scope)
  • Any UI state that isn’t explicitly saved is reset

Here’s a pattern I’ve seen in codebases more than I’d like:

class ProfileActivity : AppCompatActivity() {
    private var userProfile: UserProfile? = null  // 💀 gone on rotation

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        // Fetch every time, including on rotation
        fetchUserProfile()
    }

    private fun fetchUserProfile() {
        // Network call fires again on every rotation
        viewModel.loadProfile(userId)
    }
}

The fix isn’t complicated — it’s using ViewModel correctly:

class ProfileActivity : AppCompatActivity() {
    // ViewModel survives configuration changes
    private val viewModel: ProfileViewModel by viewModels()

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        // ViewModel already has data if we're recreating after rotation
        // loadProfile() is called inside ViewModel's init{} or handled idempotently
        observeViewModel()
    }
}

The ViewModel survives configuration changes because it’s scoped to the ViewModelStore, which is retained across recreations. The Activity is recreated; the ViewModel is not.

onPause() is your last guarantee

This is the one that surprises the most people.

onPause() is the only callback guaranteed to be called before your process can be killed.

Not onStop(). Not onDestroy(). Just onPause().

From the docs:

“If the system needs to recover memory in an emergency, it might not call onStop() and onDestroy().”

This has a direct consequence on where you save critical state:

override fun onPause() {
    super.onPause()
    // ✅ Save anything critical HERE — this is your last guaranteed checkpoint
    saveDraftMessage()
    releaseCamera()  // release exclusive resources here, not in onStop
}

override fun onStop() {
    super.onStop()
    // ✅ Safe for non-critical cleanup that can tolerate being skipped under pressure
    stopLocationUpdates()
}

override fun onDestroy() {
    super.onDestroy()
    // ✅ Cleanup that you'd expect in a clean shutdown
    // ❌ Do NOT rely on this for anything critical
}

There’s also a performance implication: onPause() blocks the transition to the next Activity. Anything slow in onPause() directly delays how quickly the new screen appears. Keep it fast.

The two-Activity transition sequence

This is a classic interview question, and the answer is not what most people expect.

What’s the callback order when Activity A starts Activity B?

// Most people answer:
A.onPause()
A.onStop()
B.onCreate()
B.onStart()
B.onResume()

// Actual order:
A.onPause()       ← A pauses FIRST
B.onCreate()      ← B starts creating
B.onStart()
B.onResume()      ← B is now visible
A.onStop()        ← A stops AFTER B is running

A.onStop() comes after B.onResume(). This matters because it means A is still technically “paused but not stopped” while B initializes. If you’re holding an exclusive resource (camera, audio focus) and releasing it in onStop() instead of onPause(), Activity B will try to acquire it before you’ve released it.

🧪 The mental model that stuck

Instead of memorizing the diagram, I now ask two questions for each callback:

1. Can this be the last callback that runs?

Callback Can be last before kill?
onPause() ✅ Yes — save critical state here
onStop() ⚠️ Usually, but not guaranteed
onDestroy() ❌ Not guaranteed

2. How many times will this run in a session?

Scenario onCreate onResume
Normal launch 1× 1×
Home → Back to app 1× 2×
Rotate screen 2× 2×
Rotate 3 times 4× 4×

If you’re doing expensive initialization in onResume() instead of onCreate(), every rotation and every return from background triggers it again. That’s a real performance bug in disguise.

❓ The three interview questions

Question 1 — Basic:

“When does onDestroy() get called, and can you rely on it always being called?”

No. Under memory pressure, the process is killed without onDestroy(). Save critical state in onPause().

Question 2 — Mid-level:

“A user starts filling out a form, receives a phone call, and comes back. How do you ensure their input is preserved?”

// Option A: ViewModel (survives configuration change, lost on process death)
class FormViewModel : ViewModel() {
    val formState = MutableStateFlow(FormState())
}

// Option B: onSaveInstanceState (survives process death, limited to Bundle size)
override fun onSaveInstanceState(outState: Bundle) {
    super.onSaveInstanceState(outState)
    outState.putString("draft_input", binding.editText.text.toString())
}

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    savedInstanceState?.getString("draft_input")?.let {
        binding.editText.setText(it)
    }
}

// Option C: SavedStateHandle (best of both — ViewModel + process death survival)
class FormViewModel(private val savedStateHandle: SavedStateHandle) : ViewModel() {
    var draftInput: String
        get() = savedStateHandle["draft_input"] ?: ""
        set(value) { savedStateHandle["draft_input"] = value }
}

The senior answer explains all three options, their tradeoffs, and reaches for SavedStateHandle as the modern solution — because it survives both configuration changes and process death.

Question 3 — Senior:

“Activity A launches Activity B. In Activity B, the user performs an action that should update a result visible in Activity A. How do you handle this, and what lifecycle considerations apply?”

The legacy approach — startActivityForResult() — is deprecated. The modern answer:

// In Activity A — register before onCreate completes
private val launcher = registerForActivityResult(
    ActivityResultContracts.StartActivityForResult()
) { result ->
    if (result.resultCode == RESULT_OK) {
        val data = result.data?.getStringExtra("updated_value")
        viewModel.handleUpdate(data)
    }
}

// Launch Activity B
launcher.launch(Intent(this, ActivityB::class.java))

// In Activity B — send result back
setResult(RESULT_OK, Intent().putExtra("updated_value", newValue))
finish()

The lifecycle consideration: the result callback fires during onResume() of Activity A. If you’re observing a StateFlow in onStart() with repeatOnLifecycle, you don’t need the callback at all — just update the ViewModel from B and let A observe the change.

What surprised me revisiting this

The lifecycle diagram gives you the what but not the why. Going back through the documentation carefully, three things stood out:

  1. onPause() blocking — I knew it was called, but I hadn’t internalized that slow onPause() directly delays the next screen appearing. I’ve definitely shipped code that violated this.

  2. The two-Activity transition order — I knew A pauses before B starts, but I hadn’t thought through the implication for exclusive resources until I re-read the ordering carefully.

  3. SavedStateHandle being the real answer — I’d used onSaveInstanceState for years and ViewModel for a few more, but SavedStateHandle genuinely makes both feel like workarounds in comparison.

Tomorrow

Day 4 → Fragment Lifecycle vs Activity Lifecycle — they look similar on a diagram, and that’s exactly why people keep getting them wrong.

What’s a lifecycle bug you’ve shipped to production? Drop it in the comments — no judgment, we’ve all been there.

← Day 2: The 4 Android Components — What Senior Devs Get Wrong

Your AI Agent Just Made a $50K Mistake. Can You Explain Why?

Agent Forensics Dashboard

AI Agents Are Making Decisions. Nobody’s Tracking Why.

In March 2026, Meta had a Sev-1 incident. An AI agent posted internal data to unauthorized engineers for two hours. The scariest part wasn’t the leak itself — it was that the team couldn’t reconstruct why the agent decided to do it.

This isn’t an isolated case:

  • A shopping agent asked to check egg prices decided to buy them instead. No one approved it.
  • A customer support bot gave a customer a completely fabricated explanation for a billing error — with confidence.
  • A shopping agent tasked with buying an Apple Magic Mouse bought a Logitech instead because “it was cheaper.” The user never asked for the cheapest option.

These aren’t hypothetical risks. They’re happening now. And every time, the same question comes up:

“Why did the agent do that?”

And every time, the same answer: “We don’t know.”

Monitoring ≠ Forensics

Here’s the thing — tools like Datadog, Arize, and Langfuse are great at watching agents in real time. But when something goes wrong, the question changes from “is it working?” to “why did it fail?”

That’s a fundamentally different question.

Monitoring Forensics
When Real-time Post-incident
Question “Is it working?” “Why did it fail?”
Output Alerts, dashboards Decision timeline, causal chain
Audience Engineering team Legal, compliance, regulators
Analogy Security camera Airplane black box

There was no tool that answered the forensics question. So I built one.

What the Black Box Shows You

Your shopping agent receives: “Buy me an Apple Magic Mouse.”

The agent runs. A few seconds later, it comes back: “Purchased Logitech M750 for $45.”

Wait — the user asked for an Apple Magic Mouse. What happened?

Pull the black box:

[DECISION] search_products("Apple Magic Mouse")
  → [TOOL] search_api → ERROR: product not found

[DECISION] retry with broader query "Apple wireless mouse"
  → [TOOL] search_api → OK: 3 products found

[DECISION] compare_prices
  → Logitech M750 is cheapest ($45)

[DECISION] purchase("Logitech M750")
  → SUCCESS — user never asked for this product

[FINAL] "Purchased Logitech M750 for $45"

Now you can see it: decision point 3 is where things went wrong. The agent’s standing instructions said “buy the cheapest,” which overrode the user’s specific product request. That’s a fixable bug — and without the trail, it’s invisible.

This is the kind of evidence that:

  • Engineers need to fix the agent’s behavior
  • Legal teams need to assess liability
  • Compliance teams need to report to regulators

Why This Matters Right Now

On August 2, 2026, the full weight of EU AI Act high-risk requirements takes effect:

  • Up to €35M or 7% of global annual turnover for the most serious violations
  • Up to €15M or 3% for non-compliance with high-risk AI obligations
  • Market surveillance authorities can order non-compliant systems withdrawn from the market

Article 14 specifically requires human oversight — the ability to understand and trace AI system decisions. You need documentation proving:

  1. What decision was made
  2. What information led to that decision
  3. What alternatives were considered
  4. Why this specific action was chosen

“We didn’t track it” is not a valid defense.

How It Works

Install:

pip install agent-forensics

Attach to your agent (one line):

from agent_forensics import Forensics

f = Forensics(session="order-123")

# LangChain
agent.invoke(..., config={"callbacks": [f.langchain()]})

# OpenAI Agents SDK
agent = Agent(hooks=f.openai_agents())

# CrewAI
Agent(step_callback=f.crewai().step_callback)

# Or any custom agent
f.decision("search", input={"query": "mouse"}, reasoning="User requested search")
f.tool_call("api", input={...}, output={...})

Get reports:

# Markdown report — full timeline + decision chain + root cause
print(f.report())

# Save files
f.save_markdown()   # → forensics-report-order-123.md
f.save_pdf()        # → forensics-report-order-123.pdf

# Visual dashboard
f.dashboard(port=8080)  # → http://localhost:8080

The dashboard gives you a visual timeline with color-coded events, session comparison, and causal chain visualization:

Agent Forensics Dashboard

What You Get

  • Decision timeline — every action in chronological order
  • Decision chain — each choice with its reasoning
  • Causal chain — “A led to B, which caused C to fail”
  • Incident detection — automatic error and failure identification
  • Compliance reports — Markdown + PDF, ready for regulators
  • Web dashboard — visual session browser with incident highlighting

No vendor lock-in. No cloud dependency. SQLite event store that runs anywhere. MIT licensed.

Try It

EU AI Act enforcement is 4 months away. If you’re running AI agents in production, the time to add forensic tracing is now.

The agents are getting smarter. The question is whether we can explain what they’re doing.

What’s the worst AI agent failure you’ve seen? I’d love to hear your stories in the comments.

Taming the Glucose Spike: Predicting Postprandial Peaks with Transformers and PyTorch

Living with a Continuous Glucose Monitor (CGM) is like having a dashboard for your metabolism. But for many, it’s a dashboard that only tells you when you’ve already crashed or spiked. What if we could see 30 minutes into the future?

In this guide, we’re moving from reactive monitoring to proactive health. We will leverage time-series forecasting, Transformer models, and PyTorch Forecasting to predict postprandial (after-meal) glucose peaks. By treating blood glucose, insulin doses, and carb intake as multi-dimensional time-series data, we can use the self-attention mechanism to capture long-range dependencies that traditional LSTMs often miss.

If you are looking for time-series forecasting, Continuous Glucose Monitoring (CGM), Deep Learning for health, or PyTorch Forecasting tutorials, you’re in the right place!

The Architecture: From Sensors to Predictions

Predicting glucose isn’t just about the last three readings; it’s about the context of the last four hours. Our pipeline ingests raw data from the Dexcom API, stores it in InfluxDB for high-performance time-series querying, and processes it through a Temporal Fusion Transformer (TFT).

graph TD
    A[Dexcom G6/G7 API] -->|Raw Glucose| B(InfluxDB)
    C[Insulin/Carb Logs] -->|Events| B
    B -->|Query| D[Pandas Preprocessing]
    D -->|Feature Engineering| E[PyTorch Forecasting]
    E -->|Temporal Fusion Transformer| F{Prediction Engine}
    F -->|30-min Warning| G[Mobile Alert/Dashboard]
    F -->|Feature Importance| H[Interpretability Layer]

Prerequisites

To follow this advanced tutorial, you’ll need:

  • Tech Stack: PyTorch Forecasting, InfluxDB, Pandas, and DEXCOM API.
  • A basic understanding of Attention mechanisms.
  • (Optional) A set of CGM data (or use the simulated datasets provided by PyTorch Forecasting).

Step 1: Data Ingestion & Feature Engineering

CGM data is notoriously noisy. We need to sync glucose levels (sampled every 5 mins) with sporadic events like insulin boluses and meals.

import pandas as pd
from influxdb_client import InfluxDBClient

# Connecting to InfluxDB to fetch our health metrics
client = InfluxDBClient(url="http://localhost:8086", token="my-token", org="health-dev")
query = 'from(bucket:"cgm_data") |> range(start: -7d) |> filter(fn: (r) => r._measurement == "glucose")'
df = client.query_api().query_data_frame(query)

# Feature Engineering: Adding 'Time of Day' and 'Glucose Velocity'
df['velocity'] = df['_value'].diff()
df['hour'] = df['_time'].dt.hour
df['is_sleeping'] = df['hour'].apply(lambda x: 1 if x < 6 or x > 23 else 0)

# Critical: PyTorch Forecasting requires a 'time_idx'
df['time_idx'] = (df['_time'] - df['_time'].min()) / pd.Timedelta('5min')
df['time_idx'] = df['time_idx'].astype(int)

Step 2: Defining the Temporal Fusion Transformer (TFT)

The Temporal Fusion Transformer is perfect for CGM data because it handles “Static” variables (like user ID/Age), “Known” future variables (scheduled exercise), and “Observed” past variables (past glucose).

from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer

# Define the dataset metadata
max_prediction_length = 6  # 30 minutes (6 * 5 min steps)
max_encoder_length = 24    # Look back 2 hours

training = TimeSeriesDataSet(
    df,
    time_idx="time_idx",
    target="_value",
    group_ids=["user_id"],
    min_encoder_length=max_encoder_length // 2,
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    static_categoricals=["user_id"],
    time_varying_known_reals=["hour", "is_sleeping"],
    time_varying_unknown_reals=["_value", "velocity", "insulin_units", "carbs_g"],
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_interpolation=True,
)

# Initialize the model
tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.03,
    hidden_size=16, 
    attention_head_size=4,
    dropout=0.1,
    loss=QuantileLoss(), # We want ranges, not just single points!
)

Step 3: Training & Interpretability

One of the coolest features of Transformers in health-tech is Attention Weights. The model can tell us why it predicted a spike. Was it the pizza you ate 2 hours ago or the lack of basal insulin?

While this tutorial covers the implementation basics, production-grade deployments require rigorous safety checks. For more advanced patterns on handling medical IoT data and productionizing health-tech models, check out the detailed deep-dives at WellAlly Tech Blog. They cover everything from HIPAA-compliant cloud architectures to real-time stream processing for wearables.

Step 4: Visualizing the Predicted Spike

We don’t just want a number; we want a confidence interval. Using QuantileLoss, our model predicts the 10th, 50th, and 90th percentiles.

# Convert to Dataloader and Predict
dataloader = training.to_dataloader(train=False, batch_size=32)
predictions = tft.predict(dataloader, mode="raw", return_x=True)

# Plotting the results
for idx in range(5): # Show first 5 predictions
    tft.plot_prediction(predictions.x, predictions.output, idx=idx, add_loss_to_title=True)

Conclusion: The Proactive Future

By using Transformers, we move beyond simple threshold alerts (“Your sugar is high!”) to predictive intelligence (“Your sugar will be 220 mg/dL in 30 minutes based on your current carb-to-insulin ratio”).

This “Learning in Public” project shows that with the right tech stack—PyTorch Forecasting for the heavy lifting, InfluxDB for the data foundation, and Transformers for the logic—we can build tools that significantly improve quality of life.

What’s next?

  1. Try adding heart rate data (Apple Watch/Fitbit) to see how exercise affects your model’s accuracy.
  2. Experiment with different max_encoder_length settings to see how much “history” the model really needs.

Got questions or better ideas for feature engineering? Let’s chat in the comments! 👇

Comprehensive Guide to Twitter/X Scraping Frameworks and Tools in 2026

Scraping data from Twitter (now X) has become increasingly complex due to aggressive anti-bot measures, the deprecation of legacy free API tiers, and the rise of sophisticated fingerprinting technologies. However, the ecosystem of tools has evolved to meet these challenges. This report provides a deep and wide comparison of the best frameworks, packages, and tools available in 2026 for scraping X data, ranging from official APIs and open-source libraries to AI-powered browser agents like Browser Use.

1. The Official X API (The Expensive Baseline)

The landscape of Twitter data collection shifted dramatically when X gutted its legacy free API. As of February 2026, X introduced a new “Pay-Per-Use” consumption-based billing model [1].

The official API is the most reliable method for accessing X data, but it comes with significant limitations and costs for data extraction:

  • Free Tier: Limited to 1,500 posts per month, but crucially, it is write-only. You cannot use the free tier to read or scrape data [2].
  • Basic Tier: Costs $200 per month and allows reading up to 10,000 tweets [2].
  • Pro Tier: Costs $5,000 per month for 1,000,000 tweets [2].

For developers using Python, Tweepy remains the standard library for interacting with the official API. It is actively maintained and fully supports the X API v2 endpoints [3]. However, due to the prohibitive costs of read access, most developers looking to scrape data at scale are turning to alternative frameworks.

2. Open-Source Python Libraries & Frameworks

For those who want to build their own scraping pipelines without paying exorbitant API fees, several open-source frameworks have emerged that bypass X’s protections.

Twikit

Twikit is a highly popular Python library (with over 4.2k stars on GitHub) that interacts with Twitter’s internal API [4].
Unlike the official API, Twikit does not require an API key. Instead, it requires the user to log in using their Twitter username, email, and password, and it manages the session cookies.

  • Capabilities: It can create tweets, search for tweets, retrieve user timelines, fetch trending topics, and send direct messages.
  • Pros: Completely free, supports asynchronous operations, and is actively maintained.
  • Cons: Because it uses an actual account, aggressive scraping can lead to account suspension. It is best used for moderate, rate-limited extraction.

Scrapling

Scrapling is a newer, highly adaptive web scraping framework built in Python that is gaining significant traction among AI agent developers [5].

  • Capabilities: It features advanced stealth capabilities, including a StealthyFetcher that natively bypasses anti-bot systems like Cloudflare Turnstile and Interstitial screens without needing brittle selectors [6].
  • Pros: Excellent for bypassing the strict bot detection on X. It spoofs Chromium fingerprints and handles multi-session scraping gracefully.
  • Cons: It is a general-purpose scraper, meaning developers must write the specific parsing logic for X’s DOM or network responses.

Proxidize Open-Source Scraper (Playwright + GraphQL)

A highly effective approach in 2026 involves intercepting Twitter’s internal GraphQL requests rather than parsing HTML [7]. Proxidize released an open-source scraper that utilizes Playwright to achieve this.

  • How it works: The scraper uses Playwright to navigate X and intercepts the XHR/fetch requests made to endpoints like UserTweets and TweetDetail. It extracts the clean JSON data directly from the network tab [7].
  • Pros: Highly accurate and resilient to UI changes. Playwright’s native proxy support and anti-detection flags make it much more stable than older Selenium-based approaches.
  • Cons: Requires high-quality residential proxies (which can cost around $15/GB) to prevent IP bans during infinite scrolling [7].

3. AI Agent Browsers: Browser Use

Browser Use represents the cutting edge of interactive web scraping in 2026. It is an open-source AI agent framework that automates browser interactions using natural language prompts [8].

Instead of writing brittle CSS selectors or complex network interception scripts, developers can simply instruct Browser Use to “go to my personal Twitter and extract the latest tweets into a Google Sheet” [9].

Key Features for Twitter Scraping

  • Stealth Infrastructure: Browser Use utilizes a custom Chromium fork with C++ and OS-level stealth patches. It successfully bypasses Cloudflare, Akamai, and DataDome, boasting an 81% success rate on stealth benchmarks (significantly higher than competitors like Browserbase) [9].
  • Dynamic Interaction: X is a heavily dynamic Single Page Application (SPA). Browser Use excels here because it can visually understand the page, handle pop-ups, manage cookie banners, and naturally scroll through infinite timelines [10].
  • Built-in CAPTCHA Solving: It includes free CAPTCHA solving for all users, which is critical when X flags a session as suspicious [9].

Comparison with Traditional Scrapers

While traditional tools like BeautifulSoup or Firecrawl fail on heavily protected sites like X (Firecrawl explicitly gets blocked by advanced anti-bot protections), Browser Use operates exactly like a human user [9].

  • Pros: No scripting required for element selection; handles dynamic content flawlessly; highest stealth success rate.
  • Cons: It is slower and more computationally expensive than basic HTTP fetchers, as it requires running a full headless browser and invoking LLMs (like OpenAI or ChatBrowserUse) to make navigation decisions [9].

4. Managed Commercial APIs

For teams that need data immediately and want to outsource the headache of proxy management and anti-bot bypass, managed APIs are the pragmatic choice.

twitterapi.io

Widely considered the best unofficial API in 2026, twitterapi.io acts as a proxy wrapper around X’s internal endpoints [7].

  • Pricing: Offers 100,000 free credits on signup, after which it costs $0.15 per 1,000 tweets [7].
  • Pros: Extremely fast (capable of 140+ requests per second), highly reliable for production apps, and provides an OpenAPI spec for instant integration [7].

Apify Twitter Scrapers

Apify hosts a marketplace of “Actors” (pre-built scrapers). Their Twitter scrapers are highly popular for data scientists [7].

  • Pricing: Approximately $0.25 to $0.45 per 1,000 tweets depending on the specific actor used [7].
  • Pros: Point-and-click configuration, built-in proxy rotation, and native exports to AWS S3, BigQuery, and CSV. Excellent for massive data mining jobs.
  • Cons: Usage-based pricing can balloon quickly if the scraping parameters are too broad.

5. The Nitter Workaround

Nitter is an open-source, privacy-focused frontend for Twitter. Because Nitter serves static HTML without JavaScript or anti-bot protections, it is incredibly easy to scrape using standard tools like BeautifulSoup or Firecrawl.

  • How it works: You scrape a Nitter instance (e.g., nitter.net/elonmusk) instead of x.com/elonmusk.
  • Pros: Completely free, no API keys needed, and no rate limits (other than the instance’s own limits).
  • Cons: Public Nitter instances are frequently taken offline or rate-limited by X. Self-hosting a Nitter instance requires maintaining a pool of guest accounts and proxies, which has a high failure rate in production [7].

Summary Comparison

Tool / Framework Type Cost Anti-Bot Bypass Best Use Case
Official X API REST API $200 – $5,000+/mo N/A (Official) Enterprise apps needing guaranteed, legal read/write access.
Twikit Python Library Free Low (Requires Account) Hobby projects and lightweight automated accounts.
Proxidize (Playwright) Python Script Free (Proxy costs apply) High (GraphQL intercept) Developers wanting total control over the data pipeline.
Browser Use AI Agent Open Source / Cloud API Very High (Custom Chromium) Complex, dynamic scraping requiring human-like interaction and visual understanding.
twitterapi.io Managed API $0.15 / 1k tweets High (Managed) Production applications needing fast, reliable JSON data.
Apify Cloud Scraper ~$0.40 / 1k tweets High (Managed) Large-scale data mining and one-off CSV exports.

Conclusion

In 2026, the “best” tool depends entirely on your constraints.
If you are building an AI Agent that needs to browse X, read context, and act autonomously, Browser Use is the clear winner due to its unmatched stealth infrastructure and natural language navigation [9].
If you need raw data at scale for a database, a managed service like twitterapi.io or Apify is the most pragmatic choice, saving hundreds of hours in proxy maintenance [7].
For developers who want total control without paying API fees, building a custom Playwright scraper that intercepts GraphQL requests is the most robust programmatic approach currently available [7].

References

[1] DevCommunity X, “Announcing the Launch of X API Pay-Per-Use Pricing” https://devcommunity.x.com/t/announcing-the-launch-of-x-api-pay-per-use-pricing/256476
[2] OpenTweet, “Best X (Twitter) APIs for AI Agents in 2026: Developer Guide” https://opentweet.io/blog/best-twitter-apis-for-ai-agents-2026
[3] Tweepy GitHub Repository https://github.com/tweepy/tweepy/
[4] Twikit GitHub Repository https://github.com/d60/twikit
[5] Wired, “OpenClaw Users Are Allegedly Bypassing Anti-Bot Systems” https://www.wired.com/story/openclaw-users-bypass-anti-bot-systems-cloudflare-scrapling/
[6] Scrapling Documentation https://scrapling.readthedocs.io/en/latest/index.html
[7] Proxidize, “Twitter Scraper: How to Scrape Twitter for Free” https://proxidize.com/blog/twitter-scraper/
[8] ScrapingBee, “BrowserUse: How to use AI Browser Automation to Scrape” https://www.scrapingbee.com/blog/browseruse-how-to-use-ai-browser-automation-to-scrape/
[9] Browser Use, “The Ultimate Guide to Web Scraping (2026)” https://browser-use.com/posts/web-scraping-guide-2026
[10] Labelerr, “Browser-Use: Open-Source AI Agent For Web Automation” https://www.labellerr.com/blog/browser-use-agent/

500+ Free Online Tools I Built — Here Are the Most Useful Ones

Over the past year, I’ve been building free browser-based tools at devtools-site-delta.vercel.app. What started as a JSON formatter I built for myself has turned into 500+ tools.

Most of them run entirely client-side — your data never leaves your browser. No accounts, no paywalls, no “5 free uses per day” nonsense.

Here are the 15 I use the most.

1. JSON Formatter & Validator

/json-formatter

Paste in ugly JSON, get it pretty-printed with syntax highlighting. Also catches errors and tells you exactly where the problem is (line and character). I use this multiple times a day when debugging API responses.

2. PDF Merge

/pdf-merge

Drag in multiple PDFs, reorder them, merge into one file. Runs entirely in your browser using pdf-lib. No upload, no server. I wrote a whole post about this one.

3. Image Compressor

/image-compress

Drop in an image, adjust the quality slider, download a smaller version. Uses the Canvas API. I compress every image before adding it to a blog post or README. Usually saves 60-80% file size.

4. Regex Tester

/regex

Write a regex pattern, paste in test text, see matches highlighted in real time. Shows capture groups, match indices, and flags. Way faster than tab-switching to regex101 when I’m already in the zone.

5. JWT Decoder

/jwt-decode

Paste a JWT token and instantly see the decoded header and payload. Shows expiration time in a human-readable format. Essential for auth debugging. Also has a JWT creator if you need to generate test tokens.

6. QR Code Generator

/qr

Type in a URL, text, or WiFi credentials and get a QR code you can download as PNG or SVG. The WiFi QR generator is great — print one out, stick it on the fridge, and guests can connect without asking for the password.

7. Color Picker & Palette Generator

/color-picker

Pick a color, get the HEX, RGB, HSL, and CSS values. The palette generator creates harmonious color schemes from a seed color. There’s also a contrast checker for WCAG accessibility compliance.

8. Base64 Encoder/Decoder

/base64

Encode and decode Base64 strings. Also handles image-to-base64 conversion, which is useful for embedding small images directly in CSS or HTML.

9. Hash Generator

/hash-generator

Generate MD5, SHA-1, SHA-256, and SHA-512 hashes from any text input. Good for verifying file integrity or generating quick checksums. All computed locally using the Web Crypto API.

10. CSS Gradient Generator

/css-gradient-generator

Visual gradient builder with live preview. Supports linear, radial, and conic gradients. Pick your colors, adjust stops, copy the CSS. I use this instead of guessing gradient values and refreshing the browser 50 times.

11. Background Remover

/background-remover

Remove image backgrounds using an ML model that runs in the browser. Works great for product photos and headshots. No upload required — the model processes everything locally.

12. Markdown Preview

/markdown-preview

Split-pane editor with live rendering. I use this when writing READMEs or documentation. Also has Markdown to HTML conversion and an HTML to Markdown converter for going the other direction.

13. CSS Box Shadow Generator

/box-shadow

Visual editor for CSS box shadows with sliders for offset, blur, spread, and color. Supports multiple shadow layers. Copy the CSS when it looks right. Saves a ton of trial-and-error.

14. JSON to TypeScript

/json-to-typescript

Paste a JSON object and get TypeScript interfaces generated automatically. Handles nested objects, arrays, and optional fields. I use this constantly when working with API responses — paste the response, get the types, move on.

15. Image Resizer

/image-resize

Resize images to exact dimensions with aspect ratio lock. Great for making Open Graph images (1200×630), favicons, or thumbnails. Runs client-side via the Canvas API.

Honorable Mentions

A few more that didn’t make the top 15 but I still reach for regularly:

Why I Built This

I was spending way too much time on ad-covered, rate-limited tool sites. Need to format some JSON? Here’s 3 pop-ups and a newsletter prompt. Need to merge two PDFs? Upload your sensitive documents to our server.

So I just started building. One tool turned into ten. Ten turned into a hundred. Now there are 500+.

Everything is free. Most tools run client-side. No accounts needed.

Bookmark devtools-site-delta.vercel.app and stop giving your data to random tool sites.

Full site: devtools-site-delta.vercel.app

Free Image Tools That Run Entirely in Your Browser (No Server Upload)

I have a thing about uploading files to random websites. Call it paranoia, call it good practice — either way, I built a bunch of image tools that work 100% in your browser. No server. No account. No “processing on our end.”

Here’s what’s available and how they work under the hood.

Image Compressor

devtools-site-delta.vercel.app/image-compress

This one gets the most use from me personally. I compress screenshots and blog images before uploading them anywhere. The tool uses the Canvas API to re-encode images at a lower quality setting:

function compressImage(file, quality = 0.7) {
  return new Promise((resolve) => {
    const img = new Image();
    img.onload = () => {
      const canvas = document.createElement('canvas');
      canvas.width = img.width;
      canvas.height = img.height;

      const ctx = canvas.getContext('2d');
      ctx.drawImage(img, 0, 0);

      canvas.toBlob(resolve, 'image/jpeg', quality);
    };
    img.src = URL.createObjectURL(file);
  });
}

The quality parameter (0 to 1) controls the compression level. At 0.7, you usually get 60-80% file size reduction with barely visible quality loss. The tool gives you a slider so you can find the sweet spot for each image.

What I like about this approach: the Canvas API is built into every modern browser. No external libraries needed. Your image gets drawn onto a canvas element, then re-exported as a blob with your chosen quality.

Image Resizer

devtools-site-delta.vercel.app/image-resize

Same Canvas API approach, different use case. Need a 1200×630 image for an Open Graph tag? Need to scale down a 4000px photo for your blog? Punch in the dimensions, done.

The resizer preserves aspect ratio by default (you can unlock it if you want exact dimensions). It also shows you the original and new file sizes side by side so you know exactly what you’re getting.

function resizeImage(file, maxWidth, maxHeight) {
  return new Promise((resolve) => {
    const img = new Image();
    img.onload = () => {
      let { width, height } = img;

      // Maintain aspect ratio
      if (width > maxWidth) {
        height = (height * maxWidth) / width;
        width = maxWidth;
      }
      if (height > maxHeight) {
        width = (width * maxHeight) / height;
        height = maxHeight;
      }

      const canvas = document.createElement('canvas');
      canvas.width = width;
      canvas.height = height;
      canvas.getContext('2d').drawImage(img, 0, 0, width, height);

      canvas.toBlob(resolve, file.type);
    };
    img.src = URL.createObjectURL(file);
  });
}

Image Converter

devtools-site-delta.vercel.app/image-converter

PNG to JPG. JPG to WebP. WebP to PNG. Whatever you need.

The trick here is that canvas.toBlob() accepts a MIME type as the second argument. Want WebP? Pass 'image/webp'. Want JPEG? 'image/jpeg'. The browser handles the encoding natively.

This is especially useful for converting to WebP. Most sites should be serving WebP images (they’re typically 25-35% smaller than equivalent JPEGs), but a lot of design tools still export PNG or JPEG by default.

Background Remover

devtools-site-delta.vercel.app/background-remover

This is the flashiest one. Drop in a photo, get back the subject with a transparent background. It uses a pre-trained ML model that runs in the browser via ONNX Runtime or TensorFlow.js.

The model downloads once and then runs locally. Your images never leave your machine. It works surprisingly well for product photos, headshots, and simple compositions. Complex scenes with hair or translucent objects can be trickier, but for most everyday use cases, it gets the job done.

Image Cropper

devtools-site-delta.vercel.app/image-cropper

Visual cropping with preset aspect ratios (1:1 for Instagram, 16:9 for YouTube thumbnails, 4:3 for presentations) or freeform. Drag the crop area, preview the result, download. Nothing fancy, just works.

Extract Colors from Images

devtools-site-delta.vercel.app/image-colors

This one’s handy for design work. Upload an image and it extracts the dominant color palette. I use it when I’m building a landing page and want colors that match a hero image. It reads pixel data from the canvas and clusters similar colors together.

Why Browser-Based?

Honestly? Three reasons.

1. Privacy. Your images might contain GPS metadata, sensitive content, or just stuff you don’t want on someone’s server. Client-side processing means the data stays on your device.

2. Speed. No upload time, no waiting for server processing, no download time. For a 5MB image, that can save 10-15 seconds compared to a server-based tool.

3. No limits. No file size caps, no “5 free conversions per day,” no watermarks. Process as many images as your browser can handle.

Part of a Bigger Toolkit

These image tools are part of devtools-site-delta.vercel.app, where I’ve built 500+ free tools for developers and anyone who’s tired of ad-heavy utility sites. PDF tools, JSON formatters, text utilities, color pickers, and a lot more.

Everything’s free. No sign-ups. Bookmark whatever you use regularly.

Links:

How to Build a Subscription Churn Early-Warning System With Apify and Google Sheets

Most SaaS founders discover churn the same way: the cancellation email arrives. By that point, the decision was made weeks ago. The customer had already stopped logging in, stopped using key features, maybe even clicked “downgrade” before closing the tab. You just didn’t know.

This guide shows you how to build a churn early-warning system that catches those signals — before the cancellation — using Apify to pull usage data and Google Sheets to score risk automatically.

The Problem: You’re Diagnosing Churn Retrospectively

Most subscription businesses track churn as a lagging metric — customers leave, you count them, you calculate the rate. This is useful for reporting but useless for retention.

The decision to cancel is rarely impulsive. It follows a pattern:

  • Login frequency drops
  • Core feature usage declines
  • The user stops completing their primary workflow
  • Sometimes they try to downgrade first

This behavioral decay typically unfolds over 2 to 4 weeks before cancellation. That window is your intervention opportunity — and right now, most founders are missing it entirely.

The Signal: What Usage Decay Looks Like

If you have any form of event tracking (Mixpanel, Amplitude, your own database logs, or even server-side API call logs), you’re already collecting the data you need. The three highest-signal indicators:

1. Last login date
A user who hasn’t logged in for 10+ days on a weekly-use tool is at elevated risk. The threshold depends on your product’s natural usage cadence — daily tools have a different baseline than monthly tools.

2. Feature usage count (trailing 14 days)
Count how many times each account triggered your core feature in the last two weeks. A drop of 50%+ vs their prior 14-day baseline is a meaningful signal.

3. Downgrade attempt
Any click on a downgrade CTA, pricing page revisit, or billing portal open — even if not completed — is a strong behavioral signal. These events are often tracked but rarely surfaced to the team.

Alone, each signal is noisy. Together, two or more active at the same time identifies accounts that deserve a proactive outreach.

The Setup: Apify + Google Sheets Pipeline

Here’s the architecture:

Usage data source → Apify HTTP Actor → Google Sheets → Conditional risk score → Alert

Step 1: Export Usage Data

Your usage data likely lives in one of three places:

  • A product analytics platform with a JSON/CSV export API
  • Your own database (Postgres, MySQL) exposed via an internal API endpoint
  • A third-party tool like Mixpanel or Amplitude with REST API access

For this walkthrough, assume you have an internal endpoint that returns a JSON array of account activity:

[
  {
    "account_id": "acc_001",
    "email": "[email protected]",
    "last_login_days_ago": 12,
    "feature_uses_14d": 3,
    "downgrade_attempt": false
  },
  {
    "account_id": "acc_002",
    "email": "[email protected]",
    "last_login_days_ago": 18,
    "feature_uses_14d": 1,
    "downgrade_attempt": true
  }
]

Step 2: Pull Data With Apify

Create an Apify actor using the HTTP request source to fetch from your internal endpoint. If your endpoint requires auth, pass the token as a header input.

Basic actor input:

{
  "url": "https://yourapp.com/api/usage-export",
  "method": "GET",
  "headers": {
    "Authorization": "Bearer YOUR_API_TOKEN"
  }
}

The actor runs, fetches the JSON array, and stores results in the Apify dataset. You can then push those results directly to a Google Sheet using Apify’s built-in Google Sheets integration — or export as CSV and import manually for a first test.

Step 3: Score Each Account in Google Sheets

Once your data is in Google Sheets, add a Churn Risk Score column using a simple formula:

=IF(B2>10,1,0) + IF(C2<5,1,0) + IF(D2=TRUE,1,0)

Where:

  • B2 = last login days ago (threshold: >10)
  • C2 = feature uses in 14 days (threshold: <5)
  • D2 = downgrade attempt (TRUE/FALSE)

This gives each account a score from 0 to 3. Accounts scoring 2 or 3 are your intervention list.

Add a conditional formatting rule: highlight rows where the risk score column is ≥ 2 in red. Now your weekly review starts with a visible list of at-risk accounts.

Calculating Thresholds for Your Product

The numbers above (10 days since login, <5 feature uses) are starting points, not universal rules. Calibrate them against your own data:

  1. Pull the last 90 days of usage for all accounts that churned
  2. Calculate their average last_login_days_ago and feature_uses_14d in the 2 weeks before they cancelled
  3. Set your thresholds at the median values from that cohort

This turns guesswork into a data-derived baseline. Even a rough calibration will outperform no system.

Step 4: Automate the Weekly Run

Schedule your Apify actor to run every Monday morning:

  1. Open your actor in Apify Console
  2. Go to Schedules → New Schedule
  3. Set: 0 8 * * 1 (8am every Monday)
  4. Enable Automatically push results to Google Sheets or trigger a webhook to your sheet

Now every Monday you have an updated risk dashboard without touching it manually.

For the alert: add a Google Sheets formula that emails you when any account hits score ≥ 2:

// Google Apps Script — trigger on sheet edit or time-based
function sendChurnAlert() {
  const sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
  const data = sheet.getDataRange().getValues();
  const atRisk = [];

  for (let i = 1; i < data.length; i++) {
    const score = data[i][4]; // column E = risk score
    if (score >= 2) {
      atRisk.push(data[i][1]); // column B = email
    }
  }

  if (atRisk.length > 0) {
    MailApp.sendEmail(
      "[email protected]",
      `Churn Alert: ${atRisk.length} at-risk accounts`,
      `Accounts needing outreach this week:n${atRisk.join("n")}`
    );
  }
}

Run this script on a Monday trigger, 30 minutes after the Apify actor fires.

The Cost: $0 Until It Saves Your First Subscriber

Apify charges based on compute. A simple HTTP fetch actor pulling a few hundred records runs in under 5 seconds. At Apify’s pay-per-use pricing, that’s fractions of a cent per run — roughly $0.00 to $0.02 per weekly run depending on dataset size.

Google Sheets is free. Apps Script is free.

The total infrastructure cost rounds to zero. The ROI depends entirely on what a retained subscriber is worth to you — and whether the outreach works.

What to Do With the Alert

When you get the Monday email listing at-risk accounts, the response doesn’t need to be elaborate:

  • Score 2: A short personal email. “Hey — noticed you haven’t been in recently. Anything blocking you?”
  • Score 3: Direct offer — a 15-minute call, an extended trial, or a feature unlock if relevant

The goal isn’t to beg customers to stay. It’s to have the conversation before they’ve made the decision. A customer who’s fading out is still recoverable. One who’s already cancelled is a much harder conversation.

Summary

The system has three parts:

  1. Data pull — Apify actor fetches your usage JSON weekly
  2. Risk scoring — Google Sheets formula flags accounts with 2+ warning signals
  3. Alert — Apps Script emails you the intervention list

Setup time: 2 to 3 hours. Running cost: effectively zero. The payoff shows up the first time you send an email to an at-risk account and they reply.

If you’re building on Apify or want to explore HTTP-source actors for other internal data pipelines, the Apify actor library has HTTP and API-based actors you can fork and adapt without writing infrastructure from scratch.

Built with Apify and Google Sheets. No proprietary tooling required.