Most developers approach XRPL event monitoring like it’s a simple polling problem. They spin up a script that checks wallet balances every few minutes, maybe set up a cron job to scan for new transactions. When they need real-time updates, they build a WebSocket listener that connects directly to the ledger.

This approach breaks down fast. WebSocket connections drop. Rate limits kick in. Transaction parsing gets complex when you’re dealing with DEX trades, escrow releases, and payment channel updates. The script that worked fine in development starts missing events in production.

The correct frame: XRPL event monitoring is an infrastructure problem, not a feature problem.

You need reliable delivery, retry logic, signature verification, and dead-letter queues. You need to handle 22 different event types across seven categories without your application logic becoming a mess of conditional statements. Most importantly, you need this infrastructure to work when your application scales beyond a few users.

I built XRNotify because every XRPL developer was solving the same infrastructure problem independently. The results were predictable: brittle listeners, missed events, no monitoring, and hours spent debugging webhook delivery failures instead of building product features.

The Infrastructure Layer XRPL Was Missing

XRNotify sits between the XRP Ledger and your application. It maintains persistent connections to the network, parses transaction data in real-time, and delivers structured events to your endpoints via webhooks.

The delivery mechanism uses exponential backoff retry with dead-letter queues for failed deliveries. Every payload includes HMAC-SHA256 signatures for verification. When your endpoint is down for maintenance, events queue up and deliver in order once you’re back online.

The event categorization covers wallet activity, DEX operations, escrow management, payment channels, NFT transfers, network state changes, and validator updates. Instead of parsing raw XRPL transaction data, you receive structured JSON with the specific information your application needs.

Beyond Simple Notifications

The real value emerges when you consider XRNotify as part of a larger data infrastructure. Network state data flows to The Neutral Bridge for financial research. Anomaly patterns feed into H.U.N.I.E.’s intelligence layer. The same infrastructure that delivers your application webhooks also powers the circuit breaker logic in H.U.N.I.E. Sentinel.

This isn’t accidental architecture. XRPL generates massive amounts of transaction data, but most of it gets processed once and discarded. XRNotify captures that data at the moment it becomes available, then routes it to multiple systems that extract different types of value.

Your application gets the specific events it needs. The research infrastructure gets comprehensive network state data. The security layer gets real-time anomaly detection. The same infrastructure investment serves multiple purposes.

Technical Implementation

The system runs on Next.js 14 with PostgreSQL for persistence and Redis for event queuing. Node.js workers handle the XRPL connections and webhook delivery. XRPL.js provides the ledger interface, but the heavy lifting happens in the retry logic and delivery infrastructure.

The webhook endpoints receive POST requests with JSON payloads. Each payload includes event metadata, transaction details, and the HMAC signature for verification. Your application validates the signature, processes the event, and returns a 200 status code. If validation fails or processing errors occur, the retry mechanism handles redelivery.

Event filtering happens at the subscription level. You specify which wallets to monitor, which event types to receive, and which endpoint should handle each category. The infrastructure handles the complexity of maintaining multiple XRPL connections, parsing different transaction types, and routing events to the correct destinations.

The Result

Developers connect their systems to XRPL events without building listener infrastructure. Applications respond to network activity in real-time without polling delays. The webhook delivery guarantees mean events reach your system even during deployment downtime or temporary outages.

This is infrastructure that works when you need it to work. No missed events, no parsing edge cases, no webhook delivery debugging at 2 AM.

https://www.xrnotify.io

As part of my volunteering with AbilityNet, I recently attended a training session on motor neurone disease (MND) — and it quietly shifted how I think about what technology is actually for.

What is MND?

MND affects nerves in the brain and spinal cord called motor neurones — the nerve cells that tell your muscles what to do. It’s a rare but progressive condition. There is no cure, and it’s usually life-shortening.

Around 5,000 people in the UK are living with MND at any one time. Six people are diagnosed every single day.

Voice banking

Voice banking was something I hadn’t encountered before this session. It allows a person to record their own voice or create personalised messages for use with speech-generating devices — and the MND Association recommends starting early, even before speech is significantly affected.

The goal is to preserve something that can’t be replaced once it’s gone: the sound of you.

Research shows that “preserving identity” is the overarching motivation for people who choose to voice bank. Yet only around 12% of people diagnosed with MND in the UK actually go through with it — because the process has historically been too long, too hard, and too clinical.

That gap between what’s available and what people can actually access is exactly where technology and human support need to meet.

Good technology serves the people

You don’t need to be in tech to show up for this. Anyone can volunteer, learn, and help bridge that gap.

If you want to understand more about what living with MND actually looks like, this is a good place to start:

Further reading:

• NHS — Motor Neurone Disease
• MND Association
• AbilityNet volunteer programme

Stop Paying $20/Month for Email You Could Run for $1.50

Most teams handle programmatic email one of two ways.

The first: sign up for SendGrid or Mailgun, grab an API key, and start sending. It works great — until you hit 50K emails and the bill jumps to $20/month, then you add inbound parsing and it’s another plan tier, then you need bounce webhooks and suddenly you’re deep in a third-party dashboard you barely understand, paying for features you don’t use.

The second: use Google Workspace’s catch-all rule to funnel everything into one inbox. Free-ish, but completely dumb — no processing, no routing, no automation. Every contact form submission and support@ email lands in the same pile for a human to sort manually.

There’s a better way that most AWS developers walk right past. Amazon SES catches the mail, S3 stores it, Lambda processes it. The logic lives in your codebase, the infrastructure is fully in your AWS account, and the bill last month was $1.48.

Let me show you exactly how it works.

The idea in one diagram

That’s it. SES stores the raw email in S3, then fires Lambda. Lambda rewrites the headers and re-sends via SES. Your team gets the email; they hit Reply; it goes back to whoever sent it.

The S3 step isn’t optional — it’s actually the smart part. Every email is durably stored before Lambda touches it. Parser bug? Replay from S3. Need an audit trail? It’s there, with 30-day auto-expiry built in.

This is not a replacement for Google Workspace or AWS WorkMail

Worth saying clearly before going further.

Google Workspace and AWS WorkMail are mailbox products — your team logs in, reads emails, manages a calendar, sends from their phone. That’s a completely different problem from what this stack solves.

The honest recommendation: run both. Keep Workspace or WorkMail for your team’s human mailboxes. Use this stack for everything your application touches — the support@ alias that should file a ticket, the contact@ that should notify Slack, the noreply@ that sends password resets.

They’re not competing. They solve different problems.

Why not just use SendGrid / Postmark / Mailgun?

Nothing wrong with them. But look at the math for a typical small company:

SES is $0.10 per 1,000 emails. That rate has not changed since 2011. At 500K sends, you’re paying $50 while Postmark charges $650 for the same volume. The gap is real.

The catch: SES is infrastructure, not a product. No drag-and-drop campaign builder, no deliverability dashboard (unless you pay extra for VDM). If you want someone to hold your hand through email, pay for managed. If you’re comfortable in AWS, this stack is hard to beat.

Keep Google Workspace or Microsoft 365 for your human mailboxes. This pattern is for the email your code handles — contact forms, support@ aliases, alerts, password resets.

What the deploy actually creates

One SAM template provisions everything:

• S3 bucket — stores raw emails under inbox/, auto-deletes after 30 days
• SES identity — verifies your domain with DKIM signing enabled
• Route 53 records — MX, SPF, DMARC, and all 3 DKIM CNAMEs, created automatically
• Lambda + IAM — reads from S3, forwards via SES
• Receipt rule — catches all mail at your domain, stores then invokes Lambda
• Custom resource — activates the rule set automatically so you don’t have to

sam build
sam deploy --guided 
  --parameter-overrides 
    EmailDomain=yourcompany.com 
    ForwardTo=team@yourcompany.com 
    ForwardFrom=relay@yourcompany.com 
    HostedZoneId=ZXXXXXXXXXXXXX

One command. Everything wired up.

The code

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Serverless email infrastructure (SES + S3 + Lambda)

Parameters:
  EmailDomain:
    Type: String
    Description: Domain configured to receive email (e.g. yourcompany.com)
  ForwardTo:
    Type: String
    Description: Address to forward incoming mail to
  ForwardFrom:
    Type: String
    Description: Verified sender used as the From on forwards
  HostedZoneId:
    Type: String
    Description: Route 53 Hosted Zone ID for the domain

Resources:

  # -------------------------------------------------------
  # S3 — raw email storage
  # -------------------------------------------------------
  EmailBucket:
    Type: AWS::S3::Bucket
    Properties:
      LifecycleConfiguration:
        Rules:
          - Id: ExpireRawEmails
            Status: Enabled
            ExpirationInDays: 30

  EmailBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref EmailBucket
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AllowSESPuts
            Effect: Allow
            Principal:
              Service: ses.amazonaws.com
            Action: s3:PutObject
            Resource: !Sub '${EmailBucket.Arn}/*'
            Condition:
              StringEquals:
                AWS:SourceAccount: !Ref AWS::AccountId

  # -------------------------------------------------------
  # Lambda — parse and forward
  # -------------------------------------------------------
  ProcessEmailFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.12
      Handler: app.lambda_handler
      CodeUri: ./src/
      MemorySize: 256
      Timeout: 30
      Environment:
        Variables:
          FORWARD_TO: !Ref ForwardTo
          FORWARD_FROM: !Ref ForwardFrom
          BUCKET_NAME: !Ref EmailBucket
          BUCKET_PREFIX: inbox/
      Policies:
        - S3ReadPolicy:
            BucketName: !Ref EmailBucket
        - Statement:
            - Effect: Allow
              Action: ses:SendRawEmail
              Resource: '*'

  ProcessEmailFunctionSESPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt ProcessEmailFunction.Arn
      Principal: ses.amazonaws.com
      SourceAccount: !Ref AWS::AccountId

  # -------------------------------------------------------
  # SES — identity, receipt rule set and rule
  # -------------------------------------------------------
  SESEmailIdentity:
    Type: AWS::SES::EmailIdentity
    Properties:
      EmailIdentity: !Ref EmailDomain
      DkimAttributes:
        SigningEnabled: true

  ReceiptRuleSet:
    Type: AWS::SES::ReceiptRuleSet
    Properties:
      RuleSetName: !Sub '${AWS::StackName}-rules'

  ReceiptRule:
    Type: AWS::SES::ReceiptRule
    DependsOn:
      - EmailBucketPolicy
      - SESEmailIdentity
    Properties:
      RuleSetName: !Ref ReceiptRuleSet
      Rule:
        Name: store-and-process
        Enabled: true
        ScanEnabled: true
        Recipients:
          - !Ref EmailDomain
        Actions:
          - S3Action:
              BucketName: !Ref EmailBucket
              ObjectKeyPrefix: inbox/
          - LambdaAction:
              FunctionArn: !GetAtt ProcessEmailFunction.Arn
              InvocationType: Event

  # -------------------------------------------------------
  # Route 53 — all DNS records created automatically
  # -------------------------------------------------------
  DkimRecord1:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref HostedZoneId
      Name: !Sub "${SESEmailIdentity.DkimDNSTokenName1}"
      Type: CNAME
      TTL: 300
      ResourceRecords:
        - !Sub "${SESEmailIdentity.DkimDNSTokenValue1}"

  DkimRecord2:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref HostedZoneId
      Name: !Sub "${SESEmailIdentity.DkimDNSTokenName2}"
      Type: CNAME
      TTL: 300
      ResourceRecords:
        - !Sub "${SESEmailIdentity.DkimDNSTokenValue2}"

  DkimRecord3:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref HostedZoneId
      Name: !Sub "${SESEmailIdentity.DkimDNSTokenName3}"
      Type: CNAME
      TTL: 300
      ResourceRecords:
        - !Sub "${SESEmailIdentity.DkimDNSTokenValue3}"

  MXRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref HostedZoneId
      Name: !Sub "${EmailDomain}."
      Type: MX
      TTL: 300
      ResourceRecords:
        - !Sub "10 inbound-smtp.${AWS::Region}.amazonaws.com"

  SPFRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref HostedZoneId
      Name: !Sub "${EmailDomain}."
      Type: TXT
      TTL: 300
      ResourceRecords:
        - '"v=spf1 include:amazonses.com -all"'

  DMARCRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref HostedZoneId
      Name: !Sub "_dmarc.${EmailDomain}."
      Type: TXT
      TTL: 300
      ResourceRecords:
        - !Sub '"v=DMARC1; p=none; rua=mailto:dmarc@${EmailDomain}"'

  # -------------------------------------------------------
  # Custom resource — activates the rule set automatically
  # -------------------------------------------------------
  ActivateRuleSetFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.12
      Handler: index.handler
      InlineCode: |
        import boto3, cfnresponse
        def handler(event, context):
          try:
            if event['RequestType'] in ('Create', 'Update'):
              boto3.client('ses').set_active_receipt_rule_set(
                RuleSetName=event['ResourceProperties']['RuleSetName']
              )
            cfnresponse.send(event, context, cfnresponse.SUCCESS, {})
          except Exception as e:
            cfnresponse.send(event, context, cfnresponse.FAILED, {'Error': str(e)})
      Policies:
        - Statement:
            - Effect: Allow
              Action: ses:SetActiveReceiptRuleSet
              Resource: '*'

  ActivateRuleSet:
    Type: AWS::CloudFormation::CustomResource
    DependsOn: ReceiptRuleSet
    Properties:
      ServiceToken: !GetAtt ActivateRuleSetFunction.Arn
      RuleSetName: !Ref ReceiptRuleSet

Outputs:
  BucketName:
    Description: S3 bucket storing raw incoming emails
    Value: !Ref EmailBucket
  RuleSetName:
    Description: Active receipt rule set
    Value: !Ref ReceiptRuleSet
  LambdaFunction:
    Description: Email processing function
    Value: !GetAtt ProcessEmailFunction.Arn

src/app.py

The key insight here is to mutate the original message in-place rather than building a new MIME object from scratch. This preserves attachments, encoding, and HTML content without any extra parsing. The critical step is stripping the headers that would cause DKIM verification to fail on the re-sent message.

import os
import email
import logging

import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3  = boto3.client("s3")
ses = boto3.client("ses")

FORWARD_TO    = os.environ["FORWARD_TO"]
FORWARD_FROM  = os.environ["FORWARD_FROM"]
BUCKET_NAME   = os.environ["BUCKET_NAME"]
BUCKET_PREFIX = os.environ["BUCKET_PREFIX"]  # "inbox/"

# These headers are tied to the original message signature.
# Keeping them causes DKIM failure on the forwarded copy.
HEADERS_TO_REMOVE = [
    "DKIM-Signature",
    "Sender",
    "Return-Path",
    "Message-ID",
]


def lambda_handler(event, context):
    for record in event["Records"]:
        mail    = record["ses"]["mail"]
        receipt = record["ses"]["receipt"]

        msg_id     = mail["messageId"]
        recipients = receipt["recipients"]
        s3_key     = f"{BUCKET_PREFIX}{msg_id}"

        logger.info("Processing %s for recipients %s", msg_id, recipients)

        raw = s3.get_object(Bucket=BUCKET_NAME, Key=s3_key)["Body"].read()
        msg = email.message_from_bytes(raw)

        original_from    = msg.get("From", "unknown")
        original_subject = msg.get("Subject", "(no subject)")

        for header in HEADERS_TO_REMOVE:
            if header in msg:
                del msg[header]

        # Rewrite From so SES accepts the send
        if "From" in msg:
            msg.replace_header("From", FORWARD_FROM)
        else:
            msg["From"] = FORWARD_FROM

        # Reply-To preserves the original sender — your team hits Reply
        # and it goes back to the person who sent it, not to your relay address
        if "Reply-To" in msg:
            msg.replace_header("Reply-To", original_from)
        else:
            msg["Reply-To"] = original_from

        # Subject prefix tells you which alias received it
        original_recipient = recipients[0] if recipients else ""
        new_subject = f"[{original_recipient}] {original_subject}"
        if "Subject" in msg:
            msg.replace_header("Subject", new_subject)
        else:
            msg["Subject"] = new_subject

        if "To" in msg:
            msg.replace_header("To", FORWARD_TO)
        else:
            msg["To"] = FORWARD_TO

        ses.send_raw_email(
            Source=FORWARD_FROM,
            Destinations=[FORWARD_TO],
            RawMessage={"Data": msg.as_bytes()},
        )
        logger.info("Forwarded %s → %s → %s", msg_id, original_recipient, FORWARD_TO)

    return {"statusCode": 200}

Deploying it

Prerequisites: AWS CLI configured, SAM CLI v1.129+, a domain in Route 53.

1. Get your hosted zone ID

aws route53 list-hosted-zones 
  --query "HostedZones[?Name=='yourcompany.com.'].{Id:Id,Name:Name}"

2. Build and deploy

sam build
sam deploy --guided 
  --parameter-overrides 
    EmailDomain=yourcompany.com 
    ForwardTo=team@yourcompany.com 
    ForwardFrom=relay@yourcompany.com 
    HostedZoneId=ZXXXXXXXXXXXXX

SAM will ask a few questions on first run — accept the defaults, confirm the changeset. The deploy creates everything: S3, Lambda, SES identity, all DNS records, and activates the rule set automatically.

3. Request production access

New SES accounts are sandboxed — you can only send to verified addresses until you request production access. Go to SES Console → Account dashboard → Request production access, describe your use case in two sentences. AWS typically responds within 24 hours.

4. Test it

Send an email to any address at your domain and watch the logs:

aws logs tail /aws/lambda/YOUR_STACK_NAME-ProcessEmailFunction 
  --follow 
  --region us-east-1

You should see the message ID logged, then a forwarded confirmation within a couple of seconds.

If domain verification stays Pending

After deploy, SES verifies your domain by polling the DKIM CNAME records. It usually completes within 5–10 minutes. If it’s been 30+ minutes and still shows Pending, check that DNS is actually resolving:

# Replace with one of your actual DKIM token values
nslookup -type=CNAME abc123._domainkey.yourcompany.com 8.8.8.8

If that resolves correctly, nudge SES to recheck immediately rather than waiting for its next polling cycle:

aws ses verify-domain-identity 
  --domain yourcompany.com 
  --region us-east-1

# Then check status
aws ses get-identity-verification-attributes 
  --identities yourcompany.com 
  --region us-east-1

If DNS isn’t resolving at all, the CNAME records weren’t created cleanly — check the CloudFormation events for any Route 53 errors during deploy.

A few gotchas learned the hard way

SES only allows one active rule set per region per account. If you have multiple environments in the same region, they share this single slot. The solution is one rule set with multiple rules — one rule per domain — rather than multiple rule sets competing for the active slot.

New accounts start in sandbox mode. You can only send to verified addresses until you request production access. Do it on day one — AWS usually approves within 24 hours and there’s no reason to wait.

Not every region supports inbound. Stick to us-east-1, us-west-2, or eu-west-1. Pick your region before pointing your MX record.

Watch your bounces. SES will throttle your account if you keep sending to addresses that bounce. Enable Account-Level Suppression in the SES console — it’s free and it protects you automatically.

Want to go further?

This same stack is the backbone for a contact form endpoint — a Lambda Function URL receives the POST, validates the fields, and calls ses.send_email. Same verified domain, same IAM, a few dozen extra lines. Add a honeypot field and you get basic bot protection for free.

You can also extend the forwarder to route different aliases to different people (support@ → helpdesk, billing@ → finance) by checking recipients[0] in the Lambda and maintaining a simple routing table in the code or in DynamoDB.

The foundation is solid. Build on it.

The Problem We Were Actually Solving

I was trying to sell digital products online from a country that was blocked by most major payment platforms, including PayPal, Stripe, Gumroad, and Payhip. As the founder of a small business, this was a major obstacle to my ability to reach customers and generate revenue. I spent countless hours researching alternative payment solutions, but every door I tried to open seemed to be met with a new set of restrictions and requirements. It was clear that this was a platform problem, not a problem with my business or my products.

What We Tried First (And Why It Failed)

My first approach was to try to use VPNs and other workarounds to pretend to be located in a country that was not blocked by these platforms. However, this approach was not only risky and potentially illegal, but it also did not work. The payment platforms had sophisticated systems in place to detect and prevent this type of activity, and I was quickly flagged and banned from using their services. I also tried to use smaller, lesser-known payment platforms, but these often had their own set of problems, including high fees, poor customer support, and limited functionality. After months of trying different solutions, I was on the verge of giving up.

The Architecture Decision

It was at this point that I made the decision to take a completely different approach. Instead of trying to work with existing payment platforms, I decided to build my own custom payment solution using a combination of open-source software and local payment providers. This was a daunting task, as it would require significant development and testing resources, but I was convinced that it was the only way to truly bypass the restrictions and achieve my business goals. I chose to use a modular architecture, with separate components for payment processing, order management, and customer management. This would allow me to easily integrate with different payment providers and switch between them as needed. I also decided to use a microservices-based approach, with each component running as a separate service. This would provide greater flexibility and scalability, and would allow me to quickly respond to changes in the market or in the requirements of my business.

What The Numbers Said After

The results of this decision were nothing short of astonishing. Within six months, I was able to launch my custom payment solution and start processing transactions. The fees were significantly lower than those charged by the major payment platforms, and the customer support was vastly improved. I was also able to customize the solution to meet the specific needs of my business, including integrating with my existing order management and customer management systems. In terms of metrics, I saw a 30% increase in sales within the first year, and a 25% decrease in payment processing fees. The solution also provided greater scalability and reliability, with uptime of 99.99% and average transaction processing time of under 2 seconds. I used tools such as Prometheus and Grafana to monitor the performance of the system, and was able to quickly identify and resolve any issues that arose.

What I Would Do Differently

In retrospect, I would have made the decision to build a custom payment solution much earlier. I would have also invested more in testing and quality assurance, as this would have helped to identify and resolve issues more quickly. I would also have chosen to use more cloud-based services, such as AWS Lambda and API Gateway, to provide greater scalability and flexibility. Additionally, I would have implemented more robust security measures, such as two-factor authentication and encryption, to protect sensitive customer data. However, overall, I am extremely satisfied with the decision to build a custom payment solution, and I believe that it has been a key factor in the success of my business. I used tools such as Jenkins and GitLab CI/CD to automate the testing and deployment of the solution, and was able to quickly and easily make changes to the system as needed.

The anatomy of a container tells you what the walls are made of. This article tells you when they go up, what happens inside them, and what the kernel does the moment they come down.

In the previous article, we dissected a running container at a single moment in time: the namespaces building the illusion of isolation, the cgroups enforcing the resource constitution, OverlayFS merging layers into a coherent filesystem, the veth pair wiring the container into the network. A cross-section. A photograph.

But a photograph doesn’t tell you how the subject got there, or what happens after the shutter closes.

That’s what this piece is for. We’ll follow a single container from before it exists to after it’s gone — and at each transition, we’ll look at what actually changes at the kernel level. The machinery is the same as before. Now we watch it move.

We’ll also cover the three places where production systems quietly break in ways that monitoring won’t catch until it’s too late: signal handling, zombie processes, and the OOM killer.

Before the Container: The Image as a Promise

Before anything runs, the runtime needs something to run from. An image isn’t a virtual machine disk or a binary blob — it’s a stack of read-only filesystem layers, each one a delta on the last, stored under /var/lib/docker/overlay2/ and identified by SHA256 digest rather than name.

When you run docker pull, the daemon fetches a manifest first: a JSON document listing every layer by digest. It then checks which of those digests already exist locally and downloads only the missing ones. This is why pulling a new version of an image that shares a base layer with one you already have takes seconds — the common layers are already there. The registry protocol is content-addressed, which means layers are inherently deduplicated across every image on the host that uses them.

Each downloaded layer is verified against its digest before being committed to the store. A tampered or corrupted layer fails this check before it touches anything.

The image at rest is inert. It’s a promise of what the container’s filesystem will look like when it starts — nothing more. The OverlayFS that merges those layers into a live, writable filesystem doesn’t exist yet. That comes with create.

Stage 1: docker create — Assembling the Environment

Here is something most engineers don’t know: docker run is not a single operation. It is docker create followed immediately by docker start. Knowing this distinction is practically useful — but more importantly, it clarifies when each piece of kernel infrastructure comes into existence.

docker create builds the complete environment the container will inhabit. Nothing executes. No process runs. It is pure allocation.

The OverlayFS mount. The runtime creates a new upperdir — an empty, writable directory unique to this container instance. The image’s read-only layers become the lowerdir. The OverlayFS is configured to present both as a single unified filesystem: reads fall through to the lower layers, writes land in the upper layer, and the container will see a seamless root filesystem when it starts. The layers you read about in the anatomy piece are present here. The writable layer that captures everything your container does is created here.

The namespace allocation. The six namespaces — PID, NET, MNT, IPC, UTS, USER — are prepared. At this point they’re reserved but not yet active. There’s nothing to isolate, so isolation hasn’t started.

The cgroup hierarchy. A cgroup is created for this container under /sys/fs/cgroup/. The memory limit from --memory 512m and the CPU quota from --cpus 0.5 are written into the cgroup’s control files right now. The cgroup exists. The limits are set. But with no processes inside it, it enforces nothing — a constitution with no citizens.

The network configuration. The veth pair is created: one end destined for the container’s network namespace, one end connecting to the docker0 bridge on the host. An IP is assigned. NAT rules are written to iptables for port mappings. The virtual cable exists, but nothing is plugged in on the container end yet.

After docker create, the container is a complete kernel data structure — every wall built, every rule written, every wire run — with no tenant. Its state is "created". Nothing has been charged to the CPU. Nothing has touched the writable layer.

The practical implication: you can pre-warm containers during off-peak hours and start them in milliseconds when demand arrives. The expensive work of building the environment is already done.

Stage 2: docker start — Crossing the Threshold

This is the moment the environment becomes inhabited. Three components work in sequence: dockerd, containerd, and containerd-shim.

dockerd instructs containerd to start the container. containerd forks a containerd-shim — the small, dedicated supervisor we covered in the daemon article — which then calls runc.

runc is where the kernel call happens. It takes the container’s prepared configuration and calls clone() with the namespace flags: CLONE_NEWPID, CLONE_NEWNET, CLONE_NEWNS, and the rest. The namespaces activate. The OverlayFS is mounted. The cgroup begins accounting. The veth pair connects. Then runc calls execve() with whatever you specified as ENTRYPOINT or CMD.

That process — your application — becomes PID 1 inside the container.

Once execve() returns, runc exits. It has done its job. The containerd-shim stays behind, holding the container’s stdin/stdout file descriptors and watching PID 1. If dockerd crashes, the shim keeps the container alive — which is the architecture we walked through in detail previously.

The moment docker start completes, everything the anatomy article described is live and active: the isolated process tree, the private network stack, the merged filesystem, the cgroup enforcing limits. The photograph is now a film.

And this is the first place things go quietly wrong. PID 1 carries obligations that your application almost certainly doesn’t know about. We’ll come back to that shortly.

Stage 3: Running — What the Commands Actually Do

A running container looks opaque from the outside. Under the hood, every docker command in this state is a thin wrapper over kernel operations you already understand.

docker exec is the most commonly misunderstood. When you run docker exec -it myapp-prod /bin/sh, Docker doesn’t create a new container. It calls setns() — a system call that lets a process join existing namespaces. Your shell enters the container’s PID and network namespaces and sees the same environment the application sees. Nothing new is created. You’re walking into an existing apartment, not building a new one.

docker logs reads from the file descriptors the containerd-shim has been holding since start. The shim inherited the container’s stdout and stderr pipes and has been buffering them since the first byte was written. No daemon magic — the shim is just a persistent pipe holder.

docker stats reads cgroup control files directly: /sys/fs/cgroup/memory.current for RSS, CPU accounting files for usage. The numbers you see in the terminal are the same numbers the kernel is maintaining to enforce your limits.

docker pause is the most elegant operation in this set. It uses the cgroup freezer subsystem to send SIGSTOP to every process in the container simultaneously — not one at a time, but atomically, via the cgroup. They freeze mid-instruction with no opportunity to catch the signal or react. docker unpause sends SIGCONT through the same path and they resume exactly where they stopped. In-flight operations, open sockets, memory contents — all preserved.

Stage 4: docker stop — The Two-Phase Shutdown

This is where most production problems originate, and where the PID 1 decision made at start time comes due.

docker stop is a two-phase protocol. Phase one: SIGTERM is sent to PID 1. The application has a grace period — ten seconds by default, adjustable with -t — to finish what it’s doing and exit cleanly. Close database connections. Drain in-flight requests. Flush logs. Phase two: if PID 1 is still alive when the timer expires, SIGKILL is sent. There is no negotiation with SIGKILL. The kernel terminates the process immediately and unconditionally.

The failure mode that bites teams most often: the shell wrapper.

Many containers are launched like this: CMD ["sh", "-c", "node server.js"]. The shell becomes PID 1. When Docker sends SIGTERM to PID 1, the shell receives it — and by default, shells do not forward signals to their children. Your Node.js process never sees SIGTERM. Ten seconds pass. SIGKILL arrives. The server dies mid-request, connections drop, and the exit code suggests a crash rather than a clean shutdown.

The fix is to make your application PID 1 directly. Use the exec form: CMD ["node", "server.js"]. Or, if you need a shell script for setup, end it with exec node server.js — the exec replaces the shell process rather than forking a child, so your application inherits PID 1.

Ten seconds is also almost always too short. An application with database connection pools to drain and requests to finish needs more. Set -t 30 as a floor. For services handling long-lived connections, -t 60 or higher is appropriate. The grace period is your only window for a clean shutdown — size it honestly.

After a successful stop, the container’s state becomes "exited". Crucially, the writable OverlayFS layer is preserved — every file the application created or modified is still there. The container can be restarted with docker start and will resume from exactly the filesystem state it stopped in.

The First Danger Zone: Zombie Processes

On a normal Linux host, PID 1 is the init system. One of init’s core responsibilities — one so fundamental it’s baked into the kernel’s design — is reaping dead child processes. When a process exits, it doesn’t fully disappear. Its kernel entry stays open, holding its exit code, until its parent calls wait() to collect that code. Until then, it sits in state Z: a zombie. Dead, but not yet buried.

Init’s job is to call wait() on every process that loses its parent, so the kernel slot gets freed. Without init doing this, zombie entries accumulate indefinitely.

In a container, your application is PID 1. And your application — a web server, an API, a worker — was written to serve requests, not to supervise processes. It almost certainly never calls wait().

The scenario: your web server forks a worker to handle a long request. The worker finishes and calls exit(). It waits for its parent to acknowledge the exit. The parent never does. The worker’s kernel entry sits in Z state. Do this at scale, over days of production traffic, and the PID table fills. Eventually fork() fails — the container cannot spawn new processes — and the application breaks in a way that looks nothing like its actual cause.

docker stats will show the container as healthy throughout. CPU and memory are fine. The zombie count isn’t surfaced by any standard metric.

The fix is a proper init process sitting in front of your application. tini is the canonical choice — purpose-built for containers, less than a thousand lines of C, does one thing: reap zombies and forward signals correctly.

RUN apt-get install -y tini
ENTRYPOINT ["/tini", "--", "/usr/local/bin/myapp"]

If you don’t want to touch the image, docker run --init injects a bundled init binary automatically. Use one or the other on every production container, as a default, without exception.

To check for zombies in a running container:

docker exec myapp-prod ps aux | grep ' Z '

Any output means you have a problem. The fix is tini. The time to add it is before the deployment, not during the incident.

The Second Danger Zone: The OOM Killer

The cgroup memory limit — set at create time, written to memory.max in cgroup v2 — is not a suggestion. When a container’s resident memory usage reaches that ceiling, the kernel’s Out-Of-Memory killer activates.

The OOM killer scores every process in the cgroup by a “badness” calculation: how much memory the process uses relative to total system memory, adjusted by that process’s oom_score_adj value. The highest scorer is sent SIGKILL. In a single-process container, there is exactly one candidate. PID 1 always loses.

The critical distinction from docker stop: the OOM killer sends no SIGTERM. There is no grace period. No warning, no chance to flush state, close connections, or write a final log line. The process simply stops existing. Docker records this as OOMKilled: true in the container’s inspect output:

docker inspect myapp-prod --format '{{.State.OOMKilled}}'
true

The kernel also logs it to dmesg:

dmesg | grep -i "oom|killed process"

Mitigation works at two levels. The first is setting a soft limit below the hard limit:

docker run --memory 512m --memory-reservation 400m myapp

--memory-reservation tells the kernel to apply gentle memory pressure — nudging the container’s garbage collector, encouraging page reclamation — before the hard ceiling is reached. It’s an early warning system rather than a wall. The container can still exceed the reservation temporarily; only --memory is the hard stop.

The second level is treating OOM kills as bugs, not tuning parameters. A container that is repeatedly OOM-killed has either a memory leak or limits set below its actual working set. Raising the limit is a short-term fix. Finding the leak is the real work.

Stage 5: docker rm — Dismantling the Environment

docker rm is the exact inverse of docker create. Every kernel structure that create allocated is released in reverse.

The OverlayFS upper layer is deleted. Every file your application created, modified, or deleted since the container started — the entire writable history — is gone. If your application wrote logs or state directly to the container filesystem, it’s unrecoverable. This is the reason containerised applications should write persistent data to mounted volumes, not the container filesystem: docker rm treats the writable layer as ephemeral by design.

The veth pair is removed. The IP address returns to Docker’s pool. The iptables NAT rules for port mappings are deleted. The network configuration from create is fully unwound.

The cgroup hierarchy is destroyed. The entries under /sys/fs/cgroup/ that tracked this container’s memory and CPU are removed. The kernel stops accounting for this container entirely.

What survives: the image layers. The read-only lowerdir that formed the container’s base filesystem is untouched — available immediately for the next container. And named volumes survive unless you pass --volumes to docker rm. Persistent data stored in volumes outlives the container that created it, which is the intended design.

The Full Picture

The anatomy article showed you the ingredients. This one showed you the recipe — the same namespaces, cgroups, OverlayFS, and veth pairs assembled in sequence, operated during a container’s life, and dismantled at its end.

The mental model to carry forward:

Create is allocation, start is execution. Every kernel structure — namespaces, cgroups, OverlayFS, veth pairs — exists before your application runs a single instruction. This is why pre-warming works and why restart is fast.

PID 1 is a contract, not just a position. The kernel gives PID 1 responsibilities that init normally handles: signal forwarding and zombie reaping. If your application holds that position, it inherits those responsibilities. Use tini or --init by default.

SIGTERM is your only warning. The grace period between SIGTERM and SIGKILL is the only window you get for a clean shutdown. Your application must handle the signal, and the window must be sized for your actual workload. Ten seconds is almost always too short.

The OOM killer gives no warning at all. There is no SIGTERM, no grace period, no log line from your application. If OOMKilled: true appears in production, treat it as a bug — because it is one.

The writable layer is temporary. docker rm destroys it completely. Write persistent state to volumes. Treat the container filesystem as scratch space that disappears when the container does.

docker run looks like a single command. It is actually the sequential execution of eight distinct kernel operations, each with observable state and specific failure modes. The engineers who debug containers fastest aren’t the ones who know the most flags — they’re the ones who can trace a problem back to the operation where the kernel did something unexpected.

Now you can.

Part of an ongoing series on container internals. Previous: A Deep Dive into Docker: What Ticks Under the Hood?

The problem

When a port is already in use, developers have to chain together lsof, grep, and kill commands to find and stop the offending process — there is no interactive way to scan all listening ports and act on them in one place.

If you’ve hit EADDRINUSE :::3000 before, you know the ritual: lsof -i :3000, then grep, then kill. Every time.

As a solution, I created port-peek

Interactive TUI for browsing all listening ports and killing the process behind any port with a keypress

Zero dependencies. Run it immediately:

npx port-peek

Output:

port-peek v1.0.0 — Listening ports

  PORT    PID       PROCESS
  3000    91234     node
▶ 5432    823       postgres
  8080    45612     python3
  8888    45990     node

  ↑↓ navigate  Enter kill  r refresh  q quit

Arrow keys to navigate, Enter to kill, r to refresh.

How it works

Pure Node.js using child_process.execSync to run lsof -iTCP -sTCP:LISTEN on macOS (with ss/netstat fallback on Linux), then renders a full-screen ANSI TUI with raw mode stdin for arrow-key navigation and Enter-to-kill. No dependencies.

Why I built it

Found recurring complaints on r/node and r/webdev about the EADDRINUSE error forcing multi-step shell rituals to find and kill the blocking process. The top Stack Overflow answer for ‘kill process on port’ has millions of views, confirming this is an extremely common pain point. No popular zero-dep npm package provides an interactive alternative — they are all one-shot kill commands that require you to know the port number upfront. An interactive TUI felt like the right format because developers often want to survey all listening ports before deciding which to kill.

Try it

npx port-peek --help

Part of µ micro — one new developer tool, shipped every day.

Se você trabalha remoto, com certeza já passou por isso: você está em uma reunião super importante, começa a digitar suas notas ou responder um chat, e alguém solta o clássico: “Ewerton, dá uma segurada aí que o seu teclado está parecendo uma metralhadora!”.

O barulho de cliques mecânicos ou digitação mais agressiva é o terror das chamadas de vídeo. Embora ferramentas como Google Meet e Zoom tenham seus próprios filtros, eles nem sempre dão conta ou acabam pesando demais no navegador.

Se você usa Linux (como o Ubuntu), existe uma solução open-source, ultra leve e que roda direto na sua máquina (sem mandar seu áudio para nuvens terceiras): o NoiseTorch.

Neste post, vou te mostrar o passo a passo definitivo para instalar, corrigir os bugs clássicos de ícone no Ubuntu e colocar ele para rodar na sua próxima reunião.

O que é o NoiseTorch?
O NoiseTorch cria um microfone virtual no seu sistema. Ele captura o áudio do seu microfone físico, passa esse som por uma biblioteca de inteligência artificial leve (chamada RNNoise) que filtra o que é voz e o que é ruído (teclado, ventilador, cachorro latindo), e entrega um áudio totalmente limpo para o Zoom, Meet, Slack ou Teams.

Passo a Passo da Instalação no Ubuntu
Como o NoiseTorch não está na loja padrão do Ubuntu, nós vamos fazer a instalação pelo terminal. Preparado? Abre o terminal (Ctrl + Alt + T) e vamos nessa:

Passo 1: Instalar as dependências básicas
Primeiro, precisamos garantir que o sistema tem as ferramentas necessárias para baixar e descompactar os arquivos:

sudo apt update && sudo apt install -y wget tar gtk-2.0

Passo 2: Baixar a versão estável e correta
Para evitar links quebrados ou arquivos HTML corrompidos que dão erro no terminal, baixe diretamente a versão estável (v0.12.2) direto para a sua pasta de Downloads:

cd ~/Downloads && wget https://github.com/noisetorch/NoiseTorch/releases/download/v0.12.2/NoiseTorch_x64_v0.12.2.tgz

Passo 3: Extrair para os diretórios locais
Os pacotes do NoiseTorch já vêm organizados na estrutura de pastas que o Linux usa. Vamos descompactar tudo direto na pasta local do seu usuário (~/.local):

tar -C $HOME -h -xzf NoiseTorch_x64_v0.12.2.tgz

(Se quiser limpar a casa, já pode apagar o instalador rodando rm NoiseTorch_x64_v0.12.2.tgz).

Passo 4: Dar permissões de sistema (Essencial!)
O NoiseTorch precisa gerenciar os fluxos de áudio do seu PulseAudio ou PipeWire sem precisar rodar como administrador (root). Para dar essa permissão especial a ele, rode:

sudo setcap 'CAP_SYS_RESOURCE=+ep' ~/.local/bin/noisetorch

Corrigindo o Bug do “Ícone de Engrenagem” no Ubuntu
Se você tentar rodar o NoiseTorch agora, o Ubuntu pode exibir apenas uma engrenagem genérica no dock e não vai deixar você fixar o aplicativo nos favoritos.

Para resolver isso e deixar o atalho com o ícone roxo oficial bonitinho, rode este bloco de comandos para atualizar o cache do GNOME:

# Corrige o caminho do executável dentro do atalho desktop
sed -i "s|Exec=noisetorch|Exec=$HOME/.local/bin/noisetorch|g" ~/.local/share/applications/noisetorch.desktop

# Garante que o ícone do app vá para a pasta correta de renderização
mkdir -p ~/.local/share/icons/hicolor/apps/
cp ~/.local/share/icons/hicolor/scalable/apps/noisetorch.svg ~/.local/share/icons/hicolor/apps/ 2>/dev/null || true

# Força o Ubuntu a ler as modificações de aplicativos
update-desktop-database ~/.local/share/applications
gtk-update-icon-cache -f ~/.local/share/icons/hicolor

Como Usar na Prática
Abra o menu de aplicativos do Ubuntu (tecla Super / Windows) e busque por NoiseTorch.

Abra o app, selecione o seu Microfone Real na lista que aparecer na tela e clique no botão grande “Load NoiseTorch”.

Agora, quando você entrar no Google Meet, Zoom ou Slack, vá nas configurações de áudio da chamada e mude o microfone para:

NoiseTorch Virtual Microphone

Como testar se funcionou?
Quer ter certeza de que o teclado sumiu antes de entrar na sala com o cliente ou com o time? Você pode gravar um áudio local de 10 segundos para testar. Rode no terminal:

arecord -D pulse -f cd -d 10 teste_ruido.wav

Nos primeiros 5 segundos: Fale normalmente enquanto digita no teclado.

Nos últimos 5 segundos: Fique em silêncio e apenas digite com força.

Para ouvir o resultado, execute:

aplay teste_ruido.wav

Se tudo deu certo, na metade final do áudio você vai ver que o NoiseTorch simplesmente dropou o som das suas teclas, deixando um silêncio absoluto. Salvação pura para o dia a dia do home office!

E você, já conhecia o NoiseTorch ou sofria configurando o áudio em cada app diferente? Deixa nos comentários se o tutorial funcionou para você!

This is a submission for the Google I/O Writing Challenge

Google I/O 2026 wrapped on May 20 at the Shoreline Amphitheater in Mountain View, and I’ve spent the last two days reading every blog post, watching every session replay almost but not all, and digging through the actual documentation. The headline everyone ran with was “Gemini Spark is your new 24/7 AI agent” and “Antigravity 2.0 built an operating system from scratch.” Those are flashy. They make good tweets.

But something much more interesting happened at this I/O, something that actually changes how Android developers will write code starting after the keynote address, and most of the coverage missed it entirely. I want to talk about that. I also want to walk through Antigravity 2.0 in detail, because Google fundamentally rebuilt it, and the technical decisions they made reveal a lot about where developer tooling is heading. Finally, I want to look at the bigger picture of what Google announced and what it means for the people who actually have to ship apps.

There’s a lot to cover. I’ll try to keep it useful.

The big picture: Agents aren’t a demo anymore

Before diving into the specific tools, let’s establish what actually launched.

Gemini 3.5 Flash is the new default model powering most of this. It outperforms last year’s Gemini 3.1 Pro on coding and agentic benchmarks: 76.2% on Terminal-Bench 2.1, 83.6% on MCP Atlas, 1656 Elo on GDPval-AA, and 84.2% on CharXiv Reasoning for multimodal understanding.That last number matters because it means the same model handles code and visual output, which is how agents can now render Compose previews and understand what they’re looking at. It runs about four times faster than other frontier models, and the pricing is $1.50 per million input tokens and $9.00 per million output tokens, with cached input at $0.15 per million.

Gemini Spark is Google’s 24/7 personal AI agent. It runs on Google Cloud infrastructure so it doesn’t need your device to stay on. It connects to Gmail, Docs, Calendar, and can interact with the web through Chrome and third-party services.You feed it context (wedding plans, a research project, whatever) and it keeps working. It’s exclusive to Google AI Pro and Ultra subscribers, rolling out in beta in the U.S. first.

Antigravity 2.0 graduated from an AI coding tool into a full agent orchestration platform. Standalone desktop app. No IDE. CLI. SDK. Multiple agents running in parallel. I’ll break this down in detail below.

Android CLI 1.0 reached stable. This is the one I keep coming back to. More on this in a moment.

Android XR smart glasses made with Samsung. Gemini Omni for video generation. Native Android app building directly in Google AI Studio. A new intelligent Search box that generates custom UIs in real time. There’s a lot.

I genuinely don’t know how to feel about all of it. The technology is real. The demos are real. A personal agent that monitors your inbox and handles calendar gymnastics while you sleep? That’s genuinely useful. An AI that can port an iOS app to native Android in hours instead of weeks? That solves a real problem for small teams. But there’s something unsettling about agents churning through your email at 3am while nobody’s watching, and I don’t think that tension goes away just because the demos are impressive.

For developers specifically, the message was unmistakable: Google doesn’t care which AI agent you use anymore. They shipped Android CLI 1.0, which gives Claude Code, OpenAI Codex, and Antigravity programmatic access to Android Studio’s toolchain from the command line.That’s a remarkable concession from a company that spent years building walled gardens. They looked at how developers actually work (switching between Claude Code, Codex, and Gemini depending on the task) and decided to make Android tooling available to all of them instead of forcing everyone into their own stack.

Pichai said the industry is in a period of “hyper progress” but also “where people want to see real value in the products they use on a day-to-day basis.” That’s the right thing to say. Whether the products deliver is a separate question.

Antigravity 2.0: What actually changed under the hood

Antigravity 2.0 is not an IDE update. Google ripped the agent manager out of the original Antigravity IDE and built it into a standalone desktop application available on macOS, Linux, and Windows.The IDE itself still exists as a separate product, but 2.0 is a completely different thing. It’s a desktop app where the primary interaction model is conversation with an agent, not a code editor with AI features bolted on.

The interface is built around conversations and artifacts. You describe what you want. The agent works on it. You review the artifacts it produces (code, documents, screenshots, browser operation recordings) and provide feedback directly on those artifacts to guide toward your desired outcome.

This is a philosophical shift. In the old model, you wrote code and the AI helped. In the new model, the AI does the work and you supervise. Whether that feels like progress or like being demoted to code reviewer depends on your disposition. I’m still figuring out where I land.

Dynamic subagents

The most technically interesting feature in 2.0 is dynamic subagents. The main agent can define and invoke child agents during execution to handle focused subtasks.These subagents are spawned dynamically, run in parallel, and get destroyed when their work is done.

Why this matters: every LLM has a finite context window. If you ask an agent to build a complex system, the intermediate steps (reading files, making edits, running tests) fill up that window fast. The agent starts forgetting what it was doing, or starts making decisions based on stale context. Dynamic subagents solve this by isolating subtasks into separate context windows that don’t pollute the main agent’s working memory.

Google’s launch demo used 93 parallel subagents to build a functional operating system from scratch. The primary agent acted like a CTO: it understood the system architecture, broke the goal into domains, and spawned specialized subagents (one for the kernel, one for memory management, one for the filesystem, one for video drivers, one for keyboard drivers). Those subagents worked independently and returned their results to the main agent for integration.

The numbers: 15,314 model requests, 339 million input tokens (with cache reads, output, and thinking tokens, that goes to over 2.6 billion), 12 hours of wall-clock time, and under $1,000 in API costs using Gemini 3.5 Flash.The OS could run FreeDoom, which is a real open-source Doom implementation.

The OS was not production quality. It had no floating-point math support, no hardware acceleration, no complex multi-threading, no sandboxing, no JIT compilation.But it was built from a single high-level prompt with zero human guidance. That’s the part that sticks with me.

Asynchronous task management and scheduled tasks

Tasks in Antigravity 2.0 can run asynchronously. You can fire off a long-running task and continue having other conversations with the agent while it works in the background.The results eventually surface in the main conversation when the task completes.

Scheduled Tasks are cron-style invocations. You define a schedule and the agent triggers automatically at the specified time.The /schedule slash command lets you write things like “every morning at 9am, summarize my inbox” directly in natural language.This moves the agent from a tool you call to something closer to a persistent automation pipeline.

JSON hooks

Hooks let you intercept and control agent behavior at key execution points. You define them in a JSON configuration file. When the agent hits a trigger event, the hook fires and can allow, deny, or ask the user before proceeding.The hooks system supports deny, allow, ask_user, and enforce rules per tool, which gives you fine-grained control over what the agent can and cannot do without reading the underlying code.

Projects, not repositories

In Antigravity 1.0, agent conversations were grouped by workspace, which meant one repository. In 2.0, they’re grouped by “project,” which can span multiple folders and enforce its own agent settings and permissions.This matters for real-world development where a single product often spans multiple repositories or codebases.

Slash commands

The four new slash commands are worth knowing:

/goal tells the agent to keep running until the specified task is completely finished, no intermediate input from you.This is for fire-and-forget work where you trust the agent enough to not need check-ins.

/grill-me flips the interaction: before starting to implement, the agent asks you clarifying questions to align on the specific details of the plan.I’ve found this genuinely useful for avoiding the “agent confidently builds the wrong thing” problem.

/schedule creates a one-time timer or recurring schedule for agent invocation.

/browser explicitly tells the agent to use browser primitives. Google found that agents were still not capable enough to determine on their own when to use the browser, so they made it an explicit command for now.

Voice input

The mic icon next to text input boxes now does live transcription instead of collecting raw audio files to pass to the model.It’s a small feature, but it changes the interaction dynamic. You can think out loud while the agent works, which feels more natural than typing everything.

The SDK

The Antigravity SDK gives you programmatic access to the same agent harness that powers Antigravity 2.0 and the Antigravity CLI. Your agent inherits a rich built-in toolset, a declarative safety-policy engine, lifecycle hooks for observing and steering every tool call, and stateful multi-turn sessions that persist across interactions.You can define custom agent behaviors and host them on your own infrastructure.

The SDK supports MCP integration for connecting to external Model Context Protocol servers, and the policy system lets you define permission rules at the tool level.

Pricing and availability

Antigravity 2.0 is free for all users. The new Google AI Ultra plan starts at $100/month with a 5x higher usage limit than the Pro plan, and there’s a limited-time $100 bonus credit for new and existing subscribers who hit their quota limit.The top-tier Ultra plan was cut from $250 to $200/month (20x the Pro limit).

Gemini Spark is gated behind Pro and Ultra subscriptions. Android CLI 1.0 requires Android Studio Quail 2 Canary 1 or later for the android studio commands.

Android CLI 1.0: The announcement that actually changes things

Everyone’s talking about Antigravity 2.0 building an operating system. That’s a great demo. But the announcement that actually matters for working Android developers is Android CLI 1.0 reaching stable. Here’s why.

For years, if you wanted to build Android apps with an AI coding agent, you had two options. Option one: use Google’s tools (Android Studio with Gemini). Option two: struggle. Third-party agents like Claude Code couldn’t access Android Studio’s static analysis engine, its refactoring tools, its Compose preview renderer, or its dependency management. They were flying blind. They generated code without being able to verify it against the actual Android toolchain.

Android CLI 1.0 fixes that. Through a new android studio command group, any AI agent can now perform semantic symbol resolution, analyze files for warnings, render Jetpack Compose previews, and execute end-to-end UI tests, all from the terminal without opening the IDE.

Google is positioning this explicitly as supporting whatever agent you want to use: Gemini in Android Studio, Antigravity, Claude Code, or Codex.That is a real shift in strategy.

The android studio command

The April preview release of Android CLI gave agents a clean binary for scaffolding projects, managing emulators, and running apps. Useful, but nothing that forced a workflow change. The stable 1.0 release adds the ‘android studio’ command group, which is the part that actually matters.

Here’s what agents can now do from the terminal:

android studio find-declaration performs semantic symbol resolution. Your agent can ask “where is HotelDetailScreen defined?” and get the actual answer, not a text-match guess.

android studio find-usages finds every reference to a symbol across the entire project.

android studio render-compose-preview renders a Jetpack Compose preview to an image file from the terminal.This is the one that changes things. Your agent can literally see what your Composable looks like before suggesting changes, instead of pattern-matching on text and hoping the layout looks right.

android studio analyze-file runs Android Studio’s full static analysis engine on a file: lint warnings, R8 hints, deprecation alerts.

android studio version-lookup queries live repository data and returns the actual latest stable version of a dependency.This is quietly excellent because agents guessing at Compose or AGP versions and hallucinating dependency coordinates is a common failure mode. version-lookup eliminates that entire class of error.

These commands require Android Studio Quail 2 Canary 1 or later to be running alongside the CLI. If you’re on the stable Android Studio channel, the android studio commands do nothing.

Why the token numbers are real

Google’s internal experiments showed that Android CLI reduced LLM token usage by over 70% for project setup tasks, with agents completing work up to 3x faster.These numbers are not vague marketing.

The reduction comes from how Android CLI structures its output. Traditional agent workflows dump full text logs, entire file contents, and verbose SDK manager output at the model. Android CLI returns structured JSON and supports a --diff mode that sends only the elements that changed since the last snapshot.

Here’s a concrete example: android layout --diff. Instead of re-processing the entire UI hierarchy on every iteration, the agent sees only what changed. On a complex screen with dozens of composables, that’s the difference between 8,000 tokens and 400.Multiplied across a development session, the savings compound quickly and translate directly into lower API costs.

Agent Skills

Android CLI ships with a skills system: modular markdown instruction sets that auto-trigger when a prompt matches their metadata. No manual context-stuffing, no copying documentation into system prompts.

The stable release adds five skills not present in the preview: build for display glasses (Jetpack Compose Glimmer for Android XR glasses), implement AppFunctions (expose app workflows to Android’s AI agent layer), analyze Perfetto traces (root cause latency, memory, and jank issues), use Perfetto SQL (natural language to Perfetto SQL query translation), and set up testing strategy (unit, UI, and screenshot test scaffolding).

There are 16 skills total available. The full library lives at github.com/android/skills and follows the agentskills.io open standard, which means community-contributed skills work in the same pipeline. Third-party repositories for Compose performance optimizations and Jetpack-specific workflows already exist.

Adding a skill for Claude Code is one command:

android skills add --agent=claude-code --skill=migrate-xml-views-to-jetpack-compose

The skills cover XML-to-Compose migration, AGP 9 upgrade, Navigation 3 setup, edge-to-edge display modernization, and R8 configuration auditing, among others.

Journeys: agent-driven UI testing

Stable 1.0 also ships Journeys support, which lets agents execute natural language user experience tests.Instead of writing Espresso test code, you describe a user flow in plain English and the agent navigates through it, reporting what it finds.

Installation

Android CLI is free. Installation varies by platform:

# macOS
brew install android-cli

# Linux and Windows
# Follow the setup guide at:
# https://developer.android.com/tools/agents/android-cli

If you already have it installed from the preview, update with android update.

Once installed, creating a new project and verifying it works:

android --version
android create project --name MyApp --package com.example.myapp

For Antigravity 2.0 users, the Android bundle (including Android CLI and skills) can be installed during onboarding or later from Settings > Customizations > Build With Google Plugins.

Getting started: what to try first

If you want to cut through the noise and try something that’ll actually make a difference this week, here’s my recommendation.

Option A: Install Android CLI and pair it with your existing agent.

This is the low-friction path. If you already use Claude Code or Codex, add Android CLI and watch your agent suddenly get smarter about Android development. The difference is immediate. Agents stop hallucinating Compose APIs. They stop guessing at dependency versions. They can actually check their own work.

After installing, try asking your agent to scaffold a new project, render a Compose preview, and check for warnings, all in one flow. That workflow used to require switching between terminal, IDE, and manual inspection. Now it’s programmatic.

Option B: Download Antigravity 2.0 and try the /grill-me workflow.

Go to antigravity.google/download and grab the installer for your platform. Start a new project. Don’t try to build an operating system. Ask it something concrete: “create a Jetpack Compose screen with a list of items, pull-to-refresh, and a detail view using Navigation 3.” Then use /grill-me to force the agent to ask clarifying questions before it starts coding. Review the artifacts it produces. Get a feel for the interaction model.

The voice transcription button (the mic icon) is surprisingly useful. Talking through what you want while watching the agent work feels more natural than typing prompts, and you catch things you’d miss in text.

8Option C: Both.*

Install Android CLI for your agent of choice. Download Antigravity 2.0. Spend an afternoon with both. They address different needs: Android CLI is infrastructure that makes any agent better at Android, while Antigravity 2.0 is an opinionated platform that wants to be your primary interface to agents. You might end up using both.

What to expect

Antigravity 2.0 feels different from traditional coding. You’re not writing code line by line with AI autocomplete. You’re describing goals and reviewing what comes back. It takes adjustment. I found myself over-specifying things at first (old habits), then gradually learning how much detail the agent actually needs. The /grill-me command helps with that by forcing the agent to ask questions instead of charging ahead with assumptions.

Android CLI is more straightforward. It’s a bridge. If you already use Claude Code or Codex for non-Android work, adding Android CLI means those same agents can suddenly speak Android Studio’s language. The productivity gains come from fewer context switches and fewer hallucinated APIs. There’s no learning curve; the agent just gets better at its job.

What I actually think

Here’s what gets me about this I/O.

The demos are spectacular. An operating system built from a single prompt. Agents that never sleep. Video generated from any input. These are genuinely impressive technical achievements, and I’m not going to pretend otherwise.

But spectacular demos have a short half-life. What lasts is whether the tools actually make developers’ lives easier three months from now, when the keynote glow has faded and you’re debugging a Compose layout at 11pm and wondering why your agent suggested a deprecated API from three versions ago.

Android CLI 1.0 will make developers’ lives easier. I’m confident about that one. It’s boring infrastructure, but boring infrastructure is what lets the exciting stuff actually ship. The 70% token reduction isn’t marketing. The --diff mode on layout inspection isn’t marketing. The version-lookup command that eliminates dependency hallucination isn’t marketing. These are concrete improvements to the development workflow that compound across hours and days of work.

Antigravity 2.0 is harder to evaluate. The subagent architecture is smart. The 93-subagent OS demo proves the orchestration layer works at scale. The scheduled tasks and JSON hooks are genuinely useful primitives. But the interaction model (describe a goal, wait for artifacts, review, repeat) is still unproven for day-to-day development work. Some developers will love it. Some will find it slower than just writing the code themselves. The truth is probably somewhere boring in the middle.

The agents? I’m still figuring out where I land. The technology is real. The demos are real. But there’s a gap between “look what it can do” and “look what it does reliably in production,” and that gap is where most developer skepticism lives. Google seems to know this. The Android CLI release, the openness to third-party agents, the emphasis on guardrails (JSON hooks, per-project permissions, /grill-me for alignment) are all signs they understand that adoption depends on trust, not just capability.

What I keep coming back to is the openness. Google spent years building walled gardens around Android development. Android Studio was the center of gravity and everything else orbited it. Android CLI 1.0 is the opposite: it makes Android Studio’s capabilities available to whatever agent you prefer, from Claude Code to Codex to your own custom setup. That’s not a concession born of weakness. It’s a strategic bet that if you make Android the easiest platform to develop for with any AI tool, more developers will build for Android regardless of which agent they use. It’s smart, and it’s more developer-friendly than anything Google has done with Android tooling in years.

Go install Android CLI and see if your agent of choice gets smarter about Android. That’s a bet I’m comfortable making today. Antigravity 2.0 is worth trying too, especially if you’re curious about what agent-first development actually feels like in practice.

We’ll see how it plays out. The tools are free. The documentation is live. The rest is up to us.

Note: The above cover image generated by AI

Mentoring juniors has taught me as much as it’s taught them. Here’s what actually helps—and what doesn’t.

1. Explain the “Why,” Not Just the “How”

“We do it this way because…” sticks better than “just do it like this.” When you share context (e.g. “we use this pattern so we can test without the API”), they start to generalize and make better decisions on their own. One sentence of why can save a lot of repeated corrections.

2. Code Review as Teaching

PR feedback is a chance to teach. Instead of only “change this,” add “because…” or “an alternative is X, which is better when Y.” Point them to a doc or example when you have one. Over time, they’ll anticipate the same points and need fewer comments.

3. Pair When It’s Sticky

When someone’s stuck for a while, pair for 30–60 minutes. Share screen, talk through the problem, and debug together. You unblock them and see where they get confused—useful for docs or future onboarding. Don’t take over; guide so they drive.

4. Give Them Ownership of a Small Area

Assign a small feature, module, or bug area they own. They get to design, implement, and maintain it. They’ll ask questions and make mistakes—that’s how they learn. You stay available for review and unblocking, but they lead.

5. Normalize “I Don’t Know”

Make it clear that nobody knows everything. Saying “I’m not sure, let’s look it up” or “good question, I’ll find out” models that learning is ongoing. That reduces the pressure to pretend they know and encourages asking questions.

6. Set Clear Expectations

Spell out what “good” looks like: “We expect PRs to have tests for new logic,” “we comment complex bits,” “we update the README when we change setup.” When expectations are clear, they can self-check and improve without guessing.

7. Don’t Fix Everything Yourself

If you always jump in and fix their code, they don’t learn. Let them fix their bugs (with hints if needed). Step in when it’s blocking the team or when they’re truly stuck, but default to “what have you tried? what do you think we should do next?”

8. Check In Regularly

Short, regular check-ins (“how’s the task? any blockers?”) beat long, rare meetings. They get help when they need it, and you notice when they’re stuck or when scope is unclear.

Mentoring well takes time, but it pays off: the team gets stronger, and you get a clearer picture of how your systems and docs work for someone new. The goal isn’t to create clones of you—it’s to help them think and ship with confidence.

Saad Mehmood — Portfolio