When Catalogs Are Embedded in Storage

yazar
Standalone catalogs versus embedded catalogs showing how the architecture simplifies

This is Part 8 of a 15-part Apache Iceberg Masterclass. Part 7 covered the traditional catalog landscape. This article examines a newer approach: embedding the catalog directly inside the storage layer.

Traditional Iceberg architectures have three components: the query engine, a standalone catalog, and object storage. Embedded catalogs collapse the catalog into the storage layer itself, reducing the number of services to manage while providing built-in table maintenance.

Table of Contents

  1. What Are Table Formats and Why Were They Needed?
  2. The Metadata Structure of Current Table Formats
  3. Performance and Apache Iceberg’s Metadata
  4. Technical Deep Dive on Partition Evolution
  5. Technical Deep Dive on Hidden Partitioning
  6. Writing to an Apache Iceberg Table
  7. What Are Lakehouse Catalogs?
  8. Embedded Catalogs: S3 Tables and MinIO AI Stor
  9. How Iceberg Table Storage Degrades Over Time
  10. Maintaining Apache Iceberg Tables
  11. Apache Iceberg Metadata Tables
  12. Using Iceberg with Python and MPP Engines
  13. Streaming Data into Apache Iceberg Tables
  14. Hands-On with Iceberg Using Dremio Cloud
  15. Migrating to Apache Iceberg

The Embedded Catalog Model

Standalone catalogs versus embedded catalogs showing how the architecture simplifies

In a traditional setup, a separate catalog service (Polaris, Glue, Nessie) runs alongside object storage. The engine talks to the catalog to get metadata pointers, then reads data from storage. Two services, two sets of credentials, two operational concerns.

In an embedded model, the storage service itself manages Iceberg metadata. When you create a table, the storage system creates the metadata files internally and handles atomic commits, compaction, and snapshot management. The engine interacts with a single endpoint that serves both catalog operations and data access.

AWS S3 Tables

S3 Tables architecture showing the built-in Iceberg catalog with automatic compaction

AWS launched S3 Tables in late 2024 as a new S3 bucket type designed specifically for Iceberg tables. When you create an S3 table bucket, AWS manages the Iceberg catalog internally.

How it works: You create tables through the S3 Tables API or through engines like Athena and EMR. S3 Tables stores the Iceberg metadata alongside the data in the same bucket, handling the catalog pointer, manifest management, and atomic commits behind the scenes.

Built-in maintenance: S3 Tables runs automatic compaction in the background, merging small files into optimally-sized ones without any user configuration. It also handles snapshot expiry and orphan file cleanup. This eliminates one of the biggest operational burdens of Iceberg (covered in Part 10).

Access via REST API: S3 Tables exposes tables through a REST-catalog-compatible interface. Dremio, Spark, Trino, and other engines that support the Iceberg REST catalog can connect to S3 Tables directly.

Built-in lifecycle management: Beyond compaction, S3 Tables handles the entire table maintenance lifecycle. Snapshot expiry happens automatically based on configurable retention policies. Orphan files are cleaned up without user intervention. For teams that do not want to manage maintenance schedules, this is a significant operational advantage.

Limitations: S3 Tables is AWS-only. Tables are stored exclusively in S3 and cannot be moved to other cloud providers without migration. Cross-engine governance is limited to what AWS IAM provides. If you need fine-grained access control beyond IAM policies (column-level masking, row-level filters), you need a standalone catalog layer on top.

Cost model: S3 Tables uses a different pricing model than standard S3. Storage and request costs are similar, but the built-in maintenance operations (compaction, expiry) are included in the service price. Compare this to running Spark compaction jobs on EMR, which adds compute costs on top of storage.

Table bucket vs. general-purpose bucket: S3 Tables uses a new “table bucket” type, separate from standard S3 buckets. You cannot mix table data with other objects in a table bucket, and standard S3 operations (ls, cp, rm) do not work on table bucket contents. All interaction goes through the S3 Tables API or through Iceberg-compatible engines.

MinIO AI Stor

MinIO AI Stor takes a similar approach for on-premises and private cloud deployments. MinIO, the leading S3-compatible object storage system, embeds Iceberg catalog functionality directly into the storage layer.

How it works: MinIO manages Iceberg table metadata as part of its storage operations. When data is written, MinIO handles the catalog updates, file tracking, and maintenance internally.

Key differentiator: MinIO is designed for on-premises deployments and private clouds, making it the embedded catalog option for organizations that cannot use public cloud services. It also integrates vector storage capabilities for AI workloads alongside Iceberg tables.

S3 compatibility: Because MinIO implements the S3 API, engines that work with S3 (Spark, Trino, Dremio) can interact with MinIO-managed Iceberg tables with minimal configuration changes. This makes it a drop-in replacement for S3 in on-premises environments.

GPU-accelerated analytics: MinIO AI Stor integrates with GPU-aware processing frameworks, enabling direct analytics on Iceberg data without moving it to a separate compute layer. This is relevant for organizations running AI/ML workloads alongside traditional analytics.

When Embedded Catalogs Make Sense

Decision tree for choosing between embedded and standalone catalogs

Scenario Recommendation
AWS-only, want minimal ops S3 Tables
On-premises, private cloud MinIO AI Stor
Multi-cloud portability needed Standalone catalog (Dremio Open Catalog)
Cross-engine governance needed Standalone catalog (Polaris)
Multiple storage systems Standalone catalog
Single storage, simple setup Embedded catalog

Embedded catalogs are the right choice when you have a single storage system and want to minimize operational complexity. They trade flexibility for simplicity.

Standalone catalogs remain the better choice when you need multi-cloud support, cross-engine governance, or the ability to query data across multiple storage systems through federation.

The Hybrid Approach

Many organizations use both. An embedded catalog handles the storage-managed tables (S3 Tables for their AWS data), while a standalone catalog like Dremio Open Catalog provides a unified view across all data sources. Dremio can connect to S3 Tables, AWS Glue tables, and standalone catalog tables simultaneously, presenting them all through a single semantic layer.

This hybrid approach lets you pick the simplest catalog for each use case while maintaining a unified analytics experience.

Operational Planning for Embedded Catalogs

When adopting an embedded catalog, plan for these considerations:

Vendor dependency: An embedded catalog ties your tables to the storage vendor’s lifecycle. If the vendor changes pricing, deprecates features, or discontinues the product, migrating away requires converting all tables to a different catalog. With a standalone catalog, switching storage providers only requires changing the storage configuration.

Monitoring limitations: Embedded catalogs provide limited visibility into their internal maintenance operations. You cannot inspect the compaction schedule, tune the target file size, or monitor orphan cleanup progress as precisely as you can with manual maintenance via Spark procedures.

Cross-region access: Embedded catalogs are scoped to a storage region. If your analytics workloads run in a different region than your storage, the embedded catalog adds cross-region latency. A standalone catalog can be deployed in the same region as your compute for lower latency.

Integration testing: Before committing to an embedded catalog for production, test your full query stack (dashboards, notebooks, scheduled pipelines) against the embedded catalog endpoint. Verify that your engines handle the catalog’s REST API implementation correctly, as there can be subtle differences between implementations.

Part 9 covers how table storage degrades over time and why maintenance matters regardless of which catalog you use.

Books to Go Deeper

Free Resources

SafeMind AI: Instant Health & Safety Intelligence

yazar

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

The Problem It Solves: During health or safety emergencies, people often face panic, confusion, and language barriers, leading to delayed decision-making. Access to immediate, clear medical guidance or rapid SOS capabilities is critical but often scattered across different apps or blocked by slow response times.

The Experience It Creates: SafeMind AI serves as an all-in-one digital guardian. It provides users with a calm, instantly accessible dashboard that centralizes critical safety tools. The experience is designed to be extremely fast, secure, and easy to use under stress—giving users peace of mind that intelligent help is just a click away.

Core Functions (In Short)
1.AI Health Assistant: A chat interface powered by Google Gemma 4 that instantly analyzes symptoms and provides extremely brief, actionable first-aid bullet points and urgent care advice.
2.Emergency SOS System: A one-tap rapid-response button that visually activates emergency protocols and simulates notifying authorities and emergency contacts.
3.Live Location Tracker: A real-time geolocation map that pinpoints the user’s coordinates and maps out nearby medical and emergency facilities.
4.Community Incident Reports: A logged feed where users can report, view, and track local safety hazards (like road blockages or fires) to maintain situational awareness.
5.Real-time Translation: A communication tool designed to break down language barriers during emergencies, ensuring users can get help no matter where they are.

Demo

https://youtu.be/BHoQZhcPzXw?si=Lqtj0_FmDrfzZ3ZG

Code

https://github.com/V-2007-P/personal-health-assistant.git

How I Used Gemma 4

Gemma 4 acts as the platform’s core medical reasoning engine. When users input emergency symptoms, Gemma 4 instantly analyzes the text and outputs highly-formatted, concise first-aid bullet points and urgent care advice—saving critical seconds when panic sets in.

Model We Chose & Why We utilize a dual-model approach:

1.31B Dense (Cloud/API): Used as our primary model because complex symptom analysis requires deep reasoning and high accuracy to prevent medical hallucinations and strictly adhere to our brief formatting rules.

2.E4B (Local Fallback): Integrated for offline execution. Emergencies often happen without internet access, and the highly-efficient E4B model allows users to run the AI entirely on their local device without sacrificing speed.

What Is PKCE, How It Works & Flow Examples

yazar
Fig: Authorization Code Flow

This blog was originally published on Descope.

You’d think the most secure OAuth flow wouldn’t need a patch, but the standard Authorization Code flow has a blind spot. It can’t guarantee that the app redeeming an authorization code is the same one that requested it. That gap opens the door to interception and Cross-Site Request Forgery (CSRF) attacks. Proof Key for Code Exchange (PKCE) closes it.

In this guide, we’ll explore what PKCE is and how it stops these attacks. We’ll break down the standard Authorization Code flow, pinpoint where PKCE adds value, and examine why organizations are embracing it, even before it’s officially mandatory in the latest OAuth standard.

Main points

  • Authorization Code flow has a verification gap. It can’t confirm the app exchanging the code is the one that requested it.
  • PKCE binds request and token exchange. A dynamic code challenge verifies the legitimacy of the client.
  • Public clients need it, all clients benefit. PKCE protects apps with or without client secrets.
  • PKCE is mandatory in OAuth 2.1. It’s no longer optional—it’s the new standard.

What is PKCE?

PKCE, pronounced “pixie,” is a security extension for OAuth 2.0’s Authorization Code flow. While it’s designed for scenarios where the client secret cannot be securely stored, all applications can benefit from PKCE. In fact, while it’s already recommended in the best practices for OAuth, PKCE is a requirement for all clients using the in-development OAuth 2.1 specification.

As an enhancement for standard OAuth, PKCE can benefit all types of applications for two big reasons:

  • CSRF attacks: When a malicious site or app initiates an authorization request without the user’s knowledge.
  • Authorization code interception/injection: When an attacker intercepts an authorization code and exchanges it for access tokens before the legitimate application does.

The standard Authorization Code flow and where PKCE steps in

OAuth provides several different “grant types”—standardized methods for obtaining access tokens. Grant types are associated with different scenarios, with each one offering a different balance of security and convenience. One of these is the Authorization Code grant type, widely considered to be the most versatile and secure of the bunch.

Authorization Code flow

The “vanilla” Authorization Code flow is meant for applications that can maintain a server-side component to securely store credentials. Here’s how it works:

Fig: Authorization Code Flow

  1. User initiates login: The user chooses to grant your application permissions via OAuth, such as by choosing “Log in with Service (e.g., Google)” in your app.
  2. Authorization Code request: The client requests an Authorization Code from the authorization server, including information about what the app is and what permissions it’s requesting.
  3. Prompt for consent: The authorization server asks the user to authenticate and provide consent for the app to access their resources.
  4. User authentication: The user logs in to the authorization server and approves the requested permissions.
  5. Authorization code return: The authorization server redirects the user back to your application with a temporary Authorization Code
  6. Token exchange: Your application sends this code to the authorization server along with your app credentials.
  7. Access token issued: The authorization server validates everything and returns ID and access tokens
  8. Resource access: Your application uses the access token to request protected resources

In this flow, the application never sees the user’s credentials. Instead, the user authenticates directly with the authorization server, which then provides the app with a temporary authorization code.

The security gap

The standard Authorization Code flow has a fundamental flaw: there’s no way to verify that the client exchanging an authorization code for tokens is the same client that initiated the request. This raises several concerns:

  • An attacker who intercepts the authorization code can use it to obtain access tokens because there’s no verification that ties the code to the original requesting application.
  • Even if the application uses a client secret (essentially a password shared between the app and authorization server), this only proves the client’s identity, not that this specific client originally requested this specific authorization code.
  • Since there’s nothing binding the initial request to the token exchange, the flow is also vulnerable to CSRF attacks, in which a user could be tricked into initiating an unintended authorization flow.

This security gap affects all types of applications, though it’s especially problematic for clients that can’t securely store client secrets. This is where PKCE comes in, binding the initial authorization request and the token exchange.

PKCE flow

Without PKCE, OAuth authorization code flows don’t have a way to verify which specific client sent this specific request. To understand how PKCE eliminates this vulnerability, we merely need to look at its name: Proof Key for Code Exchange, meaning you need proof you originated the authorization request to exchange the code for tokens.

To achieve this, PKCE has the requesting application create a new type of secret, a “code verifier.” This is used to create a “code challenge,” which the authorization server uses to confirm which app sent the request. Here’s how it works:

Fig: Authorization Code Flow With PKCE

  1. User initiates login: Just like in a standard Authorization Code flow, the user initiates the process by selecting the prompt associated with granting your application permissions via OAuth, like “Log in with Service (e.g., Google).”
  2. Code verifier creation: Before starting the flow, your application generates a random secret. This is not the same as a client secret—it’s a special component called a “code verifier.” Client secrets can still be used alongside it. Your application also creates a “code challenge” by transforming the verifier, usually by hashing it (a one-way process that can’t be reversed or decoded).
  3. Authorization code request: The application requests an authorization code from the server and includes the code challenge (along with the hashing method, like SHA-256).
  4. Prompt for user consent: The authorization server prompts the user to authenticate and provide consent for the requested permissions (same as standard flow).
  5. User authentication: The user logs in and approves the permissions (same as standard flow).
  6. Authorization code return: The authorization server redirects back to your application with the code.
  7. Token exchange (with verification): Your application sends the code to the authorization server along with the original code verifier.
  8. Server verification: The authorization server compares the previously shared code challenge to the recently shared original code verifier before issuing any tokens. For example, if the method used was hashing with SHA-256, the server will also hash the code verifier and ensure it matches the code challenge; the two strings should be the same.
  9. Access token issued: If the code verifier matches up with the code challenge, the authorization server returns ID and access tokens.
  10. Resource access: Your application can now use these tokens to request the necessary resources (as in the standard flow).

Using PKCE in your endpoints enables the authorization server to verify that the client requesting tokens is the same one that made the request. Even if an attacker intercepts the authorization code, they can’t exchange it for tokens without the original code verifier. Only the legitimate application has this in its untransformed (e.g., unhashed) state.

PKCE in public vs. confidential clients

In OAuth terminology, clients are either “public” or “confidential” based on their ability to securely store credentials. Public clients are apps that cannot safely store a client secret because their code is fully exposed to the user or can be extracted easily. Public apps include:

  • Single-page apps (SPAs) running entirely in the browser
  • Native mobile apps
  • Certain types of desktop apps, like those that don’t use TPMs (Trusted Platform Modules) to securely store credentials

Public clients need PKCE because they can’t rely on client secrets for security. Confidential clients, on the other hand, can securely store credentials because they either run in controlled server environments or the code and secrets are otherwise inaccessible to end users. So, why would you want to use PKCE for a confidential client?

  • Protection against authorization code injection attacks and CSRF: As previously mentioned, CSRF and authorization code injection attacks are potential threats to all types of applications. OAuth 2.0 Best Practices recommend PKCE for confidential clients because it “provides strong protection against misuse and injection of authorization codes” and “prevents CSRF even in the presence of strong attackers.”
  • Adjunctive security that complements (not replaces) client secrets: The OAuth PKCE spec makes no bones about it—PKCE is not a replacement for client secrets or authentication. While it was originally designed to protect public clients, PKCE proved useful as an add-on to existing mechanisms by preventing CSRF attacks, prompting the question: “Why not add PKCE if you can?”
  • Protection against implementation vulnerabilities: In the absence of PKCE, it’s possible for a client implementation to fail at properly verifying state parameters (an OAuth mechanism used to deter CSRF attacks). Because PKCE is verified by the authorization server, it provides protection even when client-side verification is flawed or incomplete.

PKCE benefits

PKCE Benefit Public Clients Confidential Clients
Eliminates need for client secret ✅ Required — can’t store secrets securely ❌ Still uses client secret
Prevents authorization code interception ✅ Essential to prevent token theft ✅ Adds an extra layer of protection
Mitigates CSRF attacks ✅ Protects flows in exposed environments ✅ Strengthens existing CSRF defenses
Recommended in OAuth 2.0 best practices ✅ Strongly recommended ✅ Strongly recommended
Required in OAuth 2.1 ✅ Mandatory ✅ Mandatory
Protects against poor client implementations ✅ Helps when state checks are missing or weak ✅ Useful fallback in misconfigured setups

PKCE use cases and examples

Although PKCE was originally developed to secure authorization code flows on public clients, the reasons for adoption can vary across use cases and application types. Even so, there are virtually no scenarios in which an additional layer of seamless security is unwelcome.

Ideal PKCE use cases include:

  • Native mobile applications: In the original PKCE scenario, native mobile apps are public clients that can’t securely store secrets. Without PKCE, these applications couldn’t use the authorization code flow without exposing credentials to anyone with the knowledge and tools to find them.
  • Single-page applications (SPAs): SPAs benefit from PKCE in the same way native mobile apps do: all their code runs in the browser, meaning there’s no server-side storage for client secrets. SPAs rely on PKCE to use the authorization code flow safely.
  • Desktop applications: Some desktop applications can leverage TPMs and other mechanisms to secure client secrets even if the application runs entirely on a user device, but many don’t have this design. Like other public apps, these desktop clients need PKCE to use the authorization code flow securely.
  • OAuth 2.0 best practices and 2.1 compliance: OAuth 2.0 best practices recommend PKCE be used for every client, not just public ones. OAuth 2.1, despite being in its draft stage, has already been adopted by many organizations. It makes PKCE mandatory.

PKCE and MCP adoption

The Model Context Protocol (MCP) is a standardized way for Large Language Models (LLMs) and AI agents to connect with external tools, APIs, and data sources. The authorization specification for MCP formally adopted OAuth 2.1, requiring developers leveraging the protocol to “implement OAuth 2.1 with appropriate security measures for both confidential and public clients.” That means using PKCE.

Because MCP serves as the “universal remote” that serves up AI agents to connect with external tools, OAuth was the logical choice for authorization. OAuth already has a strong, proven foundation spanning over a decade, making developing a new standard just for AI use cases a moot point. OAuth 2.1 provides the latest and most secure set of requirements, and PKCE adds a crucial layer of protection to a budding ecosystem still finding its footing.

MCP’s embrace of OAuth 2.1 (and thus PKCE) is particularly significant because of the AI protocol’s widespread acceptance and sudden rise to prominence. It’s already seen adoption by OpenAI’s Agents SDK and industry leaders like Google, with the tech giant releasing their own “complementary” Agent2Agent protocol that can work hand-in-hand with MCP. Looking to the future, as MCP becomes more prevalent, so does PKCE.

No-hassle OAuth Authorization Code flows with PKCE

PKCE may sound like a complicated identity concept at first glance, but it’s easily integrated with the right tools. Descope is a comprehensive external identity and access management solution that makes complex auth challenges drag & drop simple.

Descope can be configured as an OIDC Provider to easily add PKCE-based flows to your app. Our MCP Auth SDKs help make remote MCP servers OAuth-compliant by implementing OAuth, PKCE, dynamic client registration and more in just three lines of code.

Sign up for a Free Forever account with Descope to see how PKCE-enhanced OAuth flows can enhance your users’ auth journey. Got questions about Descope? Book time with our auth experts.

Fig: Descope MCP Auth SDKs and APIs

AI Agent Failure Modes Beyond Hallucination

yazar
Generation outruns review

AI can make mistakes, models hallucinate, models make stuff up – those are well-known complaints. Yet they are barely practical when it comes to agentic engineering. What does the knowledge that models make mistakes leave you with, except not trusting any output, or expecting every line to be double-checked, killing all the productivity?

I do use agentic tools a lot, and I am fascinated by how much they have improved over the past half year. At the same time, I am often pissed off by how badly many large tasks drift from common sense and the spirit of the task.

Lately, while reading plenty of material about AI agents, I pay more attention to what sort of failure modes people call out. Often those resonate with me heavily. It is gold when someone distills a pattern into a short characteristic of models or AI agents: the “jaggedness.” This sort of knowledge helps build your own intuition around AI agent capabilities and reasonable ways to shape your work around agents. It helps with healthy expectations without buying into the over-sold dark factories and other made-up AI capability BS claims around us.

Below is my attempt to categorize and outline the failure modes called out in a few blog posts and conference talks that align with my observations.

Failure Modes

Failure Mode Few Words Source
One-shotting Tries to eat the whole app in one bite, runs out of context, and leaves a half-built mess. Anthropic long-running agents: “try to do too much at once…to one-shot the app.”
Progress-as-completion Sees activity in the repo and mistakes partial progress for the whole job being done. Anthropic long-running agents: “see that progress had been made, and declare the job done.”
Cold-start amnesia Fresh sessions inherit neither memory nor runbook, then waste time guessing what happened and how to check it. Anthropic long-running agents: “each new session begins with no memory”; “figuring out how to run the app.”
Ugly wish-granting You state a wish too loosely and the agent grants it literally, completely, and uglier than if you had never asked. My observation: less like delegation, more like telling a genie your wish and getting the cursed version back.
Spec-deliverable confusion Treats the temporary plan or design doc as part of the actual deliverable, bundling scaffolding with the thing it was supposed to build. My observation: especially visible in plan-mode, e.g. asking to create and agent skill and it comes back with the planning artifact inside the skill.
Default-fill slop Unspecified parts of the task get filled with mediocre training-prior defaults: cargo-cult code, safe UI, generic product choices. Mario Zechner: “If you leave blanks in your spec…it fills it in with the garbage”; Anthropic app harness: “safe, predictable layouts.”
Overengineering by default Adds abstractions, duplication, backwards compatibility, and defense-in-depth because internet-shaped code taught it those moves. Mario Zechner: “agents…have learned complexity.”
Working-memory rot Important facts sit in the context but stop being reliably available as the window grows. Random Labs Slate: “the model’s ability to attend…degrades as the context length grows.”
Hidden harness control The tool mutates context, prompts, tools, reminders, observability, and extensibility in ways the user cannot inspect or steer. Mario Zechner: “my context wasn’t my context”; “zero observability…almost zero extensibility.”
Lossy compaction Compression keeps long runs alive by dropping state, sometimes exactly the state you needed. Random Labs Slate: “we can unpredictably lose important information.”
Local patching Each move looks locally reasonable while the global system gets harder to reason about. Mario Zechner: “every decision of an agent is local.”
Summary-only handoff loss Subagents isolate context, then pass back a neat summary instead of enough real state to integrate safely. Random Labs Slate: “fails to transfer information across context boundaries.”
Async reconciliation failure Parallel work creates the hard question of when results are final, which branch wins, and what actually composes. Random Labs Slate: “knowing when and how to reconcile results.”
Blind N-step execution Delegated chunks run too long without feedback; the agent discovers the wall only at the end. Random Labs Slate: “like navigating a maze blind.”
Plan drag Plans and task trees prevent early stopping until reality changes, then the structure itself resists adaptation. Random Labs Slate: “Markdown plans go stale”; “trading the flexibility…for rigidity.”
Overdecomposition Planner/implementer/reviewer stacks technically work, but add ceremony, latency, and inertia. Random Labs Slate: “It will sort of work, but you’re going to hate its guts.”
Validation interruption Diagnostics injected mid-edit confuse the model before a coherent change exists. Mario Zechner: “you finish your work and then you check the errors.”
False E2E completion Unit tests or curl pass, but the actual user path is still broken. Anthropic long-running agents: “fail recognize that the feature didn’t work end-to-end.”
Functional but wrong The result passes checks or sort of works, while still being awkward, unusable, overcomplicated, or against the spirit of the task. Long-horizon agents: “functionally OK but awkward, sloppy, or strangely overcomplicated”; “pass checks and still feel wrong.”
Self-review softness The agent grades its own mediocre work with confident praise and weak critique. Anthropic app harness: “confidently praising the work…obviously mediocre.”
Modality blind spots QA tooling misses bugs it cannot see, hear, or exercise like a real user. Anthropic app harness: “Claude can’t actually hear.”

Why This Turns Into Fatigue

Two related problems do not quite belong in the failure-mode table, but they explain why the whole thing gets so tiring so fast.

First, generation outruns review. Mario’s “slow the f.ck down” is not just a mood; it is an operational constraint. Once agents can produce code, tests, issues, and PRs faster than humans can read them, the bottleneck moves from typing to judgment. A review agent catches some issues, but it does not restore ownership. If nobody reads the code, nobody knows what is critical, and when users start screaming there is no human understanding left in the room.

Generation outruns review

Second, the same dynamic leaks outside your repo. AI issues, AI PRs, synthetic comments, generated docs, generic posts: some of them can be useful, but the channel fills with plausible text faster than people can sort it. That is the wider AI slop problem. The cognitive residue is fatigue, cynicism, AI brainrot, and eventually all-caps prompts begging the machine to stop being cute and do the actual job.

This is why “slow down” is not nostalgia or moral scolding. It is a practical rule: keep generated work inside reviewable bounds, use agents where verification is cheap, and preserve enough human understanding to say no.

Fixes And What They Break

Fix Helps with Breaks / creates
Context reset Long-task drift, context anxiety. Handoff artifact becomes critical state. Bad handoff means bad next session.
Compaction Keeps a long run going. Drops important state unpredictably.
Feature list / task list One-shotting, premature completion. Rigid plans, stale status, checkbox theater.
Strict task tree Early stopping, incomplete decomposition. Low expressivity; hard to adapt when reality changes.
Subagents Context isolation, parallel search. Thin summaries, message-passing limits, merge problems.
Separate evaluator Self-praise and weak review. Evaluator still misses things; criteria can create rubric-shaped slop.
Browser / E2E testing False completion from local checks. Tool blind spots remain; perception limits remain.
User-owned minimal harness Hidden vendor behavior, opacity, shallow extensibility. Security, workflow design, and maintenance move back to the user.

Sources

P.S.>

Mario, the creator of Pi Agent, uses the word “f.ck” too often in his talk. I find myself in a similar position with all caps and lots of F.CK in my prompts. I guess that is the AI fatigue from too many AI outputs manifesting 🙂

Fastest Way to Understand Stryker

yazar

FASTEST WAY TO UNDERSTAND STRYKER

We’ll create: 

Console App
   ↓
Class Library
   ↓
Unit Test Project
   ↓
Run Stryker

You’ll understand:

  • solution structure
  • testing flow
  • mutation testing
  • enterprise-level quality tooling

STEP 1 – Create Solution

Open terminal: 

mkdir StrykerDemo
cd StrykerDemo

dotnet new sln

STEP 2 – Create Class Library

This contains business logic. 

dotnet new classlib -n StrykerDemo.Core

Add to solution: 

dotnet sln add StrykerDemo.Core

STEP 3 – Create Test Project

Use xUnit. 

dotnet new xunit -n StrykerDemo.Tests

Add to solution: 

dotnet sln add StrykerDemo.Tests

Reference Core project: 

dotnet add StrykerDemo.Tests reference StrykerDemo.Core

STEP 4 – Create Actual Logic

Inside: 

StrykerDemo.Core

Create: 

namespace StrykerDemo.Core;

public class Calculator
{
    public int Add(int a, int b)
    {
        return a + b;
    }
}

STEP 5 – Create Unit Test

Inside test project: 

using StrykerDemo.Core;

namespace StrykerDemo.Tests;

public class CalculatorTests
{
    [Fact]
    public void Add_Should_Return_Correct_Value()
    {
        var calculator = new Calculator();

        var result = calculator.Add(2, 3);

        Assert.Equal(5, result);
    }
}

STEP 6 – Verify Tests

Run: 

dotnet test

You should see: 

Passed!

STEP 7 – Install Stryker

Install globally: 

dotnet tool install -g dotnet-stryker

Verify: 

dotnet stryker --version

STEP 8 — RUN STRYKER

Go to test project: 

cd StrykerDemo.Tests

Run: 

dotnet stryker

WHAT HAPPENS NOW

Stryker will:

  • Find your code
  • Mutate it
  • Run tests repeatedly

Example mutation:

Original: 

a + b

Mutated: 

a - b

Your test expects: 

5

Mutated result: 

-1

Test fails.

Mutation killed ✅

NOW LET’S SEE A SURVIVING MUTATION

Change test to weak test: 

Assert.True(result > 0);

instead of: 

Assert.Equal(5, result);

Now rerun Stryker.

Some mutations may survive because:

  • subtraction may still return positive
  • your assertion is too generic

THIS is where mutation testing becomes powerful.

THE BIG ENTERPRISE INSIGHT

In real enterprise systems:

  • code coverage can say 90%
  • but mutation score may say 40%

Meaning:

tests execute code but don’t truly verify behavior.

That’s why mature engineering teams use:

  • unit tests
  • integration tests
  • mutation testing
  • quality gates in CI/CD

HOW THIS LOOKS IN GITHUB

Usually: 

GitHub Actions
    ↓
dotnet test
    ↓
dotnet stryker
    ↓
Fail pipeline if mutation score low

This is where you begin seeing:

  • real engineering ownership
  • architecture visibility
  • DevOps quality flow
  • not just user story implementation

WHAT YOU SHOULD DO NEXT

After this basic example:

  1. Add more methods
  2. Add edge cases
  3. Purposely create weak tests
  4. Watch mutations survive
  5. Improve tests
  6. Re-run Stryker
    That loop teaches more than tutorials.

Solana Accounts Explained to a Web2 Developer

yazar

If you are coming from a traditional Web2 background (like building with Node.js, databases, or cloud servers), your first look at a blockchain can be confusing. You hear terms like smart contracts and state transition, but how does data get saved?

On Solana, Everything is a file.

Well, Solana calls them Accounts, but they act just like files in a computer operating system. Let’s break down how Solana stores data using terms you already know.

1. The Big Idea: Everything is a File (Account)

In Web2, you might use a Linux server. On that server, you have executable files (programs) and data files (such as .json or .txt).

Solana works the same way. Instead of a hard drive, Solana uses a giant, flat key-value store.

  • The Key: A 32-byte string called a Public Key (like So11111111111111111111111111111111111111112). This is your file name or path.
  • The Value: The account itself. This is the actual file contents and metadata.

Whether it’s a user wallet, a piece of code, a token balance, or a picture of an NFT—on Solana, it is always an account.

2. The 5 Fields Inside Every Account

Just like a file on your computer has a size, an owner, and permissions, every single Solana account contains the same five pieces of information:

Field What it means in Web2 terms
Lamports The bank balance of the file. A lamport is a tiny fraction of a SOL coin (1 SOL = 1 billion lamports).
Data The actual contents of the file. A flat array of bytes where you can save any information.
Owner The specific program (app) that is allowed to change this file’s data.
Executable A true/false switch. If true, this file is a program (code). If false, it’s just data.
Rent Epoch A background field used by the network (mostly system maintenance data).

Here is what it looks like if you inspect a real account using the Solana command line: 

{
  "lamports": 2039280,
  "data": ["SGVsbG8gV29ybGQ=", "base64"],
  "owner": "11111111111111111111111111111111",
  "executable": false,
  "rent_epoch": 18446744073709551615
}

3. Programs are Stateless

This is the biggest surprise for Web2 developers.

In Web2, your app server usually holds some data in memory, or your code handles its own database connections. On Solana, programs (smart contracts) cannot store data inside themselves. They are stateless.

Think of it like this:

  • The Program Account: This is your compiled code. Its executable flag is set to true. It sits there like a web server application, ready to run.
  • The Data Account: This is your database row. Its executable flag is false.

When a user wants to interact with an app, they pass both the Program (the code) and the Data Account (the file) to the network. The program reads the data file, updates it, and saves it back.

4. The Golden Rule: App Security

How do we stop people from changing your data account and giving themselves a million tokens? Solana has a very strict security rule built into its core:

Only the program listed as the “Owner” of an account can change its data.

If Program A tries to modify a data file owned by Program B, the Solana network rejects the change. However, anyone is allowed to send money (lamports) into any account. You can deposit money into a friend’s bank account, but only the bank’s system can update their account status.

5. Rent: Paying for Storage Space

On a cloud platform like AWS, you pay a monthly fee to keep your files stored on a hard drive. If you stop paying, AWS deletes your files.

Solana manages storage similarly, but with a twist. To keep a file on the blockchain, the account must hold a minimum amount of SOL coins. The bigger your data file, the more SOL you need to leave in it. This is called being Rent-Exempt.

For a basic wallet account with no extra data, it costs a tiny fraction of a cent (around 0.00089 SOL) to stay on the network forever. If you ever close the account and delete the file, you get all that SOL back!

Wrapping Up

To survive and build on Solana, just remember these four rules:

  1. Everything is a file called an account.
  2. Code and data live in completely separate files.
  3. Programs are stateless; they just read and write to data accounts.
  4. Only the owner program can modify a file’s data.

Once you view the blockchain as a giant, secure file system, writing applications becomes much easier to visualize!

TV Yayın Akışı Sitesi Geliştirirken Öğrendiğim Teknik Dersler

yazar

Televizyon yayın akışı sitesi geliştirmeye başladığımda bunun basit bir içerik projesi olacağını düşünmüştüm. İlk bakışta mantık oldukça düz görünüyordu:

Kanal listesi
Yayın saatleri
Program başlıkları
Kullanıcıya hızlı erişim

Fakat iş geliştirme tarafına geçince durum farklılaştı.

Özellikle mobil trafik, performans, SEO ve sürekli güncellenen veriler bir araya gelince küçük görünen bir proje zamanla ciddi optimizasyon gerektiren bir sisteme dönüştü.

Bu yazıda TV yayın akışı projesi geliştirirken karşılaştığım teknik sorunları ve uyguladığım çözümleri paylaşacağım.

  1. Sürekli Değişen Veri Yapısı Beklediğimden Zor Çıktı

Yayın akışları statik içerik değil.

Her gün:

yeni programlar ekleniyor
saatler değişiyor
özel yayınlar geliyor
içerikler güncelleniyor

Başta verileri doğrudan HTML içine yazmayı denedim. Fakat kısa sürede yönetilemez hale geldi.

İlk yaklaşım: 

<div class="program">
   <span>20:00</span>
   <span>Ana Haber</span>
</div>

<div class="program">
   <span>22:00</span>
   <span>Film Kuşağı</span>
</div>

Bu yöntem küçük ölçek için çalışıyor ancak kanal sayısı arttığında bakım zorlaşıyor.

Daha sürdürülebilir yapı JSON tabanlı veri modeli oldu:

[
{
"time":"20:00",
"title":"Ana Haber"
},
{
"time":"22:00",
"title":"Film Kuşağı"
}
]

Frontend tarafında: 

schedule.forEach(item=>{

container.innerHTML += `
<div class="program">

<span>${item.time}</span>

<span>${item.title}</span>

</div>
`

});

Arayüzü veriden ayırmak bakım sürecini ciddi şekilde kolaylaştırdı.

  1. Mobilde Yatay Kaydırma Beklediğimden Fazla Maliyet Oluşturdu

TV projelerinde genelde çok sayıda kanal logosu bulunuyor.

İlk çözüm klasik yatay kaydırma alanıydı: 

.channels{
overflow-x:auto;
white-space:nowrap;
}

Başlangıçta çalışıyordu fakat kanal sayısı büyüdükçe mobil cihazlarda bazı sorunlar ortaya çıktı:

fazla DOM yükü
kaydırma takılmaları
düşük FPS
gereksiz render işlemleri

Sonrasında görünmeyen öğeleri yüklememeye başladım.

Örnek:

const visibleItems = channels.slice(start,end);

Basit görünse de özellikle düşük donanımlı cihazlarda fark hissedildi.

  1. Lazy Loading Olmadan Sayfa Ağırlığı Hızla Artıyor

Kanal sitelerinde logo sayısı çok yüksek olabiliyor.

İlk sürüm:

<img src="kanal-logo.png">

Yüzlerce logo yüklenmeye başladığında ağ yükü hızla arttı.

Daha sonra:

<img
loading="lazy"
src="kanal-logo.png"
alt="kanal logosu">

Tarayıcının yerleşik lazy loading desteği ciddi fayda sağladı.

Ekstra JavaScript kullanmadan ilk yükleme süresi azaldı.

  1. Tekrarlanan HTML Yerine Bileşen Mantığı Daha Temiz Oldu

İlk sürümlerde her kanal kartı manuel yazılıyordu: 

<div class="channel">
<img src="kanald-logo.png">
<h3>Kanal D</h3>
</div>

Bir süre sonra yüzlerce tekrar oluştu.

Daha sonra yapı fonksiyonlaştırıldı: 

function channelCard(channel) {
  return `
    <div class="channel">
      <img src="kanald-logo.png">
      <h3>Kanal D</h3>
    </div>
  `;
}

Bu yaklaşım:

kod tekrarını azalttı
bakım sürecini kolaylaştırdı
ölçeklenebilirliği artırdı

Ayrıca Dev.to editöründe bazı template literal örneklerinin sorun çıkarabildiğini fark ettim. Bu yüzden örnek kodlarda sade yapı kullanmak daha güvenli olabiliyor.

  1. SEO Tarafında En Büyük Sorun Dinamik İçerik Oldu

Yayın akışı siteleri her gün değişiyor.

Burada temel soru şuydu:

Arama motorları sürekli güncellenen içeriği ne kadar sağlıklı algılıyor?

Bazı çözümler:

anlamlı URL yapıları
yapılandırılmış veriler
sitemap güncellemeleri
hızlı sayfa yüklenmesi
gereksiz JavaScript azaltılması

Örnek:

<script type="application/ld+json">
{
"@context":"https://schema.org",
"@type":"TVSeries",
"name":"Örnek Program"
}
</script>

Yapısal veriler küçük görünse de arama motorlarının içeriği anlamasında yardımcı olabiliyor.

  1. Fazla JavaScript Her Zaman Çözüm Değil

Bir noktada fark ettiğim önemli şeylerden biri:

Sorunların çoğu daha fazla JavaScript eklenerek çözülmüyor.

Bazen:

daha az DOM
daha az istek
daha küçük CSS
daha sade yapı

çok daha iyi sonuç verebiliyor.

Özellikle mobil ağırlıklı projelerde hafiflik önemli hale geliyor.

Sonuç

TV yayın akışı sitesi ilk bakışta basit görünüyor.

Ancak kullanıcı sayısı ve veri miktarı arttığında:

performans
veri yönetimi
SEO
mobil optimizasyon
ölçeklenebilirlik

bir anda projenin merkezine yerleşiyor.

Küçük görünen projeler bazen büyük sistemlerden daha fazla optimizasyon gerektirebiliyor.

Kullanılan Kaynaklar:

• Schema.org — BroadcastService: https://schema.org/BroadcastService
• Schema.org — TelevisionChannel: https://schema.org/TelevisionChannel
• Örnek Çalışma – TV Yayın Akışı
• Google Search Central — Video Structured Data: https://developers.google.com/search/docs/appearance/structured-data/video