Hytale Servers Will Fail Treasure Hunts Until We Fix Our Event Handling

The Problem We Were Actually Solving

At the time, we were trying to optimize our server for high latency environments. We wanted to make sure that our Treasure Hunt engine was stable and fast, even in the presence of network partitions or high packet loss. The goal was to minimize the impact of events on our server’s performance. But, as it often does, our optimization effort quickly spiraled out of control.

What We Tried First (And Why It Failed)

We initially went down the route of using a highly-optimized event bus library, one that promised to minimize the overhead of event dispatching. The idea was to use this library to decouple our event handling logic from the rest of the server, allowing us to scale our event processing independently of our main business logic. Sounds good in theory, but in practice, it was a nightmare. The library had a massive memory footprint, which, combined with our own server’s memory leaks, brought our system to its knees. The event bus library was also notoriously difficult to debug, making it almost impossible to pinpoint the root cause of our problems.

The Architecture Decision

After months of struggling with the event bus library, we were forced to take a step back and reassess our architecture. We realized that our main problem was not the event handling itself, but rather the way we were structuring our events. We were using a “publish-subscribe” model, where each event was broadcast to every subscriber, regardless of whether they needed it or not. This led to a huge amount of unnecessary event processing, which, in turn, caused our server to become overwhelmed.

To fix this, we switched to a more targeted approach, where events were only dispatched to relevant subscribers. We also introduced a caching layer to store event metadata, reducing the number of database queries and minimizing the overhead of event processing. And, most importantly, we rewrote our event handling logic to use a more efficient data structure, one that minimized memory allocations and reduced the overall latency of our event handling pipeline.

What The Numbers Said After

After implementing these changes, our system saw a significant improvement in performance. The Treasure Hunt engine, once a bottleneck that brought our server to its knees, was now able to handle thousands of concurrent requests without breaking a sweat. Our server’s memory usage dropped by 50%, and our latency improved by 90%. The load average, once a stubborn resident of the triple digits, now hovered around 1-2.

What I Would Do Differently

Looking back, I would have approached the problem differently from the start. Instead of trying to optimize our event handling system for high latency environments, I would have focused on building a more robust and scalable system from day one. I would have used a more targeted event handling model, one that minimized unnecessary event processing and reduced the overall load on our server. And, I would have avoided the event bus library altogether, opting for a more lightweight and custom-built solution. In the end, it’s a lesson learned: sometimes, the best approach is not to optimize for one specific use case, but to build a system that is inherently robust and scalable.

Redux in React: Managing Global State Like a Pro

Welcome back to Episode 12 of the “Let’s Master React Hooks Together” series

So far in this series, we’ve explored React hooks, component architecture, side effects, refs, memoization, and performance optimization. As React applications grow larger and more feature-rich, one major challenge starts appearing everywhere — managing state across multiple components efficiently.

What begins as simple prop passing between parent and child components can eventually turn into deeply nested component trees, duplicated state, inconsistent updates, and difficult-to-maintain code.

This problem becomes especially noticeable in real-world applications like:

  • E-commerce platforms
  • Social media applications
  • Admin dashboards
  • Analytics platforms
  • Project management tools

In these applications, multiple components often need access to the same shared data at the same time.

That’s exactly where Redux becomes valuable.

Redux helps React applications manage state in a centralized, predictable, and scalable way. Instead of scattering shared state across many components, Redux introduces a single source of truth that makes data flow easier to understand, debug, and maintain.

In this article, we’ll deeply understand Redux, how it works with React, why Redux Toolkit is now the standard approach, and how to structure Redux applications properly for modern frontend development.

By the end of this article, you’ll clearly understand:

  • What Redux is
  • Why Redux exists
  • Core Redux concepts
  • Redux Toolkit
  • Store setup
  • Actions and reducers
  • Async operations
  • RTK Query
  • Best practices
  • Common mistakes

What is Redux?

Redux is a predictable state management library used for JavaScript applications, especially React applications.

It provides a centralized place called a store where the entire application state is managed in a structured and organized way.

Instead of multiple components independently managing and duplicating shared data, Redux creates a single source of truth for the application.

This centralized architecture makes state changes easier to track, debug, test, and maintain as applications become larger and more complex.

Redux follows a strict one-way data flow, which makes application behavior more predictable and easier to understand during development.

Why Redux is Important

In modern frontend applications, multiple components often depend on the same piece of data.

For example:

  • Navbar → cart count
  • Product page → add to cart
  • Checkout page → cart items
  • Wishlist page → saved products
  • Profile page → user details
  • Dashboard → analytics data

Managing this shared data separately inside individual components quickly becomes difficult and error-prone.

Without centralized state management:

  • State duplication increases
  • Data synchronization becomes inconsistent
  • Prop drilling becomes excessive
  • Debugging becomes harder
  • Component communication becomes messy
  • Scaling the application becomes difficult

Redux solves these issues by introducing a centralized and predictable state container that every component can access consistently.

The Problem Redux Solves

Consider this component structure:

<App>
  <Dashboard>
    <Sidebar>
      <Profile />
    </Sidebar>
  </Dashboard>
</App>

If Profile needs user data stored inside App, the data must pass through every intermediate component.

This process is called prop drilling.

As applications grow, prop drilling creates several problems:

  • Components receive props they don’t actually use
  • Code becomes harder to maintain
  • Component trees become cluttered
  • State management becomes difficult

Redux removes this problem completely by allowing components to directly access centralized state from anywhere inside the application.

Redux Mental Model

You can think of Redux like a banking system:

  • Store → Bank database
  • Action → Request form
  • Reducer → Bank employee processing request
  • Dispatch → Submitting the request
  • Selector → Reading account information

This analogy helps visualize how data flows inside Redux applications.

Core Redux Concepts

To understand how Redux works internally, we first need to understand the five core building blocks that power the Redux architecture.

Redux mainly revolves around five concepts:

  1. Store
  2. Actions
  3. Reducers
  4. Dispatch
  5. Selectors

Let’s understand them one by one.

1. Store

The store is the central storage for the application state.

It acts as the brain of the application and contains the complete global state tree for the entire application.

Example:

{
  user: {
    name: "John"
  },

  cart: [],

  theme: "dark"
}

In this example:

  • user stores authentication or profile-related data
  • cart stores shopping cart items
  • theme stores UI preferences

The entire application state lives inside one centralized object, making shared data easier to manage consistently across the application.

2. Actions

Actions describe what happened inside the application.

They are plain JavaScript objects that tell Redux what type of state update should occur.

Example:

{
  type: "cart/addItem",

  payload: {
    id: 1,
    name: "Laptop"
  }
}

Here:

  • type identifies the action
  • "cart/addItem" describes adding an item to cart
  • payload carries extra data
  • id represents product ID
  • name represents product name

Actions help maintain predictable state updates because every state change is explicitly described.

3. Reducers

Reducers are pure functions responsible for determining how the state should change in response to specific actions.

They receive the current state and an action object, then return a brand-new updated state.

Reducers never perform API calls, side effects, or asynchronous operations directly because their behavior must remain predictable and consistent.

Example:

function cartReducer(state = [], action) {

  switch (action.type) {

    case "cart/addItem":
      return [...state, action.payload];

    default:
      return state;
  }
}

Here:

  • cartReducer is the reducer function
  • state = [] sets default cart state
  • action receives dispatched action object
  • switch(action.type) checks action type
  • "cart/addItem" handles cart item addition
  • [...state, action.payload] creates new updated array
  • default returns existing state if no match exists

Reducers are one of the most important parts of Redux because they define how state transitions happen in a predictable way.

4. Dispatch

Dispatch acts as the communication bridge between components and the Redux store.

Whenever a user performs an action like clicking a button, submitting a form, or updating data, components dispatch actions to Redux.

Redux then forwards those actions to reducers, which calculate the next state update.

Example:

dispatch({
  type: "cart/addItem",
  payload: product
});

Here:

  • dispatch() sends action to Redux store
  • type identifies the operation
  • payload contains product data
  • Reducer receives this action
  • Redux updates state automatically

5. Selectors

Selectors are used to read data from the Redux store.

They help components subscribe only to the specific pieces of state they need.

This improves performance because components re-render only when the selected state changes.

Example:

const cartItems = useSelector(
  (state) => state.cart
);

Here:

  • useSelector accesses Redux state
  • state represents complete Redux store
  • state.cart extracts cart data
  • cartItems stores extracted values

Selectors help keep components clean and focused by separating state access logic from UI rendering logic.

Redux Data Flow

Redux follows a strict one-way data flow architecture.

The flow looks like this:

  1. User interacts with UI
  2. Action gets dispatched
  3. Reducer processes action
  4. Store updates state
  5. UI re-renders automatically

This predictable structure makes applications easier to debug, scale, and maintain because every state update follows the same consistent flow.

What is Redux Toolkit?

Redux Toolkit is the official recommended way to use Redux today.

Earlier Redux implementations required developers to manually create:

  • Action types
  • Action creators
  • Reducers
  • Immutable update logic
  • Middleware configuration
  • Redux DevTools setup

This resulted in large amounts of repetitive boilerplate code.

Redux Toolkit was introduced to solve these problems and provide a modern, standardized, and simplified Redux development experience.

Benefits of Redux Toolkit:

  • Less code
  • Cleaner structure
  • Built-in best practices
  • Better readability
  • Simpler async handling
  • Automatic DevTools setup
  • Easier state updates

Redux Toolkit dramatically improves developer productivity while still following Redux principles internally.

Installing Redux Toolkit

npm install @reduxjs/toolkit react-redux
  • @reduxjs/toolkit installs Redux Toolkit
  • react-redux connects Redux with React
  • Both packages are required for React Redux applications

Creating a Redux Slice

A slice is one of the most important concepts in Redux Toolkit.

A slice groups together:

  • State
  • Reducers
  • Actions

This feature-based structure keeps Redux code cleaner and easier to scale.

Create counterSlice.js

import { createSlice } from "@reduxjs/toolkit";

const initialState = {
  value: 0
};

const counterSlice = createSlice({

  name: "counter",

  initialState,

  reducers: {

    increment: (state) => {
      state.value += 1;
    },

    decrement: (state) => {
      state.value -= 1;
    },

    incrementByAmount: (state, action) => {
      state.value += action.payload;
    }
  }
});

export const {
  increment,
  decrement,
  incrementByAmount
} = counterSlice.actions;

export default counterSlice.reducer;
  • createSlice creates Redux slice
  • initialState stores default values
  • value: 0 initializes counter
  • name: "counter" defines slice name
  • reducers contains state update functions
  • increment increases counter value
  • decrement decreases counter value
  • incrementByAmount updates value dynamically
  • action.payload receives custom value
  • counterSlice.actions auto-generates actions
  • export default counterSlice.reducer exports reducer

Slices significantly reduce Redux boilerplate and organize related logic in one place.

Why Direct Mutation Works Here

Inside Redux Toolkit:

state.value += 1;

At first glance, this looks like direct state mutation.

In traditional Redux, directly mutating state is strictly forbidden because Redux depends on immutable updates to detect state changes correctly.

However, Redux Toolkit internally uses a library called Immer.

Immer allows developers to write code that appears mutable while safely generating immutable updates behind the scenes.

So even though the syntax looks like mutation:

state.value += 1;

Redux Toolkit still preserves immutability internally.

This makes Redux code much cleaner and easier to write.

Configuring Redux Store

The Redux store holds the complete application state.

Create store.js

import { configureStore } from "@reduxjs/toolkit";

import counterReducer from "./counterSlice";

export const store = configureStore({

  reducer: {
    counter: counterReducer
  }
});
  • configureStore creates Redux store
  • counterReducer imports reducer
  • reducer object combines reducers
  • counter becomes state key
  • Entire application now accesses centralized state

Redux Toolkit’s configureStore also automatically enables useful development features like Redux DevTools and middleware support.

Connecting Redux to React

To make Redux available throughout the React application, React Redux provides a component called Provider.

Inside main.jsx

import React from "react";
import ReactDOM from "react-dom/client";

import App from "./App";

import { Provider } from "react-redux";

import { store } from "./store";

ReactDOM.createRoot(
  document.getElementById("root")
).render(

  <Provider store={store}>
    <App />
  </Provider>
);
  • Provider connects React with Redux
  • store={store} passes Redux store globally
  • <App /> gets access to Redux state
  • Every child component can now use Redux hooks

Without Provider, React components cannot access the Redux store.

Using Redux Inside Components

React Redux provides hooks that allow components to interact with Redux easily.

import React from "react";

import {
  useSelector,
  useDispatch
} from "react-redux";

import {
  increment,
  decrement
} from "./counterSlice";

function Counter() {

  const count = useSelector(
    (state) => state.counter.value
  );

  const dispatch = useDispatch();

  return (
    <div>

      <h1>{count}</h1>

      <button
        onClick={() => dispatch(increment())}
      >
        Increment
      </button>

      <button
        onClick={() => dispatch(decrement())}
      >
        Decrement
      </button>

    </div>
  );
}

export default Counter;
  • useSelector reads Redux state
  • state.counter.value accesses counter value
  • count stores extracted state
  • useDispatch returns dispatch function
  • dispatch(increment()) triggers increment action
  • dispatch(decrement()) triggers decrement action
  • UI automatically updates after state changes

This hook-based approach makes Redux integration significantly cleaner in modern React applications.

Recommended Redux Folder Structure

As Redux applications grow, organizing files properly becomes very important.

A feature-based structure improves scalability and maintainability.

src/

├── app/
│   └── store.js

├── features/
│   ├── counter/
│   │   ├── counterSlice.js
│   │   └── Counter.jsx

├── components/
  • app/ contains global store setup
  • features/ stores feature-based Redux logic
  • counter/ contains counter-related files
  • components/ stores reusable UI components

This structure keeps Redux logic modular and easier to scale in large applications.

Async Operations with createAsyncThunk

Real-world applications rarely work with static data.

Most applications fetch information from APIs such as:

  • User data
  • Product lists
  • Orders
  • Notifications
  • Dashboard analytics

Handling asynchronous operations manually in Redux can become complicated.

Redux Toolkit simplifies async logic using createAsyncThunk.

import {
  createSlice,
  createAsyncThunk
} from "@reduxjs/toolkit";

export const fetchUsers =
  createAsyncThunk(

    "users/fetchUsers",

    async () => {

      const response = await fetch(
        "https://jsonplaceholder.typicode.com/users"
      );

      return response.json();
    }
  );
  • createAsyncThunk handles async logic
  • "users/fetchUsers" defines action name
  • async() performs API request
  • fetch() calls backend API
  • response.json() converts response into JSON
  • Returned data becomes payload automatically

createAsyncThunk also automatically generates loading, success, and error action types internally.

Redux DevTools

Redux DevTools is one of Redux’s most powerful debugging features.

It helps developers:

  • Track dispatched actions
  • Inspect state updates
  • Debug application flow
  • Replay state changes
  • Analyze application behavior

Redux Toolkit automatically enables Redux DevTools during development, making debugging significantly easier.

Redux vs Context API

React Context API can also handle shared state.

However, Redux provides several advanced capabilities that become valuable in larger applications.

Redux offers:

  • Better debugging tools
  • Middleware support
  • Predictable architecture
  • Advanced async handling
  • Improved scalability
  • Better developer tooling

Context API works well for:

  • Theme switching
  • Authentication state
  • Language preferences
  • Small shared states

Redux becomes more powerful when applications grow larger and state management complexity increases.

Common Redux Mistakes

Direct State Mutation

Wrong:

state.items.push(item);

Chain-by-chain breakdown:

  • Direct array mutation occurs
  • Original state gets modified
  • Traditional Redux does not allow this
  • Mutations can break predictable state updates

Correct:

return [...state, item];
  • Creates new array
  • Keeps original state immutable
  • Redux state updates safely

Understanding immutability is one of the most important concepts in Redux architecture.

RTK Query

RTK Query is one of the most powerful features introduced in Redux Toolkit.

It eliminates much of the manual API handling logic developers traditionally wrote themselves.

RTK Query automatically manages:

  • Data fetching
  • Caching
  • Loading states
  • Error handling
  • Background refetching
  • Request deduplication

This significantly reduces boilerplate and improves application performance.

Example:

import {
  createApi,
  fetchBaseQuery
} from "@reduxjs/toolkit/query/react";

export const api = createApi({

  reducerPath: "api",

  baseQuery: fetchBaseQuery({
    baseUrl: "https://api.example.com"
  }),

  endpoints: (builder) => ({

    getUsers: builder.query({

      query: () => "/users"
    })
  })
});

export const {
  useGetUsersQuery
} = api;
  • createApi creates API service
  • reducerPath defines Redux key
  • fetchBaseQuery sets base API URL
  • endpoints defines API operations
  • builder.query() creates GET request
  • query() returns endpoint path
  • useGetUsersQuery becomes auto-generated hook

RTK Query dramatically simplifies server-state management in modern React applications.

Best Practices

Use Redux Toolkit

Redux Toolkit is now the official standard for Redux development because it reduces boilerplate, improves readability, and includes built-in best practices.

Keep State Minimal

Only store data globally if multiple components truly need access to it.

Avoid storing unnecessary UI state inside Redux.

Use Feature-Based Structure

Keep related Redux logic together inside feature folders.

This improves scalability and maintainability.

Avoid Overusing Redux

Not every piece of state belongs in Redux.

Local UI states like:

  • Modal visibility
  • Form inputs
  • Dropdown toggles
  • Tooltip visibility

are usually better managed using useState.

Final Thoughts

This was Episode 12 of the “Let’s Master React Hooks Together” series, and Redux is one of the most important concepts to understand once React applications begin scaling.

Redux may initially feel complex because it introduces several new architectural concepts. However, once you understand the predictable flow between actions, reducers, and the store, managing large-scale React applications becomes significantly more organized and maintainable.

Modern Redux with Redux Toolkit has dramatically simplified the developer experience, making Redux cleaner, easier to learn, and far more practical for real-world applications.

The goal is not just learning Redux syntax, but understanding how scalable frontend architecture works in professional applications.

Once you become comfortable with slices, reducers, actions, selectors, and async handling, building complex applications becomes significantly easier and more structured.

In the upcoming episodes, we’ll continue exploring advanced React concepts, scalable frontend architecture, performance optimization patterns, and production-level development practices used in modern applications.

See you in the next episode of “Let’s Master React Hooks Together.”

Unfreezing Your GitHub Actions: Troubleshooting Stuck Deployments and Protecting Your Git Repo Statistics

The frustration of a stuck deployment, especially when using GitHub Actions for GitHub Pages, is a common pain point for developers. It’s not just about a delayed update; it impacts delivery timelines, developer morale, and skews crucial git repo statistics related to deployment success and efficiency. This isn’t merely an inconvenience; it’s a critical bottleneck that demands immediate attention from dev team members, project managers, and CTOs alike.

When GitHub Actions Workflows Freeze: The “Failed to Cancel” Enigma

Our discussion begins with a developer, Cubic-crypto, experiencing a persistent problem: a GitHub Actions workflow attempting to deploy to GitHub Pages remained “queued” for hours, failing to push edits from a Codespace. Attempts to cancel the run resulted in a frustrating “Failed to cancel workflow” error. This scenario, as highlighted by community member Gecko51, is a classic symptom of a broader GitHub Actions incident, where a runner is never assigned, leaving the workflow in a perpetual limbo. Such incidents can severely impact your team’s productivity and throw off your meticulously tracked git repo statistics.

Checklist for GitHub Actions workflow configurationChecklist for GitHub Actions workflow configuration

Initial Checks for Deployment Issues: Rule Out the Obvious

Before assuming a platform-wide issue, it’s always wise to rule out common configuration errors. Kunalmankar852 provided an excellent checklist for typical GitHub Pages deployment problems. These are the first steps any developer or delivery manager should take:

- **Pages Source Configuration:** Ensure your repository's `Settings > Pages` is correctly set to "GitHub Actions," not a specific branch. This is a frequent oversight that can halt deployments before they even begin.

**Missing Workflow Permissions:** Your workflow YAML needs explicit permissions for deployment. Without these, GitHub Actions simply won't have the authority to push changes. Look for:
    permissions:

contents: read
pages: write
id-token: write

**Incomplete Deploy Steps:** Verify your workflow includes the essential GitHub Pages actions. These actions handle the crucial steps of configuring, uploading, and deploying your site. Missing any of these will result in an incomplete or failed deployment:
    - uses: actions/configure-pages@v4
  • uses: actions/upload-pages-artifact@v3
  • uses: actions/deploy-pages@v4

    • Workflow Not Running on Correct Branch: Confirm that your on: push or on: workflow_dispatch trigger matches your default or deployment branch. A mismatch here means your workflow won’t even activate on the intended changes.
    • Build Output Folder Wrong: Ensure you upload the correct build directory (e.g., dist, build, public, etc.) in your upload-pages-artifact step. If the artifact isn’t found, there’s nothing for GitHub Pages to deploy.

Command line interface for force-canceling GitHub ActionsCommand line interface for force-canceling GitHub Actions

When It’s Not You: Incident Recovery Strategies

If you’ve checked all the above and your workflow is still stuck with a “Failed to cancel workflow” error, it’s highly likely you’re caught in a GitHub Actions incident. This is where technical leadership needs to step in with more advanced troubleshooting, as outlined by Gecko51:

**Force-Cancel via API:** The UI cancel button is often ineffective when a runner was never assigned. Instead, use the GitHub CLI to force-cancel the run. Replace `{owner}`, `{repo}`, and `{run_id}` with your specific values. The run ID is visible in the URL of the stuck workflow.
    `gh api -X POST /repos/{owner}/{repo}/actions/runs/{run_id}/force-cancel`

- **Disable and Re-enable Actions:** If the force-cancel doesn't work, a more drastic but often effective measure is to temporarily disable and then re-enable GitHub Actions for your repository. Navigate to `Settings > Actions > General`, disable, save, then toggle back on. This clears the queued state for your repo.

**Push an Empty Commit to Retrigger:** Once the GitHub Status page shows green and your workflows are unstuck, push an empty commit to kick off a clean deploy. This ensures a fresh workflow run without introducing new code changes:
    git commit --allow-empty -m "retrigger pages deploy"

git push

- Avoid Pushing During Incidents: A critical piece of advice: do not keep pushing real commits while things are stuck. Each push queues another run that will also get caught in the incident, prolonging your recovery time.

Impact on Productivity and Delivery: Beyond the Code

Stuck deployments have a cascading effect. For dev teams, it means wasted time, context switching, and a dip in morale. For product and project managers, it translates directly into missed deadlines and delayed feature releases. CTOs and delivery managers must recognize that these incidents directly impact key performance indicators (KPIs) tracked in a software kpi dashboard, such as deployment frequency, lead time for changes, and change failure rate. Accurate git repo statistics become vital here, but they are only useful if the underlying processes are reliable. When deployments are consistently failing or getting stuck, your metrics become misleading, masking deeper issues in your CI/CD pipeline.

Software KPI dashboard showing healthy git repo statistics and deployment metricsSoftware KPI dashboard showing healthy git repo statistics and deployment metrics

Proactive Measures and Lessons Learned for Technical Leadership

To mitigate the impact of such incidents and maintain high developer productivity, technical leaders should:

- Monitor GitHub Status: Regularly check GitHub's official status page during suspected outages. Subscribe to their updates for timely notifications.

  • Implement Robust CI/CD Health Checks: Beyond just checking if a build passed, integrate monitoring that tracks the entire deployment lifecycle. Tools that offer a comprehensive software kpi dashboard can provide visibility into your pipeline's health.

  • Diversify Critical Deployments (Where Applicable): While GitHub Actions is powerful, for mission-critical applications, consider a multi-pronged deployment strategy or robust fallback mechanisms.

  • Leverage Analytics for Deeper Insights: Tools that serve as a Pluralsight Flow free alternative can help analyze your team's workflow efficiency, identify bottlenecks, and provide actionable insights from your git repo statistics, ensuring that deployment issues are not just fixed, but prevented.

  • Empower Teams with Troubleshooting Knowledge: Ensure your team members are aware of these advanced troubleshooting steps, reducing reliance on a single point of contact during an incident.

Conclusion: Resilient Deployments for Peak Productivity

While platform incidents are sometimes unavoidable, our response to them defines our resilience. By understanding common configuration pitfalls and equipping ourselves with advanced recovery strategies, we can minimize downtime and protect our critical delivery metrics. For technical leaders, ensuring smooth, reliable deployments is paramount to fostering a productive development environment and maintaining accurate git repo statistics that truly reflect your team’s efficiency and impact.

Unlocking Project Discoverability on GHES: A Key to Software Engineering Productivity

GitHub Enterprise Server (GHES) is the backbone for many organizations, providing a secure, on-premise environment for their development efforts. It offers unparalleled control and customization, making it a critical asset for teams committed to innersource principles. Yet, a common question arises: do all the intuitive features of public GitHub.com, such as the comprehensive github.com/topics discovery page, translate directly to a GHES instance? A recent community discussion illuminated this very point, clarifying expectations and, more importantly, offering actionable strategies to significantly enhance project discoverability and, by extension, overall software engineering productivity on GHES.

The github.com/topics Experience: SaaS vs. On-Premise

The initial query from donny-dont highlighted a frequent source of confusion: attempting to navigate to HOSTNAME/topics on a GHES instance often results in a 404 error. This behavior can be perplexing, especially when some documentation hints at topics support. As community experts vshivam1729 and Gecko51 succinctly clarified, the rich, browsable github.com/topics interface is a feature exclusive to GitHub’s SaaS offering. It is not bundled with GitHub Enterprise Server.

The documentation references to HOSTNAME/topics are indeed accurate, but they pertain specifically to the REST API endpoint used for managing topics on individual repositories (e.g., /api/v3/repos/{owner}/{repo}/topics). They do not imply a user-facing interface for broad topic discovery. Consequently, the 404 is an expected response, and there is no hidden admin setting to simply “turn on” an Explore-style landing page on your GHES instance. Understanding this distinction is the first step toward building effective internal discovery mechanisms.

Comparison of GitHub.comComparison of GitHub.com’s topics browse page versus a 404 error on a GHES instance, illustrating the SaaS vs. on-premise feature difference.

Strategies for Boosting Development Productivity on GHES

While the direct equivalent of github.com/topics isn’t available, GHES users have several robust strategies to improve project discoverability, foster innersource practices, and ultimately elevate software engineering productivity. These approaches empower teams to find relevant codebases, documentation, and collaborators more efficiently, directly impacting their engineering performance goals.

1. Leverage Repository-Level Topics for Granular Discovery

Individual repositories on GHES fully support topics. This is a fundamental development productivity tool that’s often underutilized. Teams can tag their projects with relevant keywords, technologies, or team names directly through the repository UI or via the API.

- **Enhanced Searchability:** Topics are fully searchable within your GHES instance. A simple search query like `topic:innersource`, `topic:backend-service`, or `topic:your-team-name` in the global search bar will quickly filter repositories. This significantly reduces the time developers spend hunting for relevant projects, accelerating project onboarding and collaboration.

- **API Integration:** For more advanced use cases, the GHES API allows programmatic access to repository topics. This means you can build custom scripts or integrations to pull lists of repositories based on specific topics, feeding into internal dashboards or reporting tools. For example, `GET /api/v3/search/repositories?q=topic:innersource` can power a custom internal portal.

By consistently applying and maintaining repository topics, organizations create a self-organizing knowledge graph that directly contributes to faster development cycles and improved code reuse.

Illustration of a search bar on GitHub Enterprise Server with Illustration of a search bar on GitHub Enterprise Server with ‘topic:innersource’ entered, highlighting the use of repository topics for project discovery.

2. Build Curated Landing Pages with GitHub Pages

For a more structured and visually appealing discovery experience, the most common and effective GHES approach is to create a dedicated GitHub Pages site. This acts as an internal directory or portal.

- **Centralized Information Hub:** Host a simple markdown-based index in a dedicated repository. This site can serve as a landing page for key projects, documentation, team information, and even links to external resources, all grouped by relevant topics or categories.

- **Low Administrative Overhead:** Maintaining such a site is typically a manual effort but requires no special administrative changes to the GHES instance itself, making it appealing to admins wary of new features or organizations. It leverages existing GHES capabilities without adding complexity.

- **Innersource Showcase:** This approach is particularly powerful for fostering innersource. It provides a clear, curated entry point for developers looking to contribute to internal projects, promoting visibility and collaboration across teams.

A well-maintained GitHub Pages site can significantly reduce friction in discovering internal assets, acting as a powerful development productivity tool that guides users through your organization’s codebase landscape.

3. Strategic Use of Dedicated “Discovery” Organizations

Admins are often cautious about creating new organizations to prevent sprawl. However, a single, strategically designed “discovery” or “innersource” organization can be a highly effective solution.

- **Minimal Footprint, Maximum Impact:** Instead of multiple new orgs, propose a single organization dedicated to housing key discovery assets. This could include the GitHub Pages repository mentioned above, as well as repositories that serve as meta-indexes or project manifests.

- **Clear Purpose and Structure:** With a well-maintained README and judicious use of pinned repositories, this org can provide a clear, concise overview of your most important internal projects. It addresses the admin's concern about sprawl by centralizing discovery efforts rather than fragmenting them.

- **Achieving Engineering Performance Goals:** This structured approach directly supports `engineering performance goals` by making it easier for new hires to onboard, for teams to find reusable components, and for leadership to gain an overview of active projects.

Image depicting a custom internal project portal built using GitHub Pages on GHES, showcasing curated content for easy discoverability.Image depicting a custom internal project portal built using GitHub Pages on GHES, showcasing curated content for easy discoverability.

Engage Your GHES Admins and GitHub Enterprise Support

Feature availability can vary significantly across GHES versions. While the core mechanisms described above are generally consistent, it’s always prudent to engage your GHES administrators. They can provide insights into your specific instance’s configuration, any custom integrations, or future upgrade plans.

Furthermore, opening a ticket with GitHub Enterprise Support is a valuable step. They can confirm the exact status of topic browsing features for your GHES version and offer official guidance on best practices for internal discoverability. This proactive engagement ensures you’re leveraging your GHES investment to its fullest potential, aligning with broader engineering performance goals for your organization.

Conclusion: Empowering Your Teams on GHES

The absence of a public github.com/topics-style page on GHES is not a limitation but an opportunity for tailored, internal solutions. By thoughtfully implementing repository topics, building curated GitHub Pages portals, and strategically organizing discovery assets, organizations can create a robust ecosystem for project visibility. These strategies are more than workarounds; they are foundational development productivity tools that empower your teams, streamline innersource adoption, and ultimately drive significant improvements in overall software engineering productivity. Embrace these approaches to transform your GHES instance into a highly discoverable, collaborative, and efficient development hub.

I Stacked 4 More Context Layers on Top of RAG. Sonnet Got 12% Better. Haiku Got 14% Worse.

Full CE vs RAG: Score deltas across Sonnet and Haiku

I read a post about “Full Context Engineering” and immediately added four more layers to my RAG pipeline. Structured output instructions. Hierarchical document layout. Role definition. Few-shot examples. The whole buffet.

The improvement on Claude Sonnet was 12%.

The improvement on Claude Haiku was minus 14%.

I had just spent two weeks building scaffolding to make my smaller model worse at its job. If you have ever wallpapered a room and stepped back to discover you covered up the light switch, you know the feeling.

This post is about what those numbers actually mean for the way you spend your context engineering effort in 2026.

What I was measuring

I was running a benchmark against my own book corpus for a previous experiment (the cheap-model post). The same scoring rubric: factual accuracy, hallucination rate, specificity, and honesty on a 0 to 15 scale.

The configurations were a ladder. Each rung adds one more thing on top of the previous one.

  1. System prompt only: the bare baseline. No retrieval, nothing.
  2. System + RAG: vector search over a curated corpus, top documents injected.
  3. Full Context Engineering: RAG + structured output instructions + hierarchical layout + role definition + few-shot examples.

What I expected: a smooth upward curve. What I got was a curve that leaned forward and then fell over.

The numbers

Full CE vs RAG: Score deltas across Sonnet and Haiku

Claude Sonnet, total score (out of 15):

Configuration Total Delta from previous
System only 8.8 —
System + RAG 10.2 +1.4 (+16%)
Full Context Engineering 11.4 +1.2 (+12%)

Claude Haiku, total score (out of 15):

Configuration Total Delta from previous
System only 3.7 —
System + RAG 11.8 +8.1 (+219%)
Full Context Engineering 10.1 -1.7 (-14%)

Two findings I did not expect.

First: RAG is doing almost all of the work. On Sonnet, RAG closed 88% of the gap between baseline and the fully tricked-out pipeline (1.4 of the total 2.6 point improvement). On Haiku, RAG over-shot the final number entirely.

Second: stacking more on top of RAG is not free. On Haiku, it actively made things worse. The hallucination score went from 1.7 to 0.5. The honesty score went from 1.3 to 0.5. The model started confidently making things up that it had previously hedged on.

Why this happens

I have a hypothesis that I think survives contact with reality.

A small model has limited working memory. RAG hands it the right facts. Once those facts are in front of it, the marginal returns from extra structure are small. But the marginal cost of extra context is not small. Every paragraph of role definition, every few-shot example, every “here is how to format the output” block competes with the retrieved documents for the model’s attention.

For Sonnet, the working memory is wide enough that the extras land in unused space. For Haiku, the extras shove the actually-useful retrieved context off to the edge of the window. The model still sees it. It just stops trusting it.

This is the same finding that recent research on long-context behavior keeps surfacing. Studies on instruction-following at high context fill report that for most frontier models in 2026, quality starts to degrade measurably at 60 to 70 percent context fill, and falls off a cliff around 90 percent. The cliff is steeper for smaller models.

The Pareto principle applies to context engineering with embarrassing accuracy. RAG is the 20 percent of effort that produces 80 percent of the result. Everything you stack on top of it is the long tail.

The 2026 reality I almost forgot to mention

When I ran the original experiment, I was on Sonnet 4 and Haiku 3 with a 200K context window. As of this writing, Sonnet 4.6 has a 1M token context window at standard pricing and prompt caching cuts the cost of repeated context by 90 percent.

This changes the math, but not in the direction you might think.

A 1M context window does not magically make stacked context cheaper to design. The model still has to pay attention to the right thing. The cliff at 60 to 70 percent fill is a percentage, not an absolute. A bigger window just means you can write more bad context before you fall off it.

Prompt caching helps if your stacked layers are static. The role definition, the few-shot examples, the structured output instructions: those parts cache cleanly. But that only saves money. It does not save quality. If your Haiku result was minus 14%, prompt caching makes minus 14% cheaper. That is not the win you wanted.

The thing nobody told me about Skills

Anthropic’s Skills feature is interesting in this light. Skills are reusable context bundles that load on demand. The right way to think about them is not “more context, all the time” but “the right context, just in time.”

That is the failure mode my Full CE experiment ran into. I was packing every layer into every request. Skills point at the alternative: keep the system prompt small, retrieve the relevant skill, and let the rest stay out of the window. It is the same lesson as RAG, applied one level up. Selective beats throwing everything in.

What I do now

If you take only one thing from this post, take this: the order of operations matters more than the number of operations.

  1. Build the retrieval first. Get RAG working with a clean corpus, decent embeddings, and a relevance threshold. This is your 80%.
  2. Run a benchmark. Real benchmark, on real questions, scored by a real rubric. Not vibes.
  3. Add one layer at a time. Structured output, then hierarchical layout, then role definition. Re-benchmark after each.
  4. If the score goes down, take that layer out. Do not assume the layer is good and your benchmark is bad. The benchmark is right more often than you think.
  5. Try the same ladder on a smaller model. The thing that helps Sonnet may hurt Haiku. Knowing which side of the line you are on saves you money.

This sounds obvious. It is not what most teams do. Most teams read a blog post about Context Engineering, add four layers in one weekend, and never measure whether the layers actually helped.

The chef and the kitchen, revisited

In an earlier post I wrote that the model is the chef and the context is the kitchen. I want to extend that.

Adding more context layers is like installing more kitchen equipment. A second oven. A pasta machine. A sous-vide. None of them make the chef worse at cooking pasta. But if the pasta machine takes up the counter space where the chef was chopping vegetables, the dinner gets worse anyway.

The chef does not need every appliance. The chef needs the right ingredients within reach.

Before you read the next breathless post about Full Context Engineering and start adding layers, run the experiment. Measure RAG alone. Measure RAG plus one thing. Find the layer that earns its keep, and leave the rest in the catalog.

The answer is almost always: do RAG well first. Everything else is decoration. Decoration that, on a small model, can flip the sign on your accuracy score and leave you wondering why.

The next time someone says “Context Engineering,” what I want to say back is: please define which 20 percent of context you mean. The other 80 has a good chance of making things worse.

The full Context Engineering system (five strategies, the RAG benchmarks behind these numbers, MCP server design, and the Agentic RAG implementation) is in Turning LLMs from Liars into Experts: Context Engineering in Practice.

OpenClaw Hit 250K Stars Faster Than React. I Spent 24 Hours Trying to Like It.

I switched my entire dev setup from Claude Code to OpenClaw on a Tuesday morning. By 11am I was googling “how to remove openclaw”. By 6pm I had written a SOUL.md file longer than the actual feature I was shipping.

This post is about that day. About what broke, what didn’t, and what 24 hours of working in the terminal agent that is now technically the most-starred open-source project in GitHub history bought me.

Yes, I am the engineer who wrote about Claude Code Skills three weeks ago and called the workflow pattern “settled for at least a year.” Then OpenClaw passed React’s all-time star count in 60 days, Peter Steinberger announced he was joining OpenAI to ship agents to everyone, and the launch tweet went past 4 million views. Settled, apparently, was a one-month forecast.

The numbers I had to verify before believing them

Let me get the facts out of the way, because half of what people quote on Twitter about OpenClaw is wrong by a factor of two.

  • OpenClaw crossed 250,000 GitHub stars on March 3, 2026, surpassing React for the all-time most-starred software repository
  • 60 days from launch to 250K. React took roughly a decade
  • 60K stars in the first 72 hours. That part is the one nobody actually believes the first time
  • Peter Steinberger announced on February 14, 2026 that he is joining OpenAI to work on agents, with OpenClaw moving to a foundation to stay open and independent
  • One mid-sized refactor session in my test run consumed 920K tokens, which on Claude 4.5 Sonnet billing came out to about USD 8.30

The Hacker News thread when it crossed React was the most upvoted submission of the week. The top comment was “this is either the best thing that happened to dev tools in five years or the most expensive way to learn what --yolo does.”

It is, somehow, both.

The setup, and the part I underestimated

Installation took less than a minute.

curl -fsSL https://get.openclaw.dev | sh
export ANTHROPIC_API_KEY=sk-ant-...
openclaw

The first surprise: OpenClaw asked me which model I wanted as default. I had four serious choices, plus Ollama for local models.

openclaw --model claude-4.5-sonnet
openclaw --model gpt-4o
openclaw --model gemini-2.5-pro
openclaw --model ollama/devstral:24b

Claude Code has a backend model. OpenClaw has a backend model dropdown. That is not a small UX difference when you are trying to land a refactor for less than ten dollars.

The second surprise: when I ran my first command, the agent asked me where the SOUL.md file was. I did not have one. It happily generated a default. The default was generic enough that I closed the session, opened my editor, and started writing my own. That is when the day quietly stopped being a benchmark and started being a personality test.

SOUL.md is the part nobody warned me about

Here is the SOUL.md I ended the day with, after rewriting it three times.

# SOUL.md
You are a senior backend engineer with strong opinions and short patience for code
that talks more than it does.

- Prefer Python over TypeScript when both fit. We're not building a frontend here.
- Never add a feature without a test. If the test would take more than 10 minutes
  to write, ask first instead of writing it.
- Performance matters but readability matters more. We're a four-person team, not
  Google.
- Do not write conversational filler. "Sure, I'll do that" is not output. Output
  is the diff.
- When in doubt, ask. Don't guess. Guessing once cost us a weekend.

The thing the docs do not tell you: SOUL.md is not a config file. It is a contract. CLAUDE.md tells Claude Code what the project is. SOUL.md tells OpenClaw who the agent is. They are two different shapes of the same trust problem, and the day I figured that out was the day OpenClaw stopped feeling worse than Claude Code and started feeling different.

I had a Claude Code session open in another window all day for a sanity check. By 4pm I noticed my CLAUDE.md was 312 lines and my SOUL.md was 14. The SOUL.md was doing more work per line.

The Gateway, and why my LGPD-anxious teammate cared

OpenClaw routes every LLM call through a local process called the Gateway.

The Gateway sits on your machine. Your prompts and code do not pass through an OpenClaw-operated cloud relay on the way to Anthropic, OpenAI, or whoever. They go straight from your laptop to the model provider you picked.

Claude Code does not have an equivalent intermediary, but it also does not need one because Anthropic is the only provider. The moment you have multi-provider support, you either need a relay (vendor lock-in risk) or a local gateway (the OpenClaw choice).

A teammate of mine who lives in Brazil and spends meaningful time worrying about LGPD compliance pinged me at lunch to ask what the network diagram looked like. He liked what I sent him. That conversation alone might be worth the day.

ClawHub vs Claude Code Skills

Claude Code Skills are markdown files plus optional resources, distributed however you distribute markdown. ClawHub is an npm-style package marketplace for OpenClaw skills.

openclaw skills search "docker"
openclaw skills install @clawhub/docker-manager
openclaw skills list

ClawHub had several thousand skills the day I tried it. The numbers Steinberger throws around at conferences are higher and probably accurate, but the count moves fast enough that any specific figure is wrong by the time you publish it.

Two real differences I felt:

  1. ClawHub skills are JavaScript. They run in a sandbox but can request shell exec privileges. That makes them more capable than Claude Code Skills and more dangerous. The ClawHavoc incident in March of 2026 saw 341 malicious skills caught, which is a real cost of an open marketplace
  2. Claude Code Skills are simpler to author. I wrote a Skill in 20 minutes my first time. The equivalent ClawHub skill took me about 90 minutes because I had to learn the SDK conventions

If you are an individual developer wanting to share a workflow, Skills are easier. If you are a team wanting a versioned, packaged, audited tool, ClawHub is better. They are not competing for the same problem.

The 3pm moment where I almost stopped

I asked OpenClaw to update some Python 3.8 code to 3.11 across a small repo, run the test suite, and report back.

It did. The session ate 920K tokens, took about 14 minutes, found three places where my colleague had used the walrus operator wrong, and quietly fixed them. I checked the diff. It was right.

Claude Code does the same thing. I have run the same prompt against it many times.

The difference was not the output. The difference was that Claude Code is in my muscle memory. I have typed claude three times a day for a year. When I typed openclaw and waited the extra 1.2 seconds for the cold start, my fingers reached for claude instead. Three times.

That is the part nobody writes about. Switching costs are not just config. They are reflexes. By 3pm I had written half a SOUL.md, almost given up, made coffee, and come back. By 6pm I was OK again.

What I would actually use each one for

I built this matrix during the second coffee.

Decision OpenClaw Claude Code
Locked into Anthropic models? No, multi-provider Yes, Anthropic only
Local model option Ollama None official
Skill distribution ClawHub package marketplace Markdown files
Personality file SOUL.md (who is the agent) CLAUDE.md (what is the project)
Network architecture Local Gateway, no relay Direct to Anthropic
Maturity 60 days old, foundation forming 18 months, Anthropic-stable
Best at Multi-model teams, regulated environments Anthropic-first dev shops, simplicity

If your team is Anthropic-only and your CLAUDE.md is already 200 lines, do not switch. Claude Code is fine. The Skills you wrote are still fine. The pattern works.

If your team is multi-provider, or your compliance team has questions about where prompts travel, or you want a backend model dropdown, OpenClaw is worth a Tuesday.

I am still on Claude Code as my default. I have OpenClaw aliased to a separate command for the cases where I want to try a different model on the same prompt without paying for two SaaS subscriptions worth of context.

Where this goes next

OpenClaw moving to a foundation while Steinberger joins OpenAI is the part I am watching most closely. Foundations are how open-source projects survive their founders. They are also how projects ossify. The first six months of governance under the OpenClaw Foundation will tell you whether the project is going to be Linux or Helm.

If you used to argue Claude Code vs Codex was a binary, OpenClaw is the answer that was supposed to be impossible: a third option that is not produced by an LLM lab. The economics of that are interesting. The next twelve months are going to teach us whether neutral, cross-provider, foundation-governed AI tooling is sustainable, or whether it gets quietly absorbed.

I am betting on sustainable. I have also been wrong about agents in roughly all of the previous quarters, so adjust accordingly.

What this all costs to know

If you take one thing from my Tuesday, take this. OpenClaw and Claude Code are not competitors. They are two answers to the same question: what should the AI inside your terminal be allowed to do without asking you first? SOUL.md and CLAUDE.md are different shapes of the same trust contract. The team that wrote each chose differently because they had different assumptions about who was sitting in front of the screen.

The right tool is the one whose assumptions match yours. Pick on assumptions, not stars.

If you want the harness-engineering frame on this (CLAUDE.md tiers, hooks, sub-agents, how to think about the shell around the prompt, not just the prompt), this is the reference:

Harness Engineering: A Practitioner’s Guide

How HTTP Reverse Proxying Lost the Argument and Won the Market

The FastCGI specification is 30 years old today. It was published on April 29, 1996. It is, on a couple of structural axes that have come to matter a great deal in 2026, the protocol that the rest of the industry should be using for proxy-to-backend communication, instead of the protocol it actually uses, which is HTTP. This is not my argument; it is Andrew Ayer’s argument, made on the 30th-birthday post on his blog at agwa.name. Ayer is the founder of SSLMate and notes the company has run FastCGI in production for over ten years. He has, in other words, the receipts.

The piece is short and operational, and the HN thread on it ran 100 comments deep — modest by 2026 front-page standards but heavy on long-tenure-operator testimony. The piece argues that HTTP-as-reverse-proxy-protocol has two structural failure modes that FastCGI does not have, and that the industry’s continued reliance on HTTP for this specific use case is the result of accreted preference rather than careful comparison. The thread argues, additionally, that the reason HTTP won despite being worse is itself worth understanding — and that the answer is the end-to-end principle, applied where it should not have been.

The two structural failure modes

The first failure mode is desync attacks, also known as request smuggling. HTTP/1.1 has the property of looking simple on the surface — “it’s just text!” in Ayer’s parenthetical — and being a nightmare to parse correctly. There is no explicit framing in the protocol. The message itself describes where it ends, via Content-Length, Transfer-Encoding, or sometimes both, and there are enough edge cases in the parsing rules that two HTTP implementations can disagree about where one message ends and the next begins. When the implementation that disagrees with its neighbor is your reverse proxy and the implementation it disagrees with is your backend, an attacker can smuggle a second request inside the body of a first, and the second request gets handled with the privileges of the wrong session.

The recent example Ayer leads with is a desync vulnerability in Discord’s media proxy, disclosed earlier this year, that allowed spying on private attachments. The class is decades old. Watchfire described it in 2005 and warned, prophetically, that the parser-divergence approach to fixing it would be a losing strategy. James Kettle at PortSwigger has spent the last several years finding new variants on a roughly annual cadence. After the most recent batch he declared, on a single-purpose website with the URL http1mustdie.com, that “HTTP/1.1 must die.”

HTTP/2 fixes desync, when consistently used between proxy and backend, by putting clear boundaries around messages. FastCGI fixed it in 1996, with a simpler protocol, by also putting clear boundaries around messages. Ayer notes — and this is the part that lands — that nginx has supported FastCGI backends since its first release, but only got HTTP/2 backend support in late 2025. Apache’s HTTP/2 backend support is, as of this writing, still experimental. FastCGI has been there the entire time.

The second failure mode is untrusted-header confusion. HTTP has no clean way for the proxy to convey trusted information about a request — the real client IP, the authenticated username, client-certificate details — separately from the headers the client itself sent. The conventional approach is to use additional HTTP headers (X-Real-IP, X-Forwarded-For, True-Client-IP) and trust the proxy to strip any client-supplied headers with those names before adding its own. In theory this works. In practice it is a minefield, because there is no structural distinction between trusted and untrusted headers — it’s all just headers — and any part of your stack that looks at the wrong header without your knowledge can be tricked. Ayer’s specific example: Go’s Chi middleware reads True-Client-IP first, falling back to X-Real-IP. Even if your proxy correctly strips X-Real-IP, an attacker who sends True-Client-IP defeats it.

FastCGI structurally cannot have this bug. Trusted parameters and HTTP headers travel in the same key/value list, but HTTP headers are prefixed HTTP_ to mark them as client-originated. The proxy sets REMOTE_ADDR directly; a client trying to forge it would have to send a header literally named HTTP_REMOTE_ADDR, which the backend would parse as a client-set field, not a trusted one. The forgery is a different shape from a successful one. There is no header-name-collision attack in the FastCGI design, because it has domain separation built into the wire format.

FastCGI is a wire protocol, not a process model

The biggest practical objection people raise to FastCGI is that it sounds dated, and the reason it sounds dated is the historical association with the .fcgi per-request-spawning pattern that nobody runs anymore. Ayer’s piece is careful to separate these. FastCGI today is just an alternate transport for HTTP requests over a TCP or Unix socket. In Go, switching is one import and one call:

l, _ := net.Listen("tcp", "127.0.0.1:8080")
fcgi.Serve(l, handler)

fcgi.Serve is a drop-in for http.Serve. The handler stays the same. http.ResponseWriter and http.Request keep their types. nginx, Apache, Caddy, and HAProxy all support FastCGI backends with one or two lines of configuration. The work to switch is roughly the same as adding TLS termination, with much less ongoing maintenance.

The piece also notes that Go’s standard net/http/fcgi automatically populates the Request.RemoteAddr field from the trusted REMOTE_ADDR parameter, and sets the TLS field to a non-nil value when the proxy reports HTTPS. The middleware most Go services use to extract the real client IP from X-Forwarded-For is unnecessary. “It Just Works,” Ayer writes, in one of the few rhetorical flourishes in an otherwise dry piece.

The honest downsides are real and Ayer lists them. FastCGI was never extended to support WebSockets. The tooling is thinner — curl doesn’t speak it, even though it speaks FTP, Gopher, and SMTP. When Ayer benchmarked Go’s FastCGI server behind various reverse proxies, some workloads had worse throughput than HTTP/1.1 or HTTP/2 — which he attributes to the FastCGI code-paths being less optimized rather than to anything inherent in the protocol. The cloud-era “just use HTTP, we’ll handle it” mindset has not made room for FastCGI even though most of the infrastructure could.

Why HTTP won anyway

The historical question Ayer’s post leaves implicit and the HN thread takes up directly is the more interesting one. If FastCGI is structurally better — better-framed, with native domain separation — why did HTTP win the reverse-proxy market?

The thread’s most useful single comment frames it through the end-to-end principle. The argument goes: HTTP everywhere lets you compose intermediate gateways arbitrarily. You can run the same backend stack with a direct browser connection in development and behind a multi-tier proxy stack in production, without code changes. You can introduce a new authentication gateway, a DDoS filter, a TLS terminator, a region-routing layer — at any position in the request chain — without each layer needing to know what the others are doing. The end-to-end principle says: keep the network agnostic to what’s being transmitted, push application logic to the endpoints. HTTP-as-everywhere-protocol is the literal embodiment of that principle for the web.

The piece’s actual recommendation, the comment continues, is the principle of least privilege as the alternative. “Allowlist your communications to only what you expect, so that you aren’t unwittingly contributing to a compromise elsewhere in the network.” Don’t trust headers you didn’t ask for. Don’t allow framing ambiguities you can’t verify. Domain-separate the things that should be domain-separated.

These are both correct principles. They disagree about which threat is the bigger one. The end-to-end principle is what gives the web its compositional flexibility — the property that has, in fact, made the web outperform basically every closed-protocol alternative for thirty years. The principle of least privilege is what catches the class of bug Ayer’s piece is about — the one where a Discord user’s private attachments leak because two parsers disagree on a Content-Length edge case. Both classes of harm are real. The argument over which to prioritize is the argument over which class of harm hurts more, and the practical answer is it depends on what you’re running.

A second thread in the discussion pushed back on the end-to-end framing entirely, arguing that connection caching and multiplexing — both of which the modern HTTP stack does compulsively — already violate the end-to-end principle in ways that explain most of the reverse-proxy exploits. The exploit class exists not because we picked HTTP, but because we picked HTTP and we cached connections and we multiplexed requests across cached connections, all the way through the stack, and pretended the resulting topology was still end-to-end. The desync attacks live exactly at the points where the pretence breaks down.

A third thread — citing Google’s internal Stubby protocol, which wraps HTTP semantics in a different wire protocol for service-to-service traffic — observed that hyperscalers solved this years ago by not using HTTP between their internal services, while continuing to use HTTP at the edge. The compositional flexibility argument is real at the network’s boundary. Inside the boundary, in the proxy-to-backend leg, the boundary doesn’t exist anymore, and the principle is being applied where it doesn’t belong.

That is, on reflection, the right way to read Ayer’s piece. The argument is not “HTTP is bad.” HTTP is fine for the browser-to-edge leg, where end-to-end composability is the whole game. The argument is that the proxy-to-backend leg is not an end-to-end leg, and using HTTP there imports a class of failure mode the leg should not have. FastCGI’s wire format is what HTTP would look like if it were designed with the proxy-to-backend leg’s actual constraints in mind: explicit framing, structural domain separation, no header-name-collision class. It is the protocol the leg should have been using all along.

What the production testimony says

The thread runs heavy on quiet production-engineering testimony. SSLMate’s ten years on FastCGI is one data point. Others in the comments described running FastCGI for “all our web customers” for a decade. The most extensively-documented alternative in the thread was uWSGI, which one commenter said had been their reverse-proxy backbone “at several places for many years” with mostly-praise; another commenter described WAS (Web Application Socket), a separately-designed open-source protocol they built at CM4all roughly fifteen years ago, used in production at CM4all for hosting environments running PHP. Multiple commenters on different stacks converged on the same observation: when you run the same backend behind a non-HTTP reverse-proxy protocol for a long time, you stop having a class of bugs you used to have.

The counter-testimony was also concrete and worth taking seriously. One commenter described founding a Web 2.0 startup in the FastCGI/SCGI/HTTP era and choosing HTTP because “instead of needing to introduce another protocol into your stack, you can just use HTTP, which you already needed to handle at the gateway.” The setup-cost saving is real. nginx came in “lots faster than most FastCGI/SCGI modules of the time, and more robust,” which is also real, and the cost of switching to a less-optimized FastCGI path was eventually worse than the security cost of staying on HTTP. By the time the security cost compounded into a continuous run of desync vulnerabilities, the migration cost had compounded too.

Andrew Ayer himself replied in the thread, on a different sub-discussion about plain CGI, with the httpoxy footgun caveat — that CGI’s use of environment variables to convey HTTP headers introduced a HTTP_PROXY-collision class that doesn’t apply to FastCGI’s parameter list. That distinction matters: the article is about FastCGI, not CGI, and the differences are structural, not just performance.

The interesting question

The interesting question is not whether your service should switch from HTTP-to-backend to FastCGI-to-backend tomorrow. For most services, the answer is the boring one: probably not, because the migration cost is real and the threat model can be managed by stripping headers carefully and keeping a current HTTP/2 stack between proxy and backend. The interesting question is why the industry’s response to the request-smuggling class of bug — twenty years old since the foundational 2005 Watchfire paper — has been patch the wire protocol harder rather than use the wire protocol that doesn’t have this class of bug.

The wire protocol you pick at the proxy-to-backend boundary is a security decision. We have been pretending it was a convenience decision for thirty years. The bill, in CVEs, has been correspondingly persistent. Ayer’s anniversary post is a good moment to notice that the bill is still arriving, and that the alternative on the table is not new, not exotic, and not even particularly hard to deploy. Happy 30th birthday, FastCGI, he closes. Worth marking the year on the calendar where the question got asked again, even if the answer doesn’t move the market for another decade.