TL;DR: We measured failover latency across three AI gateways (Bifrost, LiteLLM, Portkey) during 30 days of production traffic at Nexus Labs. Bifrost added 11ms p99 overhead with automatic provider fallback. The model is the easy part. Routing it reliably is not.

Our agent platform at Nexus Labs handles around 2.4M LLM requests per day. Half of those hit OpenAI, the rest spread across Anthropic, Bedrock, and Vertex. When OpenAI had its 4-hour incident on April 23, we lost 38 minutes of traffic before our homegrown retry logic gave up and rerouted.

That hurt. So we replaced the retry layer.

The actual problem

Most gateway benchmarks measure throughput on a cold path with no failures. That tells you very little about production. What I care about: how long does it take for a request to recover when a provider returns 429 or 503? How much p99 latency does the gateway add when nothing is wrong?

Our team of 9 engineers spent two weeks instrumenting three options. Same hardware (c6i.4xlarge, 2 nodes behind an NLB). Same upstream credentials. Same request distribution sampled from our actual logs.

Setup

Each gateway sat between our agent service and four providers. We configured identical fallback chains: OpenAI primary, Anthropic secondary, Bedrock tertiary. Cache disabled. Rate limits set to mirror our prod allocation.

Here’s the Bifrost config we used:

providers:
  openai:
    keys:
      - value: env.OPENAI_API_KEY
        weight: 1.0
  anthropic:
    keys:
      - value: env.ANTHROPIC_API_KEY
        weight: 1.0
  bedrock:
    keys:
      - value: env.AWS_BEDROCK_KEY
        weight: 1.0

fallbacks:
  - provider: openai
    model: gpt-4o
    fallback_to:
      - provider: anthropic
        model: claude-sonnet-4
      - provider: bedrock
        model: anthropic.claude-sonnet-4

Documented behavior is at https://docs.getbifrost.ai/features/retries-and-fallbacks. LiteLLM and Portkey have equivalent configs. Different YAML shape, same semantics.

Results

We ran 720 hours of mirrored traffic. Numbers below are from the actual logs, not synthetic load.

Bifrost is written in Go. LiteLLM is Python with FastAPI. That accounts for most of the gap on the hot path. Not all of it. Bifrost’s fallback chain evaluates synchronously without re-queuing the request, which matters when you’re already on retry attempt two.

Portkey was solid but the self-hosted version lagged their managed offering in feature parity. LiteLLM’s killer feature for our team was richer support for custom cost-tracking callbacks. We still use those for finance reporting.

What we used Bifrost for

Three things, specifically.

Fallback routing. When OpenAI returns 429, the request goes to Anthropic with the equivalent model. Our agent code never knows. Docs at https://docs.getbifrost.ai/features/retries-and-fallbacks.

Semantic caching. For our evaluation harness specifically. We replay 18,000 prompts against new model versions nightly. Cache hit rate is 73% because the evaluation suite asks the same questions repeatedly. That’s around 13k requests we don’t pay for each night. Reference: https://docs.getbifrost.ai/features/semantic-caching.

Prometheus metrics. Native export. We already had a Prom stack. Five-minute integration. The default dashboards aren’t great but the metrics themselves are useful. Reference: https://docs.getbifrost.ai/features/observability/default.

What we did not use

MCP gateway, governance, SSO. Our auth sits in front of the gateway, not inside it. The custom plugins interface looked interesting but we haven’t needed one yet.

Trade-offs and Limitations

Bifrost is younger than LiteLLM. The provider list is wide (23+) but if you need a niche provider, check the docs first. The plugin interface is straightforward so you can add one yourself, but that’s still work.

The web UI is decent for initial setup, not where you want to be doing complex governance. Configure things in YAML and version them in git like anything else.

If you’re already deep in LiteLLM and using its callback ecosystem, migration cost is real. LiteLLM has more community integrations because it’s been around longer. Portkey is also a fine choice if you want a managed control plane and don’t want to operate a gateway yourself. Pick based on what your team will actually maintain.

Last caveat. The numbers above are from our workload. Your traffic shape will differ. Run the test yourself before deciding.

Further Reading

• Bifrost retries and fallbacks: https://docs.getbifrost.ai/features/retries-and-fallbacks
• Bifrost semantic caching: https://docs.getbifrost.ai/features/semantic-caching
• Bifrost observability: https://docs.getbifrost.ai/features/observability/default
• Bifrost provider configuration: https://docs.getbifrost.ai/quickstart/gateway/provider-configuration
• Bifrost source: https://github.com/maximhq/bifrost

The model is the easy part. Routing it under failure is the hard part. Spend the time on the boring infrastructure problem.

The Problem We Were Actually Solving

I still remember the call from our lead developer in Douala, Cameroon, who told us that the language model we’d deployed on a popular platform store wasn’t working for a certain class of users. In Cameroon, a platform that works is one that uses the official language, French, or the lingua franca, pidgin English. Not the platform’s dominant language, English. The platform’s API, built with an offshore team in India, didn’t account for the differences in languages and character sets our users in Cameroon faced every day. Our users couldn’t even see the available models, let alone use them. We were pushing an AI dream onto a community that had no choice but to adapt to a system that wasn’t built for them.

What We Tried First (And Why It Failed)

We initially thought that building our own APIs on the platform’s infrastructure would solve the problem. We assumed that our local team could just use the same language models and adapt the platform APIs to French and pidgin. The API calls were more or less the same, we figured. We even convinced our offshore team to help us. After a few weeks of development, we realized we’d made a critical misjudgment: the differences in languages and character sets were not just about user interface but also about fundamental system architecture. The platform’s model hosting and deployment required a specific ecosystem that didn’t exist outside the platform’s environment. Our adapted API still wouldn’t work. Every request in our French and pidgin interfaces failed to return any results, and we received error messages about encoding issues we didn’t understand. We’d taken a shortcut that didn’t exist in reality.

The Architecture Decision

We eventually realized that we needed to rethink our entire approach to global access. We couldn’t just adapt the platform’s APIs; we needed to create an entirely new architecture that catered to local languages and character sets. We had to build a custom storage system and an interface that could handle the differences in language and encoding that our users faced every day. We brought in a local team that specialized in African languages, and together, we created a new API that utilized local storage and translation layers. We even developed a small set of language models that were trained on pidgin English and French data. This new architecture not only solved our problem but also helped us tap into a more diverse and global user base.

What The Numbers Said After

After deploying our new architecture, we saw a significant drop in errors related to encoding and language incompatibility. Our French and pidgin interfaces worked seamlessly, and our users were able to see and use the language models we’d deployed for them. Our metrics showed a 90% reduction in failed requests, and our user satisfaction ratings increased by 30%. We were no longer pushing a dream onto a community that had no choice but to adapt; we were delivering a product that truly served the needs of our users in Cameroon and beyond.

What I Would Do Differently

In hindsight, I would have engaged the local team and our offshore team earlier in the development process to ensure that everyone understood the complexity of our problem. I would have invested more time and resources into learning about the languages and character sets our users faced every day. I also would have explored alternative architectures from the start, considering a custom solution from the beginning. Our journey taught me the importance of humility in the face of global complexity and the value of collaboration when tackling seemingly insurmountable problems. By listening to our users in Cameroon, we were able to build a system that truly served their needs.

Target Exploitation & Privilege Escalation (Session 10 Summary)

🔬 1. Vulnerability Research Foundations

Programming: Code analysis using C/C++, Python, Perl, or Assembly.
Reverse Engineering: Analyzing compiled software binaries without original source code.
Instrumented Tools: Monitoring application runtime behaviors with debuggers and fuzzers.
Payload Construction: Engineering reliable shellcode execution pathways.

🗂️ 2. Public Exploit Directories

Public PoCs: Used by penetration testers to safely evaluate software version vulnerabilities.
Platforms: Exploit-DB , Packet Storm, SecurityFocus, NVD, US-CERT , Secunia , and XSSed.

🛠️ 3. Metasploit Console Basics (msfconsole)

Execution: Terminal interface tool for launching framework scans and exploits.
Global Show: Commands like show exploits or show payloads view platform-wide modules.
Contextual Show: Module-specific settings reveal options like show targets or `show advanced.

🚀 4. Privilege Escalation Mechanics

Vertical Escalation: Gaining higher system permissions (e.g., web user to root admin).
Horizontal Escalation: Lateral movement between accounts with matching privilege levels.
Vectors: Local kernel exploits, open home directories exposing stored SSH private keys, weak passwords, network sniffing, and packet spoofing.

🔐 5. Password Auditing Strategy & Tools

Online Guessing Attacks

Mechanics: Remote login attempts over production networks; limited by latency and lockouts.
Tools: Hydra, Medusa, and CeWL (web wordlist scraper).

Offline Cracking Attacks

Mechanics: Reversing stolen database file hashes locally on attacker hardware without network restrictions.
Tools: Hashcat, John the Ripper, Rainbow crack, Ophcrack, Crunch, and Hash-identifier.

🔄 6. Man-in-the-Middle (MitM) & ARP Spoofing

Normal Condition: Nodes exchange standard ARP requests and replies to map local IPs to hardware MAC addresses.
Spoofed Condition: The attacker broadcasts fake ARP responses to poison targets’ cache tables.
Sniffing intercept: Traffic routes through the attacker to be read via Tcpdump, Wireshark, or Dsniff before forwarding.

A missing index on a foreign key column can turn a 5ms query into a 5-second table scan. A correlated subquery in a WHERE clause can multiply your query time by the number of rows. An ORM that generates N+1 queries can bring down a production API under moderate load. SQL performance problems are almost always fixable — the hard part is knowing where to look. This guide is that map.

All examples use PostgreSQL 16 syntax. The concepts apply to MySQL, SQLite, and most relational databases.

How PostgreSQL Picks a Query Plan

Before optimizing, understand what the planner does. PostgreSQL maintains statistics about tables and uses a cost-based optimizer to choose among many possible query plans. It estimates row counts, considers available indexes, and picks the plan with the lowest estimated cost. The planner is usually right — when it is wrong, it is almost always because statistics are stale or misleading.

-- always run ANALYZE before investigating slow queries
ANALYZE orders;

-- basic EXPLAIN (shows plan, no execution)
EXPLAIN SELECT * FROM orders WHERE customer_id = 42;

-- EXPLAIN ANALYZE (runs the query, shows actual vs estimated rows)
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT o.id, o.total, c.email
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.status = 'pending'
  AND o.created_at > NOW() - INTERVAL '7 days';

The output to look for: Seq Scan (table scan — usually bad on large tables), Index Scan (good), Bitmap Index Scan (good for low selectivity), and Hash Join vs Nested Loop vs Merge Join. A large discrepancy between “rows estimated” and “rows actual” means the planner has bad statistics.

B-Tree Indexes: The Foundation

-- basic single-column index
CREATE INDEX idx_orders_customer_id ON orders (customer_id);

-- index on a frequently filtered status column
CREATE INDEX idx_orders_status ON orders (status);

-- partial index — only index the rows you actually query
-- much smaller, faster to maintain
CREATE INDEX idx_orders_pending ON orders (created_at)
WHERE status = 'pending';

-- unique index (enforces uniqueness + faster lookups)
CREATE UNIQUE INDEX idx_users_email ON users (lower(email));

-- expression index — index a computed value
CREATE INDEX idx_users_email_lower ON users (lower(email));
-- now this query uses the index:
-- SELECT * FROM users WHERE lower(email) = '[email protected]';

Composite Indexes: Column Order Matters

A composite index on (a, b, c) can satisfy queries on a, a, b, and a, b, c — but NOT on b alone or c alone. The leading column rule is the most commonly misunderstood fact about composite indexes.

-- this index supports:
-- WHERE status = 'pending'                          ✓
-- WHERE status = 'pending' AND created_at > ...    ✓
-- WHERE status = 'pending' AND created_at > ... AND customer_id = 42  ✓
-- WHERE created_at > ...                            ✗ (can't skip status)
CREATE INDEX idx_orders_status_created_customer
  ON orders (status, created_at, customer_id);

-- covering index — includes all columns the query needs, zero heap fetches
CREATE INDEX idx_orders_covering ON orders (customer_id, status)
INCLUDE (total, created_at);

-- query that uses the covering index with no heap access:
SELECT total, created_at
FROM orders
WHERE customer_id = 42 AND status = 'shipped';

Reading EXPLAIN ANALYZE Output

EXPLAIN (ANALYZE, BUFFERS)
SELECT p.name, SUM(oi.quantity * oi.unit_price) AS revenue
FROM order_items oi
JOIN products p ON p.id = oi.product_id
WHERE oi.created_at BETWEEN '2026-01-01' AND '2026-03-31'
GROUP BY p.id, p.name
ORDER BY revenue DESC
LIMIT 20;

/*
 Sample output sections to read:

 -> Hash Join  (cost=1234.56..5678.90 rows=10000 width=48)
              (actual time=45.123..98.456 rows=8734 loops=1)
   Hash Cond: (oi.product_id = p.id)
   Buffers: shared hit=1023 read=234

   -> Bitmap Heap Scan on order_items oi
         (actual time=12.3..34.5 rows=92345 loops=1)
      Recheck Cond: (created_at BETWEEN ...)
      ->  Bitmap Index Scan on idx_oi_created_at
            (actual time=8.9..8.9 rows=92345 loops=1)

 Planning Time: 2.1 ms
 Execution Time: 102.3 ms
*/

Key metrics: actual time is the real wall time. rows discrepancy is your optimization signal. Buffers: read=N means disk I/O — large values indicate missing indexes or cold cache.

Eliminating N+1 Queries

The N+1 problem happens when you fetch a list of records, then run one query per record to fetch related data. In SQL, the fix is always a JOIN.

-- BAD: 1 query for orders + N queries for customers
SELECT * FROM orders WHERE status = 'pending';
-- then for each order:
SELECT * FROM customers WHERE id = ?;

-- GOOD: single query with JOIN
SELECT
  o.id,
  o.total,
  o.created_at,
  c.id         AS customer_id,
  c.email      AS customer_email,
  c.first_name,
  c.last_name
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.status = 'pending'
ORDER BY o.created_at DESC;

-- fetching nested aggregates without N+1
-- BAD: separate query per order for item count
SELECT * FROM orders WHERE created_at > NOW() - INTERVAL '30 days';
-- then: SELECT COUNT(*) FROM order_items WHERE order_id = ?

-- GOOD: aggregate in JOIN
SELECT
  o.id,
  o.total,
  COUNT(oi.id)              AS item_count,
  SUM(oi.quantity)          AS total_units
FROM orders o
LEFT JOIN order_items oi ON oi.order_id = o.id
WHERE o.created_at > NOW() - INTERVAL '30 days'
GROUP BY o.id, o.total
ORDER BY o.created_at DESC;

CTEs vs Subqueries

-- subquery in FROM (derived table) — planner can inline and optimize
SELECT p.name, revenue_data.total_revenue
FROM products p
JOIN (
  SELECT product_id, SUM(quantity * unit_price) AS total_revenue
  FROM order_items
  WHERE created_at > NOW() - INTERVAL '90 days'
  GROUP BY product_id
) revenue_data ON revenue_data.product_id = p.id
ORDER BY revenue_data.total_revenue DESC;

-- CTE — cleaner syntax, same performance in Postgres 12+
-- (CTEs are now inlined by default unless MATERIALIZED is specified)
WITH revenue_by_product AS (
  SELECT
    product_id,
    SUM(quantity * unit_price) AS total_revenue,
    COUNT(DISTINCT order_id)   AS order_count
  FROM order_items
  WHERE created_at > NOW() - INTERVAL '90 days'
  GROUP BY product_id
),
top_products AS (
  SELECT product_id, total_revenue, order_count
  FROM revenue_by_product
  WHERE total_revenue > 1000
)
SELECT p.name, tp.total_revenue, tp.order_count
FROM top_products tp
JOIN products p ON p.id = tp.product_id
ORDER BY tp.total_revenue DESC
LIMIT 50;

-- force materialization when the CTE result is expensive + reused
WITH MATERIALIZED expensive_aggregation AS (
  SELECT customer_id, AVG(total) AS avg_order_value
  FROM orders
  WHERE created_at > NOW() - INTERVAL '1 year'
  GROUP BY customer_id
)
SELECT c.email, ea.avg_order_value
FROM expensive_aggregation ea
JOIN customers c ON c.id = ea.customer_id
WHERE ea.avg_order_value > 200;

Window Functions

Window functions compute a value across a set of rows related to the current row — without collapsing them into a single GROUP BY result. They are one of the most powerful SQL features for analytics.

-- running total
SELECT
  created_at::date      AS date,
  SUM(total)            AS daily_revenue,
  SUM(SUM(total)) OVER (
    ORDER BY created_at::date
    ROWS UNBOUNDED PRECEDING
  )                     AS cumulative_revenue
FROM orders
WHERE created_at >= '2026-01-01'
GROUP BY created_at::date
ORDER BY date;

-- rank products by revenue within each category
SELECT
  p.name,
  p.category,
  SUM(oi.quantity * oi.unit_price)          AS revenue,
  RANK() OVER (
    PARTITION BY p.category
    ORDER BY SUM(oi.quantity * oi.unit_price) DESC
  )                                          AS rank_in_category
FROM products p
JOIN order_items oi ON oi.product_id = p.id
GROUP BY p.id, p.name, p.category
ORDER BY p.category, rank_in_category;

-- compare to previous period (LAG)
SELECT
  month,
  revenue,
  LAG(revenue) OVER (ORDER BY month)  AS prev_month_revenue,
  ROUND(
    (revenue - LAG(revenue) OVER (ORDER BY month))
    / NULLIF(LAG(revenue) OVER (ORDER BY month), 0) * 100,
    2
  )                                     AS pct_change
FROM (
  SELECT
    DATE_TRUNC('month', created_at) AS month,
    SUM(total)                      AS revenue
  FROM orders
  GROUP BY 1
) monthly
ORDER BY month;

-- deduplicate: keep latest record per customer
SELECT DISTINCT ON (customer_id)
  customer_id, id AS order_id, created_at, total
FROM orders
ORDER BY customer_id, created_at DESC;

Index Maintenance

-- find unused indexes (waste space, slow writes)
SELECT
  schemaname,
  tablename,
  indexname,
  idx_scan,
  pg_size_pretty(pg_relation_size(indexrelid)) AS index_size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
  AND indexrelname NOT LIKE 'pg_%'
ORDER BY pg_relation_size(indexrelid) DESC;

-- find missing indexes on foreign keys
SELECT
  tc.table_name,
  kcu.column_name,
  ccu.table_name AS references_table
FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage kcu
  ON tc.constraint_name = kcu.constraint_name
JOIN information_schema.constraint_column_usage ccu
  ON tc.constraint_name = ccu.constraint_name
WHERE tc.constraint_type = 'FOREIGN KEY'
  AND NOT EXISTS (
    SELECT 1 FROM pg_indexes
    WHERE tablename = tc.table_name
      AND indexdef LIKE '%' || kcu.column_name || '%'
  );

-- rebuild bloated indexes concurrently (no table lock)
REINDEX INDEX CONCURRENTLY idx_orders_customer_id;

People Also Ask

When should I use a partial index instead of a full index?

Use a partial index when your queries consistently filter on a specific condition — like WHERE status = 'pending' or WHERE deleted_at IS NULL. A partial index only indexes rows matching the condition, so it is smaller (faster to build, cheaper to maintain) and has higher selectivity (more likely to be used by the planner). The tradeoff is that it only helps queries that include the matching WHERE clause.

Why does adding an index sometimes make queries slower?

For very low-selectivity queries (e.g., WHERE status IN ('a', 'b', 'c') matching 80% of rows), a sequential scan is actually faster than an index scan because the index forces random I/O to fetch each heap page individually. The planner knows this and will choose a seq scan. You can force it with hints in development, but in production you should trust the planner — and investigate its statistics if you think it is wrong.

What is the difference between EXPLAIN and EXPLAIN ANALYZE?

EXPLAIN shows the query plan the planner would use — it does not execute the query. EXPLAIN ANALYZE actually runs the query and shows both the estimated and actual row counts and timings. Always use EXPLAIN ANALYZE with BUFFERS (EXPLAIN (ANALYZE, BUFFERS)) when debugging performance issues so you can see disk I/O. Be careful: EXPLAIN ANALYZE on a DELETE or UPDATE will actually execute those statements, so wrap them in a transaction you roll back if needed.

Originally published at wowhow.cloud

On April 29, 2026, Mitchell Hashimoto announced he was moving Ghostty off GitHub. His phrasing — “GitHub is no longer a place for serious work if it just blocks you out for hours per day, every day” — landed on the front page of Hacker News via The Register and stayed there. The departure itself isn’t the story. The departure plus the four other threads on the same front page that week — about a federated-forge protocol, a security audit of GitHub’s leading alternative, the Dutch government’s Forgejo-based code platform, and Armin Ronacher’s long essay on what came before GitHub — those, taken together, are the story. Five threads, one shape.

Hashimoto is not a casual user. He is HashiCorp’s co-founder; he is the developer behind Ghostty, the terminal emulator he has been working on since leaving HashiCorp; and as he put it in his own post, he is “GitHub user 1299, joined Feb 2008.” He is, in the phrase he used to introduce his journal, the kind of person who “doom scroll[s] GitHub issues since before that was a word.” If GitHub still feels like home for anyone, it would be him.

His description of the past month is what makes the post unusual. He kept a journal of dates, putting an “X” next to every day a GitHub outage had blocked him from doing work. “Almost every day has an ‘X’,” he wrote. “On the day I am writing this post, I’ve been unable to do any PR review for ~2 hours because there is a GitHub Actions outage.” The Register noted that the post itself appeared just before an April 28 incident in which pull requests stopped completing because of an Elasticsearch failure. The official excuse circulating around the thread — that GitHub is straining under a flood of vibe-coded projects — got the obvious counter from one commenter: “if you’ve built a public SaaS before you know the job is not to host the software, it’s to put rails around people taking it down. They’ve had since 2008 to build those rails, and they’re just now hitting places that take the service down on the regular.” Whether the surge story is the real cause or the convenient one, the customer-side argument lands either way.

The closing line of his post is what gives the piece its weight: “I want to ship software and it doesn’t want me to ship software.”

This is not the rhetoric of a writer with strong views on software freedom. It is the rhetoric of someone who has just realized that the platform under his work has stopped being a platform and started being a problem.

The same week, four more threads

On April 28, Armin Ronacher published Before GitHub, an essay tracing his own pre-GitHub project life — SourceForge, his own Trac installation, Subversion repositories on infrastructure he ran himself, the Pocoo collective he ran with Georg Brandl. The piece is gentle and careful, and the central observation is one that any developer who’s been in the field for more than fifteen years can confirm in their own bones. “Subversion in particular made this ‘running your own forge’ natural. It was centralized: you needed a server, and somebody had to operate it… When Mercurial and Git arrived, they were philosophically the opposite. Both were distributed. … In principle, those distributed version control systems should have reduced the need for a single center. But despite all of this, GitHub became the center. That is one of the great ironies of modern Open Source. The distributed version control system won, and then the world standardized on one enormous centralized service for hosting it.”

That same week, the team at Tangled published their argument for why open source needs a federation of forges. Their proposal is technical: pair git for code transfer with the AT protocol for the social fabric of issues and pull requests, so that a developer on one server can open a PR against a repo on a completely different server, the way email federates today. “Centralized systems always crumble; it’s the emails, gits, and IRCs that stand the test of time,” the post observes. The post is short. The 384-comment HN thread does the surrounding work — the trade-offs, the comparisons against ActivityPub-based alternatives, the cost of social fragmentation — and the post links to ForgeFed for the ActivityPub side.

On April 29, the Dutch government soft-launched code.overheid.nl — “a government-wide code platform for publishing and developing open-source software,” fully self-hosted on Forgejo, framed as “a European, sovereign alternative to GitHub and GitLab.” The platform is initiated by the Open Source Program Office at the Ministry of the Interior and Kingdom Relations. This is not a hobbyist statement. It is the moment a sovereign government writes off the social-infrastructure assumption that GitHub will always be there.

And on the day before Ronacher’s essay, Julien Voisin (jvoisin) at dustri.org published a security write-up of Forgejo — the Gitea fork that Codeberg hosts, and that Fedora has now adopted as its primary forge. Voisin’s results are not gentle. “It took me one evening after work to find a good amount of vulnerabilities,” he writes — SSRF, missing CSP and Trusted Types, JavaScript templating issues, authentication problems across OAuth2/OTP/sessions/recovery, low-hanging DoS, information leaks, TOCTOU bugs — and chained some of them into “a full-blown RCE, some secrets leaks, a bunch of persistent account access, a handful of OAuth2 privesc.” He chose a “carrot disclosure” — publishing a redacted exploit output as evidence rather than reporting individual bugs through Forgejo’s security policy — because “the codebase (not their fault though, they inherited the gitea/gogs ones)” has “systemic issues” that won’t be fixed by playing whack-a-mole.

The comment thread on Voisin’s piece converges on the obvious corollary. The alternative forges are not ready. The alternative forges are projects with five-figure star counts, small maintainer teams, codebases inherited from a decade of patchwork development. Forgejo is not GitHub-with-the-ethics-fixed. Forgejo is a different product at a different stage of its lifecycle, with different gaps.

What GitHub actually became

Read these five pieces side by side and the question they’re collectively asking starts to come into focus. It isn’t “is GitHub broken?” — Hashimoto’s journal answers that. It isn’t “are there alternatives?” — Tangled, Codeberg, Forgejo, code.overheid.nl, Sourcehut, GitLab self-hosted, all answer that. The question they’re asking is: what was GitHub, exactly, and what would we lose if it stopped being it?

Ronacher’s essay is the most generous attempt to answer. “GitHub was, and continues to be, a tremendous gift to Open Source,” he writes. “It made creating a project easy and it made discovering projects easy. It made contributing understandable to people who had never subscribed to a development mailing list in their life. It gave projects issue trackers, pull requests, release pages, wikis, organization pages, API access, webhooks, and later CI.” But the underappreciated piece, in his telling, was archival. “GitHub became a library. It became an index of a huge part of the software commons because even abandoned projects remained findable. You could find forks, and old issues and discussions all stayed online. For all the complaints one can make about centralization, that centralization also created discoverable memory.”

That archival role wasn’t designed. It was a side-effect of being the center long enough that everything ended up there. Ronacher quotes his own earlier work to make the parallel point: in the pre-GitHub world he lived in, some of his old packages are “technically still on PyPI, but the actual packages are gone. The metadata points to my old server, and that server has long stopped serving those files.” That’s what the world before GitHub looked like at scale. “A personal domain expired, a VPS was shut down, a developer passed away, and with them went the services they paid for. The web was once full of little software homes, and many of them are gone.”

This is the part of the GitHub critique that the “just leave” faction doesn’t account for. Leaving GitHub is technically easy — git push --mirror and you’re done. Leaving the social infrastructure that made GitHub the default place to find a project, the place to verify that an npm package corresponds to a real maintainer with a real history, the place that a billion of trusted publishing handshakes flow through — that’s not a git push. That’s an institution.

It’s worth being concrete about what does and doesn’t move when a project actually leaves:

Code moves. Trust, reputation, and the conversation around the work do not. That’s the part the move surfaces.

What’s actually breaking

The visible artifact is outages. “Almost every day has an ‘X’,” in Hashimoto’s accounting. The GitHub statuses tracker, cited in the Hacker News thread by one commenter, was registering uptime down around 86% over a recent window. The user-facing symptoms are well-known by now: PR review dies for a couple of hours; the Actions queue piles up; somebody posts the latest status-page incident; a thread runs.

But the deeper change Ronacher names is more important. “People are tired of the instability, the product churn, the Copilot AI noise, the unclear leadership, and the feeling that the platform is no longer primarily designed for the community that made it valuable.” GitHub didn’t break in a single visible way. It drifted from being a developer tool with a community wrapped around it into being a generative-AI front-end with a developer tool wrapped around that. Hashimoto’s frustration, read carefully, isn’t about uptime as an SLA number. It’s about a service that has stopped acting as if its job is to let him ship software.

Ronacher puts it more directly: “the site has no leadership! It’s a miracle that things are going as well as they are.”

The hardest part of the current moment is that the alternatives are not, individually, ready. Voisin’s audit makes that point with painful concreteness on the Forgejo side — and Forgejo is the one with the most institutional momentum, hosted by Codeberg, adopted by Fedora, and chosen by the Dutch government for code.overheid.nl. The alternatives have inherited a decade of accumulated patchwork from upstream codebases. They are run by smaller teams. The Tangled federation is, by their own admission, a pitch. The Dutch platform is explicitly “a pilot using Forgejo,” and “not all government organisations can use the platform yet.” There is no drop-in replacement waiting on the shelf for the next person who decides to leave.

For someone making that calculation today, the ladder of alternatives currently looks roughly like this:

None of these is “GitHub minus the part you don’t like.” Each is a different product, on a different lifecycle, with its own gaps.

The cost of dispersion

Ronacher names the thing that “just leave” arguments tend to wave past: “Going back to many forges, many servers, many small homes, and many independent communities will increase decentralization, and in many ways it will force systems to adapt. … It can also make the web forget again. … Issues, reviews, design discussions, release notes, security advisories, and old tarballs are fragile. They disappear much more easily than we like to admit. Mailing lists, which carried a lot of this in earlier years, have not kept up with the needs of today, and are largely a user experience disaster.”

This is the warm-critical part of the argument. Dispersion is good for autonomy. It is also bad for memory. The web that came before GitHub had more autonomy and more loss. Some of that loss was in code; more of it was in the social context around the code — who wrote what, why, in response to which discussion, with which trade-offs. None of that is portable in a git push --mirror. None of it federates cleanly through a forge protocol that hasn’t been written yet.

What the five pieces from this past stretch ask, collectively, is what we’d want next. Tangled’s answer is technical — federate the social fabric the way email is federated, and let projects choose their hosts. The Dutch government’s answer is institutional — a sovereign-funded forge for civic software, run by the Ministry of the Interior, which can’t be sold and can’t be repositioned. Voisin’s answer is critical — until the alternatives are audited at scale, they cannot inherit the trust GitHub accidentally accumulated. Hashimoto’s answer is practical — Ghostty leaves, the read-only mirror stays, the personal projects stay, the day-job code goes somewhere new.

Ronacher’s answer is the largest. “What I would like to see is some public, boring, well-funded archive for Open Source software. Something with the power of an endowment or public funding to keep it afloat. Something whose job is not to win the developer productivity market but just to make sure that the most important things we create do not disappear.”

This is the right ask. It is also the hardest of the asks because it is the one with no obvious bidder. Open-source archival is uneconomical. The closest things we have — the Internet Archive’s Software Collection, the various academic mirroring projects — are chronically underfunded relative to the cultural weight they carry. The Dutch effort is the kind of model that could scale. It would take more of them.

What we owe each other

Open source is, as Ronacher writes, more than where the code lives. For most of the past two decades, it was where a lot of the community lived too — the maintainers you trusted, the discoveries that came from the issue-tracker browsing of someone two time zones away. Those things happened on top of GitHub because GitHub was where the social infrastructure had ended up. The fact that they happened anywhere is what made open source feel inclusive. Five pieces from this stretch on the front page of HN are not announcing the end of that. They are announcing the beginning of an honest conversation about what comes after — for the first time including people who built the largest projects on GitHub and the institutions that keep critical software running.

The question now is what we keep when the center moves. The answer is not going to be one platform. It is going to be a public archive, a federated forge protocol, a few sovereign-funded forges, several commercial alternatives that compete on uptime and developer experience, and many smaller homes that come and go the way they did before. Some of what we built on top of GitHub will not survive the move.

Hashimoto’s read-only mirror of Ghostty on GitHub is a small, careful gesture toward exactly that. He is leaving a toothbrush at the old apartment. He is also keeping the locks on his new place changed. The question he is asking, in moving, is the question the rest of us will be asked too — perhaps not this year, but soon enough that it is worth thinking about now: what would you take with you, if your project’s home stopped being one?

One thing that slowly wore me down over the years of frontend development wasn’t JavaScript.

It wasn’t components.

It wasn’t CSS itself.

It was the increasing amount of visual noise in the markup.

At some point, UI code started feeling less like structure and more like string management.

You’d open a component and see something like:

<div class="flex flex-col items-center justify-between gap-6 rounded-2xl border border-zinc-800 bg-zinc-900/50 p-8 shadow-lg backdrop-blur-md">

And while utility-first systems absolutely solve real problems, I eventually realized something:

I was spending more time parsing styling syntax than understanding the component itself.

That started bothering me.

A lot.

I Wanted the Markup to Read Like Structure Again

Over time, I became increasingly interested in making UI code feel:

• readable
• semantic
• intentional
• composable
• expressive

Not just:

“technically efficient.”

That thinking eventually led me toward building Juice — an attribute-driven UI system I’ve been experimenting with as part of the broader WebEngine ecosystem.

The goal wasn’t:

“replace CSS.”

And it wasn’t:

“reinvent HTML.”

It was:

“make components readable again.”

Here’s an Example

This is a pricing section component written using Juice-style attributes:

<section type="pricing" id="pricing" paddingY="roomy">
    <div container>
        <div section-header>
            <h2>Pricing That Scales With You</h2>
            <p>Start free and grow at your own pace. No hidden fees, no surprises.</p>
        </div>

        <div grid="3x1" gap="comfortable">
            <div card="pricing">

Even before reading the implementation details, the structure is understandable.

You can visually identify:

• sections
• layout intent
• spacing
• grouping
• component purpose

without parsing long utility chains.

That distinction matters to me.

Because I think frontend architecture should optimize for human readability just as much as browser rendering.

The Problem Wasn’t Utility Classes

To be clear:
utility-first frameworks solve legitimate problems.

They:

• improve consistency
• reduce CSS sprawl
• accelerate development
• encourage reusable spacing systems

I completely understand why they became popular.

But after building larger systems, I noticed a tradeoff emerging.

The markup itself became increasingly overloaded with implementation detail.

Eventually components started feeling visually dense.

Not structurally expressive.

Structure and Styling Started Blending Together

One thing I kept asking myself was:

“Can I quickly understand the purpose of this component?”

Not:

“Can I understand the CSS?”

But:

“Can I understand the architecture?”

That’s a different question entirely.

Because when UI systems scale, readability becomes incredibly important.

Especially when:

• teams grow
• components evolve
• systems become long-lived
• onboarding matters
• design systems expand

The easier a component is to mentally parse, the easier it becomes to maintain over time.

I Wanted Styling to Feel More Declarative

One thing I love about attributes is that they naturally feel declarative.

Instead of:

<div class="pt-24 pb-24">

You can express intent more directly:

<section paddingY="roomy">

Now the component describes meaning instead of implementation detail.

That subtle difference changes how the code feels to work with.

Components Start Reading More Like Systems

As I kept experimenting with the approach, I noticed something interesting.

The markup started feeling less like:

“a pile of styled divs”

and more like:

“a structured UI language.”

For example:

<div card="pricing" featured>

immediately communicates:

• this is a card
• it’s specifically a pricing card
• it has a featured state

without needing:

• multiple utility chains
• conditional class merging
• long style abstractions

The component becomes easier to reason about visually.

The Goal Was Never “Less CSS”

Ironically, this approach actually made me think more about CSS architecture.

Because now the system itself needs:

• scalable tokens
• predictable patterns
• strong conventions
• reusable layout primitives
• intentional naming

The complexity doesn’t disappear.

It shifts downward into the design system layer instead of living directly in every component.

And personally, I think that’s a healthier separation of concerns.

I Think Modern UI Has a Readability Problem

One thing I rarely hear discussed is how mentally exhausting frontend markup can become at scale.

Not because developers are bad.

Not because frameworks are bad.

But because modern UI systems often optimize heavily for:

• flexibility
• speed
• composability

while readability becomes secondary.

That tradeoff accumulates over time.

Especially in large applications.

This Isn’t About “Winning” Frontend Development

I don’t think there’s one perfect way to build UI.

And I’m not interested in framework tribalism.

Honestly, most frontend ecosystems have good ideas worth learning from.

This is more about exploring:

• readability
• semantic structure
• maintainability
• expressive architecture

and asking:

“What should UI systems feel like five years from now?”

Because I don’t think frontend architecture is finished evolving yet.

Not even close.

Why This Matters to Me

The longer I work on systems like:

• WebEngine
• KiwiPress
• Juice
• Sig.js

the more I care about creating software that:

• scales operationally
• stays understandable
• remains expressive
• reduces hidden complexity

I want components to feel approachable.

I want systems to feel intentional.

And I want markup to communicate structure clearly again.

Final Thoughts

I still appreciate utility-first systems.

I still appreciate modern frontend tooling.

But personally, I’ve found myself increasingly drawn toward UI architectures that emphasize:

• intent
• structure
• readability
• semantics
• declarative design

Because eventually frontend code stops being just implementation.

It becomes part of the language teams use to communicate architecture to each other.

And I think that language matters more than we sometimes realize.

Hi all!

We’re MacPaw – the team behind well-known products including CleanMyMac and Setapp, our unified app marketplace.

Connecting with independent developers has always been something we value at MacPaw, as we know the best solutions and collaboration can come from your creativity. We’re sharing our news here because we think the developer community may be interested in what we’re working on at Setapp.

For nine years, we’ve been focused on making Setapp a platform that connects great solutions from independent developers to a community of passionate users. In line with this, we’re expanding our approach to provide our partner developers with a more optimized experience for offering their apps on the platform.

These changes are focused on:

1. Improving accessibility and management of AI plans from different providers
2. Expanded monetization options for applications on Setapp

The two main updates are:

New AI Gateway – a secure, unified API layer that allows developers to access multiple AI models from providers including OpenAI, Anthropic, Google Gemini, and xAI

A few benefits of this approach include:

• Simplified billing
• Easier model integration into your application
• Improved cost optimization (e.g., caching repeated requests, dynamic model routing, batch processing, and more)
• Access to ready-to-use SDKs for macOS and web (iOS support will come in the future).

For more details, you can visit: https://docs.setapp.com/docs/ai-gateway

New marketplace capabilities – we’re rolling out new purchase models for our partner developers, including the ability to offer your product outside the Setapp Membership

We’re aiming to open our ecosystem more to welcome more developers to the platform. This approach offers early-stage developers benefits, including:

• The opportunity to fine-tune your product with the Setapp team
• Access to a global community of users with various distribution channels
• Opportunities to receive feedback and start monetizing your app

Additionally, apps that may not have been available as a part of our Setapp Membership will now have the opportunity to reach our user base with more diverse monetization models.

The full announcement can be found at the link here: https://setapp.com/news/company-updates

Already a part of Setapp or interested in joining? We’d love to hear your feedback in the comments below. Your valuable insights help us shape Setapp to better serve our developer community! ⬇️