Part 2 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 1: I walked through why I replaced Multipass with OrbStack, the dual-cluster architecture I settled on, and a preview of the M1 vs M4 CNI problem that’s coming in Part 4.

The cluster I am going to set up in this article is the one I spend most of my working day inside. It’s a single-node Kubernetes cluster, always on, idles at around 512 MB of memory, has real LoadBalancer IPs and wildcard DNS out of the box. No MetalLB, /etc/hosts editing,kubectl port-forward muscle memory. By the time this article is done, it will also have Istio, Vault, and Crossplane running. Total elapsed time, the first time you do it: about ten minutes.

If you’ve ever built a local Kubernetes cluster and then spent the next twenty minutes wiring up MetalLB and editing /etc/hosts so you can actually reach a service from a browser, this is going to feel almost suspicious.

# 💻 Mac
orb start k8s

# Confirm cluster is udr
kubectl get nodes
# NAME       STATUS   ROLES                  AGE   VERSION
# orbstack   Ready    control-plane,master   30s   v1.33.x

kubectl config current-context
# orbstack

That’s all! No kubeadm, no CNI configuration, no certificate management. The cluster is up and reachable in under thirty seconds the first time, and instantly on every subsequent start.

What makes OrbStack’s networking special?

This is where OrbStack genuinely earns its keep. On a typical local cluster; kind, minikube, kubeadm, LoadBalancer services stay in pending state until you install MetalLB on OrbStack:

• LoadBalancer services get a real, reachable IP automatically.
• *.k8s.orb.local wildcard DNS resolves from your Mac browser, with no /etc/hosts entry.
• cluster.local DNS also resolves from your Mac.
• Every service type is reachable without kubectl port-forward.

⚠️ The wildcard DNS only resolves on your Mac. Other devices on your network won’t see *.k8s.orb.local. If you need a service reachable from another machine, that’s what Cluster 2 (Parts 3–6) is for.

Installing Istio via Helm.

I use Helm rather than istioctl for two reasons.
First, it’s how I manage Istio on the production EKS clusters at work, so the muscle memory transfers.
Second, Helm gives fine-grained control over resource requests, which matters on a laptop.

# 💻 Mac
kubectx orbstack

# Add helm charts
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

# Step 1 - Base CRDs
helm install istio-base istio/base 
  --namespace istio-system --create-namespace 
  --set defaultRevision=default

# Step 2 - Control plane
# The PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION flag is required on OrbStack
# to prevent DNS resolution conflicts with the host network.
helm install istiod istio/istiod 
  --namespace istio-system 
  --set pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true 
  --set global.proxy.resources.requests.cpu=10m 
  --set global.proxy.resources.requests.memory=64Mi 
  --wait

# Step 3 - Ingress gateway
# OrbStack assigns a real LoadBalancer IP automatically - no MetalLB needed.
helm install istio-ingress istio/gateway 
  --namespace istio-ingress --create-namespace 
  --set service.type=LoadBalancer

# Verify the gateway got an EXTERNAL-IP
kubectl get svc -n istio-ingress
# NAME            TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)
# istio-ingress   LoadBalancer   10.x.x.x     198.19.x.x    80:xxx/TCP

# Enable sidecar injection on the default namespace
kubectl label namespace default istio-injection=enabled

Gateways and Virtual Services on the native cluster.

This is the moment OrbStack feels like cheating. You create a Gateway pointing at *.k8s.orb.local, and it just works from your Mac browser. No IP lookups. No /etc/hosts, no 127.0.0.1:8080 proxying.

How the traffic actually flows.

The Gateway resource binds to the istio-ingress LoadBalancer service, OrbStack intercepts traffic to the *.k8s.orb.localwildcard domain at the Mac level and routes it to that LoadBalancer IP. The Virtual Service then routes inside the cluster to the right service.

Example 1: Basic routing with httpbin:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  selector:
    app: httpbin
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "httpbin.k8s.orb.local"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "httpbin.k8s.orb.local"
  gateways:
  - httpbin-gateway
  http:
  - route:
    - destination:
        host: httpbin
        port:
          number: 80
EOF

Open http://httpbin.k8s.orb.local in your Mac browser. It works immediately.

Example 2: Traffic splitting (canary):

The same pattern used in production canary deployments:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-split
spec:
  hosts:
  - "myapp.k8s.orb.local"
  gateways:
  - myapp-gateway
  http:
  - route:
    - destination:
        host: myapp-v1
        port:
          number: 80
      weight: 80
    - destination:
        host: myapp-v2
        port:
          number: 80
      weight: 20
EOF

Example 3: Path-based routing:

Route /v1 and /v2 to different backend services:

# 💻 Mac
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: path-routing
spec:
  hosts:
  - "api.k8s.orb.local"
  gateways:
  - api-gateway
  http:
  - match:
    - uri:
        prefix: /v1
    route:
    - destination:
        host: api-v1-svc
        port:
          number: 80
  - match:
    - uri:
        prefix: /v2
    route:
    - destination:
        host: api-v2-svc
        port:
          number: 80
EOF

One OrbStack gotcha: don’t enable httpsRedirect!

Do not use httpsRedirect: true in a Gateway on the native cluster. OrbStack intercepts LoadBalancer traffic in a way that causes an infinite 301 redirect loop when TLS redirect is enabled.

# ❌ This breaks on OrbStack native K8s
servers:
- tls:
    httpsRedirect: true

# ✅ Use plain HTTP on the native cluster
servers:
- port:
    number: 80
    protocol: HTTP

For TLS testing, use Cluster 2 (the VM cluster, coming in Part 3), where you have full control over the network stack.

Installing the rest of the daily stack:

Vault in dev mode.

Dev mode trades durability for speed. No unsealing, no persistence concerns, instant startup. It’s the right call for the daily-driver cluster where I’m testing AppRole workflows, PKI policies, or Kubernetes auth configurations, and the value is in iteration speed, not data preservation.

# 💻 Mac
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

helm install vault hashicorp/vault 
  --namespace vault --create-namespace 
  --set "server.dev.enabled=true"

kubectl get pods -n vault
# vault-0   1/1   Running   0   30s

For production-grade Vault with HA Raft storage, use Cluster 2 (Part 3). Dev mode here is intentional — it trades durability for speed.

Crossplane:

Crossplane turns the Kubernetes cluster into a universal control plane for cloud infrastructure. I use it heavily with the AWS provider at work.

# 💻 Mac
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

# Note: --enable-composition-functions was removed in newer versions.
# Composition Functions are enabled by default now.
helm install crossplane crossplane-stable/crossplane 
  --namespace crossplane-system --create-namespace

kubectl get pods -n crossplane-system

Stop/start without losing your work.

# 💻 Mac

# Stop - releases all CPU and RAM
orb stop k8s

# Start - full state persists (deployments, services, secrets, configmaps)
orb start k8s

# Verify
kubectx orbstack
kubectl get nodes
kubectl get pods -A

The native cluster state persists across stop/start. Vault dev mode data is lost on restart by design, but everything else, Crossplane installations, Istio configuration, and your workload deployments come back exactly as provisioned.

Quick reference for this cluster:

What’s next.

The daily-driver cluster is the easy half of this lab. The hard half is the one that mirrors production, a real multi-node kubeadm cluster running on four VMs, with HashiCorp Vault as its certificate authority.

That’s where Part 3 picks up: creating the VMs, wiring their networking, and standing up a 3-tier Vault PKI that will sign every certificate in the cluster.

← Part 1: Why I Replaced Multipass with OrbStack.. | Part 3: Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts →

I’m Noah Makau, a DevSecOps engineer based in Nairobi. I run a small DevOps consultancy and hold CKA, CKAD, and AWS Solutions Architect Professional certifications, currently preparing for CKS. I write about Kubernetes, Vault, Crossplane, and the day-to-day of running platforms that actually have to stay up.

Originally published at blog.arkilasystems.com.

I built a small public experiment.

You can open a GitHub Issue and ask an AI-agent workflow to create a static page on my website:

1. If your request fits the rules, the workflow may turn it into a published page.
2. If your request does not fit the rules, the agents should reject it and explain why.

And if you want to make the experiment more interesting, do not only submit a valid request. Try to bypass the boundary. Try to make the workflow publish something it should not publish. Then inspect the issue comments, labels, pull request, and final result.

This is the game.

But under that game, there is a more serious question:

Can external user intent enter a real product repository and become a bounded, reviewable product change without giving up product ownership?

That is what I am trying to explore.

The public playground is here: Demo Pages

The challenge

The surface rule is simple:

Create a GitHub Issue that requests a small static demo page.

The workflow should read the issue, decide whether the request belongs to the allowed area, run the required agent steps, create or update the page, validate the result, and publish it only if the process succeeds.

But this is not a public CMS.

It is not a hosting service.

It is not a website builder.

It is not a place where an external user should be able to change my homepage, global layout, styles, scripts, product pages, or site structure.

The allowed result is intentionally narrow: a bounded inert static page under the Demo Pages section.

So the challenge has two sides.

First, try the normal path:

Create a small demo page request and see whether the workflow can turn it into a published static artifact.

Then try the adversarial path:

Ask for something outside the boundary and see whether the workflow stops you.

For example, you can try to request a homepage change, inject instructions into the issue body, ask the agent to modify global styles, or try to smuggle forbidden behavior into the generated content.

Please keep the challenge inside the intended boundary. This is a workflow and prompt-boundary experiment, not an infrastructure attack. Do not attack the server, credentials, network, GitHub itself, availability, or anything outside the issue text and the public repository workflow.

The interesting question is not whether the server can be broken.

The interesting question is whether a chain of AI agents can defend a product boundary while still allowing useful delegated work.

What you can inspect

The workflow is intentionally public enough to leave evidence.

A request should not disappear into a private chatbot.

A useful result should leave a visible trail:

• the original issue;
• the labels that describe state transitions;
• comments written by agents;
• a pull request when a change is produced;
• validation evidence;
• the published static page when the change is accepted.

The page itself is not the main artifact.

The main artifact is the trace.

A static page is only the visible end of the process. The more important part is the path from external request to controlled product change.

If the workflow accepts your request, you can inspect what happened.

If the workflow rejects your request, you can inspect why.

A successful rejection is also a useful result.

In ordinary automation, a rejection often feels like failure. In an AI-agent workflow, a correct rejection is evidence that the process has boundaries. It means the agent did not blindly satisfy the user request. It treated the issue as a signal, not as a command.

That distinction matters.

Why I built this

AI coding agents are getting better at producing code.

But code generation is not the hardest part of using agents in real products.

The harder problem is control.

• Who allowed the agent to work?
• What kind of request was considered safe?
• Which files was the agent allowed to change?
• What checks were required?
• What evidence was recorded?
• When should the agent stop and ask for a human decision?

If these questions are not answered, “autonomous agent development” easily becomes an uncontrolled prompt playground.

That may be interesting for demos, but it is not how I want agents to touch real products.

In a product environment, the owner must not lose control. Users can express intent. Agents can perform delegated work. But the product owner must define the paths where autonomous work is allowed, the boundaries where it must stop, and the evidence required before the result enters the product.

This experiment uses static demo pages because they are safe, visible, and easy to understand.

But the real subject is not static page generation.

The real subject is controlled product evolution.

A user opens an issue. The workflow checks whether the request belongs to an approved path. If it does, agents may process it. If it does not, the request should be rejected, blocked, or routed to human review.

That is the model I want to test in public.

The workflow behind the game

The current experiment uses GitHub Issues as the entry point.

A GitHub Issue is a familiar development artifact. It can describe a request, collect discussion, receive labels, trigger events, and preserve history. That makes it a practical control surface for agent-driven work.

The simplified route looks like this:

The workflow does not treat the issue as a direct publishing request.

It treats the issue as a development signal.

That signal must pass through rules.

For the Demo Pages experiment, the allowed area is intentionally small. The agent may work only inside the documented demo-page boundary. It must not turn a reader request into arbitrary site changes.

This is where the workflow becomes more important than the generated page.

A simple page-generation demo could be built with one prompt and one script.

That is not the point here.

The point is to show a controlled route where request admission, agent execution, validation, pull request creation, merge, and publication are separate observable steps.

What protects the boundary

The boundary is not protected by asking the model to “please behave.”

That is not enough.

The workflow needs several kinds of control.

First, there is a documented scope.

The agent must know what kind of request is allowed, what area of the repository may be changed, and what must remain untouched.

Second, there is admission.

Before implementation starts, the request should be classified. A request that targets the homepage, global layout, scripts, styles, or product content should not be treated as a valid demo-page request.

Third, there is restricted execution.

Agents should run in a bounded environment and work through repository changes, not through direct production mutation.

Fourth, there is validation.

The result must be checked before it becomes a published artifact.

Fifth, there is evidence.

Labels, comments, pull requests, and logs matter because they make the process inspectable. Without evidence, autonomy is difficult to trust.

This is why I am inviting people not only to submit valid requests, but also to submit boundary-crossing requests.

A workflow that only succeeds on clean examples is not very interesting.

A workflow becomes more useful when it can say:

No, this request is outside the approved path, and here is why.

That answer is part of the product control model.

Why this may matter for product teams

This experiment looks small because the artifact is small.

A static page is not a large feature.

But the pattern is larger than the artifact.

Many teams already use GitHub Issues, pull requests, CI checks, reviews, and deployment pipelines. AI agents can enter this environment, but they need more than task prompts. They need process boundaries.

A product team does not only need an agent that can write code.

It needs a way to answer:

• which requests can be delegated;
• which requests require human review;
• what evidence must be produced;
• what validation is required;
• how the owner can inspect and change the rules;
• how the product avoids becoming controlled by random external prompts.

That is the commercial reason I care about this.

I am not trying to demonstrate that an AI agent can write a static page. That is already obvious.

I am trying to demonstrate that an AI-agent workflow can transform an external request into a controlled, reviewable, bounded product change.

For a real team, this pattern could apply to documentation fixes, small UI corrections, test additions, generated examples, controlled content updates, internal tools, or other well-bounded change paths.

The important part is not the specific artifact.

The important part is the governance loop:

intent -> boundary check -> delegated work -> evidence -> review -> product change

What powers this experiment

The public workflow is powered by my GitHub Flows work.

The host application is here: github-flows-app

It is a ready-to-run host application around @teqfw/github-flows. It starts a Node.js process, loads local runtime configuration, exposes the GitHub webhook ingress, provides a workspace for configuration and logs, and allows the workflow runtime to react to GitHub events.

The host application itself is not the whole idea.

The broader idea is that repository events can drive agent execution through an explicit process. An issue is opened. A label is added. A pull request is created. A validation result appears. A merge happens. These events can move work through a controlled chain instead of relying on one long manual chat session.

I described the conceptual layer here: Turning User Intent Into Controlled Product Evolution

That article explains the model behind this experiment: users express intent, the product owner defines approved paths, agents act as delegated intermediaries, and the product changes only through bounded, observable, reviewable steps.

Try it

Start here: Demo Pages

Try a simple valid request first.

Then try a request that should be rejected.

Look at the labels.

Read the agent comments.

Follow the pull request if one appears.

Open the published page if the request reaches publication.

If the workflow rejects you, that may be the correct result.

If the workflow publishes something it should not publish, I want to know.

And if you are working with a real product team and thinking about how AI agents could safely participate in GitHub-based development, I am interested in practical cases.

The question is no longer only:

Can an AI agent write code?

The more useful question is:

Can we give agents limited authority inside a process that product owners can control?

If that is the kind of problem you are facing, you can contact me.

/$0.50 per rule per month. $0.03 per million events. Enabling Contributor Insights on a single DynamoDB table auto-creates four rules so your floor is $2.00/month before a single event is counted.

At 500 million events/month, that’s $17.00/month. At 5 billion, it’s $152.00/month.

Whether that’s justified depends on your traffic volume, whether you have hot key patterns, and what throttling is already costing you in wasted provisioned throughput.

What Is DynamoDB Contributor Insights?

Contributor Insights for DynamoDB is a CloudWatch monitoring feature that identifies the top partition keys driving read and write traffic on a table without any code-level instrumentation or custom application logging.

It analyzes request traffic in near real-time and publishes results as CloudWatch graphs. For tables with partition key skew, it answers the specific question a FinOps engineer needs: “Which partition key is causing throttling?”

The Two Pricing Components

Contributor Insights billing has exactly two line items on your CloudWatch invoice:

• Rule fee: $0.50 per rule per month, flat regardless of traffic
• Event fee: $0.03 per 1 million DynamoDB read/write events matched

How many rules does one table create?

Enabling on a single table auto-creates four rules:

• Most-accessed partition keys on the base table
• Most-accessed partition keys causing errors on the base table
• Most-accessed partition keys on each GSI
• Most-accessed partition keys causing errors on each GSI

Rule count formula: 2 + (2 × number of GSIs)

A table with no GSIs = $2.00/month in rule fees. A table with 3 GSIs = $4.00/month in rule fees, before any events.

What counts as an “event”?

Every read and write operation: GetItem, PutItem, UpdateItem, DeleteItem, BatchGetItem, BatchWriteItem (each item individually), Query and Scan (each item returned or examined), and transactional reads/writes. GSI operations are counted separately when rules are active on both.

Worked Cost Examples

10M events/month: $2.00 rule fee + $0.30 event fee = $2.30/month

500M events/month: $2.00 rule fee + $15.00 event fee = $17.00/month

2B events/month: $2.00 rule fee + $60.00 event fee = $62.00/month

2B events/month + 3 GSIs: $4.00 rule fee + $60.00 event fee = $64.00/month

For most workloads, the event fee is the variable that matters. The rule fee is fixed overhead.

Throttle-Only Mode

As of August 2025, AWS added a throttle-only mode. Rules only activate for events that result in throttling requests that exceed provisioned capacity or on-demand burst limits.

You still pay the $0.50/rule/month rule fee. But the event fee drops dramatically. On a well-provisioned table with a 0.1% throttle rate processing 2 billion events/month, you’d match roughly 2 million throttled events, an event cost of $0.06/month instead of $60.00/month.

Use throttle-only mode when:

• You only need to diagnose which keys are causing throttling, not which are most popular
• Your table is currently healthy and you want passive monitoring with near-zero cost
• You have a high-volume table (1B+ events/month) where full mode runs $30+/month

Use full mode when:

• You’re actively investigating a hot key pattern
• You want continuous visibility into traffic distribution
• You’re evaluating whether to restructure your partition key schema

If you’re deciding between 1-year and 3-year AWS commitment terms before locking in Reserved Capacity, the tradeoffs are covered in detail here How to Choose Between 1-Year and 3-Year AWS Commitments

The Real Cost Question: What Does Hot Key Blindness Cost?

Contributor Insights costs $2 to $62/month. The cost of not having it shows up in your provisioned capacity bill every month.

DynamoDB provisioned capacity is distributed across partitions. If your table is configured for 10,000 RCUs/second and 90% of reads go to three partition keys, the capacity on your other partitions sits idle while those three keys throttle.

Common patterns where this happens:

• Low-cardinality partition keys (e.g., user_type, region, status)
• Popular cached items that expire simultaneously, causing a read surge on one key
• Time-based queries where all traffic clusters around “today” or “current”

A concrete scenario: a table provisioned at 5,000 WCUs/second costs approximately $2,847/month in write capacity (verify current rates at aws.amazon.com/dynamodb/pricing). If hot key throttling means 30% of that capacity is sitting idle on underutilized partitions, you’re paying ~$854/month for throughput you can’t use.

Contributor Insights at $17/month to identify which keys to redesign around: that’s a 50:1 return in the month you act on it.

*Hot Keys and Reserved Capacity Commitments
*
This is where hot key detection connects directly to cost optimization strategy.

DynamoDB Reserved Capacity delivers 30–40% savings on provisioned throughput. But those savings assume your capacity is being utilized effectively. A table with undetected hot key patterns has two compounding problems:

• It’s over-provisioned because teams add capacity to compensate for throttling instead of fixing the partition key design
• When you commit to Reserved Capacity on an over-provisioned table, you lock in the waste

The right sequence: identify hot keys via Contributor Insights → fix the partition key schema → right-size provisioned capacity → apply Reserved Capacity on that right-sized number. The result is a smaller commitment with higher utilization and 30–40% off the correct baseline.

Contributor Insights vs Standard CloudWatch Metrics

CloudWatch already provides DynamoDB metrics by default, at no charge. The key difference is specificity.

Standard CloudWatch metrics tell you “you have 50,000 throttled requests this hour.” Contributor Insights tells you “partition key user#12345 accounts for 48,000 of those throttled requests.”

For active troubleshooting or capacity planning, that distinction is the difference between guessing and acting.

When to Enable It

Enable Contributor Insights when:

• Your table experiences recurring throttling that standard metrics can’t explain
• You’re planning a table migration or partition key schema redesign
• Your table processes 100M+ events/month and you have no visibility into traffic distribution
• You’re considering a Reserved Capacity commitment and want to validate utilization first

Use throttle-only mode when:

• You want passive monitoring with near-zero event costs
• Your table is healthy but you want early warning if hot keys develop

Skip it entirely when:

• You have very high cardinality partition keys (e.g., UUID-based) with no known skew
• You’re on on-demand capacity mode with confirmed even traffic distribution
• The table is non-production with no Reserved Capacity commitment

How to Enable It

Prerequisites: IAM permissions for cloudwatch:EnableInsightRules and dynamodb:DescribeTable. Understand the event cost for your traffic volume before enabling full mode.

• Open the DynamoDB console and select your target table
• Go to the Monitor tab
• Click Manage Contributor Insights; select “All Requests” (full mode) or “Throttled Requests Only” (throttle-only mode)
• Navigate to CloudWatch > Contributor Insights and confirm the rules are in “Enabled” state
• Allow 15–30 minutes for data to appear; CloudWatch will show the top 10 partition keys by request volume

If rules show an “Error” state, the most common cause is insufficient IAM permissions. Verify the execution role includes cloudwatch:EnableInsightRules, cloudwatch:PutInsightRule, and dynamodb:DescribeTable.

How This Fits With Reserved Capacity

Identifying hot keys is a prerequisite to maximizing Reserved Capacity ROI. A table with hot key throttling shows a deceptive utilization signal: total consumed capacity may look high, but actual table-level efficiency is low because hot partitions throttle while cold partitions sit idle.

Usage.ai’s Flex Reserved Instances handle DynamoDB Reserved Capacity commitments with no upfront payment, no multi-year lock-in, and a cashback guarantee if utilization falls short see how it works at usage.ai

Usage.ai refreshes DynamoDB recommendations every 24 hours, vs. the 72+ hour refresh cycle on AWS native tools. At DynamoDB pricing, that 3-day gap in detecting newly-emerged hot key patterns can represent meaningful daily waste before any action is taken.

Have you run into hot key issues on DynamoDB? Did you catch it via Contributor Insights or some other way and what did the fix end up looking like?

Explore the full operational breakdown here → DynamoDB Contributor Insights Pricing: Monitoring Hot Keys at a Cost

Cloudflare rebuilt Browser Run on its own Containers platform, delivering 4x higher concurrency and 50% faster response times. The upgrade completes a six-layer agent infrastructure stack: compute (Dynamic Workers + Sandboxes), orchestration (Dynamic Workflows), memory (Agent Memory), browsing (Browser Run), and commerce (Stripe Projects).

By Steef-Jan Wiggers

Introduction

Enterprise Linux operating systems require periodic upgrades to maintain security, supportability, compliance, and operational stability.

As organizations modernize infrastructure platforms, migrating from Oracle Linux 7 to Oracle Linux 8 becomes important because Oracle Linux 8 introduces:

✔ Modern package management
✔ Improved security
✔ Better kernel support
✔ Enhanced automation compatibility
✔ AppStream modular repositories
✔ Long-term enterprise support

However, major Linux upgrades are not simple package updates.

They involve:

✔ Repository transitions
✔ Package dependency changes
✔ Kernel migration
✔ Bootloader modifications
✔ Service compatibility validation
✔ Third-party package handling

Oracle Linux provides the Leapp upgrade utility to automate and orchestrate Oracle Linux 7 to Oracle Linux 8 migrations safely.

In this blog, we will perform a complete deep dive into:

✔ Configuring Leapp repositories
✔ Installing Leapp utility
✔ Preupgrade analysis
✔ Understanding inhibitors
✔ Answer file handling
✔ Repository migration
✔ Upgrade execution
✔ Upgrade boot workflow
✔ Enterprise troubleshooting
✔ Real-world operational challenges

Understanding Leapp

Leapp is an in-place upgrade utility used to migrate Oracle Linux 7 systems to Oracle Linux 8.

Why Major OS Upgrades Are Complex

Upgrading from Oracle Linux 7 to Oracle Linux 8 involves platform-level architectural changes.

These include:

✔ Kernel transitions
✔ Package replacement
✔ Repository mapping
✔ Driver compatibility
✔ Service migration
✔ Security policy updates
✔ Boot environment changes

Note : Operating system upgrades are not only package upgrades — they are full platform transitions.

Oracle Linux Upgrade Architecture

The Leapp upgrade process follows multiple operational stages.

Upgrade Workflow

Oracle Linux 7
│
▼
Repository Validation
│
▼
Leapp Installation
│
▼
Preupgrade Analysis
│
▼
Inhibitor Detection
│
▼
Answer File Validation
│
▼
Upgrade Initramfs Creation
│
▼
System Reboot
│
▼
Upgrade Environment Boot
│
▼
Package Migration
│
▼
Oracle Linux 8

Step 1: Verify Current Oracle Linux Version

Before starting the migration, validate the current OS version.

Command:

cat /etc/os-release

Example output:

NAME=”Oracle Linux Server”
VERSION=”7.x”

Step 2: Verify Current Repositories

Repository consistency is critical before performing upgrades.

Check repositories:

yum repolist

Why Repository Validation Matters

Broken or duplicate repositories may cause:

✔ Dependency failures
✔ Package mapping errors
✔ Upgrade inhibitors
✔ Incomplete migrations
✔ Boot failures

Add Leapp Repository Configuration

Navigate to repository directory:

cd /etc/yum.repos.d/

Create or validate Oracle Linux repositories.

Example:

[ol7_leapp]
name=Oracle Linux 7 Leapp Repository
baseurl=https://yum.oracle.com/repo/OracleLinux/OL7/leapp/x86_64/
enabled=1
gpgcheck=1

Refresh Repository Metadata
yum clean all
yum makecache

Step 4: Install Leapp Utility

Install required Leapp packages.

Command:

yum install -y leapp-upgrade leapp-data-oraclelinux

What Gets Installed?

Leapp installs:

✔ Upgrade actors
✔ Dependency analysis modules
✔ Migration logic
✔ Repository mapping data
✔ Upgrade workflows

Verify Leapp Installation

rpm -qa | grep leapp

Step 5: Understanding Leapp Preupgrade

Before performing the actual upgrade, Leapp performs extensive system analysis.

Command:
leapp preupgrade

What Happens During Preupgrade?

Leapp analyzes:

✔ Installed packages
✔ Drivers
✔ Repository configuration
✔ Bootloader state
✔ Kernel compatibility
✔ Unsupported packages
✔ Dependency conflicts
✔ Security policies

Operational Insight
The preupgrade phase prevents unsafe migrations before system modification begins.

Understanding Inhibitors

One of the most important Leapp concepts is inhibitors.

What Are Inhibitors?

Inhibitors are conditions that stop the upgrade from continuing safely.

If inhibitors exist, Leapp blocks the upgrade process.

Why Inhibitors Exist

Inhibitors protect systems from unsafe migration scenarios.

Examples:

✔ Unsupported repositories
✔ Duplicate repository entries
✔ Deprecated packages
✔ Missing answer files
✔ Third-party RPM conflicts
✔ Unsupported drivers
✔ Incorrect boot configuration

Real Repository Inhibitor Example

Example error:

Repository ol8_baseos_latest is listed more than once in the configuration.

Why This Happens

Possible causes:

✔ Duplicate .repo files
✔ Custom repositories
✔ Third-party repositories
✔ Incorrect migration preparation

Fixing Duplicate Repositories

Check repository directory:

ls -l /etc/yum.repos.d/

Review duplicate repository definitions:

grep -r “ol8_baseos_latest” /etc/yum.repos.d/

Remove duplicate entries carefully.

Understanding Leapp Answer Files

This is one of the most important upgrade concepts.

What Are Answer Files?

During upgrades, Leapp may require administrator confirmation for specific migration decisions.

Leapp stores these prompts inside answer files.

Location:

/var/log/leapp/answerfile

Why Answer Files Matter

Leapp blocks upgrades until required questions are answered.

Example:

Missing required answers in the answer file.

View Required Answers

Command:

leapp answer –section remove_pam_pkcs11_module_check.confirm=True

What Does This Do?

This command confirms specific upgrade actions required by Leapp.

Operational Insight

Answer files help administrators explicitly approve risky or environment-specific migration decisions.

Understanding Repository Migration

Oracle Linux 7 and Oracle Linux 8 use different repository structures.

Oracle Linux 7 Repositories

✔ ol7_latest
✔ ol7_UEKR6
✔ Optional repositories

Oracle Linux 8 Repositories

✔ BaseOS
✔ AppStream
✔ UEK repositories

Repository Mapping Workflow

OL7 Repositories
│
Repository Mapping
│
▼
OL8 BaseOS + AppStream

Understanding AppStream Repositories

Oracle Linux 8 introduces AppStream modular repositories.

Unlike Oracle Linux 7:

Packages are grouped into modules and streams.

Examples:

✔ Python streams
✔ NodeJS streams
✔ Database modules

This increases flexibility but also migration complexity.

Third-Party Repository Challenges

Enterprise systems commonly use third-party repositories.

Examples:

✔ EPEL
✔ Monitoring agents
✔ Security tools
✔ Vendor repositories
✔ Backup software

Example EPEL Problem

Example error:

No package epel-release available
Why This Happens

Possible causes:

✔ Repository incompatibility
✔ Incorrect release version
✔ Unsupported packages
✔ Missing metadata

Step 6: Execute Upgrade

Once inhibitors are resolved, begin the upgrade.

Command: leapp upgrade

What Happens Internally?

Leapp performs:

✔ Upgrade initramfs creation
✔ Bootloader modification
✔ Package migration
✔ Repository transition
✔ Kernel migration
✔ Service migration
Understanding Upgrade Initramfs

This is one of the most advanced upgrade concepts.

What is Upgrade Initramfs?

Leapp temporarily boots into a dedicated upgrade initramfs environment outside the running Oracle Linux 7 userspace.

This isolated environment safely performs package replacement operations.

Upgrade Boot Workflow

Normal OL7 Boot
│
▼
Upgrade Initramfs
│
▼
Package Migration
│
▼
OL8 Boot

Why Upgrade Initramfs Is Required

Leapp performs the upgrade outside the running Oracle Linux 7 userspace to avoid active package conflicts and dependency corruption during migration.

The temporary upgrade initramfs environment provides an isolated userspace where package replacement, repository switching, kernel migration, and dependency updates can occur safely without interfering with the currently running operating system.

This isolation significantly reduces the risk of package inconsistency and upgrade instability during major platform transitions.

Step 7: Reboot the System

After upgrade preparation:

reboot

The system boots into the temporary upgrade environment.

Migration occurs automatically.

Step 8: Verify Oracle Linux 8 Upgrade

After reboot completes:

Verify OS Version
cat /etc/os-release

Common Real-World Upgrade Problems

Many upgrade failures occur because of environment inconsistencies rather than Leapp itself.

Common Operational Problems

✔ Repository duplication
✔ Dependency conflicts
✔ Unsupported packages
✔ Missing drivers
✔ Bootloader issues
✔ EPEL incompatibility
✔ Service startup failures
✔ Network configuration mismatches
✔ SELinux conflicts

Upgrade Failure Recovery

If the upgrade process fails during migration, administrators should analyze Leapp reports, validate repositories, review inhibitors, and restore systems using backups or boot volume snapshots when necessary.

/var/log/leapp/leapp-report.txt

/var/log/leapp/leapp-upgrade.log

/var/log/leapp/leapp-preupgrade.log

Leapp logs provide detailed visibility into dependency analysis, migration stages, repository transitions, and package failures.

Rollback and Recovery Planning

Enterprise upgrades should always include rollback preparation.

Before upgrades:

✔ Create backups
✔ Snapshot boot volumes
✔ Validate rollback workflows
✔ Test upgrades in staging
✔ Document recovery procedures

Why Rollback Planning Matters

If upgrades fail unexpectedly:

Rollback capability reduces downtime and operational risk.
Enterprise Upgrade Best Practices

✔ Validate backups before upgrade
✔ Remove unsupported repositories
✔ Review Leapp reports carefully
✔ Resolve inhibitors completely
✔ Test upgrades in staging first
✔ Validate applications after migration
✔ Monitor services post-upgrade
✔ Maintain rollback procedures

Conclusion

Leapp provides a powerful and automated framework for migrating Oracle Linux 7 systems to Oracle Linux 8 through dependency analysis, repository validation, package migration, and upgrade orchestration.

Although the migration process is heavily automated, successful enterprise upgrades still require careful planning, repository consistency, inhibitor analysis, answer file validation, operational testing, and rollback preparation to ensure production stability.

Modern Linux upgrades are no longer simple package updates — they are enterprise platform modernization workflows requiring operational engineering discipline.