Yapay Zeka

Organizing the Use Cases of AWS Step Functions and Glue Workflow for ETL Processing with AWS Glue Jobs

yazar

Original Japanese article: Glue JobのETL処理におけるAWS Step FunctionsとGlue Workflowの使い分けを整理する

Introduction

I’m Aki, an AWS Community Builder (@jitepengin).

When building data pipelines on AWS, deciding which workflow orchestration tool to use is an important architectural decision.
Especially when designing ETL pipelines centered around Glue Jobs, the following two services are commonly compared:

  • AWS Step Functions
  • AWS Glue Workflow

Both services are workflow orchestration tools that manage job dependencies and execute processes sequentially.
However, since they are designed with different strengths and philosophies, choosing the right one based on your requirements is critical.

In this article, I’ll organize the characteristics, advantages, disadvantages, and use cases of these two services, and discuss how to decide which one to choose.

Personally, I really like Glue Workflow, but I’ve had fewer opportunities to use it recently, which is honestly a bit disappointing.
I also like Glue Python Shell, but compared to standard Glue Jobs, I rarely get to use it these days either…

Major Workflow Orchestration Tools

AWS provides multiple workflow orchestration services, but the two most commonly compared for Glue Job–based data pipelines are the following:

Service Overview
AWS Step Functions AWS managed workflow orchestration service. Can flexibly orchestrate a wide range of AWS services such as Lambda, Glue, ECS, and more
AWS Glue Workflow Glue-native workflow feature. Defines pipelines using Glue Jobs, Crawlers, and Triggers

AWS also offers Amazon Managed Workflows for Apache Airflow (MWAA).
MWAA becomes a strong option when you need highly complex dependency management or cross-cloud orchestration. However, in this article, I’ll focus specifically on comparing Step Functions and Glue Workflow.

Recently, many teams have also adopted workflow tools such as Airflow, Dagster, and Prefect. As always, selecting the right tool depends heavily on your requirements and goals.

AWS Step Functions

Overview

(Figure above: Visual workflow definition using Workflow Studio)

AWS Step Functions is a workflow orchestration service that defines workflows using Amazon States Language (ASL), a JSON/YAML-based language.

It integrates with over 200 AWS services including Lambda, Glue, ECS, SNS, and DynamoDB, and supports flexible workflow controls such as conditional branching, parallel execution, and retry handling.

Features

Step Functions provides two workflow types:

Type Characteristics
Standard Workflow Long-running execution, audit logs, exactly-once execution guarantee
Express Workflow High throughput, low cost, optimized for short-lived processing (at-least-once)

In the context of data pipelines, Standard Workflow is generally the more common choice.

Step Functions also provides AWS Step Functions Workflow Studio, a visual editor that allows workflows to be built through a GUI.

Advantages

  • Multi-service integration: Easily integrates Glue with other AWS services such as Lambda, ECS, SNS, and more
  • Flexible control flow: Supports conditional branching (Choice), parallel execution (Parallel/Map), and advanced error handling (Catch/Retry)
  • High observability: Execution history and per-step states can be visually inspected in the console
  • Event-driven integration: Easily triggered through EventBridge using S3 uploads or schedules
  • Parallel execution support (Map): Well suited for large-scale processing such as file-level or partition-level parallel execution. Distributed Map is especially useful for high-scale parallel workloads
  • Infrastructure as Code support: Can be fully managed through CloudFormation, CDK, or Terraform

Disadvantages

  • Learning curve: Requires understanding ASL, and complex workflows can become verbose
  • Cost: Standard Workflow pricing is based on state transitions, so costs can increase with larger workflows
  • Glue integration setup: IAM roles, parameter passing, and Glue Job integration must be configured manually

Use Cases

Step Functions is a strong fit for scenarios such as:

  • Hybrid pipelines combining Glue with Lambda, ECS, or other AWS services
  • Complex workflows requiring branching, parallel processing, or dynamic parameter passing
  • Pipelines that require SNS notifications or compensating actions on failures
  • Teams managing infrastructure strictly through IaC
  • Large-scale data platforms involving multiple teams

AWS Glue Workflow

Overview

(Figure above: Visual workflow definition using Glue Workflow)

AWS Glue Workflow is Glue’s native workflow orchestration feature.

It defines pipelines using three primary components:

  • Glue Jobs
  • Glue Crawlers
  • Glue Triggers

Pipelines can be configured directly from the Glue console GUI, making it easy to build Glue-centric ETL workflows quickly.

Features

The major components of Glue Workflow are as follows:

Component Role
Glue Job Actual ETL processing using Spark or Python Shell
Glue Crawler Scans data sources such as S3 and registers table metadata into the Data Catalog
Glue Trigger Defines execution conditions such as schedules, events, or conditional dependencies

Glue Triggers support three types:

  • SCHEDULED
  • ON_DEMAND
  • CONDITIONAL

This allows chained execution patterns, such as triggering downstream jobs based on upstream job success or failure.

Advantages

  • Native Glue integration: Seamlessly integrates with Glue Jobs and Crawlers with minimal additional setup
  • Simple configuration: DAG-style workflows can be built intuitively through the console GUI
  • Low cost: No additional charge for the workflow itself beyond Glue Job execution costs
  • Easy crawler integration: Natural fit for workflows that update the Data Catalog after ETL execution
  • Glue Data Catalog integration: Job execution metadata and lineage can be managed centrally within Glue

Disadvantages

  • Glue-only orchestration: Cannot directly orchestrate Lambda, ECS, or other non-Glue services
  • Limited event-driven capabilities: Primarily Trigger- and schedule-based, making advanced event integration less flexible than Step Functions
  • Limited control flow: Weak support for advanced branching and dynamic parameter handling
  • Observability limitations: Detailed execution logs often require separate CloudWatch investigation
  • More difficult IaC management: CloudFormation and Terraform management can become cumbersome compared to Step Functions
  • Limited parallel execution control: Not ideal for fine-grained parallelization or Map-style orchestration
  • Weaker retry/re-execution control: Re-running only failed portions of a workflow is less flexible than in Step Functions

Use Cases

Glue Workflow is well suited for:

  • Simple ETL pipelines composed entirely of Glue Jobs and Crawlers
  • Periodic ingestion pipelines into S3-based data lakes with automatic catalog updates
  • Small teams or early-stage projects that need fast implementation
  • Organizations that prefer operating primarily through the Glue console

Which One Should You Choose?

Based on the characteristics of both services, the selection criteria can generally be summarized as follows.

When to Choose Glue Workflow

  • Your ETL pipeline consists only of Glue Jobs and Crawlers
  • Simple sequential execution and conditional triggers are sufficient
  • You want to build quickly (prototypes or small-scale projects)
  • Your operations are centered around the Glue console
  • You want to minimize costs

When to Choose Step Functions

  • You need integration with Lambda, ECS, or other AWS services
  • You require branching, parallel processing, or advanced error handling
  • You are adopting an EventBridge-centric event-driven architecture
  • You want strict Infrastructure as Code management
  • Multiple teams are involved in operating the data platform
  • Observability and audit logging are important

Summary of Decision Criteria

Perspective Glue Workflow Step Functions
Supported Services Glue only Broad AWS integration
Control Flow Simple Flexible and advanced
Observability △ ◎
Ease of Configuration ◎ △ (learning curve)
Cost Low Depends on state transitions
IaC Management △ ◎
Crawler Integration ◎ △ (manual setup required)

Glue Workflow is fundamentally Trigger-oriented and is not designed for advanced event orchestration like Step Functions.

Unless Glue Workflow specifically satisfies your requirements better, choosing Step Functions is generally the safer long-term option.
Personally, when designing architectures, I often start by considering Step Functions first.

That said, Glue Workflow remains a strong choice when the requirement is simply:

“I want to quickly build a Glue-centric ETL pipeline.”

Combining Both Services

Instead of choosing one or the other exclusively, it is also possible to invoke Glue Workflow from Step Functions.

For example, Step Functions can handle preprocessing and postprocessing with Lambda, while delegating the ETL core to Glue Workflow.

However, this introduces additional complexity because workflow state coordination between the two services must be managed carefully.
If simplicity is important, standardizing on one orchestration tool is generally easier operationally.

Caveats When Invoking Glue Workflow from Step Functions

1. Completion Detection Requires Polling

Step Functions provides a convenient .sync integration pattern for Glue: StartJobRun, which waits for job completion automatically.

However, Glue: StartWorkflowRun does not support the .sync integration pattern.

While you can invoke Glue Workflow through SDK integration, Step Functions will immediately proceed to the next state without waiting for completion.
As a result, you must implement custom polling logic to repeatedly check the WorkflowRun status.

2. Polling Logic Becomes Complex

You typically need to implement a loop like:

  • Wait
  • GetWorkflowRun
  • Choice (RUNNING / COMPLETED / FAILED)

This increases both the number of states and the verbosity of the ASL definition.

3. Error Handling Becomes More Complicated

Glue Workflow status is returned at the WorkflowRun level rather than the individual Job level.

As a result, identifying which specific Glue Job failed requires additional parsing logic against the GetWorkflowRun response.

Because of these complications, although combining both services is technically possible, I generally recommend avoiding Step Functions → Glue Workflow orchestration unless there is a compelling reason.

One possible use case is when you need to extend an existing Glue Workflow–based system incrementally.
Even then, rebuilding the orchestration directly in Step Functions using existing Glue Jobs often feels cleaner.

Migration Considerations

Some teams initially adopt Glue Workflow during the early stages of a project and later migrate to Step Functions as the data platform grows.

When migrating, the following considerations become important:

  • Workflow definitions must be rewritten: Glue Trigger definitions need to be converted into ASL
  • IAM roles must be redesigned: Step Functions requires permissions to invoke Glue Jobs
  • Extensive testing is necessary: Existing jobs must be validated carefully after migration

Migration is certainly possible, but it is not trivial.
This is why it is important to consider future extensibility and maintainability from the beginning when selecting your orchestration tool.

Conclusion

In this article, I organized the differences between AWS Step Functions and Glue Workflow for ETL orchestration.

To summarize:

  • Glue Workflow: Glue-native, simple, low-cost, and ideal for rapidly building straightforward ETL pipelines
  • Step Functions: Better suited for multi-service orchestration, advanced workflow control, observability, and large-scale pipelines

Neither service is universally “better.”
The right choice depends on your use cases, organizational structure, operational requirements, and future scalability needs.

For small-scale ETL pipelines, Glue Workflow is often sufficient. However, as data platforms evolve, requirements such as exception handling, notifications, conditional branching, and integrations with other services tend to grow over time. In many cases, architectures gradually move toward Step Functions as complexity increases.

A practical strategy can be to start simple with Glue Workflow during the early stages, and later migrate to Step Functions when requirements become more sophisticated.

That said, considering future migration costs, building on Step Functions from the beginning can also be a very reasonable approach.

I hope this article helps anyone currently evaluating workflow orchestration options on AWS.

The Console Is the Shadow Control Plane

yazar
Rack2Cloud - Authority Layer Series

Rack2Cloud - Authority Layer Series
Most organizations believe they have one infrastructure control plane. They have two.

The declared control plane has policy gates, approval workflows, branch protections, and an audit trail that connects change to intent. The operational control plane has a browser and a credential. Both mutate production state. Only one of them is governed.

That gap — between the infrastructure authority you designed and the infrastructure authority that runs your environment — is the shadow control plane problem. It is not a tooling failure. It is not an operator discipline failure. It is an authority topology problem: modern infrastructure environments rarely operate through a single governance system. They operate through two competing ones simultaneously, and the ungoverned one has been winning for years.

shadow control plane — two competing infrastructure authority systems operating simultaneously

What a Shadow Control Plane Actually Is

The term shadow control plane is often used to mean “people clicking in the console when they shouldn’t be.” That framing is wrong, and it leads to the wrong solutions.

A shadow control plane is any execution path that retains full infrastructure authority while bypassing the declared control plane’s governance layer. The emphasis is on retaining full authority. This is not a restricted path, a read-only viewer, or a monitoring interface. It is a fully operational execution environment — provisioning, modifying, deleting, reconfiguring — with no mandatory policy enforcement, no approval gate, no blast radius boundary, and no audit trail linking the change to an approved intent.

The cloud console is the most visible instance of this pattern. But it is not the only one. The CLI running from a local workstation with production credentials is a shadow control plane. A SaaS integration writing directly to cloud APIs outside the pipeline path is a shadow control plane. An AI agent with infrastructure credentials operating outside declared governance mediation is a shadow control plane.

The defining characteristic is not the interface. It is the absence of governance mediation between the execution authority and the infrastructure it can reach.

Why Operations Falls Back to the Console

The shadow control plane does not grow because engineers are careless. It grows because operations trusts it more during failure — and in many cases, that trust is operationally justified.

During a major incident, the pipeline is often the wrong tool for recovery. Approval workflows are unavailable at 2am. Policy engines block changes that don’t match pre-declared patterns — exactly the kind of changes an incident requires. The IaC repository may not reflect current runtime state, because drift has accumulated since the last apply. Terraform plan output during an active incident can be actively misleading — showing changes against a declared state that no longer matches reality.

The console, by contrast, shows what is actually running. It allows direct intervention against the real state of the environment, without waiting for a pipeline trigger, a reviewer, or an approval queue to clear. During major incidents, the console often reflects operational reality more accurately than the IaC repository does. That is not a criticism of IaC. It is a description of what happens to state under failure conditions, and why operators reach for the tool that reflects reality rather than the tool that reflects intent.

This is the birth pattern of the shadow control plane: 

Incident occurs
↓
Console change restores service
↓
Nobody reconciles the change
↓
IaC repository diverges permanently
↓
Next terraform apply becomes dangerous
↓
Pipeline trust erodes further
↓
Console usage increases

Each incident that goes unreconciled makes the declared control plane less reliable as a representation of actual state — and makes the shadow control plane more operationally rational as a result. The problem compounds itself.

The key insight: the shadow control plane grows wherever operational urgency exceeds governance friction.

The Execution Authority Gap

Pipelines govern intent. Consoles govern capability.

That contrast is the architecture problem stated precisely. Map what each path requires to execute an identical change:

Execution path Policy check Approval gate Blast radius analysis Audit trail (change to intent)
CI/CD pipeline ✅ ✅ ✅ ✅
Cloud console ❌ ❌ ❌ ❌
CLI (local) ❌ ❌ ❌ ❌
SaaS integration Varies ❌ ❌ Rarely
AI agent (ungoverned) ❌ ❌ ❌ ❌

The Execution Authority Gap is the delta between the pipeline row and every other row. The pipeline is the only execution path that carries governance all the way through. Every other path retains full execution authority while dropping the governance layer.

shadow control plane execution authority gap — pipeline vs console vs CLI vs AI agent governance comparison

Machine-Scale Shadow Control Planes

Console drift is human-scale. One operator, one session, one set of changes.

The real exposure in 2026 is system-scale.

Infrastructure mutations are increasingly performed by systems operating entirely outside the declared governance path — at machine speed, without human review, continuously. The problem is not automation. Automation with governance mediation is precisely what the CI/CD control plane is designed to provide. The problem is autonomous mutation authority without reconciliation or intent validation.

The systems introducing machine-scale shadow control plane authority:

GitOps controllers — continuous reconciliation loops that enforce repository state, not organizational intent. The governance gap is upstream of the controller.

Terraform Cloud and remote execution platforms — runs triggered outside the pipeline path, with production credentials, bypassing branch protection and approval workflows.

CSP-native auto-remediation — AWS Config rules, Azure Policy remediations, GCP Security Command Center automated responses all write to infrastructure state outside the organization’s declared change authority model.

Security orchestration platforms — SOAR workflows that modify infrastructure in response to detections operate outside the pipeline entirely. The change is correct. The governance path is absent.

AI agents with infrastructure credentials — the most significant emerging category. An agent that can invoke cloud APIs, execute Terraform, or modify network configuration holds infrastructure mutation authority at inference speed, without governance mediation.

The distinction: not automation vs. no automation. Automation with governance mediation vs. autonomous mutation authority without reconciliation or intent validation.

machine-scale shadow control plane — autonomous infrastructure mutation authority outside governance mediation

The Audit Trail Is Not the Approval Trail

Forensics is not governance.

Audit logs record who changed something. They do not record why, under what authority, against what approved intent, or with what blast radius analysis. The audit trail creates post-event visibility. Governance requires pre-change authority control.

An organization that can reconstruct exactly what happened after a breach has a forensics capability. An organization that prevented unauthorized changes from reaching production has a governance capability. CloudTrail supports the first. It does not constitute the second.

⚠ Common mistake: Expanding log retention and improving log query tooling improves forensics. It does not close the Execution Authority Gap. The console change that caused the outage is in the audit log — the problem was that it could be made without governance mediation, not that it couldn’t be found afterward.

The Pipeline Became Documentation. The Console Became Operations.

This is the steady state for most organizations that have been operating long enough.

The IaC repository was supposed to be the authoritative representation of infrastructure state. For many teams it has become something different: a record of intended state at the time the last deployment ran, which may or may not reflect what is actually running in production.

Each unreconciled console change adds one more divergence between declared state and actual state. Each emergency fix that stays in production adds one more dependency the IaC repository doesn’t know about. Each SaaS integration that writes to cloud APIs adds one more execution path outside governance mediation.

The IaC repository becomes increasingly dangerous to apply at full scope — because applying it would overwrite operational changes that production depends on. So teams begin scoping applies more narrowly, running targeted modules, avoiding full-environment plans. The declared control plane retreats. The shadow control plane advances.

This is not a Terraform problem or an operator discipline problem. It is the natural trajectory of any governance system that cannot operate at the speed of operational reality.

shadow control plane authority drift — IaC declared state diverging from production reality over time

What Shadow Control Plane Activity Looks Like

These are not random technical messes. They are authority artifacts — evidence of uncontrolled execution authority accumulated over time.

IAM sprawl — roles and permissions created for operational needs and never removed. Each represents an authorization decision made outside the declared governance model.

Security group entropy — rules added during incidents or by individuals with console access, never reconciled into IaC. The effective network policy diverges from the declared network policy.

Orphaned DNS records — console changes that were never reflected in IaC, pointing at infrastructure that no longer exists or whose ownership is unknown.

Undocumented routing exceptions — route table entries, VPC peering connections, transit gateway attachments added outside the pipeline path.

Hidden egress paths — NAT gateway configurations, internet gateway attachments, and service endpoint policies modified outside governance mediation.

Policy drift — resource-level policies and permission boundaries modified from their declared configuration.

All share the same root cause: execution authority that reached production without passing through the declared governance model.

Reducing Uncontrolled Execution Authority

The goal is not to ban the console. The goal is to reduce execution authority that reaches production without governance mediation.

First principle: if governance is slower than operational recovery requirements, the shadow control plane will always win. Operations routes around governance systems that cannot operate at incident speed. Governance latency is an architecture problem, not a culture problem.

Make the pipeline mandatory for categories that matter. Tier changes by risk. Security group modifications, IAM changes, network topology changes, and credential rotations should be pipeline-mandatory with SCP and IAM permission boundary enforcement. Match governance friction to change risk.

SCPs and IAM permission boundaries as hard enforcement. Governance that depends on operator compliance is policy, not governance. SCPs, Azure Policy deny assignments, and GCP organization policies can make specific change categories impossible via console regardless of individual IAM permissions.

Reconciliation SLAs. Emergency console changes are operationally legitimate. Permanent console changes are not. Define a reconciliation window — emergency changes must be reflected in IaC within a defined period, or they trigger a governance review.

Drift monitoring as governance signal. Terraform plan output run on schedule against production state surfaces the gap between declared state and actual state. The delta is the shadow control plane’s footprint. Treat each unreconciled divergence as a governance event.

The shadow control plane is not hidden from the organization. It is hidden from governance.

Architect’s Verdict

The shadow control plane is not a byproduct of undisciplined operations. It is a rational response to a governance system that cannot operate at incident speed. Every organization that has been running infrastructure long enough has one — the question is not whether it exists, but how much production authority it has accumulated.

The Execution Authority Gap is the delta between the governance model you declared and the execution authority that actually reaches production. Pipelines govern intent. Consoles govern capability. The gap between those two statements is where shadow control plane authority lives, accumulates, and compounds.

The shadow control plane is not temporary operational drift. It is an alternate infrastructure authority model.

Organizations that believe they operate through Infrastructure as Code often actually operate through Infrastructure as Exception.

Originally published at rack2cloud.com

The 12 VS Code Extensions That Actually Made Me Faster in 2026

yazar

I tried more than forty VS Code extensions while working on full stack projects this year. Most of them slowed me down or added no real value. Only twelve made my workflow faster and earned a permanent spot.

This post is not another random list from the marketplace. Each extension here passed one clear test – it saved me time or solved a real problem while I was writing code.

Let’s look at the twelve that truly helped me work better.

1. GitHub Copilot

Copilot stopped being optional in 2026. It writes boilerplate, suggests entire functions and explains code I did not write. The 2026 version handles multi-file context far better than older releases.

I use it most for writing test cases and converting plain English into working code blocks.

Why it stays: It saves me roughly two hours every day on repetitive code.

Pro tip: Press Ctrl+I for inline chat. This shortcut alone changed how I refactor.

2. GitLens

GitLens turns every line of code into a story. Hover over any line and you see who wrote it when they wrote it and why. The blame annotations are not just for finding who broke production. They help me understand legacy code faster.

The 2026 update added AI commit summaries which actually work.

Why it stays: Code archaeology becomes effortless.

3. Prettier

Prettier ends every formatting debate inside a team. You set the rules once and the extension enforces them on save. My students stop arguing about semicolons and tabs versus spaces within a week.

Add this to your settings.json: 

json
{
  "editor.formatOnSave": true,
  "editor.defaultFormatter": "esbenp.prettier-vscode"
}

Why it stays: Zero brain cycles spent on style.

4. ESLint

ESLint catches bugs before you run your code. Unused variables, wrong types, broken promises. ESLint flags all of it inside the editor in real time.

Pair it with Prettier and your code becomes both clean and correct.

Why it stays: It prevents bugs I would otherwise ship.

5. Thunder Client

Thunder Client replaced Postman in my workflow. It runs API requests directly inside VS Code so I stop switching windows fifty times an hour. The free tier handles everything I need for full stack work.

The collection sync feature lets my team share API requests through Git.

Why it stays: One less app on my Mac and one less context switch.

6. Live Server

Live Server still earns its place in 2026. Open any HTML file right click and select Open with Live Server. Your browser reloads on every save. For students learning the front end this extension removes the entire build setup hurdle.

Why it stays: Instant feedback loop for HTML CSS and vanilla JavaScript.

7. Tailwind CSS IntelliSense

If you write Tailwind without this extension you are punishing yourself. It autocompletes class names, previews colors inline and warns when you misspell utilities.

Why it stays: Tailwind without IntelliSense feels like coding blindfolded.

8. Path Intellisense

This one surprised me with how much it saves. Type a file path and Path Intellisense suggests files and folders as you type. No more typos in import statements. No more wrong relative paths.

Why it stays: Tiny extension. Huge daily impact.

9. Error Lens

Error Lens displays errors and warnings inline next to the actual code. Most developers run their cursor over a red squiggle and read the popup. Error Lens shows the message right next to the line.

This single change made my debugging twice as fast.

Why it stays: Errors become impossible to ignore.

10. Dev Containers

Dev Containers killed the work on my machine problem for my team. Define your dev environment once inside a container. Every teammate opens the project and gets the same Node version, the same dependencies and the same database setup.

New hires now start coding within thirty minutes instead of two days.

Why it stays: Onboarding pain disappears.

11. Auto Rename Tag

You rename an opening HTML tag and the closing tag updates automatically. Sounds small. Saves hundreds of edits a week if you write JSX or HTML daily.

Why it stays: Pure quality of life.

12. Better Comments

Better Comments color codes your comments by type. Questions show up orange warnings show up red and TODOs show up blue. Reading your own code from six months ago has become much easier.

Add these markers: 

`js
// ! Critical warning
// ? Open question
// TODO: needs refactor
// Important note

Why it stays: Comments become scannable.

The Honest Section: Three Extensions I Removed

I want to be straight with you. Some popular extensions did not survive my workflow this year.

Bracket Pair Colorizer: VS Code now does this natively. Uninstall it.

Code Spell Checker: Useful in theory but the false positives drove me crazy.

Live Server Preview: Slower than the original Live Server. Stick with the classic.

How I Set Up VS Code for Maximum Speed

Three settings made the biggest difference in my daily speed.

Disable extensions per workspace. Heavy extensions like Docker do not need to run inside a React project. Right click any extension in the sidebar and select Disable for this Workspace.

Use the Performance view. Open the command palette and run Developer: Show Running Extensions. Anything taking over 100ms to activate is a candidate for removal.

Sync settings across machines. Sign in to VS Code with your GitHub account and turn on Settings Sync. Your extensions keybindings and themes follow you to every laptop.

Quick Recommendations by Developer Type

If you are a front end developer install Tailwind IntelliSense Live Server Auto Rename Tag and Error Lens first.

If you are a back end developer install Thunder Client GitLens Dev Containers and Prettier first.

If you are a full stack developer install all 12 and never look back.

If you are a complete beginner install GitHub Copilot Live Server Prettier and Error Lens. Skip the rest until you build at least three projects.

Final Thoughts

The right extensions do not just save seconds. They remove friction so you stay in flow. My output doubled this year and I credit roughly half of that gain to a properly configured VS Code.

Test these one at a time. Add an extension. Use it for a week. Decide if it earns a spot. Reject anything that slows your editor or distracts you.

What VS Code extensions earned a permanent spot in your workflow this year? Drop them in the comments. I am always hunting for the next hidden gem.

I train developers at TISA-TECH in Jaipur where we teach full stack and AI integrated development. If you are learning to code and want structured mentorship feel free to drop a question in the comments. Always happy to help.

Espressif ESP32-E22: First Wi-Fi 6E Tri-Band RISC-V SoC Released

yazar

ESP32-E22 is Here: Espressif First Wi-Fi 6E Tri-Band RISC-V SoC

Espressif just dropped a bombshell — the ESP32-E22, their first-ever Wi-Fi 6E Tri-Band (2.4/5/6 GHz) SoC, built entirely on RISC-V architecture. Here is everything you need to know.

What is New?

The ESP32-E22 is not just an incremental update. It is a completely different beast:

  • Wi-Fi 6E Tri-Band — 2.4 GHz, 5 GHz, AND 6 GHz in one chip
  • Dual-mode Bluetooth 6.0 — latest BT standard baked in
  • Dual-core RISC-V @ 500 MHz — that is fast for an embedded MCU
  • 1MB on-chip SRAM — generous for this class
  • Up to 2.1 Gbps — seriously, this is near-Gigabit territory

Why RISC-V Matters

Espressif is making a big architectural shift. RISC-V is an open-source ISA — no licensing fees, no ARM dependency. For hobbyists and makers, this could mean cheaper boards down the line. For professionals, it means more freedom in customization.

ESP32 Family Overview (2026)

Chip Key Feature
ESP32-E22 Wi-Fi 6E Tri-Band, RISC-V, 500 MHz
ESP32-S31 Wi-Fi 6, Bluetooth 5.4, Thread/Zigbee
ESP32-H21 Ultra-low power BLE MCU, RISC-V
ESP32-P4 Multimedia powerhouse, HDMI, no Wi-Fi
ESP32-C6 Wi-Fi 6, BLE 5.3, Zigbee, Thread, Matter

Project Idea: Tri-Band Smart Home Gateway

With the E22, you could build a Matter-compatible smart home hub that operates on 6 GHz for backhaul while 2.4/5 GHz handles client devices. This is a hot use case in 2026 as more households go all-in on IoT.

Getting Started

ESP-IDF v6.0 is already out with E22 support. If you are running VS Code, the ESP-IDF extension has been updated for AI-assisted workflows this year.

ESP32 #RISCV #WiFi6E #IoT #SmartHome #Maker #Espressif #EmbeddedSystems #TriBand #Bluetooth

ESP32-E22 is Here: Espressif First Wi-Fi 6E Tri-Band RISC-V SoC

yazar

ESP32-E22 is Here: Espressif First Wi-Fi 6E Tri-Band RISC-V SoC

Espressif just dropped a bombshell — the ESP32-E22, their first-ever Wi-Fi 6E Tri-Band (2.4/5/6 GHz) SoC, built entirely on RISC-V architecture. Here is everything you need to know.

What is New?

The ESP32-E22 is not just an incremental update. It is a completely different beast:

  • Wi-Fi 6E Tri-Band — 2.4 GHz, 5 GHz, AND 6 GHz in one chip
  • Dual-mode Bluetooth 6.0 — latest BT standard baked in
  • Dual-core RISC-V @ 500 MHz — that is fast for an embedded MCU
  • 1MB on-chip SRAM — generous for this class
  • Up to 2.1 Gbps — seriously, this is near-Gigabit territory

Why RISC-V Matters

Espressif is making a big architectural shift. RISC-V is an open-source ISA — no licensing fees, no ARM dependency. For hobbyists and makers, this could mean cheaper boards down the line. For professionals, it means more freedom in customization.

ESP32 Family Overview (2026)

Chip Key Feature
ESP32-E22 Wi-Fi 6E Tri-Band, RISC-V, 500 MHz
ESP32-S31 Wi-Fi 6, Bluetooth 5.4, Thread/Zigbee
ESP32-H21 Ultra-low power BLE MCU, RISC-V
ESP32-P4 Multimedia powerhouse, HDMI, no Wi-Fi
ESP32-C6 Wi-Fi 6, BLE 5.3, Zigbee, Thread, Matter

Project Idea: Tri-Band Smart Home Gateway

With the E22, you could build a Matter-compatible smart home hub that operates on 6 GHz for backhaul while 2.4/5 GHz handles client devices. This is a hot use case in 2026 as more households go all-in on IoT.

Getting Started

ESP-IDF v6.0 is already out with E22 support. If you are running VS Code, the ESP-IDF extension has been updated for AI-assisted workflows this year.

ESP32 #RISCV #WiFi6E #IoT #SmartHome #Maker #Espressif #EmbeddedSystems #TriBand #Bluetooth

ESP32-E22 à ¡à à à ¥à ‰à §! à Šà à › Tri-Band Wi-Fi 6E à •à à §à à £à à ˆà à Espressif

yazar

ESP32-E22 มาแล้ว! ชิป Tri-Band Wi-Fi 6E ตัวแรกของ Espressif

Espressif เพิ่งปล่อยชิปใหม่ล่าสุดออกมาเลย — ESP32-E22 🎉

นี่คือชิปตัวแรกของ Espressif ที่รองรับ Wi-Fi 6E Tri-Band (2.4/5/6 GHz) พร้อม Bluetooth 6.0 ในตัว โครงสร้างภายในเป็น RISC-V Dual-Core ความเร็ว 500 MHz และมีหน่วยความจำในตัว 1MB

Specs ที่น่าสนใจ

Spec Value
Wi-Fi Wi-Fi 6E Tri-Band (2.4/5/6 GHz)
Bluetooth Dual-mode Bluetooth 6.0
CPU Dual-core RISC-V @ 500 MHz
Memory 1MB on-chip
Max Speed 2.1 Gbps

ทำไมถึงน่าสนใจ?

  • Tri-Band หมายความว่าจะไม่ติดขัดเหมือน Wi-Fi แบบเดิมที่ทุกอย่างแย่งกันในคลื่น 2.4 GHz
  • RISC-V เป็นสถาปัตยกรรมเปิดที่ไม่ต้องพึ่ง ARM — บอร์ดราคาถูกลงได้ในระยะยาว
  • 500 MHz เร็วมากสำหรับ embedded chip — เหมาะกับงานที่ต้องประมวลผลหนัก

ไอเดียโปรเจกต์

ลองทำ Smart Home Gateway ที่รองรับทุกมาตรฐาน — Wi-Fi 6E + Bluetooth + Matter ในบอร์ดเดียว ของแบบนี้ตลาดยุโรปกำลังต้องการมาก

ESP-IDF v6.0 ก็เพิ่งออกใน Q1 2026 พร้อมรองรับ E22 แล้ว ถ้ามี ESP32 อยู่ในมือ ลองอัปเดตเลย!

ESP32 #RISCV #WiFi6E #IoT #SmartHome #Maker #ESP32E22 #Espressif #EmbeddedSystems #TriBand

Cron expressions are hard to verify — so I built `croncheck`

yazar

The problem

Cron syntax is notoriously hard to verify — developers copy expressions from Stack Overflow or write their own, then paste them into random online parsers to check them, often without offline access or confidence in edge-case handling.

If you’ve hit this before, you know how it goes — you fire up a browser tab, paste the expression into crontab.guru, and hope it interprets edge cases the same way your cron daemon does.

As a solution, I created croncheck

Parse and preview cron expressions interactively — see the next 10 run times before you deploy

It’s zero-dependency Node.js, so you can run it immediately without installing anything: 

npx croncheck

Output: 

croncheck "30 9 * * 1-5"

  Expression: 30 9 * * 1-5
  Meaning:    minute 30, hour 9, every day, every month, weekdays 1-5

  Next 10 run times:
   1. Mon May 25 2026  09:30  in 4d 23h
   2. Tue May 26 2026  09:30  in 5d 23h
   3. Wed May 27 2026  09:30  in 6d 23h
   4. Thu May 28 2026  09:30  in 7d 23h
   5. Fri May 29 2026  09:30  in 8d 23h
   6. Mon Jun  1 2026  09:30  in 11d 23h
   7. Tue Jun  2 2026  09:30  in 12d 23h
   8. Wed Jun  3 2026  09:30  in 13d 23h
   9. Thu Jun  4 2026  09:30  in 14d 23h
  10. Fri Jun  5 2026  09:30  in 15d 23h

How it works

Pure Node.js cron parser that expands each field (minute, hour, day, month, weekday) into a sorted array of valid values using set arithmetic, then advances a Date object forward one minute at a time to collect the next N matching timestamps, rendered in a raw-mode terminal UI.

Why I built it

Found repeated complaints on r/devops and HN from developers who deployed broken cron schedules after writing expressions from memory. Online tools like crontab.guru exist but require a browser and fail offline. No popular zero-dep npx tool fills the gap for quick in-terminal verification before committing a schedule to production.

Try it

npx croncheck --help

Part of µ micro — one new developer tool, shipped every day. All tools are zero-dependency Node.js and run instantly with npx.

DNS records every developer sending email must understand (SPF, DKIM, DMARC explained)

yazar

Adding DNS records to send email is one of those tasks where most developers do exactly what the dashboard tells them without understanding why. That works until it doesn’t — and when deliverability breaks, debugging without understanding the underlying mechanisms is genuinely painful.

This post covers the five DNS record types involved in email authentication, what each one actually does, which ones are required, and how to verify your setup is correct.

Why email authentication exists

Email was designed in an era of trust. The original protocols have no mechanism for verifying that the server claiming to send mail from acme.com actually has permission to do so. Anyone can configure a mail server to say it is sending from your domain. This is why spam and phishing exist at industrial scale.

SPF, DKIM, and DMARC are the three authentication layers built on top of the original protocols to address this. Mailbox providers at Google, Microsoft, Apple, and others evaluate all three when deciding whether to deliver your mail, send it to spam, or reject it outright.

SPF: who is allowed to send for your domain

SPF (Sender Policy Framework) is a TXT record on your root domain that lists the IP addresses and services authorized to send email on your behalf. A receiving mail server checks this record when it receives a message claiming to be from your domain.

A minimal SPF record looks like this:

v=spf1 include:amazonses.com ~all

This says: servers listed in Amazon SES’s SPF record are authorized to send for this domain. The ~all at the end is a soft fail — messages from unauthorized servers are flagged but not rejected. -all is a hard fail — reject them. ~all is the recommended starting point.

The include: mechanism is how you authorize third-party senders. Every service you send email through needs to be listed. If you use multiple services, you chain them: v=spf1 include:amazonses.com include:sendgrid.net ~all.

SPF has a 10-lookup limit. If your include chain requires more than 10 DNS lookups to resolve, SPF evaluation fails. This is a real problem for organizations using many email services simultaneously.

DKIM: cryptographic signing for your messages

DKIM (DomainKeys Identified Mail) lets the sending server cryptographically sign outgoing messages. The receiving server retrieves your public key from DNS, verifies the signature, and confirms the message was not modified in transit.

The DNS record is a CNAME or TXT record published at a selector subdomain. SES uses three CNAME records, each pointing to SES-managed keys:

selector1._domainkey.acme.com → selector1._domainkey.REGION.dkim.amazonses.com
selector2._domainkey.acme.com → selector2._domainkey.REGION.dkim.amazonses.com
selector3._domainkey.acme.com → selector3._domainkey.REGION.dkim.amazonses.com

All three must be present and resolving correctly. The private key lives in SES; you publish only the public key via CNAME. Key rotation is handled automatically by SES when you use the CNAME delegation approach — no manual intervention required.

DKIM is the most important of the three from a deliverability perspective. Gmail, for example, heavily weights DKIM alignment when scoring inbound mail. It is also the only record required by SendFleet before you can send — without DKIM, your domain identity is not fully verified.

DMARC: policy and reporting

DMARC (Domain-based Message Authentication, Reporting and Conformance) sits on top of SPF and DKIM and adds two things: a policy telling receivers what to do with mail that fails authentication, and a reporting mechanism that sends you data about what is happening to your email.

The record lives at _dmarc.yourdomain.com and looks like this:

v=DMARC1; p=none; rua=mailto:[email protected]

The p= field is the policy:

  • p=none: do nothing, just send reports. This is monitoring mode – the right place to start.
  • p=quarantine: send failing mail to spam
  • p=reject: reject failing mail entirely

Start with p=none and review the aggregate reports (rua) for at least a few weeks before moving to quarantine. The reports show you every source sending mail claiming to be from your domain — including misconfigured legitimate senders you might not know about.

MAIL FROM subdomain: proper bounce routing

The MAIL FROM subdomain (also called the envelope from or return path) is a subdomain you configure — typically send.yourdomain.com — that SES uses as the bounce address for your emails.

Two records are required:

An MX record on the subdomain pointing to SES’s feedback servers:
send.acme.com MX 10 feedback-smtp.us-east-1.amazonses.com.

And a separate SPF TXT record on the subdomain:
send.acme.com TXT “v=spf1 include:amazonses.com ~all”

Without a custom MAIL FROM, SES uses its own domain as the return path, which can cause SPF alignment failures under DMARC. With a custom MAIL FROM on your subdomain, the return path is under your domain and SPF alignment passes.

Domain ownership TXT record

This is the simplest of the five: a TXT record on your root domain that proves to SES (and through it, to SendFleet) that you control the domain. It looks like this:

_amazonses.acme.com TXT “some-verification-token”

This must be verified before any other verification can proceed.

What is required versus what is recommended

Required to send with SendFleet: domain ownership TXT and all three DKIM CNAMEs.

Strongly recommended: SPF, MAIL FROM subdomain, DMARC in p=none mode.

Without SPF and MAIL FROM, some mailbox providers will mark your mail as suspicious. Without DMARC, you have no visibility into authentication failures and cannot protect your domain from spoofing at the policy level.

SendFleet’s domain dashboard shows per-record verification status with individual verify buttons. Use these to confirm propagation – the dashboard checks DNS directly.

Sysvar Accounts: Solana’s Live System Data, Explained for Developers

yazar

The Accounts Nobody Talks About

If you ask a Solana developer what accounts are, they’ll tell you about wallets and programs. If you ask them about sysvar accounts, there’s a good chance they pause.

Sysvars are the third category of accounts on Solana and one of the most useful once you’re building programs that need to know anything about the current state of the network. This post explains what they are, how they work, and why they exist as accounts rather than as function calls.

What Is a Sysvar?

A sysvar is a special on-chain account that the Solana runtime maintains automatically. Every slot, the network updates these accounts with current chain data: the current time, recent blockhashes, the rent rate, stake history, and more.

Programs can read sysvar accounts as inputs to their instructions. Instead of having a get_current_time() syscall, Solana gives you a Clock account. Instead of a get_rent_rate() function, there’s a Rent account.

The design choice is intentional: by making system data available as accounts, programs access it the same way they access any other on-chain data by having the account passed as an input to the instruction.

The Core Sysvar Accounts

Clock SysvarC1ock11111111111111111111111111111111

Contains the current network time:

  • slot the current slot number (~400ms per slot)
  • epoch the current epoch (~2–3 days per epoch)
  • unix_timestamp unix timestamp, updated approximately once per slot
  • epoch_start_timestamp when the current epoch began
  • leader_schedule_epoch used for validator leader scheduling

Programs use Clock when they need time-based logic: vesting schedules, time-locked accounts, auction deadlines. It’s the on-chain equivalent of Date.now().

Rent SysvarRent111111111111111111111111111111111

Contains the current rent parameters:

  • lamports_per_byte_year the base rent rate
  • exemption_threshold how many years of rent an account needs to hold to be rent-exempt
  • burn_percent what percentage of collected rent is burned

Programs use Rent when creating new accounts they need to calculate the rent-exempt minimum to fund the account correctly. Without reading Rent, a program would have to hardcode the exemption amount, which can change via governance.

RecentBlockhashes SysvarRecentB1ockHashes11111111111111111111

Contains the ~150 most recent block hashes. Transactions must include a recent blockhash as a validity window transactions referencing a blockhash older than ~150 slots are rejected as expired.

This is the mechanism that prevents transaction replay attacks: you can’t rebroadcast an old signed transaction because its blockhash will eventually expire.

EpochSchedule SysvarEpochSchedu1e111111111111111111111111

Contains the parameters that define epoch length and how slot counts ramp up during the warmup period. Programs that need to reason about epoch boundaries can use this.

StakeHistory SysvarStakeHistory11111111111111111111111111

A record of total stake activating and deactivating per epoch. Used by the staking program to calculate how quickly stake delegations activate.

Instructions Sysvar1nstructions1111111111111111111111111

A special sysvar that contains the serialized instructions of the current transaction. Programs use this to inspect other instructions in the same transaction, useful for enforcing that certain instructions appear together.

How to Inspect Sysvars

You can read any sysvar account directly from the CLI: 

# Clock
solana account SysvarC1ock11111111111111111111111111111111 --url devnet

# Rent  
solana account SysvarRent111111111111111111111111111111111 --url devnet

# Recent blockhashes
solana account SysvarRecentB1ockHashes11111111111111111111 --url devnet

The raw output is binary-encoded data. To see decoded values, use a JSON RPC call: 

curl https://api.devnet.solana.com -X POST -H "Content-Type: application/json" -d '{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "getAccountInfo",
  "params": [
    "SysvarC1ock11111111111111111111111111111111",
    {"encoding": "jsonParsed"}
  ]
}'

The jsonParsed encoding tells the RPC to decode the known sysvar format into readable fields.

Why Accounts Instead of Syscalls?

Other blockchains implement system data differently. Ethereum has block.timestamp and block.number as built-in variables accessible inside smart contract code no account required.

Solana’s account-based approach has a specific advantage: it fits the transaction model. Solana transactions must declare every account they’ll read upfront. By making sysvars accounts, a program that reads the clock must include the Clock sysvar in its account inputs making it statically visible which system data each instruction accesses before execution.

This supports Solana’s parallel execution model. The scheduler can see exactly which accounts (including sysvars) a transaction will read, and run non-conflicting transactions simultaneously.

Sysvar Addresses Never Change

One useful property: sysvar addresses are fixed constants, known at compile time. The Clock sysvar has always been SysvarC1ock11111111111111111111111111111111 and always will be. Programs hardcode these addresses as known constants rather than looking them up dynamically.

In the Solana SDK: 

use solana_program::sysvar::clock::ID as CLOCK_SYSVAR;
// CLOCK_SYSVAR == SysvarC1ock11111111111111111111111111111111

The Full Picture

Sysvar accounts complete the “everything is an account” model:

  • Wallet accounts user-owned data, managed by the System Program
  • Program accounts executable bytecode, managed by the BPF Loader
  • Data accounts program-owned state, managed by user programs
  • Sysvar accounts network-owned runtime data, managed by the Solana runtime itself

Four types. One storage model. Every category fits the same four-field account structure just with different owners, executability flags, and data payloads.

Sysvars are the part of that model that gives programs a window into the live state of the network they’re running on.