Mert

A git-backed, cache-resilient, privacy-first CMS – and a live proof of concept born in Wittenberg.

“Where Is My Website Calling Home?”

If you’ve ever installed WordPress and added a plugin – any plugin – you’ve probably asked yourself this question at some point.

Google Analytics. Facebook Pixel. Some random jQuery CDN. A newsletter widget that reports back to a server in Virginia.

Your website is having conversations you never started.

With Forge CMS 2026.3.27, we’re doing things differently.

What Is Forge CMS?

Forge CMS is a lightweight content management system built on the Forge 4D stack. It uses SML (a declarative UI language inspired by QML) to define pages and layouts, runs on a VPS, and serves static-style content with zero JavaScript framework overhead.

No React. No Next.js. No npm install rabbit hole.

Just clean, readable markup – and now, with 2026.3.27, a proper content pipeline.

What’s New in 2026.3.27

1. Git-Backed Content – Version History for Free

All page content is now loaded from a Git repository that you specify in the server config.

content_repo: https://codeberg.org/YourOrg/your-site

That’s it. Your entire website lives in a repo. You get:

• Full version history – every edit is a commit
• Rollback – one git revert and you’re back
• Collaboration – PRs, reviews, branches – standard Git workflow
• Flexibility – Codeberg, GitHub, Gitea, your own Forgejo instance – whatever you trust

And if you want to switch repos at runtime? You can. Swap the config, restart – the new repo is live. This means you can set up a fallback repo (your own server, GitHub mirror, whatever) and pivot within minutes if your primary goes down.

During the writing of this article, Codeberg went fully offline – “Service unavailable.”

The live site at atesti.crowdware.info kept serving without interruption – from VPS cache.

Same day as the release tag. Unplanned. Perfect.

2. Resilient Caching – The Site Stays Up

Even if the source repo becomes unreachable, Forge CMS keeps serving from its VPS-side cache.

This isn’t a CDN. There’s no Cloudflare account, no monthly bill, no Terms of Service to read. It’s just the server remembering what it already fetched – and staying calm when upstream has a bad day.

Practically: your site survives your Git host’s outage.

We tested this accidentally today. It worked.

3. Umami Analytics – Opt-In, Commented Out by Default

WordPress ships with Google Analytics baked into half its popular themes. You install a theme, you get tracking. Surprise.

In Forge CMS, analytics is a single SML element:

// Umami analytics - uncomment to enable
// Umami {
//     websiteId: "your-id-here"
//     src: "https://your-umami-instance/script.js"
// }

Commented out. By default. On purpose.

When you do enable it, it points to Umami – an open source, self-hostable, GDPR-friendly analytics tool. No cookies. No cross-site tracking. No phone calls to Mountain View.

You know exactly what it tracks, where the data goes, and how to turn it off.

That’s the deal.

Live Proof: Atesti para Dana

The first real-world deployment of Forge CMS 2026.3.27 is atesti.crowdware.info – a site for Atesti para Dana, a local currency experiment born on March 25, 2026 in Wittenberg, Germany.

The concept: every existing currency begins with debt or promise. Atesti begins with a completed act of giving – witnessed by three people. Only then is a “GUT-Schein” (goodness-certificate) issued. No debt. No promise. Just something that already happened.

The site is intentionally local-first. It’s in German. It’s for Wittenberg. The seed doesn’t need to travel fast – it needs to land well.

But the code that serves it? That’s for everyone.

The Stack, Briefly

Why This Matters

The web is full of CMS platforms that are technically free but philosophically expensive. They trade your users’ attention and data for convenience.

Forge CMS makes a different trade: a little more intentionality upfront, in exchange for knowing exactly what your site does.

Git for content. Cache for resilience. Umami for honest analytics.

And when your Git host has a bad day – your site just shrugs.

Get Involved

• 🔧 Code: codeberg.org/CrowdWare
• 🌱 Live example: atesti.crowdware.info
• ✉️ Contact: [email protected]

Aho. 🌱

Our atesti site may be loading a bit slow, because it checks against the repo if there are newer versions. And when the host is slow, this will also make the website slow, even when content is cached.

Also I studied Web- and GraphicsDesign, but I am too lazy to write CSS. Just wanted to show a very basic website 😉

Art – Wittenberg, March 27, 2026

License: GPL3 + commercial dual license

Here is something that surprises almost every engineering manager the first time they live through it: growth slows you down before it speeds you up.

You hire five more engineers. You expect output to increase proportionally. Instead, things get messier. PRs take longer to review. Standups run over. Someone built a feature that conflicts with something another team was working on. The planning process that worked for eight people completely breaks down at fifteen. Velocity, by most measures, actually drops.

This is not a failure of hiring. It is a failure of org design. And it happens so reliably, at so many companies, that it has become one of the most predictable traps in engineering leadership.

The good news is that it is largely avoidable if you think about the structural side of growth before you need it, rather than after things are already falling apart. That is what this article is about. Not headcount strategy in the abstract, but the practical mechanics of how teams grow: org structure, hiring cycles, cross-team coordination, and the specific bottlenecks that kill velocity as you scale.

I want to start with the thing most scaling guides skip, which is the relationship between team size and communication overhead. It is not intuitive until you see the numbers.

When you have a team of four engineers, there are six possible communication paths between them. Add a fifth person and you get ten. Add a sixth and you get fifteen. The formula is n(n-1)/2, and the point is not to memorise it but to understand the curve. Every person you add increases the number of relationships, dependencies, and potential misalignments on the team. At some point, that overhead exceeds the capacity benefit of adding another person.

This is what Brooks’ Law is actually pointing at when it says adding manpower to a late software project makes it later. It is not that people are useless. It is that onboarding and coordination costs are real, and if you add people without managing the structural overhead, you pay more than you gain.

The practical implication is that teams above a certain size need organisational structure to remain effective, not because structure is inherently good, but because it reduces the surface area of coordination that each person has to manage. Smaller, autonomous teams with clear ownership can each operate with low overhead while the aggregate output scales. This is the core logic behind the two-pizza team concept, and it holds up in practice even if the pizza metric itself is a bit silly.

So when should you split a team? The signals I watch for:

• when more than half your standup is irrelevant to most of the people in it
• when engineers are regularly blocked waiting for decisions from a single point of authority
• when the codebase has grown to the point where nobody fully understands the system end-to-end
• when you have grown past eight or nine people and the coordination overhead is visibly slowing things down.

Any one of those is a reasonable trigger. Two or more at once and you are already overdue.

How you split matters as much as when. The worst splits are by technical layer; frontend team, backend team, infrastructure team. That structure creates handoff-heavy workflows where a single feature requires coordination across three teams and nobody owns the outcome end-to-end. Conway’s Law tells you that your org structure will reflect your system architecture, so if you want loosely coupled systems, you need loosely coupled teams with bounded ownership. Organise around product domains or user-facing capabilities instead. A team that owns the payments experience end-to-end – from the frontend interaction to the backend logic to the database schema – can move fast without waiting on anyone else.

The flip side of this is that you need to be deliberate about shared infrastructure. When teams own their domains end-to-end, they will inevitably have shared concerns: authentication, logging, deployment pipelines, internal libraries. Left unaddressed, every team solves these independently and you end up with seven slightly different implementations of the same thing. The solution is a platform or infrastructure function – a small team explicitly chartered to own the things that are shared – with a strong bias toward building self-service tooling rather than becoming a bottleneck that product teams have to request work from.

Now let’s talk about hiring cycles, because this is where a lot of scaling plans go wrong in a different way.

Hiring has a lag. From the moment you open a role to the moment a new hire is meaningfully contributing is typically four to six months, sometimes more. That includes recruiting time, interview process, notice period, onboarding, and ramp-up. If you wait until your team is visibly overloaded to start hiring, you will be waiting for relief for a long time while the people already on your team burn out.

The right time to start a hiring cycle is before you need the capacity, not when you need it. That requires a planning horizon most engineering managers are not used to operating on. You need to look three to six months out and ask: what does this team need to look like at that point, given what we have committed to delivering? Work backward from that and you will usually find that you needed to start hiring two months ago.

This requires building a relationship with your recruiting function and being a genuine partner rather than just a consumer. Understand what is in the pipeline. Give fast, specific feedback on candidates. Keep your job descriptions current. Show up to sourcing conversations. Engineering managers who treat recruiting as someone else’s job tend to have longer time-to-hire and weaker candidate pools than those who engage actively.

One thing worth doing if you have the data: track your actual time-to-productivity for new hires by role and seniority. Not time-to-hire, but time-to-first-meaningful-contribution and time-to-full-productivity. That data tells you how far ahead you need to plan, and it often reveals onboarding gaps that are adding weeks of unnecessary ramp-up time.

Cross-team handoffs are where velocity goes to die once you have multiple teams. I have seen this pattern so many times it is almost a cliche: two teams are working on adjacent parts of a system, the interface between them is not well defined, and what should be a clean integration turns into a weeks-long negotiation over API contracts, data ownership, and who is responsible for what.

The solution is not more meetings between teams. It is clearer ownership and better written contracts upfront. Before work begins on anything that spans team boundaries, you want a written definition of: who owns what, what the interface looks like, what the expected behaviour is, and what the escalation path is when something goes wrong. This does not have to be elaborate. A short technical spec or an RFC that both teams have reviewed and agreed to is usually enough.

Some teams formalise this with an internal RFC process, where any significant technical change that affects other teams requires a written proposal with a comment period before work begins. Done lightly, this is genuinely useful. Done heavily, it becomes bureaucratic overhead that slows everything down. The right level depends on your team size and the pace of change in your system. At ten engineers, a quick shared doc and a Slack thread is probably enough. At fifty, you probably want something more structured.

Dependency management is a related problem that gets worse as teams scale. When team A is blocked on team B’s work, that is a velocity killer. The standard advice is to minimise cross-team dependencies, which is correct but not always achievable. When dependencies are unavoidable, the goal is to surface them as early as possible, sequence work so that the dependent team can start on other things, and have a clear escalation path when a dependency is at risk of slipping.

This is one of the things that makes engineering planning genuinely hard at scale. You are not just sequencing work within a team. You are managing a dependency graph across multiple teams, each with their own priorities and constraints. The teams that do this well treat inter-team dependencies as first-class items in planning – visible, tracked, and actively managed – rather than background assumptions that only surface when they blow up.

Let me talk about a specific bottleneck that I see consistently kill velocity in growing teams: the single point of decision authority.

It usually starts innocuously. A senior engineer or tech lead has strong opinions and good judgment, so decisions naturally flow through them. This works fine at small scale. At scale, it creates a queue. Every architectural decision, every significant PR, every cross-cutting concern sits waiting for one person’s input. That person becomes increasingly overloaded and increasingly a blocker, not because they are doing anything wrong but because the structure has not kept up with the team’s growth.

The solution is to distribute decision-making authority deliberately. That means being explicit about who has authority over what class of decision, investing in documentation and principles that allow people to make good decisions independently, and creating a culture where “I made a call and here is my reasoning” is the norm rather than “I waited for someone to tell me what to do.”

This requires accepting that some decisions made without your input will be decisions you would have made differently. That is an uncomfortable tradeoff. But a slightly suboptimal decision made quickly and autonomously is almost always better for team velocity than the perfect decision that took two weeks to reach because everything had to go through one person.

The other bottleneck worth naming explicitly is the release process. Teams that can deploy independently and safely move faster than teams that share a deployment pipeline or a release schedule. If your release process requires coordination across multiple teams or a human approval gate that creates a queue, that is a structural constraint on your velocity that hiring more people will not fix. Investing in a deployment pipeline that gives teams independent, safe, fast release capability is one of the highest-leverage infrastructure investments you can make as you scale.

There is a meta-principle underlying most of what I have described: the goal of organisational design as you scale is to keep the blast radius of any individual team’s decisions small while maximising their autonomy within that boundary. Small blast radius means that when a team makes a mistake, it does not cascade across the whole system. High autonomy means the team can move fast without waiting for permission or coordination.

Those two things are in tension, and the job of engineering leadership is to find the right balance for your organisation at its current size and stage. The balance shifts as you grow. What worked at fifteen engineers will break at forty. What works at forty will need to change at a hundred. The managers who navigate this well are the ones who keep asking “is our structure still serving us, or are we serving our structure?” and are willing to change the answer when it needs to change.

Growth is not just an execution problem. It is a design problem. The teams that scale well treat their org structure, hiring cadence, and cross-team coordination as things that need to be actively designed and maintained, not defaults that exist until they break.

Start designing before you need to. You will be glad you did.

Next in the series: Building Psychological Safety in Engineering — creating a culture where failure is a learning opportunity, not a liability.

I built a WordPress plugin that solves one of the most tedious yet critical SEO tasks: internal linking. The problem? Manually connecting hundreds (or thousands) of posts is time-consuming, error-prone, and nearly impossible to scale. But blind automation risks irrelevant links or over-optimized anchor text. My solution bridges this gap with AI-powered linking that respects editorial control.

Under the hood, the plugin uses semantic embeddings to understand content relationships, not just keyword matching. When you publish a post, it analyzes the entire site’s indexed content, scores potential link targets based on relevance and authority, and suggests the best connections. The twist? You retain full control. Every suggestion appears in a dedicated editor panel where you can tweak anchor text, reject links, or bulk-apply with a threshold for confidence scores.

For large sites, a bulk processor handles site-wide linking in the background. It identifies orphan pages (content with no incoming links) and rescues them by connecting them to high-authority sources. A visual graph maps your entire link structure, letting you spot weak connections or overlinked content instantly. The plugin even flags repetitive anchor text to avoid SEO penalties.

The real magic is the hybrid approach: AI does the heavy lifting of analyzing content and generating suggestions, while you decide what gets applied. It’s automation without sacrificing precision. Check it out here if you’re tired of manual linking but wary of losing control.

Hiring is one of the highest-leverage things you will do as an engineering manager. A single great hire compounds over years. A single bad hire – or more precisely, a bad hiring process that lets the wrong person through while filtering out the right ones; costs you more than you think. Not just in time and salary, but in team morale, velocity, and the invisible tax of managing a poor fit.

Most engineering hiring processes are broken in ways that managers do not fully recognise until they have been on the other side of them. I have sat in hiring debriefs where five engineers gave five different verdicts on the same candidate and nobody could articulate why they felt the way they did. I have seen candidates with brilliant portfolios fail whiteboard tests that had nothing to do with the actual job. I have watched great engineers get filtered out because they had a quiet interview style and the interviewer mistook silence for incompetence.

The goal of this article is to help you build a process that is structured enough to reduce those failure modes, flexible enough to surface genuine talent, and honest enough to actually tell candidates what working on your team is like. That last one matters more than most people realise. Hiring is a two-way evaluation, and the best candidates have options.

Let’s walk through the full arc: structure, technical assessment, culture and values alignment, and onboarding. Not as separate checklists but as a coherent system that you design intentionally.

Defining what you are actually hiring for is the step most hiring processes skip or do half-heartedly. A job description is not a definition of success. “Strong communication skills and a passion for technology” is not a definition of success. Before you post a role, sit down with a blank document and answer three questions: what does this person need to be able to do in the first ninety days to be considered a strong hire? What do they need to be able to do after a year? What specific gaps on the team are we trying to close?

Those answers should drive everything else. The technical assessment should test for the actual skills the role requires. The interview questions should probe for the actual behaviours the role demands. The evaluation criteria should map directly to those definitions of success. When you have that clarity upfront, the rest of the process becomes much easier to design and much harder to game.

It also forces an honest conversation about scope. Are you hiring a senior engineer to independently own complex technical problems, or a mid-level engineer to execute on well-defined work? Those are different roles, different assessments, and different interview conversations. Conflating them (which is extremely common) leads to either overhiring for the work or underhiring for the scope and setting someone up to fail.

The interview structure itself should follow a consistent format for every candidate for the same role. Not because consistency is a bureaucratic virtue but because without it, you cannot make fair comparisons. If candidate A had a rigorous technical discussion and candidate B got a casual conversation about their career history, your debrief is comparing two different things. Structured interviews reduce that problem significantly.

A reasonable interview structure for a senior engineering role looks something like this. A recruiter or hiring manager screen that covers career background, motivations, and basic role fit; thirty minutes tops, mostly conversational. A technical assessment that gives you signal on the actual skills the role requires; more on format in a moment. A values and working-style conversation with one or two people on the team. And a final conversation with the hiring manager that covers the role more deeply, answers the candidate’s questions, and gives you a chance to assess how they think about their work at a higher level.

The exact structure will vary by seniority and team. But the principle is the same:

• each stage should have a clear purpose
• a clear set of things it is trying to evaluate
• a consistent format so that different candidates are being measured against the same bar.

Technical assessment is where the most disagreement happens and where the most damage is done. Let me be direct about what I think: timed algorithm puzzles and whiteboard problems that bear no resemblance to the actual job are a poor way to assess most engineering roles. They test a specific kind of performance under artificial pressure that correlates poorly with day-to-day engineering work. They also systematically disadvantage experienced engineers who have been in industry roles and simply do not practice leetcode.

That does not mean technical assessment is not important. It absolutely is. It means the assessment should reflect the actual work.

For most backend engineering roles, a practical take-home that asks the candidate to do something roughly analogous to what they would do on the job is a better signal than a timed algorithm test. Something like: here is a small API, extend it with this feature, write tests, and leave notes on any tradeoffs you made. That tells you how someone actually writes code, how they think about testing, how they communicate their decisions; all things that matter in the role. You can review it asynchronously, which is fairer to candidates in different timezones or with jobs that make synchronous sessions difficult.

If you do a live technical session, make it collaborative rather than evaluative in the traditional sense. Work through a problem together. Let the candidate look things up. Ask them to walk you through their reasoning. That is a much closer simulation of actual engineering work than asking someone to produce a correct answer under observation with no resources.

One thing worth doing regardless of format: review your technical assessment regularly to check whether it is actually predicting performance. If you can look back at candidates you hired and compare their assessment performance to their on-the-job performance, you will often find that the correlation is weaker than you assumed. That is useful information for calibrating the assessment over time.

Culture fit is a phrase that gets misused constantly, so let’s reframe it. What you are actually evaluating in a “culture fit” conversation is values alignment and working style compatibility. Not whether someone shares your hobbies or went to the same kind of school. The distinction matters because culture fit as commonly practiced is one of the most reliable vectors for homogeneity in hiring.

Values alignment is about the things that actually affect how someone works: how they handle disagreement, how they respond to ambiguity, how they communicate when something is going wrong, whether they default to transparency or opacity under pressure, how they think about ownership. Those are things you can probe for directly with behavioural questions.

Behavioural questions should be specific and past-oriented. “Tell me about a time when you had to push back on a technical decision you disagreed with” gives you real signal. “Are you someone who speaks up when you disagree” gives you the answer the candidate thinks you want. The difference in data quality between those two questions is significant.

A few questions that reliably surface useful signal across seniority levels:

• Tell me about a time a project you were responsible for went significantly off track. What happened and what did you do?
• Tell me about a piece of feedback that genuinely changed how you work.
• How do you typically approach a technical decision when the right answer is not obvious?
• Tell me about a time you worked with someone whose working style was very different from yours.

What you are listening for is not the story but the self-awareness. Does this person understand their own patterns? Do they take ownership of things without being defensive? Do they demonstrate genuine curiosity and willingness to learn? Those qualities matter across almost every technical role and almost every team culture.

The debrief structure matters as much as the interview itself. A common failure mode is the unstructured debrief where the first person to speak sets the frame and everyone else anchors to their opinion. This is how groupthink gets baked into hiring decisions.

A better approach: everyone writes their evaluation independently before the debrief begins. Not a detailed report, just a hire or no-hire recommendation and a few bullet points on the key evidence for each dimension you were evaluating. Then in the debrief, go around the room before any open discussion. Once everyone has shared their read, then you discuss disagreements.

This surfaces more honest signal and makes disagreements more productive. When two interviewers have very different reads on the same candidate, that disagreement is itself information worth understanding. Sometimes it means the candidate gave different answers to different people. Sometimes it means the interviewers were evaluating for different things. Either way, working through it leads to a better decision.

Define your evaluation rubric before the process starts, not after. Dimensions like technical depth, communication clarity, problem-solving approach, and ownership mindset should have concrete descriptions of what strong, acceptable, and weak looks like for the role in question. Rubrics are not bureaucratic overhead. They are the thing that makes your process defensible and improvable.

Onboarding is the part of the hiring process that most engineering managers treat as someone else’s responsibility. It is not. The way you integrate a new engineer into the team in their first sixty days has a direct effect on how quickly they become productive, how connected they feel to the team, and whether they stay past the twelve-month mark.

A good engineering onboarding plan has a few consistent elements. It starts before day one: send the new hire an onboarding doc before they start so they know what to expect in week one. Not a firehose of information, just an overview of what the first few weeks look like and who they will be meeting.

In the first week, the goal is orientation not productivity. Get their environment set up, walk them through the architecture at a high level, introduce them to the key people they will work with, and give them something small but real to ship. That first commit or merged PR is important for psychological reasons; it makes the new hire feel like a contributor rather than an observer.

In weeks two through four, graduate the complexity. Give them increasingly meaningful work with explicit context about why it matters and what good looks like. Run a proper one-on-one at the end of week one, week two, and week four specifically to check in on how the onboarding is going and surface any confusion or friction early.

Assign a buddy. Not a formal mentor relationship, just someone on the team who has been around for a while and is available to answer the questions the new hire is too self-conscious to ask their manager. Questions like “where does this documentation actually live” or “is it normal that this service takes fifteen minutes to build” are exactly the kind of thing a buddy handles well and that a new hire will not raise in a one-on-one for fear of seeming underprepared.

At the sixty-day mark, have an explicit conversation about how the onboarding has gone. What worked? What was confusing? What would have helped to know earlier? This is partly for the new hire’s benefit and partly for yours. The feedback you get from new engineers about your onboarding process is some of the most valuable signal you have for improving it, because they just experienced it with fresh eyes.

One broader principle worth naming: the hiring process is a product. It has users (candidates and interviewers), it produces an output (a hiring decision), and it can be improved iteratively based on data. Treat it that way.

• Track your offer acceptance rate.
• Track time-to-hire.
• Talk to candidates who declined your offers and find out why.
• Talk to new hires at the sixty-day mark and find out where the process was unclear or misleading.
• Run a retrospective on your process once a quarter.

Most engineering teams never do any of this. They run the same hiring process for years without measuring whether it works, then wonder why their candidate pool is shallow or their offer acceptance rate is low. A small amount of deliberate iteration goes a long way.

The last thing I want to say about hiring is about honesty. Be honest with candidates about the role, the team, and the company. Tell them what is genuinely hard about working there. If your codebase has serious technical debt, say so. If the team is going through a restructuring, say so. If the role requires a lot of on-call, say so clearly and early.

This feels counterintuitive because you want to sell the role. But candidates who join with an accurate picture of what they are getting into are better retained and faster to trust you than candidates who discover the gap between the pitch and the reality in their first month. The best hiring conversations I have seen treat the candidate like an intelligent adult who deserves real information, not a polished brand narrative.

That respect carries forward into the relationship if they join. And it saves everyone a painful offboarding conversation six months later.

Next in the series: Scaling Engineering Teams Without Losing Velocity – org design, hiring cycles, and avoiding the bottlenecks that slow teams down as they grow.

I kept hearing that fasthttp was faster than Go’s standard net/http. So I decided to stop guessing and just measure it.

The result? 5.6x faster. Zero heap allocations. And the benchmark is only two files.

Keeping It Fair

The trick to an honest benchmark: remove everything except what you’re testing.

Both servers use the same in-memory listener — no TCP, no network noise. Same endpoints, same response bodies. The only difference is the HTTP stack.

// net/http
func SimpleHandler(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Hello, World!"))
}

// fasthttp
func SimpleFastHandler(ctx *fasthttp.RequestCtx) {
    ctx.Write([]byte("Hello, World!"))
}

Same job. Very different performance.

The Numbers

Apple M2 Pro, Go 1.23.4

Zero allocations on simple requests. That’s not a typo.

The parallel gap is smaller (2.5x) because net/http already handles concurrency well. But even there — 63 allocations vs. 0.

What Makes fasthttp So Fast?

It comes down to one idea: don’t allocate, reuse.

net/http creates fresh Request and ResponseWriter objects for every single request. fasthttp pools them with sync.Pool and resets them between requests. That’s why you see 63 allocs drop to 0.

It also skips things most apps don’t need — header map cloning, chunked encoding by default, HTTP/2. Less work per request = less time per request.

So Should You Switch?

Use fasthttp when:

• You’re building a proxy, gateway, or high-throughput service
• Allocation pressure and GC pauses are a real problem
• You need every microsecond on hot paths

Stick with net/http when:

• You need HTTP/2
• You rely on the Go middleware ecosystem
• context.Context propagation matters to you

For most apps, net/http is the right choice. But when performance is the priority, fasthttp earns its name.

Run It Yourself

git clone https://github.com/soyvural/fasthttp-vs-nethttp.git
cd fasthttp-vs-nethttp
go test -bench=. -benchmem -count=3

Two files. No dependencies beyond fasthttp. Clone it, run it, see your own numbers.

Full source: soyvural/fasthttp-vs-nethttp

Got different results on your machine? I’d love to see them in the comments.

Remote engineering leadership is not a soft skill. It is a systems problem.

I have watched a lot of engineering managers struggle with distributed teams, and the pattern is almost always the same. They take the management habits that worked in an office, transplant them into Slack and Zoom, and then wonder why everything feels slower, noisier, and harder to trust. The problem is not the tools. The problem is that remote work requires a fundamentally different operating model, and most managers never build one.

This article is about building that model. We will cover async communication rhythms, documentation as infrastructure, how to build genuine trust without physical proximity, and how to protect your team’s well-being when the office and home become the same place. These are not abstract principles. They are specific systems you can start implementing this week.

The Mental Model Shift That Changes Everything

Before we get into tactics, there is one mental model shift that underpins all of it.

In a co-located team, the default medium is conversation. Information lives in people’s heads and travels through spoken exchanges. When you need to know something, you walk over and ask. The cost of communication is low, so the system works even without much structure.

In a remote team, the default medium has to be writing. Not because conversations are bad, but because you cannot afford to have knowledge trapped in a single timezone or dependent on someone being available at 2pm on a Tuesday. When you make writing the default, you shift from a pull model (someone asks, someone answers) to a push model (important information is documented and available). That shift is what makes remote teams actually work.

Every tactic in this article flows from that principle. Async rhythms exist because writing enables async. Documentation-as-infrastructure exists because written knowledge compounds. Trust-building at a distance exists because when people cannot see each other working, shared written context is what replaces ambient presence.

Keep that framing in mind as we go.

Async Rhythms: Designing Your Team’s Cadence

The biggest mistake in remote engineering management is treating async as a lesser version of real-time communication. Async is not a fallback. Done well, it is actually more productive than most synchronous alternatives, because it respects deep work, reduces context switching, and creates a written record by default.

But async does not mean “respond whenever you feel like it.” It means designing intentional rhythms so that the team stays aligned without being constantly interrupted.

Here is the framework I recommend:

• Define response time expectations explicitly. Not “reply promptly”: that means different things to different people. Something like: Slack messages get a response within four hours during working hours. Anything urgent uses a dedicated channel or a direct ping with the word “urgent” in it. Anything that can wait for the next day goes into a thread or a document. When these norms are written down and agreed on, a lot of the ambient anxiety around “is this person ignoring me” disappears.
• Protect focus blocks. Remote engineers are often more interrupted than office engineers, not less, because the barrier to sending a message is so low. Push back on this. Establish team-wide focus blocks where Slack notifications are off and meetings do not get scheduled. Two hours in the morning, two hours in the afternoon — something like that. Treat these as seriously as you would treat a production incident.
• Run fewer, better meetings. Every meeting on a remote team has a higher cost than it appears, because it requires people in different timezones and different focus states to synchronize at a specific moment. Before scheduling anything, ask: can this be an async update, a thread, or a document instead? If the answer is yes, do not schedule the meeting. When you do run meetings, have an agenda written in advance, assign a note-taker, and publish the notes publicly within an hour. Meetings that produce no written artifact are meetings that happened twice: once in real time, and again when people have to ask what was decided.
• Have a weekly team update ritual. One of the most useful habits I have seen remote engineering managers develop is a simple weekly async standup. Each engineer writes three or four sentences: what they shipped this week, what they are working on next week, any blockers. It takes five minutes to write and five minutes to read, and it gives the team a shared sense of momentum without requiring a meeting. You can do this in Notion, Confluence, a GitHub discussion, or even a dedicated Slack channel. The format matters less than the consistency.

Documentation as Infrastructure

Documentation in most engineering orgs is treated as a chore; something you write after the fact to satisfy a process requirement. That approach fails in remote teams, because documentation is the primary way context travels across the organization.

Think of documentation as infrastructure. It is the road network that lets information move. When the roads are bad, everything slows down. When the roads are good, everything flows.

There are three categories of documentation that matter most for remote engineering teams:

• Decision records. When your team makes a significant technical or product decision, write down the context, the options you considered, the tradeoffs, and what you chose. This does not have to be long; a few paragraphs is often enough. The value is that six months later, when someone asks “why did we do it this way,” you have an answer that does not depend on someone’s memory. Lightweight Architecture Decision Records (ADRs) are a good format for this. Keep them in the repo alongside the code they describe.
• Runbooks and operational guides. Every repeatable operational task; deploying, running migrations, handling a specific class of incident, onboarding a new service – should have a written guide. Not a perfect guide, but a good-enough guide that someone can follow without asking five questions. Remote teams that lack runbooks create invisible dependencies on specific individuals, and those dependencies become serious bottlenecks.
• Onboarding documentation. The quality of your onboarding docs tells you a lot about the overall health of your remote team’s documentation culture. When a new engineer joins, can they get their environment set up, understand the architecture, and ship something meaningful in their first two weeks, mostly by following written guides? If not, you have documentation gaps that are slowing down your entire team, not just new hires.

One practical tip: assign a rotating “documentation duty” to engineers on the team. Each week or sprint, one person is responsible for updating docs that are out of date, filling gaps they notice, and adding anything that was only in someone’s head. This turns documentation from a solo chore into a team practice.

Building Trust Without Physical Proximity

Trust is harder to build at a distance, and it is worth understanding why. In an office, a lot of trust-building happens through ambient signals; you see someone arrive early, stay late, help a colleague, handle a difficult conversation calmly. Those signals accumulate over time without anyone consciously tracking them.

Remote teams do not have ambient signals. Trust has to be built deliberately, through consistent behavior over time and through enough human contact that people actually know each other.

A few things that work:

• Show up consistently to the few meetings you do have. In a remote context, meetings are often the primary venue where people see each other as humans rather than text on a screen. Being reliably present, engaged, and prepared signals that you take the relationship seriously.
• Create low-stakes social touchpoints. Virtual coffee chats, optional Friday hangouts, a random channel in Slack for non-work things; these feel frivolous but they matter. People work harder for and communicate more openly with people they actually know. You do not need to force this, but you do need to create the infrastructure for it to happen. In my experience mentoring engineers who have moved between fully remote and hybrid roles, the ones who invest in these informal relationships consistently have an easier time navigating difficult work conversations later. The social foundation carries over.
• Give feedback frequently and specifically. One of the most trust-eroding patterns in remote management is the feedback vacuum; where engineers go weeks without hearing whether their work is landing. Specific, timely feedback, even small amounts of it, communicates that you are paying attention. “That PR review you wrote last Thursday was really thorough, I appreciated it” takes ten seconds and has a disproportionate effect on someone’s sense of being seen.
• Be transparent about decisions and context. Remote engineers who feel out of the loop tend to fill that vacuum with their own narratives, and those narratives are usually more negative than reality. When you make a decision that affects the team, explain why. When something is uncertain, say it is uncertain. When the company is going through something difficult, acknowledge it. Information hoarding in a remote context creates distrust far faster than it would in an office.
• Trust people’s work, not their activity. This is the one that trips up a lot of managers who are new to remote. The instinct to monitor activity; who is online, how quickly they respond, whether they log into the project management tool every morning – is natural but counterproductive. It signals distrust, and it measures the wrong thing. What matters is whether the work is getting done and getting done well. Build your feedback loops around outcomes and output quality. Evaluate work in PRs and demos and shipped features, not presence signals.

Cross-Timezone Collaboration

If your team spans timezones, you have an additional set of challenges worth addressing specifically.

The most important thing is to identify and protect your overlap window. If your team has people in London, New York, and San Francisco, you probably have one or two hours where everyone is theoretically available. Do not fill that window with status meetings. Use it for the things that genuinely require synchronous interaction: difficult decisions, thorny technical discussions, anything emotionally sensitive. Reserve everything else for async.

Rotate meeting times when necessary. If your standup is always at 9am Eastern, your engineers in Singapore are always staying up late or waking up early. Rotating the inconvenience is a small gesture that signals you take their time seriously.

Be explicit about whose timezone is default and when. If your planning docs say “we will ship on Friday,” which Friday? Which timezone? These details seem trivial until they are not. Building a habit of including timezone context in time-sensitive communications prevents a class of miscommunications that accumulate into real friction.

Well-Being and Sustainable Remote Work

Remote work is fantastic for autonomy and focus. It is genuinely difficult for isolation and boundary erosion. As an engineering manager, you have a responsibility to pay attention to both sides of that equation.

Watch for isolation signals. Remote engineers who are struggling often go quiet before they surface the issue explicitly. If someone who used to be active in threads and reviews goes noticeably quiet, check in. Not with “is everything okay” as a formality, but with a real one-on-one conversation where you ask how things are going and actually listen to the answer.

Normalize the end of the workday. One of the more insidious patterns in remote teams is the slow expansion of working hours. When the office and home are the same place, the natural stopping signals disappear. Engineers who routinely work evenings and weekends are not a sign of commitment. They are a sign of an unsustainable pattern that will eventually lead to burnout or attrition. As a manager, model the behavior you want: sign off at a reasonable hour, do not send non-urgent messages late in the evening, and acknowledge when someone is clearly overextended.

Check in on workload regularly. Capacity conversations should happen in one-on-ones, not just during planning ceremonies. Ask directly: is the workload sustainable right now? Where are you feeling stretched? This gives you information you need before a problem becomes a crisis, and it signals that you actually care about the answer.

Invest in the physical workspace. If your budget allows, provide equipment stipends or home office reimbursements. A good chair, a good monitor, and reliable internet are not luxuries for a remote engineer — they are the tools of the job. Teams whose employers invest in their physical setup tend to be more productive and more loyal.

Putting It Together: The Remote Engineering Operating System

Remote engineering leadership works when you treat it as a system design problem. Here is the short version of what that system looks like:

Communication has explicit norms; response time expectations, escalation paths, and async as the default. Focus time is protected. Meetings are scarce and well-run.

Knowledge is written down. Decisions have records. Operational tasks have guides. Onboarding works without someone holding your hand through it.

Trust is built through consistent behavior, genuine human contact, and outcome-focused evaluation rather than activity monitoring.

Timezones are planned around. Overlap windows are used strategically. The burden of inconvenient meeting times is shared.

Well-being is actively monitored. Workload is a real conversation, not a formality. Boundaries around working hours are modeled from the top.

None of this is complicated. Most of it is just consistent application of basic leadership principles in a context where the defaults do not work and you have to build the structure yourself.

That is what good remote engineering management looks like: not doing more, but building the right systems so that the team can do their best work without you holding everything together by force of personality.

Start with the thing that is most obviously broken in your current setup, build one system around it, and then move to the next. You do not need to implement everything at once. You just need to keep building.

If you found this useful, the next article in this series covers the transition from individual contributor to engineering manager; specifically how to let go of the work you love doing and build the habits that actually make you effective in a leadership role.

The first thing I want to tell you is that the discomfort you are feeling is not a warning sign. It is the job working correctly.

When you step into an engineering leadership role for the first time, almost everything that made you good at your previous job stops being directly useful. The skills that got you promoted – the ability to reason through a hard problem, write clean code, ship features reliably – those are no longer your primary tools. Your primary tool is now other people. And working with people is a fundamentally different craft from working with code.

That transition is jarring for most engineers, and it catches a lot of first-time managers completely off guard. Not because they are not capable, but because nobody told them what to actually expect.

So let’s talk about what to actually expect.

What Just Changed (And What Didn’t)

Here is the uncomfortable truth about moving into engineering management: your output is no longer measurable in the same way. When you were an IC, you could point to pull requests, shipped features, performance improvements, and bug fixes. There was a feedback loop. You could tell, more or less, whether you were doing well.

As a manager, your output is the team’s output. That is a much longer feedback loop, and it is far less legible. Did that one-on-one conversation you had three weeks ago lead to someone shipping better work today? Did the process change you introduced last month reduce friction on the team? You often will not know for weeks or months, and sometimes you will never know for certain.

This drives a lot of new engineering managers straight back to coding. Not because they need to – they just miss the feedback loop. They miss feeling productive in a way they can measure.

I am not going to tell you to never write code again. That is both unrealistic and unnecessary. But I will tell you that if you are writing code to avoid the uncomfortable parts of your new job, you are solving the wrong problem. The goal is to get comfortable with the longer feedback loop, not to escape it.

What did not change: you still need to understand the technical work deeply. Not at a “I could have written that PR myself” level necessarily, but at a “I can have a real conversation about this tradeoff and push back when something feels wrong” level. Technical credibility matters enormously in engineering leadership. You earn it by staying engaged with the work, asking good questions in code review, understanding the architecture, and remembering what it actually feels like to ship something under pressure.

The shift is not from technical to non-technical. It is from doing the technical work to enabling others to do it better than you could alone.

Letting Go of the Keyboard

This is the hardest part for most engineers, so let’s spend some real time here.

You are probably good at coding. Maybe very good. You have built up years of intuition about how to structure a problem, where the edge cases are, what a clean solution looks like. And now you are sitting in meetings watching your team write code that you could see the issues in immediately, and you have to… not fix it yourself.

That feeling does not go away quickly. But here is a reframe that helped me when I was working through this with engineers making the management transition: your job is no longer to write the best code. Your job is to build a team that writes better code than you ever could alone.

That reframe changes the question from “why am I not fixing this?” to “how do I help this person grow into someone who catches this themselves?” Those are very different questions, and they lead to very different actions.

Concretely, letting go of the keyboard looks like this:

Reviewing instead of rewriting. When you see a PR that you would have written differently, write a thoughtful review comment explaining your reasoning instead of just fixing it yourself. This is slower in the short term and faster in the long term. The engineer learns something, and next time they will write it better without needing your input.

Asking questions instead of giving answers. When an engineer comes to you with a problem, resist the instinct to solve it. Ask questions instead. “What have you already tried? What do you think is causing it? What would you do if you had to take a guess?” This feels slower and sometimes frustrating (for both of you) but it builds problem-solving independence. Engineers who get answers handed to them stay dependent. Engineers who are coached through problems become more capable.

Delegating things that make you nervous. The tasks you are most reluctant to hand off are usually the tasks you most need to hand off. That is often because they are high-visibility or technically complex, and you do not fully trust someone else to handle them yet. That distrust is sometimes justified, but often it is just anxiety. Start with lower-stakes delegation and build up. Give people room to own something end-to-end, and resist the urge to hover.

Managing Your Time as a New Leader

Your calendar is about to become your main work surface. That is a real adjustment.

As an IC, your calendar was mostly something that interrupted your work. Meetings were overhead. Focus time was the actual job. As a manager, your calendar is the actual job. One-on-ones, team syncs, cross-team coordination, recruiting conversations, performance check-ins – these are not the things getting in the way of your work. They are your work.

That said, there are a few time management habits that separate effective new managers from ones who burn out quickly.

Protect one or two focus blocks per week. Even as a manager, you need uninterrupted time to think. Not to write code, but to read documents carefully, think through an organizational problem, draft a strategy, or just catch up on what is actually happening in the codebase. If you let your entire week become back-to-back meetings, you will always be reacting and never thinking ahead.

Time-box your one-on-ones and run them consistently. One-on-ones are your single most important management tool, and new managers often either skip them when things get busy or run them without a clear purpose. Do not do either. Hold them every week, keep them to thirty or forty-five minutes, and treat them as the engineer’s time – not your status update meeting. Come with a light agenda, ask real questions, and actually listen. “How is everything going” is not a real question. “What is the most frustrating part of your work right now” is a real question.

Batch your administrative overhead. Expense reports, tool approvals, interview scheduling, performance review paperwork – this stuff has to get done but it does not require your best brain hours. Block thirty minutes at the end of a Tuesday for admin. Do not let it colonize your mornings.

Learn to say no to your own instincts. You will constantly want to jump in, add yourself to things, take on tasks you see falling through cracks. Some of that instinct is good leadership. A lot of it is just difficulty letting go of the doing. Before you take something on, ask: is this actually mine to handle, or am I adding myself here because it feels more comfortable than delegating?

The Expectation Gap

Here is something that does not get talked about enough in the IC-to-manager transition: the expectations of the people around you shift before you have had time to develop new competencies.

Your team expects you to have answers about career growth, organizational direction, and team priorities. Your manager expects you to surface problems early, have a handle on your team’s capacity, and represent your team in cross-functional conversations. Your peers across other teams expect you to be reliable and aligned.

And you are three weeks in, still figuring out what one-on-ones are supposed to accomplish.

The gap between those expectations and your current capabilities is real, and trying to close it by pretending you have everything figured out is a trap. The better approach is direct acknowledgment. “I am still finding my footing on this” is a perfectly acceptable thing to say in your first ninety days. It signals self-awareness and honesty, which are actually two of the more important traits in a new manager.

What you should not do is fake competence in areas where you are genuinely unsure, because that creates expectations you will then have to maintain. Better to set honest expectations early and then exceed them than to promise things you cannot deliver.

The areas where most new engineering managers genuinely struggle in the first six months:

Performance conversations. Giving critical feedback to someone whose work is not meeting expectations is one of the hardest things managers have to do, and most ICs have almost no experience with it. The instinct is to soften, delay, or avoid. Push against that instinct. Clear, specific, compassionate feedback delivered early prevents the much harder conversation later.

Prioritization and saying no. As an IC, someone else decided what was on the roadmap. Now you are involved in those decisions, and you have to be willing to push back when the team is being asked to do too much. That requires a different kind of conversation than most engineers have had to have before.

Navigating organizational ambiguity. Companies are messier from the inside than they look from a distance. There are competing priorities, unclear ownership, processes that exist for reasons nobody remembers, and political dynamics that take time to understand. Your job is to make sense of enough of that context to give your team clarity, even when you do not have full clarity yourself.

Skills to Start Building Right Now

If you are a new engineering manager reading this, here are the things I would focus on in your first ninety days:

Get really good at listening. Not just hearing – actually listening. When an engineer tells you something in a one-on-one, are you thinking about your response while they are talking, or are you actually taking in what they are saying? Active listening is a learnable skill and it is foundational to everything else in this job.

Start a manager journal. Write down what you tried, what happened, and what you would do differently. Management has a long feedback loop, which means you need to be deliberate about capturing the signal. A weekly reflection of ten minutes is enough. Over six months it becomes an invaluable record of your own development.

Find a peer group or mentor. The transition into management is isolating in a specific way: the problems you are dealing with are not something you can fully discuss with your direct reports, and they are different from the technical problems your IC friends are working on. Finding other people at a similar stage – through communities, networks, or formal mentoring – gives you a sounding board that is genuinely hard to replace. Some of the most useful conversations I have had with engineers making the management transition were not about tactics at all. They were just about realizing that the disorientation they were feeling was normal.

Learn the business context. As an IC, you needed to understand the technical context of your work. As a manager, you need to understand the business context too. Why does your company prioritize certain things? How does your team’s work connect to revenue, users, or strategic goals? The better you understand that, the better equipped you are to make good prioritization decisions and advocate for your team effectively.

Be patient with yourself. This is easier to say than to live, but it is true. Most engineers who become managers underestimate how long the real learning curve is. You will make mistakes in your first year. Some of them will be painful. That is not a sign that you made the wrong choice. It is a sign that you are actually doing the job.

The Part Nobody Tells You

There is something genuinely satisfying about watching someone you have been working with closely ship something significant, get promoted, or work through a hard problem they could not have solved six months ago. It is a different kind of satisfaction from shipping something yourself – less immediate, more durable.

You will not feel it right away. In the first few months, it mostly just feels hard. But it shows up eventually, and when it does, it tends to make the tradeoffs feel worth it.

The move from IC to manager is one of the more significant professional transitions you will make. It is not for everyone, and there is no shame in deciding the role is not the right fit. But if you go in with clear eyes about what is actually changing, build the right habits early, and resist the pull of the familiar, you have a real shot at being genuinely good at it.

Start there.

Next up in this series: Hiring Engineers – a practical playbook for interview structure, technical assessment, and building an onboarding process that actually works.

The framework sits at the center of most data pipelines, and interviewers test for it accordingly. Whether you are preparing as a candidate or building a question bank as a hiring manager, this set of 100 Spark interview questions for data engineer roles covers every seniority level and the most common technical topics.

Preparing for the Spark Interview

Both recruiters and technical specialists benefit from a structured question bank. It speeds up candidate screening and helps engineers close knowledge gaps before the real conversation.

How Sample Spark Interview Questions Help Recruiters

A structured set of Apache Spark interview questions data engineer candidates face lets recruiters screen technical depth without engineering support. Compare answers side by side and move qualified profiles forward faster.

How Sample Spark Interview Questions Help Data Engineers

Working through data engineer Spark interview questions before the call exposes blind spots in shuffle internals and pipeline design. Pair this list with Spark developer interview questions if your work extends into application-level code.

List of 100 Spark Interview Questions for Data Engineers

Each group opens with five bad-and-good answer pairs. These Spark data engineer interview questions cover fundamentals through production edge cases.

Spark Interview Questions for Junior Data Engineer

Start with the fundamentals every entry-level candidate should explain clearly.

1: What is Apache Spark and why do data engineers use it?

Bad Answer: It is a programming language for big data.

Good Answer: It is a distributed compute engine that processes data in memory across a cluster. It handles batch ETL, streaming, and SQL in one framework.

2: What is the difference between a transformation and an action?

Bad Answer: Transformations run the code, actions save it.

Good Answer: Transformations build a plan but do not execute until an action like count or write triggers the job.

3: What are RDDs and when would you encounter them?

Bad Answer: RDDs are old and never used anymore.

Good Answer: An RDD is a low-level distributed collection with no schema. You see them with custom partitioning or legacy code.

4: How does the framework handle fault tolerance?

Bad Answer: It copies all data to every node.

Good Answer: Each DataFrame tracks its lineage. If a partition is lost, the engine recomputes it from the lineage.

5: What is the difference between a DataFrame and a Dataset?

Bad Answer: A Dataset is a DataFrame with another name.

Good Answer: A DataFrame is Dataset[Row], untyped. A Dataset carries compile-time type safety in Scala and Java.

6: What is a SparkSession?

The unified entry point from 2.0 replacing SparkContext, SQLContext, and HiveContext.

7: What does lazy evaluation mean?

Transformations build a plan but nothing runs until an action is called.

8: What is a partition?

A chunk of data processed by one task. Partition count controls parallelism.

9: What file formats are supported natively?

Parquet, ORC, JSON, CSV, Avro, and text. Parquet and ORC support predicate pushdown.

10: What is the driver program?

It converts code into a DAG, negotiates resources, sends tasks, and collects results.

11: What does an executor do?

Runs tasks on a worker node, stores cached data, and reports back to the driver.

12: What is a DAG?

A Directed Acyclic Graph of transformations split into stages at shuffle boundaries.

13: How do you read a CSV into a DataFrame?

Spark.read.option(‘header’,’true’).csv(‘path’). Add inferSchema or supply a StructType.

14: What is the difference between cache and persist?

cache() uses MEMORY_ONLY. persist() accepts a storage level for memory, disk, or both.

15: What is schema inference?

The engine samples the data to detect column types. Slower than an explicit schema.

16: What is a narrow transformation?

Each input partition maps to one output partition. No shuffle. Examples: map, filter.

17: What is a wide transformation?

Needs data from multiple partitions, triggering a shuffle. Examples: groupBy, join.

18: What does repartition do?

Redistributes data through a full shuffle. Can increase or decrease partition count.

19: How is coalesce different from repartition?

Coalesce merges partitions without a shuffle. It can only reduce, not increase the count.

20: What is a broadcast variable?

A read-only copy sent once to every executor to avoid shipping data with each task.

21: What is an accumulator?

A write-only variable tasks add to. The driver reads the total after the job.

22: What does Spark-submit do?

Packages the application and submits it to the cluster manager with config flags.

23: What is the default storage level for cache?

MEMORY_ONLY for RDDs, MEMORY_AND_DISK for Datasets.

24: How do you write to Parquet?

df.write.parquet(‘path’). Add mode and partitionBy as needed.

25: What is the web UI for?

Shows jobs, stages, tasks, storage, and SQL plans to spot skew and slow stages.

Spark Interview Questions for Middle Data Engineer

These questions target mid-level engineers who run production jobs and handle tuning on a regular basis.

1: How does the Catalyst optimizer work?

Bad Answer: It caches queries automatically.

Good Answer: Catalyst parses a logical plan, applies predicate pushdown and constant folding, then generates a physical plan compiled by Tungsten.

2: What is the difference between groupByKey and reduceByKey?

Bad Answer: They are the same.

Good Answer: groupByKey shuffles all values before aggregation, risking OOM. reduceByKey reduces locally first, cutting shuffle volume.

3: How would you handle data skew in a join?

Bad Answer: Add more memory.

Good Answer: Salt the skewed key, replicate the other side to match, and join on the salted key. AQE does this automatically when enabled.

4: What is Adaptive Query Execution?

Bad Answer: It just means faster queries.

Good Answer: AQE re-optimizes the plan at runtime using shuffle statistics. It coalesces small partitions and switches join strategies.

5: How does partitioning affect shuffle performance?

Bad Answer: It does not matter with enough memory.

Good Answer: Poor partitioning forces extra shuffles. Partitioning by the join key eliminates the shuffle for that operation.

6: What is predicate pushdown?

Pushes filters to the source so only matching rows are read.

7: How does broadcast join work?

The small table is sent to every executor. Each joins locally without shuffling the large side.

8: What is whole-stage code generation?

Tungsten compiles operators in a stage into one Java function, removing virtual calls.

9: How do you tune shuffle partitions?

Default is 200. Raise for large data, lower for small jobs. AQE auto-coalesces.

10: What is a sort-merge join?

Both sides shuffle by key and sort. Good for large-to-large joins but costly.

11: What is the Tungsten engine?

Manages memory off-heap, avoids GC, and generates bytecode at runtime.

12: Client vs. cluster deploy mode?

Client runs the driver locally. Cluster runs it inside the cluster. Cluster is standard for production.

13: How does dynamic allocation work?

Requests executors when tasks queue, releases idle ones. Needs the external shuffle service.

14: What serialization formats are supported?

Java (slow default), Kryo (faster), and Tungsten binary for DataFrames.

15: When do you use checkpointing?

To truncate deep lineage in iterative or streaming jobs by writing to reliable storage.

16: How do you monitor a running job?

Web UI for stage metrics. Prometheus or Graphite sinks for production alerting.

17: What is speculative execution?

A duplicate launches for slow tasks. Whichever finishes first wins.

18: How do you size executor memory and cores?

Four to five cores per executor. Memory at 75% of node RAM with OS headroom.

19: What is a shuffle spill?

Shuffle data exceeds memory and spills to disk, slowing the job.

20: What happens at a stage boundary?

Shuffle output is written to disk, then the next stage reads and sorts it by key.

21: How do you handle nulls in a pipeline?

na.fill, na.drop, or coalesce/when expressions. Define handling early.

22: What is column pruning?

The optimizer drops unused columns so only needed data is read from storage.

23: What is the cost-based optimizer?

Uses ANALYZE TABLE statistics to pick the cheapest join order and strategy.

24: How do you read from Kafka?

readStream.format(‘kafka’) with broker addresses and topic subscription.

25: Append vs. complete output mode?

Append writes new rows only. Complete rewrites the full result table.

Spark Interview Questions for Senior Data Engineer

This section covers Spark interview questions for experienced data engineer profiles and Spark data engineer technical interview questions on architecture, streaming, and cluster management.

1: How would you design a multi-hop lakehouse architecture?

Bad Answer: Write everything to one big table.

Good Answer: Bronze for raw data, silver for cleaning and dedup, gold for aggregates. Delta Lake or Iceberg adds ACID across layers.

2: What is the impact of GC on long-running jobs?

Bad Answer: GC pauses are too small to notice.

Good Answer: Long pauses stall tasks and bloat stage duration. Tune G1GC regions and keep objects off-heap through Tungsten.

3: How do you achieve exactly-once in Structured Streaming?

Bad Answer: The engine handles it automatically.

Good Answer: Checkpoint to durable storage and use an idempotent sink. The engine replays uncommitted batches after failure.

4: When would you pick RDDs over DataFrames in production?

Bad Answer: Always. RDDs give more control.

Good Answer: Only for custom partitioning, non-tabular data, or low-level APIs without a DataFrame equivalent.

5: How do you manage backpressure in streaming?

Bad Answer: Just add more executors.

Good Answer: Cap batch size with maxOffsetsPerTrigger. Monitor processing time vs trigger interval and scale before lag grows.

6: How does the external shuffle service work?

A separate process per node serves shuffle files so executors can leave without losing data.

7: V1 vs. V2 DataSource API?

V2 adds columnar reads, partition pushdown, transactional writes, and streaming.

8: How do you debug Container killed by YARN?

Memory plus overhead exceeds the container limit. Check broadcasts, UDFs, and concurrency.

9: How does the framework integrate with Hive?

SparkSession reads the metastore for schemas and partitions, replacing Hive’s engine with Catalyst.

10: How do you implement SCD Type 2 with Delta Lake?

MERGE on the business key. Close old versions, insert new ones in one transaction.

11: What is Z-ordering?

Interleaves column bits into one sort order so related values land in the same files.

12: How do you manage cluster concurrency?

YARN or K8s queues with per-team limits. Cap per-app resources to prevent starvation.

13: What problems do small files cause?

High metadata overhead, excess tasks, and namenode load. Compact before writing.

14: How do you migrate a MapReduce pipeline?

Mapper becomes filter/select/withColumn. Reducer becomes groupBy/agg. Validate on a sample.

15: How do you test data pipelines?

Unit-test transforms with static DataFrames. Integration-test writes on a local cluster.

16: Micro-batch vs. continuous processing?

Micro-batch is simpler with full Catalyst support. Continuous has lower latency but fewer ops.

17: How does the scheduler assign tasks?

By data locality: PROCESS_LOCAL first, then NODE_LOCAL, RACK_LOCAL, ANY.

18: What triggers a stage retry?

A FetchFailedException from a lost executor or missing shuffle file.

19: How do you profile executor memory?

Attach async-profiler via extraJavaOptions. Check Storage tab and GC logs.

20: Local vs. distributed checkpointing?

Local stores on executor disk. Distributed writes to HDFS or S3, surviving executor loss.

21: How do you optimize multi-source reads?

Read sources in parallel, push filters, join after reducing row counts.

22: What causes serialization overhead?

Large closures, UDF arguments, and objects shipped between driver and executors.

23: How do you write a custom partitioner?

Extend Partitioner, override numPartitions and getPartition, pass to rdd.partitionBy.

24: What is bucket pruning?

Reads only matching bucket files when filtering or joining on the bucket column.

25: How do you handle schema evolution with Parquet?

mergeSchema on read. Nullable types for new columns. Delta Lake blocks incompatible changes.

Practice-Based Spark Questions for Data Engineer

These Spark data engineering interview questions focus on pipeline problems. Combine them with Spark scenario-based interview questions and answers for wider situational coverage.

1: A 10 TB join keeps failing with OOM. What do you do?

Bad Answer: Request more memory.

Good Answer: Check the UI for skew. Salt the hot key or use AQE. Try broadcast if one side fits after filters.

2: How do you build an incremental pipeline for new files only?

Bad Answer: Reprocess everything each time.

Good Answer: Track processed paths in a metadata table or use Streaming file source to pick up new files automatically.

3: Pipeline output has duplicates. How do you investigate?

Bad Answer: Add .distinct() and move on.

Good Answer: Check source duplication, many-to-many joins, and retry-caused double writes. Fix the root cause first.

4: How do you reduce file count in a partitioned Parquet table?

Bad Answer: Delete files manually.

Good Answer: Coalesce or repartition before writing. For Delta, run OPTIMIZE. Use maxRecordsPerFile to cap sizes.

5: Streaming throughput drops after an upstream schema change. What do you check?

Bad Answer: Roll back the schema change.

Good Answer: Verify deserialization works. Check for null-heavy columns causing GC pressure and checkpoint compatibility.

6: How do you orchestrate jobs in Airflow?

SparkSubmitOperator in a DAG. Parameterize by date, set retries and timeouts.

7: How do you validate data quality?

Assert row counts, null ratios, and key uniqueness per stage. Use Deequ or Great Expectations.

8: How do you roll back a bad Delta Lake write?

RESTORE TABLE to a previous version. Time travel for read-only access.

9: How do you handle late data with watermarks?

Set a watermark on event time. Records outside the window are dropped.

10: How do you break a monolithic job into modules?

One function per stage, intermediate writes to storage, Airflow for orchestration.

11: How do you estimate cluster sizing?

Profile on a sample, measure shuffle and peak memory, then scale linearly.

12: How do you migrate from YARN to Kubernetes?

Container image, –master k8s://, replace queues with namespace limits.

13: How do you manage config across environments?

Separate config files per environment. Pass the env flag at submit time.

14: How do you deduplicate by event time in streaming?

dropDuplicatesWithinWatermark on key and event-time column.

15: How do you set up production alerting?

Push metrics to Prometheus. Alert on batch duration, lag, and executor loss.

Tricky Spark Questions for Data Engineer

These questions test edge-case knowledge and tend to surface in senior-level rounds.

1: Why might a broadcast join fail even when the table seems small?

Bad Answer: It would not fail if the table is small.

Good Answer: Pre-filter size estimates can be wrong. After runtime filters actual data may exceed the threshold or driver memory.

2: What happens when you call collect on a huge DataFrame?

Bad Answer: The engine handles it efficiently.

Good Answer: All partitions ship to the driver as one array. Driver OOM if the result is too large. Use take or write instead.

3: How can a UDF silently produce wrong results?

Bad Answer: UDFs are correct if the code is correct.

Good Answer: Retries on exceptions create duplicates. Null handling differs from SQL. Type mismatches corrupt data silently.

4: Why does adding partitions sometimes slow a job?

Bad Answer: More partitions are always better.

Good Answer: Each partition adds scheduling and serialization cost. Tiny partitions make launch time dominate compute.

5: Why can reusing a cached DataFrame hurt performance?

Bad Answer: Caching is always beneficial.

Good Answer: Cached data pins memory, shrinking shuffle buffers. Large or rarely used caches cause spills that slow other stages.

6: What is the risk of coalesce(1)?

One task writes everything. No parallelism, possible OOM.

7: How does partition pruning interact with bucketing?

Partition pruning cuts hive partitions. Bucket pruning cuts bucket files. Both apply together.

8: repartition(n).write vs. maxRecordsPerFile?

Repartition shuffles. maxRecordsPerFile splits output without a shuffle.

9: Driver OOM vs. executor OOM?

Driver holds DAG, broadcasts, collected results. Executor holds cache and shuffle buffers. Driver OOM kills the app.

10: Why might SQL and RDD code return different results?

Catalyst applies null-safe comparisons and predicate ordering that RDD operations skip.

Tips for Spark Interview Preparation for Data Engineers

A few habits sharpen preparation beyond memorizing answers to Apache Spark interview questions data engineering teams commonly ask.

• Run a local cluster, submit a pipeline, and study the web UI stage graphs.
• Build a small ETL that reads, transforms, and writes Parquet. Break it with skewed data and fix it.
• Practice explaining DAG stages on a whiteboard.
• Compare plans with explain(true) and learn physical plan operators.
• Time yourself. Two minutes per answer is a solid pace.

Conclusion

These 100 questions cover everything from core distributed-computing concepts to debugging and streaming edge cases. Use them to map your weak spots, rehearse under time pressure, and build the kind of technical fluency that stands out in a live interview.

Find the Right Scala Talent with Our Specialized Platform

Post a Job

The post 100 Spark Interview Questions for Data Engineer first appeared on Jobs With Scala.

Optimizing performance in React applications is essential to create smooth, fast, and responsive user experiences. React’s flexibility and component-based architecture allow developers to implement various strategies to enhance performance.

In this article, we’ll explore key techniques such as dynamic imports, list virtualization, and others to help you boost your app’s speed.

Dynamic Imports: Code Splitting for Faster Load Times

One of the most effective performance optimizations in React is code splitting, which allows you to load JavaScript files only when needed. This can drastically reduce the initial load time, especially for large applications. React’s dynamic imports enable you to load components and modules on demand, which means users only download the code they need.

With dynamic imports, you can split your application into smaller chunks and only load parts of the app when required. For example, you might load a component for a specific route only when the user navigates to it, rather than including it in the initial bundle.

By implementing dynamic imports, you improve your app’s initial loading time and reduce the amount of unnecessary code loaded upfront.

import React, { Suspense } from 'react';

const LazyLoadedComponent = React.lazy(() => import('./LazyLoadedComponent'));

function App() {
  return (
    <div>
      <Suspense fallback={<div>Loading...</div>}>
        <LazyLoadedComponent />
      </Suspense>
    </div>
  );
}

List Virtualization: Rendering Only What’s Visible

In applications that display large lists of data, rendering every single item at once can be a performance bottleneck. List virtualization solves this problem by only rendering the items visible on the screen at any given time, drastically reducing the number of DOM elements.

Libraries like react-window help implement list virtualization by rendering a small subset of items based on the visible area, improving rendering speed and reducing memory usage.

Memoize Expensive Calculations with useMemo

If you perform heavy calculations or transformations (like sorting, filtering, or mapping large arrays), memoize the result with useMemo so it’s only recalculated when the inputs change.

const sortedList = useMemo(() => {
  return items.sort((a, b) => a.name.localeCompare(b.name));
}, [items]);

This avoids unnecessary recalculations on every render.

Preload Critical Resources

Use <link rel="preload"> or tools like React Helmet to preload fonts, critical images, or even API data you know the user will need immediately. This can significantly reduce LCP and improve perceived performance.

Smart Data Fetching with React Query

Managing server state efficiently is key to performance. Instead of manually fetching and storing data in state variables, libraries like React Query handle caching, background refetching, and request deduplication out of the box.

React Query can:

• Cache requests and avoid redundant fetches.
• Refetch data in the background without blocking the UI.
• Handle loading, error, and success states automatically.

Optimize Images and Assets

Large or unoptimized images are one of the most common causes of slow-loading apps. To avoid this:

• Use modern formats like WebP or AVIF
• Compress images before loading
• Use lazy loading (<img loading="lazy" />) for below-the-fold images
• Leverage responsive images with srcSet for different screen sizes

<picture>
  <source media="(min-width:768px)" srcset="flowers-tablet.jpg">
  <img src="ppizza.jpg" alt="Pozza" style="width:auto;">
</picture>

Keep Components Small and Focused

Smaller components are easier to optimize, understand, and reuse. If a component does too many things, it’s more likely to re-render often. Split logic-heavy components into smaller children where only relevant parts update.

Analyze and Monitor with Performance Tools

• Lighthouse (via Chrome DevTools or CI)
• Web Vitals extension
• React Profiler
• Sentry/Datadog for performance monitoring

Measure, benchmark, and monitor performance to track regressions over time.

Use useEffect and useCallback Wisely

React’s useEffect and useCallback hooks are powerful, but they can easily become sources of performance issues if not used thoughtfully.

With useEffect, it’s important to limit side effects to what’s necessary and avoid including unstable dependencies (like inline functions or objects) unless absolutely required. This reduces the risk of unnecessary re-renders or even infinite loops. Also, always clean up subscriptions or event listeners inside effects to prevent memory leaks.

On the other hand, useCallback helps memoize functions so they retain a stable reference between renders, useful when passing callbacks to memoized child components or when including functions as dependencies in useEffect. However, overusing useCallback can clutter your code and offer little benefit if the function isn’t passed as a prop or used in dependency arrays.

In short, use useEffect to manage external interactions and side effects with minimal, clear logic, and reserve useCallback for stabilizing functions that really need it. Used wisely, these hooks help you keep your components performant and predictable.

Conclusions

Performance optimization in React is not just about writing faster code. It’s about making thoughtful architectural choices, leveraging the right tools, and reducing unnecessary work for the browser.

Focus on rendering only what’s needed, when it’s needed, and keep an eye on the network, re-renders, and the structure of your components. With these strategies in place, you’ll create React apps that are not only functional but delightfully fast.